Publications

2024

1. Towards Holistic Human Evaluation of Automatic Text Simplification

   Luisa Carrer, Andreas Säuberli, Martin Kappus, and 1 more author

   In Proceedings of the 4th Workshop on Human Evaluation of NLP Systems (HumEval), 2024

   Abs PDF

   Text simplification refers to the process of rewording within a single language, moving from a standard form into an easy-to-understand one. Easy Language and Plain Language are two examples of simplified varieties aimed at improving readability and understanding for a wide-ranging audience. Human evaluation of automatic text simplification (ATS) is usually done by employing experts or crowdworkers to rate the generated texts. However, this approach does not include the target readers of simplified texts and does not reflect actual comprehensibility. In this paper, we explore different ways of measuring the quality of automatically simplified texts. We conducted a multi-faceted evaluation study involving end users, post-editors, and Easy Language experts, and applied a variety of qualitative and quantitative methods. We found differences in the perception and actual comprehension of the texts by different user groups. In addition, qualitative surveys and behavioral observations proved to be essential in interpreting the results. Finally, we discuss the advantages of comprehensive evaluations of ATS and provide recommendations for future work.

2. Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models

   Andreas Säuberli, and Simon Clematide

   In Proceedings of the 3rd Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI), 2024

   Abs PDF Code Data

   Reading comprehension tests are used in a variety of applications, reaching from education to assessing the comprehensibility of simplified texts. However, creating such tests manually and ensuring their quality is difficult and time-consuming. In this paper, we explore how large language models (LLMs) can be used to generate and evaluate multiple-choice reading comprehension items. To this end, we compiled a dataset of German reading comprehension items and developed a new protocol for human and automatic evaluation, including a metric we call \emphtext informativity, which is based on guessability and answerability. We then used this protocol and the dataset to evaluate the quality of items generated by Llama 2 and GPT-4. Our results suggest that both models are capable of generating items of acceptable quality in a zero-shot setting, but GPT-4 clearly outperforms Llama 2. We also show that LLMs can be used for automatic evaluation by eliciting item reponses from them. In this scenario, evaluation results with GPT-4 were the most similar to human annotators. Overall, zero-shot generation with LLMs is a promising approach for generating and evaluating reading comprehension test items, in particular for languages without large amounts of available data.

3. Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual Disabilities

   Andreas Säuberli, Franz Holzknecht, Patrick Haller, and 4 more authors

   In Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024

   Abs DOI PDF

   Text simplification refers to the process of increasing the comprehensibility of texts. Automatic text simplification models are most commonly evaluated by experts or crowdworkers instead of the primary target groups of simplified texts, such as persons with intellectual disabilities. We conducted an evaluation study of text comprehensibility including participants with and without intellectual disabilities reading unsimplified, automatically and manually simplified German texts on a tablet computer. We explored four different approaches to measuring comprehensibility: multiple-choice comprehension questions, perceived difficulty ratings, response time, and reading speed. The results revealed significant variations in these measurements, depending on the reader group and whether the text had undergone automatic or manual simplification. For the target group of persons with intellectual disabilities, comprehension questions emerged as the most reliable measure, while analyzing reading speed provided valuable insights into participants’ reading behavior.

2023

1. Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile application

   Andreas Säuberli, Silvia Hansen-Schirra, Franz Holzknecht, and 4 more authors

   Frontiers in Communication, 2023

   Abs DOI PDF Code

   In research on Easy Language and automatic text simplification, it is imperative to evaluate the comprehensibility of texts by presenting them to target users and assessing their level of comprehension. Target readers often include people with intellectual or other disabilities, which renders conducting experiments more challenging and time-consuming. In this paper, we introduce Okra, an openly available touchscreen-based application to facilitate the inclusion of people with disabilities in studies of text comprehensibility. It implements several tasks related to reading comprehension and cognition and its user interface is optimized toward the needs of people with intellectual disabilities (IDs). We used Okra in a study with 16 participants with IDs and tested for effects of modality, comparing reading comprehension results when texts are read on paper and on an iPad. We found no evidence of such an effect on multiple-choice comprehension questions and perceived difficulty ratings, but reading time was significantly longer on paper. We also tested the feasibility of assessing cognitive skill levels of participants in Okra, and discuss problems and possible improvements. We will continue development of the application and use it for evaluating automatic text simplification systems in the future.

2022

1. Eye-tracking based classification of Mandarin Chinese readers with and without dyslexia using neural sequence models

   Patrick Haller, Andreas Säuberli, Sarah Kiener, and 3 more authors

   In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), 2022

   Abs DOI PDF Code

   Eye movements are known to reflect cognitive processes in reading, and psychological reading research has shown that eye gaze patterns differ between readers with and without dyslexia. In recent years, researchers have attempted to classify readers with dyslexia based on their eye movements using Support Vector Machines (SVMs). However, these approaches (i) are based on highly aggregated features averaged over all words read by a participant, thus disregarding the sequential nature of the eye movements, and (ii) do not consider the linguistic stimulus and its interaction with the reader’s eye movements. In the present work, we propose two simple sequence models that process eye movements on the entire stimulus without the need of aggregating features across the sentence. Additionally, we incorporate the linguistic stimulus into the model in two ways—contextualized word embeddings and manually extracted linguistic features. The models are evaluated on a Mandarin Chinese dataset containing eye movements from children with and without dyslexia. Our results show that (i) even for a logographic script such as Chinese, sequence models are able to classify dyslexia on eye gaze sequences, reaching state-of-the-art performance, and (ii) incorporating the linguistic stimulus does not help to improve classification performance.

2. LauSAn at eRisk 2022: Simply and Effectively Optimizing Text Classification for Early Detection

   Andreas Säuberli, Sooyeon Cho, and Laura Stahlhut

   Working Notes of CLEF, 2022

   Abs PDF

   The goal of early detection tasks at eRisk is to classify social media users as early as possible, based on streams of posts written by those users. We present two simple strategies of adapting standard text classification models in order to optimize them for early detection: concatenating the posts in different ways during training and inference, and continuously moving the decision boundary at inference time. We applied these approaches to two different text classification architectures based on pre-trained language models in eRisk 2022’s Task 2 (early detection of depression), and were able to reach top 5 placements in all time-sensitive evaluation metrics. A systematic post-submission ablation study confirmed that both strategies were effective at optimizing for early detection.

3. Automatic Text Simplification for German

   Sarah Ebling, Alessia Battisti, Marek Kostrzewa, and 4 more authors

   Frontiers in Communication, 2022

   Abs DOI PDF

   The article at hand aggregates the work of our group in automatic processing of simplified German. We present four parallel (standard/simplified German) corpora compiled and curated by our group. We report on the creation of a gold standard of sentence alignments from the four sources for evaluating automatic alignment methods on this gold standard. We show that one of the alignment methods performs best on the majority of the data sources. We used two of our corpora as a basis for the first sentence-based neural machine translation (NMT) approach toward automatic simplification of German. In follow-up work, we extended our model to render it capable of explicitly operating on multiple levels of simplified German. We show that using source-side language level labels improves performance with regard to two evaluation metrics commonly applied to measuring the quality of automatic text simplification.

2021

1. Measuring Text Comprehension for People with Reading Difficulties Using a Mobile Application

   Andreas Säuberli

   In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, 2021

   Abs DOI

   Measuring text comprehension is crucial for evaluating the accessibility of texts in Easy Language. However, accurate and objective comprehension tests tend to be expensive, time-consuming and sometimes difficult to implement for target groups of Easy Language. In this paper, we propose using computer-based testing with touchscreen devices as a means to simplify and accelerate data collection using comprehension tests, and to facilitate experiments with less proficient readers. We demonstrate this by designing and implementing a mobile touchscreen application and validating its effectiveness in an experiment with people with intellectual disabilities. The results suggest that there is no difference in terms of task difficulty between measuring comprehension using the mobile application and a traditional paper-and-pencil test. Moreover, reading times appear to be faster in the application than on paper.

2. A New Dataset and Efficient Baselines for Document-level Text Simplification in German

   Annette Rios, Nicolas Spring, Tannon Kew, and 4 more authors

   In Proceedings of the Third Workshop on New Frontiers in Summarization, 2021

   Abs DOI PDF Code

   The task of document-level text simplification is very similar to summarization with the additional difficulty of reducing complexity. We introduce a newly collected data set of German texts, collected from the Swiss news magazine 20 Minuten (‘20 Minutes’) that consists of full articles paired with simplified summaries. Furthermore, we present experiments on automatic text simplification with the pretrained multilingual mBART and a modified version thereof that is more memory-friendly, using both our new data set and existing simplification corpora. Our modifications of mBART let us train at a lower memory cost without much loss in performance, in fact, the smaller mBART even improves over the standard model in a setting with multiple simplification levels.

2020

1. Benchmarking Data-driven Automatic Text Simplification for German

   Andreas Säuberli, Sarah Ebling, and Martin Volk

   In Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI), 2020

   Abs PDF

   Automatic text simplification is an active research area, and there are first systems for English, Spanish, Portuguese, and Italian. For German, no data-driven approach exists to this date, due to a lack of training data. In this paper, we present a parallel corpus of news items in German with corresponding simplifications on two complexity levels. The simplifications have been produced according to a well-documented set of guidelines. We then report on experiments in automatically simplifying the German news items using state-of-the-art neural machine translation techniques. We demonstrate that despite our small parallel corpus, our neural models were able to learn essential features of simplified language, such as lexical substitutions, deletion of less relevant words and phrases, and sentence shortening.

2. A Corpus for Automatic Readability Assessment and Text Simplification of German

   Alessia Battisti, Dominik Pfütze, Andreas Säuberli, and 2 more authors

   In Proceedings of the Twelfth Language Resources and Evaluation Conference, 2020

   Abs PDF

   In this paper, we present a corpus for use in automatic readability assessment and automatic text simplification for German, the first of its kind for this language. The corpus is compiled from web sources and consists of parallel as well as monolingual-only (simplified German) data amounting to approximately 6,200 documents (nearly 211,000 sentences). As a unique feature, the corpus contains information on text structure (e.g., paragraphs, lines), typography (e.g., font type, font style), and images (content, position, and dimensions). While the importance of considering such information in machine learning tasks involving simplified language, such as readability assessment, has repeatedly been stressed in the literature, we provide empirical evidence for its benefit. We also demonstrate the added value of leveraging monolingual-only data for automatic text simplification via machine translation through applying back-translation, a data augmentation technique.