@misc{UcrelHex,
	title        = {{UCREL - Hex}; A shared, hybrid multiprocessor system},
    author       = {Vidler, John AND Rayson, Paul},
	abstract     = {Hex is a collection of GPU equipped hosts onto which single- multi-
                    or GPU-processor jobs can be executed hosted at Lancaster University,
                    UK as part of the School of Computing and Communications and the
                    UCREL group.},
	howpublished = {\url{https://github.com/UCREL/hex}},
    note         = {Accessed: 2024}
}

(2026) Creating a Hybrid Rule and Neural Network Based Semantic Tagger Using Silver Standard Data: The PyMUSAS Framework for Multilingual Semantic Annotation

Word Sense Disambiguation (WSD) has been widely evaluated using the semantic frameworks of WordNet, BabelNet, and the Oxford Dictionary of English. However, for the UCREL Semantic Analysis System (USAS) framework, no open extensive evaluation has been performed beyond lexical coverage or single language evaluation. In this work, we perform the largest semantic tagging evaluation of the rule based system that uses the lexical resources in the USAS framework covering five different languages using four existing datasets and one novel Chinese dataset. We create a new silver labelled English dataset, to overcome the lack of manually tagged training data, that we train and evaluate various mono and multilingual neural models in both mono and cross-lingual evaluation setups with comparisons to their rule based counterparts, and show how a rule based system can be enhanced with a neural network model. The resulting neural network models, including the data they were trained on, the Chinese evaluation dataset, and all of the code will be released as open resources.

Moore, Andrew and Rayson, Paul and Archer, Dawn and Czerniak, Tim and Knight, Dawn and Lal, Daisy Monika and 'O Donnchadha, Gear'oid and 'O Meachair, M'iche'al J. and Piao, Scott and U'i Dhonnchadha, Elaine and Vuorinen, Johanna and Yabo, Yan and Yang, Xiaobin

(2025) Geografia strachu, smutku, gniewu i rado'sci. Odkrywanie krajobraz'ow emocjonalnych w 'swiadestwach ocalalych z Zaglady.

Holocaust survivors testimonies offer a rich source on the personal experiences of thosewho witnessed and endured the horrors of the Nazi genocide of Jews and other persecutedgroups. These narratives contain references to the emotions felt while recalling memoriesof people, places, and events. Analyzing the spatiality of these human emotions helpsus understand how they connect to the places around them. We focus on fear, sadness,anger, and joy to examine the interplay of these emotional experiences across multipleindividuals, places, times, and circumstances. Understanding these complex emotionallandscapes, especially from large collections of textual data, requires a carefully designedtechnique that can apply both existing and new technologies effectively and efficiently.Therefore, in this work, we explore the possibility of extracting and analyzing theseemotions and their related geographies by employing a combination of natural languageprocessing methods, including large language models.

Ezeani, Ignatius and Rayson, Paul and Gregory, Ian and Cole, Tim and Steiner, Erik and Frank, Zephyr

(2025) Evaluating the LLM and NMT Models in Translating Low-Resourced Languages

Machine translation has significantly advanced due to the development of transformer architecture, which is utilised by many modern deep-learning models. However, low-resource languages, such as Lithuanian, still face challenges stemming from the limited availability of training data and resource constraints. This study examines the translation capabilities of Neural Machine Translation (NMT) models and Large Language Models (LLMs), comparing their performance in low-resource translation tasks. Furthermore, it assesses the impact of parameter scaling and fine-tuning on their effectiveness in enhancing model performance. The evaluation showed that while LLMs demonstrated proficiency in low-resource translation, their results were lower compared to NMT models, which remained consistent across smaller variants. However, as model size increased, the lead was not as prominent, correlating with automatic and human evaluations. The effort to enhance translation accuracy through fine-tuning proved to be an effective strategy, demonstrating improvements in vocabulary expansion and structural coherence in both architectures. These findings highlight the importance of diverse datasets, comprehensive model design, and fine-tuning techniques in addressing the challenges of low-resourced language translation. This project, one of the first studies to focus on the low-resourced Lithuanian language, aims to contribute to the broader discourse and ongoing efforts to enhance accessibility and inclusivity in Natural Language Processing.

Pucinskaite, Julita JP and Mitkov, Ruslan

(2025) LLM-based embedders for prior case retrieval

In common law systems, legal professionals such as lawyers and judges rely on precedents to build their arguments. As the volume of cases has grown massively over time, effectively retrieving prior cases has become essential. Prior case retrieval (PCR) is an information retrieval (IR) task that aims to automatically identify the most relevant court cases for a specific query from a large pool of potential candidates. While IR methods have seen several paradigm shifts over the last few years, the vast majority of PCR methods continue to rely on traditional IR methods, such as BM25. The state-of-the-art deep learning IR methods have not been successful in PCR due to two key challenges: i. Lengthy legal text limitation; when using the powerful BERT-based transformer models, there is a limit of input text lengths, which inevitably requires to shorten the input via truncation or division with a loss of legal context information. ii. Lack of legal training data; due to data privacy concerns, available PCR datasets are often limited in size, making it difficult to train deep learning-based models effectively. In this research, we address these challenges by leveraging LLM-based text embedders in PCR. LLM-based embedders support longer input lengths, and since we use them in an unsupervised manner, they do not require training data, addressing both challenges simultaneously. In this paper, we evaluate state-of-the-art LLM-based text embedders in four PCR benchmark datasets and show that they outperform BM25 and supervised transformer-based models.

Premasiri, Damith and Ranasinghe, Tharindu and Mitkov, Ruslan

(2025) Complex concept-based readability estimation from arabic curriculum

This article presents an approach to readability estimation that focuses on conceptual rather than linguistic complexity, using the extensive SaudiTextBooks textbooks. We introduce DARES 2.0, an enhanced concept-based readability training dataset designed to estimate the readability of Saudi educational texts. Building on DARES 1.0, DARES 2.0 extends the scope of conceptual complexity by replacing repetitive concepts and manually revising the input features with unique terms and their surrounding contexts from the SaudiTextBooks, spanning grades 1 to 12. The refined DARES 2.0 is employed to fine-tune pre-trained transformer models, including XLM-R Base, mBERT, AraELECTRA, AraBERTv2, and CAMeLBERTmix. The findings suggest that both the dataset and experimental setup require further development to ensure a larger, higher-quality dataset and to support more extensive fine-tuning experiments, in addition to exploring transfer learning from other languages and enhancing the diversity and richness of Arabic concepts. These developments pave the way for further advancements in concept-based readability estimation in educational contexts in future work.

Almujaiwel, Sultan and Premasiri, Damith and Ranasinghe, Tharindu and El-Haj, Mo and Mitkov, Ruslan

(2025) Sinhala encoder-only language models and evaluation

Recently, language models (LMs) have produced excellent results in many natural language processing (NLP) tasks. However, their effectiveness is highly dependent on available pre-training resources, which is particularly challenging for low-resource languages such as Sinhala. Furthermore, the scarcity of benchmarks to evaluate LMs is also a major concern for low-resource languages. In this paper, we address these two challenges for Sinhala by (i) collecting the largest monolingual corpus for Sinhala, (ii) training multiple LMs on this corpus and (iii) compiling the first Sinhala NLP benchmark (SINHALA-GLUE) and evaluating LMs on it. We show that the Sinhala LMs trained in this paper outperform the popular multilingual LMs, such as XLM-R and existing Sinhala LMs in downstream NLP tasks. All the trained LMs are publicly available. We also make SINHALA-GLUE publicly available as a public leaderboard, and we hope that it will enable further advancements in developing and evaluating LMs for Sinhala.

Ranasinghe, Tharindu and Hettiarachchi, Hansi and Pathirana, Nadeesha Chathurangi Naradde Vidana and Premasiri, Damith and Uyangodage, Lasitha and Arachchige, Isuri Nanomi and Plum, Alistair and Rayson, Paul and Mitkov, Ruslan

(2025) Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development

Predicting semantic textual similarity (STS) is a complex and ongoing challenge in natural language processing (NLP). Over the years, researchers have developed a variety of supervised and unsupervised approaches to calculate STS automatically. Additionally, various benchmarks, which include STS datasets, have been established to consistently evaluate and compare these STS methods. However, they largely focus on high-resource languages, mixed with datasets annotated focusing on relatedness instead of similarity and containing automatically translated instances. Therefore, no dedicated benchmark for multilingual STS exists. To solve this gap, we introduce the Multilingual Semantic Textual Similarity Benchmark (MUSTS), which spans 13 languages, including low-resource languages. By evaluating more than 25 models on MUSTS, we establish the most comprehensive benchmark of multilingual STS methods. Our findings confirm that STS remains a challenging task, particularly for low-resource languages.

Ranasinghe, Tharindu and Hettiarachchi, Hansi and Orasan, Constantin and Mitkov, Ruslan

(2025) SENTimental-A Simple Multilingual Sentiment Annotation Tool

Here we present SENTimental, a simple and fast web-based, mobile-friendly tool for capturing sentiment annotations from participants and citizen scientist volunteers to create training and testing data for low-resource languages. In contrast to existing tools, we focus on assigning broad values to segments of text over specific tags for tokens or spans to build datasets for training and testing LLMs. The SENTimental interface minimises barriers to entry with a goal of maximising the time a user spends in a flow state whereby they are able to quickly and accurately rate each text fragment without being distracted by the complexity of the interface. Designed from the outset to handle multilingual representations, SENTimental allows for parallel corpus data to be presented to the user and switched between instantly for immediate comparison. As such this allows for users in any loaded languages to contribute to the data gathered, building up comparable rankings in a simple structured dataset for later processing.

Vidler, John and Rayson, Paul and Knight, Dawn

(2024) The Geography of 'Fear', 'Sadness', 'Anger' and 'Joy': Exploring the Emotional Landscapes in the Holocaust Survivors' Testimonies.

Holocaust survivors testimonies provide a rich source of evidence about the personal experiences of survivors who witnessed and endured horrors during the Nazi genocide of Jews and other persecuted groups. The narratives contain references to the emotions experienced when describing memories of people, places, and events. Analysing the spatiality of these human emotions enables us to understand how they are connected to the places around them. We focus on fear, sadness, anger, as well as joy to examine the interplay of these emotional experiences by multiple individuals at dierent places and times and in dierent circumstances. Understanding these complex emotional landscapes, especially from very large collections of textual data requires a carefully designed technique that can eectively and eciently apply existing and new technologies. In this work, therefore, we explore the possibility of extracting and analysing these emotions as well as their related geographies by applying a combination of natural language processing methods including large language models.

Ezeani, Ignatius and Rayson, Paul and Gregory, Ian N and Cole, Tim and Steiner, Erik and Frank, Zephyr

(2024) LiSAScore: Exploring Linear Sum Assignment on BertScore

Metrics play a crucial role in evaluating the performance of machine learning models. In the context of Natural Language Processing (NLP) tasks, such as text summarization and machine translation, Natural Language Generation (NLG) metrics such as Bleu and Rouge have been widely used. However, these metrics are based on n-gram matching and do not capture the semantic similarity between the generated and reference texts. To address this, BertScore has emerged as a popular evaluation metric that uses a pre-trained Large Language Model (LLM) to measure semantic similarity between two sentences. Unlike n-gram-based metrics, BertScore uses the contextual and semantic embeddings of words, allowing flexible semantic evaluation. We outline a number of hypotheticals in which the dependence of BertScore on token embedding cosine similarity may be exploited. The comparative distribution of BertScores on a set of reference - prediction pairs mean that results often scale differently with training to traditional metrics, which requires more expertise when interpreting results.

Mander, Stephen and Phillips, Jesse

(2024) Towards Generalized Offensive Language Identification

The prevalence of offensive content on the internet, encompassing hate speech and cyberbullying, is a pervasive issue worldwide. Consequently, it has garnered significant attention from the machine learning (ML) and natural language processing (NLP) communities. As a result, numerous systems have been developed to automatically identify potentially harmful content and mitigate its impact. These systems can follow two approaches; (1) Use publicly available models and application endpoints, including prompting large language models (LLMs) (2) Annotate datasets and train ML models on them. However, both approaches lack an understanding of how generalizable they are. Furthermore, the applicability of these systems is often questioned in off-domain and practical environments. This paper empirically evaluates the generalizability of offensive language detection models and datasets across a novel generalized benchmark. We answer three research questions on generalizability. Our findings will be useful in creating robust real-world offensive language detection systems.

Alphaeus Dmonte and Tejas Arya and Tharindu Ranasinghe and Marcos Zampieri