Fine-tuning AI for Low-Resource Languages

“...you begin anyway and you see it through no matter what.”

To Kill A Mockingbird, Harper Lee, 1960

With the ever-advancing development of large language models, the combined knowledge of humanity is now more accessible than ever before.

Yet the vast majority of world languages, the so-called low-resource (LR) languages, have little-to-no data available for training, placing their speakers at risk of being sidelined from the benefits of artificial intelligence.

As the capstone of my postgraduate studies in computer science at UCL, I had the unique opportunity to investigate this technological inequality. With a multidisciplinary academic background in both linguistics and computer science, I found myself strategically poised to explore something that was as fascinating to me as it was valuable to others. Here, before me, was a chance to make the world just that little bit better.

This blog covers my experiences developing Kestrel, a family of fine-tuned local language models for Mauritian Creole built on top of IBM’s Granite 4.1 decoder-only architecture. From tackling resource scarcity to grappling with days of overseeing fine-tuning on my own hardware, this thesis battled me and I won.

With no prior experience in AI development, an unrivalled unfamiliarity with Mauritian Creole, and no guarantee that this model would even work under such constraints, I took the first step down a path I only wish I had discovered sooner.

Short on time? Skip to the results here.

Introduction

To understand this project, we should first analyse the context in which these models were developed.

Minority and Low-Resource Languages

The world’s languages are often classified as either minority or majority, loosely defined by the proportion of speakers they have within a population in a given region or country. Catalan (as a rather personal example) is a minority language of Spain, yet it may also be considered a majority language in both the country’s autonomous region of Catalonia, as well as the neighbouring state of Andorra.

The speakership of a language by no means determines the complexity of developing a bespoke language model. For AI machine translation it is far more relevant to focus on a secondary classification – that of a language’s available written resources – instead. Languages can be divided into the high-resource (HR) languages, ones for which considerable written corpora exist, and the low-resource (LR) languages, ones for which language data is challengingly scarce.

Low resource availability and low speakership are not mutually exclusive. A language may be a linguistic minority but not low-resource (MiHR), such as with Catalan, which boasts an abundance of language material and is the language of government and education in Catalonia. Alternatively, a language may be neither a linguistic minority nor low-resource (MaHR), into which fall English, Spanish, and all major world languages. A language may be both a linguistic minority and low-resource (MiLR), as is the case with any of the Okinawan languages of the Japonic language family, which suffer from both a low digital representation (Eberhard et al. 2025), but also a low speaker population as Japanese continues to supplant the languages in day-to-day life. Finally, a language may be low-resource, but not a linguistic minority (MaLR). This is the case with Mauritian Creole.

Mauritian Creole Data Scarcity

Mauritian Creole (endonymically, Kreol Morisien) is the lingua franca of Mauritius, where it is spoken by over one million people. It is a predominantly oral language, with English and French assuming the role of administration and media (Eberhard et al., 2025) and Mauritian Creole being used in day-to-day life. As a consequence of the historical written and social dominance of English and French, the digital presence of Mauritian Creole is still, to this day, extremely limited.

Modern transformer-based large language model architectures rely on unfathomably vast quantities of data to produce predictive models capable of outputting natural language. For the strongest world economies this is not an issue – English and Chinese both revel in a near-limitless supply of language data (granted, one might challenge how limitless ‘limitless’ truly is) – but Mauritian Creole is not as fortunate. Nearly three decades into the 21st Century and the spectre of colonialism still rears its ugly head: historical stigmatisation has suppressed the written uptake of Mauritian Creole to such an extent that the development of equivalently-powerful models is just not possible.

Let us not forget that this isn’t even an issue of not having grammars and dictionaries for the language; Mauritian Creole has both. What it lacks is language-in-context, a crucial ingredient in the machine learning recipe from which machine learning can capture the patterns of language in much the same way as a child first learning to speak.

Thankfully, researchers far more experienced than I have been steadily advancing on a solution.

Transfer Learning for LR Languages

How can you make a large language model without the language? The answer lies in transfer learning.

Advances in fine-tuning – the process of refining an existing base model with new data to output a specialised new model – have shown that the linguistic capabilities of large language models can act as a stepping stone to develop LR language models in spite of data limitations. By capturing the underlying patterns of human grammar structures, fine-tuned models can be created for previously unseen languages with much smaller quantities of data required.

From the perspective of a language learner, this does make sense. We are able to learn our second or third language far more quickly than our first, which takes years of near-constant input and immersion (and many more to build the refined vocabularies of adult speakers). Yet, also from the perspective of a language learner, this notion of transfer learning in humans reveals a major challenge for machine learning:

It is far easier to learn a language with similar grammar and semantics.

As English speakers, our easiest languages are those with the most similar set of features to latch on to. These are languages like Swedish, Spanish, or French – all close genealogical relatives – while our hardest languages tend to stray far away in terms of both features and vocabulary – think Japanese, Korean, or even Nahuatl (though I can’t claim to have ever delved into that conduit of precolombian mesoamerican history).

Over my near-decade of learning Japanese and Spanish, I can say with the utmost confidence that the former’s grammar and vocabulary causes the intensity of learning both languages to be not even remotely comparable. How do you teach an English speaker, whose language is rich in relativisers, that Japanese only expresses its wheres and thats not just implicitly, but ahead of the noun they modify? How can a speaker of Japanese master the articles of Spanish, so definitely indefinite and so singularly plural?

This is the unsung challenge of transfer learning. Languages are not ciphers; they are an inseparable pair of morphosyntax and semantopragmatics that commands resource. Resource that Mauritian Creole does not have.

In linguistics we have a term for this feature set of a language, its linguistic typology. We’ll get back to that later, but for now let’s make peace with knowing that though transfer learning for machine translation is theoretically straightforward, an understanding of linguistics tells us otherwise.

Prior Mauritian Creole Language Models

Contemporary efforts have been made to fine-tune language models for Mauritian Creole. Most prominently, Dabre and Sukhoo (2022) created the open-weight KreolMorisienMT models and released their datasets publicly so that others could further their work. The dataset is still limited, and heavily reliant on biblical texts and dictionary entries over more quotidian phrases, but it established that transfer learning for Mauritian Creole was possible.

The work has since been carried forward. Robinson et al. (2024) attempted to address the data limitations for Mauritian Creole and many other languages by fine-tuning a massive multilingual model on a combined dataset of 41 different Creoles. Results were mostly successful, but there was a lot of variance in quality across such a large set of languages. Rajcoomar (2025) returned to the narrower scope of developing a model specifically for Mauritian Creole, and developed a fine-tuned model under a pipeline that much more closely influenced my own (though it also involved considerable pretraining steps that I did not investigate myself). These models produced some of the strongest results.

Results

A total of six models (four fine-tuned, two base) were evaluated using the two leading machine translation metrics – BLEU (Papineni, 2002) and chrF++ (Popovic, 2017) – to provide a quantitative result for their improved translation quality. Figure 1 below illustrates the chrF++ results, while BLEU results can be found on the Kestrel repository on GitHub.

There are three significant facets that stand out from these results.

Low-Resource Transfer Learning is Effective

Firstly, transfer learning clearly works for low-resource machine learning. Versus the zero-shot base Granite 4.1 models, the fine-tuned models improved by up to 2.85x, and all of this was achieved on a single 12 GB consumer-grade GPU.

Low-Resource Machine Translation Exhibits Bidirectional Asymmetry

Secondly, improvements versus the base models were notably stronger for generating translations into Mauritian Creole, a concept known as bidirectional asymmetry. The Kestrel models were able to acquire a significant understanding of Mauritian Creole despite being limited to a dataset of only several thousand words and phrases.

The English Models Outperformed the French Models

Thirdly, the English models performed better than their French counterparts. Even though Mauritian Creole is French-lexified (it has a large part of its vocabulary derived directly from French), it is possible that the English-Mauritian Creole dataset, which contains fewer one-word dictionary entries and more sample sentences, contributed to this result.

What would I have done differently

In the closing thoughts of my thesis, I wrote that I had initially sought to extend the Kestrel models into a distributable command-line interface. Constrained by limited time, I did – fleetingly – endeavour to create a standalone executable, but I was unable to bring it under the 2 GB limit of the Llama.cpp tooling, and it was much more important to me to focus on the proof-of-concept of my paper than to refine this final distribution step. I forfeited flashiness for furthering theory.

I reason that, although the dedicated CLI would facilitate the usage of the models by third parties, ultimately what would benefit speakers the most would be to extend the research into Mauritian Creole machine translation in several key areas. It is these areas that I would focus on, were I to have had more time.

A Dedicated Tokeniser

Towards the end of my literature review, I became acutely aware of tokenisation and the benefits this could have on the quality of a fine-tuned translation model.

Tokenisation describes the process of breaking up an input text into segments of meaning. In linguistics, we might consider this equivalent to the domain of Morphology where each unit of meaning is known as a morpheme.

While ‘tokens’ can be more-or-less the same as morphemes, they by no means correspond identically. Tokenisation focuses on identifying the n-most frequent segments of text in a given corpus, and these are often single words. Semantically-speaking, this does make sense. For a word like ‘breakfast’, it is reasonable to tokenise it into a single part, rather than divide it into its constituent parts ‘break’ and ‘fast’, whence it derives. For a word like ‘ebike’, on the other hand, it could make sense to tokenise this either as a single component, or as two: ‘e-’ is commonly prepended to words to describe an electric variation, and an ebike is still a bike.

Unfortunately, I have not personally dived much into the connection between linguistic morphology and machine learning tokenisation, so I will leave this anecdote for another day. I would be interested to see if there is a relationship between tokenisation and free/bound morphemes… one for the linguists out there?

Adding a dedicated tokeniser for Mauritian Creole could help the model in two ways. Firstly, there is a chance that a dedicated tokeniser could greatly improve the final translation quality. A fine-tuned model with a custom tokeniser would have a stronger understanding of Mauritian Creole words, and even a subset of just 1,000 additional tokens would already cover the majority of words used in the language (thanks to Zipf’s Law).

Secondly, a dedicated tokeniser could significantly lower the cost of using a model, both in terms of compute time and compute amount. A tokeniser optimised for Mauritian Creole, English, and French could produce results that are stronger and more economical.

And you know what? I actually went and did this! I trained the Kestrel models on my own hardware, and I still have that hardware. So, following my thesis, I sat down for a week, tweaked my scripts to build and use a custom tokeniser, and supervised my GPU as it pulled twelve-hour shifts to train four new models. At some point I will share what happened.

Synthetic Data

Naturally, the biggest constraint with low-resource fine-tuning is the lack of usable datasets. Some prior research went around this by combining datasets from other Creole languages, but this didn’t feel like the right solution. I want to see a world where someday we can build models for low-resource languages without any neighbours at all, and this doesn’t step in that direction.

One intriguing option was to generate synthetic data. I do not know much of this process – that’s why I didn’t investigate it – but if large language models are truly dependent on data, then this might be the only way forward.

Training Other Models

Had I had more time, I would also have wanted to experiment with training other models and parameter sizes. Because the experiment only measured two sizes, I couldn’t pinpoint with statistical confidence at which point performance truly plateaus or drops off. Testing sub-3B architectures could reveal whether Mauritian Creole translation could run natively even on low-end smartphones.

Beyond scale, running the exact same QLoRA pipeline across alternative open-weight models, such as Llama, Qwen, or Gemma, could help isolate how much of Kestrel’s behaviour was dependent on Granite’s pre-training corpus.

Machine Translation and Linguistic Typology

More than any of the above, I would love to have had the opportunity to jump much deeper into the intersection between linguistics and machine learning.

Linguistic typology is the classification of languages according to their structural and grammatical features rather than their genealogy (the languages which they are related to). Back when I studied it at university, it was the perfect medium through which to learn about how the world’s languages take different approaches to communication – approaches including everything from subject alignment to tone and suprasegmentals to reduplication.

To me, this ties in perfectly with transfer learning. Surely fine-tuning a model to a language with a near-identical feature set to the source language would produce better results than fine-tuning for a language rich in typological outliers. The most immediate of these that springs to mind is reduplication, which I know to exist in Mauritian Creole, but not in English.

Closing Thoughts

Undergoing this project was personally transformative. I had never felt too energetic about AI: the loudest voices in the industry have seldom come without an agenda; personal projects I toiled countless hours over no longer seem significant; and the joy of problem-solving through code is a thing of the past.

But AI research was different. I wasn’t using AI because of the flashy label, but because it really was the best tool for the job.

What’s more, this thesis brought me face-to-face with computational linguistics. I mean, can you even imagine anything more apt for someone such as myself?

And I wasn’t alone. The guidance from my supervisors, professors Dean Mohamedally and Ajit Jaokar, and personal tutor professor Jens Krinke, really proved that all you need is someone cheering you on and you will find the resilience to do the rest. The merit of support must never be overlooked.

Named after the Mauritius Kestrel – a bird whose wild population once dwindled down to only four individuals in the wild – the Kestrel models showed me what happens when computer science and linguistics work in tandem. By combining modern decoder-only architectures and optimisation strategies, I was able to develop a capable Mauritian Creole translator that can be deployed on everyday devices.

Were you to have told me even five years ago that this is where I’d be today, I’d have been in disbelief.

And yet, despite my doubts in this four-year journey from modern languages into computer science, I decided to begin anyway, and see it through no matter what.