Modeling Second Language Acquisition with pre-trained neural language models

Álvaro J.Jiménez Palenzuela, Flavius Frasincar, Maria Mihaela Truşcǎ*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

2 Citations (Scopus)
54 Downloads (Pure)


Prediction of language mistakes is a task introduced by Duolingo as part of the Second Language Acquisition Modeling topic that aims to learn from the history of mistakes to improve the experience of language learners. Using transfer learning by means of pre-trained language models, we propose a framework that can learn the actual mistakes distribution according to which faraway words of a sentence have a higher chance to produce errors. To adapt the information provided by the pre-trained language models, more approaches based on feature extraction or fine-tuning were tried. However, according to our experiments, integrating these two options in a stack-and-finetune approach seems to be more appropriate for our task. Regarding the comparison of language models in terms of model distillation, we notice that distillation does not affect the effectiveness while significantly reducing the training time. We conclude that the model complexity should be adjusted to the specifics of the analyzed problem and the distillation is an efficient option for low complexity corpora without considerably affecting the overall performance.

Original languageEnglish
Article number117871
JournalExpert Systems with Applications
Publication statusPublished - 30 Nov 2022

Bibliographical note

Publisher Copyright:
© 2022 Elsevier Ltd


Dive into the research topics of 'Modeling Second Language Acquisition with pre-trained neural language models'. Together they form a unique fingerprint.

Cite this