The Analysis of a GPT-based Sepedi Text Generation Model

Abstract

Text generation is defined as a component of  natural language processing that makes use of  computational linguistics techniques to produce  text that cannot be distinguished from human written text. This study aims to develop and  analyse a Generative Pre-Trained Transformer 2  (GPT-2) language model to generate Sepedi  phrases. The under-resourced Sepedi language is  regarded as a disjunctive language. The Sepedi  language orthographic representation presents  challenges and has limited resources. The GPT-2  transformer requires large datasets, as well as  state-of-the-art computational resources. The  unstructured National Centre for Human Language Technology (NCHLT) Sepedi text  dataset was used. The text generation model  developed with the small dataset managed to get  the lowest loss value of 2.36. The output text  generated using this model produces a text that is  syntactically correct with instances of  grammatical errors. The model performed better  than previously developed Sepedi text generation  models by using transformer-based technique.

https://doi.org/10.59200/ICONIC.2022.016