Advanced Neural Model for Spanish Spell-Checking

Introduction

Spelling correction is a fundamental task in natural language processing (NLP), with significant implications for various applications, including text composition, writing assistance, and automatic editing. In the literature, ¹⁾ suggests four classic misspelled word categories, and ² analyze around 76K misspellings found in real-life texts produced by humans (Table I). Many errors tend to be insertions, deletions, substitutions, and transpositions of letters. ² also found that many misspelling errors in Spanish are due to

1. Omissions, mainly of accents or of one character

2. The use of lowercase instead of uppercase at the beginning of a proper noun

3. The addition of a letter

4. The substitution of one character

5. The transposition of a letter

Table I: Common spelling errors by humans 2

Table I shows that a full 51 % of the misspellings found are omissions of a diacritic on a vowel. A notable example of this is the prevalence of word errors in millions of tweets and other massive datasets. In the context of Spanish, a language characterized by rich regional and orthographic variability, achieving accurate and efficient spelling correction poses unique challenges.

Various research efforts have focused on improving the accuracy and correctness of written content, using methods such as linguistic analysis, extraction, annotation, and correction based on dictionaries or statistical analysis. These methods have been applied to a range of tasks, including lemmatization, morphosyntactic labeling, syntactic analysis, sentiment analysis, and conceptual annotation 3 ^{) (}6 .

On the other hand, large language models (LLMs) have revolutionized various NLP tasks, including spelling correction. Models such as GPT-4 and BERT have shown significant advances in identifying and correcting errors using deep neural networks. These models leverage large volumes of data to learn complex and contextual patterns, enhancing their ability to correct spelling errors in various languages, including Spanish. However, despite these advancements, the current approaches still face significant challenges, particularly in adapting to different contexts and reducing errors in more complex situations within Spanish texts.

Recent studies have indicated that, although modern neural models achieve competitive results, there are still limitations in adapting to regional variations and specific contexts in Spanish. Less common spelling errors and linguistic peculiarities can reduce the effectiveness of current systems. For example, research has shown that LLMs can improve about precision and coverage if trained with more diverse and specific data ^{7)- (11).}

One innovative approach to addressing these challenges is to computationally simulate the brain’s process for recognizing and correcting misspelled words. This paper introduces a novel neural model designed to enhance the accuracy and efficiency of spelling correction systems for Spanish texts. The proposed model employs a deep hierarchical framework with specialized recognition neurons and formal strategies to identify and correct misspelled words. This approach not only enhances the accuracy of error detection and correction but also provides a more nuanced understanding of language processing. The new neural model for detecting misspelled words simulates the process that the human brain, specifically the neocortex, follows by reusing information. This efficient approach is based on the lexical and syntactic analysis of words in Spanish.

Like the brain’s neocortex, a neural network uses multiple layers, with each layer handling progressively more complex aspects, starting from character-level analysis to higher-order syntactic structures. Additionally, specialized modules, akin to cortical columns in the brain, are employed to detect and process various spelling errors, from simple typos to more contextually inaccurate words. The model also leverages contextual cues, analyzing surrounding words and the overall context to enhance the accuracy of its corrections, thereby surpassing basic pattern recognition. Moreover, it integrates a mechanism for reusing previously acquired knowledge, analogous to the brain’s ability to apply past experiences to novel situations. This adaptability allows the model to accommodate different writing styles and error patterns, making it more versatile and robust.

This new deep neural model emulates certain aspects of human brain function ^(12-14): memory is organized as a hierarchy of patterns, and, if only part of a pattern is perceived (through sight, hearing, or smell), it can still be recognized. The model also assumes several hypotheses regarding the structure of the biological neocortex, such as the uniformity of the basic neocortical structure, known as the cortical column. In addition, pattern recognition neurons are constantly interconnected.

Our model improves upon the state of the art by recognizing input patterns through a process of self-association in a hierarchy of patterns. It enables the decomposition of the pattern recognition problem into simpler patterns, allowing to analyze input patterns regardless of their level of complexity or their nature (a line, a word, a sentence, a paragraph, etc.). In addition, the neural model is easily parallelizable, as its calculations, defined in the theorems, are simpler and distributed across a hierarchy. Moreover, computational cost can be improved with respect to other approaches, with more efficient use of memory due to a single abstract data structure that can be instanced by various text patterns. Finally, the model is adaptable since it can learn both the possible changes in pattern descriptors (such as their importance weights) and new neurons (components) if the atomic patterns are known, which is very useful in the context of a language for self-learning of words and idiomatic sentences.

In synthesis, the deep neural architecture is characterized by recognition neuron hierarchies, which increase the levels of complexity, i.e., the pattern recognition neurons that constitute the lowest level levels (or X _j−1 ), will always be less complex than the neurons of the upper level (or X _j , for j = 1,...,m). This is an innovative approach with respect to classical recognition models.

The primary contribution of this study lies in an innovative system for recognizing and correcting misspelled words in Spanish texts. This system

• demonstrates high levels of recursion and uniformity;

• operates on a self-associative principle within a hierarchical pattern framework;

• exhibits adaptability, given its ability to assimilate new patterns (words); and

• efficiently analyzes extensive Spanish texts containing words of varying structural complexity.

This method draws inspiration from cognitive neuroscience, particularly from the functioning of the neocortex, which plays a crucial role in complex language processing tasks. The pattern recognition theory of mind (PRTM) describes the basic algorithm of the neocortex, which is characterized by a hierarchy of patterns, uniformity in its basic structure, and continuous connectivity between its pattern recognition modules ¹⁴.

This work is organized as follows. Section 2 presents the research trends and related work, providing a comprehensive overview of the existing approaches to spelling correction systems for Spanish texts. Section 3 details the methodology employed, including the formal definition of a neuron within the neuron module and a description of the activation functions used within the neural network model. This section outlines the architecture of the neural model and its optimization, explaining how the model has been tailored for efficient performance in spelling correction. Section 4 presents the experiments conducted, describing the experimental setup, datasets, and performance evaluations for the proposed model in comparison with other existing methods. Finally, Section 5 states the conclusions and proposes directions for future work, highlighting the contributions of this study and discussing potential improvements and extensions of the model.

Research trends and related work

The field of NLP has witnessed remarkable advancements in recent years, particularly in the domains of spell-checking, grammatical error correction (GEC), and overall text improvement. These developments have been largely driven by the advent of sophisticated neural network architectures and the increasing availability of large-scale datasets. Transformer-based models have emerged as the dominant paradigm, demonstrating unprecedented performance across various NLP tasks. ¹⁵ introduced GECToR, a transformer-based approach for GEC that achieves state-of-the-art results in several benchmarks. Building upon this work, in ¹⁶, the GECToR architecture was adapted for the Russian language (RuGECToR). Furthermore, ¹⁷ proposed a unified pre-training approach for monolingual and multilingual GEC. Their method leverages massive amounts of synthetic data and multi-task learning, achieving new state-of-the-art results in standard GEC benchmarks in English and extending well to other languages. ¹⁸ expanded the evaluation of GEC systems beyond essays from non-native learners by introducing CWEB, a new benchmark for GEC that comprises website texts written by English speakers of varying proficiency levels. This work highlights how the lower error density in these domains poses significant challenges for current GEC systems, demonstrating the need for models that better generalize across different topics and genres.

The field of spell-checking and GEC has therefore seen substantial advancements in recent years. ¹⁹ provided an extensive overview of the field of GEC, which addresses the automatic detection and correction of various errors in texts, including grammatical, orthographic, and semantic discrepancies. Over the past decade, significant advancements have been driven by five major shared tasks, catalyzing the evolution from rule-based methods and statistical classifiers to advanced neural machine translation systems. The integration of deep learning techniques, particularly transformer-based models, has led to significant improvements regarding accuracy and capability. Furthermore, the focus has expanded beyond simple error correction to include context-aware corrections, fluency enhancement, and multilingual support. The emphasis on efficiency and real-time performance demonstrates the field’s maturation and its readiness for widespread practical application. As these technologies continue to evolve, we can expect to see more sophisticated, efficient, and widely applicable text improvement systems that not only correct errors but also enhance the overall quality and fluency of written communication across multiple languages and domains.

Some particularly interesting works related to the proposed model include STILUS ²⁰, which distinguishes four types of errors: grammatical, orthographic, semantic, and style. In the case of orthographic revision, STILUS corrects words in three stages: the generation of alternatives to the incorrect word, the weighting of alternatives, and the arrangement of alternatives. Another system is ArText, a prototype automatic help system for writing texts in Spanish in specialized domains ²¹. The system has three modules: the first module handles aspects of structure, content, and phraseology; the second focuses on format and linguistic revision; and the last allows users to linguistically revise their texts. XUXEN ²²⁾ is a spell checker/corrector defined based on two morphological formalisms. It uses a highly inflected standardized language with a broad relationship between nouns and verbs as well as a lexicon that contains approximately 50 000 items across different grammatical categories.

On the other hand, ²³ proposed a spelling and grammar checker algorithm for texts where mistakes are not detected through tagging and parsing, but rather through statistical analysis, comparing combinations of two words used in the text against a corpus of one hundred million words. In ²⁴, the JHU FLuency-Extended GUG corpus (JFLEG) aimed at developing and evaluating GEC was presented. It represents a broad range of language proficiency levels and uses holistic fluency edits to not only correct grammatical errors but also to make the original text sound more natural.

Additionally, ³ presented a general approach to various NLP applications, such as translation and recognition, using modern techniques like deep learning. Finally, to date, there are no works based on a hierarchical approach to pattern recognition (in our case, words) as a fundamental mechanism for reusing text patterns, which is an efficient way to recognize many words, some of which may be complex. The next sections detail our proposal.

Methodology

This section presents the mathematical formalization of the neural model, including the neuron or recognition module, its activation functions, and its recursive architecture, as well as the computational algorithm that integrates the entire model.

Formal definition of a neuron within the neuron module

A pattern recognition module (or neuron) is formally defined as a 3-tuple. The Γρ notation is used to represent the module that recognizes the ρ pattern (ρ: shapes, letters, words, etc.). Γρ =< E,U,S _o >, where E is an array composed of the 2-tuple E =<S, C> (Table I), S =<Signal, State>is an array that represents the set of signals that make up the pattern recognized by Γ and their corresponding states, and C is an array that encodes information about the pattern, as defined by the 3-tuple C =<D, V, W>, where D represents the descriptors of Γ, V is the domain vector for each D (i.e., the possible values of each descriptor), and W is the importance weight of the descriptor for the ρ pattern. Additionally, U denotes the thresholds vector used by the module (Γ) to recognize its respective pattern.

Table II shows one artificial neuron, i.e., a neocortical pattern recognition module according to the PRTM theory.

Table II: Neuron: matrix E = <S, C>

In the neural model, each neuron/module or pattern recognition module can recognize and observe every aspect of the input pattern s(), as well as the way in which the different parts of the data in the input pattern may or may not relate to each other.

There are two types of thresholds: ∆U ₁ is the threshold for recognition by key signals, and ∆U ₂ is the threshold for recognition by partial or total mapping. ∆U ₁ should be stricter than ∆U ₂, given that the process of recognition by key signals utilizes only a few signals. Finally, each module produces a recognition or petition signal (S _o ) towards lower levels. As petition, S _o becomes the input signal s() for the pattern-matching neurons of the lower levels. When there is a recognition signal, it is diffused to its higher attainable levels, in order to modify the state of the signal to "trueïn the patterns of said levels.

Thus, a pattern is represented as a set of lower-level sub-patterns that conform to it (n descriptors), and it also serves as the sub-pattern of a higher-level pattern. The value of n depends on the pattern to be recognized (the descriptors of the pattern). The values of W are normalized (0,1), and δU ₁ or ∆U ₂ are thresholds that must be overcome to recognize the pattern. These values are defined in a supervised manner, according to the domain of application. In the context of text analysis, the main patterns to be recognized (ρ) are letters, words, special signs, and numbers.

Architecture of the neural model

This section describes the neural network model instanced for the specific case of text analysis. Particularly, the hierarchical system in Fig. 1 represents the recursive and iterative process for the recognition and correction of words. Each layer in the hierarchy is an interpretation space X _i from i = 1 to m. X ₁ is the level of recognition for atomic patterns ( e.g., letters or letterforms), and X _m is the level of recognition for complex patterns (e.g., words and compound words). Each level is composed of γ _ji recognition modules (for j = 1,2,3... neurons at level i). X _ij is the pattern recognized by module j at level i. The function of each recognition module is to recognize its corresponding pattern.

Figure 1: Recursive architecture of the neural model

Fig. 1 shows the architecture of the neural network (i.e., the hierarchical pattern recognition system). The multiple hidden layers are the recognition spaces of level i or the levels of complex pattern recognition (X _i ). This is how the neural model can find extremely complex patterns via bottom-up or top-down approaches.

Here, X _i is the recognition space of the 1-level, Γ _i1 : is the j-neuron of the 1-level, which recognizes p _i1 , i.e., the pattern recognized by the neuron j at level 1.

This architecture is marked by hierarchies of recognition neurons that increase in complexity. Lower-level neurons are less complex than those at higher levels, which makes this approach innovative compared to traditional recognition models ^26,27.

Two analytical processes characterize the algorithm. The first process, called top-down, recognizes the input pattern through decomposition (Fig. 1). The top-level module invokes the recognition neurons of lower-level constituents, and these recursively perform the same function. The second process, called bottom-up, is used for recognizing atomic patterns. Here, the output signal of the recognized pattern is sent to the neurons that make up the top-level representation. These top-level neurons will be activated (or not) depending on whether they exceed a recognition threshold.

Optimized computational neural model

The algorithm works as follows. It receives the input text (y = s(): sentences, words (s) (line 2). Then, it decomposes this input into its sub-patterns, which are stored in L (lines 4-5). On line 7, it determines the level of the hierarchy where the recognition of the input pattern (y) should start and end. Afterwards, L recognition requests for (y) are created (line 12). The algorithm enters a loop that iterates over each pattern in L (line 13). If the recognition is successful at the current level (line 14), the information is updated using the updateSubpatterns function (line 18). If the input pattern is composed of lower-level signals or has recognized atomic signals, a bottom-up process is conducted; otherwise, a top-down procedure is followed (line 16). If the recognition fails at the lowest level (level == 1) and no pattern has been recognized, the algorithm ends (lines 22-24). This process continues until all patterns in L have been processed or their recognition has failed at the lowest level. Finally, the function returns the result of the hierarchical recognition process (line 28).

Figure 2: Algorithm for the implementation of the recursive neural recognition model

Experiments

This section presents the results and an evaluation of the neural model in comparison with other works, for which three additional systems were selected: STILUS, SpanishChecker, and Microsoft word. These three systems were chosen because they are tools designed to analyze spelling and find basic grammar and style mistakes in Spanish texts. All of them show errors automatically. For these test scenarios, several paragraphs were artificially made, with misspelled words in Spanish. In these tests, the inputs, i.e., the paragraphs, were introduced into the neural model as an array of words, while, in the other systems, they were entered as a plain text file. The standard precision (P) and coverage (C) metrics, as well as the F-measure (F) ²⁸, were used during these experiments to compare the performance of the evaluated systems.

The first row of Table III presents the results of applying the neural model to a paragraph containing 205 words, out of which more than 5 % contained errors due to added letters. The neural model effectively detects all real errors caused by the addition of letters, as in ’entrdada’, ’tellevisión’, and ’teléfonoo’. In contrast, SpanishChecker generates three false positives corresponding to the names ’Pantoja’, ’Goya’, and ’Chabelita’. STILUS produces similar results, with two false positives (’Pantoja’ and ’Goya’), while Microsoft Word also flags ’Chabelita’ and ’Goya’ as errors. The neural model recognizes ’Chabelita’, ’Goya’, and ’Pantoja’ as well-written words.

Table III: Results of applying different systems to various types of spelling errors

The second row corresponds to a paragraph containing 203 words, with 5 % of the words exhibiting omitted letters. According to these results, the neural model effectively detects all errors related to omitted letters, just like Microsoft Word and STILUS. In contrast, SpanishChecker generates four false positives corresponding to the words ’expresidente’, ’Aznar’, ’populismo’, and ’patrióticamente’ (with the latter being two false positives) while omitting ’Congres’ (resulting in one false negative).

The third row uses a paragraph that contains 203 words, with 20 % of the words exhibiting cognitive errors. The neural model detects 14 errors, and Microsoft Word identifies a false negative: ’Confexionar’. Although SpanishChecker identifies the errors, many of its correction recommendations are syntactically and semantically far from the correct word. For example, for the word ’Confexionar’, SpanishChecker suggests: {conexionar, confinar, confinara, confinare, confinará, confinaré, confinaría, confina}; for ’vioseguridad’, it suggests: {vio seguridad, vio-seguridad, seguridad, esguardad, resguardad, seguridades, etc.}; and, for ’varato’, it suggests: {va rato, va-rato, verato, grato, arto, parto, urato, verte, varío, barato, etc.}. Among its false positives are ’Cofeccionistas’ {confusionistas, confesionistas, confusionista, cancionista, etc.} and ’Cucuteños’ {cicutinas, cacereños, cicateros, etc.}. The two false negatives are ’Crusada’ and ’Biernes’. Finally, STILUS generates the false positives ’Oquendo’ and ’Cucuteña’.

The fourth row corresponds a paragraph that contains 5 % of words with two exchanged letters. In this case, all systems incorrectly detect more errors. The neural model incorrectly detects only three errors, while the other systems detect significantly more.

In the fifth row, the results obtained by the different systems are shown for a paragraph with 5 % of words containing errors related to digits or special characters. In this case, only the neural model correctly detects the errors. The other systems either fail to detect the errors or incorrectly identify more (as is the case with STILUS).

Finally, the last row of the dataset used in the experiment consists of 32 texts from the digital version of El País, dated May 17 ^th , 2001, which were copied into a Word document. Approximately 9000 words were analyzed, with 14 spelling errors. SpanishChecker identifies the fewest errors ¹¹, while Microsoft Word and STILUS are more effective, detecting 13 and 12 errors, respectively. Our approach detects all errors but incorrectly identifies nine additional ones. The other approaches incorrectly detect many more. The main issue with these systems is that they flag errors arising from the inclusion of digits or special characters, which is not applicable in this case, resulting in many false positives. Overall, their results are significantly lower than those achieved by our system.

Conclusions and future works

The proposed model aims to advance the state of the art in spelling correction for Spanish - and potentially other languages - by offering a solution informed by biological processes and optimized for practical application. This innovative approach could lead to significant improvements in automated text correction systems, benefiting users across various linguistic contexts. The neural model detects a wider range of orthographic errors and generates fewer false positives, although it requires a supervised definition of the weights assigned to the variables used for recognition.

This work presents a new neural model for detecting misspelled words, closely mirroring how the human brain (specifically the neocortex) addresses this problem. The model reuses information to propose an efficient approach based on the lexical analysis of word syntax in Spanish.

This system demonstrates high levels of recursion and uniformity, operating on a self- associative principle within a hierarchical pattern framework. It showcases adaptability by assimilating new patterns (words) and efficiently analyzes extensive Spanish texts containing words of varying structural complexity.

A comparative evaluation indicated that the precision and coverage of the neural model are competitive with those of other spell-checkers. In the experiments, the system outperformed three other tools; it achieved an F-measure of 83 %, surpassing the other spell-checkers’ 73 %. Our neural model can detect a broader variety of orthographic errors while producing fewer false positives. As future work, the architecture of the neural model should be extended to incorporate unsupervised learning mechanisms, allowing it to improve its performance by learning new words. Additionally, it must be adapted to handle other languages. The model could also be developed to simultaneously correct texts written in both English and Spanish, which would be valuable in translation tasks. To this effect, the model needs to be expanded with more recognition neurons in different languages, enhancing its lexical basis.

Finally, a comparison with other approaches in the domain of NLP is not presented herein, as our primary focus is on the spell-checking (auto-correction) problem. While there are several NLP approaches for machine translation, cognitive dialogue systems, sentiment analysis, text classification, and text summarization - utilizing techniques from natural language understanding and generation present in state-of-the-art NLP - a future research direction could involve analyzing the application of our approach in these contexts for comparison with existing techniques. By leveraging the capabilities of LLMs like GPT-3.5, researchers can further refine and extend these systems, leading to more accurate, efficient, and sophisticated solutions for spelling correction in Spanish and beyond