Van Huyssteen, Gerhard B., E. Roald Eiselen, and Martin J. Puttkammer. 2004. “Re-evaluating evaluation metrics for spelling checker evaluations.” Proceedings of First Workshop on International Proofing Tools and Language Technologies. Patras: University of Patras.:91-99.
Van Huyssteen, Eiselen & Puttkammer 2004
English: evaluation, human language technology, proofing tools, spelling checker
Afrikaans: evaluering, mensetaaltegnologie, skryfhulpmiddels, speltoetser
English: In the increasingly competitive proofing tools market; it is becoming ever more important to find evaluation methods and metrics that provide stable; invariable measurements. This article focuses on the evaluation metrics currently used in spelling checker evaluation. We commence with a brief explication of some presuppositions relevant to the evaluation of spelling checkers. We then give a detailed description of metrics currently used in various evaluations of proofing tools; discussing and demonstrating their shortcomings. Subsequently; we illustrate that; by adjusting some of these metrics; by incorporating additional information (such as the percentage of errors in the text; and a measurement of the suggestion adequacy); and by adapting the evaluation methodology; a single metric can be designed that evaluates the overall linguistic performance of spelling checkers effectively. In conclusion; we suggest how these metrics can be further improved in future research; in order to open up the possibility to set a single benchmark that can be used for any spelling checker; irrespective of the language being evaluated; or the nature of the text used for the evaluation.
Afrikaans:
In: English
On: Afrikaans