FEMTI report: characteristics of a translation task

FEMTI (Framework for the Evaluation of Machine Translation) is a resource that helps MT evaluators define contextual evaluation plans, and it consists of two interrelated classifications:

1. One of them lists the possible characteristics of the contexts of use that can be applied to MT systems.
2. The other one lists the possible characteristics of an MT system, along with the metrics that were proposed to measure them.

FEMTI is used by many evaluators every day, so it proposes some characteristics of high quality that are relevant to that context, using its embedded knowledge base. Evaluators can modify this set of characteristics and choose an evaluation metric for each of them by browsing the second classification. Then, evaluators can print the evaluation plan and execute the it.

According to the FEMTI report, a translation task has specific characteristics. This refers to the information flow intended for the output, from the point of view of the agent (human or otherwise) who receives the translation. As J.C. Sager noted for Machine Translation systems, two types of use [are] to be considered: (a) the un-edited output; (b) the edited output. The output may be acceptable for either use or both and the evaluation should determine this. In the case of edited output the cost of revision, editing etc. has to be established and compared with the cost of manual translation. Since the type of use is related to the type of text, these types have to be established and taken into account.

In the other hand, In Toward Finely Differentiated Evaluation Metrics for Machine Translation, Hovy suggests dividing all the possible translation tasks into three main groups. He saidthat in order to make the taxonomization of features useful to people who do not already know much about MT and do not wish to become experts in evaluation, it is important to articulate its layers and choices in terms they can intuitively understand. This part of the present evaluation taxonomy describes three principal types of use in such a way that users can identify the particular type of work they want to have done, while developers can define in strict terms what their MT system can do.

Apart from that, the main characteristics of a translation task according to the FEMTI report are the following:

• Assimilation: its purpose is to monitor a relatively large volume of texts produced by people outside the organization, in several languages. This characteristic includes document routing / sorting, information extraction or summarization and a search process.
• Dissemination: it pretends to deliver to others a translation of documents produced inside the organization. Internal dissemination (routine internal dissemination and experimental internal dissemination) and external dissemination-publication (single client external dissemination and multi-client external dissemination) are included in here.
• Communication: its aim is to support multi-turn dialogues between people who speak different languages. The translation quality must be high enough for good and fluent conversation. The ultimate purpose of dissemination is to deliver to others a translation of documents produced inside the organization. Synchronous or interactive communication and synchronous or delayed communication must be mentioned at this point too.

Sources:

• Eduard Hovy, Margaret King & Andrei Popescu-Belis. (2003). FEMTI – Framework for the Evaluation of Machine Translation in ISLE. Retrieved April 9, 2008, from http://www.issco.unige.ch:8080/cocoon/femti/st-home.html