"The real test of machine translation is whether or not it is effective in large-scale operations." [Vasconcellos, Bostad 1992] Therefore the evaluation of how MT worked at the Pan American Health Organization (PAHO) will provide evidence of whether MT can be used efficiently already.
Development of MT at PAHO was undertaken with two main purposes. First, to meet international translation demands more efficiently and second to propagate health information in the Latin American and Caribbean countries. Only a few of the translations are for information only. The majority of translations must be of a high quality and stand close inspection.
The first language combination introduced at PAHO was Spanish-English with the SPANAM system. In 1985, a more sophisticated system was used for translations from English to Spanish (ENGSPAN). Both systems are written in the programming language PL/1 and run on mainframe computers. Later, the SPANAM system was rebuilt using the more advanced architecture of ENGSPAN.
SPANAM and ENGSPAN have deeply coded dictionaries with 63,000 and 55,000 terms respectively. They are capable of handling a wide range of subjects: Medicine, agriculture, public health, management, law and computer science. They also have to cope with a variety of styles:
Journal articles and abstracts, textbooks, manuals (both for human health and for software), proposals for funding, reports of missions, contracts and agreements, minutes of meetings, business letters, diplomatic exchange, certificates, product specifications, supply lists, captions for displays and even promotional materials and film scripts.
[Vasconcellos, Bostad 1992, p. 59]
The PAHO environment is very special as the system is developed in-production. Suggestions for changes are best brought up by the people actually working with the system. They are able to tag the machine output for later review by a terminologist. They can also suggest general system improvements.
The computational linguists, for their part, conduct their research on production text and are constantly monitoring the output to ensure that the two systems are performing up to standard.
[Vasconcellos, Bostad 1992, p. 64]
In 1989, MT was the primary mode of translation at PAHO as it supported at least 60 % of all regular production for the two languages involved. Around 1990, the machine output averagead around 6,000 pages (1,5 million words) a year.
Machine translation was not introducted overnight. At first, a seperate unit was created at PAHO in direct competition to the existing human translation service. A certified translator was hired to work full-time on post-editing the translated texts. He/she also updated the dictionaries based on experience.
For the first five years, post-editing was done with no additional cost for the requesting office. The end consumer had the choice to use MT or not to use it. The new service was well accepted and the good word spread. MT had the advantage that the output was already machine readable which was not true for the results of human translation at this time. MT could also be used for informational purposes by limiting post-editing.
In succession to structural changes, an eleven-month experiment was undertaken in October 1987 to figure out whether MT at PAHO was cost-effective, fast in turnaround and could produce more than human translation within a year. MT ceased to be optional, all texts had to be submitted on diskette. The incoming work was checked to determine whether it was suitable for the computer.
The following characteristics were checked to determine whether to use MT or HT:
[Vasconcellos, Bostad 1992, p. 68]
It turned out that post-editing seems to be a special skill. It is difficult not to rewrite the complete machine output on the one hand but to meet the high standards of a translation on the other hand.
In order to come up to this standard, the post-editor must be certain of all technical terminology, capture every nuance, provide accurate interpretations for coherence [...], ensure idiomatic equivalence, maintain a uniformly appropriate register, build in cohesiveness where it is lacking, provide adequate 'staging' [...], and preserve the information structure [...]
[Vasconcellos, Bostad 1992, p. 69]
The experiment was a great success for machine translation. A survey was made to get feedback on the service. The clients asked did not know wether their text had been translated by human or by a machine with human post-processing. The quality of machine translated and post-edited texts was sometimes rated higher than the quality of human translated texts. The goal of 30 per cent savings over in-house human translation was surpassed.
Finally, the MT effort at PAHO can be regarded as an successful application of machine translation. It reduced costs and raised productivity without sacrifcing quality. But this is also the result of a well-planned process and a lot of work.
MT is now the principal mode of translation for these two language combinations [English <->Spanish] and is an integral part of the organization's translation and terminology service.
[Vasconcellos, Bostad 1992, p. 49]
The second application of MT I want to discuss here is possibly the most successful use of MT. It is the introduction of MT at Perkins Engines, Peterborough, England.
It is an example of what can be achieved when a system is introduced in a thoroughly planned and methodical way into a restricted domain environment to process controlled-language source texts.
[Newton 1992c, p. 46]
It was found that existing translations had minor semantic and stylistic differences and also omissions and introduced elements. Therefore, more control from Peterborough was desirable while maintaining the quality of the translations.
The Technical Publications Manager, Peter Pym, decided to take a look at using MT. Fortunately, he had a foundation to build on -- his department was already using controlled English known as Perkins Approved Clear English (PACE) for all documents.
A lot of problems of MT can be circumvente by restricting the source texts to a certain vocabulary (around 2,500 words for PACE), fixing the meaning of allowed words (one word = one meaning), avoiding ambiguities at word as well as at sentence level6, avoiding synonyms and emphasizing clarity of expression.
The PACE dictionary lists all allowed words (nouns, verbs, adjectives and even articles, conjunctions, pronouns and prepositinos) with their definition and examples. Homographs for instance are only listed with their "allowed" meaning(s)7. There is also a set of rules for syntax and sentence patterns.
This approach to writing grew out of a desire to convey technical information and instructions in as precise, clear and unambiguous a form as possible in the interests of safety and efficiency.
[Newton 1992c, p. 47]
Pym and his team examined the few available systems and decided to give MicroCat a try. They had to build an English/French version of the PACE dictionary for the test. The core dictionary could not be modified so they had to copy entries for non-technical terms from the core dictionary and modify them accordingly, e.g. limiting the parts of speech in homographic entries. This optimization also increased the system's performance as the number of alternatives was reduced.
A problem with the input of the dictionary was that the system automatically generated inflected forms of target-language verbs, nouns and adjectives which could collide with the translation of identical forms functioning as other parts of speech. Therefore, all source-language words are entered uninflected8.
The correct dictionary updating plays a key role in the quality of MicroCat's output. Multiple-word-entries were neccessary to compensate weaknesses of the system. Standard phrases and idioms were among them. MicroCat's feature of variable idioms was very useful here. Variable idioms are idioms with "holes" in them, e.g. be % % away where the "%" marks a hole to be filled in with a word. These words can be further specified by selecting word type and allowed gender, number, person etc. There was also the possibility to specify "holes" which can be filled in or left empty. With these features, it is possible to enter generic "idioms" which handle a theoretically infinite number of permutations.
A trial and error process with iterative refinements of the dictionary led to a significantly improved quality of the raw output. It is difficult to get the right balance between useful additions and cluttering the dictionary with less frequent occurences since this increases processing time and may confuse the system.
After the impression was gained that little further improvement could be made by manipulating the dictionary, the English source texts underwent closer inspection to figure out whether changes could be made to improve the quality of translation. Critical sentences found during the repetitive updating of the dictionary were rewritten without sacrificing their naturalness. During this process, the number of verbs in the PACE dictionary was risen from 80 to around 250. Pre-editing is aimed at improving the system's output for an individual source text. The original, unedited version remains the official source language document.
Pre-editing led to a significantly improved quality of the output requiring only very little post-editing, although some problems could not be solved. The MicroCat system used for the tests is limited to the analysis of sentences. It cannot trace inter-sentence references. The translated texts therefore often lack coherence which needs to be added during post-editing.
The translation experiment showed great promise. Peter Pym decided to order the software to translate the four language pairs his department had to handle.
The English/French language pair was almost complete but it was put through a broader test with previously human translated texts. A series of dictionary updating followed until it seemed to match Perkin's environment. The updates only afftected structures, not vocabulary which was already complete.
In 1986 the first text written under the modified PACE rules was translated by MicroCat. The result was better than expected. The raw translation was sent to Perkins in France for editing and proofreading.
After final proofreading by Perkins France, the text was approved for publication. Painstaking preparation had created conditions in which very short turnaround times were achievable with minimal post-editing.
[Newton 1992c, p. 54]
As a nice side-effect, the English source texts are very comprehensible which not only eases understanding by non-native readers forced to use the English version since their mother tongue is not available, but also for native speakers. Writing in controlled language leads to texts which are easier to understand and translate and also increases homogeneity between translated version of different target languages.
Conformance to PACE should not lead to untranslated words. If there are any words left untranslated by the system, they can be checked for potential addition to the PACE dictionary. Then, appropiate target-language translations can be determined and added to the MicroCat system. Additionally, post-editors are encouraged to report recurring structural problems.
MT at Perkins is a another nice example of a successful application. This was not achieved by introducing MT but during the process of introducing MT. By "looking even more closely at what was being written and how it was being written" [Newton 1992c, p. 55], the field was leveled for successful machine translation.
The same procedures would also have eased human translation since a lot of general translation problems were simply avoided instead of trying to solve them.
Categorizing MT at Perkins as FAHQMT, HAMT or MAHT is not easy. The core of the translation is fully automatic. The translation result is also of a high quality and requires little post-processin; it would qualify as FAHQT. On the other hand, there is a lot of human preparation neccessary -- the documents need to adhere to PACE writing rules and sometimes have to be restructured before translation. The mode of translation is therefore a mixture of HAMT and FAHQMT, it could be called Human Aided Automatic High Quality Translation.