On the Translation Quality of Google Translate: With a Concentration on Adjectives

[1]Abouzar Oraki

Lecturer

Abstract

Translation, whose first traces date back at least to 3000 BC (Newmark, 1988), has always been considered time-consuming and labor-consuming. In view of this, experts have made numerous efforts to develop some mechanical systems which can reduce part of this time and labor. The advancement of computers in the second half of the twentieth century paved the ground for the invention of machine translation (MT). One of the most commonly used MT systems is Google Translate, which currently supports 64 languages, including Persian. In consideration of the fact that Google Translate is easily accessible, it is almost always the first MT system to which Iranian users resort for meeting their translation needs, the reason which prompted the researcher to conduct a study on the output quality of the English-to-Persian translations produced by this MT system. To narrow down the study, the researcher decided to investigate the translation of adjectives of all types, i.e. simple adjectives, adjectives of similarity, comparatives and superlatives. To this end, a test suite of 140 sentences (randomly chosen among 1400 sentences) containing all types of adjectives was prepared and translated by Google Translate. The analysis of the results revealed that Google Translate translates simple, comparative and superlative adjectives fairly accurately and naturally, while it fails to distinguish adjectives of similarity and, thus, translates them quite inaccurately.

Keywords: machine Translation, Google Translate, adjectives, accuracy

1. Introduction

The amount of translation worldwide has increased considerably over the last fifty years and in this workplace, the role of technology, in particular, the Internet and Machine Translation has been prominent. Machine Translation, also known as MT, is a sub-category of computational linguistics which analyses the use of computer software to translate written or spoken text from one language to another.To elaborate on the MT, it should be mentioned that the first real developments in MT appeared quite a while after the Second World War, a time during which the first computers had been newly designed in the UK (Hatim and Munday, 2004).

Among all MT software, Google Translate is the most commonly used worldwide. The owners of Google Translate assert that Google Translate is a free, automatic translator.

They also go further to say that it currently supports translation between dozens of languages. But there has always been some skepticism about the translation quality of such software. As Newmark (1988) puts it, the concept of quality in translation is somehow "relative".

Many different translation scholars have worked on the concept of translation quality. Among such scholars Reiss, 1972, Wilss, 1977, Amman, 1990, Gerzymisch-Abrogasts, 1977 can be referred to. But, according to Cary and Jumpelt (1963, cited in Khoshsima and Rostami, 2010), defining the quality of translation was first discussed in the Third Conference of the International Federation of Translators on Quality in 1959.

Needless to say, finding out the quality of a translation is no easy job. As House (1997, cited in Khoshsima and Rostami, ibid) asserts:

"Translation quality is a problematical concept if it is taken to involve individual and externally motivated value judgment alone. Passing any "final judgment" on the quality of a translation that fulfills the demands of scientific objectivity is very difficult indeed (p. 2)."

To find out the quality of such software, the researcher carried out the present research. In order to narrow down the research, the decision was made to carry out the research on the different types of adjectives only. The present study, in brief, has tried to answer the following question and sub-questions:

To what extent is Google Translate able to translate the followings as accurately as possible?

a)    Simple adjectives

b)   Comparative adjectives

c)    Superlative adjectives

d)   Adjectives of similarity

2. Review of Literature

As pointed earlier, it is thought that in the short term, MT is likely to be of most benefit to large corporate organizations doing a lot of translation. It is of paramount importance from different aspects. As Tauschel (2008) puts it, the social or political significance of MT emanates from the socio-political significance of translation in communities where more than one language is generally spoken. The commercial importance of MT arises from the fact that translation itself is commercially important. Scientifically, MT is interesting, because it is an obvious application and testing ground for many ideas in Computer Science, Artificial Intelligence, and Linguistics, and some of the most important developments in these fields have begun in MT. Philosophically, MT is interesting, because it represents an attempt to automate an activity that can require the full range of human knowledge — that is, for any piece of human knowledge, it is possible to think of a context where the knowledge is required.

But it should also be noted that MT output must be examined and evaluated so that corporate organizations and, on the whole, those who are intend to make use of  this technology get to know the ups and downs of what they are longing for.

As Linares (2008) asserts, in the context of MT system development, evaluation methods are necessary for three main purposes:

• Error Analysis, i.e., to detect and analyze possible cases of error. Fine knowledge of the system capabilities is essential for improving its behavior.

• System Comparison, i.e., to measure the effectiveness of the suggested mechanisms. This is done by comparing different versions of the same system. It is also common to compare translations by different systems, so system developers may borrow successful mechanisms from each other. This allows the research community to advance together.

• System Optimization, i.e., the adjustment of internal parameters. Typically, these parameters are adjusted so as to maximize overall system quality as measured according to an evaluation method at choice.

Evaluating the MT is not an easy job. As Arnold et al (2001) put it:

“The evaluation of MT systems is a complex task. This is not only because many different factors are involved, but because measuring translation performance is itself difficult. The first important step for a potential buyer is to determine the translational needs of her organization. (p. 157)”

Arnold (ibid) goes further to say that evaluating translation quality is not only a problem for MT. He asserts that it is a practical problem that human translators face, and one which translation theorists have puzzled over. As he asserts, for human translators, the problem is that there are typically many possible translations, some of them faithful to the original in some respects (e.g. literal meaning), while others try to preserve other properties (e.g. style, or emotional impact).

Arnold (ibid) introduces some of the methods that have been used to date. The followings are what he has mentioned:

Intelligibility:

A traditional way of assessing the quality of translation is to assign scores to output sentences. A common aspect to score for is Intelligibility, where the intelligibility of a translated sentence is affected by grammatical errors, mistranslations and untranslated words. Some studies also take style into account, even though it does not really affect the intelligibility of a sentence. Scoring scales reflect top marks for those sentences that look like perfect target language sentences and bottom marks for those that are so badly degraded as to prevent the average translator/evaluator from guessing what a reasonable sentence might be in the context.

Accuracy:

By measuring intelligibility we get only a partial view of translation quality. A highly intelligible output sentence need not be a correct translation of the source sentence. It is important to check whether the meaning of the source language sentence is preserved in the translation. This property is called Accuracy or Fidelity. Scoring for accuracy is normally done in combination with (but after) scoring for intelligibility.

Error Analysis:

Rather than using broad indicators as guides to score assignments, you could focus on the errors the MT system makes. The technique of error analysis tries to establish how seriously errors affect the translation output.

Test Suite:

In software development, a test suite, less commonly known as a validation suite, is a collection of test cases that are intended to be used to test a software program to show that it has some specified set of behaviors. A test suite often contains detailed instructions or goals for each collection of test cases and information on the system configuration to be used during testing. To succeed in correctly translating all the sentences in a source language (SL) test suite into a target language (TL) would definitely be encouraging. However, standard test suites are rather blunt instruments for probing translation performance in the sense that they tend to ignore typical differences between the languages involved in translation.

3. Methodology and procedures

In order to answer the above-mentioned question and sub-questions, a test suite of 140 sentences (randomly chosen among 1400 sentences), as the raw material, containing all of the aforementioned types of adjectives was prepared and translated by Google Translate:

Table1. Sentences containing Simple adjectives

As it can be noticed from the above table, all of the adjectives (all of the 35 adjectives which equals with %100) have been translated almost accurately. The following chart is representative of what has happened.

Figure 1. Sentences containing simple adjectives

Table 2. Sentences containing comparative adjectives

As it can be seen from the above table, Google Translate has translated comparative adjectives in the same way as it did in the case of simple adjectives. For the second time, all the 35 adjectives (%100 of the adjectives) have been translated in the most accurate way for a machine. The following chart is a clear representation of the above-mentioned data.

Figure 2. Sentence containing comparative adjectives

Table3. Sentences containing superlative adjectives

For the third time, a great number of the adjectives have been translated accurately according to the above table. 33 out of 35 adjectives (%94) have been translated accurately. The following Figure represents the above-mentioned data.

Figure 3. Sentences containing superlative adjectives

Table 4. Sentences containing adjectives of similarity

As the above table shows, only 2 out of 35 adjectives have been translated accurately, which makes up only % 5.7. The following figure represents what has happened in the case of adjectives of similarity.

Figure 4. Sentences containing adjectives of similarity

4. Discussion of the Results

The information inserted under the analysis part of each different kind of adjective, which is the representation of the data gathered in this study showed that simple adjectives were translated accurately by %100, i.e. all the 35 adjectives were translated in an accurate way. The same happened to the comparative adjectives (35 out of 35= %100). In the case of the superlative adjectives, 33 adjectives out of the 35 adjectives were translated accurately, which makes up %94. However, in the case of the adjectives of similarity, little success was achieved. In other words, out of the 35 adjectives, only 2 (%5.7) were translated almost accurately.

5. Conclusions

The present study was an attempt to find out the translation quality of Google Translate, as a widely used device, with a concentration on the different types of adjectives. In this study, all the four types of adjectives (simple adjectives, comparative adjectives, superlative adjectives, and adjectives of similarity) were translated separately by Google Translate. The analysis of the data showed that Google Translate is very successful in translating simple adjectives (%100), comparative adjectives (%100), and superlative adjectives (%94), but fails to translate adjectives of similarity as accurately as possible. Only %5.7 of the adjectives was translated (almost) accurately. This analysis showed that the users can rely on this device for translating simple, comparative, as well as superlative adjectives, but in case of adjectives of similarity a kind of human aided machine translation (machine translation with the help of human translator) must be applied if they are to get good results.