We conducted a comparive analysis of popular automatic translation softwares and our Kitchen Translator, and found that Kitchen Translator was far more usable when we considered its high-quality outputs.
Sample Outputs and Their Scores
Softwares and Method
We used the following on-line translation softwares as well as Kitchen Translator for our experiment:
Scoring was done by an experienced human evaluator (me) by using the following criterion specifically made by us for evaluating the performance of translating food recipes. The final scores are shown by the percentage (100 as the highest). Each MT translated about 150 sentences (lines) for each translation direction (J to E and E to J).
- 5 points：All words and phrases are correctly translated. Has high fluency (can be used with no or little correction)
- 4：Can convey what the target sentence (line) says but lacks fluency.
- 3：Some mistakes (less than 50 percent).
- 2：More than half of the contents are wrongly translated.
- 1：Most of the content are wrongly translated. No output.*1
- 0：Completely wrong translation.
*1 I believe that giving totally wrong information is worse than giving no output since users may not know that it is wrong translation if it is at least fluent.
We evaluated MTs according to the following three aspects:
- Overall performance
- No fluency asked
- Usability (whether you can use the results as they are for your documents)
[1] Overall Performance
I am so glad and relieved to tell that our KITCHEN TRANSLATOR was given very high scores, ranking first among all the systems evaluated!
English to Japanese
1：KITCHEN TRANSLATOR - 84/100
2：GOOGLE - 63/100
3：YAHOO! - 59/100
4：EXCITE - 55/100
Japanese to English
1：KITCHEN TRANSLATOR - 93/100
2：GOOGLE - 73/100
3：YAHOO! - 72/100
4：EXCITE - 69/100
[2] No Fluency Asked
We calculated MT's performance of conveying just the overall meaning ignoring their fluency. Method: All the sentences given 4 points were given 5.
English to Japanese
1：KITCHEN TRANSLATOR - 84/100
2：GOOGLE - 69/100
3：YAHOO! - 68/100
4：EXCITE - 59/100
Japanese to English
1：KITCHEN TRANSLATOR - 93/100
2：GOOGLE - 80/100
3：YAHOO! - 76/100
4：EXCITE - 77/100
Trustiness and Practical Usability
Method: All the sentences given 5 points were given 1, and the rest (0 - 4) were given zero.
English to Japanese
1：KITCHEN TRANSLATOR - 72/100
2：GOOGLE - 25/100
3：EXCITE - 16/100
4：YAHOO! - 10/100
Japanese to English
1：KITCHEN TRANSLATOR - 87/100
2：GOOGLE - 52/100
3：YAHOO! - 26/100
4：EXCITE - 25/100
Speediness
As we have seen, KITCHEN TRANSLATOR is a very reliable recipe ingredient translation tool, but I have to add that it is true that it lacks speediness and easiness shown by other on-line auto translators. It is slower because you have to input the ingredient names one by one. However it takes only one to three minutes to translate a list containing 10 to 20 ingredients.
Conclusion
Our experimental evaluation shows that KITCHEN TRANSLATOR is much superior to other on-line translators when translating lists of recipe ingredients. Although it can give users highly usable translations, it is slower and sometimes needs users' judgement.
Sample Outputs and Their Scores
Tweet
