What language is this word?

Question

Your should write a program or function that determines the language of a given word.

The task is to recognize some of the 5000 most common words in 4 languages:

• English
• German
• Italian
• Hungarian

The word lists can be found in this GitHub repository.

You are allowed to make mistakes in 40% of the provided test cases. I.e. you can miscategorize 8000 of the 20000 inputs.

Details

• The lists only contain words with lowercase letters a-z so e.g. won't and möchte are not included.
• A few words appear in multiple languages which means your code can't always guess the expected output correctly.
• For convenience you can download all the test cases as one list. In each line a number indicates the language of the word. (1 for English, 2 for German, 3 for Italian and 4 for Hungarian.)
• Standard loopholes are disallowed.
• Using word lists our similar data provided by your programming language is forbidden.

Input

• A string containing only lowercase English letters (a-z).
• Trailing newline is optional.

Output

• You can categorize the words by providing a distinct and consistent (always the same) output for each language. (E.g. 1 for English, 2 for German, 3 for Italian and 4 for Hungarian.)

This is code golf so the shortest program or function wins.

Related code golf question: Is this even a word?

^{The word lists were taken from wiktionary.org and 101languages.net.}

Answer 1

Retina, 51 bytes

.*[aeio]$
1
A`en$|ch|ei|au
^$
2
A`[jkz]|gy|m$
\D+
4

I came up with the regexes and @MartinBüttner did the conversion to/golfing in Retina so ... hurray for team effort?

The mapping is 1 -> Italian, 2 -> German, (empty) -> Hungarian, 4 -> English, with the amount classified in each category being 4506 + 1852 + 2092 + 3560 = 12010.

Try it online! | Modified multiline version

Explanation

First off, the equivalent Python is something like this:

import re
def f(s):
  if re.search("[aeio]$", s):
    return 1
  if re.search("en$|ch|ei|au", s):
    return 2
  if re.search("[jkz]|gy|m$", s):
    return ""
  return 4

Let me just say that o$ is an excellent indicator of Italian.

The Retina version is similar, with pairs of lines forming replacement stages. For example, the first two lines

.*[aeio]$
1

replaces matches of the first line with the contents of the second.

The next three lines do the same, but using Retina's anti-grep mode - anti-grep (specified with A`) removes the line if it matches the given regex, and the following two lines is a replacement from an empty line to the desired output.

A`en$|ch|ei|au
^$
2

The following line uses anti-grep again, but doesn't replace the empty line, giving the fixed output for Hungarian.

A`[jkz]|gy|m$

Finally, the last two lines

\D+
4

replaces a non-empty non-digit line with 4. All of the substitutions can only happen if no previous substitution activated, simulating an if/else if chain.

Answer 2

LabVIEW, 29 LabVIEW Primitives and 148.950 Bytes

cycles through the languages and puts the iterator into an array if the word is there. This gets checked by the inner loop, picking the ith line and doing =. In LabVIEW it only gives a true if the Strings are exactly the same.

Now take the first element of output array so englishgoes over the rest.

The output for now is 0 for English, 1 for German, 2 for Italian and 3 for Hungarian.

Answer 3

Java, 3416 bytes, 62%

this is my solution, i analyze list of given words and find 60 most commons bigrams and trigrams for each languages. Now i'm checking my n-grams against word, and choosing language with most n-grams in word.

public class Classificator {
     
    String[][] triGr = {
            {"ing","ion","ent","tio","ted","nce","ter","res","ati","con","ess","ate","pro","ain","est","ons","men","ect","red","rea","com","ere","ers","nte","ine","her","ble","ist","tin","for","per","der","ear","str","ght","pre","ver","int","nde","the","igh","ive","sta","ure","end","enc","ned","ste","dis","ous","all","and","anc","ant","oun","ten","tra","are","sed","cti"},
            {"sch","che","ver","gen","ten","cht","ich","ein","ste","ter","hen","nde","nge","ach","ere","ung","den","sse","ers","and","eit","ier","ren","sen","ges","ang","ben","rei","est","nen","nte","men","aus","der","ent","hei","her","lle","ern","ert","uch","ine","ehe","auf","lie","tte","ige","ing","hte","mme","end","wei","len","hre","rau","ite","bes","ken","cha","ebe"},
            {"ent","are","ato","nte","ett","ere","ion","chi","con","one","men","nti","gli","pre","ess","att","tto","par","per","sta","tra","zio","and","iam","end","ter","res","est","nto","tta","acc","sci","cia","ver","ndo","amo","ant","str","tro","ssi","pro","era","eri","nta","der","ate","ort","com","man","tor","rat","ell","ale","gio","ont","col","tti","ano","ore","ist"},
            {"sze","ere","meg","ett","gye","ele","ond","egy","enn","ott","tte","ete","unk","ban","tem","agy","zer","esz","tet","ara","nek","hal","dol","mon","art","ala","ato","szt","len","men","ben","kap","ent","min","ndo","eze","sza","isz","fog","kez","ind","ten","tam","nak","fel","ene","all","asz","gon","mar","zem","szo","tek","zet","elm","het","eve","ssz","hat","ell"}

                    };
    static String[][] biGr = {
        {"in","ed","re","er","es","en","on","te","ng","st","nt","ti","ar","le","an","se","de","at","ea","co","ri","ce","or","io","al","is","it","ne","ra","ro","ou","ve","me","nd","el","li","he","ly","si","pr","ur","th","di","pe","la","ta","ss","ns","nc","ll","ec","tr","as","ai","ic","il","us","ch","un","ct"},
        {"en","er","ch","te","ge","ei","st","an","re","in","he","ie","be","sc","de","es","le","au","se","ne","el","ng","nd","un","ra","ar","nt","ve","ic","et","me","ri","li","ss","it","ht","ha","la","is","al","eh","ll","we","or","ke","fe","us","rt","ig","on","ma","ti","nn","ac","rs","at","eg","ta","ck","ol"},
        {"re","er","to","ar","en","te","ta","at","an","nt","ra","ri","co","on","ti","ia","or","io","in","st","tt","ca","es","ro","ci","di","li","no","ma","al","am","ne","me","le","sc","ve","sa","si","tr","nd","se","pa","ss","et","ic","na","pe","de","pr","ol","mo","do","so","it","la","ce","ie","is","mi","cc"},
        {"el","en","sz","te","et","er","an","me","ta","on","al","ar","ha","le","gy","eg","re","ze","em","ol","at","ek","es","tt","ke","ni","la","ra","ne","ve","nd","ak","ka","in","am","ad","ye","is","ok","ba","na","ma","ed","to","mi","do","om","be","se","ag","as","ez","ot","ko","or","cs","he","ll","nn","ny"}

                    };

    public int guess(String word) {

        if (word.length() < 3) {
            return 4; // most words below 2 characters on list are hungarians
        }
        int score[] = { 0, 0, 0, 0 };
        for (int i = 0; i < 4; i++) {
            for (String s : triGr[i]) {
                if (word.contains(s)) {
                    score[i] = score[i] + 2;
                }
            }
            for (String s : biGr[i]) {
                if (word.contains(s)) {
                    score[i] = score[i] + 1;
                }
            }
        }
        int v = -1;
        int max = 0;
        for (int i = 0; i < 4; i++) {
            if (score[i] > max) {
                max = score[i];
                v = i;
            }
        }
        v++;
        return v==0?Math.round(4)+1:v;
    }
}

and this is my testcase

public class Test {

    Map<String, List<Integer>> words = new HashMap<String, List<Integer>>();

    boolean validate(String word, Integer lang) {
        List<Integer> langs = words.get(word);
        return langs.contains(lang);
    }

    public static void main(String[] args) throws FileNotFoundException {

        FileReader reader = new FileReader("list.txt");
        BufferedReader buf = new BufferedReader(reader);
        Classificator cl = new Classificator();
        Test test = new Test();
        buf.lines().forEach(x -> test.process(x));
        int guess = 0, words = 0;
        for (String word : test.words.keySet()) {
            int lang = cl.guess(word);
            if (lang==0){
                continue;
            }
            boolean result = test.validate(word, lang);
            words++;
            if (result) {
                guess++;
            }
        }
        System.out.println(guess+ " "+words+ "    "+(guess*100f/words));
    }

    private void process(String x) {
        String arr[] = x.split("\\s+");
        String word = arr[0].trim();
        List<Integer> langs = words.get(word);
        if (langs == null) {
            langs = new ArrayList<Integer>();
            words.put(word, langs);
        }
        langs.add(Integer.parseInt(arr[1].trim()));

    }

}