3
\$\begingroup\$

Your task is to write a program or function that detects accents in UTF-8 input. The catch is that your code must be written entirely in an 8-bit character encoding. The code will output a single integer which is the count of the number of accented letters in the input. Since Unicode is huge, the test input will consist solely of the characters found in Windows-1252, stored in the UTF-8 encoding, then treated as ASCII with extended characters (0x09, 0x0A, 0x0D, 0x20-0xFF only). The input won't contain any characters below 0x20, except for the three common ones, 0x09 (tab), 0x0A (line feed) & 0x0D (carriage return). It also won't contain 0x7F.

Unicode permits most common accented characters to be represented in multiple ways (only two for the characters in this challenge). é could be a pre-composed character: U+00E9 (C3 A9 in UTF-8) or a basic letter plus a combining mark: U+0065+U+0301 (65+CC 81 in UTF-8). Your code must correctly handle both.

In the following table * marks an accented letter. All letters that have two forms will be considered accented, plus Œ, œ, Æ & æ. All other characters will be considered as symbols or non-accented letters.

1252  Char  Unicode   Char  Unicode
         Bytes           Bytes
   Accented?
-----------------------------------------
80 _  €  3  U+20AC
81 Undefined
82 _  ‚  3  U+201A
83 _  ƒ  2  U+0192
84 _  „  3  U+201E
85 _  …  3  U+2026
86 _  †  3  U+2020
87 _  ‡  3  U+2021
88 _  ˆ  2  U+02C6
89 _  ‰  3  U+2030
8A *  Š  2  U+0160    Š  3  U+0053 U+030C
8B _  ‹  3  U+2039
8C *  Œ  2  U+0152
8D Undefined
8E *  Ž  2  U+017D    Ž  3  U+005A U+030C
8F Undefined
90 Undefined
91 _  ‘  3  U+2018
92 _  ’  3  U+2019
93 _  “  3  U+201C
94 _  ”  3  U+201D
95 _  •  3  U+2022
96 _  –  3  U+2013
97 _  —  3  U+2014
98 _  ˜  2  U+02DC
99 _  ™  3  U+2122
9A *  š  2  U+0161    š  3  U+0073 U+030C
9B _  ›  3  U+203A
9C *  œ  2  U+0153
9D Undefined
9E *  ž  2  U+017E    ž  3  U+007A U+030C
9F *  Ÿ  2  U+0178    Ÿ  3  U+0059 U+0308
A0 _     2  U+00A0
A1 _  ¡  2  U+00A1
A2 _  ¢  2  U+00A2
A3 _  £  2  U+00A3
A4 _  ¤  2  U+00A4
A5 _  ¥  2  U+00A5
A6 _  ¦  2  U+00A6
A7 _  §  2  U+00A7
A8 _  ¨  2  U+00A8
A9 _  ©  2  U+00A9
AA _  ª  2  U+00AA
AB _  «  2  U+00AB
AC _  ¬  2  U+00AC
AD _  2  U+00AD
AE _  ®  2  U+00AE
AF _  ¯  2  U+00AF
B0 _  °  2  U+00B0
B1 _  ±  2  U+00B1
B2 _  ²  2  U+00B2
B3 _  ³  2  U+00B3
B4 _  ´  2  U+00B4
B5 _  µ  2  U+00B5
B6 _  ¶  2  U+00B6
B7 _  ·  2  U+00B7
B8 _  ¸  2  U+00B8
B9 _  ¹  2  U+00B9
BA _  º  2  U+00BA
BB _  »  2  U+00BB
BC _  ¼  2  U+00BC
BD _  ½  2  U+00BD
BE _  ¾  2  U+00BE
BF _  ¿  2  U+00BF
C0 *  À  2  U+00C0    À  3  U+0041 U+0300
C1 *  Á  2  U+00C1    Á  3  U+0041 U+0301
C2 *  Â  2  U+00C2    Â  3  U+0041 U+0302
C3 *  Ã  2  U+00C3    Ã  3  U+0041 U+0303
C4 *  Ä  2  U+00C4    Ä  3  U+0041 U+0308
C5 *  Å  2  U+00C5    Å  3  U+0041 U+030A
C6 *  Æ  2  U+00C6
C7 *  Ç  2  U+00C7    Ç  3  U+0043 U+0327
C8 *  È  2  U+00C8    È  3  U+0045 U+0300
C9 *  É  2  U+00C9    É  3  U+0045 U+0301
CA *  Ê  2  U+00CA    Ê  3  U+0045 U+0302
CB *  Ë  2  U+00CB    Ë  3  U+0045 U+0308
CC *  Ì  2  U+00CC    Ì  3  U+0049 U+0300
CD *  Í  2  U+00CD    Í  3  U+0049 U+0301
CE *  Î  2  U+00CE    Î  3  U+0049 U+0302
CF *  Ï  2  U+00CF    Ï  3  U+0049 U+0308
D0 _  Ð  2  U+00D0
D1 *  Ñ  2  U+00D1    Ñ  3  U+004E U+0303
D2 *  Ò  2  U+00D2    Ò  3  U+004F U+0300
D3 *  Ó  2  U+00D3    Ó  3  U+004F U+0301
D4 *  Ô  2  U+00D4    Ô  3  U+004F U+0302
D5 *  Õ  2  U+00D5    Õ  3  U+004F U+0303
D6 *  Ö  2  U+00D6    Ö  3  U+004F U+0308
D7 _  ×  2  U+00D7
D8 _  Ø  2  U+00D8
D9 *  Ù  2  U+00D9    Ù  3  U+0055 U+0300
DA *  Ú  2  U+00DA    Ú  3  U+0055 U+0301
DB *  Û  2  U+00DB    Û  3  U+0055 U+0302
DC *  Ü  2  U+00DC    Ü  3  U+0055 U+0308
DD *  Ý  2  U+00DD    Ý  3  U+0059 U+0301
DE _  Þ  2  U+00DE
DF _  ß  2  U+00DF
E0 *  à  2  U+00E0    à  3  U+0061 U+0300
E1 *  á  2  U+00E1    á  3  U+0061 U+0301
E2 *  â  2  U+00E2    â  3  U+0061 U+0302
E3 *  ã  2  U+00E3    ã  3  U+0061 U+0303
E4 *  ä  2  U+00E4    ä  3  U+0061 U+0308
E5 *  å  2  U+00E5    å  3  U+0061 U+030A
E6 *  æ  2  U+00E6
E7 *  ç  2  U+00E7    ç  3  U+0063 U+0327
E8 *  è  2  U+00E8    è  3  U+0065 U+0300
E9 *  é  2  U+00E9    é  3  U+0065 U+0301
EA *  ê  2  U+00EA    ê  3  U+0065 U+0302
EB *  ë  2  U+00EB    ë  3  U+0065 U+0308
EC *  ì  2  U+00EC    ì  3  U+0069 U+0300
ED *  í  2  U+00ED    í  3  U+0069 U+0301
EE *  î  2  U+00EE    î  3  U+0069 U+0302
EF *  ï  2  U+00EF    ï  3  U+0069 U+0308
F0 _  ð  2  U+00F0
F1 *  ñ  2  U+00F1    ñ  3  U+006E U+0303
F2 *  ò  2  U+00F2    ò  3  U+006F U+0300
F3 *  ó  2  U+00F3    ó  3  U+006F U+0301
F4 *  ô  2  U+00F4    ô  3  U+006F U+0302
F5 *  õ  2  U+00F5    õ  3  U+006F U+0303
F6 *  ö  2  U+00F6    ö  3  U+006F U+0308
F7 _  ÷  2  U+00F7
F8 _  ø  2  U+00F8
F9 *  ù  2  U+00F9    ù  3  U+0075 U+0300
FA *  ú  2  U+00FA    ú  3  U+0075 U+0301
FB *  û  2  U+00FB    û  3  U+0075 U+0302
FC *  ü  2  U+00FC    ü  3  U+0075 U+0308
FD *  ý  2  U+00FD    ý  3  U+0079 U+0301
FE _  þ  2  U+00FE
FF *  ÿ  2  U+00FF    ÿ  3  U+0079 U+0308
  • Input will be in any 8-bit encoding which is a super-set of ASCII, i.e. the lower half of the code points will match exactly with ASCII, the upper half (0x80-0xFF) will be used exclusively for multi-byte UTF-8 encoded codepoints, e.g. (U+201C) will be treated as both three bytes and three characters by your code: E2 80 9C
  • Language features or pre-existing libraries that convert character encodings or handle Unicode are forbidden, e.g. JSON (can convert Unicode text to hex), Unicode flag on regex, etc.

Sample output and input:

  • 3 Zoë reads the encyclopædia in a café
  • 0 Zoe reads the encyclopaedia in a cafe
  • 1 Zoë (pre-composed character, 3 codepoints, 4 bytes)
  • 1 Zoë (base letter + combining mark, 4 codepoints, 5 bytes)
  • 2 déjà vu

Scoring

Simple code-golf: lowest number of bytes wins!

This is a re-write as my computer crashed just before posting the first version and I lost all but the first few lines of the challenge. Apologies if I've left anything out!

\$\endgroup\$
16
  • 1
    \$\begingroup\$ 1. At the start of the question you say that the input will be in UTF-8, but later on you say "any 8-bit encoding which is a super-set of ASCII". I don't think that the latter is well-specified. There are a potentially unlimited number of ASCII extensions. 2. What about ñ, ø, etc? I believe that technically neither of those is an accented character, at least in the languages which use them: rather they are full letters of the alphabet. 3. Could you please list the combining characters which we must take into account? \$\endgroup\$ Commented Jan 6, 2016 at 12:22
  • \$\begingroup\$ @PeterTaylor They are accented in English, i.e. they look like an English letter with an accent, therefore they will be considered as accented letters. The input will be transmitted as UTF-8, but mistakenly interpreted by the code as ASCII, since it doesn't know about encodings other than 8-bits. \$\endgroup\$
    – CJ Dennis
    Commented Jan 6, 2016 at 12:26
  • 4
    \$\begingroup\$ Please provide a full list of all accented characters, using all permitted variations. \$\endgroup\$
    – Dennis
    Commented Jan 6, 2016 at 14:25
  • 1
    \$\begingroup\$ What is the character between ¬ and ® - it renders as invisible for me on chrome, firefox and safari. (looks like some sort of hyphen/dash when pasted into emacs. codepoint: 0xAD html: ­) \$\endgroup\$ Commented Jan 6, 2016 at 16:34
  • 1
    \$\begingroup\$ Outside of this challenge, a lot of those characters are far from being accents, e.g. æ, œ, ß, ð, þ \$\endgroup\$
    – Fatalize
    Commented Jan 7, 2016 at 13:13

1 Answer 1

2
\$\begingroup\$

Mathematica, 46 bytes

Count[Characters@#,c_/;RemoveDiacritics@c!=c]&

Explanations:

Chatacters extract every characters in the string. Note that there is a difference between two Zoë even if they look the same:

Characters["Zoë"]
(* {"Z", "o", "ë"} *)

Characters["Zoë"]
(* {"Z", "o", "e", "̈ ̈"} *)

Here, the pattern c_/;RemoveDiacritics@c!=c will be matched with those characters that change with diacritics removed. For the first case, only ë will match since ë becomes e. While for the second case, only ̈ will match since it becomes nothing. Whatever case the string is, the counting will be correct.

Test cases:

f=%; (* assign the function to f. *)

f["Zoë reads the encyclopædia in a café"]
(* 3 *)
f["Zoe reads the encyclopaedia in a cafe"]
(* 0 *)
f["Zoë"]
(* 1 *)
f["Zoë"]
(* 1 *)
f["déjà vu"]
(* 2 *)
\$\endgroup\$
4
  • \$\begingroup\$ Unfortunately, this answer is invalid because it uses a built in library/feature of the language. \$\endgroup\$
    – CJ Dennis
    Commented Jan 11, 2016 at 13:54
  • \$\begingroup\$ @CJDennis I don't really get it. The challange forbiddens libraries that convert character encodings or handle Unicode, however neither "Characters" nor "RemoveDiacritics" convert encodings. \$\endgroup\$ Commented Jan 11, 2016 at 14:08
  • \$\begingroup\$ How does the built-in function work if it isn't just handling Unicode? Your code certainly doesn't initially treat "ë" as two bytes/characters. \$\endgroup\$
    – CJ Dennis
    Commented Jan 11, 2016 at 14:13
  • \$\begingroup\$ @CJDennis I'm adding some explanations. \$\endgroup\$ Commented Jan 11, 2016 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.