When you export data in JSON from Facebook, the multi-byte Unicode characters are broken. Example Unicode characters and their representation inside JSON:
'\u00c2\u00b0' : '°',
'\u00c3\u0081' : 'Á',
'\u00c3\u00a1' : 'á',
'\u00c3\u0089' : 'É',
'\u00c3\u00a9' : 'é',
'\u00c3\u00ad' : 'í',
'\u00c3\u00ba' : 'ú',
'\u00c3\u00bd' : 'ý',
'\u00c4\u008c' : 'Č',
'\u00c4\u008d' : 'č',
'\u00c4\u008f' : 'ď',
'\u00c4\u009b' : 'ě',
'\u00c5\u0098' : 'Ř',
'\u00c5\u0099' : 'ř',
'\u00c5\u00a0' : 'Š',
'\u00c5\u00a1' : 'š',
'\u00c5\u00af' : 'ů',
'\u00c5\u00be' : 'ž',
In the last example the bytes 0xC5
and 0xBE
are the UTF-8 encoding for U+017E
. If those individual bytes are treated as unicode codepoints, they are Å
and ¾
. So Facebook wrote each byte as an Unicode codepoint instead of handling multi-byte UTF-8 sequences appropriately.
This incorrect representation is used for every multi-byte UTF character. In short, to fix it, one needs to convert any string sequences like \u00c5\u00be\u00c5\u0099
into byte data where first byte is \xc5
and second byte is \xbe
, third is \xc5
and fourth is \x99
, then read these bytes as a UTF-8 string. This can happen by direct manipulation of raw JSON text or by first parsing JSON, then do some magic with the incorrectly parsed strings or whatever comes to your mind.
Note that while "\u00c5\u00be\u00c5\u0099"
is valid JSON, "\xc5\xbe\xc5\x99"
is not. Also
A working Ruby and Python (not verified) solution can be seen in the referenced StackOverflow question at the bottom.
The task is to fix JSON generated by Facebook with the shortest code.
Winning criteria: code-golf - the shortest code.
As input you can take a file name, STDIN or the file text as a variable. The output should be on STDOUT, a file name or a string in a variable.
You don't have precise JSON input format (except that it is valid JSON), because exported data can be different. Also result should be valid JSON.
Solutions
- should accept as input both - pretty and non-pretty formatted JSON
- should handle properly 1,2,3 and 4 octets UTF character. With the assurance (if that matters) that any multi-byte UTF characters in the input will always be split into multiple
\u00##
sequences (i.e. you will never get\u017E
or another sequence where first byte is non-zero). - can output any - pretty formatted or non-pretty JSON
- doesn't matter whether characters in the output are represented by plain Unicode characters or escaped Unicode
\u017E
- you are allowed to use any JSON parsing libraries
Example input:
{
"name": "\u00e0\u00a4\u0085\u00e0\u00a4\u0082\u00e0\u00a4\u0095\u00e0\u00a4\u00bf\u00e0\u00a4\u00a4 G \u00e0\u00a4\u00b6\u00e0\u00a5\u0081\u00e0\u00a4\u0095\u00e0\u00a5\u008d\u00e0\u00a4\u00b2\u00e0\u00a4\u00be",
"timestamp": 1599213799,
"whatever": "someth\u0022\u00d0\u0092\u00d0\u00b8\u00d0\u00ba\u00d0\u00b8",
"test": "\\u0041"
}
Example acceptable output:
{
"name": "अंकित G शुक्ला",
"timestamp": 1599213799,
"whatever": "someth\"Вики",
"test": "\\u0041"
}
Credit goes to original question https://stackoverflow.com/q/52747566/520567
s.encode("latin-1").decode("utf-8")
. \$\endgroup\${"key":" \u00E4\u00BE\u008B\u00E5\u00AD\u0090"}
, but never{" \u00E4\u00BE\u008B\u00E5\u00AD\u0090":"value"}
\$\endgroup\$