Fix Facebook exported JSON

Question

When you export data in JSON from Facebook, the multi-byte Unicode characters are broken. Example Unicode characters and their representation inside JSON:

'\u00c2\u00b0' : '°',
'\u00c3\u0081' : 'Á',
'\u00c3\u00a1' : 'á',
'\u00c3\u0089' : 'É',
'\u00c3\u00a9' : 'é',
'\u00c3\u00ad' : 'í',
'\u00c3\u00ba' : 'ú',
'\u00c3\u00bd' : 'ý',
'\u00c4\u008c' : 'Č',
'\u00c4\u008d' : 'č',
'\u00c4\u008f' : 'ď',
'\u00c4\u009b' : 'ě',
'\u00c5\u0098' : 'Ř',
'\u00c5\u0099' : 'ř',
'\u00c5\u00a0' : 'Š',
'\u00c5\u00a1' : 'š',
'\u00c5\u00af' : 'ů',
'\u00c5\u00be' : 'ž',

In the last example the bytes 0xC5 and 0xBE are the UTF-8 encoding for U+017E. If those individual bytes are treated as unicode codepoints, they are Å and ¾. So Facebook wrote each byte as an Unicode codepoint instead of handling multi-byte UTF-8 sequences appropriately.

This incorrect representation is used for every multi-byte UTF character. In short, to fix it, one needs to convert any string sequences like \u00c5\u00be\u00c5\u0099 into byte data where first byte is \xc5 and second byte is \xbe, third is \xc5 and fourth is \x99, then read these bytes as a UTF-8 string. This can happen by direct manipulation of raw JSON text or by first parsing JSON, then do some magic with the incorrectly parsed strings or whatever comes to your mind.

Note that while "\u00c5\u00be\u00c5\u0099" is valid JSON, "\xc5\xbe\xc5\x99" is not. Also

A working Ruby and Python (not verified) solution can be seen in the referenced StackOverflow question at the bottom.

The task is to fix JSON generated by Facebook with the shortest code.

Winning criteria: code-golf - the shortest code.

As input you can take a file name, STDIN or the file text as a variable. The output should be on STDOUT, a file name or a string in a variable.

You don't have precise JSON input format (except that it is valid JSON), because exported data can be different. Also result should be valid JSON.

Solutions

• should accept as input both - pretty and non-pretty formatted JSON
• should handle properly 1,2,3 and 4 octets UTF character. With the assurance (if that matters) that any multi-byte UTF characters in the input will always be split into multiple \u00## sequences (i.e. you will never get \u017E or another sequence where first byte is non-zero).
• can output any - pretty formatted or non-pretty JSON
• doesn't matter whether characters in the output are represented by plain Unicode characters or escaped Unicode \u017E
• you are allowed to use any JSON parsing libraries

Example input:

{
   "name": "\u00e0\u00a4\u0085\u00e0\u00a4\u0082\u00e0\u00a4\u0095\u00e0\u00a4\u00bf\u00e0\u00a4\u00a4 G \u00e0\u00a4\u00b6\u00e0\u00a5\u0081\u00e0\u00a4\u0095\u00e0\u00a5\u008d\u00e0\u00a4\u00b2\u00e0\u00a4\u00be",
   "timestamp": 1599213799,
   "whatever": "someth\u0022\u00d0\u0092\u00d0\u00b8\u00d0\u00ba\u00d0\u00b8",
   "test": "\\u0041"
}

Example acceptable output:

{
  "name": "अंकित G शुक्ला",
  "timestamp": 1599213799,
  "whatever": "someth\"Вики",
  "test": "\\u0041"
}

Credit goes to original question https://stackoverflow.com/q/52747566/520567

Answer 1

JavaScript, 85 bytes

let d=s=>new TextDecoder().decode(new Uint8Array(s.split('').map(r=>r.charCodeAt())))

Test:

let s = "nejni\u00c5\u00be\u00c5\u00a1\u00c3\u00ad bod: 0 mnm Ben\u00c3\u00a1tky\n";
s = d(s);
console.log(s);

As I said in the other page [1], the Facebook data is valid UTF-8. However JavaScript doesnt use UTF-8, it uses UTF-16. So you have to convert from UTF-8 to UTF-16, to get the text to play nice with JavaScript.

1. https://stackoverflow.com/a/68237082

Answer 2

APL (Dyalog Unicode), 51 bytes

Anonymous prefix lambda taking a string as argument.

'(\\u00..)+'⎕R{'UTF-8'⎕UCS⎕UCS⎕JSON'"',⍵.Match,'"'}

Try it!

'(\\u00..)+'⎕R{…} Replace any sequence of \u00 followed by two other characters with:

 '"',⍵.Match,'"' the match, surrounded by double quotes

 ⎕JSON interpret as JSON

 ⎕UCS convert to code points

 'UTF-8'⎕UCS interpret as UTF-8 bytes

Answer 3

Javascript, 79 bytes

f = 

s=>s.replace(/(\\u00[8-f].)+|\\./g,c=>c[2]?decodeURI(c.replace(/.u00/g,'%')):c)

; function ch() { try { console.log(f(val.value)) } catch (e) { console.log(e); } } ch();

<textarea id="val" onchange="ch()">
{
   "name": "\u00e0\u00a4\u0085\u00e0\u00a4\u0082\u00e0\u00a4\u0095\u00e0\u00a4\u00bf\u00e0\u00a4\u00a4 G \u00e0\u00a4\u00b6\u00e0\u00a5\u0081\u00e0\u00a4\u0095\u00e0\u00a5\u008d\u00e0\u00a4\u00b2\u00e0\u00a4\u00be",
   "timestamp": 1599213799,
   "whatever": "someth\u0022\u00d0\u0092\u00d0\u00b8\u00d0\u00ba\u00d0\u00b8",
   "test": "\\u0041"
}
</textarea>

Still use [8-f] rather than [^2] in case \u000a exist

Answer 4

Bash, 15 bytes

jq .|iconv -tl1

TIO version of jq requires the identity filter . but later versions can get away with jq|iconv -tl1 for 13 bytes

Try it online!

Answer 5

Perl 5 (-p), 34 bytes

s/(?<!\\)\\u00([^2].)/chr hex$1/ge

Try it online!

Answer 6

JavaScript (Node.js), 79 bytes

s=>JSON.stringify(eval('s='+s),(k,v)=>v===v+''?decodeURIComponent(escape(v)):v)

Try it online!

It only fix non printable ascii strings in JSON value, but not JSON keys. But as we accept JSON exported from Facebook which doesn't handle non ASCII correctly, we may safely assume keys won't contain such strings by design.

Answer 7

Retina, 239 213 bytes

/\\\\|\\u00[cde].(\\u00[89ab].)+/_(/\w/_(`[89a-f]
g$&
T`89l`d0
.
*
)4+`^(_*)\1(_)?
$1$#2
\\(0{7}(0{5})110(.{5})|0{12}1110(.{4})\\0{12}10(.{6}))\\0{12}10(.{6})
\u$2$3$4$5$6
/\d{4}/_(`1
01
+`10
011
C`1
T`d`l`..
b\B

Try it online! Retina isn't really designed for bit-twiddling. Outputs in escaped Unicode. Explanation:

/\\\\|\\u00[cde].(\\u00[89ab].)+/_(`

Run the rest of the script on pairs of backslashes and Unicode escapes of UTF-8 byte sequences. (The rest of the script leaves pairs of backslashes unchanged so this simply avoids treating them as the start of a Unicode escape.)

/\w/_(`[89a-f]
g$&
T`89l`d0
.
*
)4+`^(_*)\1(_)?
$1$#2

Change each letter into its binary equivalent. (This also changes u to 0000 but that doesn't matter a lot.)

\\(0{7}(0{5})110(.{5})|0{12}1110(.{4})\\0{12}10(.{6}))\\0{12}10(.{6})
\u$2$3$4$5$6

Shuffle the bits from the UTF-8 bytes into a single Unicode character.

/\d{4}/_(`

Run the rest of the script on each set of four bits.

1
01
+`10
011
C`1

Convert to decimal.

T`d`l`..

Change 10-15 to ba-bf.

b\B

Remove any leading b.

The previous version converted from hexadecimal to binary in a single substitution, but I was able to shave off 26 bytes by using a per-digit script:

/\w/_(`
)`

Run the script on each letter i.e. hexadecimal digit or u.

[89a-f]
g$&

Prepend g to the digits a-f.

T`89l`d0

Subtract 8 from the digits 8-f, but turn g into 8 as compensation; also turn u into 0.

.
*

Convert each digit to unary.

4+`^(_*)\1(_)?
$1$#2

Convert the unary to four bits of binary.