8
\$\begingroup\$

When you export data in JSON from Facebook, the multi-byte Unicode characters are broken. Example Unicode characters and their representation inside JSON:

'\u00c2\u00b0' : '°',
'\u00c3\u0081' : 'Á',
'\u00c3\u00a1' : 'á',
'\u00c3\u0089' : 'É',
'\u00c3\u00a9' : 'é',
'\u00c3\u00ad' : 'í',
'\u00c3\u00ba' : 'ú',
'\u00c3\u00bd' : 'ý',
'\u00c4\u008c' : 'Č',
'\u00c4\u008d' : 'č',
'\u00c4\u008f' : 'ď',
'\u00c4\u009b' : 'ě',
'\u00c5\u0098' : 'Ř',
'\u00c5\u0099' : 'ř',
'\u00c5\u00a0' : 'Š',
'\u00c5\u00a1' : 'š',
'\u00c5\u00af' : 'ů',
'\u00c5\u00be' : 'ž',

In the last example the bytes 0xC5 and 0xBE are the UTF-8 encoding for U+017E. If those individual bytes are treated as unicode codepoints, they are Å and ¾. So Facebook wrote each byte as an Unicode codepoint instead of handling multi-byte UTF-8 sequences appropriately.

This incorrect representation is used for every multi-byte UTF character. In short, to fix it, one needs to convert any string sequences like \u00c5\u00be\u00c5\u0099 into byte data where first byte is \xc5 and second byte is \xbe, third is \xc5 and fourth is \x99, then read these bytes as a UTF-8 string. This can happen by direct manipulation of raw JSON text or by first parsing JSON, then do some magic with the incorrectly parsed strings or whatever comes to your mind.

Note that while "\u00c5\u00be\u00c5\u0099" is valid JSON, "\xc5\xbe\xc5\x99" is not. Also

A working Ruby and Python (not verified) solution can be seen in the referenced StackOverflow question at the bottom.

The task is to fix JSON generated by Facebook with the shortest code.

Winning criteria: code-golf - the shortest code.

As input you can take a file name, STDIN or the file text as a variable. The output should be on STDOUT, a file name or a string in a variable.

You don't have precise JSON input format (except that it is valid JSON), because exported data can be different. Also result should be valid JSON.

Solutions

  • should accept as input both - pretty and non-pretty formatted JSON
  • should handle properly 1,2,3 and 4 octets UTF character. With the assurance (if that matters) that any multi-byte UTF characters in the input will always be split into multiple \u00## sequences (i.e. you will never get \u017E or another sequence where first byte is non-zero).
  • can output any - pretty formatted or non-pretty JSON
  • doesn't matter whether characters in the output are represented by plain Unicode characters or escaped Unicode \u017E
  • you are allowed to use any JSON parsing libraries

Example input:

{
   "name": "\u00e0\u00a4\u0085\u00e0\u00a4\u0082\u00e0\u00a4\u0095\u00e0\u00a4\u00bf\u00e0\u00a4\u00a4 G \u00e0\u00a4\u00b6\u00e0\u00a5\u0081\u00e0\u00a4\u0095\u00e0\u00a5\u008d\u00e0\u00a4\u00b2\u00e0\u00a4\u00be",
   "timestamp": 1599213799,
   "whatever": "someth\u0022\u00d0\u0092\u00d0\u00b8\u00d0\u00ba\u00d0\u00b8",
   "test": "\\u0041"
}

Example acceptable output:

{
  "name": "अंकित G शुक्ला",
  "timestamp": 1599213799,
  "whatever": "someth\"Вики",
  "test": "\\u0041"
}

Credit goes to original question https://stackoverflow.com/q/52747566/520567

\$\endgroup\$
5
  • \$\begingroup\$ I actually ran into this when doing a certain small project. I fixed it by doing s.encode("latin-1").decode("utf-8"). \$\endgroup\$
    – xigoi
    Jul 1, 2021 at 19:46
  • \$\begingroup\$ @xigoi, looks fantastic. Maybe write an answer. \$\endgroup\$ Jul 1, 2021 at 20:34
  • \$\begingroup\$ It's only for a single string, so to make it work for the whole JSON, I'd have to recursively descend into it, which seems quite long-winded… but maybe it's not. \$\endgroup\$
    – xigoi
    Jul 1, 2021 at 20:49
  • 2
    \$\begingroup\$ May I assume the non-printable ASCII string only occurred as string values not object keys? For example, we have {"key":" \u00E4\u00BE\u008B\u00E5\u00AD\u0090"}, but never {" \u00E4\u00BE\u008B\u00E5\u00AD\u0090":"value"} \$\endgroup\$
    – tsh
    Jul 2, 2021 at 1:13
  • \$\begingroup\$ @tsh, I think so. I don't see Facebook using non-Latin letters in keys. \$\endgroup\$ Jul 5, 2021 at 8:15

7 Answers 7

2
\$\begingroup\$

JavaScript, 85 bytes

let d=s=>new TextDecoder().decode(new Uint8Array(s.split('').map(r=>r.charCodeAt())))

Test:

let s = "nejni\u00c5\u00be\u00c5\u00a1\u00c3\u00ad bod: 0 mnm Ben\u00c3\u00a1tky\n";
s = d(s);
console.log(s);

As I said in the other page [1], the Facebook data is valid UTF-8. However JavaScript doesnt use UTF-8, it uses UTF-16. So you have to convert from UTF-8 to UTF-16, to get the text to play nice with JavaScript.

  1. https://stackoverflow.com/a/68237082
\$\endgroup\$
2
\$\begingroup\$

APL (Dyalog Unicode), 51 bytes

Anonymous prefix lambda taking a string as argument.

'(\\u00..)+'⎕R{'UTF-8'⎕UCS⎕UCS⎕JSON'"',⍵.Match,'"'}

Try it!

'(\\u00..)+'⎕R{}Replace any sequence of \u00 followed by two other characters with:

'"',⍵.Match,'"' the match, surrounded by double quotes

⎕JSON interpret as JSON

⎕UCS convert to code points

'UTF-8'⎕UCS interpret as UTF-8 bytes

\$\endgroup\$
1
  • 1
    \$\begingroup\$ This is really cool, sometimes the toolset APL brings to the table just blows my mind. \$\endgroup\$
    – Etheryte
    Jul 3, 2021 at 17:40
2
\$\begingroup\$

Javascript, 79 bytes

f = 

s=>s.replace(/(\\u00[8-f].)+|\\./g,c=>c[2]?decodeURI(c.replace(/.u00/g,'%')):c)

; function ch() { try { console.log(f(val.value)) } catch (e) { console.log(e); } } ch();
<textarea id="val" onchange="ch()">
{
   "name": "\u00e0\u00a4\u0085\u00e0\u00a4\u0082\u00e0\u00a4\u0095\u00e0\u00a4\u00bf\u00e0\u00a4\u00a4 G \u00e0\u00a4\u00b6\u00e0\u00a5\u0081\u00e0\u00a4\u0095\u00e0\u00a5\u008d\u00e0\u00a4\u00b2\u00e0\u00a4\u00be",
   "timestamp": 1599213799,
   "whatever": "someth\u0022\u00d0\u0092\u00d0\u00b8\u00d0\u00ba\u00d0\u00b8",
   "test": "\\u0041"
}
</textarea>

Still use [8-f] rather than [^2] in case \u000a exist

\$\endgroup\$
1
\$\begingroup\$

Bash, 15 bytes

jq .|iconv -tl1

TIO version of jq requires the identity filter . but later versions can get away with jq|iconv -tl1 for 13 bytes

Try it online!

\$\endgroup\$
1
  • \$\begingroup\$ I can't miss to highlight this answer by accepting it as it requires very little effort for anybody to use even without coding skills. Also it's short and at the same time legible. Not sure it is a good practice to accept an answer in a code-golf challenge. But I think in this instance it is well deserves. Without undermining all the other awesome answers of course. \$\endgroup\$ Jul 5, 2021 at 8:19
1
\$\begingroup\$

Perl 5 (-p), 34 bytes

s/(?<!\\)\\u00([^2].)/chr hex$1/ge

Try it online!

\$\endgroup\$
1
\$\begingroup\$

JavaScript (Node.js), 79 bytes

s=>JSON.stringify(eval('s='+s),(k,v)=>v===v+''?decodeURIComponent(escape(v)):v)

Try it online!

It only fix non printable ascii strings in JSON value, but not JSON keys. But as we accept JSON exported from Facebook which doesn't handle non ASCII correctly, we may safely assume keys won't contain such strings by design.

\$\endgroup\$
3
  • \$\begingroup\$ Assuming all values are either number or string, 1/v is true if v is number(or number-like string) and false otherwise. If it's number-like, then there's no \x80-\xff chars and can skip convertion \$\endgroup\$
    – l4m2
    Jul 2, 2021 at 11:44
  • \$\begingroup\$ Also your code fails if input contain # \$\endgroup\$
    – l4m2
    Jul 2, 2021 at 11:49
  • \$\begingroup\$ @l4m2 that wont work for large number, nested object, null. sharp should got fixed \$\endgroup\$
    – tsh
    Jul 2, 2021 at 15:54
0
\$\begingroup\$

Retina, 239 213 bytes

/\\\\|\\u00[cde].(\\u00[89ab].)+/_(/\w/_(`[89a-f]
g$&
T`89l`d0
.
*
)4+`^(_*)\1(_)?
$1$#2
\\(0{7}(0{5})110(.{5})|0{12}1110(.{4})\\0{12}10(.{6}))\\0{12}10(.{6})
\u$2$3$4$5$6
/\d{4}/_(`1
01
+`10
011
C`1
T`d`l`..
b\B

Try it online! Retina isn't really designed for bit-twiddling. Outputs in escaped Unicode. Explanation:

/\\\\|\\u00[cde].(\\u00[89ab].)+/_(`

Run the rest of the script on pairs of backslashes and Unicode escapes of UTF-8 byte sequences. (The rest of the script leaves pairs of backslashes unchanged so this simply avoids treating them as the start of a Unicode escape.)

/\w/_(`[89a-f]
g$&
T`89l`d0
.
*
)4+`^(_*)\1(_)?
$1$#2

Change each letter into its binary equivalent. (This also changes u to 0000 but that doesn't matter a lot.)

\\(0{7}(0{5})110(.{5})|0{12}1110(.{4})\\0{12}10(.{6}))\\0{12}10(.{6})
\u$2$3$4$5$6

Shuffle the bits from the UTF-8 bytes into a single Unicode character.

/\d{4}/_(`

Run the rest of the script on each set of four bits.

1
01
+`10
011
C`1

Convert to decimal.

T`d`l`..

Change 10-15 to ba-bf.

b\B

Remove any leading b.

The previous version converted from hexadecimal to binary in a single substitution, but I was able to shave off 26 bytes by using a per-digit script:

/\w/_(`
)`

Run the script on each letter i.e. hexadecimal digit or u.

[89a-f]
g$&

Prepend g to the digits a-f.

T`89l`d0

Subtract 8 from the digits 8-f, but turn g into 8 as compensation; also turn u into 0.

.
*

Convert each digit to unary.

4+`^(_*)\1(_)?
$1$#2

Convert the unary to four bits of binary.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.