1
\$\begingroup\$

This question already has an answer here:

Given a UTF-8 string, give a string that represents byte counts of each characters.

Rules:

  1. As long as the input string is encoded in UTF-8 (without BOM), its type doesn't matter. In C++, it can be char[] or std::string. In Haskell, it can be [Int8] or [CChar].

  2. The output must be (in usual case) a string of ASCII digit characters. Its encoding doesn't matter.

  3. When given an invalid UTF-8 sequence (including representation of surrogates), every bytes of the sequence are to be represented as the replacement character �.

  4. Reserved characters and noncharacters are still considered as valid UTF-8 sequences, and thus must have corresponding output digit.

  5. As this is a code-golf, the shortest code in bytes wins.

Example:

When given the following byte sequence:

00 80 C0 80 EB 80 80

The output must be:

1���3
\$\endgroup\$

marked as duplicate by Adám code-golf Sep 29 at 7:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • \$\begingroup\$ Recommended test case: b'Hej d\xc3\xc5!' (i.e. 48 65 6A 20 64 C3 C5 21) (invalid continuation byte (C5) after a valid start byte (C3)) (expected output: 11111��1) \$\endgroup\$ – pizzapants184 Sep 29 at 6:42
  • 1
    \$\begingroup\$ Closely related \$\endgroup\$ – Adám Sep 29 at 7:25
1
\$\begingroup\$

Python 3, 135 bytes

def f(b):
	B=b.decode(errors="ignore")[:1].encode();l=len(B)
	if B and b[:l]!=B:return"�"+f(b[1:])
	return str(l)+f(b[l:])if b else''

Try it online!

Explanation:

def f(b):
	B=b.decode(errors="ignore")[:1].encode();l=len(B)
	 # B is the UTF-8-encoded version of the first Unicode character in the given bytes
	 # (Or the Nth, if the first N-1 bytes were invalid Unicode bytes)
	if B and b[:l]!=B:return"�"+f(b[1:])
	 # If the two encoded byte-strings don't match,
	 # then the first byte in b is invalid,
	 # so append "�" to the returned string, and recurse
	return str(l)+f(b[l:])if b else''
	 # If the byte-string is not empty,
	 #  append the length of the first character and recurse.
	 # Else (the input is empty) return the empty string (recursion base case)
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.