Is it Valid Modified UTF-8?

Question

UTF-8 is a relatively simple way to encode Unicode codepoints in a variable-width format such that it doesn't easily confuse code that isn't Unicode aware.

UTF-8 overview

• Bytes in the range of 1-0x7F, inclusive, are normally valid
• Bytes with the bit pattern 10XX XXXX are considered continuation bytes, with the six least significant bits being used to encode part of a codepoint. These must not appear unless they are expected by a preceding byte.
• Bytes with the pattern 110X XXXX expect one continuation byte afterward
• Bytes with the pattern 1110 XXXX expect two continuation bytes afterward
• Bytes with the pattern 1111 0XXX expect three continuation bytes afterward
• All other bytes are invalid and should not appear anywhere in a UTF-8 stream. 5, 6, and 7 byte clusters are possible in theory, but will not be allowed for the purposes of this challenge.

Overlong encodings

UTF-8 also requires that a codepoint should be represented with the minimum number of bytes. Any byte sequence that could be represented with fewer bytes is not valid. Modified UTF-8 adds one exception to this for null characters (U+0000), which should be represented as C0 80 (hex representation)), and instead disallows null bytes to appear anywhere in the stream. (This makes it compatible with null-terminated strings)

Challenge

You are to make a program that, when given a string of bytes, will determine if that string represents valid Modified UTF-8 and will return a truthy value if valid and a falsy value otherwise. Note that you must check for overlong encodings and null bytes (since this is Modified UTF-8). You do not need to decode the UTF-8 values.

Examples

41 42 43  ==> yes (all bytes are in the 0-0x7F range)
00 01 02  ==> no (there is a null byte in the stream)
80 7F 41  ==> no (there is a continuation byte without a starter byte)
D9 84 10  ==> yes (the correct number of continuation bytes follow a starter byte)
F0 81 82 41  ==> no (there are not enough continuation bytes after F0)
EF 8A A7 91  ==> no (too many continuation bytes)
E1 E1 01  ==> no (starter byte where a continuation byte is expected)
E0 80 87  ==> no (overlong encoding)
41 C0 80  ==> yes (null byte encoded with the only legal overlong encoding)
F8 42 43  ==> no (invalid byte 'F8')

Rules

• Standard rules and loopholes apply
• Input and output can be in any convenient format as long as all values in the unsigned byte range (0-255) can be read in.
  • You may need to use an array or file rather than a null-terminated string. You need to be able to read null bytes.
• Shortest code wins!
• Note that using builtins to decode the UTF-8 is not guaranteed to conform to the requirements given here. You may need to work around it and create special cases.

EDIT: added bonus for not using builtins that decode UTF-8

EDIT2: removed bonus since only the Rust answer qualified and it's awkward to define.

Answer 1

Elixir, 69 bytes

import String
&valid? replace replace(&1,<<0>>,"\xFF"),"\xC0\x80","0"

Try it online!

Makes use of built-in string validation function. Takes input as Elixir binary.

Answer 2

APL (Dyalog Unicode), 41 39 bytes^SBCS

Anonymous tacit prefix function. Takes a Unicode string as argument where the characters' code points represent the input bytes.

{0::0⋄×⌊/'UTF-8'⎕UCS⍣2⎕UCS⍵}'À\x80'⎕R⎕A

Try it online!

'À\x80'⎕R⎕A Replace C0 80s with the uppercase Alphabet

{…} apply the following anonymous function, where the argument is ⍵:

 0:: if any error happens:

  0 return zero

 ⋄ try:

  ⎕UCS⍵ convert the string to code points

  'UTF-8'⎕UCS⍣2 interpret as UTF-8 bytes and convert resulting text back to bytes

  ⌊/ lowest byte (zero if a null byte is present, positive if not, "infinity" if empty string)

  × sign (zero if null byte is present, one if not)

Answer 3

Python 2, 104 102 bytes

''.join(chr(int(c,16))for c in input().replace('00','-').replace('C0 80','0').split()).decode('utf-8')

Try it online!

Outputs via exit code

Answer 4

Rust - 191 bytes 313 bytes

Per comment below original did not work properly. New and improved version. No libraries are used, because The Mighty Rust Has No Need For You And Your Libraries. This code uses pattern matching with a state machine. By shamelessly ripping off the UTF8 spec, after finding it via reference and discussion by Jon Skeet, we can copy the spec almost character for character into a rust Match pattern match block. At the end, we add in Beefster's special Mutf8 requirement for C0 80 to be considered valid. Ungolfed:

/* http://www.unicode.org/versions/corrigendum1.html
 Code Points        1st Byte    2nd Byte    3rd Byte    4th Byte
U+0000..U+007F      00..7F           
U+0080..U+07FF      C2..DF      80..BF           
U+0800..U+0FFF      E0          A0..BF      80..BF       
U+1000..U+FFFF      E1..EF      80..BF      80..BF       
U+10000..U+3FFFF    F0          90..BF      80..BF      80..BF
U+40000..U+FFFFF    F1..F3      80..BF      80..BF      80..BF
U+100000..U+10FFFF  F4          80..8F      80..BF      80..BF
*/

let m=|v:&Vec<u8>|v.iter().fold(0, |s, b| match (s, b) {
        (0, 0x01..=0x7F) => 0,
        (0, 0xc2..=0xdf) => 1,
        (0, 0xe0) => 2,
        (0, 0xe1..=0xef) => 4,
        (0, 0xf0) => 5,
        (0, 0xf1..=0xf3) => 6,
        (0, 0xf4) => 7,
        (1, 0x80..=0xbf) => 0,
        (2, 0xa0..=0xbf) => 1,
        (4, 0x80..=0xbf) => 1,
        (5, 0x90..=0xbf) => 4,
        (6, 0x80..=0xbf) => 4,
        (7, 0x80..=0x8f) => 4,
        (0, 0xc0) => 8, // beefster mutf8 null
        (8, 0x80) => 0, // beefster mutf8 null
        _ => -1,
    })==0;

try it on the rust playground