The goal is to create a fully compliant converter between the official Unicode encodings as given in the UTF FAQ. Given that this is centred on Unicode, I will accept the answer with the lowest byte count using the best possible of the involved encodings (which will probably be UTF-8, unless maybe you program it in APL). I apologize for the long post, but a lot of it is explaining the encodings which can also be accessed in the official specification (pdf, section 3.9 D90 - D92), or Wikipedia.
If at any time your language of choice cannot exactly meet a requirement, substitute it with something that sticks the spirit of the rules given. Eg. not every language has built-in arrays, functions etc.
No using string libraries/functions, or encoding libraries/functions. The point of this code golf is to implement the converter using bit/byte manipulation. Using strings themselves in their capacity as a character or byte array is allowed though. Oh, and no OS calls which perform the conversion either.
The converter is a function which will take three parameters: a byte array representing the encoded input string, and the "input" and "output" encodings represented as numbers. Arbitrarily we will assign
UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF32LEnumbers from 0 to 6 in that order. There is no need to check if the number is
> 6, we will assume these parameters are correct. The converter will return a valid byte array in the desired output encoding.
We will use the null character (
U+0000) as a string terminator. Anything after this doesn't matter. We will assume that the input array has the null character somewhere so you do not need to do a bounds check.
As per the FAQ, if the input byte array is invalid for its declared encoding, we must signal an error. We will do this in one of the following ways: crash the program, throw an exception, return null or return an array whose first four bytes are all 0 (so that it can be recognized as
U+0000in every encoding).
The official specifications must be followed, but Wikipedia provides a good (and as far as I believe correct) explanation of the encodings, and I will summarize them here for completeness. Note that UTF-16 and UTF-32 have variants for endianness.
UTF-32, UTF-32LE, UTF-32BE
The simplest encoding, each code point is simply encoded in 4 bytes equal to its numeric value. LE/BE represents endianness (little endian/big endian).
UTF-16, UTF-16LE, UTF-16BE
Code points from
U+0000 - U+FFFF are encoded in 2 bytes equal to its numeric value. Larger values are encoded using a pair of surrogates which are reserved values from
U+D800 - U+DFFF. So to encode points greater than
U+FFFF, the following algorithm can be used (shamelessly copied from Wikipedia):
- 0x010000 is subtracted from the code point, leaving a 20 bit number in the range 0..0x0FFFFF.
- The top ten bits (a number in the range 0..0x03FF) are added to 0xD800 to give the first code unit or lead surrogate, which will be in the range 0xD800..0xDBFF [...].
- The low ten bits (also in the range 0..0x03FF) are added to 0xDC00 to give the second code unit or trail surrogate, which will be in the range 0xDC00..0xDFFF [...].
Code points from
U+0000 - U+007F are encoded as 1 byte equal to its numeric value. From
U+0080 - U+07FF they are encoded as
U+0800 - U+FFFF is
1110xxxx 10xxxxxx 10xxxxxx, higher values are
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx. The
x's are the bits from the numeric value of the code point.
The byte-order mark (BOM,
U+FEFF) is used as the first code point to indicate endianness. Following the FAQ guidelines on BOMs, the BOM will be used as follows: For
UTF-8, UTF-16 and UTF-32 it is optional. If the BOM is absent in
UTF-32, it is assumed to be big endian. The BOM must not appear in
UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE.
Common Pitfalls Causing Invalid UTF
Various things may cause a byte sequence to be invalid UTF.
- UTF-8 and UTF-32: Directly encoding surrogate code points (
U+D800 - U+DFFF), or code points greater than
- UTF-8: Many invalid byte sequences.
- UTF-16: Unpaired, or improperly paired surrogates.
- BOM: Must be used as specified in the encoding section. Note that when outputting
UTF-32(no inherent endianness specified) you can pick, but with little endian, you must include the BOM.
Note that non-characters and unassigned code points (both distinct from surrogates) are to be treated like regular characters.
''⎕R''⍠'InEnc' 'UTF16BE' 'OutEnc' 'UTF8-BOM'. \$\endgroup\$