13
\$\begingroup\$

UTF-9 was an April Fool's Day RFC joke specifications for encoding Unicode suitable for 9-bit nonet platforms.

Challenge

Your task is to implement a program or a function or a subroutine that takes one integer, who represents the codepoint of a character, to return a sequence of integers that represents its corresponding UTF-9 value.

Definition of UTF-9

In this problem, the definition of the UTF-9 shall be as follows:

From section 3 of RFC 4042:

A UTF-9 stream represents [ISO-10646] codepoints using 9 bit nonets. The low order 8-bits of a nonet is an octet, and the high order bit indicates continuation.

UTF-9 does not use surrogates; consequently a UTF-16 value must be transformed into the UCS-4 equivalent, and U+D800 - U+DBFF are never transmitted in UTF-9.

Octets of the [UNICODE] codepoint value are then copied into successive UTF-9 nonets, starting with the most-significant non-zero octet. All but the least significant octet have the continuation bit set in the associated nonet.

Constraints

Input is nonnegative integers that is defined on Unicode: which is 0 to 0x7FFFFFFF (inclusive)

Rules

  • Standard loopholes apply.
  • Standard I/O rules apply.
  • Shortest code wins.

Test cases

Leftmost column: Input in hexadecimal.

Right items: output as a sequence of octed integers.

0000     000
0041     101
00c0     300
0391     403 221
611b     541 033
10330    401 403 060
e0041    416 400 101
10fffd   420 777 375
345ecf1b 464 536 717 033

Hint Notes

Section 5.2 of RFC4042 has an example of the impmenentation.

Here is a non-competing Python function that represents the algorithm:

def ucs4utf9(x:int)->[int]:
 l=[]
 if x>0x100:
  if x>0x10000:
   if x>0x1000000:
    l.append(0x100|(x>>24)&0xff)
   l.append(0x100|(x>>16)&0xff)
  l.append(0x100|(x>>8)&0xff)
 l.append(x&0xff)
 return l

Test it online!

Those two programs were incorrect, as they convert codepoints 0x100, 0x10000, and 0x1000000 incorrectly; they should be converted to 257,0, 257,256,0, 257,256,256,0 respectively.

\$\endgroup\$
6
  • \$\begingroup\$ Sandbox. \$\endgroup\$
    – user100411
    Commented Nov 15, 2021 at 11:45
  • 3
    \$\begingroup\$ The sample code both here and in the RFC does not actually match the description in the overview for the code points 0x100 and 0x10000 (and 0x1000000, although that's not a valid Unicode code point). \$\endgroup\$
    – Neil
    Commented Nov 15, 2021 at 12:05
  • \$\begingroup\$ @Neil Thank you; I may need to move back to the sandbox again. \$\endgroup\$
    – user100411
    Commented Nov 15, 2021 at 12:23
  • \$\begingroup\$ Also I noticed the description of the RFC itself has some errors such as mismatching of section name and function. Also in section 2, a duplicate can be seen: codepoints in the range U+0100 - U+FFFF … are represented by two nonets; and codepoints in the range U+1000 - U+10FFFF … are represented by three nonets. \$\endgroup\$
    – user100411
    Commented Nov 15, 2021 at 12:30
  • 2
    \$\begingroup\$ Suggest add testcase where input is 0. \$\endgroup\$
    – tsh
    Commented Nov 16, 2021 at 6:24

12 Answers 12

7
\$\begingroup\$

Jelly, 8 bytes

b⁹+⁹_0¦⁹

Try it online!

Explanation

b⁹+⁹_0¦⁹
b        Convert to base
 ⁹         256
  +      Add
   ⁹       256
      ¦  At index
     0     last
    _      subtract
       ⁹   256
\$\endgroup\$
6
\$\begingroup\$

Python 3, 104 103 97 bytes

def f(n):b=f'{0:08}{n:b}';l=len(b);return[256*(i<l-8)+int(b[i:i+8],2)for i in range(l%8or 8,l,8)]

Try it online!

-1 thanks to Kevin Cruijssen

Explanation:

  • The program converts the number to binary and left-pads it with 8 zeroes
  • It takes each full chunk of 8 bytes (discarding additional zeroes at the start)
  • The chunk is converted from a binary string to an int. 256 is added if it isn't the last chunk.
\$\endgroup\$
0
6
\$\begingroup\$

INTERCAL, 164 162 154 bytes

I made this program before noticing the first comment of the OP. In my comment section the two example programs on both RFC document and my Python implementation says values 0x100, 0x10000, and 0x1000000 are converted incorrectly, while this program converts 256 to [257,0]; which seems to be correct.

DOWRITEIN:1DOCOMEFROM.3PLEASE.1<-!2$:1~#255'~#54613DOSTASH.1DO.2<-#128PLEASE:1<-:1~#65520$#65520(1)DO.3<-':1~:1'~#1(9)DORETRIEVE.1DOREADOUT.1PLEASE(9)NEXT

Try it online!

Usage

  • Input as INTERCAL-72 integer input (e.g. ONE TWO THREE FOUR FIVE).
  • Outputs several integers as roman numbers, from first to last elements of the list.

Ungolfed

DOWRITEIN:1

DONOTE from label (1) if .3 got at least one bit of one
DOCOMEFROM.3
DONOTE :1~#255 is done first on CLCI
PLEASENOTE obtain lowest 8 bits
DONOTE then do C operation such twospot[1]|spot[2]
DONOTE spot[2] is 256 if second time to visit here 0 otherwise
PLEASE.1<-!2$:1~#255'~#54613
PLEASENOTE this and RETRIEVE statements below
DOSTASH.1
DO.2<-#128
DONOTE like (twospot[1]>>8)
PLEASE:1<-:1~#65520$#65520
(1)DO.3<-':1~:1'~#1

PLEASENOTE finally, until no more .1 stacks
(9)DORETRIEVE.1
DOREADOUT.1
PLEASE(9)NEXT

Algorithm

  1. Write in to the codepoint.
  2. Make a list from backward, using STASH and RETRIEVE statements and a stack.
  3. Then output each item.
\$\endgroup\$
5
\$\begingroup\$

JavaScript (ES6), 41 bytes

If returning an empty array for \$n=0\$ is not acceptable, this version returns [0] instead. Thanks to @tsh for pointing this out.

f=(n,q)=>n|!q?[...f(n>>8,256),n%256|q]:[]

Try it online!


JavaScript (ES6), 38 bytes

f=(n,q)=>n?[...f(n>>8,256),n%256|q]:[]

Try it online!

\$\endgroup\$
2
  • \$\begingroup\$ To my understanding, the expected output for 0 should be [0] \$\endgroup\$
    – tsh
    Commented Nov 16, 2021 at 6:24
  • \$\begingroup\$ @tsh Good point. Waiting for OP confirmation, but I've added a version supporting that. \$\endgroup\$
    – Arnauld
    Commented Nov 16, 2021 at 9:41
5
\$\begingroup\$

R, 51 bytes

f=function(x,s=0,b=256)c(if(x>=b)f(x%/%b,b),x%%b+s)

Try it online!

\$\endgroup\$
4
\$\begingroup\$

Python 3.8, 81 bytes

def f(n):x=n.to_bytes(4,'big');return[i+256 for i in x.lstrip(b'\0')[:-1]]+[x[-1]]
\$\endgroup\$
3
\$\begingroup\$

05AB1E (legacy), 7 bytes

₁вā€₁¨+

Port of @xigoi's Jelly answer.

I/O as integers.

Try it online or verify all test cases.

Explanation:

₁в       # Convert the (implicit) input-integer to base-256 as list
  ā      # Push a list in the range [1,length] (without popping)
   ¨     # Remove the final item to make the range [1,length)
    €₁   # Map each to 256
         # (we now have a list of 256s of size length-1)
      +  # Add the values at the same positions in the lists together
         # (where the larger list keeps its trailing items as is)
         # (after which this list is output implicitly as result)

In the new version of 05AB1E both €₁ and + wouldn't work here. €₁ will add a 256 before each item instead of replacing them (an alternative could be Ā₁*), and + will cause the final item to be removed, since it will adjust the list to the shortest. In 05AB1E it therefore requires a different approach, which would be 8 bytes instead:

05AB1E, 8 bytes

₁в₁+`₁-)

Try it online or verify all test cases.

Explanation:

₁в       # Convert the (implicit) input-integer to base-256 as list
  ₁+     # Add 256 to each
    `    # Pop and dump the contents of the list separated to the stack
     ₁-  # Subtract 256 from the top item (the last item of the dumped list)
       ) # Wrap the entire stack into a list again
         # (after which this list is output implicitly as result)
\$\endgroup\$
3
\$\begingroup\$

Charcoal, 20 bytes

I⮌E⮌∨↨N²⁵⁶⟦⁰⟧⁺ι∧κ²⁵⁶

Try it online! Link is to verbose version of code. I/O is in decimal. Would have been 16 bytes if empty output was acceptable for an input of 0. Explanation:

      N                 Input as an integer
     ↨ ²⁵⁶              Converted to base 256 as a list
    ∨                   Logical Or
          ⟦⁰⟧           List of literal `0`
   ⮌                    Reversed i.e. LSB first
  E                     Map over bytes
              ι         Current byte
             ⁺          Plus
                κ       Current index
               ∧        Logical And
                 ²⁵⁶    Literal 256
 ⮌                      Reversed i.e. MSB first again
I                       Cast to string
                        Implicitly print

Converting to groups of 8 bits also takes 20 bytes:

I⮌E⪪⮌⍘N²¦⁸⁺⍘⮌ι²∧κ²⁵⁶

Try it online! Link is to verbose version of code. I/O is in decimal. Explanation:

      N                 Input as an integer
     ⍘ ²                Converted to base 2 as a string
    ⮌                   Reversed i.e. LSB first
   ⪪     ⁸              Split into groups of up to 8 bits
  E                     Map over bytes
             ι          Current byte
            ⮌           Reversed i.e. MSB first again
           ⍘  ²         Converted from base 2
          ⁺             Plus
                κ       Current index
               ∧        Logical And
                 ²⁵⁶    Literal 256
 ⮌                      Reversed
I                       Cast to string
                        Implicitly print
\$\endgroup\$
3
\$\begingroup\$

Retina 0.8.2, 71 bytes

.+
$*
+`(1+)\1
$+0
01
1
^
7$*0
r`.{8}
¶$&
1A`
.+¶
1$&
1
01
+`10
011
%`1

Try it online! Link includes test suite. I/O is in decimal. Explanation:

.+
$*

Convert to unary.

+`(1+)\1
$+0
01
1

Convert to binary.

^
7$*0
r`.{8}
¶$&
1A`

Split into bytes of 8 bits.

.+¶
1$&

Prefix a 1 to all bytes except the last.

1
01
+`10
011

Convert to unary.

%`1

Convert to decimal.

\$\endgroup\$
2
\$\begingroup\$

Ruby, 62 42 bytes

f=->n,w=0{w+=n%z=256;n<z ?[w]:f[n/z,z]<<w}

Try it online!

\$\endgroup\$
1
\$\begingroup\$

Perl 5, 59 bytes

sub{my@o;do{unshift@o,$_[0]%256|256*!!@o}while$_[0]>>=8;@o}
  • do-while loop to make sure that 0 outputs (0) and not ()
  • unshift pushes onto the array backwards
  • !!@o is 0 if the output is empty (this will be the last nonet) or 1 otherwise
  • consumes its input
  • could lose my@o; for -5 points under the old-school Perl golf rules.

TIO.

\$\endgroup\$
1
\$\begingroup\$

Rust, 93 bytes

|n|(0..4).rev().map(|i|n>>i*8&255|(i>0)as i32*256).skip_while(|&n|n==256).collect::<Vec<_>>()

Try it online!

Ungolfed

|n| {
    (0..4).rev()                 // Iterator over integers from 3 down to 0
    .map(|i|
        (n >> i*8) & 255         // Extract the i'th octet from n
        | ((i > 0) as i32) * 256 // Set the continuation bit if not the last byte
    )
    .skip_while(|&n| n == 256)   // Remove 0x100 bytes at the start
    .collect::<Vec<_>>()         // Evaluate into a vector
}
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.