12
\$\begingroup\$

UTF-1 is one of the methods to transform ISO/IEC 10646 and Unicode into a sequence of bytes. It was originally for ISO 10646.

The UTF-1 encodes every character with variable length of byte sequence, just like UTF-8 does. It was designed to avoid control codes. But it was not designed to avoid duplication of a slash character ('/'), which is path delimiter for many operating systems. Additionally it takes a bit long to process the algorithm, because it is based on modulo 190, which is not power of two. Eventually UTF-8 was better than that.

In this challenge we are reviving UTF-1.

How it works

Every constant value in tables on this section is represented in hexadecimal value.

The symbol / is the integer division operator, % is the integer modulo operator, and ^ means power. The precedence of operators is ^ first, / second, % third, and others last. Here is how to convert each codepoint to UTF-1 (sorry for typo last time):

codepoint x UTF-1
U+0000 to U+009F (x)
U+00A0 to U+00FF (A0,x)
U+0100 to U+4015 y=x-100 in (A1+y/BE,T(y%BE))
U+4016 to U+38E2D y=x-4016 in (F6+y/BE^2,T(y/BE%BE),T(y%BE))
U+38E2E to U+7FFFFFFF y=x-38E2E in (FC+y/BE^4,T(y/BE^3%BE),T(y/BE^2%BE),T(y/BE%BE),T(y%BE))

T(z) is a function such that:

z T(z)
0 to 5D z+21
5E to BD z+42
BE to DE z-BE
DF to FF z-60

Challenge

Your task is to implement a program or a function or a subroutine that takes one integer, who represents the codepoint of a character, to return a sequence of integers that represents its corresponding UTF-1 value.

Constraints

Input shall be an nonnegative integer up to 0x7FFFFFFF.

Rules

  • Standard loopholes apply.
  • Standard I/O rules apply.
  • Shortest code wins.

Test cases

Taken from the Wikipedia article.

    U+007F  7F
    U+0080  80
    U+009F  9F
    U+00A0  A0 A0
    U+00BF  A0 BF
    U+00C0  A0 C0
    U+00FF  A0 FF
    U+0100  A1 21
    U+015D  A1 7E
    U+015E  A1 A0
    U+01BD  A1 FF
    U+01BE  A2 21
    U+07FF  AA 72
    U+0800  AA 73
    U+0FFF  B5 48
    U+1000  B5 49
    U+4015  F5 FF
    U+4016  F6 21 21
    U+D7FF  F7 2F C3
    U+E000  F7 3A 79
    U+F8FF  F7 5C 3C
    U+FDD0  F7 62 BA
    U+FDEF  F7 62 D9
    U+FEFF  F7 64 4C
    U+FFFD  F7 65 AD
    U+FFFE  F7 65 AE
    U+FFFF  F7 65 AF
   U+10000  F7 65 B0
   U+38E2D  FB FF FF
   U+38E2E  FC 21 21 21 21
   U+FFFFF  FC 21 37 B2 7A
  U+100000  FC 21 37 B2 7B
  U+10FFFF  FC 21 39 6E 6C
U+7FFFFFFF  FD BD 2B B9 40
\$\endgroup\$
6
  • 1
    \$\begingroup\$ for the 3rd formula, is the x in (A1+x/BE,T(y%BE)) a typo? Did you mean A1+y/BE? \$\endgroup\$
    – Mayube
    Nov 22 at 14:56
  • \$\begingroup\$ @mayube Thank you, it REALLY was. \$\endgroup\$ Nov 22 at 20:20
  • 1
    \$\begingroup\$ I always thought UTF-8 was so-named due to the fact that it uses 8 binary bits to encode each character. This is interesting. Great question and cool info as well. \$\endgroup\$
    – Nate T
    Nov 23 at 3:11
  • 2
    \$\begingroup\$ Won't all value passed to function T will be less than BE since they are mod of BE? Then the 3rd and 4th rules of T seems useless. \$\endgroup\$
    – tsh
    Nov 23 at 3:15
  • 1
    \$\begingroup\$ @NateT With just 8 binary bits, you can only encode 256 different characters. \$\endgroup\$
    – Abigail
    Nov 23 at 22:24
4
\$\begingroup\$

Python 3, 289 262 241 240 232 230 bytes

lambda n:[p(*[[n-d,4,252],[n-16406,2,246],[n-256,1,161]][(n<16406)+(n<d)]),[160]*(n>159)+[n]][n<256]
c=190
d=233006
t=lambda n:n+[-96,-c,66,33][(n<94)+(n<c)+(n<223)]
p=lambda y,x,z:[z+y//c**x]+[t(y//c**i%c)for i in range(x)][::-1]

Try it online!

-27 bytes cos my implementation of T was wrong
-21 bytes thanks to @wasif's implementation of f, which works on t too
-1 byte thanks to storing 233006 in a variable
-8 bytes thanks to @pxeger in TNB
-2 bytes thanks to unneeded parens

It's a naive approach, and I'm sure there's lots of room for golfing, but it works!

Python 3.8, 210 bytes

lambda n:[[(y:=n-[d:=233006,e:=16406,256][j:=(n&lt;e)+(n&lt;d)])//c**(x:=[4,2,1][j])+(z:=[252,246,161][j])]+[(g:=y//c**i%c)+[-96,-c,66,33][(g&lt;94)+(g&lt;c)+(g&lt;223)]for i in range(x)][::-1],[160]*(n&gt;159)+[n]][n&lt;256]
c=190

Attempt This Online!

Same as the Python 3 answer, but uses 3.8's := operator to cram it all into 1 lambda

Explanation

The lambda t is equivalent to the T() defined in the challenge.

The lambda p takes 3 integers, y, x, and z, and creates:

[z+y/190^x, t(y/190^(x-1)%190), t(y/190^(x-2)%190)...t(y/190^0%190)]

We only use x as 1, 2, or 4, depending on which formula we're handling:

(y = n-256, x = 1, z = 161):
    [161+y/190, t(y%190)]
(y = n-16406, x = 2, z = 246):
    [246+y/190^2, t(y/190%190), t(y%190)]
(y = n-233006, x = 4, z = 252):
    [252+y/190^4, t(y/190^3%190), t(y/190^2%190), t(y/190%190), t(y%190)]

Finally the lambda f handles the 5 formulas and their ranges for input n:

def f(n):
    if n < 160: return [n]
    if n < 256: return [160, n]
    if n < 16406: return p(n-256, 1, 161)    #first formula
    if n < 233006: return p(n-16406, 2, 246) #second formula
    return p(n-233006, 4, 252)               #third formula
\$\endgroup\$
2
  • \$\begingroup\$ -18 bytes \$\endgroup\$
    – wasif
    Nov 22 at 16:57
  • \$\begingroup\$ @wasif nice! I already fixed my implementation of T such that I don't need the list comprehension to turn -1 into 255 anymore, but your implementation of f saves 7 bytes! \$\endgroup\$
    – Mayube
    Nov 22 at 17:01
3
\$\begingroup\$

Burlesque, 270 bytes

190s1256s216406s7233006s8{g1./}s9%t={g1.%J{{33.+}{66.+}{g1.-}{g6.-}}{94g1223g2}(.<)bcz[{}Z]cne!}J{qbx{40960.+bx}{g2.-{{g9e!161.+}q%t!}M-}{g7.-{{g92E!246.+}{g9e!%t!}q%t!}M-}{g8.-{{g94E!252.+}{g93E!%t!}{g92E!%t!}{g9e!%t!}q%t!}M-}}{160g2g7g82147483648}(.<)bcz[{}Z]cne!qb6\m

Try it online!

Pretty sure this can be golfed further/better, but this is already pretty terrifying. Main changes vs. ungolfed are using zips to build the conditional blocks.

190s1                       # Store 190
256s2                       # Store 256
16406s7                     # Store 16406
233006s8                    # Store 233006
{g1./}s9                    # Store divide by 190
%t={                        # Assign T
 g1.%                       # Mod 190
 J                          # Duplicate (conditional is destructive)
 {
  {94.<}{33.+}              # <94 add 33 (0x21)
  {g1.<}{66.+}              # <190 add 66 (0x42)
  {223.<}{g1.-}             # <223 sub 190 (0xBE)
  {g2.<}{96.-}              # <256 sub 96 (0x60)
 }
  cne!}                     # Conditional evualuate resulting block
            # END T
J                           # Duplicate (conditional is destructive)
{{160.<}        qbx         # <160 Put in box
 {g2.<}         {40960.+bx} # <256 add 40960 (0xA000) and box
 {g7.<}         {           # <16406 (0x4016)
        g2.-       # Sub 256
        {
         {
          g9e!     # Divide by 190
          161.+    # Add 161
         }
         q%t!      # Apply T
        }M-}       # Cool map (create copy for each block and
 {g8.<}         {  # eval and return list of each result)
        g7.-       # Sub 16406
        {
         {
          g92E!    # Divide by 190 twice
          246.+    # Add 246
         }
         {
          g9e!%t!  # Divide by 190 apply T
         }
         q%t!      # Apply T
        }M-}
 {2147483648.<} {           # < 2147483648
        g8.-       # Sub 233006
        {
         {
          g94E!    # Divide by 190 4x
          252.+    # Add 252
         }
         {
          g93E!%t! # Divide by 190 3x
         } #2x       1x       0x
         {g92E!%t!}{g9e!%t!}q%t!}
        M-}
       }
}cne!              # Evaluate the conditional
qb6\m              # Map to hex and concatenate
\$\endgroup\$
2
\$\begingroup\$

JavaScript (ES6), 175 bytes

Expects the input as a string. Returns a comma-separated string of bytes.

x=>(v=190,p=16406,q=233006,g=(o,n,d,k=1,y=(x-o)/k%v|0)=>--n?g(o,n,d,k*v)+[,y+=y<94?33:y<v?66:y<223?-v:-96]:160-~d+y,x<160?x:x>>8?x<p?g(256,2):x<q?g(p,3,85):g(q,5,91):160+[,x])

Try it online!

\$\endgroup\$
2
\$\begingroup\$

JavaScript (Node.js), 138 bytes

f=x=>x<160?[x]:x>>8?(g=y=>++l<5?[...g(y/190|0),(y%=190)-33*~(y>93)]:[f(n)[0]-~y])(x+~[233005,,16405,255].find((m,j)=>(l=j,n=m)<x)):[160,x]

Try it online!

\$\endgroup\$
1
\$\begingroup\$

Charcoal, 99 bytes

I⎇›¹⁶⁰θθ⎇›²⁵⁶θ⟦¹⁶⁰θ⟧E↨⁺θ⎇›¹⁶⁴⁰⁶θ¹²⁴⎇›²³³⁰⁰⁶θ³¹²⁴²⁹⁴I”)¶∧‴+νO﹪”¹⁹⁰⁺ι⎇κ⎇‹¹⁸⁹ι⎇‹²²²ι±⁹⁶±¹⁹⁰׳³⊕‹⁹³ι¹⁵⁹

Try it online! Link is to verbose version of code. Explanation:

   ¹⁶⁰                                                      Literal integer 160
  ›                                                         Is greater than
    θ                                                       Input integer
 ⎇                                                          If true then
     θ                                                      Input integer
 ⎇                                                          Else
        ²⁵⁶                                                 Literal 256
       ›                                                    Is greater than
         θ                                                  Input integer
      ⎇                                                     If true then
          ⟦  ⟧                                              List of
           ¹⁶⁰                                              Literal integer 160
            θ                                               Input integer
      ⎇                                                     Else
                 θ                                          Input integer
                ⁺                                           Plus
                    ¹⁶⁴⁰⁶                                   Literal integer 16406
                   ›                                        Is greater than
                     θ                                      Input integer
                  ⎇                                         If true then
                      ¹²⁴                                   Literal integer 124
                  ⎇                                         Else
                         ²³³⁰⁰⁶                             Literal integer 233006
                        ›                                   Is greater than
                          θ                                 Input integer
                       ⎇                                    If true then
                           ³¹²⁴²⁹⁴                          Literal integer 3124294
                       ⎇                                    Else
                             ”)¶∧‴+νO﹪”                     Compressed string "121198296994"
                            I                               Cast to integer
               ↨                                            Convert to base
                              ¹⁹⁰                           Literal integer 190
              E                                             Map over base 190 digits
                                ι                           Current value
                               ⁺                            Plus
                                  κ                         Current index
                                 ⎇                          If not first digit then
                                     ¹⁸⁹                    Literal integer 189
                                    ‹                       Is less than
                                      ι                     Current value
                                   ⎇                        If true then
                                         ²²²                Literal integer 222
                                        ‹                   Is less than
                                          ι                 Current value
                                       ⎇                    If true then
                                            ⁹⁶              Literal integer 96
                                       ⎇                    Else
                                           ±                Negated
                                              ¹⁹⁰           Literal integer 190
                                             ±              Negated
                                   ⎇                        Else
                                                ³³          Literal 33
                                               ×            Times
                                                   ⁹³       Literal 93
                                                  ‹         Is less than
                                                    ι       Current value
                                                 ⊕          Incremented
                                 ⎇                          If first digit then
                                                     ¹⁵⁹    Literal integer 159
I                                                           Cast to string
                                                            Implicitly print
\$\endgroup\$
1
\$\begingroup\$

Haskell, 234 bytes

t z|z<94=33+z|z<130=z+66|z<223=z-190|0<1=z-96
u x|x<256=[160|x>159]++[x]|1>0=let[b,c,d]=head$dropWhile((x<).head)[[233005,5,252],[16406,3,246],[256,2,161]];w=reverse.take c.map(`mod`190).iterate(`div`190)$x-b in d+head w:map t(tail w)

Try it online!

\$\endgroup\$
1
\$\begingroup\$

Rust, 250 bytes

|n|{let t=|i,b,z|if i<2{b+z}else{z%190+33+(z%190>93)as i32*33};let g=|b,y,p|(1..=p).map(|i|t(i,b,y/190i32.pow(p-i))).collect();match n{0..=159=>vec![n],0..=255=>vec![160,n],0..=16405=>g(161,n-256,2),0..=233005=>g(246,n-16406,3),_=>g(252,n-233006,5)}}

Try it online!

Ungolfed

|n| {
    // Modified T: Returns T(Z%0xBE) if i > 1 else B+Z
    let t = |i, b, z|
        if i < 2 {
            b+z
        } else {
            z % 0xBE + 0x21 + if (z % 0xBE) > 0x5D { 0x21 } else { 0 }
        };

    // Helper: Generates the p UTF-1 bytes for codepoints > U+FF
    let g = |b, y, p|
        (1..=p).map(
            |i| t(i, b, y / 190_i32.pow(p-i))
        ).collect();

    // Select encoding variant
    match n{
        0..=159    => vec![n],
        0..=255    => vec![0xA0, n],
        0..=16405  => g(0xA1, n - 0x100,   2),
        0..=233005 => g(0xF6, n - 0x4016,  3),
        _          => g(0xFC, n - 0x38E2E, 5)
    }
}
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.