13
\$\begingroup\$

UTF-1 is one of the methods to transform ISO/IEC 10646 and Unicode into a sequence of bytes. It was originally for ISO 10646.

The UTF-1 encodes every character with variable length of byte sequence, just like UTF-8 does. It was designed to avoid control codes. But it was not designed to avoid duplication of a slash character ('/'), which is path delimiter for many operating systems. Additionally it takes a bit long to process the algorithm, because it is based on modulo 190, which is not power of two. Eventually UTF-8 was better than that.

In this challenge we are reviving UTF-1.

How it works

Every constant value in tables on this section is represented in hexadecimal value.

The symbol / is the integer division operator, % is the integer modulo operator, and ^ means power. The precedence of operators is ^ first, / second, % third, and others last. Here is how to convert each codepoint to UTF-1 (sorry for typo last time):

codepoint x UTF-1
U+0000 to U+009F (x)
U+00A0 to U+00FF (A0,x)
U+0100 to U+4015 y=x-100 in (A1+y/BE,T(y%BE))
U+4016 to U+38E2D y=x-4016 in (F6+y/BE^2,T(y/BE%BE),T(y%BE))
U+38E2E to U+7FFFFFFF y=x-38E2E in (FC+y/BE^4,T(y/BE^3%BE),T(y/BE^2%BE),T(y/BE%BE),T(y%BE))

T(z) is a function such that:

z T(z)
0 to 5D z+21
5E to BD z+42
BE to DE z-BE
DF to FF z-60

Challenge

Your task is to implement a program or a function or a subroutine that takes one integer, who represents the codepoint of a character, to return a sequence of integers that represents its corresponding UTF-1 value.

Constraints

Input shall be an nonnegative integer up to 0x7FFFFFFF.

Rules

  • Standard loopholes apply.
  • Standard I/O rules apply.
  • Shortest code wins.

Test cases

Taken from the Wikipedia article.

    U+007F  7F
    U+0080  80
    U+009F  9F
    U+00A0  A0 A0
    U+00BF  A0 BF
    U+00C0  A0 C0
    U+00FF  A0 FF
    U+0100  A1 21
    U+015D  A1 7E
    U+015E  A1 A0
    U+01BD  A1 FF
    U+01BE  A2 21
    U+07FF  AA 72
    U+0800  AA 73
    U+0FFF  B5 48
    U+1000  B5 49
    U+4015  F5 FF
    U+4016  F6 21 21
    U+D7FF  F7 2F C3
    U+E000  F7 3A 79
    U+F8FF  F7 5C 3C
    U+FDD0  F7 62 BA
    U+FDEF  F7 62 D9
    U+FEFF  F7 64 4C
    U+FFFD  F7 65 AD
    U+FFFE  F7 65 AE
    U+FFFF  F7 65 AF
   U+10000  F7 65 B0
   U+38E2D  FB FF FF
   U+38E2E  FC 21 21 21 21
   U+FFFFF  FC 21 37 B2 7A
  U+100000  FC 21 37 B2 7B
  U+10FFFF  FC 21 39 6E 6C
U+7FFFFFFF  FD BD 2B B9 40
Improve this question
\$\endgroup\$
6
  • 1
    \$\begingroup\$ for the 3rd formula, is the x in (A1+x/BE,T(y%BE)) a typo? Did you mean A1+y/BE? \$\endgroup\$
    – Mayube
    Nov 22, 2021 at 14:56
  • \$\begingroup\$ @mayube Thank you, it REALLY was. \$\endgroup\$
    – user100411
    Nov 22, 2021 at 20:20
  • 1
    \$\begingroup\$ I always thought UTF-8 was so-named due to the fact that it uses 8 binary bits to encode each character. This is interesting. Great question and cool info as well. \$\endgroup\$
    – Nate T
    Nov 23, 2021 at 3:11
  • 2
    \$\begingroup\$ Won't all value passed to function T will be less than BE since they are mod of BE? Then the 3rd and 4th rules of T seems useless. \$\endgroup\$
    – tsh
    Nov 23, 2021 at 3:15
  • 1
    \$\begingroup\$ @NateT With just 8 binary bits, you can only encode 256 different characters. \$\endgroup\$
    – Abigail
    Nov 23, 2021 at 22:24

7 Answers 7

Reset to default
4
\$\begingroup\$

Python 3, 289 262 241 240 232 230 bytes

lambda n:[p(*[[n-d,4,252],[n-16406,2,246],[n-256,1,161]][(n<16406)+(n<d)]),[160]*(n>159)+[n]][n<256]
c=190
d=233006
t=lambda n:n+[-96,-c,66,33][(n<94)+(n<c)+(n<223)]
p=lambda y,x,z:[z+y//c**x]+[t(y//c**i%c)for i in range(x)][::-1]

Try it online!

-27 bytes cos my implementation of T was wrong
-21 bytes thanks to @wasif's implementation of f, which works on t too
-1 byte thanks to storing 233006 in a variable
-8 bytes thanks to @pxeger in TNB
-2 bytes thanks to unneeded parens

It's a naive approach, and I'm sure there's lots of room for golfing, but it works!

Python 3.8, 210 bytes

lambda n:[[(y:=n-[d:=233006,e:=16406,256][j:=(n&lt;e)+(n&lt;d)])//c**(x:=[4,2,1][j])+(z:=[252,246,161][j])]+[(g:=y//c**i%c)+[-96,-c,66,33][(g&lt;94)+(g&lt;c)+(g&lt;223)]for i in range(x)][::-1],[160]*(n&gt;159)+[n]][n&lt;256]
c=190

Attempt This Online!

Same as the Python 3 answer, but uses 3.8's := operator to cram it all into 1 lambda

Explanation

The lambda t is equivalent to the T() defined in the challenge.

The lambda p takes 3 integers, y, x, and z, and creates:

[z+y/190^x, t(y/190^(x-1)%190), t(y/190^(x-2)%190)...t(y/190^0%190)]

We only use x as 1, 2, or 4, depending on which formula we're handling:

(y = n-256, x = 1, z = 161):
    [161+y/190, t(y%190)]
(y = n-16406, x = 2, z = 246):
    [246+y/190^2, t(y/190%190), t(y%190)]
(y = n-233006, x = 4, z = 252):
    [252+y/190^4, t(y/190^3%190), t(y/190^2%190), t(y/190%190), t(y%190)]

Finally the lambda f handles the 5 formulas and their ranges for input n:

def f(n):
    if n < 160: return [n]
    if n < 256: return [160, n]
    if n < 16406: return p(n-256, 1, 161)    #first formula
    if n < 233006: return p(n-16406, 2, 246) #second formula
    return p(n-233006, 4, 252)               #third formula
Improve this answer
\$\endgroup\$
2
  • \$\begingroup\$ -18 bytes \$\endgroup\$
    – Wasif
    Nov 22, 2021 at 16:57
  • \$\begingroup\$ @wasif nice! I already fixed my implementation of T such that I don't need the list comprehension to turn -1 into 255 anymore, but your implementation of f saves 7 bytes! \$\endgroup\$
    – Mayube
    Nov 22, 2021 at 17:01
3
\$\begingroup\$

Burlesque, 270 bytes

190s1256s216406s7233006s8{g1./}s9%t={g1.%J{{33.+}{66.+}{g1.-}{g6.-}}{94g1223g2}(.<)bcz[{}Z]cne!}J{qbx{40960.+bx}{g2.-{{g9e!161.+}q%t!}M-}{g7.-{{g92E!246.+}{g9e!%t!}q%t!}M-}{g8.-{{g94E!252.+}{g93E!%t!}{g92E!%t!}{g9e!%t!}q%t!}M-}}{160g2g7g82147483648}(.<)bcz[{}Z]cne!qb6\m

Try it online!

Pretty sure this can be golfed further/better, but this is already pretty terrifying. Main changes vs. ungolfed are using zips to build the conditional blocks.

190s1                       # Store 190
256s2                       # Store 256
16406s7                     # Store 16406
233006s8                    # Store 233006
{g1./}s9                    # Store divide by 190
%t={                        # Assign T
 g1.%                       # Mod 190
 J                          # Duplicate (conditional is destructive)
 {
  {94.<}{33.+}              # <94 add 33 (0x21)
  {g1.<}{66.+}              # <190 add 66 (0x42)
  {223.<}{g1.-}             # <223 sub 190 (0xBE)
  {g2.<}{96.-}              # <256 sub 96 (0x60)
 }
  cne!}                     # Conditional evualuate resulting block
            # END T
J                           # Duplicate (conditional is destructive)
{{160.<}        qbx         # <160 Put in box
 {g2.<}         {40960.+bx} # <256 add 40960 (0xA000) and box
 {g7.<}         {           # <16406 (0x4016)
        g2.-       # Sub 256
        {
         {
          g9e!     # Divide by 190
          161.+    # Add 161
         }
         q%t!      # Apply T
        }M-}       # Cool map (create copy for each block and
 {g8.<}         {  # eval and return list of each result)
        g7.-       # Sub 16406
        {
         {
          g92E!    # Divide by 190 twice
          246.+    # Add 246
         }
         {
          g9e!%t!  # Divide by 190 apply T
         }
         q%t!      # Apply T
        }M-}
 {2147483648.<} {           # < 2147483648
        g8.-       # Sub 233006
        {
         {
          g94E!    # Divide by 190 4x
          252.+    # Add 252
         }
         {
          g93E!%t! # Divide by 190 3x
         } #2x       1x       0x
         {g92E!%t!}{g9e!%t!}q%t!}
        M-}
       }
}cne!              # Evaluate the conditional
qb6\m              # Map to hex and concatenate
Improve this answer
\$\endgroup\$
2
\$\begingroup\$

JavaScript (ES6), 175 bytes

Expects the input as a string. Returns a comma-separated string of bytes.

x=>(v=190,p=16406,q=233006,g=(o,n,d,k=1,y=(x-o)/k%v|0)=>--n?g(o,n,d,k*v)+[,y+=y<94?33:y<v?66:y<223?-v:-96]:160-~d+y,x<160?x:x>>8?x<p?g(256,2):x<q?g(p,3,85):g(q,5,91):160+[,x])

Try it online!

Improve this answer
\$\endgroup\$
2
\$\begingroup\$

JavaScript (Node.js), 138 bytes

f=x=>x<160?[x]:x>>8?(g=y=>++l<5?[...g(y/190|0),(y%=190)-33*~(y>93)]:[f(n)[0]-~y])(x+~[233005,,16405,255].find((m,j)=>(l=j,n=m)<x)):[160,x]

Try it online!

Improve this answer
\$\endgroup\$
1
\$\begingroup\$

Charcoal, 99 bytes

Ｉ⎇›¹⁶⁰θθ⎇›²⁵⁶θ⟦¹⁶⁰θ⟧Ｅ↨⁺θ⎇›¹⁶⁴⁰⁶θ¹²⁴⎇›²³³⁰⁰⁶θ³¹²⁴²⁹⁴Ｉ”)¶∧‴+νＯ﹪”¹⁹⁰⁺ι⎇κ⎇‹¹⁸⁹ι⎇‹²²²ι±⁹⁶±¹⁹⁰׳³⊕‹⁹³ι¹⁵⁹

Try it online! Link is to verbose version of code. Explanation:

   ¹⁶⁰                                                      Literal integer 160
  ›                                                         Is greater than
    θ                                                       Input integer
 ⎇                                                          If true then
     θ                                                      Input integer
 ⎇                                                          Else
        ²⁵⁶                                                 Literal 256
       ›                                                    Is greater than
         θ                                                  Input integer
      ⎇                                                     If true then
          ⟦  ⟧                                              List of
           ¹⁶⁰                                              Literal integer 160
            θ                                               Input integer
      ⎇                                                     Else
                 θ                                          Input integer
                ⁺                                           Plus
                    ¹⁶⁴⁰⁶                                   Literal integer 16406
                   ›                                        Is greater than
                     θ                                      Input integer
                  ⎇                                         If true then
                      ¹²⁴                                   Literal integer 124
                  ⎇                                         Else
                         ²³³⁰⁰⁶                             Literal integer 233006
                        ›                                   Is greater than
                          θ                                 Input integer
                       ⎇                                    If true then
                           ³¹²⁴²⁹⁴                          Literal integer 3124294
                       ⎇                                    Else
                             ”)¶∧‴+νＯ﹪”                     Compressed string "121198296994"
                            Ｉ                               Cast to integer
               ↨                                            Convert to base
                              ¹⁹⁰                           Literal integer 190
              Ｅ                                             Map over base 190 digits
                                ι                           Current value
                               ⁺                            Plus
                                  κ                         Current index
                                 ⎇                          If not first digit then
                                     ¹⁸⁹                    Literal integer 189
                                    ‹                       Is less than
                                      ι                     Current value
                                   ⎇                        If true then
                                         ²²²                Literal integer 222
                                        ‹                   Is less than
                                          ι                 Current value
                                       ⎇                    If true then
                                            ⁹⁶              Literal integer 96
                                       ⎇                    Else
                                           ±                Negated
                                              ¹⁹⁰           Literal integer 190
                                             ±              Negated
                                   ⎇                        Else
                                                ³³          Literal 33
                                               ×            Times
                                                   ⁹³       Literal 93
                                                  ‹         Is less than
                                                    ι       Current value
                                                 ⊕          Incremented
                                 ⎇                          If first digit then
                                                     ¹⁵⁹    Literal integer 159
Ｉ                                                           Cast to string
                                                            Implicitly print
Improve this answer
\$\endgroup\$
1
\$\begingroup\$

Haskell, 234 bytes

t z|z<94=33+z|z<130=z+66|z<223=z-190|0<1=z-96
u x|x<256=[160|x>159]++[x]|1>0=let[b,c,d]=head$dropWhile((x<).head)[[233005,5,252],[16406,3,246],[256,2,161]];w=reverse.take c.map(`mod`190).iterate(`div`190)$x-b in d+head w:map t(tail w)

Try it online!

Improve this answer
\$\endgroup\$
1
\$\begingroup\$

Rust, 250 bytes

|n|{let t=|i,b,z|if i<2{b+z}else{z%190+33+(z%190>93)as i32*33};let g=|b,y,p|(1..=p).map(|i|t(i,b,y/190i32.pow(p-i))).collect();match n{0..=159=>vec![n],0..=255=>vec![160,n],0..=16405=>g(161,n-256,2),0..=233005=>g(246,n-16406,3),_=>g(252,n-233006,5)}}

Try it online!

Ungolfed

|n| {
    // Modified T: Returns T(Z%0xBE) if i > 1 else B+Z
    let t = |i, b, z|
        if i < 2 {
            b+z
        } else {
            z % 0xBE + 0x21 + if (z % 0xBE) > 0x5D { 0x21 } else { 0 }
        };

    // Helper: Generates the p UTF-1 bytes for codepoints > U+FF
    let g = |b, y, p|
        (1..=p).map(
            |i| t(i, b, y / 190_i32.pow(p-i))
        ).collect();

    // Select encoding variant
    match n{
        0..=159    => vec![n],
        0..=255    => vec![0xA0, n],
        0..=16405  => g(0xA1, n - 0x100,   2),
        0..=233005 => g(0xF6, n - 0x4016,  3),
        _          => g(0xFC, n - 0x38E2E, 5)
    }
}
Improve this answer
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.