Bake me some moji

Question

Given a string, character list, byte stream, sequence… which is both valid UTF-8 and valid Windows-1252 (most languages will probably want to take a normal UTF-8 string), convert it from (that is, pretend it is) Windows-1252 to UTF-8.

Walked-through example

The UTF-8 string
I       ♥        U  T  F  -  8
is represented as the bytes
49 20  E2 99 A5  20 55 54 46 2D 38
these byte values in the Windows-1252 table gives us the Unicode equivalents
49 20 E2 2122 A5 20 55 54 46 2D 38
which render as
I    â  ™  ¥    U  T  F  -  8

Examples

£ → £

£ → £

£ → £

I ♥ UTF-8 → I ♥ UTF-8

árvíztűrő tükörfúrógép → árvÃztűrÅ‘ tükörfúrógép

Answer 1

bash, 14 bytes

iconv -fCP1252

Try it online!

Answer 2

Java 8, 72 66 36 25 bytes

s->new String(s,"cp1252")

Try it online.

s->  // Method with byte-array (UTF-8 by default) as parameter and String return-type
  new String(s,"cp1252")
     //  Pretend this UTF-8 input is (and convert it to) Windows-1252,
     //  and return it as UTF-8 String (by default) as well

cp1252 is an alias for Windows-1252. This alias cp1252 is the Canonical Name for the java.io and java.lang APIs, while the full name Windows-1252 is the Canonical Name for the java.nio API. See here for a full list of supported Java encodings, where we'd always want to use the shortest of the two for codegolfing.

Answer 3

R 3.5.0 or higher, 32 20 bytes

scan(,"",e="latin1")

Try it online!

Oddly short for a string challenge in R... thanks to JayCe for golfing down 12 more bytes!

scan optionally takes an encoding argument to set the encoding of the input string. latin1 corresponds to, according to the documentation of Encoding

There is some ambiguity as to what is meant by a ‘Latin-1’ locale, since some OSes (notably Windows) make use of character positions used for control characters in the ISO 8859-1 character set. How such characters are interpreted is system-dependent but as from R 3.5.0 they are if possible interpreted as per Windows codepage 1252 (which Microsoft calls ‘Windows Latin 1 (ANSI)’) when converting to e.g. UTF-8.

Answer 4

Python 2, 40 38 bytes

-2 bytes thanks to Erik the Outgolfer.

lambda s:s.decode('1252').encode('u8')

Try it online!

u8 is an alias for utf-8.

Answer 5

Ruby, 31 bytes

->s{s.encode("UTF-8","CP1252")}

Try it online!

Tests cases are included in the TIO

Answer 6

Python 3, 38 36 34 bytes

lambda s:s.encode().decode('1252')

Try it online!

note: After I had a working function I used ovs's python2 answer answer to learn about the header and footer fields for tio, so the header and footer are the same

edit: Trimmed it a little thanks to python3 defaulting to utf8 and a tip from ovs's submission :)

Answer 7

C#, 81 bytes

using e=System.Text.Encoding;s=>e.GetEncoding(1252).GetString(e.UTF8.GetBytes(s))

Try it online!

Thanks to Schmalls for 3 bytes

Answer 8

180 bytes, machine code (16-bit x86)

I noticed most answers use builtin encode/decode (which I believe is perfectly fine), but I thought I'll continue my 16-bit quest.

As with previous ones, this was done without compiler using mostly HT hexeditor and ICY's hexplorer.

00000000: eb40 ac20 0000 1a20 9201 1e20 2620 2020  .@. ... ... &                     
00000010: 2120 c602 3020 6001 3920 5201 0000 7d01  ! ..0 `.9 R...}.                  
00000020: 0000 0000 1820 1920 1c20 1d20 2220 1320  ..... . . . " .                   
00000030: 1420 dc02 2221 6101 3a20 5301 0000 7e01  . .."!a.: S...~.                  
00000040: 7801 89f7 4646 89fa 89d9 4143 4bb4 3fcd  x...FF....ACK.?.                  
00000050: 2185 c074 288a 053c 8073 05e8 1700 ebec  !..t(..<.s......                  
00000060: 3ca0 721a d440 0d80 c050 86c4 e806 0058  <[email protected]                  
00000070: e802 00eb d7b4 4088 05b3 01cd 21c3 2c80  ......@.....!.,.                  
00000080: d0e0 89c3 8b00 89cb 85c0 74c0 3dff 0773  ..........t.=..s                  
00000090: 08c1 c002 c0e8 02eb cd50 c1e8 0c0c e0e8  .........P......                  
000000a0: d3ff 5825 ff0f c1c0 02c0 e802 0d80 8050  ..X%...........P                  
000000b0: 86c4 ebb8                                ....                              

Dissection

Implementation is pretty straight-forward, although I haven't given much thought to flow upfront so there is SOME spaghetti there.

I'll mix order a bit, to make it easier to follow...

0000 eb40               jmp         0x42

Skip over table that maps chars >= 0x80 < 0xa0, to unicode codes.

data db ACh,20h, 00h,00h, 1Ah,20h, ...

Invalid ones are encoded as 0, they are not mapped to anything

0075 b440               mov         ah, 0x40   
0077 8805               mov         [di], al   
0079 b301               mov         bl, 0x1    
007b cd21               int         0x21       
007d c3                 ret                    

Helper function used to print char in al, will be called few times.

0042 89f7               mov         di, si     
0044 46                 inc         si         
0045 46                 inc         si         
0046 89fa               mov         dx, di     
0048 89d9               mov         cx, bx     
004a 41                 inc         cx         
004b 43                 inc         bx         

Prepare registers. Data will be read into 0x100, let si point into translation table above.

004c 4b                 dec         bx         
004d b43f               mov         ah, 0x3f   
004f cd21               int         0x21       
0051 85c0               test        ax, ax     
0053 7428               jz          0x7d       

Read char from stdin, jump to 0x7d if EOF.

Sidenote: This actually is a small (but pretty well known) trick, 0x7d contains ret, this will cause pop sp, sp at start points to end of a segment, there's 00 00 there, and cs:0 in DOS contains CD 20, which causes application to exit.

0055 8a05               mov         al, [di]   
0057 3c80               cmp         al, 0x80   
0059 7305               jnc         0x60       
005b e81700             call        0x75       
005e ebec               jmp         0x4c       

If char is < 0x80, just print it out, and go to beginning of loop (because helper function is setting BX to 1 - stdout, jumps will go to dec bx)

0060 3ca0               cmp         al, 0xa0   
0062 721a               jc          0x7e       
0064 d440               aam         0x40       
0066 0d80c0             or          ax, c080   
0069 50                 push        ax         
006a 86c4               xchg        ah, al     
006c e80600             call        0x75       
006f 58                 pop         ax         
0070 e80200             call        0x75       
0073 ebd7               jmp         0x4c       

This part deals with chars >= 0xa0, splits ascii code into "high" two bits and "low" 6 bits and applies utf-8 mask c080 for two bytes, then prints both of them

007e 2c80               sub         al, 0x80   
0080 d0e0               shl         al, 0x1    
0082 89c3               mov         bx, ax     
0084 8b00               mov         ax, [bx+si]
0086 89cb               mov         bx, cx     
0088 85c0               test        ax, ax     
008a 74c0               jz          0x4c       
008c 3dff07             cmp         ax, 07ff   
008f 7308               jnc         0x99       
0091 c1c002             rol         ax, 0x2    
0094 c0e802             shr         al, 0x2    
0097 ebcd               jmp         0x66       

This part deals with chars >= 0x80 < 0xa0, it finds proper utf-8 code in the table at the top, if code equals 0, just skip to beginning, if it's below 0x7ff (ergo: fits on two UTF-8 bytes), just adjust the value and re-use previous code at 0x166.

0099 50                 push        ax         
009a c1e80c             shr         ax, 0xc    
009d 0ce0               or          al, e0     
009f e8d3ff             call        0x75       
00a2 58                 pop         ax         
00a3 25ff0f             and         ax, 0fff   
00a6 c1c002             rol         ax, 0x2    
00a9 c0e802             shr         al, 0x2    
00ac 0d8080             or          ax, 8080   
00af 50                 push        ax         
00b0 86c4               xchg        ah, al     
00b2 ebb8               jmp         0x6c       

Final part, deals with codes that are above 0x7FF, drop low 12 bits, apply 0xE0 (see UTF-8 encoding description for reference) and print it out, adjust lower 12 bits and apply 8080 mask and again reuse part that spits out two chars.

Answer 9

JavaScript, 64 bytes

x=>new TextDecoder('cp1252').decode(new TextEncoder().encode(x))

f =
x=>new TextDecoder('cp1252').decode(new TextEncoder().encode(x))

<p><input id=i oninput="o.value=f(i.value)" style="width:100%" /></p>
<p><output id=o></output></p>

Even longer than Java answer. So sad. :(

Answer 10

PHP+mbstring, 63 49 bytes

<?=mb_convert_encoding($argv[1],'UTF8','CP1252');

It doesn't work on TIO due to the lack of mbstring. The third parameter force mbstring to interpret the string as Windows-1252 encoded

-14 bytes thanks to Ismael Miguel

Answer 11

C (gcc) + libiconv, 119 117 bytes

*f(s,t,u)void*s,*t,*u;{long i=strlen(s),j=i*4;u=t=malloc(j);iconv(iconv_open("UTF8","CP1252"),&s,&i,&u,&j);return t;}

Try it online!

Answer 12

Scala 3, 18 bytes

Simply a port of Kevin Cruijssen's Java answer

String(_,"cp1252")

Try it in Scastie!

Ordinarily, I wouldn't post such a trivial answer, but this demonstrates Scala 3's universal apply methods, which let you omit new.