Bytes/Character

Question

Task

Given a UTF-8 string (by any means) answer (by any means) an equivalent list where every element is the number of bytes used to encode the corresponding input character.

Examples

! → 1

Ciao → 1 1 1 1

tʃaʊ → 1 2 1 2

Adám → 1 1 2 1

ĉaŭ → 2 1 2 (single characters)

ĉaŭ → 1 2 1 1 2 (uses combining overlays)

チャオ → 3 3 3

(empty input) → (empty output)

!±≡𩸽 → 1 2 3 4

� (a null byte) → 1

Null bytes

If the only way to keep reading input beyond null bytes is by knowing the total byte count, you may get the byte count by any means (even user input).

If your language cannot handle null bytes at all, you may assume the input does not contain nulls.

Answer 1

Python 3, 42 36 bytes

lambda x:[len(i.encode())for i in x]

Answer 2

C, 68 65 bytes

b;main(c){for(;~c;b=c/64^2?b?putchar(b+48)/48:1:b+1)c=getchar();}

Thanks to @FryAmTheEggman for golfing off 3 bytes!

Test it on Ideone.

Answer 3

APL, 15 chars

≢¨'UTF-8'∘⎕ucs¨

In English: convert each character to UTF-8 (meaning: vector of bytes representation) and get its tally.

Answer 4

Pyth, 9 7 bytes

Thanks to @Maltysen for saving 2 bytes!

mlc.Bd8

Test suite

Converts every character of the input to it's binary representation and then splits this into chunks of length 8. The number of those chunks is then the amount of bytes needed to encode that character.

Answer 5

GolfScript, 16 bytes

{64/2=}%1,/{,)}*

Try it online!

Background

GolfScript doesn't have a clue what Unicode is; all strings (input, output, internal) are composed of bytes. While that can be pretty annoying, it's perfect for this challenge.

UTF-8 encodes ASCII and non-ASCII characters differently:

• All code points below 128 are encoded as 0xxxxxxx.

• All other code points are encoded as 11xxxxxx 10xxxxxx ... 10xxxxxx.

This means that the encoding of each Unicode character contains either a single 0xxxxxxx byte or a single 11xxxxxx byte and 1 to 5 10xxxxxx bytes.

By dividing all bytes of the input by 64, we turn 0xxxxxxx into 0 or 1, 11xxxxxx into 3, and 10xxxxxx into 2.

If we compare the quotient with 2 – pushing 1 for 2; and 0 for 0, 1, and 3 – each character will be turned into a 0, followed by 1 to 5 1's.

All that's left is to split the resulting string at occurrences of 0, count the number of 1's between those zeroes and add one to the amount.

How it works

{     }%          Map the following over all bytes in the input.
 64/                Divide the byte by 64.
    2=              Compare the quotient with 2, pushing 1 or 0.
        1,        Push range(1), i.e., [0].
          /       Split the array of Booleans around zeroes.
           {  }*  Fold; for each run of ones but the first:
            ,       Push its length.
             )      Increment.

Answer 6

PowerShell v4, 58 bytes

[char[]]$args[0]|%{[Text.Encoding]::UTF8.GetByteCount($_)}

---

NB

OK, this should work, and does in almost all of the test cases except for 𩸽 which is somehow counted as 3,3 on my machine. That character even shows as 7 bytes on my computer. I suspect this is due to some sort of bug in the Windows or .NET version that I'm running locally, as @Mego doesn't have that issue. (Edit: @cat points out this is due to BOM. Thanks for solving that mystery, @cat!)

However, that still doesn't account for all of the problem. I think I know where some of the problems are coming from, though. Inside .NET, all strings are composed of UTF-16 code units (which is the System.Char type). With the very loose typecasting that PowerShell uses, there's a lot of implicit casting and conversion between types in the background. Likely this is a contributing factor to the behavior we're seeing -- for example, [system.text.encoding]::utf8.getchars([System.Text.UTF8Encoding]::UTF8.GetBytes('𩸽')) returns two unprintables, rather than a single character.

---

Explanation

Very straightforward code. Takes the input $args[0] and explicitly casts it as a char-array so we can loop through each component of the string |%{...}. Each iteration, we use the .NET call [System.Text.Encoding]::UTF8.GetByteCount() (the System. is implied) to get the byte count of the current character $_. That's placed on the pipeline for later output. Since it's a collection of [int]s that are returned, casting to an array is implicit.

Test Runs

PS C:\Tools\Scripts\golfing> .\bytes-per-character.ps1 'tʃaʊ'
1
2
1
2

PS C:\Tools\Scripts\golfing> .\bytes-per-character.ps1 'Adám'
1
1
2
1

PS C:\Tools\Scripts\golfing> .\bytes-per-character.ps1 'ĉaŭ'
2
1
2

PS C:\Tools\Scripts\golfing> .\bytes-per-character.ps1 'ĉaŭ'
1
2
1
1
2

PS C:\Tools\Scripts\golfing> .\bytes-per-character.ps1 'チャオ'
3
3
3

PS C:\Tools\Scripts\golfing> .\bytes-per-character.ps1 '!±≡𩸽'
1
2
3
3
3

Edited to add This does properly account for the null-bytes requirement that was added to the challenge after I originally posted, provided you pull the data from a text file and pipe it as follows:

PS C:\Tools\Scripts\golfing> gc .\z.txt -Encoding UTF8|%{.\bytes-per-character.ps1 $_}
2
1
1
1

Answer 7

JavaScript (ES6), 54 45 43 bytes

s=>[...s].map(c=>encodeURI(c).length/3-8&7)

Edit: Saved 2 bytes with help from @l4m2.

Answer 8

Ruby, 33 bytes

Barely edges out Python, yay! Try it online.

->s{s.chars.map{|c|c.bytes.size}}

Answer 9

Perl 6,  77 69  63 bytes

put +$0 if $_».base(2).fmt("%8d")~~/^(1)**2..*|^(" ")/ while $_=$*IN.read: 1

put +$0 if $_».fmt("%8b")~~/^(1)**2..*|^(" ")/ while $_=$*IN.read: 1

put 1+$0 if $_».fmt("%8b")~~/^1(1)+|^" "/while $_=$*IN.read: 1

put 1+$0 if $_».fmt("%0.8b")~~/^1(1)+|^0/while $_=$*IN.read: 1

Since Perl 6 uses NFG strings I have to pull in the bytes directly, which sidesteps the feature.
(NFG is like NFC except it also creates synthetic composed codepoints)

The output is separated by newlines.

Test:

for text in '!' 'Ciao' 'tʃaʊ' 'Adám' 'ĉaŭ' 'ĉaŭ' 'チャオ' '' '!±≡𩸽' '𩸽\0𩸽';
do
  echo -en $text |
  perl6 -e 'put 1+$0 if $_».fmt("%8b")~~/^1(1)+|^" "/while $_=$*IN.read: 1' |

  # combine all of the lines into a single one for display purposes
  env text=$text perl6 -e 'put qq["%*ENV<text>"], "\t\t", lines.gist'
done

"!"     (1)
"tʃaʊ"      (1 2 1 2)
"Adám"      (1 1 2 1)
"ĉaŭ"       (2 1 2)
"ĉaŭ"     (1 2 1 1 2)
"チャオ"       (3 3 3)
""      ()
"!±≡𩸽"     (1 2 3 4)
"𩸽\0𩸽"        (4 1 4)

Explanation:

# turns the list in ｢$0｣ into a count, and adds one
# ｢put｣ prints that with a trailing newline
put 1+$0 

   # if the following is true
   if

       # format the input byte to base 2 and pad it out to 8 characters
       $_».fmt("%8b")

       ~~ # smart match against

       # check to see if it starts with more than one 1s, or a space
       # ( also sets ｢$0｣ to a list that is 1 shorter
       # than the number of bytes in this codepoint )
       / ^1 (1)+ | ^" " /

           # for every byte in STDIN
           while
               $_ = $*IN.read: 1

This works because the first byte in a multi-byte codepoint has the number of bytes encoded inside of it, and the other bytes in the codepoint have the highest bit set, but not the next highest. While the single byte codepoints don't have the highest bit set.

Answer 10

Python 3, 82 bytes

import math
lambda x:[ord(i)<128and 1or int((math.log2(ord(i))-1)//5+1)for i in x]

This is much longer than the other Python answer, and the majority of the other answers, but uses an approach involving logarithms that I haven't yet seen.

An anonymous function that takes input, via argument, as a string and returns a list.

Try it on Ideone

How it works

This method relies on the way in which UTF-8 encodes the code-point of a character. If the code-point is less than 128, the character is encoded as in ASCII:

0xxxxxxx

where x represents the bits of the code point. However, for code-points greater than or equal to 128, the first byte is padded with the same number of 1 s as the total number of bytes, and subsequent bytes begin 10. The bits of the code-point are then entered to give the shortest possible multibyte sequence, and any remaining bits become 0.

No. of bytes  Format
1             0xxxxxxx
2             110xxxxx 10xxxxxx
3             1110xxxx 10xxxxxx 10xxxxxx
4             11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
...           ...

and so forth.

It can now be noticed that for each number of bytes n, the upper limit for the number of code-point bits is given by (-n+7)+6(n-1) = 5n+1. Hence, the upper limit code-point c for each n is given, in decimal, by c= 2^(5n+1). Rearranging this gives n = (log2(c)-1)/5. So for any code-point, the number of bytes can be found by evaluating the above expression, and then taking the ceiling.

However, this does not work for code points in the range 64 <= c <= 127, since the lack of a padding 1 due to the ASCII-like encoding for 1 byte characters means that the wrong upper limit is predicted, and log2 is undefined for c = 0, which happens if a null byte is present in the input. Therefore, if c <= 127, a value of 1 is returned for n.

This is exactly what the code is doing; for each character i in the string x, the code-point is found using the ord function, and the ceiling of the expression is found by using integer rather than float division by 5 and then adding 1. Since Python's float type always represents integers as x.0, even after integer division, the result is passed to the int function to remove the trailing zero. If ord(i) <= 127, logical short-circuiting means that 1 is instead returned. The number of bytes for each character is stored as an element in a list, and this list is returned.

Answer 11

Java 10, 100 96 95 67 61 bytes

a->{for(var c:a)System.out.print(c.getBytes("utf8").length);}

-4 bytes removing spaces because this is allowed in the comments
-1 byte changing UTF-8 to utf8
-28 bytes going from Java 7 to 8 (a->{...} instead of void c(char[]i)throws Exception{...})
-3 bytes taking the input as String-array instead of character-array, and
-3 bytes going from Java 8 to 10 (var instead of String)

Explanation:

Try it online.

a->{                      // Method with String-array parameter and no return-type
  for(var c:a)            //  Loop over the input-array
    System.out.print(     //   Print:
      c.getBytes("utf8")  //    The bytes as array in UTF-8 of the current item,
       .length);}         //    and print the amount of bytes in this array

Answer 12

Julia, 34 bytes

s->s>""?map(sizeof,split(s,"")):[]

This is an anonymous function that accepts a string and returns an integer array. To call it, assign it to a variable.

The approach is quite straightforward: If the input is empty, the output is empty. Otherwise we map the sizeof function, which counts the number of bytes in a string, to each one-character substring.

Try it online! (includes all test cases)

Answer 13

PHP, 92 57 bytes

On second thought you can do this with much less faffing around:

<?php for(;$a=strlen(mb_substr($argv[1],$i++,1));)echo$a;

Try it online note that this is slightly longer as it uses stdin rather than a program argument.
This version requires you to ignore notices sent to stderr but that's fine.

old version:
Uses a rather different approach to the other php answer. Relies on the lack of native support for multi-byte strings in php.

<?php for($l=strlen($a=$argv[1]);$a=mb_substr($a,1);$l=$v)echo$l-($v=strlen($a));echo$l?:'';

Answer 14

Emacs Lisp, 55 49 bytes

(lambda(s)(mapcar'string-bytes(mapcar'string s)))

First dissects the string into a list of characters with (mapcar 'string s). The string function in Emacs Lisp takes a list of characters and builds a string out of them. Due to the way Emacs splits strings with mapcar (i.e. into a list of integers, not characters or strings), this explicit conversion is needed. Then maps the string-bytes function onto that list of strings.

Example:

(mapcar 'string "abc") ; => ("a" "b" "c")
(mapcar 'string-bytes '("a" "b" "c")) ; => (1 1 1) 

Testcases:

(mapcar
 (lambda(s)(mapcar'string-bytes(mapcar'string s)))
 '("!""Ciao""tʃaʊ""Adám""ĉaŭ""ĉaŭ""チャオ""""!±≡𩸽""\0"))
;; ((1) (1 1 1 1) (1 2 1 2) (1 1 2 1) (2 1 2) (1 2 1 1 2) (3 3 3) nil (1 2 3 4) (1))

Old answer:

(lambda(s)(mapcar(lambda(s)(string-bytes(string s)))s))

Ungolfed:

 (lambda (s)
   (mapcar
    ;; we can't use string-bytes directly,
    ;; since Emacs mapcar yields a list of ints instead of characters
    ;; therefore we need a wrapper function here. 
    (lambda (s)
      (string-bytes (string s)))
    s))

Testcases:

(mapcar
 (lambda(s)(mapcar(lambda(s)(string-bytes(string s)))s))
 '("!""Ciao""tʃaʊ""Adám""ĉaŭ""ĉaŭ""チャオ""""!±≡𩸽""\0"))
;; ((1) (1 1 1 1) (1 2 1 2) (1 1 2 1) (2 1 2) (1 2 1 1 2) (3 3 3) nil (1 2 3 4) (1))

Answer 15

JavaScript (Node), 27 bytes

s=>s.map(Buffer.byteLength)

This takes input as an array of individual characters, and returns an array of byte counts.

Buffer is a method of representing raw binary data. Buffer.byteLength(string) gives the number of bytes in the string. UTF-8 is the default encoding. Note that only Node.js has buffers, not browser JS. The rough browser equivalent is called Blob, which comes in at 31 bytes:

s=>s.map(e=>new Blob([e]).size)

Test

Save this file and run it through node, or try it online.

var f =
  s=>s.map(Buffer.byteLength)

var tests = [
  ["!"],
  ["C","i","a","o"],
  ["t","ʃ","a","ʊ"],
  ["A","d","á","m"],
  ["ĉ","a","ŭ"],
  ["c","̂","a","u","̆"],
  ["チ","ャ","オ"],
  [],
  ["!","±","≡","𩸽"]
];

tests.forEach(test => {
  console.log(test, f(test));
});

This should be the result:

$ node bytes.js
[ '!' ] [ 1 ]
[ 'C', 'i', 'a', 'o' ] [ 1, 1, 1, 1 ]
[ 't', 'ʃ', 'a', 'ʊ' ] [ 1, 2, 1, 2 ]
[ 'A', 'd', 'á', 'm' ] [ 1, 1, 2, 1 ]
[ 'ĉ', 'a', 'ŭ' ] [ 2, 1, 2 ]
[ 'c', '̂', 'a', 'u', '̆' ] [ 1, 2, 1, 1, 2 ]
[ 'チ', 'ャ', 'オ' ] [ 3, 3, 3 ]
[] []
[ '!', '±', '≡', '�' ] [ 1, 2, 3, 4 ]

Answer 16

Bash, 74 bytes

Golfed

xxd -p|fold -2|cut -c1|tr -d '89ab'|echo `tr -t '01234567cbef' '[1*]2234'`

Algorithm

hexdump input string, fold 2 chars per line, cut the first char only

echo -ne '!±≡𩸽' | xxd -p|fold -2|cut -c1

2
c
b
e
8
a
f
a
b
b

(4 high order bits of an each input byte as a hex char, one per line)

Remove "continuation bytes" 0x80..0xBF

tr -d '89ab'

2
c

e


f

(what is left, is 4 bits of the first byte of an each unicode char)

map the first bits into the char length, collapse the output and print

echo `tr -t '01234567cbef' '[1*]2234'`

1 2 3 4

Test

 U() { xxd -p|fold -2|cut -c1|tr -d '89ab'|echo `tr -t '01234567cbef' '[1*]2234'`;}

 echo -ne '!' | U 
 1

 echo -ne 'Ciao' | U
 1 1 1 1

 echo -ne 'tʃaʊ' | U
 1 2 1 2

 echo -ne 'Adám' | U
 1 1 2 1

 echo -ne 'ĉaŭ' | U
 2 1 2

 echo -ne 'ĉaŭ' | U
 1 2 1 1 2

 echo -ne 'チャオ' | U
 3 3 3
 echo -ne '!±≡𩸽' | U
 1 2 3 4

 echo -ne "\x0" | U
 1

 echo -ne '' | U

Answer 17

PHP, 126 bytes

<?php $s=fgets(STDIN);echo $s!=''?implode(' ',array_map(function($x){return strlen($x);},preg_split('/(?<!^)(?!$)/u',$s))):'';

Try it online!

Answer 18

C#, 89 82 bytes

I=>{var J="";foreach(char c in I){J+=Encoding.UTF8.GetByteCount(c+"");}return J;};

A simple C# lambda that iterates through the string and returns the space separated list.

Edit: saved 6 bytes thanks to some very nice comments.

Answer 19

Haskell, 85 bytes

import Data.ByteString as B
import Data.ByteString.UTF8
(B.length.fromString.pure<$>)

Answer 20

Pyth, 17 bytes

mhxS+11+16,7lCdlC

Try it online!

Use the code-point of the characters with some arithmetics.

Answer 21

C, 85 bytes.

l(unsigned char* c){while(*c){int d=(*c>>4)-11;
d=d<0?1:d+(d==1);putchar(48+d);c+=d;}}

Examines the high 4 bits of each byte to determine the encoding and the number of subsequent bytes to skip;

Answer 22

Factor, 57 87 82 80 bytes

[ [ dup zero? [ drop "1"] [ >bin length 4 /i 10 >base ] if ] { } map-as ""join ]

Explained:

USING: kernel math math.parser sequences ;
IN: byte-counts

: string>byte-counts ( str -- counts )
  [                  ! new quotation: takes a char as a fixnum
    dup zero?        ! true if this is a NUL byte
    [ drop "1" ]     ! NUL bytes have length 1
    [ >bin           ! else, convert to binary string
      length         ! length of binary string
      4              ! the constant 4
      /i             ! integer division
      number>string  ! 4 -> "4"
    ] if             ! conditionally execute one of the previous quotations
  ]                  ! end
  { } map-as         ! map and clone-like an { } array
  "" join ;          ! join array of 1strings on empty string

Unit tests:

USING: tools.test byte-counts ;
IN: byte-counts.tests

{ "1" } [ "!" string>byte-counts ] unit-test
{ "1111" } [ "Ciao" string>byte-counts ] unit-test
{ "1212"} [ "tʃaʊ" string>byte-counts ] unit-test
{ "1121" } [ "Adám" string>byte-counts ] unit-test
{ "212" } [ "ĉaŭ" string>byte-counts ] unit-test
{ "12112" } [ "ĉaŭ" string>byte-counts ] unit-test
{ "333" } [ "チャオ" string>byte-counts ] unit-test
{ "" } [ "" string>byte-counts ] unit-test
{ "1234" } [ "!±≡𩸽" string>byte-counts ] unit-test
{ "1" } [ "\0" string>byte-counts ] unit-test

They all pass, now. c:

Answer 23

Swift 2.2, 67 52 50 bytes

for c in i.characters{print(String(c).utf8.count)}

Horribly ugly. There's no way to get the UTF-8 length of a Character in Swift, so I need to iterate through the string by character, convert the Character to a String, and find the count of that single-character String (hey, at least there's a built-in method to do that). Looking for optimizations, possibly using a scanner.

Revision 1: Saved 15 bytes by using count instead of underestimateCount().

Revisions 2: Saved another 2 character by using a for-in loop instead of a for each closure.

Answer 24

Rust, 53 bytes

|s:&str|for c in s.chars(){print!("{}",c.len_utf8())}

Rust has utf-8 char primitives, iterators, and lambdas, so this was straightforward. Test code:

fn main() {
    let s = "Löwe 老虎 Léopard💖💖💖💖";
    let f =|s:&str|for c in s.chars(){print!("{}",c.len_utf8())};
    f(s);
}

Outputs

1211133112111114444 

Answer 25

jq, 26 characters

(23 characters code + 3 characters command line option)

(./"")[]|utf8bytelength

Hopefully competing. Although utf8bytelength was added 9++ months before this question, it is still not included in released version.

Sample run:

bash-4.3$ ./jq -R '(./"")[]|utf8bytelength' <<< 'tʃaʊ'
1
2
1
2

bash-4.3$ ./jq -R '(./"")[]|utf8bytelength' <<< 'ĉaŭ '
1
2
1
1
2
1

bash-4.3$ ./jq -R '(./"")[]|utf8bytelength' <<< 'チャオ'
3
3
3

bash-4.3$ ./jq -R '(./"")[]|utf8bytelength' <<< ''

bash-4.3$ ./jq -R '(./"")[]|utf8bytelength' <<< '!±≡𩸽'
1
2
3
4

Answer 26

C (gcc), 53 bytes

k=49;f(char*s){*++s/64-2?k=puts(&k)+47:++k;*s&&f(s);}

Try it online!

Answer 27

SmileBASIC, 69 bytes

DEF C B
WHILE I<LEN(B)Q=INSTR(BIN$(B[I],8),"0")I=I+Q+!Q?Q+!Q
WEND
END

Input is an array of bytes.

The number of bytes in a UTF-8 character is equal to the number of leading 1 bits in the first byte (unless there are no 1s, in which case the character is 1 byte). To find the number of leading 1s, the program finds the first 0 in the binary representation, then adds 1 if this was 0.

0xxxxxxx - no leading ones, 1 byte
110xxxxx 10xxxxxx - 2 leading ones, 2 bytes
1110xxxx 10xxxxxx 10xxxxxx - 3 leading ones, 3 bytes
etc.

Answer 28

F#, 59 54 66 bytes

(s)=seq{for c in s->System.Text.Encoding.UTF8.GetByteCount([|c|])}

Technically, s is a char sequence, but it turns out there's an implicit conversion that allows a string to be passed in.

When testing this in the console with !±≡𩸽, it splits the kanji into two characters, each 3 bytes long. All the other test cases work fine.

Edit: It turns out common namespace imports are not implicit. Up another 12 chars.

Answer 29

05AB1E, 15 bytes

ÇεDžy‹i1ë.²<5÷>

Try it online.
Header ε is used to for-each over all the test cases;
Footer ï]J]» to pretty-print the output character-lists (ï: decimals and characters to integers; ]: close if-else and for-each; J: Join digits together; }: close header foreach; »: Join by new-lines).

Explanation:

Ç                   # Convert each character to its unicode value
 εD                 # Foreach over this list
      i             #  If the current item
     ‹              #  is smaller than
   žy               #  128
       1            #   Use 1
        ë           #  Else
         .²         #   Use log_2
           <        #   minus 1
            5÷      #   integer-divided by 5
              >     #   plus 1

Since 05AB1E doesn't have any builtins to convert characters to amount of bytes used, I use Ç to convert the characters to their unicode values, and in a for-each do the following in pseudo-code:

if(unicodeValue < 128)
  return 1
else
  return log_2(unicodeValue-1)//5+1    # (where // is integer-division)

Inspired by @TheBikingViking's Python 3 answer.

Answer 30

Zsh, 41 bytes

for c (${(s::)1})set +o multibyte&&<<<$#c

Try it online!

Zsh is UTF-8 aware, so we split the string on characters, then disable multibyte and print each character's length.