# Tokenize a Stack-Based language

I've been working on another stack-based golfing language called Stackgoat. In this challenge you'll be writing a Tokenizer for Stackgoat (or really any general stack-based languages).

## Examples

"PPCG"23+
["PPCG", '23', '+']

'a "bc" +
['"a"', '"bc"', '+']

12 34+-"abc\"de'fg\\"
['12', '34', '+', '-', '"abc\"de'fg\\"']

"foo
['"foo"']

(empty input)
[]

' ""
['" "', '""']


## Specification

The three types you'll need to handle are:

• Strings, anything within ""
• Numbers, any sequence of digits
• Operators, any other single character besides whitespace

Whitespace is essentially ignored unless it is within a string or separates two numbers.

String / char spec:

• Strings are delimited by a ", and when a \ is encountered, the next character should be escaped.
• Chars are prepended by a ' and the character after the ' should be converted into a string literal. 'a -> "a"
• ' will always have a character after it
• Closing quotes should be auto-inserted

Rules:

• No form of eval is allowed

Input / Output:

• Input can be taken through STDIN, function parameters, or your language's equivalent.
• Output should be an array or your language's closest equivalent.
• @Doorknob, seriously? Jan 9, 2016 at 23:23
• @LegionMammal978 Yes, seriously. Jan 9, 2016 at 23:44
• Can output be to STDOUT? Jan 10, 2016 at 0:04
• @ZachGates Well yes, most languages do handle \ as an escape character too, so yes, you will need to escape that if your language needs it obviously. Jan 13, 2016 at 4:19
• Also, in the first example, should the first element of the result be '"PPCG"' instead of just "PPCG"?
– Nic
Jan 13, 2016 at 17:24

## Retina, 6864 63 bytes

M!s"(\\.|[^"])*"?|'.|\d+|\S
ms^'(.)|^"(([^\\"]|\\.)*$) "$1$2"  or s\s*((")(\\.|[^"])*(?<-2>")?|'.|\d+|.)\s*$1$2¶ \ms^'(.) "$1"


I think this covers all the funky edge cases, even those not covered by the test cases in the challenge.

Try it online!

• Dang, this is short. Nicely done!
– Nic
Jan 13, 2016 at 19:37
• I was able to translate this into a 95 byte ES6 function. It would have been 80 except that the regexps don't work the other way around (too many edge cases).
– Neil
Jan 18, 2016 at 13:29

### Ruby, 234 bytes

puts"[#{$stdin.read.scan(/("(?:(?<!\\)\\"|[^"])+(?:"|$))|'(.)|(\d+)|(.)/).map{|m|(m[0]?(m[0].end_with?('"')?m[0]: m[0]+'"'): m[1]?"\"#{m[1]}\"": m.compact[0]).strip}.reject(&:empty?).map{|i|"'#{/\d+|./=~i ?i: i.inspect}'"}.join', '}]"


I tried using the find(&:itself) trick that I saw... somewhere, but apparently .itself isn't actually a method. Also, I'm working on golfing the regex down, but it's already unreadable.

If we don't have to output in any fancy way (i.e. strings don't have to be quoted in the array) I can save a whole lotta bytes:

### Still Ruby, 194 bytes:

p$stdin.read.scan(/("(?:(?<!\\)\\"|[^"])+(?:"|$))|'(.)|(\d+)|(.)/).map{|m|(m[0]?(m[0].end_with?('"')?m[0]: m[0]+'"').gsub(/\\(.)/,'\1'): m[1]?"\"#{m[1]}\"": m.compact[0]).strip}.reject(&:empty?)


I'm sure I can golf it more, but I'm not quite sure how.

Ungolfed coming soon. I started fiddling with the golfed directly at some point and I'll have to tease it out.

# Python 3, 228 bytes

import re;L=list
print(L(map(lambda i:i+'"'if i[0]=='"'and not i[-1]=='"'else i,map(lambda i:'"%s"'%i[1]if i[0]=="'"else i,filter(None,sum([L(i)for i in re.findall('(\'.)|(".*")|(\d+)|([^\w\"\'\s\\\])|(".*"?)',input())],[]))))))


Here's a nice, long, two-liner.

Test it out in Python 3. Here's some examples:

$python3 test.py "PPCG"23+ ['"PPCG"', '23', '+']$ python3 test.py
'a "bc" +
['"a"', '"bc"', '+']

$python3 test.py 12 34+-"abc"de'fg\" ['12', '34', '+', '-', '"abc"de\'fg\\"']$ python3 test.py
"foo
['"foo"']

$python3 test.py []$ python3 test.py
' ""
['" "', '""']