15
\$\begingroup\$

I've been working on another stack-based golfing language called Stackgoat. In this challenge you'll be writing a Tokenizer for Stackgoat (or really any general stack-based languages).

Examples

"PPCG"23+
["PPCG", '23', '+']

'a "bc" +
['"a"', '"bc"', '+']

12 34+-"abc\"de'fg\\"
['12', '34', '+', '-', '"abc\"de'fg\\"']

"foo
['"foo"']

(empty input)
[]

' ""
['" "', '""']

Specification

The three types you'll need to handle are:

  • Strings, anything within ""
  • Numbers, any sequence of digits
  • Operators, any other single character besides whitespace

Whitespace is essentially ignored unless it is within a string or separates two numbers.

String / char spec:

  • Strings are delimited by a ", and when a \ is encountered, the next character should be escaped.
  • Chars are prepended by a ' and the character after the ' should be converted into a string literal. 'a -> "a"
  • ' will always have a character after it
  • Closing quotes should be auto-inserted

Rules:

  • No form of eval is allowed

Input / Output:

  • Input can be taken through STDIN, function parameters, or your language's equivalent.
  • Output should be an array or your language's closest equivalent.
\$\endgroup\$
  • 5
    \$\begingroup\$ @Doorknob, seriously? \$\endgroup\$ – LegionMammal978 Jan 9 '16 at 23:23
  • 4
    \$\begingroup\$ @LegionMammal978 Yes, seriously. \$\endgroup\$ – Alex A. Jan 9 '16 at 23:44
  • 1
    \$\begingroup\$ Can output be to STDOUT? \$\endgroup\$ – Doorknob Jan 10 '16 at 0:04
  • 2
    \$\begingroup\$ @ZachGates Well yes, most languages do handle \ as an escape character too, so yes, you will need to escape that if your language needs it obviously. \$\endgroup\$ – Downgoat Jan 13 '16 at 4:19
  • 1
    \$\begingroup\$ Also, in the first example, should the first element of the result be '"PPCG"' instead of just "PPCG"? \$\endgroup\$ – Nic Hartley Jan 13 '16 at 17:24
8
+50
\$\begingroup\$

Retina, 68 64 63 bytes

M!s`"(\\.|[^"])*"?|'.|\d+|\S
ms`^'(.)|^"(([^\\"]|\\.)*$)
"$1$2"

or

s`\s*((")(\\.|[^"])*(?<-2>")?|'.|\d+|.)\s*
$1$2¶
\ms`^'(.)
"$1"

I think this covers all the funky edge cases, even those not covered by the test cases in the challenge.

Try it online!

\$\endgroup\$
  • \$\begingroup\$ Dang, this is short. Nicely done! \$\endgroup\$ – Nic Hartley Jan 13 '16 at 19:37
  • \$\begingroup\$ I was able to translate this into a 95 byte ES6 function. It would have been 80 except that the regexps don't work the other way around (too many edge cases). \$\endgroup\$ – Neil Jan 18 '16 at 13:29
2
\$\begingroup\$

Ruby, 234 bytes

puts"[#{$stdin.read.scan(/("(?:(?<!\\)\\"|[^"])+(?:"|$))|'(.)|(\d+)|(.)/).map{|m|(m[0]?(m[0].end_with?('"')?m[0]: m[0]+'"'): m[1]?"\"#{m[1]}\"": m.compact[0]).strip}.reject(&:empty?).map{|i|"'#{/\d+|./=~i ?i: i.inspect}'"}.join', '}]"

I tried using the find(&:itself) trick that I saw... somewhere, but apparently .itself isn't actually a method. Also, I'm working on golfing the regex down, but it's already unreadable.

If we don't have to output in any fancy way (i.e. strings don't have to be quoted in the array) I can save a whole lotta bytes:

Still Ruby, 194 bytes:

p$stdin.read.scan(/("(?:(?<!\\)\\"|[^"])+(?:"|$))|'(.)|(\d+)|(.)/).map{|m|(m[0]?(m[0].end_with?('"')?m[0]: m[0]+'"').gsub(/\\(.)/,'\1'): m[1]?"\"#{m[1]}\"": m.compact[0]).strip}.reject(&:empty?)

I'm sure I can golf it more, but I'm not quite sure how.


Ungolfed coming soon. I started fiddling with the golfed directly at some point and I'll have to tease it out.

\$\endgroup\$
0
\$\begingroup\$

Python 3, 228 bytes

import re;L=list
print(L(map(lambda i:i+'"'if i[0]=='"'and not i[-1]=='"'else i,map(lambda i:'"%s"'%i[1]if i[0]=="'"else i,filter(None,sum([L(i)for i in re.findall('(\'.)|(".*")|(\d+)|([^\w\"\'\s\\\])|(".*"?)',input())],[]))))))

Here's a nice, long, two-liner.


Test it out in Python 3. Here's some examples:

$ python3 test.py
"PPCG"23+
['"PPCG"', '23', '+']

$ python3 test.py
'a "bc" +
['"a"', '"bc"', '+']

$ python3 test.py
12 34+-"abc"de'fg\"
['12', '34', '+', '-', '"abc"de\'fg\\"']

$ python3 test.py
"foo
['"foo"']

$ python3 test.py

[]

$ python3 test.py
' ""
['" "', '""']
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.