The source code is tokenized linearly (single-pass), using simple rules about which characters are allowed to be adjacent within a token. When a character is encountered that cannot be adjacent to the last character of the current token, the current token is deemed complete and the new character begins a new token. Some characters (such as {
or ,
) cannot be adjacent to any other characters and are therefore their own token. Others (like >
or =
) are only allowed to be adjacent to themselvesother characters within their class, and can thus form tokens like >>>
or, ==
, or >=
, but not like =2
. Whitespace characters force a boundary between tokens but aren't themselves included in the result. The most difficult character to tokenize is -
because it can both represent subtraction and unary negation, and thus requires some special-casing.