tokenize
Transforms the given input into a list of tokens using this Grammar's lexemes.
The lexer processes input sequentially, attempting to match each Lexeme in order at the current position. When discardEnabled is true, discardable tokens (such as whitespace) are excluded from the result.
Return
A list of tokens representing the lexical structure of input
Parameters
input
The character sequence to tokenize
discard
Whether to exclude discardable tokens from the result
Throws
if the input cannot be fully tokenized