Gramur

3.2.0-RC1

Gramur/net.archandle.gramur.lexer/Lexer

Lexer

class Lexer(val lexemes: List<Lexeme<*>>, val factoryOverrides: Map<Lexeme<*>, (value: CharSequence) -> Token> = emptyMap())

The Lexer transforms raw character input into a stream of tokens based on defined lexemes.

A Lexer processes input sequentially, attempting to match each Lexeme, in declaration order, at the current position. When a Lexeme produces a match, the corresponding Token is emitted and the Lexer advances. If no Lexeme matches at the current position, a LexerException is thrown.

A Lexer may include a definable map of factory overrides for any Lexeme declared in lexemes; if a particular Lexeme has a factory override defined, it will be used to generate Tokens for each match of that Lexeme, with precedence over Lexeme.defaultFactory (but not over factories individually specified by LexemeMatch.factory).

See also

Constructors

constructor(vararg lexemes: Lexeme<*>)

Creates a Lexer from the given lexemes.

constructor(factoryOverrides: Map<Lexeme<*>, (value: CharSequence) -> Token>, vararg lexemes: Lexeme<*>)

Creates a Lexer from the given lexemes and factoryOverrides.

constructor(lexemes: List<Lexeme<*>>, factoryOverrides: Map<Lexeme<*>, (value: CharSequence) -> Token> = emptyMap())

Types

class LexerException : RuntimeException

Exception thrown when lexing fails.

Properties

factoryOverrides

val factoryOverrides: Map<Lexeme<*>, (value: CharSequence) -> Token>

An optional Map of Lexemes to a factory override which produces Tokens, given the CharSequence which captures the Token (equal to Token.value)

val lexemes: List<Lexeme<*>>

The ordered list of Lexeme definitions to match against input

Functions

fun ~~lex~~(input: CharSequence, discardEnabled: Boolean = true): List<Token>

fun tokenize(input: CharSequence, discardEnabled: Boolean = true): List<Token>

Transforms the given character input into a list of tokens.