Token

abstract class Token

Represents a lexical token produced by matching a Lexeme against input.

A Token encapsulates the matched text (value), its source position (line and column), and a reference to the Lexeme that produced it. Tokens are created by the Lexer during the tokenization process (Lexer.tokenize).

Custom token types can extend this class to carry additional metadata beyond the matched text.

Inheritors

Generic

Constructors

Token

constructor()

Types

Generic

open class Generic : Token

A generic Token implementation with no supplementary metadata and value-based equality.

Properties

column

var column: Int

The column number where this token begins in the source input.

isDiscardable

open val isDiscardable: Boolean

Indicates whether this token should be discarded during lexing (when discard mode is enabled).

lexeme

lateinit var lexeme: Lexeme<*>

The Lexeme that produced this token.

line

var line: Int

The line number where this token begins in the source input.

value

lateinit var value: CharSequence

The character sequence matched from the input that this token represents.

Functions

toString

open override fun toString(): String