ANTLR for Ruby

Tokens

updated Sunday, August 04, 2013 at 09:59PM EDT

Tokens are basic currency within your grammar file; each one represents a slice of source code text of a particular type. Lexers exist soley to create tokens; parsers exist soley to react when a sequence of tokens fits a specific pattern. Furthermore, as far as ANTLR is concerned, Abstract Syntax Trees (ASTs) are just tokens that have been organized hierarchically. Since tokens are a crucial aspect of ANTLR’s recognition semantics, I want to cover their structure and characteristics in Ruby code.

At a minimum, a token must track:

Moreover, recognition tasks often require additional information associated with tokens:

In the antlr3 runtime library, tokens are implemented as simple data objects which track all of the properties listed above. While you are able to add your own customized token implementation, the Struct-based class, ANTLR3::CommonToken, should track all token data required to perform your recognition task.

Token Objects

Attributes

Name Data Type Description
type Integer
text String or nil
input_stream ANTLR3::CharacterStream
start Integer
stop Integer
index Integer
channel Symbol

Providing a Custom Token Implementation

Token Schemes