com.raritantechnologies.utils.program
Class Lexer

java.lang.Object
  com.raritantechnologies.utils.program.Lexer

public class Lexer

extends java.lang.Object

This class implements a small scanner (aka lexical analyzer or lexer).

In addition to the scanner proper (called first via init() then with next_token() to get each Symbol) this class provides simple error and warning routines and keeps a count of errors and warnings that is publicly accessible.

Field Summary

protected int	absolute_position Character position in entire query.
protected boolean	at_eof State of the query stream
protected char[]	buf build the token here.
protected int	buffer_position Character position in the token buffer.
protected static int	BUFSIZ
protected int	current_line Current line number for use in error messages.
protected int	current_position Character position in current line.
protected static int	EOF_CHAR EOF constant.
int	error_count Count of total errors detected so far.
protected int	ifLevel If level (to support nested ifs)
protected SymbolTable	keywords Symbol table holding keywords.
protected int	next_char First character of lookahead.
protected int	next_char2 second character of lookahead.
protected int	parenLevel Parenthesis level
protected java.io.Reader	reader_in
protected java.io.InputStream	stream_in Stream containing query to be parsed
protected boolean	then_clause in the then clause or the else
int	warning_count Count of warnings issued so far

Constructor Summary

Lexer(java.io.InputStream lexIn)
The constructor takes an inputStream as a parameter .

Lexer(java.io.Reader lexIn)
The constructor takes a Reader as a parameter .

Method Summary

protected void	advance() Advance the scanner one character in the input stream.
Token	debug_next_token(java.util.HashMap typedef) Debugging version of next_token().
void	emit_error(java.lang.String message) Emit an error message.
void	emit_warn(java.lang.String message) Emit a warning message.
int	getCurrentLine() Return the current line number
int	getCurrentPosition() Return the current character position
int	getIfLevel() Access to the ifLevel .
int	getParenLevel() Access to the parenLevel .
protected boolean	id_char(int ch) Determine if a character is ok for the middle of an id.
protected boolean	id_start_char(int ch) Determine if a character is ok to start an id.
void	init_reader() Initialize the scanner.
void	init_stream() Initialize the scanner.
Token	next_token(java.util.HashMap typedef) Return one Token.
void	push_token(Token t) Pushes a token back onto the queue for re-evaluation
protected Token	real_next_token(java.util.HashMap typedef) The actual routine to return one Token.
protected void	swallow_comment() Handle swallowing up a comment.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EOF_CHAR

protected static final int EOF_CHAR

EOF constant.

See Also:

Constant Field Values

BUFSIZ

protected static final int BUFSIZ

See Also:

Constant Field Values

stream_in

protected java.io.InputStream stream_in

Stream containing query to be parsed

reader_in

protected java.io.Reader reader_in

at_eof

protected boolean at_eof

State of the query stream

keywords

protected SymbolTable keywords

Symbol table holding keywords.

next_char

protected int next_char

First character of lookahead.

next_char2

protected int next_char2

second character of lookahead.

buf

protected char[] buf

build the token here.

parenLevel

protected int parenLevel

Parenthesis level

ifLevel

protected int ifLevel

If level (to support nested ifs)

then_clause

protected boolean then_clause

in the then clause or the else

current_line

protected int current_line

Current line number for use in error messages.

current_position

protected int current_position

Character position in current line.

absolute_position

protected int absolute_position

Character position in entire query.

buffer_position

protected int buffer_position

Character position in the token buffer.

error_count

public int error_count

Count of total errors detected so far.

warning_count

public int warning_count

Count of warnings issued so far

Constructor Detail

Lexer

public Lexer(java.io.InputStream lexIn)
      throws java.io.IOException

The constructor takes an inputStream as a parameter .

Lexer

public Lexer(java.io.Reader lexIn)
      throws java.io.IOException

The constructor takes a Reader as a parameter .

Method Detail

getParenLevel

public int getParenLevel()

Access to the parenLevel .

getIfLevel

public int getIfLevel()

Access to the ifLevel .

init_stream

public void init_stream()
                 throws java.io.IOException

Initialize the scanner. This sets up the keywords tables and reads the first two characters of lookahead.

Throws:

java.io.IOException

init_reader

public void init_reader()
                 throws java.io.IOException

Initialize the scanner. This sets up the keywords tables and reads the first two characters of lookahead.

Throws:

java.io.IOException

advance

protected void advance()
                throws java.io.IOException

Advance the scanner one character in the input stream. Also set the eof state when no more characters are available.

Throws:

java.io.IOException

getCurrentLine

public int getCurrentLine()

Return the current line number

getCurrentPosition

public int getCurrentPosition()

Return the current character position

emit_error

public void emit_error(java.lang.String message)

Emit an error message. The message will be marked with both the current line number and the position in the line. Error messages are printed on standard error (System.err).

Parameters:

message - the message to print.

emit_warn

public void emit_warn(java.lang.String message)

Emit a warning message. The message will be marked with both the current line number and the position in the line. Messages are printed on standard error (System.err).

Parameters:

message - the message to print.

id_start_char

protected boolean id_start_char(int ch)

Determine if a character is ok to start an id.

Parameters:

ch - the character in question.

id_char

protected boolean id_char(int ch)

Determine if a character is ok for the middle of an id.

Parameters:

ch - the character in question.

swallow_comment

protected void swallow_comment()
                        throws java.io.IOException

Handle swallowing up a comment. Both old style C and new style C++ comments are handled.

Throws:

java.io.IOException

next_token

public Token next_token(java.util.HashMap typedef)
                 throws java.io.IOException

Return one Token. This is the main external interface to the scanner. It consumes sufficient characters to determine the next input Token and returns it. To help with debugging, this routine actually calls real_next_token() which does the work. If you need to debug the parser, this can be changed to call debug_next_token() which prints a debugging message before returning the symbol as a Token.

Throws:

java.io.IOException

debug_next_token

public Token debug_next_token(java.util.HashMap typedef)
                       throws java.io.IOException

Debugging version of next_token(). This routine calls the real scanning routine, prints a message on System.out indicating what the Token is, then returns it.

Throws:

java.io.IOException

push_token

public void push_token(Token t)

Pushes a token back onto the queue for re-evaluation

real_next_token

protected Token real_next_token(java.util.HashMap typedef)
                         throws java.io.IOException

The actual routine to return one Token. This is normally called from next_token(), but for debugging purposes can be called indirectly from debug_next_token().

Throws:

java.io.IOException