com.raritantechnologies.vqlparser
Class Lexer

java.lang.Object
  com.raritantechnologies.vqlparser.Lexer

public class Lexer

extends java.lang.Object

This class implements a small scanner (aka lexical analyzer or lexer). Creates a series of Node objects that represent query tokens.

In addition to the scanner proper (called first via init() then with next_token() to get each Symbol) this class provides simple error and warning routines and keeps a count of errors and warnings that is publicly accessible.

Developed by Raritan Technologies Inc..

Author:

Kepler Gelotte

Field Summary

protected int	absolute_position Character position in entire query.
protected boolean	at_eof State of the query stream: at end of file
protected char[]	buf build the token here.
protected int	buffer_position Character position in the token buffer.
protected static int	BUFSIZ
protected int	current_line Current line number for use in error messages.
protected int	current_position Character position in current line.
protected static int	EOF_CHAR EOF constant.
int	error_count Count of total errors detected so far.
protected SymbolTable	keywords Symbol table holding keywords.
protected int	level Parenthesis level
protected int	next_char First character of lookahead.
protected int	next_char2 second character of lookahead.
protected java.io.Reader	reader_in
protected java.io.InputStream	stream_in Stream containing query to be parsed
int	warning_count Count of warnings issued so far

Constructor Summary

Lexer(java.io.InputStream lexIn)
This constructor takes an inputStream as a parameter.

Lexer(java.io.Reader lexIn)
This constructor takes a Reader as a parameter.

Lexer(java.lang.String vqlString)

Method Summary

protected void	advance() Advance the scanner one character in the input stream.
Node	debug_next_token() Debugging version of next_token().
void	emit_error(java.lang.String message) Emit an error message.
void	emit_warn(java.lang.String message) Emit a warning message.
int	getLevel() Access to the level.
protected boolean	id_char(int ch) Determine if a character is ok for the middle of an id.
protected boolean	id_start_char(int ch) Determine if a character is ok to start an id.
Node	next_token() Return one Node.
protected Node	real_next_token() The actual routine to return one Node.
protected void	swallow_comment() Handle swallowing up a comment.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EOF_CHAR

protected static final int EOF_CHAR

EOF constant.

See Also:

Constant Field Values

BUFSIZ

protected static final int BUFSIZ

See Also:

Constant Field Values

stream_in

protected java.io.InputStream stream_in

Stream containing query to be parsed

reader_in

protected java.io.Reader reader_in

at_eof

protected boolean at_eof

State of the query stream: at end of file

keywords

protected SymbolTable keywords

Symbol table holding keywords.

next_char

protected int next_char

First character of lookahead.

next_char2

protected int next_char2

second character of lookahead.

buf

protected char[] buf

build the token here.

level

protected int level

Parenthesis level

current_line

protected int current_line

Current line number for use in error messages.

current_position

protected int current_position

Character position in current line.

absolute_position

protected int absolute_position

Character position in entire query.

buffer_position

protected int buffer_position

Character position in the token buffer.

error_count

public int error_count

Count of total errors detected so far.

warning_count

public int warning_count

Count of warnings issued so far

Constructor Detail

Lexer

public Lexer(java.lang.String vqlString)
      throws java.io.IOException

Lexer

public Lexer(java.io.InputStream lexIn)
      throws java.io.IOException

This constructor takes an inputStream as a parameter.

Lexer

public Lexer(java.io.Reader lexIn)
      throws java.io.IOException

This constructor takes a Reader as a parameter.

Method Detail

advance

protected void advance()
                throws java.io.IOException

Advance the scanner one character in the input stream. Also set the eof state when no more characters are available.

Throws:

java.io.IOException

getLevel

public int getLevel()

Access to the level.

emit_error

public void emit_error(java.lang.String message)

Emit an error message. The message will be marked with both the current line number and the position in the line. Error messages are printed on standard error (System.err).

Parameters:

message - the message to print.

emit_warn

public void emit_warn(java.lang.String message)

Emit a warning message. The message will be marked with both the current line number and the position in the line. Messages are printed on standard error (System.err).

Parameters:

message - the message to print.

id_start_char

protected boolean id_start_char(int ch)

Determine if a character is ok to start an id.

Parameters:

ch - the character in question.

id_char

protected boolean id_char(int ch)

Determine if a character is ok for the middle of an id.

Parameters:

ch - the character in question.

swallow_comment

protected void swallow_comment()
                        throws java.io.IOException

Handle swallowing up a comment. Both old style C and new style C++ comments are handled.

Throws:

java.io.IOException

next_token

public Node next_token()
                throws java.io.IOException

Return one Node. This is the main external interface to the scanner. It consumes sufficient characters to determine the next input Node and returns it. To help with debugging, this routine actually calls real_next_token() which does the work. If you need to debug the parser, this can be changed to call debug_next_token() which prints a debugging message before returning the symbol as a Node.

Throws:

java.io.IOException

debug_next_token

public Node debug_next_token()
                      throws java.io.IOException

Debugging version of next_token(). This routine calls the real scanning routine, prints a message on System.out indicating what the Node is, then returns it.

Throws:

java.io.IOException

real_next_token

protected Node real_next_token()
                        throws java.io.IOException

The actual routine to return one Node. This is normally called from next_token(), but for debugging purposes can be called indirectly from debug_next_token().

Throws:

java.io.IOException