com.raritantechnologies.verity.collection
Class FlatFileGateway

java.lang.Object
  com.raritantechnologies.verity.collection.FlatFileGateway

All Implemented Interfaces:

ICollectionGateway, java.lang.Runnable

Direct Known Subclasses:

BIFFileGateway

public class FlatFileGateway

extends java.lang.Object

implements ICollectionGateway

Reads a flat file or files to create sets of records (IResultSet) for indexing.

Configuration file defines the columns in the flat file - or in other words how the columns in the flat file map to fields in the XML record.

XML Configuration Template:

 <CollectionGateway name="FlatFiles" 
      class="com.raritantechnologies.verity.collection.FlatFileGateway"
      rootPath="C:\some\path\to\the\files"
      numRecordsPerSet="100" 
      keyFieldName="[ name of key field ]" >

   <Files>
     <File name="*.txt"
            delimiter="[delimChar] or tab|cr|lf|crlf"
            valueDelimiter=","
            comment="#"
            skipLines="1"
            VdkVgwKey="[ field that has key ]"  >

       <!-- Filter that is applied to each line of the text file -->
       <LineFilter class="[ class of com.raritantechnologies.utils.filter.IStringFilter" ]" >

       </LineFilter>

       <!-- Filter that is applied to each field value -->
       <FieldFilter class="[ class of com.raritantechnologies.utils.filter.IStringFilter" ]" >

       </FieldFilter>

       <Columns>
         <Column fieldID="firstFieldID" />
         <Column fieldID="secondFieldID" />
         <Column />  <!-- Ignore this one -->
         <Column fieldID="thirdFieldID" />
       </Columns>
     </File>

     <File name="DataFile*.csv" delimiter=":" comment="rem" skiplines="0" >
        <Columns>
          <Column ID|fieldID="secondFieldID" />
          <Column ID|fieldID="thirdFieldID" />
          <Column ID|fieldID="firstFieldID" />
        </Columns>
     </File>

     <File name="DataFile*.csv" delimiter=":" comment="rem" skiplines="0" >
        <Columns>
          <Column>secondFieldID
          <Column>thirdFieldID
          <Column>firstFieldID
        </Columns>
     </File>

       <!-- File definition that uses 1st line of file as column definition -->
     <File name="*.txt"
            delimiter="[delimChar] or tab|cr|lf|crlf"
            valueDelimiter=","
            comment="#"
            skipLines="1"
            firstLineDefinesColumns="true|false (default)"/> 

   </Files>

 </CollectionGateway>
 

Developed by Raritan Technologies .

Author:

Ted Sullivan

Nested Class Summary

class

FlatFileGateway.FileProcessor

Constructor Summary

FlatFileGateway()

FlatFileGateway(java.lang.String rootPath, boolean recurseSubdirectories)

Method Summary

void	addFileProcessor(java.lang.String name, java.lang.String delimiter, java.lang.String comment, int skipLines, java.lang.String[] columns)
java.lang.String	getConfigurationXML()
protected FlatFileGateway.FileProcessor	getFileProcessor(java.io.File file)
IGatewayListener	getGatewayListener()
java.lang.String	getKeyField()
int	getNumRecordsPerSet()
boolean	getRecurseSubdirectories()
java.lang.String	getRootDirectory()
void	initialize(org.w3c.dom.Element gatewayElem, ISearchFieldMap sfMap) Initialize the ICollectionGateway from a Configuration XML element.
boolean	isPaused()
void	pause()
protected void	processFile(java.io.File aFile)
void	resume()
void	run()
void	run(ILoginInfo userInfo) Need to have loginInfo for sources which require login/pw access.
void	setDataAvailable(IResultSet rs)
void	setDataComplete()
void	setGatewayListener(IGatewayListener listener) Sets the IGatewayListener interface.
void	setInputParameters(OrderedMap inputParams) set input parameters...
void	setKeyField(java.lang.String keyField)
void	setNumRecordsPerSet(int numRecordsPerSet)
void	setRecurseSubdirectories(boolean recurseSubdirectories)
void	setRootDirectory(java.lang.String rootDirectory)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlatFileGateway

public FlatFileGateway()

FlatFileGateway

public FlatFileGateway(java.lang.String rootPath,
                       boolean recurseSubdirectories)

Method Detail

initialize

public void initialize(org.w3c.dom.Element gatewayElem,
                       ISearchFieldMap sfMap)

Description copied from interface: ICollectionGateway

Initialize the ICollectionGateway from a Configuration XML element.

Specified by:

initialize in interface ICollectionGateway

setInputParameters

public void setInputParameters(OrderedMap inputParams)

Description copied from interface: ICollectionGateway

set input parameters...

Specified by:

setInputParameters in interface ICollectionGateway

run

public void run()

Specified by:

run in interface ICollectionGateway

processFile

protected void processFile(java.io.File aFile)

getFileProcessor

protected FlatFileGateway.FileProcessor getFileProcessor(java.io.File file)

run

public void run(ILoginInfo userInfo)

Need to have loginInfo for sources which require login/pw access.

Specified by:

run in interface ICollectionGateway

setGatewayListener

public void setGatewayListener(IGatewayListener listener)

Description copied from interface: ICollectionGateway

Sets the IGatewayListener interface.

Specified by:

setGatewayListener in interface ICollectionGateway

setDataAvailable

public void setDataAvailable(IResultSet rs)

setDataComplete

public void setDataComplete()

pause

public void pause()

Specified by:

pause in interface ICollectionGateway

resume

public void resume()

Specified by:

resume in interface ICollectionGateway

isPaused

public boolean isPaused()

Specified by:

isPaused in interface ICollectionGateway

getGatewayListener

public IGatewayListener getGatewayListener()

addFileProcessor

public void addFileProcessor(java.lang.String name,
                             java.lang.String delimiter,
                             java.lang.String comment,
                             int skipLines,
                             java.lang.String[] columns)

setRootDirectory

public void setRootDirectory(java.lang.String rootDirectory)

getRootDirectory

public java.lang.String getRootDirectory()

setRecurseSubdirectories

public void setRecurseSubdirectories(boolean recurseSubdirectories)

getRecurseSubdirectories

public boolean getRecurseSubdirectories()

setNumRecordsPerSet

public void setNumRecordsPerSet(int numRecordsPerSet)

getNumRecordsPerSet

public int getNumRecordsPerSet()

setKeyField

public void setKeyField(java.lang.String keyField)

getKeyField

public java.lang.String getKeyField()

getConfigurationXML

public java.lang.String getConfigurationXML()

Specified by:

getConfigurationXML in interface ICollectionGateway