com.raritantechnologies.searchApp.dataCollection
Class WebCrawlerGateway

java.lang.Object
  com.raritantechnologies.searchApp.dataCollection.WebCrawlerGateway

All Implemented Interfaces:

ICollectionGateway, java.lang.Runnable

public class WebCrawlerGateway

extends java.lang.Object

implements ICollectionGateway

Collection Gateway that uses a Web Crawler or "spider" to collect document URLs for processing. Can incorporate a configurable HTMLScraper to extract metadata from HTML pages and/or an HTMLFilter to pre-process HTML pages.

XML Configuration Template:

Developed by Raritan Technologies .

Author:

Ted Sullivan

Constructor Summary

WebCrawlerGateway()

Method Summary

java.lang.String	getConfigurationXML()
void	initialize(org.w3c.dom.Element gatewayElem, ISearchFieldMap sfMap) Initialize the ICollectionGateway from a Configuration XML element.
boolean	isPaused()
void	pause()
void	resume()
void	run()
void	run(ILoginInfo userInfo) Need to have loginInfo for sources which require login/pw access.
void	setGatewayListener(IGatewayListener listener) Sets the IGatewayListener interface.
void	setInputParameters(OrderedMap inputParams) set input parameters...

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebCrawlerGateway

public WebCrawlerGateway()

Method Detail

run

public void run()

Specified by:

run in interface ICollectionGateway

run

public void run(ILoginInfo userInfo)

Description copied from interface: ICollectionGateway

Need to have loginInfo for sources which require login/pw access.

Specified by:

run in interface ICollectionGateway

setInputParameters

public void setInputParameters(OrderedMap inputParams)

Description copied from interface: ICollectionGateway

set input parameters...

Specified by:

setInputParameters in interface ICollectionGateway

setGatewayListener

public void setGatewayListener(IGatewayListener listener)

Description copied from interface: ICollectionGateway

Sets the IGatewayListener interface.

Specified by:

setGatewayListener in interface ICollectionGateway

initialize

public void initialize(org.w3c.dom.Element gatewayElem,
                       ISearchFieldMap sfMap)

Description copied from interface: ICollectionGateway

Initialize the ICollectionGateway from a Configuration XML element.

Specified by:

initialize in interface ICollectionGateway

pause

public void pause()

Specified by:

pause in interface ICollectionGateway

resume

public void resume()

Specified by:

resume in interface ICollectionGateway

isPaused

public boolean isPaused()

Specified by:

isPaused in interface ICollectionGateway

getConfigurationXML

public java.lang.String getConfigurationXML()

Specified by:

getConfigurationXML in interface ICollectionGateway