Hi All,
Is there a way to retrieve the documents being imported in a dataimport
request from a EventListener configured to run at onImportEnd?
I need to get the set of values of the field:content of all the documents
imported to perform an enhancement task. Is there a way to retrieve the
documents imported in dataimport from my EventListener?
My example data-config is as below;
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/test" user="usr1" password="pass1"
batchSize="1" />
<document name="stanboldata"
onImportEnd="com.solr.stanbol.processor.StanbolEventListener">
<entity name="stanbolrequest" query="SELECT * FROM documents">
<field column="id" name="id" />
<field column="content" name="content" />
<field column="title" name="title" />
</entity>
</document>
</dataConfig>
Currently what I do is as below;
1. The /dataimport requesthandler is configured with a custom
UpdateRequestProcessor which intercepts the documents being imported, gets
the value of the field I want and updates a static Map<long, String>
contents in my custom EventListener class with the documentID and the
content String.
2. At the end of the import process the StanbolEventListener is triggered
onImportEnd of the dataimport; in the onEvent(Context cntxt) method, the
contents Map is iterated and all the content field values are sent to an
external Server to be enhanced.
3. The documents with the IDs (keys of contents Map) are updated with the
enhanced fields and committed.
This mechanism works fine for a single dataimport process at a time.
But when there are concurrent dataimport requests, the system behaves
abruptly. I suspect the static Map<long,String>contents is updated abruptly
by concurrent update requests initiated by the dataimport process. To make
the contents Map thread safe, I used a ConcurrentHashMap implementation.
However I still get abrupt results in the update process.
What I'm looking for is an alternative to bypass data concurrency handling
in EventListeners.
I think this can be achieved if the whole dataimport process is executed as
a single transaction and at the onImportEnd EventListener, all the
documents imported are retrieved to get the content field of each document.
Is there a way to access the set of documents imported in the
onEvent(Context context) method of EventListener in Solr? Can I use the
Context object to access my documents?
Any suggestions
