Re: Commit after every document - alternate approach

2016-03-04 Thread Shawn Heisey
On 3/3/2016 11:36 PM, sangs8788 wrote: > When a commit fails, the document doesnt get cleared out from MQ and there is > a task which runs in a background to republish the files to SOLR. If we do a > batch commit we will not know we will end up redoing the same batch commit > again. We currenlty

Re: Commit after every document - alternate approach

2016-03-04 Thread Emir Arnautovic
.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html Sent from the Solr - User mailing list archive at Nabble.com. -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: Commit after every document - alternate approach

2016-03-03 Thread sangs8788
this message in context: http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261587.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread Walter Underwood
> -- > View this message in context: > http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread Walter Underwood
commit failure the > document can be resent from MQ. > > Thanks > Sangeetha > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread sangs8788
in context: http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261576.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-03 Thread sangs8788
failure the document can be resent from MQ. Thanks Sangeetha -- View this message in context: http://lucene.472066.n3.nabble.com/Commit-after-every-document-alternate-approach-tp4260946p4261575.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Commit after every document - alternate approach

2016-03-02 Thread Varun Thacker
Hi Sangeetha, Well I don't think you need to commit after every document add. You can rely on Solr's transaction log feature . If you are using SolrCloud it's mandatory to have a transaction log . So every documents get written to the tlog . Now say a node crashes even if documents were not

Re: Commit after every document - alternate approach

2016-03-02 Thread Emir Arnautovic
Hi Sangeetha, What is sure is that it is not going to work - with 200-300K doc/hour, there will be >50 commits/second, meaning there are <20ms time for doc+commit. You can do is let Solr handle commits and maybe use real time get to verify doc is in Solr or do some periodic sanity checks. Are

Commit after every document - alternate approach

2016-03-02 Thread sangeetha.subraman...@gtnexus.com
Hi All, I am trying to understand on how we can have commit issued to solr while indexing documents. Around 200K to 300K document/per hour with an avg size of 10 KB size each will be getting into SOLR . JAVA code fetches the document from MQ and streamlines it to SOLR. The problem is the