Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-26 Thread Vincenzo D'Amore
Thanks Erick for your willingness and patience, if I understood well when autoCommit with openSearcher=true at first commit (soft or hard) all new documents will be automatically available for search. But when openSearcher=false, the commit will flush recent index changes to stable storage, but

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-26 Thread Erick Erickson
right, autoCommit (in solrconfig.xml) will 1 close the current Lucene segments and open a new one 2 close the tlog and start a new one. Those actions are independent of whether openSearcher=true or false. if (and only if) openSearcher=true, then the commits will be immediately visible to a query.

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-25 Thread Vincenzo D'Amore
Hi Erick, thanks for your support. Reading the post I realised that my scenario does not apply the autoCommit configuration, now we don't have autoCommit in our solrconfig.xml. We need docs are searchable only after the indexing process, and all the documents are committed only at end of index

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-25 Thread Erick Erickson
OK, assuming you're not doing any commits at all until the very end, then the tlog contains all the docs for the _entire_ run. The article really doesn't care whether the commits come from the solrconfig.xml or SolrJ client or curl. The tlog simply is not truncated until a hard commit happens, no

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-25 Thread Vincenzo D'Amore
Hi Erick, I have tried indexing code I have few times, this is the behaviour I have tried out: When an indexing process starts, even if one or more tlog file exists, a new tlog file is created and all the new documents are stored there. When indexing process ends and does an hard commit, older

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-25 Thread Erick Erickson
The design is that the latest successfully flushed tlog file is kept for peer sync in SolrCloud mode. When a replica comes up, there's a chance that it's not very many docs behind. So, if possible, some of the docs are taken from the leader's tlog and replayed to the follower that's just been

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-24 Thread Erick Erickson
Vincenzo: Here's perhaps more than you want to know about hard commits, soft commits and transaction logs: http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Best, Erick On Sun, May 24, 2015 at 12:04 AM, Vincenzo D'Amore v.dam...@gmail.com wrote:

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-24 Thread Vincenzo D'Amore
Thanks Shawn for your prompt support. Best regards, Vincenzo On Sun, May 24, 2015 at 6:45 AM, Shawn Heisey apa...@elyograg.org wrote: On 5/23/2015 9:41 PM, Vincenzo D'Amore wrote: Thanks Shawn, may be this is a silly question, but I looked around and didn't find an answer... Well,

SolrCloud 4.8 - Transaction log size over 1GB

2015-05-23 Thread Vincenzo D'Amore
Hi, looking at tlog size I see there are many collection that have keep more than 1GB of space. Tlog are growing and the code that adds new documents never does an hard commit. The question is must I fix the code that update the collections or can I do an hard commit externally using collection

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-23 Thread Shawn Heisey
On 5/23/2015 8:56 PM, Vincenzo D'Amore wrote: looking at tlog size I see there are many collection that have keep more than 1GB of space. Tlog are growing and the code that adds new documents never does an hard commit. The question is must I fix the code that update the collections or can I

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-23 Thread Vincenzo D'Amore
Thanks Shawn, may be this is a silly question, but I looked around and didn't find an answer... Well, could I update solrconfig.xml for the collection while the instances are running or should I restart the cluster/reload the cores? On Sun, May 24, 2015 at 5:07 AM, Shawn Heisey

Re: SolrCloud 4.8 - Transaction log size over 1GB

2015-05-23 Thread Shawn Heisey
On 5/23/2015 9:41 PM, Vincenzo D'Amore wrote: Thanks Shawn, may be this is a silly question, but I looked around and didn't find an answer... Well, could I update solrconfig.xml for the collection while the instances are running or should I restart the cluster/reload the cores? You can

Re: transaction log size

2014-04-02 Thread Gurfan
-in-sorlcloud/ How can we reduce Tlog size at its lowest, so that our system restart up time will less. Thanks, --Gurfan -- View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354p4128547.html Sent from the Solr - User mailing list archive

Re: transaction log size

2014-04-02 Thread Erick Erickson
/transaction-log-size-tp4128354p4128547.html Sent from the Solr - User mailing list archive at Nabble.com.

transaction log size

2014-04-01 Thread Gurfan
transactionLog/master/* transactionLog/slave/* May you please give us some pointer so that we can control on Transaction log(Tlog) generation. Thanks, --Gurfan -- View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354.html Sent from the Solr - User mailing

Re: transaction log size

2014-04-01 Thread Shawn Heisey
the small size of transaction log. Transaction log size is purely controlled by hard commits (autoCommit), soft commits have no influence at all. To test the aforesaid statement we executed some Run: Document Size: ~2KB. 1st Run: AutoCommit: 30 Sec autoSoftCommit: 20 Sec openSearcher