Thanks. Temporarily got over the problem by specifying custom limits through jute.maxbuffer=<customlimit>
On Tue, Mar 18, 2014 at 9:45 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > Sorry guys I spoke too fast. I looked at the code again. No it doesn't > correlate with commits at all. I was mistaken. > > On Wed, Mar 19, 2014 at 10:06 AM, Chris W <chris1980....@gmail.com> wrote: > > Thanks, Shawn and Shalin > > > > How does the frequency of commit affect zookeeper? > > > > > > Thanks > > > > > > On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar < > > shalinman...@gmail.com> wrote: > > > >> SolrCloud will update Zookeeper on state changes (node goes to > >> recovery, comes back up etc) or for leader election and during > >> collection API commands. It doesn't correlate directly with indexing > >> but is correlated with how frequently you call commit. > >> > >> On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey <s...@elyograg.org> > wrote: > >> > On 3/18/2014 5:46 PM, Chris W wrote: > >> >> > >> >> I am running a 3 node zookeeper 3.4.5 Quorum. I am running into > issues > >> >> with Zookeeper transaction logs > >> >> > >> >> [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database > on > >> disk > >> >> java.io.IOException: Unreasonable length = 1048587 > >> >> at > >> >> > >> > org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100) > >> >> at > >> >> > org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233) > >> >> at > >> >> > >> >> > >> > org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602) > >> >> at > >> >> > >> >> > >> > org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157) > >> >> at > >> >> > org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223) > >> >> > >> >> To unblock temporarily, i deleted the most recent txn log. How do i > tell > >> >> zookeeper to not grow the transaction log beyond x MegaBytes? > >> >> > >> >> How often does the transaction log get updated? Does zk transactions > log > >> >> grow everytime we index data into a new collection? > >> > > >> > > >> > Zookeeper is a separate project at Apache. ZK file management is > >> discussed > >> > in the ZK documentation. > >> > > >> > > >> > http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance > >> > > >> > There is a bug filed on Zookeeper for the issue you are seeing, with a > >> > fairly simple workaround. It is fixed in the 3.4.6 version, which was > >> > released last week. I will see whether we can get ZK upgraded to > 3.4.6 > >> in > >> > the Solr 4.8.0 release. I don't think we want to risk doing that > >> upgrade in > >> > 4.7.1, but I could be wrong. > >> > > >> > https://issues.apache.org/jira/browse/ZOOKEEPER-1513 > >> > http://zookeeper.apache.org/releases.html > >> > > >> > I am actually not sure how often SolrCloud updates Zookeeper. It > happens > >> > whenever the collections API is called for sure, and it may happen > >> anytime > >> > you index data as well. > >> > > >> > Thanks, > >> > Shawn > >> > > >> > >> > >> > >> -- > >> Regards, > >> Shalin Shekhar Mangar. > >> > > > > > > > > -- > > Best > > -- > > C > > > > -- > Regards, > Shalin Shekhar Mangar. > -- Best -- C
