Re: Zookeeper upconfig files to upload big config files

[Shawn Heisey]

On 2/17/2016 1:04 PM, Aswath Srinivasan (TMS) wrote:
> I’m tyring to upconfig my config files and my synonyms.txt file is
> about 2 MB. Whenever I try to do this, I get the following expection.
> It’s either a “broken pipe” expection or the following expection. Any
> advice for me to fix it?

The default size limit on a zookeeper node is about 1 megabyte.

Adjusting this limit is possible, but involves some invasive changes. 
You will need to add "-Djute.maxbuffer=NNN" to the command-line of
every single java invocation related to Solr and Zookeeper, including
the zkcli script, to allow a larger node size.

http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Unsafe+Options

The zookeeper project calls this an "unsafe" option because zookeeper is
not really designed to store large data.

I would also expect such a large list of synonyms to slow down analysis.

Thanks,
Shawn

Zookeeper upconfig files to upload big config files

[Aswath Srinivasan (TMS)]

Hi fellow Solr developers,

I'm tyring to upconfig my config files and my synonyms.txt file is about 2 MB. 
Whenever I try to do this, I get the following expection. It's either a "broken 
pipe" expection or the following expection. Any advice for me to fix it?

If I remove most of the synonym entries and keep my synonyms.txt file simple 
and short then upconfig works without any problem.

WARN  - 2016-02-17 11:48:55.514; org.apache.zookeeper.ClientCnxn$SendThread; 
Session 0x252e439e59f0017 for server zookeeperhost/zookeeperhost:2181, 
unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[abc@dc1-abc ~]$ ^C
[abc@dc1-abc ~]$ /t3/apps/solr-5.3.2/server/scripts/cloud-scripts/zkcli.sh 
-zkhost 
t3solr.test1.abc.com:2181,t3solr.test2.abc.com:2181,t3solr.test3.abc.com:2181 
-cmd upconfig -confname test_config_2 -confdir 
/t3/apps/solr-5.3.2/example/example-DIH/solr/TestEnvCore/conf
Exception in thread "main" java.io.IOException: Error uploading file 
/t3/apps/solr-5.3.2/example/example-DIH/solr/TestEnvCore/conf/synonyms.txt to 
zookeeper path /configs/test_config_2/synonyms.txt
at 
org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:68)
at 
org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:58)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:135)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
at java.nio.file.Files.walkFileTree(Files.java:2602)
at java.nio.file.Files.walkFileTree(Files.java:2635)
at 
org.apache.solr.common.cloud.ZkConfigManager.uploadToZK(ZkConfigManager.java:58)
at 
org.apache.solr.common.cloud.ZkConfigManager.uploadConfigDir(ZkConfigManager.java:120)
at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:220)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /configs/test_config_2/synonyms.txt
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
at 
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:362)
at 
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:359)
at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at 
org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:359)
at 
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:529)
at 
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:408)
at 
org.apache.solr.common.cloud.ZkConfigManager$1.visitFile(ZkConfigManager.java:66)
... 9 more
[abc@dc1-abc ~]$


[cid:image002.png@01D1697B.64FB26D0]
Thank you,
Aswath NS