Re: ZK CLI script giving IOException doing upconfig
On 4/4/2018 1:51 PM, Shawn Heisey wrote: On 4/4/2018 12:13 PM, Doug Turnbull wrote: [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of session 0x10024db7e280006: *Len error 5327937* With that information, I think I can tell you what went wrong. I asked the ZK list a question related to this situation, to figure out how to get certain information from the ZK client so Solr could log warnings when files are too large. They told me that the ZK client should have indicated what went wrong, as the server log did. Would it be possible for you to provide entire logs (both the script and the ZK server) from an upload that fails this way, so I can review what changes we should make and whether we need to file a bug on ZK? I did come up with a preliminary patch for Solr that logs warnings about file sizes. It doesn't attempt to change or abort the operation, just log warnings. Any error messages would be the same as they are now. https://www.dropbox.com/s/uzi607evjt9gacc/upload-zk-size-warnings.patch?dl=0 Thanks, Shawn
Re: ZK CLI script giving IOException doing upconfig
On 4/5/2018 7:01 AM, Doug Turnbull wrote: My upconfig was also trying to upload the data dir (I had used this as a solr home in a standalone non cloud Solr), I'm missing *conf* here Oops. :) Uploading an entire core would be a problem! Glad you figured it out. I wonder too if there's anything that can be changed in zkcli to see if the confdir is a reasonable configuration directory? The process could look at what's being uploaded and log a warning that says it didn't see any files with names that looked like a config, but I wouldn't want it to make any decisions (like aborting the upload before it begins) based on that information. As an example of something we would want to allow: It is perfectly acceptable to make a change to an existing config in zookeeper by uploading a directory with only one file in it, say a synonym list. The upconfig action never *deletes* anything from the config in zookeeper. Thanks, Shawn
Re: ZK CLI script giving IOException doing upconfig
Shawn, that's the ticket... I see where I screwed up now. My upconfig was also trying to upload the data dir (I had used this as a solr home in a standalone non cloud Solr), I'm missing *conf* here -confdir solr_home/foo/ Changing to: -confdir solr_home/foo/conf works... I wonder too if there's anything that can be changed in zkcli to see if the confdir is a reasonable configuration directory? Thanks -Doug On Wed, Apr 4, 2018 at 3:51 PM Shawn Heisey wrote: > On 4/4/2018 12:13 PM, Doug Turnbull wrote: > > Thanks for the responses. Yeah I thought they were weird errors too... :) > > > > Below are the logs from zookeeper running in foreground after a > connection > > attempt. But this Exception looks suspicous to me: > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - > Exception > > causing close of session 0x10024db7e280006: *Len error 5327937* > > With that information, I think I can tell you what went wrong. > > It looks like one of the files you're trying to upload is 5 megabytes in > size. ZooKeeper doesn't allow anything bigger than about 1 megabyte by > default, because is not designed for handling large amounts of data. > > I think that the ZK uploading functionality probably needs to check the > size of what it is uploading against the max buffer setting and log a > useful error message. > > You can get this to work, but to do so will require setting a system > property on *all* ZK clients and servers. The clients will include Solr > itself and the zkcli script. The system property to set is > "jute.maxbuffer". Info can be found in ZK documentation. > > https://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html > > Thanks, > Shawn > > -- CTO, OpenSource Connections Author, Relevant Search http://o19s.com/doug
Re: ZK CLI script giving IOException doing upconfig
On 4/4/2018 12:13 PM, Doug Turnbull wrote: > Thanks for the responses. Yeah I thought they were weird errors too... :) > > Below are the logs from zookeeper running in foreground after a connection > attempt. But this Exception looks suspicous to me: > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception > causing close of session 0x10024db7e280006: *Len error 5327937* With that information, I think I can tell you what went wrong. It looks like one of the files you're trying to upload is 5 megabytes in size. ZooKeeper doesn't allow anything bigger than about 1 megabyte by default, because is not designed for handling large amounts of data. I think that the ZK uploading functionality probably needs to check the size of what it is uploading against the max buffer setting and log a useful error message. You can get this to work, but to do so will require setting a system property on *all* ZK clients and servers. The clients will include Solr itself and the zkcli script. The system property to set is "jute.maxbuffer". Info can be found in ZK documentation. https://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html Thanks, Shawn
Re: ZK CLI script giving IOException doing upconfig
Hi Doug, are you able to connect to Zookeeper thru Zookeeper zkCli.sh or does Zookeeper.out show anything useful. Thnx On Wed, Apr 4, 2018 at 2:13 PM, Doug Turnbull < dturnb...@opensourceconnections.com> wrote: > Thanks for the responses. Yeah I thought they were weird errors too... :) > > Below are the logs from zookeeper running in foreground after a connection > attempt. But this Exception looks suspicous to me: > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception > causing close of session 0x10024db7e280006: *Len error 5327937* > > Has anyone seen this before? The LenError seems to be a thread to google... > > 2018-04-04 14:06:01,210 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket > connection > from /127.0.0.1:55078 > 2018-04-04 14:06:01,218 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@938] - Client attempting to establish > new session at /127.0.0.1:55078 > 2018-04-04 14:06:01,219 [myid:] - INFO [SyncThread:0:ZooKeeperServer@683] > - Established session 0x10024db7e280006 with negotiated timeout 3 for > client /127.0.0.1:55078 > 2018-04-04 14:06:01,361 [myid:] - WARN [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of > session 0x10024db7e280006: Len error 5327937 > 2018-04-04 14:06:01,362 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for > client /127.0.0.1:55078 which had sessionid 0x10024db7e280006 > 2018-04-04 14:06:01,956 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket > connection > from /0:0:0:0:0:0:0:1:55079 > 2018-04-04 14:06:01,959 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@931] - Client attempting to renew > session 0x10024db7e280006 at /0:0:0:0:0:0:0:1:55079 > 2018-04-04 14:06:01,960 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@683] - Established session > 0x10024db7e280006 with negotiated timeout 3 for client > /0:0:0:0:0:0:0:1:55079 > 2018-04-04 14:06:03,223 [myid:] - WARN [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@376] - Unable to read additional data > from client sessionid 0x10024db7e280006, likely client has closed socket > 2018-04-04 14:06:03,223 [myid:] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for > client /0:0:0:0:0:0:0:1:55079 which had sessionid 0x10024db7e2800 > > On Wed, Apr 4, 2018 at 11:15 AM Shawn Heisey wrote: > > > On 4/4/2018 7:14 AM, Doug Turnbull wrote: > > > I've been struggling to do a basic upconfig both with embedded and > actual > > > Zookeeper in Solr 7.2.1 using the zkcli script on OSX. > > > > > > One variable, I recently upgraded to Java 9. I get slightly different > > > errors on Java 8 vs 9 > > > > > > > > > Java 9: > > > > > > doug@wiz$~/ws/foo(mas) $ > > > /Users/doug/bin/solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh > -zkhost > > > localhost:2181 -cmd upconfig -confdir solr_home/foo/ -confname foo_conf > > > WARN - 2018-04-04 09:05:28.194; > > > org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004 > for > > > server localhost/127.0.0.1:2181, unexpected error, closing socket > > > connection and attempting reconnect > > > java.io.IOException: Connection reset by peer > > > > > > > > > Java 8 gives the error > > > > > > java.io.IOException: Protocol wrong type for socket > > > > > > WARN - 2018-04-04 09:10:11.879; > > > org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002 > for > > > server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing > socket > > > connection and attempting reconnect > > > java.io.IOException: Protocol wrong type for socket > > > > I'm with Erick on this. These are REALLY weird errors. The stacktraces > > for the errors are entirely in ZooKeeper and Java code, not Solr code. > > The log for Java 9 does have an entry that mentions Solr classes, but > > that's a disconnect after the error, not part of the error. > > > > Are you getting any corresponding log messages in the ZK server log? > > > > The ZkCLI class is part of Solr, and does interface to ZK through Solr > > internals, but ultimately it's ZK doing the work. > > > > The ZK client that's in Solr 7.2.1 is version 3.4.10. > > > > Thanks, > > Shawn > > > > -- > CTO, OpenSource Connections > Author, Relevant Search > http://o19s.com/doug >
Re: ZK CLI script giving IOException doing upconfig
Thanks for the responses. Yeah I thought they were weird errors too... :) Below are the logs from zookeeper running in foreground after a connection attempt. But this Exception looks suspicous to me: [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of session 0x10024db7e280006: *Len error 5327937* Has anyone seen this before? The LenError seems to be a thread to google... 2018-04-04 14:06:01,210 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /127.0.0.1:55078 2018-04-04 14:06:01,218 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@938] - Client attempting to establish new session at /127.0.0.1:55078 2018-04-04 14:06:01,219 [myid:] - INFO [SyncThread:0:ZooKeeperServer@683] - Established session 0x10024db7e280006 with negotiated timeout 3 for client /127.0.0.1:55078 2018-04-04 14:06:01,361 [myid:] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of session 0x10024db7e280006: Len error 5327937 2018-04-04 14:06:01,362 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /127.0.0.1:55078 which had sessionid 0x10024db7e280006 2018-04-04 14:06:01,956 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /0:0:0:0:0:0:0:1:55079 2018-04-04 14:06:01,959 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@931] - Client attempting to renew session 0x10024db7e280006 at /0:0:0:0:0:0:0:1:55079 2018-04-04 14:06:01,960 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@683] - Established session 0x10024db7e280006 with negotiated timeout 3 for client /0:0:0:0:0:0:0:1:55079 2018-04-04 14:06:03,223 [myid:] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@376] - Unable to read additional data from client sessionid 0x10024db7e280006, likely client has closed socket 2018-04-04 14:06:03,223 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /0:0:0:0:0:0:0:1:55079 which had sessionid 0x10024db7e2800 On Wed, Apr 4, 2018 at 11:15 AM Shawn Heisey wrote: > On 4/4/2018 7:14 AM, Doug Turnbull wrote: > > I've been struggling to do a basic upconfig both with embedded and actual > > Zookeeper in Solr 7.2.1 using the zkcli script on OSX. > > > > One variable, I recently upgraded to Java 9. I get slightly different > > errors on Java 8 vs 9 > > > > > Java 9: > > > > doug@wiz$~/ws/foo(mas) $ > > /Users/doug/bin/solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh -zkhost > > localhost:2181 -cmd upconfig -confdir solr_home/foo/ -confname foo_conf > > WARN - 2018-04-04 09:05:28.194; > > org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004 for > > server localhost/127.0.0.1:2181, unexpected error, closing socket > > connection and attempting reconnect > > java.io.IOException: Connection reset by peer > > > > > Java 8 gives the error > > > > java.io.IOException: Protocol wrong type for socket > > > > WARN - 2018-04-04 09:10:11.879; > > org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002 for > > server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket > > connection and attempting reconnect > > java.io.IOException: Protocol wrong type for socket > > I'm with Erick on this. These are REALLY weird errors. The stacktraces > for the errors are entirely in ZooKeeper and Java code, not Solr code. > The log for Java 9 does have an entry that mentions Solr classes, but > that's a disconnect after the error, not part of the error. > > Are you getting any corresponding log messages in the ZK server log? > > The ZkCLI class is part of Solr, and does interface to ZK through Solr > internals, but ultimately it's ZK doing the work. > > The ZK client that's in Solr 7.2.1 is version 3.4.10. > > Thanks, > Shawn > > -- CTO, OpenSource Connections Author, Relevant Search http://o19s.com/doug
Re: ZK CLI script giving IOException doing upconfig
On 4/4/2018 7:14 AM, Doug Turnbull wrote: I've been struggling to do a basic upconfig both with embedded and actual Zookeeper in Solr 7.2.1 using the zkcli script on OSX. One variable, I recently upgraded to Java 9. I get slightly different errors on Java 8 vs 9 Java 9: doug@wiz$~/ws/foo(mas) $ /Users/doug/bin/solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd upconfig -confdir solr_home/foo/ -confname foo_conf WARN - 2018-04-04 09:05:28.194; org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer Java 8 gives the error java.io.IOException: Protocol wrong type for socket WARN - 2018-04-04 09:10:11.879; org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002 for server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Protocol wrong type for socket I'm with Erick on this. These are REALLY weird errors. The stacktraces for the errors are entirely in ZooKeeper and Java code, not Solr code. The log for Java 9 does have an entry that mentions Solr classes, but that's a disconnect after the error, not part of the error. Are you getting any corresponding log messages in the ZK server log? The ZkCLI class is part of Solr, and does interface to ZK through Solr internals, but ultimately it's ZK doing the work. The ZK client that's in Solr 7.2.1 is version 3.4.10. Thanks, Shawn
Re: ZK CLI script giving IOException doing upconfig
I haven't seen those errors before, so it's puzzling. Is there any chance there are conflicting _zookeeper_ jars somewhere in your classpath? This looks like a problem with ZK talking to itself. You may find it easier just to use the bin/solr script, we tried to put useful ZK commands there. Whether that would get around your error I don't know. The command you used _should_ work. "bin/solr zk -help" Best, Erick On Wed, Apr 4, 2018 at 6:14 AM, Doug Turnbull wrote: > I've been struggling to do a basic upconfig both with embedded and actual > Zookeeper in Solr 7.2.1 using the zkcli script on OSX. > > One variable, I recently upgraded to Java 9. I get slightly different > errors on Java 8 vs 9 > > This is probably me being dumb, but googling / searching Jira hasn't really > yielded anything fruitful. Perhaps my google fu is weak this morning. > > Thanks for any help > -Doug > > > > Java 9: > > doug@wiz$~/ws/foo(mas) $ > /Users/doug/bin/solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh -zkhost > localhost:2181 -cmd upconfig -confdir solr_home/foo/ -confname foo_conf > WARN - 2018-04-04 09:05:28.194; > org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004 for > server localhost/127.0.0.1:2181, unexpected error, closing socket > connection and attempting reconnect > java.io.IOException: Connection reset by peer > at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:192) > at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:382) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) > WARN - 2018-04-04 09:05:28.310; > org.apache.solr.common.cloud.ConnectionManager; Watcher > org.apache.solr.common.cloud.ConnectionManager@3eed2369 name: > ZooKeeperConnection Watcher:localhost:2181 got event WatchedEvent > state:Disconnected type:None path:null path: null type: None > > Java 8 gives the error > > java.io.IOException: Protocol wrong type for socket > > WARN - 2018-04-04 09:10:11.879; > org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002 for > server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket > connection and attempting reconnect > java.io.IOException: Protocol wrong type for socket > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) > > -- > CTO, OpenSource Connections > Author, Relevant Search > http://o19s.com/doug
ZK CLI script giving IOException doing upconfig
I've been struggling to do a basic upconfig both with embedded and actual Zookeeper in Solr 7.2.1 using the zkcli script on OSX. One variable, I recently upgraded to Java 9. I get slightly different errors on Java 8 vs 9 This is probably me being dumb, but googling / searching Jira hasn't really yielded anything fruitful. Perhaps my google fu is weak this morning. Thanks for any help -Doug Java 9: doug@wiz$~/ws/foo(mas) $ /Users/doug/bin/solr-7.2.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd upconfig -confdir solr_home/foo/ -confname foo_conf WARN - 2018-04-04 09:05:28.194; org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Connection reset by peer at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method) at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:192) at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:382) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) WARN - 2018-04-04 09:05:28.310; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@3eed2369 name: ZooKeeperConnection Watcher:localhost:2181 got event WatchedEvent state:Disconnected type:None path:null path: null type: None Java 8 gives the error java.io.IOException: Protocol wrong type for socket WARN - 2018-04-04 09:10:11.879; org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002 for server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Protocol wrong type for socket at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) -- CTO, OpenSource Connections Author, Relevant Search http://o19s.com/doug