Here's supervisor.log with TRACE level: http://pastebin.com/nSX9Uuup
2015-11-18 15:30 GMT+02:00 Dmitry Sergeev <[email protected]>: > I think the problem is in Storm. Because previous version works fine. > Maybe I need to add some security configuration in Storm conf? > > thanks, Dmitry. > > 2015-11-18 10:17 GMT+02:00 Dmitry Sergeev <[email protected]>: > >> So where you think I need to search? In Zoo configurations? >> I've come to so many processes connected to Zoo because of many >> services(Kafka, Spark etc.) running on this machine. >> >> thanks! >> >> 2015-11-17 15:02 GMT+02:00 Nick R. Katsipoulakis <[email protected]>: >> >>> Hello again, >>> >>> The connection is not the problem then. It is clear that Nimbus connects >>> properly to ZooKeeper. As I mentioned earlier, the problem is with the >>> acess control list you are providing for a given session. By the way, how >>> come so many processes are connected to your ZooKeeper and your nimbus is >>> not running? >>> >>> Nick >>> >>> On Tue, Nov 17, 2015 at 7:55 AM, Dmitry Sergeev <[email protected]> >>> wrote: >>> >>>> Hi Nick, >>>> thanks for response. >>>> 1. My ZooKeeper instance in running on localhost:5181, here is lsof >>>> -i:5181 output: >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >>>> >>>>> java 1267 user 32u IPv6 11008 0t0 TCP *:5181 (LISTEN) >>>>> java 1267 user 33u IPv6 12398 0t0 TCP >>>>> localhost:5181->localhost:41394 (ESTABLISHED) >>>>> java 1267 user 36u IPv6 12402 0t0 TCP >>>>> localhost:5181->localhost:41395 (ESTABLISHED) >>>>> java 1267 user 37u IPv6 12405 0t0 TCP >>>>> localhost:5181->localhost:41396 (ESTABLISHED) >>>>> java 1267 user 38u IPv6 12234 0t0 TCP >>>>> localhost:5181->localhost:41397 (ESTABLISHED) >>>>> java 1267 user 39u IPv6 12236 0t0 TCP >>>>> localhost:5181->localhost:41398 (ESTABLISHED) >>>>> java 1267 user 40u IPv6 12238 0t0 TCP >>>>> localhost:5181->localhost:41399 (ESTABLISHED) >>>>> java 1267 user 41u IPv6 12428 0t0 TCP >>>>> localhost:5181->localhost:41400 (ESTABLISHED) >>>>> java 1267 user 42u IPv6 12665 0t0 TCP >>>>> localhost:5181->localhost:41401 (ESTABLISHED) >>>>> java 1267 user 43u IPv6 82902 0t0 TCP >>>>> localhost:5181->localhost:41467 (ESTABLISHED) >>>>> java 1267 user 44u IPv6 13313 0t0 TCP >>>>> localhost:5181->localhost:41403 (ESTABLISHED) >>>>> java 1267 user 45u IPv6 98702 0t0 TCP >>>>> localhost:5181->localhost:41469 (ESTABLISHED) >>>>> java 1267 user 46u IPv6 32367 0t0 TCP >>>>> localhost:5181->localhost:41456 (ESTABLISHED) >>>>> java 1648 user 118u IPv6 12279 0t0 TCP >>>>> localhost:41401->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 127u IPv6 82901 0t0 TCP >>>>> localhost:41467->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 132u IPv6 12397 0t0 TCP >>>>> localhost:41394->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 137u IPv6 12232 0t0 TCP >>>>> localhost:41395->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 141u IPv6 12404 0t0 TCP >>>>> localhost:41396->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 145u IPv6 12233 0t0 TCP >>>>> localhost:41397->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 149u IPv6 12235 0t0 TCP >>>>> localhost:41398->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 154u IPv6 12237 0t0 TCP >>>>> localhost:41399->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 158u IPv6 12272 0t0 TCP >>>>> localhost:41400->localhost:5181 (ESTABLISHED) >>>>> java 1648 user 164u IPv6 12288 0t0 TCP >>>>> localhost:41403->localhost:5181 (ESTABLISHED) >>>>> java 3893 user 38u IPv6 31428 0t0 TCP >>>>> localhost:41456->localhost:5181 (ESTABLISHED) >>>>> java 15889 user 138u IPv6 98701 0t0 TCP >>>>> localhost:41469->localhost:5181 (ESTABLISHED) >>>> >>>> 2. nimbus logs: >>>> >>>>> 2015-11-17 11:28:49.676 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] >>>>> Starting >>>>> 2015-11-17 11:28:49.676 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating >>>>> client connection, connectString=localhost:5181/storm sessionTimeout=20000 >>>>> watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@5057ae47 >>>>> 2015-11-17 11:28:49.677 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>>>> connection to server localhost/127.0.0.1:5181. Will not attempt to >>>>> authenticate using SASL (unknown error) >>>>> 2015-11-17 11:28:49.677 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket >>>>> connection established to localhost/127.0.0.1:5181, initiating session >>>>> 2015-11-17 11:28:49.679 o.a.s.s.o.a.z.ClientCnxn [INFO] Session >>>>> establishment complete on server localhost/127.0.0.1:5181, sessionid >>>>> = 0x151153299cd000e, negotiated timeout = 20000 >>>>> 2015-11-17 11:28:49.680 o.a.s.s.o.a.c.f.s.ConnectionStateManager >>>>> [INFO] State change: CONNECTED >>>> >>>> Cleaning storm-local directory doesn't help, the problem persists >>>> >>>> >>>> 2015-11-17 14:27 GMT+02:00 Nick R. Katsipoulakis <[email protected] >>>> >: >>>> >>>>> Hello Dmitry, >>>>> >>>>> KeeperException has to do with ZooKeeper. Why don't you check the >>>>> following first: >>>>> >>>>> >>>>> 1. Check if your ZooKeeper instance is running properly and on the >>>>> machine:port you are expecting it to run. >>>>> 2. Check that your Nimbus attempts to connect to the right >>>>> ZooKeeper >>>>> >>>>> The ACL Exception you are getting might be because there is a mismatch >>>>> between your Nimbus's previous state and the current one stored on >>>>> ZooKeeper. If you verify that Steps 1 and 2 are correct and the problem >>>>> persists, I would consider re-initializing Nimbus (erasing the nimbus >>>>> directory in your storm-local directory) and restarting ZooKeeper. Keep in >>>>> mind that re-initialization might cause the loss of data. Hence, do the >>>>> previous only if you are ok with losing any data. >>>>> >>>>> Cheers, >>>>> Nick >>>>> >>>>> On Tue, Nov 17, 2015 at 6:36 AM, Dmitry Sergeev <[email protected]> >>>>> wrote: >>>>> >>>>>> First, I started Nimbus with >>>>>> ${storm.home}/bin/storm nimbus >>>>>> It starts successfully >>>>>> When I'm trying to start supervisor with this command: >>>>>> ${storm.home}/bin/storm supervisor >>>>>> It prints JVM options and exit. >>>>>> In supervisor.log I see next message: >>>>>> >>>>>>> 2015-11-17 11:28:56.111 b.s.d.supervisor [ERROR] Error on >>>>>>> initialization of server mk-supervisor >>>>>>> java.lang.RuntimeException: >>>>>>> org.apache.storm.shade.org.apache.zookeeper.KeeperException$InvalidACLException: >>>>>>> KeeperErrorCode = InvalidACL for >>>>>>> /supervisors/62a290be-610d-4177-9702-58479ebc80d2 >>>>>> >>>>>> I guess it is problem with security configuration. >>>>>> Storm 0.9.4 works fine. >>>>>> Can you help me? >>>>>> >>>>>> Thanks >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Nick R. Katsipoulakis, >>>>> Department of Computer Science >>>>> University of Pittsburgh >>>>> >>>> >>>> >>> >>> >>> -- >>> Nick R. Katsipoulakis, >>> Department of Computer Science >>> University of Pittsburgh >>> >> >> >
