Re: Node can be created and listed, but not updated or deleted
I created same node/path on a different cluster and delete works, so yes, it seems length or characters used in the path is not an issue. What else could it be? Corrupted data? On Wed, Sep 27, 2017 at 8:29 PM, Patrick Hunt <ph...@apache.org> wrote: > I spun up a 3.4.10 zk server (standalone) and tried to reproduce the issue > - the delete works fine for me. Perhaps the length is a red herring (or I > had some other diff I didn't notice) and it's something else? > > zk: localhost:2181(CONNECTED) 12] ls > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2 > [ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545] > [zk: localhost:2181(CONNECTED) 13] ls > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > [] > [zk: localhost:2181(CONNECTED) 14] get > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > foo > cZxid = 0x7 > ctime = Wed Sep 27 11:24:56 PDT 2017 > mZxid = 0x7 > mtime = Wed Sep 27 11:24:56 PDT 2017 > pZxid = 0x7 > cversion = 0 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 3 > numChildren = 0 > [zk: localhost:2181(CONNECTED) 15] delete > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > [zk: localhost:2181(CONNECTED) 16] ls > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > Node does not exist: > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > [zk: localhost:2181(CONNECTED) 17] get > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > Node does not exist: > /dd/r-/mmm/ttt/ww. > c.h-hhh$qqq.qqqq-qqQqq2/ > ww-dd-k-0_ww-dd-k_ > nnn_nnn_mmm_m$887960620903477545 > > On Wed, Sep 27, 2017 at 3:09 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > In ZK shell: > > > > ls > > /dd/r-/mmm/ttt/ww. > > c.h-hhh$qqq.-qqQqq2 > > [ww-dd-k-0_ww-dd-k_ > > nnn_nnn_mmm_m$887960620903477545] > > > > get > > /dd/r-/mmm/ttt/ww. > > c.h-hhh$qqq.-qqQqq2/ > > ww-dd-k-0_ww-dd-k_ > > nnn_nnn_mmm_m$887960620903477545 > > _ > > cZxid = 0x432e7 > > ctime = Fri Sep 22 14:51:22 CEST 2017 > > mZxid = 0x432e7 > > mtime = Fri Sep 22 14:51:22 CEST 2017 > > pZxid = 0x432e7 > > cversion = 0 > > dataVersion = 0 > > aclVersion = 0 > > ephemeralOwner = 0x0 > > dataLength = 13 > > numChildren = 0 > > > > delete > > /dd/r-/mmm/ttt/ww. > > c.h-hhh$qqq.-qqQqq2/ > > ww-dd-k-0_ww-dd-k_ > > nnn_nnn_mmm_m$887960620903477545 > > Node does not exist: > > /dd/r-/mmm/ttt/ww. > > c.h-hhh$qqq.-qqQqq2/ > > ww-dd-k-0_ww-dd-k_ > > nnn_nnn_mmm_m$887960620903477545 > > > > > > From one of ZK server logs: > > 2017-09-27 11:07:19,654 [myid:3] - INFO [ProcessThread(sid:3 > > cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException > when > > processing sessionid:0x15ec269280a000a type:delete cxid:0x73e > > zxid:0xc0017 txntype:-1 reqpath:n/a Error > > Path:/dd/r-/mmm/ttt/ww.c.h- > > hhh$qqq.-qqQqq2/ww-dd- > > k-0_ww-dd-k_nnn_nnn_mmm_ > > m$887960620903477545 > > Error:KeeperEr
Re: Node can be created and listed, but not updated or deleted
In ZK shell: ls /dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2 [ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545] get /dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545 _ cZxid = 0x432e7 ctime = Fri Sep 22 14:51:22 CEST 2017 mZxid = 0x432e7 mtime = Fri Sep 22 14:51:22 CEST 2017 pZxid = 0x432e7 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 13 numChildren = 0 delete /dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545 Node does not exist: /dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545 >From one of ZK server logs: 2017-09-27 11:07:19,654 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x15ec269280a000a type:delete cxid:0x73e zxid:0xc0017 txntype:-1 reqpath:n/a Error Path:/dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545 Error:KeeperErrorCode = NoNode for /dd/r-/mmm/ttt/ww.c.h-hhh$qqq.-qqQqq2/ww-dd-k-0_ww-dd-k_nnn_nnn_mmm_m$887960620903477545 On Mon, Sep 25, 2017 at 4:36 PM, Jordan Zimmerman < jor...@jordanzimmerman.com> wrote: > There's no internal limit on path lengths. Can you post a sample showing > the problem? > > -Jordan > > > On Sep 25, 2017, at 3:53 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > > Hello Apache ZooKeeper community, > > > > While on 3.4.10 I just experienced odd issue with a node that has > > relatively lengthy path. It was created and had data stored. In the > shell I > > could list it and retrieve the data. Problem is that the attempt from the > > app to update the data as well as attempt from the shell to delete the > node > > would fail with NoNode. > > > > Path length (with separators) is 190 characters, and longest path > segment / > > node name is 95 characters long. > > > > After shortening some of the prefixes for few path segments, keeping > > everything else same, it started working properly. > > > > Is there a limit on max path length and max path segment / node name > length? > > If so, why is creation passing and update/deletion fails? Is it a known > > issue? > > > > Kind regards, > > Stevo Slavic. > >
Node can be created and listed, but not updated or deleted
Hello Apache ZooKeeper community, While on 3.4.10 I just experienced odd issue with a node that has relatively lengthy path. It was created and had data stored. In the shell I could list it and retrieve the data. Problem is that the attempt from the app to update the data as well as attempt from the shell to delete the node would fail with NoNode. Path length (with separators) is 190 characters, and longest path segment / node name is 95 characters long. After shortening some of the prefixes for few path segments, keeping everything else same, it started working properly. Is there a limit on max path length and max path segment / node name length? If so, why is creation passing and update/deletion fails? Is it a known issue? Kind regards, Stevo Slavic.
Re: Multiple credentials associated with same principal?
Is this natively supported by ZooKeeper or does this require some customization? On Thu, Feb 2, 2017 at 7:06 PM, Martin Gainty <mgai...@hotmail.com> wrote: > > Stevo- > -- > *From:* Stevo Slavić <ssla...@gmail.com> > *Sent:* Thursday, February 2, 2017 7:38 AM > *To:* user@zookeeper.apache.org > *Subject:* Re: Multiple credentials associated with same principal? > > Alternatively, is it possible to set ACL that would grant given permissions > to any successfully authenticated user? > > MG>acl group should match subject=GroupOfPeople > > MG>ZooKeeperServer: > MG>best to configure each ACL on server for Group where > MG>ServerGroupACL=JAASSubject > MG>e,g. GroupACL=JAASSubject=EmergencyRoomNurses > > MG>ZooKeeperClient: > MG>since there can be multiple principals per subject > MG>each Principal would represent an individual within the group > MG>e.g. JAASPrincipal=NurseCratchett > > > On Wed, Feb 1, 2017 at 1:16 PM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Hello Apache ZooKeeper community, > > > > Is it valid in JAAS config file to associate more than one password per > > user, and if so, will ZooKeeper server authenticate user correctly if > > provided password matches any of the configured ones? > > > > Kind regards, > > Stevo Slavic. > > >
Re: Multiple credentials associated with same principal?
Hello Patrick, Thanks for reply! That feature would be appreciated, but it's not what I had in mind, it would not be sufficient. I need a way to change credentials without ZK client or cluster downtime, ideally with no ACL changes. Option of configuring two valid passwords for same user would help - then I could along with old password configure new one, roll ZK cluster with new settings, and then gradually roll out new credentials to all different clients, later remove old expired password. In one ZK client app, both zkclient and curator client libraries are being used to access two different ZK subtrees. I managed to configure each client to set ACLs appropriate for each subtree, but I couldn't find way yet to configure each client with different user, with sasl scheme. So had to fallback to single user. Still ACLs are different in the two subtrees. One subtree allows world to read, and creator all permissions. Other subtree just allows creator all permissions. It would help with credentials expiration if I could instead of (creator, all permissions) ACLs, set (any authenticated user, all permissions) ACL, while still keeping ACL for first subtree that world can read it. If it was possible, I'd expire not only password but replace it with new user, and no changes to ACLs would be needed. Thinking again, even if it was possible to set such ACL (any authenticated user, all permissions) in ZK nodes, it wouldn't help me now, since I cannot configure it to all clients managing nodes in subtree, some have ACLs that they set hardcoded, would have to fork large OSS project which is not really an option, and making ACL configurable in that OSS project would take some time. Kind regards, Stevo Slavic. On Thu, Feb 2, 2017 at 4:39 PM, Patrick Hunt <ph...@apache.org> wrote: > Hi Stevo, you might be talking about one of the following variants? (see > the jiras linked to from this jira) > https://issues.apache.org/jira/browse/ZOOKEEPER-1634 > > Patrick > > On Thu, Feb 2, 2017 at 4:38 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Alternatively, is it possible to set ACL that would grant given > permissions > > to any successfully authenticated user? > > > > On Wed, Feb 1, 2017 at 1:16 PM, Stevo Slavić <ssla...@gmail.com> wrote: > > > > > Hello Apache ZooKeeper community, > > > > > > Is it valid in JAAS config file to associate more than one password per > > > user, and if so, will ZooKeeper server authenticate user correctly if > > > provided password matches any of the configured ones? > > > > > > Kind regards, > > > Stevo Slavic. > > > > > >
Re: Multiple credentials associated with same principal?
Hello Patrick, Thanks for reply! That feature would be appreciated, but it's not what I had in mind, it would not be sufficient. I need a way to change credentials without ZK client or cluster downtime, ideally with no ACL changes. Option of configuring two valid passwords for same user would help - then I could along with old password configure new one, roll ZK cluster with new settings, and then gradually roll out new credentials to all different clients, later remove old expired password. In one ZK client app, both zkclient and curator client libraries are being used to access two different ZK subtrees. I managed to configure each client to set ACLs appropriate for each subtree, but I couldn't find way yet to configure each client with different user, with sasl scheme. So had to fallback to single user. Still ACLs are different in the two subtrees. One subtree allows world to read, and creator all permissions. Other subtree just allows creator all permissions. It would help with credentials expiration if I could instead of (creator, all permissions) ACLs, set (any authenticated user, all permissions) ACL, while still keeping ACL for first subtree that world can read it. If it was possible, I'd expire not only password but replace it with new user, and no changes to ACLs would be needed. Thinking again, even if it was possible to set such ACL (any authenticated user, all permissions) in ZK nodes, it wouldn't help me now, since I cannot configure it to all clients managing nodes in subtree, some have ACLs that they set hardcoded, would have to fork large OSS project which is not really an option, and making ACL configurable in that OSS project would take some time. Kind regards, Stevo Slavic. On Thu, Feb 2, 2017 at 4:39 PM, Patrick Hunt <ph...@apache.org> wrote: Hi Stevo, you might be talking about one of the following variants? (see the jiras linked to from this jira) https://issues.apache.org/jira/browse/ZOOKEEPER-1634 Patrick On Thu, Feb 2, 2017 at 4:38 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Alternatively, is it possible to set ACL that would grant given permissions > to any successfully authenticated user? > > On Wed, Feb 1, 2017 at 1:16 PM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Hello Apache ZooKeeper community, > > > > Is it valid in JAAS config file to associate more than one password per > > user, and if so, will ZooKeeper server authenticate user correctly if > > provided password matches any of the configured ones? > > > > Kind regards, > > Stevo Slavic. > > >
Re: Multiple credentials associated with same principal?
Alternatively, is it possible to set ACL that would grant given permissions to any successfully authenticated user? On Wed, Feb 1, 2017 at 1:16 PM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello Apache ZooKeeper community, > > Is it valid in JAAS config file to associate more than one password per > user, and if so, will ZooKeeper server authenticate user correctly if > provided password matches any of the configured ones? > > Kind regards, > Stevo Slavic. >
Multiple credentials associated with same principal?
Hello Apache ZooKeeper community, Is it valid in JAAS config file to associate more than one password per user, and if so, will ZooKeeper server authenticate user correctly if provided password matches any of the configured ones? Kind regards, Stevo Slavic.
Re: How to determine cause of InvalidACL
It's my bad - java.security.auth.login.config system property was not being properly set for the app. On Mon, Jul 4, 2016 at 3:30 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Thanks for the tips to both! > > Still struggling with this. > > If I got it right, issue it seems is in different response of ZooKeeper > server (3.4.6) for same request - when authentication is on or off, > ZooKeeper responds differently when trying to create a node that already > exists. > > Not using Kerberos, using DIGEST-MD5 as SASL mechanism. > > I'm trying to enable just ZooKeeper authentication, between Kafka brokers > and ZooKeeper, and an app and ZooKeeper. InvalidACL is not thrown and > everything works OK, without authentication enabled. > App has logic to explicitly create topic using AdminUtils.createTopic with > custom topic configuration. That fails now with InvalidACL. No custom ACL > is being used in the calls. > > What I can see from stack trace and Kafka source code, when trying to > write topic configuration data > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala#L436, > underlying ZkUtils call first gets NoNode for the topic config path > "/config/topics/foo" and then tries to create parent and that fails with > InvalidACL > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L424 > > Will check with Kafka community, why does the AdminUtils, when tying to > update topic config node data assumes that it got NoNode because parent > node doesn't exist - checked manually and /config/topics exists (likely > broker creates it on startup) and has ACLs set (world can read, and Kafka's > user can cdrwa). App trying to create the topic explicitly has same JAAS > config file as Kafka broker, and zkUtils passed to AdminUtils.createTopic > has isSecure set to true, so it shouldn't be permissions or authentication > issue. > > Without authentication, same ZkUtils call to create parent fails with > ZkNodeExistsException but that exception is swallowed by ZkClient (see > https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L295 > ) so ZkUtils never sees it. While it seems with ZooKeeper 3.4.6 server and > authentication on, client trying to create a node that already exists gets > InvalidACL, so topic creation fails - if true, is this inconsistency in > response a feature or a bug in ZooKeeper? > > Kind regards, > Stevo Slavic. > > On Sat, Jul 2, 2016 at 2:40 AM, Raúl Gutiérrez Segalés < > r...@itevenworks.net> wrote: > >> Another way to debug this is by using zk-dump [0] to dump requests/replies >> and see if the expected ACL is being send across the wire. >> >> -rgs >> >> [0] https://github.com/twitter/zktraffic >> On Jul 1, 2016 7:03 AM, "Flavio Junqueira" <f...@apache.org> wrote: >> >> > Hi Stevo, >> > >> > The logs indicate whether it has connected and authorized properly. Make >> > sure your keytab file is all good. >> > >> > One other suggestion is to try with the zkCli against the ensemble >> before >> > you try to run Kafka to make sure that your configuration is correct. >> > >> > -Flavio >> > >> > > On 01 Jul 2016, at 14:48, Stevo Slavić <ssla...@gmail.com> wrote: >> > > >> > > Hello Apache ZooKeeper community, >> > > >> > > As subject states, I'm having difficulties determining what could be >> > > causing InvalidACL to be thrown, any help would be appreciated. >> > > >> > > Kafka's ZkUtils I assume works OK and sends default ACLs with create >> path >> > > request >> > > >> > >> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L80 >> > > but server replies with InvalidACL error. Even at trace log level I >> > cannot >> > > see any useful information in the console. >> > > >> > > Kind regards, >> > > Stevo Slavic. >> > >> > >> > >
Log dir locked after ZK shutdown
Hello ZooKeeper community, In ZK (3.4.6) related tests, if I try to delete ZK data directory immediatelly after calling shutdown on server connection factory, only on Windows I get an IOException that a file could not be deleted. On Linux and Mac same code works well. I'm creating a temporary director using Java 7 Files.createTempDirectory(zookeeper-) resulting in directory similar to the following: C:\Users\Foo\AppData\Local\Temp\zookeeper-4563523978878660799\ Then ZooKeeper is configured to use this as data directory. To delete it I'm using FileUtils.deleteDirectory from commons-io (2.4) It walks the directory tree cleaning it up. It fails deleting log file C:\Users\Foo\AppData\Local\Temp\zookeeper-4563523978878660799\version-2\log.1 Is there something else I have to call to make sure ZooKeeperServer/ServerCnxnFactory is stopped and released all locks? If not, is this a known bug/feature? Kind regards, Stevo Slavic.
Re: Log dir locked after ZK shutdown
Found out through experiment - after shoutdown I also had to close ZK server database. I wonder why isn't this part of the server shutdown? ZooKeeperServer shutdown currently (ZK 3.4.6) calls only clear on database. Kind regards, Stevo Slavic. On Tue, Sep 2, 2014 at 5:19 PM, Stevo Slavić ssla...@gmail.com wrote: Hello ZooKeeper community, In ZK (3.4.6) related tests, if I try to delete ZK data directory immediatelly after calling shutdown on server connection factory, only on Windows I get an IOException that a file could not be deleted. On Linux and Mac same code works well. I'm creating a temporary director using Java 7 Files.createTempDirectory(zookeeper-) resulting in directory similar to the following: C:\Users\Foo\AppData\Local\Temp\zookeeper-4563523978878660799\ Then ZooKeeper is configured to use this as data directory. To delete it I'm using FileUtils.deleteDirectory from commons-io (2.4) It walks the directory tree cleaning it up. It fails deleting log file C:\Users\Foo\AppData\Local\Temp\zookeeper-4563523978878660799\version-2\log.1 Is there something else I have to call to make sure ZooKeeperServer/ServerCnxnFactory is stopped and released all locks? If not, is this a known bug/feature? Kind regards, Stevo Slavic.
ServerCnxnFactory startup order and registering ZooKeeperServer JMX MBean
Hello ZooKeeper community,
When running ZK (3.4.6) related tests in parallel in same JVM, multiple
servers although assigned a different local port clash when trying to
register JMX MBean for server (see [1]) - the two get assigned same MBean
name. Registering JMX MBean for ZooKeeperServer it seems cannot be
disabled, not even for tests.
Looking into ZooKeeperServerBean constructor, and
ZooKeeperServer.getClientPort one can see that the MBean name will clash
only when server connection factory is not bound to server, before startup
method of server connection factory gets called.
This is exactly what happens in startup methods of both
NIOServerCnxnFactory and NettyServerCnxnFactory, because of IMO wrong order
that the methods are being called. Instead of current [2] IMO it should be
[3] or at least [4].
Is my understanding wrong? If not, is this a (known) bug?
Kind regards,
Stevo Slavic
[1] example build output fragment with failing ZK server JMX MBean
registration
javax.management.InstanceAlreadyExistsException:
org.apache.ZooKeeperService:name0=StandaloneServer_port-1
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at
org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96)
at
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
at
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
[2]
@Override
public void startup(ZooKeeperServer zks) throws IOException,
InterruptedException {
start();
zks.startdata();
zks.startup();
setZooKeeperServer(zks);
}
[3]
@Override
public void startup(ZooKeeperServer zks) throws IOException,
InterruptedException {
start();
setZooKeeperServer(zks);
zks.startdata();
zks.startup();
}
[4]
@Override
public void startup(ZooKeeperServer zks) throws IOException,
InterruptedException {
start();
zks.startdata();
setZooKeeperServer(zks);
zks.startup();
}
