2020-07-24 11:05:31 UTC - Walter: Failed to build bookie server in kubernetes environment
we have deployed pulsar using helm chart in openshfit k8s environment. when we scale up the deleted volume bookie we encounter with below issue. `2.284: Total time for which application threads were stopped: 0.0001663 seconds, Stopping threads took: 0.0001040 seconds` `10:28:55.441 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]` `10:28:55.441 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer` `2.404: Total time for which application threads were stopped: 0.0002793 seconds, Stopping threads took: 0.0001671 seconds` `2.405: Total time for which application threads were stopped: 0.0002462 seconds, Stopping threads took: 0.0001874 seconds` `2.406: Total time for which application threads were stopped: 0.0002902 seconds, Stopping threads took: 0.0001068 seconds` `2.413: Total time for which application threads were stopped: 0.0005099 seconds, Stopping threads took: 0.0003770 seconds` `2.413: Total time for which application threads were stopped: 0.0001357 seconds, Stopping threads took: 0.0000875 seconds` `2.415: Total time for which application threads were stopped: 0.0004179 seconds, Stopping threads took: 0.0001708 seconds` `2.417: Total time for which application threads were stopped: 0.0001661 seconds, Stopping threads took: 0.0000410 seconds` `2.418: Total time for which application threads were stopped: 0.0003534 seconds, Stopping threads took: 0.0002680 seconds` `2.419: Total time for which application threads were stopped: 0.0003882 seconds, Stopping threads took: 0.0003552 seconds` `2.419: Total time for which application threads were stopped: 0.0000419 seconds, Stopping threads took: 0.0000138 seconds` `2.420: Total time for which application threads were stopped: 0.0000399 seconds, Stopping threads took: 0.0000124 seconds` `10:28:55.461 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server` `org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:` `at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:468) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]` `2.425: Total time for which application threads were stopped: 0.0000542 seconds, Stopping threads took: 0.0000156 seconds` *Steps to reproduce the behavior :* 1. scale down the bookie pod. 2. delete persistent volume claim and persistent volume of deleted bookie pod. 3. scale up the bookie pod ---- 2020-07-24 13:12:24 UTC - Joshua Decosta: So i noticed when I try to produce a message with a custom AuthenticationProvider that the pulsar-client relies on the getAuthMethodName to run the correct AuthenticationProvider. I was originally just reusing the AuthenticationToken class. But this seems to be an issue with the pulsar-client. Is this true? ---- 2020-07-24 15:33:02 UTC - Chris: Is there a way for the pulsar websocket service to run in a proxy, or does it have to be a broker? ---- 2020-07-24 16:11:07 UTC - Chris: For others wondering, the websocket can run totally standalone using `bin/pulsar websocket` , but it looks like 2.6.0 doesn't pick up the cluster name from the config file correctly. 2.5.2 does however. ---- 2020-07-24 16:39:16 UTC - Addison Higham: @Walter did you decomission the bookie (<https://bookkeeper.apache.org/docs/latest/admin/decomission/>) or delete the cookie some other way? That is expected behavior. Via zookeeper, bookkeeper tracks cookies that are tied to a hostname. If you have the same hostname but don't have the cookie on disk, that is an error state. Decommissioning the bookie (or deleting the cookie via some other mechanism) is how you inform bookkkeeper that you do want to rejoin the node but with a new disk and that none of the old data is present ---- 2020-07-24 16:43:00 UTC - Addison Higham: @Chris can you provide some more details on that? Sounds like it may be a regression or an issue with docs, either way, if you have chance to file an issue that would be helpful ---- 2020-07-24 16:43:47 UTC - Chris: Yeah, I thought I remembered 2.5.0 had a similar issue with missing websocket cluster data ---- 2020-07-24 16:44:01 UTC - Chris: Perhaps it was fixed when using broker+websocket but not websocket standalone? ---- 2020-07-24 16:47:07 UTC - Addison Higham: @Arthur yeah, so ingresses don't support TCP really, but you can do it with nginx using this sort of "hack" <https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/> But you can use this <https://kubernetes.github.io/ingress-nginx/deploy/baremetal/> as well to get nginx to act the way service load balancers do! ---- 2020-07-24 16:50:58 UTC - Frank Kelly: Question on Proxy vs. Broker - I have my pulsar cluster inside the same Kubernetes namespace as my services. The pulsar cluster 1 proxy, 3 brokers. Should my services connect to the proxy vs. the broker (k8s service)? Thanks in advance. ---- 2020-07-24 17:26:46 UTC - Addison Higham: I know there is a way to handle that... but not immediately aware of what it is ---- 2020-07-24 17:27:58 UTC - Addison Higham: for service inside k8s, you probably just want to connect direct to the broker, as they will just discover the names of the rest of the brokers. Proxy is more useful for exposing pulsar externally ---- 2020-07-24 17:28:38 UTC - Frank Kelly: Great - Thank you! ---- 2020-07-24 17:44:17 UTC - Joshua Decosta: I think i figured out my issue and it has to do with the different protocols being used. I had to extend the AuthenticationToken class and override the auth method type to fix it. ---- 2020-07-24 18:00:27 UTC - Evgeny Sorokin: @Evgeny Sorokin has joined the channel ---- 2020-07-25 02:40:14 UTC - Shivam Arora: I have a similar doub. We have a cluster running on K8. When 2 nodes fails out of 3, auto recovery pod will create desired replica. What will happen when 2 failed node join back ? or if I add 2 new nodes ? Thanks in advance. ---- 2020-07-25 02:40:56 UTC - Shivam Arora: I have a similar doub. We have a cluster running on K8. When 2 nodes fails out of 3, auto recovery pod will create desired replica. What will happen when 2 failed node join back ? or if I add 2 new nodes ? Thanks in advance. ----
