2019-04-01 12:25:47 UTC - Sébastien de Melo: Hello @Sijie Guo, ok thank you! We have added the line extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent to bookkeeper.conf before starting the bookie. The option seems to be taken into account but the error still shows up. Could you please give us a hint about what could be missing? +1 : Laurent Chriqui ---- 2019-04-01 13:17:19 UTC - Matti-Pekka Laaksonen: I'm setting up a new Pulsar cluster in a new environment, I am actually having this same problem right now. The client connections to the proxy time out, but I'm able to connect directly to the broker. It seems that I'm unable to start the Pulsar proxy for some reason. I only see the error message "[sun.misc.Launcher$AppClassLoader@4d405ef7] error Uncaught exception in thread main: Failed to start HTTP server on ports [8080]" in the logs, nothing else. This directs me to <https://github.com/apache/pulsar/blob/branch-2.3/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/WebServer.java#L216>, but it doesn't offer much help ---- 2019-04-01 13:19:26 UTC - Matti-Pekka Laaksonen: The hosts are Azure VMs, and I have no DNS names for any of my servers. Zookeeper servers are configured as a string of ip-address:port -entries and the client service url is also in the style of <protocol://ip:port> ---- 2019-04-01 15:44:19 UTC - Ryan Samo: @Matteo Merli I have attempted to create readers, 1 per partition, and connect in just fine. The problem is I see them subscribe as exclusive and the backlog builds but they never read any messages. They just sit there connected and idle. I am starting with MessageId.Latest. Any idea why this would occur? ---- 2019-04-01 16:04:43 UTC - Matteo Merli: Not really, can you paste the topic stats ?
`pulsar-admin topics stats $TOPIC` and `pulsar-admin topics stats-internal $TOPIC` ---- 2019-04-01 16:12:28 UTC - Ryan Samo: Sure, one second ---- 2019-04-01 16:16:03 UTC - Devin G. Bost: The Pulsar Admin CLI function to terminate a topic... What does it do? It doesn't seem to delete the topic, and my understanding is that a consumer will immediately re-create the topic anyway. ---- 2019-04-01 16:18:19 UTC - Matteo Merli: Terninate operation will “seal” a topic. No more messages will be allowed. Consumers will receive special error code when they reach the end of it +1 : Devin G. Bost ---- 2019-04-01 16:24:00 UTC - Ryan Samo: I think I figured it out, it’s a big in my code forcing all the readers to look at 1 partition, not all ---- 2019-04-01 16:24:24 UTC - Ryan Samo: Those commands you gave helped me figure it out, thanks! ---- 2019-04-01 16:24:55 UTC - Matteo Merli: :+1: ---- 2019-04-01 16:36:20 UTC - Ryan Samo: If you have multiple partitions to read from do you recommend new client connections for each with 1 reader per or 1 connection with multiple readers? ---- 2019-04-01 16:36:40 UTC - Matteo Merli: 1 client and multiple readers ---- 2019-04-01 16:37:07 UTC - Ryan Samo: Ok thanks ---- 2019-04-01 16:51:38 UTC - Sanjeev Kulkarni: @bhagesharora does the command work without specifying any namespace/tenant(essentially launching the source in public/default) ---- 2019-04-01 16:53:51 UTC - Devin G. Bost: We have a namespace where we have deleted all of the sinks, sources, and functions; but we still have a topic with a subscription, and that subscription still has an active consumer, so we're unable to delete the subscription. Is there a way to force deletion of the subscription? ---- 2019-04-01 16:54:46 UTC - Matteo Merli: You can force the deletion of the topic, though when the consumer reconnects they’ll be recreating the topic immediately ---- 2019-04-01 16:57:07 UTC - Devin G. Bost: Would that create a race condition if we tried deleting the namespace (after forcing deletion of the topic) before the consumer tries recreating the topic? ---- 2019-04-01 16:58:47 UTC - Matteo Merli: Correct. If you have authz enabled, you could just remove permissions to consumers +1 : Devin G. Bost ---- 2019-04-01 16:58:56 UTC - Matteo Merli: and then delete ---- 2019-04-01 17:08:45 UTC - Devin G. Bost: @Matteo Merli Which method on the Pulsar Admin CLI would we use to do that? ---- 2019-04-01 17:10:23 UTC - Matteo Merli: To remove permissions? `pulsar-admin namespaces revoke-permission my-tenant/my-namespace --role my-consumer-principal` ---- 2019-04-01 17:10:31 UTC - Devin G. Bost: Thanks! ---- 2019-04-01 17:18:22 UTC - Fredrick P Eisele: Did you have any luck with your upgrade? ---- 2019-04-01 17:21:14 UTC - Devin G. Bost: @Matteo Merli After running that command, we still were unable to delete the topic. ---- 2019-04-01 17:24:51 UTC - Matteo Merli: pulsar-admin topics delete $TOPIC --force ---- 2019-04-01 17:34:46 UTC - Devin G. Bost: Thanks. That worked. However, it appears that the topic is still getting recreated. ---- 2019-04-01 17:35:06 UTC - Matteo Merli: Is authorization turned on? ---- 2019-04-01 17:35:13 UTC - Devin G. Bost: Yes. ---- 2019-04-01 17:35:33 UTC - Matteo Merli: If you remove permission from producers and consumers, they won’t be able to re-create the topic ---- 2019-04-01 18:40:05 UTC - Ryan Samo: Ok so I have tried only connecting 1 reader to partition 0 but still no luck. It just creates a backlog but no reads. If I take that same reader and point it to a non-partitioned topic it works fine ---- 2019-04-01 18:41:05 UTC - Ryan Samo: Maybe the way earliest and latest position is not quite right on partitions? Hard to tell ---- 2019-04-01 18:41:30 UTC - Matteo Merli: There shouldn’t be any difference from the 2 scenarios ---- 2019-04-01 18:42:16 UTC - Ryan Samo: Yeah that’s the strange part, using a basic reader works fine on a normal topic. Just does nothing on a partition ---- 2019-04-01 18:47:21 UTC - Ryan Samo: Does the MessageId.latest and earliest work when you target a single partition? ---- 2019-04-01 18:49:11 UTC - Matteo Merli: Yes, they’re supposed to ---- 2019-04-01 18:49:49 UTC - Matteo Merli: I’m trying out right now. Saw that it’s working fine with “latest” but seeign your same issue with earliest ---- 2019-04-01 18:50:17 UTC - Ryan Samo: Oh interesting ---- 2019-04-01 18:50:45 UTC - Matteo Merli: I don’t think it’s a partition vs non-partition thing.. but there’s something wrong ---- 2019-04-01 18:51:59 UTC - Ryan Samo: Ok, is there a way to query for the ledger I’d of a partition from the client? Think I could set my message Id via a loop, targeting the proper ledgers accordingly ---- 2019-04-01 18:52:15 UTC - Ryan Samo: ID ---- 2019-04-01 18:53:04 UTC - Matteo Merli: Yes, you can specify the message id directly when creating the reader ---- 2019-04-01 18:53:39 UTC - Ryan Samo: Yeah I’m thinking that might do the trick, as a workaround ---- 2019-04-01 18:53:59 UTC - Matteo Merli: looking into the issue though… ---- 2019-04-01 18:54:23 UTC - Ryan Samo: :+1: ---- 2019-04-01 18:56:02 UTC - Matteo Merli: btw: do you get these errors in broker log: ``` 11:55:14.231 [bookkeeper-ml-workers-OrderedExecutor-2-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/my-topic] Error opening ledger for reading at position 0:0 - org.apache.bookkeeper.mledger.ManagedLedgerException: Unknown exception 11:55:14.231 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [public/default/persistent/my-topic][null] read failed from ledger at position:0:0 : Unknown exception ``` ? ---- 2019-04-01 18:56:39 UTC - Ryan Samo: Checking ---- 2019-04-01 18:58:30 UTC - Matteo Merli: actually. in my cases it was because of tiered-storage and config changes… (eg: data was offloaded but I changed my config so it’s not able to fetch the data) ---- 2019-04-01 18:58:30 UTC - Matteo Merli: .. so, probably not same issue as yours ---- 2019-04-01 19:00:22 UTC - Ryan Samo: Nope not seeing that on my side. Using regular storage ---- 2019-04-01 19:02:15 UTC - Matteo Merli: Anyway, now I can see your issue on partition reading. ---- 2019-04-01 19:03:01 UTC - Ryan Samo: Yeah I see that earliest is -1:-1:-1, but it can’t pick that up ---- 2019-04-01 19:04:06 UTC - Ryan Samo: Latest actually doesn’t work for me either. Using 18 partitions, it resolves to numbers that seem to be larger than the ledger. Thanks for looking into it ---- 2019-04-01 19:40:59 UTC - Matteo Merli: @Ryan Samo just created PR with fix: <https://github.com/apache/pulsar/pull/3960> ---- 2019-04-01 19:41:49 UTC - Ryan Samo: :+1: Thanks @Matteo Merli for the support! ---- 2019-04-01 20:41:37 UTC - Devin G. Bost: Perhaps I should be asking if there are any guidelines for the deployment part of CI/CD for Pulsar at-scale. Are there any recommendations for how to gracefully roll-out Pulsar deployments? ---- 2019-04-01 20:41:53 UTC - Devin G. Bost: I didn't see much on the website. ---- 2019-04-01 21:03:32 UTC - Devin G. Bost: Where in the source code do deletes actually happen internally? I see that a request is constructed, for example, with the `deleteFunction` in: > org/apache/pulsar/client/admin/internal/FunctionsImpl.java:183 but it's not clear where the request is processed. (I'm investigating what would be involved to improve the delete functionality in the source code.) ---- 2019-04-01 21:06:48 UTC - Devin G. Bost: I'm guessing that a request hits Zookeeper somewhere. ---- 2019-04-01 21:36:29 UTC - Ali Ahmed: @Matti-Pekka Laaksonen @bhagesharora what’s the jvm version being used ? ---- 2019-04-01 21:38:41 UTC - Ali Ahmed: @Devin G. Bost there are no specific guidelines you can always use geo replication to replicate traffic to a test cluster upgrade or deploy functions in a sandbox there and then roll out to production ---- 2019-04-02 05:22:42 UTC - Matti-Pekka Laaksonen: @Ali Ahmed The Java version is "OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)", so the latest openjdk-8-jre-headless package installs ---- 2019-04-02 06:34:41 UTC - bhagesharora: @Ali Ahmed openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) ---- 2019-04-02 06:39:29 UTC - bhagesharora: @Sanjeev Kulkarni root@pulsar-setup-2-3-0-n1-standalone:/home/bhagesharora93/apache-pulsar-2.3.0# ./bin/pulsar-admin source --name generator --destinationTopicName generator_test --source-type data-generator Unknown option: --name Exception in thread "main" com.beust.jcommander.ParameterException: Asking description for unknown command: null at com.beust.jcommander.JCommander.getCommandDescription(JCommander.java:1003) at com.beust.jcommander.JCommander.usage(JCommander.java:988) at com.beust.jcommander.JCommander.usage(JCommander.java:980) at com.beust.jcommander.JCommander.usage(JCommander.java:972) at org.apache.pulsar.admin.cli.CmdBase.run(CmdBase.java:49) at org.apache.pulsar.admin.cli.PulsarAdminTool.run(PulsarAdminTool.java:185) at org.apache.pulsar.admin.cli.PulsarAdminTool.main(PulsarAdminTool.java:227) ---- 2019-04-02 06:59:25 UTC - bhagesharora: Hello everyone, I have installed pulsar in my GCP ubuntu VM, and able to start stanadlone cluster using ./bin/pulsar standalone. now I am trying to implement pulsar SQL and following Pulsar SQL Getting Started from apache document Ref. URL - <https://pulsar.apache.org/docs/en/sql-getting-started/> I followed all the steps(skip Data generator command step I am doing in a programatically way), whcih is mention in the same document. In last section there is one program Test.java, which I have create and able to RUN. after that I am executing query on "public/default" schema. presto> show tables in pulsar."public/default"; Query 20190402_055635_00013_fh96w failed: Failed to get tables/topics in public/default: Read timed out It's giving some Read timed out error. What is the reason for Read time out Error ?? see the attached screen shot, which includes java program, and presto shell. ---- 2019-04-02 07:00:07 UTC - bhagesharora: ---- 2019-04-02 07:00:26 UTC - bhagesharora: ---- 2019-04-02 07:00:39 UTC - bhagesharora: ----
