2019-10-15 10:19:29 UTC - Takahiro Hozumi: @Takahiro Hozumi has joined the channel ---- 2019-10-15 12:25:09 UTC - Chris Bartholomew: @Luke Lu Websocket auth definitely works with the bearer token in the Authorization header when running websocket proxy as a separate component. In that case, you have to configure token auth in websocket.conf. When its running with the broker, I believe it picks up the config from broker.conf. +1 : Luke Lu ---- 2019-10-15 13:17:35 UTC - Olivier Chicha: Hi all, For non-persistent topic I couldn't find how broker acknowledgement happens on *non-persistent* topic here is my scenario: A producer P sends a message to the Broker B on the topic T for which there are free different subscription S1, S2, S3 What I would like to know is: Is there a way to configure the Broker to acknowledge the message only once S1, S2 and S3 have acknowledged the message (if it is not the behavior by default) Is there a way to configure a consumer to automatically unsubscribe when the client connection is lost
What I am trying to achieve "deliver at least once" to all connected consumers without using persistent topics (for performance and ease of deployment). ---- 2019-10-15 14:14:12 UTC - jamie: @jamie has joined the channel ---- 2019-10-15 14:26:01 UTC - jamie: Hello :wave: Hoping to get something cleared up around message retention... Scenario: I've got message retention set up to retain infinitely. A message is produced on a persistent topic. There may be existing subscribers at this point. A new subscriber comes along and wants to get all retained messages (or from a point in time, either works) - can it do this? An example: ``` ./pulsar-admin tenants create mytenant ./pulsar-admin namespaces create mytenant/myns ./pulsar-admin namespaces set-retention mytenant/myns --size 100M --time 10h ./pulsar-client produce <persistent://mytenant/myns/mypersistenttopic> -m "hello persistent!" ./pulsar-client consume <persistent://mytenant/myns/mypersistenttopic> -s persistent-sub - no messages returned as I subscribed after it was produced - can the subscriber get to them somehow? ``` Apologies if I've missed the obvious in the docs / searching through here. ---- 2019-10-15 14:35:59 UTC - jamie: Maybe I just needed to search a bit harder... <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1568506719027800?thread_ts=1568505498.026600&cid=C5Z4T36F7> ---- 2019-10-15 14:37:32 UTC - Alexandre DUVAL: Hi, how to keep subscription alive even on functions restart ? ---- 2019-10-15 15:56:32 UTC - Matteo Merli: > I couldn’t find how broker acknowledgement happens on *non-persistent* topic Producer will get an ack as soon as message reaches the broker. > Is there a way to configure the Broker to acknowledge the message only once S1, S2 and S3 have acknowledged the message (if it is not the behavior by default) That would kind of stretch the pub/sub semantic that is more centered on decoupling producers and consumers. Also, for non-persistent that would mean to keep the messages in memory for an extended amount of time. > Is there a way to configure a consumer to automatically unsubscribe when the client connection is lost Yes, you could use a `Reader` instead of a `Consumer`. ---- 2019-10-15 15:58:45 UTC - Matteo Merli: Correct. You can also pre-create the subscription on set it to the earliest message: ``` pulsar-admin topics create-subscription $TOPIC --subscription $SUBSCRIPTION --messageId earliest ``` ---- 2019-10-15 15:59:38 UTC - Matteo Merli: Subscriptions are durable and are not affected when a function is restarted. If a function is *deleted* it would, by default, also unsubscribe from the topic. ---- 2019-10-15 16:21:26 UTC - Alexandre DUVAL: Change inputs of a function requires a delete/create, right? You said by default deleting function unsubscribes it, it is possible to not unsubscribe? (The function should have the same fqdn as the subscription name is used, correct?). ---- 2019-10-15 18:17:48 UTC - Oleg Kozlov: Hello, I'm trying out Pulsar for our project. Have the the standalone Pulsar running locally on my mac, and dashboard also in docker. When i go to dashboard - there are no brokers/namespaces/topics - it's empty. The only thing i was able to get is the name of the cluster (standalone) to show up in the Clusters box , but when i click on it -no other info is displayed. ---- 2019-10-15 18:18:17 UTC - Oleg Kozlov: ---- 2019-10-15 18:18:31 UTC - Oleg Kozlov: using this to start dashboard: ---- 2019-10-15 18:18:33 UTC - Oleg Kozlov: SERVICE_URL=<http://172.16.3.99:8080> docker run -p 80:80 -e SERVICE_URL=$SERVICE_URL apachepulsar/pulsar-dashboard:2.4.1 ---- 2019-10-15 18:18:52 UTC - Oleg Kozlov: where the ip address is my machine's laptop ---- 2019-10-15 18:22:44 UTC - Chris Bartholomew: Having you done anything in the cluster that would generate stats (ex sent some message to a topic)? The dashboard generally runs off the stats, so until you run some traffic through the broker, it doesn't display much. Once you do that, brokers/namespaces/topics will show up. ---- 2019-10-15 18:26:22 UTC - Oleg Kozlov: yes, i produced about 1mil messages after i started it ---- 2019-10-15 18:26:34 UTC - David Kjerrumgaard: @Alexandre DUVAL You can try using the function update command. <http://pulsar.apache.org/docs/en/pulsar-admin/#update-1> ---- 2019-10-15 18:27:07 UTC - Oleg Kozlov: i also ran consumer to read those messages ---- 2019-10-15 18:27:31 UTC - Chris Bartholomew: Ok, then that's not the issue :grinning: ---- 2019-10-15 18:27:34 UTC - Alexandre DUVAL: Update gives error when changing inputs @David Kjerrumgaard ---- 2019-10-15 18:28:17 UTC - David Kjerrumgaard: Are you using the --inputs switch? What is the error? ---- 2019-10-15 18:28:18 UTC - Oleg Kozlov: right, i saw some threads online that mentioned that.. my issue is that nothing is showing up :disappointed: ---- 2019-10-15 18:28:59 UTC - Oleg Kozlov: i also tried ssh'ing to broker docker instance and running "pulsar-admin clusters list", i think someone said that would work, but it didn't in my case.. ---- 2019-10-15 18:29:14 UTC - Chris Bartholomew: Do you see anything in the Pulsar logs that show the dashboard is polling the broker? ---- 2019-10-15 18:30:01 UTC - Oleg Kozlov: hm, not sure, what should i look for specifically? ---- 2019-10-15 18:35:34 UTC - Chris Bartholomew: There should be some logs about the REST GET commands to the /admin/broker-stats/destinations endpoint ---- 2019-10-15 18:38:44 UTC - Chris Bartholomew: When you started the standalone Pulsar, did you specify your laptop's IP address: ```$ bin/pulsar standalone --advertised-address 1.2.3.4``` ---- 2019-10-15 18:39:38 UTC - Chris Bartholomew: Otherwise, it will bind to localhost, which won't be reachable from the Docker container ---- 2019-10-15 18:46:14 UTC - Oleg Kozlov: i'm running pulsar in docker can i pass that parameter pulsar running in docker? they run in separate containers ---- 2019-10-15 18:46:24 UTC - Oleg Kozlov: (cluster and dashboard) ---- 2019-10-15 18:53:44 UTC - Oleg Kozlov: ok, i just tried passing --advertised-address with my ip address running pulsar standalone w/o docker, i can see broker and topics in the dashboard now ---- 2019-10-15 18:55:35 UTC - Chris Bartholomew: Cool. I am sure there is a way to get this to work running both Pulsar and the dashboard in Docker, though. ---- 2019-10-15 18:56:23 UTC - Oleg Kozlov: yep, i'll figure it out, thanks for your help! ---- 2019-10-15 19:29:31 UTC - Luke Lu: zk 3.5+ server appears to have this regression/bug (that we also encountered): <https://issues.apache.org/jira/browse/ZOOKEEPER-3466> It appears that zk 3.5+ client can work with zk 3.4.6+ servers though. ---- 2019-10-15 20:45:15 UTC - Naby: Hi Is there a way to create a pulsar producer without tying it to a specific topic and use it for producing to several topics? I am using a python client. ---- 2019-10-15 20:46:03 UTC - Matteo Merli: No, a producer is always tied to a specific topic ---- 2019-10-15 21:06:22 UTC - Alexandre DUVAL: For custom usage, we are talking about shade all the pulsar project for a side project, what do you think? ;) ---- 2019-10-15 21:18:37 UTC - Junli Antolovich: Hello, I am getting this error trying to 'Set up a standalone Pulsar in Docker' in powershell on windows. This is the first time I am using docker and I made sure both docker and Kubernetes are running. Do I need to download Pulsar before install it in docker? ---- 2019-10-15 21:18:41 UTC - Junli Antolovich: PS C:\Program Files\docker\docker\resources\bin> docker run -it \ -p 6650:6650 \ -p 8080:8080 \ -v "$PWD/data:/pulsar/data".ToLower() \ apachepulsar/pulsar:2.4.1 \ bin/pulsar standalone docker : C:\Program Files\Docker\Docker\Resources\bin\docker.exe: invalid reference format. At line:1 char:1 + docker run -it \ + ~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (C:\Program File...ference format.:String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'. -p : The term '-p' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:2 char:3 + -p 6650:6650 \ + ~~ + CategoryInfo : ObjectNotFound: (-p:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException -p : The term '-p' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:3 char:3 + -p 8080:8080 \ + ~~ + CategoryInfo : ObjectNotFound: (-p:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException -v : The term '-v' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:4 char:3 + -v "$PWD/data:/pulsar/data".ToLower() \ + ~~ + CategoryInfo : ObjectNotFound: (-v:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException apachepulsar/pulsar:2.4.1 : The term 'apachepulsar/pulsar:2.4.1' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:5 char:3 + apachepulsar/pulsar:2.4.1 \ + ~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (apachepulsar/pulsar:2.4.1:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException bin/pulsar : The term 'bin/pulsar' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:6 char:3 + bin/pulsar standalone + ~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (bin/pulsar:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException ---- 2019-10-15 21:20:05 UTC - Tim Howard: so it would probably be ideal in the future to use a snippet or start a thread under your message before you paste a wall of text into chat ---- 2019-10-15 21:20:30 UTC - Tim Howard: my guess is that you are trying to use a bash shell syntax in Powershell, but I can't be sure, i don't know Powershell ---- 2019-10-15 21:21:16 UTC - Tim Howard: does powershell allow new-line breaks in its commands like that? ---- 2019-10-15 21:25:10 UTC - Junli Antolovich: Sorry, have not used slacker before either. will try your suggestion in the future ---- 2019-10-15 21:25:26 UTC - Tim Howard: np- i see now that you were using the command from the docs for Windows ---- 2019-10-15 21:25:50 UTC - Tim Howard: oh- but it doesn't say on the page explicitly Powershell ---- 2019-10-15 21:25:58 UTC - Tim Howard: can you just try from the Command Prompt window? ---- 2019-10-15 21:26:13 UTC - Tim Howard: my guess is it's not Powershell syntax ---- 2019-10-15 21:37:27 UTC - Junli Antolovich: You are right, PS does not recognize it. I put the command into one line and run it cmd and get this error: ---- 2019-10-15 21:37:31 UTC - Junli Antolovich: docker: Error response from daemon: create $PWD/data: "$PWD/data" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path. ---- 2019-10-15 21:38:35 UTC - Junli Antolovich: Do I need to replace "$PWD/data" to something else? ---- 2019-10-15 21:39:01 UTC - Tim Howard: i think you need to use the Command Prompt (not powershell) or learn the differences and translate on your own ---- 2019-10-15 21:39:18 UTC - Tim Howard: I don't have any of those tools so I can only speculate but I don't think it's meant for Powershell ---- 2019-10-15 21:40:13 UTC - Tim Howard: `cmd.exe` i think ---- 2019-10-15 21:41:18 UTC - Tim Howard: oh did you run it in the Command Prompt? ---- 2019-10-15 21:41:30 UTC - Tim Howard: sorry- i think I missed that in your above message `cmd` ---- 2019-10-15 21:41:56 UTC - Junli Antolovich: I am using Command Prompt as admin. It does not understand --v"$PWD/data:/pulsar/data".ToLower() apachepulsar/pulsar:2.4.1 ---- 2019-10-15 21:42:57 UTC - Tim Howard: i'm honestly unsure now, when I look at the command again, the fact that it has .ToLower() almost certainly means it is actually powershell ---- 2019-10-15 21:45:09 UTC - Junli Antolovich: The cmd did pull 'apachepulsar/pulsar:2.4.1' to local, which is better than PS ---- 2019-10-15 21:45:39 UTC - Junli Antolovich: I will try convert the command to lower case and try ---- 2019-10-15 21:46:04 UTC - Tim Howard: sorry I can't help more, I infrequently use windows and never for anything serious :slightly_smiling_face: ---- 2019-10-15 21:49:07 UTC - Chris Bartholomew: @Junli Antolovich You just need to change ```$PWD:data``` to a valid Windows path. The environment variable $PWD is Linux specific. This will be the directory where the Pulsar data will be persisted. ---- 2019-10-15 21:50:13 UTC - Tim Howard: i sort of thought that also but the fact that it was in the Windows example on the Pulsar site I thought it might just be old knowledge on my part :confused: ---- 2019-10-15 21:50:48 UTC - Tim Howard: Windows also won't use forward slashes for its path will it? ---- 2019-10-15 21:51:50 UTC - Junli Antolovich: Thanks @Chris Bartholomew. trying ... :slightly_smiling_face: ---- 2019-10-15 21:54:41 UTC - Chris Bartholomew: @Tim Howard I don't run Docker on Windows very often, but I think the Docker command will accept the forward slashes. I found this example in a blog post: ```D:\>docker run -v d:/data:/data alpine ls /data``` which suggests it will work. ---- 2019-10-15 21:55:14 UTC - Tim Howard: interesting, ok.. thanks... stale knowledge ---- 2019-10-15 21:55:55 UTC - Tim Howard: oh- docker is using that, right... not the shell... makes sense then ---- 2019-10-15 22:05:05 UTC - Junli Antolovich: @Chris Bartholomew, a step closer. I am getting '[main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service. java.io.IOException: Error open RocksDB database". Do I need to install RocksDB? ---- 2019-10-15 22:09:35 UTC - Junli Antolovich: any reference on how to install RocksDB? ---- 2019-10-15 22:11:19 UTC - Junli Antolovich: Run it from PS and get the same error... ---- 2019-10-15 22:15:33 UTC - Junli Antolovich: @Chris Bartholomew here are part of the stdout from the command: ---- 2019-10-15 22:19:10 UTC - Junli Antolovich: 22:10:06.860 [main] ERROR org.apache.bookkeeper.bookie.Journal - Problems reading from data/standalone/bookkeeper0/current/lastMark (this is okay if it is the first time starting this bookie 22:10:06.862 [main] INFO org.apache.bookkeeper.bookie.Bookie - Using ledger storage: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage 22:10:06.865 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Started Db Ledger Storage 22:10:06.866 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - - Number of directories: 1 22:10:06.866 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - - Write cache size: 1024 MB 22:10:06.866 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - - Read Cache: 1024 MB 22:10:06.874 [main] INFO org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage - Creating single directory db ledger storage on data/standalone/bookkeeper0/current 22:10:07.044 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 22:10:07.058 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service. java.io.IOException: Error open RocksDB database at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:182) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:58) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4. 9.2] at org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:160) ~[org.apache.bookkeeper-bookkeeper-server-4.9. 2.jar:4.9.2] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:148) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.j ar:4.9.2] at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:128) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:790) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:106) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:272) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:396) ~[org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:259) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:121) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1] Caused by: org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument at org.rocksdb.RocksDB.open(Native Method) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?] at org.rocksdb.RocksDB.open(RocksDB.java:231) ~[org.rocksdb-rocksdbjni-5.13.3.jar:?] at org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:179) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] ... 13 more 22:10:07.063 [Thread-1] ERROR org.apache.pulsar.PulsarStandaloneStarter - Shutdown failed: null ---- 2019-10-15 22:28:15 UTC - Quentin ADAM: . ---- 2019-10-16 00:58:41 UTC - Poule: is there a plan for integration with Knative as PulsarChannel or source/sink ---- 2019-10-16 00:59:36 UTC - Ali Ahmed: not currently but it should be pretty simple to do with the pulsar io framework heart_eyes : Poule ---- 2019-10-16 01:59:09 UTC - Junli Antolovich: Opened a bug in github -<https://github.com/apache/pulsar/issues/5392> ---- 2019-10-16 05:03:51 UTC - vikash: @Vladimir Shchur i have used Pulsar Client 0.5.0 in .net FrameWork after implementing i am getting Error TypeLoadException: Could not load type 'Microsoft.Extensions.Logging.Abstractions.Internal.NullScope' from assembly 'Microsoft.Extensions.Logging.Abstractions, Version=3.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. ---- 2019-10-16 05:04:21 UTC - vikash: Error after implementation ---- 2019-10-16 05:07:13 UTC - vikash: but same code working in 0.1.1 ---- 2019-10-16 05:14:36 UTC - Vladimir Shchur: Hi! Hm.. Looks like you haven't assigned any logger? ---- 2019-10-16 05:16:11 UTC - Vladimir Shchur: In this case null logger should have been used, but looks like it doesn't work, I will take a look soon ---- 2019-10-16 05:26:10 UTC - Vladimir Shchur: By the way, have you tried some more binding redirects? ---- 2019-10-16 05:42:52 UTC - vikash: no i have not triyed binding ---- 2019-10-16 06:43:56 UTC - Vladimir Shchur: Can you please try installing Microsoft.Extensions.Logging package. If it fixes the issue for you, I'll add it to dependent packages in next version ---- 2019-10-16 07:23:23 UTC - Maak: @Maak has joined the channel ---- 2019-10-16 07:24:21 UTC - Gilles Barbier: @Gilles Barbier has joined the channel ----
