2018-11-27 09:16:00 UTC - Sijie Guo: no you don’t need to start 2 zookeeper. you can read this doc <http://pulsar.apache.org/docs/en/deploy-bare-metal/> to understand how pulsar components are deployed. ---- 2018-11-27 09:26:28 UTC - bossbaby: but i see need 3 entry zookeeperServers ---- 2018-11-27 09:26:57 UTC - bossbaby: i'm run in local not use cloud ---- 2018-11-27 10:06:43 UTC - bossbaby: i run success, thanks you bro +1 : jia zhai ---- 2018-11-27 10:14:47 UTC - Yifan: Hi I deployed Pulsar on Azure AKS with helm. It deploys bastion instead of pulsar-admin node as in standard deployment. I couldn’t find any information on bastion in pulsar, should I deploy my own pulsar admin? I couldn’t find anything on pulsar bastion ---- 2018-11-27 10:24:41 UTC - Sijie Guo: I think bastion is the “admin node” (the contributor who contributes this helmchart probably call it bastion). you can run `pulsar-admin` on bastion. ---- 2018-11-27 10:25:02 UTC - Sijie Guo: np ---- 2018-11-27 10:35:05 UTC - Yifan: it works, thank you @Sijie Guo ---- 2018-11-27 11:14:42 UTC - Ivan Kelly: bastion is admin? it would make more sense for the bastion to be the proxy ---- 2018-11-27 11:14:48 UTC - Ivan Kelly: linguistically ---- 2018-11-27 12:34:56 UTC - Yifan: Yes, it is admin, and there is a proxy pod with the name ‘proxy’ ---- 2018-11-27 16:46:45 UTC - Yifan: Hi, when ‘pulsar-admin source create’, in what format should I write the config argument? ‘’' --source-config Source config key/values ‘’' ---- 2018-11-27 16:52:59 UTC - Sijie Guo: I think he was referring to one of the k8 deployment doc, there is one pod deployed called admin. And in the helmchart, it was called bastion. They are serving as same purpose, which is bastion. ---- 2018-11-27 16:59:31 UTC - Sijie Guo: It should something like
config: param1: param2: You can find the actual parameters at each connector’s doc : <http://pulsar.apache.org/docs/en/io-connectors/> ---- 2018-11-27 17:00:32 UTC - Yifan: I see, basically the content of the config file in command line, right? ---- 2018-11-27 17:02:20 UTC - Yifan: `com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 path $` ---- 2018-11-27 17:02:47 UTC - Yifan: it looks like JSON is expected ---- 2018-11-27 17:04:21 UTC - Sijie Guo: I remember you can provide a yaml file as config file or a jsonified string ---- 2018-11-27 17:04:28 UTC - Yifan: got it, it is indeed jsonfield string ---- 2018-11-27 17:04:31 UTC - Sijie Guo: I need to double check ---- 2018-11-27 17:04:32 UTC - Yifan: thanks. ---- 2018-11-27 17:29:13 UTC - Yifan: @Sijie Guo how should I debug connectors? I mean, where can I see logs from my connector source? ---- 2018-11-27 17:35:27 UTC - Emma Pollum: Are there other authentication providers besides Athenz? Is it possible to use other authentication and plug it in broker.conf "authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderAthenz" ---- 2018-11-27 17:42:45 UTC - Emma Pollum: And if I were to use TLS auth, how do I generate a role token? ---- 2018-11-27 17:43:01 UTC - Sanjeev Kulkarni: @Yifan connectors use the same backend as functions. Thus you can get the connectors log at logs/functions/tenant/namespace/connectorname/ ---- 2018-11-27 18:14:29 UTC - Yifan: thanks, I am not sure I understand. is this a topic I access with <persistent://logs/functions/tenant/namespace/connectorname/>? I see no such topics in dashboard. pulsar-admin source create doesn’t take log-topics argument ---- 2018-11-27 18:17:36 UTC - Matteo Merli: @Emma Pollum There’s doc for enabling TLS auth: <http://pulsar.apache.org/docs/en/security-tls-authentication/> We’re also working on adding a simplified auth plugin based on “tokens” for 2.3 release. See <https://github.com/apache/pulsar/wiki/PIP-25%3A-Token-based-authentication> ---- 2018-11-27 18:48:33 UTC - Sanjeev Kulkarni: @Yifan when you run connectors(pulsar-admin sink/source create/localrun), they spit out logs on the machine that they are running. On that machine you can go to <pulsar-root>/logs/functions/<tenant>/<namespace>/<source/sinkname>/<source/sinkname>.log ---- 2018-11-27 18:48:46 UTC - Sanjeev Kulkarni: are you running the connector using localrun? ---- 2018-11-27 18:48:51 UTC - Sanjeev Kulkarni: or doing a create ---- 2018-11-27 18:53:35 UTC - Grant Wu: Does anyone know what “topic hierarchy fan-out” would mean in the context of Pulsar? ---- 2018-11-27 18:53:49 UTC - Grant Wu: Someone here at work is saying Pulsar has this feature, but I have no clue what they mean. ---- 2018-11-27 18:53:53 UTC - Grant Wu: Did they perhaps mean partitioned topics? ---- 2018-11-27 18:56:25 UTC - Yifan: @Sanjeev Kulkarni I did create. it keeps restarting ```{ "functionStatusList": [ { "numRestarts": "3", "instanceId": "0", "workerId": "c-geared-possum-pulsar-fw-10.244.14.24-8080" } ] }``` ---- 2018-11-27 18:57:34 UTC - Yifan: I guess I can use localrun on admin machine, will do that now ---- 2018-11-27 18:58:14 UTC - Emma Pollum: Are there any plans to implent oauth? ---- 2018-11-27 19:00:57 UTC - Yifan: ``` 18:58:35.493 [main] INFO org.apache.pulsar.functions.runtime.ProcessRuntime - Started process successfully 18:58:53.712 [main] INFO org.apache.pulsar.admin.cli.CmdFunctions - RuntimeSpawner quit because of ``` ---- 2018-11-27 19:02:43 UTC - Sanjeev Kulkarni: Is this master? or 2.2? ---- 2018-11-27 19:03:09 UTC - Yifan: I deployed using helm ---- 2018-11-27 19:03:41 UTC - Yifan: I had to remove rbac for it to work on my AKS cluster though ---- 2018-11-27 19:05:50 UTC - Yifan: found the log file ---- 2018-11-27 19:06:54 UTC - Yifan: ```ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable Uncaught exception in Java Instance java.lang.NullPointerException: null at org.apache.pulsar.io.kafka.KafkaAbstractSource.open(KafkaAbstractSource.java:59) ~[pulsar-io-kafka-2.2.0.nar-unpacked/:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:570) ~[java-instance.jar:2.2.0] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:162) ~[java-instance.jar:2.2.0] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:188) [java-instance.jar:2.2.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] ``` ---- 2018-11-27 19:07:47 UTC - Yifan: something wrong with my kafka input maybe ---- 2018-11-27 19:11:22 UTC - Matteo Merli: Not immediate, though it should be easy to add on the server side, on top of the existing token auth ---- 2018-11-27 19:12:40 UTC - Yifan: thanks for the help @Sanjeev Kulkarni , I will revisit it tomorrow I guess. At least I can see the exact error message now. ---- 2018-11-27 19:21:54 UTC - Sanjeev Kulkarni: I believe there might be some configuration error for the source ---- 2018-11-27 21:05:18 UTC - Sijie Guo: no idea. does he mean some kind of “parent-child” topics? ---- 2018-11-27 21:05:44 UTC - Grant Wu: Apparently it is something of their own creation involving Pulsar functions ---- 2018-11-27 21:05:56 UTC - Sijie Guo: gotcha ---- 2018-11-27 23:42:38 UTC - Emma Pollum: Are there any plans to fix the ByteBuffer issue that requires us to use java8 with pulsar? ---- 2018-11-27 23:42:55 UTC - Emma Pollum: I'm hoping to update to java9 but continue to get teh java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear() exception ---- 2018-11-27 23:43:43 UTC - Matteo Merli: master is already working with java-11. We plan to support properly from 2.3 release ---- 2018-11-27 23:44:05 UTC - Emma Pollum: :thumbsup: ---- 2018-11-28 00:06:53 UTC - Emma Pollum: What exactly is the publish latency measuring in the openmessaging benchmark tests? ---- 2018-11-28 00:07:54 UTC - Matteo Merli: the time it takes for `producer.send()`, meaning when a message is successfully published (with the configured replication factor) ---- 2018-11-28 00:14:45 UTC - Emma Pollum: Thanks. I'm seeing my backlog go negative occassionally in the benchmarking tests... any ideas what I'm doing wrong? ---- 2018-11-28 00:15:46 UTC - Matteo Merli: The stats are collected from multiple nodes independently, so it can go negative ---- 2018-11-28 00:47:00 UTC - Samuel Sun: got it, thanks @Matteo Merli ---- 2018-11-28 02:25:58 UTC - xiaoyong: @xiaoyong has joined the channel ---- 2018-11-28 06:36:30 UTC - Sanjay K: @Sanjay K has joined the channel ---- 2018-11-28 09:07:52 UTC - Yifan: apparently the problem with KafkaSource connector is that the optional options are not optional in configuration, those with default ‘null’ will crash ----
