2019-02-05 11:16:07 UTC - dba: Hi guys Having read the (somewhat sparse and partial outdated) documentation of Pulsar Functions for Python, I have a few questions and I'm hoping you are willing to help :slightly_smiling_face: Is there an ETA for the following (all in the context of Pulsar Functions for Python): * Docker deployment * PEXes deployment * Support for state And a few questions: * It there an ETA on support for Go and JavaScript? * You can define multiple input topics for a function, but can they have different data types? Meaning one topic with a complex type A and another with complex type B? If yes, is there an example of that somewhere? * While waiting for docker and/or PEXes deployment, how do you deploy python functions that make use of external packages? * I can see that there are two built-in (de)serializers for python. Since protobuf is cross language and already there as a dependency, would it make sense to create a built-in SerDe for it? If not, it there an example of one somewhere? * Is there a way to have my function's cursor reset? So that it will reprocess the entire topic? Hope the questions make sense. I am new to Pulsar Functions and I am not a Python developer. Any help is appreciated :slightly_smiling_face: ---- 2019-02-05 11:32:10 UTC - Ivan Kelly: At least for state, a lot of it should be in 2.3.0 (finishing touches going on now) ---- 2019-02-05 11:32:36 UTC - Ivan Kelly: lots of go support in that too ---- 2019-02-05 11:32:47 UTC - Ivan Kelly: js, not so much ---- 2019-02-05 11:33:22 UTC - Ivan Kelly: > • While waiting for docker and/or PEXes deployment, how do you deploy python functions that make use of external packages? The machine has to have the packages installed (it's sucky) ---- 2019-02-05 11:35:11 UTC - Ivan Kelly: > • Is there a way to have my function's cursor reset? So that it will reprocess the entire topic? I think so. If you know the functions subscription name, you can reset the cursor (<http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Topics.html#resetCursor-java.lang.String-java.lang.String-MessageId->) ---- 2019-02-05 11:35:57 UTC - Ivan Kelly: I link the javadoc because it's generally more accurate, but this should be available via the rest api or cli tool (pulsar-admin topics ...) ---- 2019-02-05 11:36:17 UTC - Ivan Kelly: to find the function subscription <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Topics.html#getSubscriptions-java.lang.String-> ---- 2019-02-05 12:24:28 UTC - Sijie Guo: > Docker deployment
Kubernetes support will be included in 2.3.0 > PEXes deployment I don’t think we chose PEX. but in 2.3.0, it supports wheel package. > Support for state it will be in 2.3.0 > It there an ETA on support for Go and JavaScript? Javascript will be depending on a js client. I think folks from Yahoo Japan are working on a js client. as far as I know, there are a few Chinese contributors looking into Go function support. > but can they have different data types? Meaning one topic with a complex type A and another with complex type B? If yes, is there an example of that somewhere? currently it doesn’t support different POJOs. However a workaround would be using `byte[]`, so you can deserialize in the function based on the topics. > While waiting for docker and/or PEXes deployment, how do you deploy python functions that make use of external packages? you can use wheel files and define a requirement file. it is coming in 2.3.0. > I can see that there are two built-in (de)serializers for python. Since protobuf is cross language and already there as a dependency, would it make sense to create a built-in SerDe for it? If not, it there an example of one somewhere? python schema is added in 2.3.0. so we will have better python serde supports in functions, once the python schema is in. ---- 2019-02-05 12:24:55 UTC - Sijie Guo: @dba I added a few more comments based on Ivan’s comment ---- 2019-02-05 15:24:05 UTC - dba: @Sijie Guo @Ivan Kelly That sounds great! Thanks! Looking forward to 2.3.0 then :slightly_smiling_face: ---- 2019-02-05 16:59:41 UTC - Karthik Palanivelu: Team, I am just upgrading from version 2.1.1 to 2.2.1. I updated the package alone and I kept the same configuration files. I use to containerize it. When I run bookie, it just run and stops. I do not see any errors other than below. If I revert the version to earlier it works fine. Can you please let me know what I am missing? When running manually inside the container seeing the below error: ``` ./pulsar bookie Error: Could not find or load main class " ``` ``` 11:47:27.174 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=<ZK-IP>:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@2fc6f97f 11:47:27.223 [main-SendThread(HOST:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server <ZK-IP>:2181. Will not attempt to authenticate using SASL (unknown error) 11:47:27.232 [main-SendThread(HOST:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to <ZK-IP>:2181, initiating session 11:47:27.239 [main-SendThread(HOST:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server <ZK-IP>:2181, sessionid = 0x10000003f8e0030, negotiated timeout = 30000 11:47:27.250 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 11:47:27.270 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10000003f8e0030 closed 11:47:27.270 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10000003f8e0030 ``` ---- 2019-02-05 17:19:28 UTC - Ichsan Hariadi: @Ichsan Hariadi has joined the channel ---- 2019-02-05 19:52:44 UTC - Emma Pollum: Hey team, I recently removed two bookkeepers from my pulsar cluster. Now I'm getting messages that some ledgers are under replicated and its still trying to replicate to the removed bookies. How do I force it to try a different bookie? ---- 2019-02-05 20:02:29 UTC - Ali Ahmed: @dba <http://pulsar.apache.org/docs/en/next/functions-quickstart/#packaging-python-dependencies> +1 : dba ---- 2019-02-06 00:35:20 UTC - David Kjerrumgaard: @Emma Pollum Do you have 3 bookies now or just the one? ---- 2019-02-06 01:12:45 UTC - Emma Pollum: I have 6 bookies running, I did have 8 ---- 2019-02-06 03:01:39 UTC - David Kjerrumgaard: @Emma Pollum Can you share the messages that indicate the messages are trying to replicate to the removed bookies? ---- 2019-02-06 04:18:59 UTC - Karthik Palanivelu: @Matteo Merli @Sijie Guo Any help here? I am facing this issue only on Kubernetes cluster. Same image has no issues locally - runs fine on Mac. It looks weird to me. ---- 2019-02-06 05:03:34 UTC - Sijie Guo: @Karthikeyan Palanivelu - interesting. have you configured PULSAR_MEM? ---- 2019-02-06 05:14:35 UTC - Karthik Palanivelu: Yes I did - `PULSAR_MEM: "\" -Xms2048m -Xmx4096m -XX:MaxDirectMemorySize=4g\""`. But it is complaining about main class not found. ---- 2019-02-06 05:15:30 UTC - Karthik Palanivelu: But this works fine with version 2.1.1. ---- 2019-02-06 05:15:35 UTC - Sijie Guo: yeah. I think the problem is coming from PULSAR_MEM. ---- 2019-02-06 05:15:52 UTC - Sijie Guo: one second let me check something quickly ---- 2019-02-06 05:24:20 UTC - Sijie Guo: I think it is related to <https://github.com/apache/pulsar/pull/2484> ---- 2019-02-06 05:24:48 UTC - Sijie Guo: are you using helm for your deployment? ---- 2019-02-06 05:34:11 UTC - Sijie Guo: is there any changes about your deployment script? or was it just a version change? ----
