2019-02-18 05:02:23 UTC - Jacob O'Farrell: Hey all! Currently trying to deploy pulsar functions onto a kubernetes deployment, however when running the `pulsar-admin functions create` command the python function in question, we are running into the following error: ```Function worker service is not done initializing. Please try again in a little while.
Reason: Function worker service is not done initializing. Please try again in a little while. command terminated with exit code 1``` This happens both with our function, and the example functions available in the docs. Any suggestions as to where I should start looking would be appreciated! ---- 2019-02-18 05:03:18 UTC - Sijie Guo: have you deployed function worker? or enable function worker at your brokers? ---- 2019-02-18 05:04:21 UTC - Jacob O'Farrell: @Sijie Guo thanks for the quick response! I've just followed the getting started guides, and used most of the default configuration. Where would I find these settings/how would I configure the function worker? I've had a poke around the documentation but couldn't find anything (either that, or I'm blind - happy to be shown where I should be looking!) ---- 2019-02-18 05:05:26 UTC - Sijie Guo: are you using helm chart to deploy pulsar to your k8s cluster? ---- 2019-02-18 05:06:44 UTC - Jacob O'Farrell: I believe we deployed using the AWS yaml files, with a few modifications for our environment/in response to some issues we ran into ---- 2019-02-18 05:07:24 UTC - Sijie Guo: <http://pulsar.apache.org/docs/en/deploy-bare-metal/#enabling-pulsar-functions-optional> ---- 2019-02-18 05:07:41 UTC - Sijie Guo: I think the instructions were added to bare-metal deployment. but not added to k8s. ---- 2019-02-18 05:08:05 UTC - Sijie Guo: but the instructions should be applied to k8s ---- 2019-02-18 05:08:30 UTC - Sijie Guo: you need to change conf/broker.conf and conf/function_worker.yml ---- 2019-02-18 05:11:47 UTC - Jacob O'Farrell: Okay - any suggestions for how to do this in the k8s context? Would I add `functionsWorkerEnabled=true` to the `broker-config` ConfigMap? (as defined in `broker.yaml`) ---- 2019-02-18 05:13:38 UTC - Sijie Guo: <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/aws/broker.yaml#L36> you can add `functionsWorkerEnabled: 'true'` after this line ---- 2019-02-18 05:14:14 UTC - Jacob O'Farrell: Understood! awesome ---- 2019-02-18 05:15:30 UTC - Jacob O'Farrell: In terms of `functions_worker.yaml` - where do I set this? <https://gyazo.com/2869c51b9b6fcf039a510f7272e8e899> ---- 2019-02-18 05:15:44 UTC - Jacob O'Farrell: (Thank you for your help - you can't begin to understand how thankful I am) ---- 2019-02-18 05:16:10 UTC - Jacob O'Farrell: I'm happy to work on the documentation to reflect how to set this up as well ---- 2019-02-18 05:17:33 UTC - Sijie Guo: good question about the functions_worker.yaml part. I think it requires some more work. let me see how to do that. thank you in advance for contributing fixes to documentation. :slightly_smiling_face: ---- 2019-02-18 05:18:07 UTC - Jacob O'Farrell: That would be very much appreciated! ---- 2019-02-18 05:26:05 UTC - Jacob O'Farrell: If there's anything I can help with please let me know - eager to see functions working on k8s ---- 2019-02-18 05:26:29 UTC - Sijie Guo: so I think you need to run this script - <https://github.com/apache/pulsar/blob/master/docker/pulsar/scripts/gen-yml-from-env.py> basically insert ``` gen-yml-from-env.py conf/functions_worker.yml && ``` after <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/aws/broker.yaml#L72> then you can configure function worker settings with “PF_” prefix in the config map. e.g. PF_pulsarFunctionsCluster: test-cluster ---- 2019-02-18 05:27:04 UTC - Sijie Guo: @Jacob O'Farrell that should be working. although I have tried that yet. :slightly_smiling_face: ---- 2019-02-18 05:44:13 UTC - Jacob O'Farrell: Okay just confirming I've understood, I should modify the `args`block of the broker.yaml file to look like: ``` args: - > mkdir logs && bin/apply-config-from-env.py conf/broker.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && gen-yml-from-env.py conf/functions_worker.yml && bin/pulsar broker ``` Slack has ruined this formatting but the idea is there ---- 2019-02-18 05:44:55 UTC - Sijie Guo: correct. ---- 2019-02-18 05:44:57 UTC - Sijie Guo: gen-yml-from-env.py => bin/gen-yml-from-env.py ---- 2019-02-18 05:45:03 UTC - Sijie Guo: I think the file should be in `bin` directory ---- 2019-02-18 05:45:23 UTC - Jacob O'Farrell: Is the fact the other steps are using config-from-env vs. this new one using gen-yml-from.env significant? ---- 2019-02-18 05:46:07 UTC - Sijie Guo: the conf/broker.conf and conf/pulsar_env.sh are properties file. while function_worker.yml is a yaml file. ---- 2019-02-18 05:58:42 UTC - Jacob O'Farrell: Giving it a test now. Is there a good place to have a look for what the setting sin the functions_worker.yml do/change/recommendations? ---- 2019-02-18 06:00:28 UTC - Sijie Guo: <https://github.com/apache/pulsar/blob/master/conf/functions_worker.yml> it is the probably the best to checkout the settings and their comments here. by default, you don’t really need to change any settings, except setting the cluster name. ---- 2019-02-18 06:02:23 UTC - Jacob O'Farrell: Okay sounds good! ---- 2019-02-18 06:03:23 UTC - Jacob O'Farrell: Should/will I need to change things like `pulsarServiceUrl: <pulsar://localhost:6650>` to point to the kubernetes service endpoint? ---- 2019-02-18 06:04:31 UTC - Sijie Guo: you don’t need to change that when running as part of broker. broker will configure those settings for you. ---- 2019-02-18 06:04:48 UTC - Sijie Guo: if you run function_worker as a separate service, then you need to configure those settings. ---- 2019-02-18 06:06:58 UTC - Jacob O'Farrell: I was unaware I could run function_worker as a separate service. Is this advisable? ---- 2019-02-18 06:13:23 UTC - Jacob O'Farrell: And if I wanted to - is there documentation available for this? Would this be advisable for high throughout processing situations or similar? ---- 2019-02-18 06:13:31 UTC - Jacob O'Farrell: Really appreciate all your assistance! ---- 2019-02-18 06:18:35 UTC - Sijie Guo: in kubernetes, we would recommend running function worker as part of broker, but enabling kubernetes runtime factory, so functions can be run as kubernenete jobs. <https://github.com/apache/pulsar/blob/master/conf/functions_worker.yml#L88> however the kubenentes runtime factory is requires some configuration. it is lacking of documentation. ---- 2019-02-18 06:25:59 UTC - Sijie Guo: (kubernetes runtime is not released yet. it will be included in 2.3.0 release. 2.3.0 release is under voting now. should be out in 2~3 days) ---- 2019-02-18 06:32:01 UTC - Jacob O'Farrell: Okay - so I'm best to wait for the 2.3.0 release by the sounds of things then? ---- 2019-02-18 06:32:40 UTC - Jacob O'Farrell: Also, do you have any comment on the sort of volume we should be able to process with Pulsar Functions? ---- 2019-02-18 06:33:06 UTC - Sijie Guo: you mean throughput? ---- 2019-02-18 06:33:39 UTC - Jacob O'Farrell: Yep - we'd be looking to process a few thousand records a second (processing clickstream data) with Pulsar functions ---- 2019-02-18 06:35:32 UTC - Sijie Guo: so the throughput of per function instance, will be depending on the avg time of calling the function. you can scale the processing but specifying the parallelism of the function (aka multiple instances invoking the function). so a few thousand records per second is not a problem.. ---- 2019-02-18 07:03:57 UTC - dba: @Sanjeev Kulkarni Ah ok, do you know if Python support is on the roadmap? ---- 2019-02-18 07:21:32 UTC - Jacob O'Farrell: @Sijie Guo Looks like your recommendations for the functions worker settings have worked! Thank you ---- 2019-02-18 07:21:46 UTC - Sijie Guo: cool ---- 2019-02-18 07:21:51 UTC - Sijie Guo: glad to hear it worked ---- 2019-02-18 07:21:53 UTC - Sijie Guo: :slightly_smiling_face: ----
