[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544802423 This may be the solution... - If you are starting from fresh you need to make sure to configure `PF_pulsarFunctionsCluster` right before deploying the worker. - If you already started the worker with a wrong configuration, you can try this out: ```bash $ bin/pulsar-admin namespaces set-clusters --cluster= public/functions ``` And then redeploy the worker with a fixed configuration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure
when set functionsWorkerEnabled=true
URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544799548
@rocketraman you were right. My solution does not make any sense. When I
tried it for the first time I was working on a new cluster. Before that, I
tried many other solutions but none of them work (including setting the `PF_`
env variables in `broker.yaml`). And I think that's the issue here, once the
clusters of `/admin/policies/public/functions` are set, they can't change.
Check this code from
`pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java`:
```java
// getting namespace policy
log.info("Initializing Pulsar Functions namespace...");
try {
try {
admin.namespaces().getPolicies(workerConfig.getPulsarFunctionsNamespace());
} catch (PulsarAdminException e) {
if (e.getStatusCode() == Response.Status.NOT_FOUND.getStatusCode()) {
// if not found than create
try {
Policies policies = new Policies();
policies.retention_policies = new RetentionPolicies(-1, -1);
policies.replication_clusters = new HashSet<>();
policies.replication_clusters.add(workerConfig.getPulsarFunctionsCluster());
admin.namespaces().createNamespace(workerConfig.getPulsarFunctionsNamespace(),
policies);
} catch (PulsarAdminException e1) {
// prevent race condition with other workers starting up
if (e1.getStatusCode() !=
Response.Status.CONFLICT.getStatusCode()) {
log.error("Failed to create namespace {} for pulsar
functions", workerConfig
.getPulsarFunctionsNamespace(), e1);
throw e1;
}
}
} else {
log.error("Failed to get retention policy for pulsar function
namespace {}",
workerConfig.getPulsarFunctionsNamespace(), e);
throw e;
}
}
try {
internalConf = admin.brokers().getInternalConfigurationData();
} catch (PulsarAdminException e) {
log.error("Failed to retrieve broker internal configuration", e);
throw e;
}
} finally {
admin.close();
}
```
It looks like the values are only set if they are not found. What do you
think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544788820 How did you initialize you cluster metadata? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544722192 I'm not adding that value to the ZooKeeper configuration. It is being added to `functions_worker.yml`. Both for the broker and ZooKeeper services. After reading (https://github.com/apache/pulsar/issues/2328#issuecomment-411013973)[this] I got the idea that maybe there was an error during the creation of the ZooKeeper data. I tried this workaround and it looks like it's working. Keep in mind that`PF_` env variales are only applied if you add this line: ``` bin/gen-yml-from-env.py conf/functions_worker.yml && ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544706581 I spent more than 12 hours on this... There are 2 problems here: - The functions worker needs to know the cluster name. - The `public/functions` namespace needs to have that cluster set. The list of `public/functions` clusters is set when the ZooKeeper service is created. Here is my solution: - `zookeeper.yaml` ``` apiVersion: v1 kind: ConfigMap metadata: name: zookeeper-config data: ... PF_pulsarFunctionsCluster: rings PF_numFunctionPackageReplicas: "2" ... containers: - name: zookeeper image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/generate-zookeeper-config.sh conf/zookeeper.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/pulsar zookeeper ... ``` - `broker.yaml`: ``` apiVersion: v1 kind: ConfigMap metadata: name: broker-config data: ... functionsWorkerEnabled: "true" PF_pulsarFunctionsCluster: rings PF_numFunctionPackageReplicas: "2" ... containers: - name: broker image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/broker.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/pulsar broker ... ``` I hope this helps you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
