[GitHub] [pulsar] Guchman commented on a change in pull request #5429: fix: java doc on MessageListener - acknowledgement part
Guchman commented on a change in pull request #5429: fix: java doc on MessageListener - acknowledgement part URL: https://github.com/apache/pulsar/pull/5429#discussion_r337335216 ## File path: pulsar-client-1x-base/pulsar-client-1x/src/main/java/org/apache/pulsar/client/api/MessageListener.java ## @@ -33,6 +33,8 @@ * * This method will only be called once for each message, unless either application or broker crashes. * + * Implementation should acknowledge messages by calling consumer.acknowledge(msg). Review comment: Hm, 1. who will ack messages then? 2. Fair This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli.
MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli. URL: https://github.com/apache/pulsar/pull/5148#issuecomment-544810227 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5434: Fix couple functions related integration tests
jerrypeng commented on issue #5434: Fix couple functions related integration tests URL: https://github.com/apache/pulsar/pull/5434#issuecomment-544809803 rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5434: Fix couple functions related integration tests
jerrypeng commented on issue #5434: Fix couple functions related integration tests URL: https://github.com/apache/pulsar/pull/5434#issuecomment-544809476 rerun integration tests rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli.
MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli. URL: https://github.com/apache/pulsar/pull/5148#issuecomment-544805122 run java8 tests run integration tests run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544802423 This may be the solution... - If you are starting from fresh, you need to make sure to configure `PF_pulsarFunctionsCluster` correctly before deploying the worker. - If you already started the worker with a wrong configuration, you can try this out: ```bash $ bin/pulsar-admin namespaces set-clusters --cluster= public/functions ``` And then redeploy the worker with a fixed configuration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544802423 This may be the solution... - If you are starting from fresh you need to make sure to configure `PF_pulsarFunctionsCluster` right before deploying the worker. - If you already started the worker with a wrong configuration, you can try this out: ```bash $ bin/pulsar-admin namespaces set-clusters --cluster= public/functions ``` And then redeploy the worker with a fixed configuration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544802423 This may be the solution... - If you are starting from fresh you need to make sure to configure `PF_pulsarFunctionsCluster` correctly before deploying the worker. - If you already started the worker with a wrong configuration, you can try this out: ```bash $ bin/pulsar-admin namespaces set-clusters --cluster= public/functions ``` And then redeploy the worker with a fixed configuration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544706581 I spent more than 12 hours on this... There are 2 problems here: - The functions worker needs to know the cluster name. - The `public/functions` namespace needs to have that cluster set. Here is my solution (edit: this is pointless and does not work if the broker service was already deployed. Do not use this solution!): - `zookeeper.yaml` ``` apiVersion: v1 kind: ConfigMap metadata: name: zookeeper-config data: ... PF_pulsarFunctionsCluster: PF_numFunctionPackageReplicas: "2" ... containers: - name: zookeeper image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/generate-zookeeper-config.sh conf/zookeeper.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/pulsar zookeeper ... ``` - `broker.yaml`: ``` apiVersion: v1 kind: ConfigMap metadata: name: broker-config data: ... functionsWorkerEnabled: "true" PF_pulsarFunctionsCluster: PF_numFunctionPackageReplicas: "2" ... containers: - name: broker image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/broker.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/pulsar broker ... ``` I hope this helps you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure
when set functionsWorkerEnabled=true
URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544799548
@rocketraman you were right. My solution does not make any sense. When I
tried it for the first time I was working on a new cluster. Before that, I
tried many other solutions but none of them work (including setting the `PF_`
env variables in `broker.yaml`). And I think that's the issue here, once the
clusters of `/admin/policies/public/functions` are set, they can't change.
Check this code from
`pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java`:
```java
// getting namespace policy
log.info("Initializing Pulsar Functions namespace...");
try {
try {
admin.namespaces().getPolicies(workerConfig.getPulsarFunctionsNamespace());
} catch (PulsarAdminException e) {
if (e.getStatusCode() == Response.Status.NOT_FOUND.getStatusCode()) {
// if not found than create
try {
Policies policies = new Policies();
policies.retention_policies = new RetentionPolicies(-1, -1);
policies.replication_clusters = new HashSet<>();
policies.replication_clusters.add(workerConfig.getPulsarFunctionsCluster());
admin.namespaces().createNamespace(workerConfig.getPulsarFunctionsNamespace(),
policies);
} catch (PulsarAdminException e1) {
// prevent race condition with other workers starting up
if (e1.getStatusCode() !=
Response.Status.CONFLICT.getStatusCode()) {
log.error("Failed to create namespace {} for pulsar
functions", workerConfig
.getPulsarFunctionsNamespace(), e1);
throw e1;
}
}
} else {
log.error("Failed to get retention policy for pulsar function
namespace {}",
workerConfig.getPulsarFunctionsNamespace(), e);
throw e;
}
}
try {
internalConf = admin.brokers().getInternalConfigurationData();
} catch (PulsarAdminException e) {
log.error("Failed to retrieve broker internal configuration", e);
throw e;
}
} finally {
admin.close();
}
```
It looks like the values are only set if they are not found. What do you
think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] tuteng merged pull request #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
tuteng merged pull request #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated (be6a102 -> c4035d3)
This is an automated email from the ASF dual-hosted git repository. guangning pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from be6a102 [Issue: 3161][Go client] Return Message ID for send (#4811) add c4035d3 Revert "Add default loader for latest pyyaml (#4974)" (#5432) No new revisions were added by this update. Summary of changes: docker/pulsar/scripts/gen-yml-from-env.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [pulsar] MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli.
MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli. URL: https://github.com/apache/pulsar/pull/5148#issuecomment-544795823 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544788820 How did you initialize you cluster metadata? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng edited a comment on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
tuteng edited a comment on issue #5432: Revert "[Issue 4956]Add default loader
for latest pyyaml"
URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544786813
@jerrypeng My guess is that python-yaml is the package that caused this
problem. instead of installing python-pip, I directly use the following command
to package it in
[Dockerfile](https://github.com/apache/pulsar/blob/master/docker/pulsar/Dockerfile).
it can be successful and the integration test can also pass.
```
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python2.7 get-pip.py
RUN pip2.7 install kazoo pyyaml==5.1.2
RUN python3.7 get-pip.py
RUN pip3.7 install kazoo pyyaml==5.1.2
```
```
root@c4b89a6b7021:/pulsar# pip2.7 list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as Python 2.7 won't be maintained after that date. A
future version of pip will drop support for Python 2.7. More details about
Python 2 support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
-
apache-bookkeeper-client 4.9.2
certifi 2019.9.11
chardet 3.0.4
enum34 1.1.6
fastavro 0.22.5
futures 3.3.0
grpcio 1.24.1
idna 2.8
kazoo2.6.1
pip 19.3.1
prometheus-client0.7.1
protobuf 3.10.0
pulsar-client2.5.0
pymmh3 0.0.5
pytz 2019.3
PyYAML 5.1.2
ratelimit2.2.1
requests 2.22.0
setuptools 40.8.0
six 1.12.0
urllib3 1.25.6
wheel0.33.6
root@c4b89a6b7021:/pulsar# pip uninstall PyYAML==5.1.2
Uninstalling PyYAML-5.1.2:
Would remove:
/usr/local/lib/python3.7/dist-packages/PyYAML-5.1.2.dist-info/*
/usr/local/lib/python3.7/dist-packages/yaml/*
Proceed (y/n)? y
Successfully uninstalled PyYAML-5.1.2
root@c4b89a6b7021:/pulsar# pip install PyYAML==5.1.2
Processing
/root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030/PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl
Installing collected packages: PyYAML
Successfully installed PyYAML-5.1.2
```
After the integration test passes, I will try to upgrade to the new version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
tuteng commented on issue #5432: Revert "[Issue 4956]Add default loader for
latest pyyaml"
URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544786813
My guess is that python-yaml is the package that caused this problem.
instead of installing python-pip, I directly use the following command to
package it in
(Dockerfile)[https://github.com/apache/pulsar/blob/master/docker/pulsar/Dockerfile].
it can be successful and the integration test can also pass.
```
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python2.7 get-pip.py
RUN pip2.7 install kazoo pyyaml==5.1.2
RUN python3.7 get-pip.py
RUN pip3.7 install kazoo pyyaml==5.1.2
```
```
root@c4b89a6b7021:/pulsar# pip2.7 list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as Python 2.7 won't be maintained after that date. A
future version of pip will drop support for Python 2.7. More details about
Python 2 support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
-
apache-bookkeeper-client 4.9.2
certifi 2019.9.11
chardet 3.0.4
enum34 1.1.6
fastavro 0.22.5
futures 3.3.0
grpcio 1.24.1
idna 2.8
kazoo2.6.1
pip 19.3.1
prometheus-client0.7.1
protobuf 3.10.0
pulsar-client2.5.0
pymmh3 0.0.5
pytz 2019.3
PyYAML 5.1.2
ratelimit2.2.1
requests 2.22.0
setuptools 40.8.0
six 1.12.0
urllib3 1.25.6
wheel0.33.6
root@c4b89a6b7021:/pulsar# pip uninstall PyYAML==5.1.2
Uninstalling PyYAML-5.1.2:
Would remove:
/usr/local/lib/python3.7/dist-packages/PyYAML-5.1.2.dist-info/*
/usr/local/lib/python3.7/dist-packages/yaml/*
Proceed (y/n)? y
Successfully uninstalled PyYAML-5.1.2
root@c4b89a6b7021:/pulsar# pip install PyYAML==5.1.2
Processing
/root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030/PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl
Installing collected packages: PyYAML
Successfully installed PyYAML-5.1.2
```
After the integration test passes, I will try to upgrade to the new version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] congbobo184 removed a comment on issue #5227: [PIP-44] Separate schema compatibility checker for producer and consumer
congbobo184 removed a comment on issue #5227: [PIP-44] Separate schema compatibility checker for producer and consumer URL: https://github.com/apache/pulsar/pull/5227#issuecomment-542104823 run Integration Tests run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 removed a comment on issue #5227: [PIP-44] Separate schema compatibility checker for producer and consumer
congbobo184 removed a comment on issue #5227: [PIP-44] Separate schema compatibility checker for producer and consumer URL: https://github.com/apache/pulsar/pull/5227#issuecomment-544374157 run Integration Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 removed a comment on issue #4847: Pulsar sql avro support schema version
congbobo184 removed a comment on issue #4847: Pulsar sql avro support schema version URL: https://github.com/apache/pulsar/pull/4847#issuecomment-540982085 run Integration Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 removed a comment on issue #4847: Pulsar sql avro support schema version
congbobo184 removed a comment on issue #4847: Pulsar sql avro support schema version URL: https://github.com/apache/pulsar/pull/4847#issuecomment-540991733 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 removed a comment on issue #4847: Pulsar sql avro support schema version
congbobo184 removed a comment on issue #4847: Pulsar sql avro support schema version URL: https://github.com/apache/pulsar/pull/4847#issuecomment-541027219 run Integration Tests run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
jerrypeng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544780358 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544722192 I'm not adding that value to the ZooKeeper configuration. It is being added to `functions_worker.yml`. Both for the broker and ZooKeeper services. After reading [this](https://github.com/apache/pulsar/issues/2328#issuecomment-411013973) I got the idea that maybe there was an error during the creation of the ZooKeeper data. I tried this workaround and it looks like it's working. Keep in mind that `PF_` env variales are only applied if you add this line: ``` bin/gen-yml-from-env.py conf/functions_worker.yml && ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rocketraman edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
rocketraman edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544778549 My ZK, which is a standalone ZK rather that the one started by `bin/pulsar zookeeper` does have this key `/admin/policies/public/functions`, and it does have the wrong value in it: ``` "replication_clusters":["standalone"] ``` so this means two things: 1) This value is being initialized from somewhere, but it isn't the `functions_worker.yaml` in the ZK container, because my ZK container does not have `functions_worker.yaml` in it, nor does it have any scripts that would initialize this value to anything. 2) It isn't one of the broker containers, because my broker container has this line in `functions_worker.yaml`: ``` # grep pulsarFunctionsCluster conf/functions_worker.yml pulsarFunctionsCluster: local ``` which is the correct value, indicating that `gen-yml-from-env.py` is applying the config to this file correctly, but it does not match what is in ZK. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rocketraman edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
rocketraman edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544778549 My ZK, which is a standalone ZK rather that the one started by `bin/pulsar zookeeper` does have this key `/admin/policies/public/functions`, and it does have the wrong value in it: ``` "replication_clusters":["standalone"] ``` so this means two things: 1) This value is being initialized from somewhere, but it isn't the `functions_worker.yaml` in the ZK container, because my ZK container does not have `functions_worker.yaml` in it, nor does it have any scripts that would initialize this value to anything. 2) It isn't one of the broker containers, because my broker container has this line in `functions_worker.yaml`: ``` # grep pulsarFunctionsCluster conf/functions_worker.yml pulsarFunctionsCluster: local ``` which is the correct value, and does not match what is in ZK. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rocketraman commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
rocketraman commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544778549 My ZK, which is a standalone ZK rather that the one started by `bin/pulsar zookeeper` does have this key `/admin/policies/public/functions`, and it does have the wrong value in it: ``` "replication_clusters":["standalone"] ``` so this means two things: 1) This value is being initialized from somewhere, but it isn't the `functions_worker.yaml` in the ZK container, because my ZK container does not have `functions_worker.yaml` in it. 2) It isn't one of the broker containers, because my broker container has this line in `functions_worker.yaml`: ``` # grep pulsarFunctionsCluster conf/functions_worker.yml pulsarFunctionsCluster: local ``` which is the correct value, and does not match what is in ZK. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rocketraman commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
rocketraman commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544777605 I'd be very surprised if the container running ZooKeeper was responsible for reading `functions_worker.yml`, and initializing any data inside zookeeper from that file. If that is the case, I'd say that's a major bug. Nothing in the Pulsar docs indicates you need to initialize ZK this way, or indeed, even use the ZooKeeper provided by Pulsar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5391: [pulsar-broker] Fix: race condition : Failed to read-more entries on dispatcher
jiazhai commented on issue #5391: [pulsar-broker] Fix: race condition : Failed to read-more entries on dispatcher URL: https://github.com/apache/pulsar/pull/5391#issuecomment-544773699 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0
jiazhai commented on issue #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0 URL: https://github.com/apache/pulsar/pull/5431#issuecomment-544773405 @yjshen to take a look This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5428: Added Documentation to add connectors avaliable
sijie commented on a change in pull request #5428: Added Documentation to add connectors avaliable URL: https://github.com/apache/pulsar/pull/5428#discussion_r337303189 ## File path: distribution/io/src/assemble/README ## @@ -7,4 +7,43 @@ contains: * the connector code plus all the dependencies * META-INF/DEPEDENCIES file with licensing information for all transitive - dependencies \ No newline at end of file + dependencies + + +The source/sink connectors are avaliable for below external interfaces Review comment: what is the motivation of this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #5429: fix: java doc on MessageListener - acknowledgement part
sijie commented on a change in pull request #5429: fix: java doc on MessageListener - acknowledgement part URL: https://github.com/apache/pulsar/pull/5429#discussion_r337302959 ## File path: pulsar-client-1x-base/pulsar-client-1x/src/main/java/org/apache/pulsar/client/api/MessageListener.java ## @@ -33,6 +33,8 @@ * * This method will only be called once for each message, unless either application or broker crashes. * + * Implementation should acknowledge messages by calling consumer.acknowledge(msg). Review comment: I am not sure the changes here. 1) I don't think an implementation need to ack messages. *should* here is a bit misleading. 2) there are many methods to ack a message, ack vs ackCumulative, sync vs async. Also it is recommended to call async methods in the message listener otherwise a sync method can block the listener thread. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng opened a new pull request #5434: Fix couple functions related integration tests
jerrypeng opened a new pull request #5434: Fix couple functions related integration tests URL: https://github.com/apache/pulsar/pull/5434 ### Motivation There are a couple of function related integration tests that were either not executed or broken. FunctionsCLITest.java is broken and not executed PulsarStateTest.java is not executed Tests should pass after #5432 is merged This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng edited a comment on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
jerrypeng edited a comment on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544759460 @tuteng I have tried that but there are problems with installing a more updated version of PyYAML: ``` root@28f8b6cfc440:/pulsar# pip3.7 install pyyaml==5.1.2 Collecting pyyaml==5.1.2 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB) || 266kB 1.1MB/s Building wheels for collected packages: pyyaml Building wheel for pyyaml (setup.py) ... done Created wheel for pyyaml: filename=PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl size=44103 sha256=c9d64793aadb42102719829ba39030b121cd514b3139fcefe9e0584eff3b14ab Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030 Successfully built pyyaml Installing collected packages: pyyaml Found existing installation: PyYAML 3.13 ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. ``` feel free to try to get this the work. I just wanted to do something really fast to unblock all PR getting merged because of failing integration tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
jerrypeng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544759460 @tuteng I have tried that but there are problems with installing a more updated version of PyYAML: ``` root@28f8b6cfc440:/pulsar# pip3.7 install pyyaml==5.1.2 Collecting pyyaml==5.1.2 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB) || 266kB 1.1MB/s Building wheels for collected packages: pyyaml Building wheel for pyyaml (setup.py) ... done Created wheel for pyyaml: filename=PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl size=44103 sha256=c9d64793aadb42102719829ba39030b121cd514b3139fcefe9e0584eff3b14ab Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030 Successfully built pyyaml Installing collected packages: pyyaml Found existing installation: PyYAML 3.13 ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5418: Unable to Configure a JDBC sink
tuteng commented on issue #5418: Unable to Configure a JDBC sink URL: https://github.com/apache/pulsar/issues/5418#issuecomment-544756480 Have you created a schema for your topic, http://pulsar.apache.org/docs/en/io-quickstart/#configure-a-jdbc-sink This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
tuteng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544752697 The pyyaml version here https://github.com/apache/pulsar/blob/master/docker/pulsar/Dockerfile#L48 seems a bit old. Why not try to install the new pyyaml https://github.com/apache/pulsar/blob/branch-2.4/docker/pulsar/Dockerfile#L61? ``` pip2.7 install pyyaml==5.1.2 pip3.7 install pyyaml==5.1.2 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] eolivelli commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.5
eolivelli commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.5 URL: https://github.com/apache/pulsar/pull/5043#issuecomment-544750345 Fyi now 3.5.6 is available :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5361: [proxy] Fix proxy to be able to re-send request body
addisonj commented on issue #5361: [proxy] Fix proxy to be able to re-send request body URL: https://github.com/apache/pulsar/pull/5361#issuecomment-544749702 rerun integration tests rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
jerrypeng commented on issue #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432#issuecomment-544739781 rerun integration tests rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wmccarley opened a new issue #5433: Implementing a Sink to a non thread-safe destination
wmccarley opened a new issue #5433: Implementing a Sink to a non thread-safe destination URL: https://github.com/apache/pulsar/issues/5433 I am implementing a Sink to a proprietary socket-based system. The API provided by the destination system does not supply thread-safe classes and write methods. Is the write method of a class implementing the Pulsar Sink interface expected to be thread safe? If yes, can someone explain the lifecycle of a Sink within a given JVM? - When exactly is open/close called? - Can multiple instances of the same Sink class be instantiated in one JVM? - What happens when I trigger a sink "update" command. The developer of the destination system suggested just use a ThreadLocal instance of the socket thingy. That _might be possible_ contingent upon the answers to the questions above but I don't like it. I am worried about leaking the socket thingys if I stop/start sinks etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng opened a new pull request #5432: Revert "[Issue 4956]Add default loader for latest pyyaml"
jerrypeng opened a new pull request #5432: Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5432 Reverts apache/pulsar#4974 Lets revert this for now because its causing Pulsar Function Workers to fail starting up in our integration tests. ``` 2019-10-21T21:03:32.806743794Z + python -V 2019-10-21T21:03:32.809746507Z Python 2.7.16 2019-10-21T21:03:32.979390472Z Traceback (most recent call last): 2019-10-21T21:03:32.979437479Z File "bin/gen-yml-from-env.py", line 49, in 2019-10-21T21:03:32.979445879Z conf = yaml.load(open(conf_filename), Loader=yaml.FullLoader) 2019-10-21T21:03:32.979452110Z AttributeError: 'module' object has no attribute 'FullLoader' ``` This seems to be a problem for python 2.7. Once we move are images to use python 3.7 by default we can bring this change back. Full log for function worker starting up: ``` 2019-10-21T21:03:32.804486087Z ++ echo 2019-10-21T21:03:32.804626995Z ++ tr '[:upper:]' '[:lower:]' 2019-10-21T21:03:32.806562619Z + FLAG= 2019-10-21T21:03:32.806632205Z + '[' '' = true ']' 2019-10-21T21:03:32.806672570Z + echo 'Python Version ...' 2019-10-21T21:03:32.806723034Z Python Version ... 2019-10-21T21:03:32.806743794Z + python -V 2019-10-21T21:03:32.809746507Z Python 2.7.16 2019-10-21T21:03:32.979390472Z Traceback (most recent call last): 2019-10-21T21:03:32.979437479Z File "bin/gen-yml-from-env.py", line 49, in 2019-10-21T21:03:32.979445879Z conf = yaml.load(open(conf_filename), Loader=yaml.FullLoader) 2019-10-21T21:03:32.979452110Z AttributeError: 'module' object has no attribute 'FullLoader' 2019-10-21T21:03:33.378413384Z 2019-10-21 21:03:33,377 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2019-10-21T21:03:33.378506024Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/bookie.conf" during parsing 2019-10-21T21:03:33.378518130Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/broker.conf" during parsing 2019-10-21T21:03:33.378527184Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/functions_worker.conf" during parsing 2019-10-21T21:03:33.378535634Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/global-zk.conf" during parsing 2019-10-21T21:03:33.378544050Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/local-zk.conf" during parsing 2019-10-21T21:03:33.378552477Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/presto_worker.conf" during parsing 2019-10-21T21:03:33.378574774Z 2019-10-21 21:03:33,378 INFO Included extra file "/etc/supervisord/conf.d/proxy.conf" during parsing 2019-10-21T21:03:33.390787507Z 2019-10-21 21:03:33,390 INFO RPC interface 'supervisor' initialized 2019-10-21T21:03:33.390874023Z 2019-10-21 21:03:33,390 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-10-21T21:03:33.391123089Z 2019-10-21 21:03:33,390 INFO supervisord started with pid 1 2019-10-21T21:03:34.395601299Z 2019-10-21 21:03:34,394 INFO spawned: 'functions-worker' with pid 21 2019-10-21T21:03:35.397666715Z 2019-10-21 21:03:35,397 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-10-21T21:04:40.473995238Z 2019-10-21 21:04:40,473 INFO exited: functions-worker (exit status 255; not expected) 2019-10-21T21:04:41.478742777Z 2019-10-21 21:04:41,478 INFO spawned: 'functions-worker' with pid 215 2019-10-21T21:04:42.480653712Z 2019-10-21 21:04:42,480 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-10-21T21:05:47.522984806Z 2019-10-21 21:05:47,522 INFO exited: functions-worker (exit status 255; not expected) 2019-10-21T21:05:48.527747430Z 2019-10-21 21:05:48,527 INFO spawned: 'functions-worker' with pid 409 2019-10-21T21:05:49.529612725Z 2019-10-21 21:05:49,529 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-10-21T21:06:54.316756347Z 2019-10-21 21:06:54,316 INFO exited: functions-worker (exit status 255; not expected) 2019-10-21T21:06:55.321097684Z 2019-10-21 21:06:55,320 INFO spawned: 'functions-worker' with pid 603 2019-10-21T21:06:56.323222911Z 2019-10-21 21:06:56,322 INFO success: functions-worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-10-21T21:08:01.154232192Z 2019-10-21 21:08:01,153 INFO exited: functions-worker (exit status 255; not expected) 2019-10-21T21:08:02.158816137Z 2019-10-21 21:08:02,158 INFO spawned: 'functions-worker' with pid 797
[pulsar] branch revert-5335-python371 created (now efbbd13)
This is an automated email from the ASF dual-hosted git repository. jerrypeng pushed a change to branch revert-5335-python371 in repository https://gitbox.apache.org/repos/asf/pulsar.git. at efbbd13 Revert "Build python 37 artifacts and use them for the docker image (#5335)" This branch includes the following new commits: new efbbd13 Revert "Build python 37 artifacts and use them for the docker image (#5335)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[pulsar] 01/01: Revert "Build python 37 artifacts and use them for the docker image (#5335)"
This is an automated email from the ASF dual-hosted git repository.
jerrypeng pushed a commit to branch revert-5335-python371
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit efbbd139ff9707304e8f95526ca5ae7c4ee85eab
Author: Boyang Jerry Peng
AuthorDate: Mon Oct 21 15:21:21 2019 -0700
Revert "Build python 37 artifacts and use them for the docker image (#5335)"
This reverts commit e8adb0aa5a524c1eddda7607ab979192f293cfc2.
---
docker/pulsar/Dockerfile | 16 +++-
docker/pulsar/pom.xml| 4 ++--
...l-pulsar-client-37.sh => install-pulsar-client-35.sh} | 4 ++--
3 files changed, 15 insertions(+), 9 deletions(-)
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index d4b8646..de4470b 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -32,7 +32,7 @@ COPY scripts/pulsar-zookeeper-ruok.sh /pulsar/bin
COPY scripts/watch-znode.py /pulsar/bin
COPY scripts/set_python_version.sh /pulsar/bin
COPY scripts/install-pulsar-client-27.sh /pulsar/bin
-COPY scripts/install-pulsar-client-37.sh /pulsar/bin
+COPY scripts/install-pulsar-client-35.sh /pulsar/bin
### Create 2nd stage from OpenJDK image
@@ -44,21 +44,27 @@ COPY --from=pulsar /pulsar /pulsar
# Install some utilities
RUN apt-get update \
&& apt-get install -y netcat dnsutils less procps iputils-ping \
- python2.7 python-setuptools python-yaml python-kazoo \
- python3.7 python3-setuptools python3-yaml python3-kazoo \
+ python2.7 python-setuptools \
+ python3-setuptools \
libreadline-gplv2-dev libncursesw5-dev libssl-dev
libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev \
curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+# Install python3.5.6
+RUN curl -O -L https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz
+RUN tar -zxvf Python-3.5.6.tgz
+RUN cd Python-3.5.6 && ./configure --enable-optimizations && make altinstall
+
RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
RUN python2.7 get-pip.py
-RUN python3.7 get-pip.py
+RUN pip install kazoo pyyaml
+RUN python3.5 get-pip.py
ADD target/python-client/ /pulsar/pulsar-client
ADD target/cpp-client/ /pulsar/cpp-client
RUN /pulsar/bin/install-pulsar-client-27.sh
-RUN /pulsar/bin/install-pulsar-client-37.sh
+RUN /pulsar/bin/install-pulsar-client-35.sh
RUN echo networkaddress.cache.ttl=1 >>
$JAVA_HOME/jre/lib/security/java.security
RUN apt-get update \
&& apt install -y /pulsar/cpp-client/*.deb \
diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
index 4af9a77..cc0d783 100644
--- a/docker/pulsar/pom.xml
+++ b/docker/pulsar/pom.xml
@@ -80,8 +80,8 @@
${project.basedir}/target
${project.basedir}/../../pulsar-client-cpp/docker/build-wheels.sh
-
-3.7 cp37-cp37m
+
+3.5 cp35-cp35m
diff --git a/docker/pulsar/scripts/install-pulsar-client-37.sh
b/docker/pulsar/scripts/install-pulsar-client-35.sh
similarity index 88%
rename from docker/pulsar/scripts/install-pulsar-client-37.sh
rename to docker/pulsar/scripts/install-pulsar-client-35.sh
index 9d3714c..058eefd 100755
--- a/docker/pulsar/scripts/install-pulsar-client-37.sh
+++ b/docker/pulsar/scripts/install-pulsar-client-35.sh
@@ -20,5 +20,5 @@
set -x
-WHEEL_FILE=$(ls /pulsar/pulsar-client | grep cp37)
-pip3.7 install /pulsar/pulsar-client/${WHEEL_FILE}
+WHEEL_FILE=$(ls /pulsar/pulsar-client | grep cp35)
+pip3.5 install /pulsar/pulsar-client/${WHEEL_FILE}
[GitHub] [pulsar] MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli.
MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli. URL: https://github.com/apache/pulsar/pull/5148#issuecomment-544726947 run java8 tests run integration tests run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544722192 I'm not adding that value to the ZooKeeper configuration. It is being added to `functions_worker.yml`. Both for the broker and ZooKeeper services. After reading [this](https://github.com/apache/pulsar/issues/2328#issuecomment-411013973) I got the idea that maybe there was an error during the creation of the ZooKeeper data. I tried this workaround and it looks like it's working. Keep in mind that`PF_` env variales are only applied if you add this line: ``` bin/gen-yml-from-env.py conf/functions_worker.yml && ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544722192 I'm not adding that value to the ZooKeeper configuration. It is being added to `functions_worker.yml`. Both for the broker and ZooKeeper services. After reading (https://github.com/apache/pulsar/issues/2328#issuecomment-411013973)[this] I got the idea that maybe there was an error during the creation of the ZooKeeper data. I tried this workaround and it looks like it's working. Keep in mind that`PF_` env variales are only applied if you add this line: ``` bin/gen-yml-from-env.py conf/functions_worker.yml && ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rocketraman commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
rocketraman commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544714109 @stupidusername None of the zookeeper conf files have anything related to function worker cluster names in them (nor would I expect them to). I'm not certain why setting `PF_pulsarFunctionsCluster` for the zookeeper container would solve the issue. I don't even use the zookeeper that comes with Pulsar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername edited a comment on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544706581 I spent more than 12 hours on this... There are 2 problems here: - The functions worker needs to know the cluster name. - The `public/functions` namespace needs to have that cluster set. The list of `public/functions` clusters is set when the ZooKeeper service is created. Here is my solution: - `zookeeper.yaml` ``` apiVersion: v1 kind: ConfigMap metadata: name: zookeeper-config data: ... PF_pulsarFunctionsCluster: PF_numFunctionPackageReplicas: "2" ... containers: - name: zookeeper image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/generate-zookeeper-config.sh conf/zookeeper.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/pulsar zookeeper ... ``` - `broker.yaml`: ``` apiVersion: v1 kind: ConfigMap metadata: name: broker-config data: ... functionsWorkerEnabled: "true" PF_pulsarFunctionsCluster: PF_numFunctionPackageReplicas: "2" ... containers: - name: broker image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/broker.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/pulsar broker ... ``` I hope this helps you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true
stupidusername commented on issue #5325: Pulsar on k8s: broker start failure when set functionsWorkerEnabled=true URL: https://github.com/apache/pulsar/issues/5325#issuecomment-544706581 I spent more than 12 hours on this... There are 2 problems here: - The functions worker needs to know the cluster name. - The `public/functions` namespace needs to have that cluster set. The list of `public/functions` clusters is set when the ZooKeeper service is created. Here is my solution: - `zookeeper.yaml` ``` apiVersion: v1 kind: ConfigMap metadata: name: zookeeper-config data: ... PF_pulsarFunctionsCluster: rings PF_numFunctionPackageReplicas: "2" ... containers: - name: zookeeper image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/generate-zookeeper-config.sh conf/zookeeper.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/pulsar zookeeper ... ``` - `broker.yaml`: ``` apiVersion: v1 kind: ConfigMap metadata: name: broker-config data: ... functionsWorkerEnabled: "true" PF_pulsarFunctionsCluster: rings PF_numFunctionPackageReplicas: "2" ... containers: - name: broker image: apachepulsar/pulsar:2.4.1 command: ["sh", "-c"] args: - > bin/apply-config-from-env.py conf/broker.conf && bin/gen-yml-from-env.py conf/functions_worker.yml && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/pulsar broker ... ``` I hope this helps you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.5
addisonj commented on issue #5043: [zookeeper] bump zookeeper to version 3.5.5 URL: https://github.com/apache/pulsar/pull/5043#issuecomment-544697045 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5400: [functions] Allow functions to pass runtime specific options
addisonj commented on issue #5400: [functions] Allow functions to pass runtime specific options URL: https://github.com/apache/pulsar/pull/5400#issuecomment-544696751 okay @jerrypeng I changed that, will see if I can get the tests happy, let me know if you have any other feedback! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] fhalim opened a new pull request #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0
fhalim opened a new pull request #5431: [Issue 5430][pulsar-flink] Flink connector sink should implement consumeDataStream for 1.9.0 URL: https://github.com/apache/pulsar/pull/5431 Fixes #5430 ### Motivation Should be able to write to Pulsar under Flink 1.9.0 application ### Modifications Added an implementation for `StreamTableSink#consumeDataStream(DataStream)` in the sink to make it compatible with Flink 1.9.0 blink. ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change added tests and can be verified as follows: * Added unit test that verifies that a DataStreamSink is returned and is not null ### Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (yes / no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) - If a feature is not applicable for documentation, explain why? (Covered by Flink APIs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] fhalim opened a new issue #5430: Flink connector sink doesn't work on blink SQL context
fhalim opened a new issue #5430: Flink connector sink doesn't work on blink SQL context URL: https://github.com/apache/pulsar/issues/5430 **Describe the bug** A clear and concise description of what the bug is. **To Reproduce** Steps to reproduce the behavior: 1. Create a Flink SQL application in Flink 1.9.0, using the blink optimizer, and register a PulsarJsonTableSink (for instance as described in https://flink.apache.org/2019/05/03/pulsar-flink.html). 2. Run the application. **Expected behavior** Sink should be available for writing. **Actual output** ``` Exception in thread "main" org.apache.flink.table.api.TableException: The StreamTableSink#consumeDataStream(DataStream) must be implemented and return the sink transformation DataStreamSink. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5398: [functions] Distribute the CA for KubernetesSecretsTokenAuthProvider
addisonj commented on issue #5398: [functions] Distribute the CA for KubernetesSecretsTokenAuthProvider URL: https://github.com/apache/pulsar/pull/5398#issuecomment-544692130 rerun java8 tests rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jm7647 commented on issue #5418: Unable to Configure a JDBC sink
jm7647 commented on issue #5418: Unable to Configure a JDBC sink
URL: https://github.com/apache/pulsar/issues/5418#issuecomment-544664056
I looked directly at sink log and it seems to be some schema issue:
19:09:46.263 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.PulsarClientImpl - Configuring schema for topic
pulsar-mysql-jdbc-sink-topic : {
"name": "public/default/pulsar-mysql-jdbc-sink-topic",
"schema": "",
"type": "NONE",
"properties": {}
}
19:09:46.264 [public/default/pulsar-mysql-jdbc-sink-0] ERROR
org.apache.pulsar.functions.instance.JavaInstanceRunnable - Source open
produced uncaught exception:
java.util.concurrent.CompletionException: java.lang.RuntimeException:
Currently auto consume only works for topics with avro or json schemas
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] addisonj commented on issue #5398: [functions] Distribute the CA for KubernetesSecretsTokenAuthProvider
addisonj commented on issue #5398: [functions] Distribute the CA for KubernetesSecretsTokenAuthProvider URL: https://github.com/apache/pulsar/pull/5398#issuecomment-544660392 rerun java8 tests rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng closed pull request #5427: DON'T MERGE TESTING IF THIS SOLVES INTEGRATION TESTS ISSUE Revert "[Issue 4956]Add default loader for latest pyyaml"
jerrypeng closed pull request #5427: DON'T MERGE TESTING IF THIS SOLVES INTEGRATION TESTS ISSUE Revert "[Issue 4956]Add default loader for latest pyyaml" URL: https://github.com/apache/pulsar/pull/5427 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli.
MarvinCai commented on issue #5148: Validate topic name before creating partition/non partition topic via admin cli. URL: https://github.com/apache/pulsar/pull/5148#issuecomment-544636966 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jm7647 commented on issue #5418: Unable to Configure a JDBC sink
jm7647 commented on issue #5418: Unable to Configure a JDBC sink
URL: https://github.com/apache/pulsar/issues/5418#issuecomment-544632418
I'm using now useSSL=false. Notice SSL warning is gone but jdbc sink
connector continue to fail:
17:50:54.652 [function-timer-thread-86-1] INFO
org.apache.pulsar.functions.runtime.ProcessRuntime - ProcessBuilder starting
the process with args java -cp
/pulsar/instances/java-instance.jar:/pulsar/instances/deps/*
-Dpulsar.functions.extra.dependencies.dir=/pulsar/instances/deps
-Dpulsar.functions.instance.classpath=/pulsar/conf:::/pulsar/lib/*:
-Dlog4j.configurationFile=java_instance_log4j2.xml
-Dpulsar.function.log.dir=/pulsar/logs/functions/public/default/pulsar-mysql-jdbc-sink
-Dpulsar.function.log.file=pulsar-mysql-jdbc-sink-0 -Xmx1073741824
org.apache.pulsar.functions.instance.JavaInstanceMain --jar
/tmp/pulsar_functions/public/default/pulsar-mysql-jdbc-sink/0/pulsar-io-jdbc-2.4.1.nar
--instance_id 0 --function_id be5a9234-2964-4b96-8899-b92e045c9ef0
--function_version 72757313-5529-44d0-91ff-05f7d1f21860 --function_details
'{"tenant":"public","namespace":"default","name":"pulsar-mysql-jdbc-sink","className":"org.apache.pulsar.functions.api.utils.IdentityFunction","autoAck":true,"parallelism":1,"source":{"typeClassName":"org.apache.pulsar.client.api.schema.GenericRecord","inputSpecs":{"pulsar-mysql-jdbc-sink-topic":{}},"cleanupSubscription":true},"sink":{"className":"org.apache.pulsar.io.jdbc.JdbcAutoSchemaSink","configs":"{\"userName\":\"root\",\"password\":\"jdbc\",\"jdbcUrl\":\"jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink\",\"tableName\":\"pulsar_mysql_jdbc_sink\",\"useSSL\":\"false\"}","typeClassName":"org.apache.pulsar.client.api.schema.GenericRecord"},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"componentType":"SINK"}'
--pulsar_serviceurl pulsar://mtn21cvaas04.mt.att.com:6650
--max_buffered_tuples 1024 --port 38206 --metrics_port 43625
--state_storage_serviceurl bk://127.0.0.1:4181 --expected_healthcheck_interval
30 --secrets_provider
org.apache.pulsar.functions.secretsprovider.ClearTextSecretsProvider
--cluster_name standalone
17:50:54.656 [function-timer-thread-86-1] INFO
org.apache.pulsar.functions.runtime.ProcessRuntime - Started process
successfully
Using function root classloader: sun.misc.Launcher$AppClassLoader@7852e922
Using function instance classloader: java.net.URLClassLoader@5c647e05
Starting function instance...
17:51:02.473 [pulsar-web-57-8] INFO org.eclipse.jetty.server.RequestLog -
135.25.67.33 - - [21/Oct/2019:17:51:02 +] "GET
/admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 837 "-"
"Pulsar-Java-v2.4.1" 2
17:51:24.651 [function-timer-thread-86-1] ERROR
org.apache.pulsar.functions.runtime.ProcessRuntime - Extracted Process death
exception
java.lang.RuntimeException:
at
org.apache.pulsar.functions.runtime.ProcessRuntime.tryExtractingDeathException(ProcessRuntime.java:380)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
at
org.apache.pulsar.functions.runtime.ProcessRuntime.isAlive(ProcessRuntime.java:367)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
at
org.apache.pulsar.functions.runtime.RuntimeSpawner.lambda$start$0(RuntimeSpawner.java:88)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_212]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_212]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_212]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
17:51:24.652 [function-timer-thread-86-1] ERROR
org.apache.pulsar.functions.runtime.RuntimeSpawner -
public/default/pulsar-mysql-jdbc-sink-java.lang.RuntimeException: Function
Container is dead with exception.. restarting
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #5391: [pulsar-broker] Fix: race condition : Failed to read-more entries on dispatcher
rdhabalia commented on issue #5391: [pulsar-broker] Fix: race condition : Failed to read-more entries on dispatcher URL: https://github.com/apache/pulsar/pull/5391#issuecomment-544620963 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] adeora commented on issue #3671: pulsar-admin topics list public/default fails
adeora commented on issue #3671: pulsar-admin topics list public/default fails URL: https://github.com/apache/pulsar/issues/3671#issuecomment-544575808 We are seeing this issue in a Pulsar cluster with version 2.4.1, see issue #5417 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers
wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers URL: https://github.com/apache/pulsar/pull/5389#issuecomment-544572069 > java.lang.IllegalArgumentException: No pulsar-functions-worker is alive at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at org.apache.pulsar.tests.integration.topologies.PulsarCluster.getAnyContainer(PulsarCluster.java:461) at org.apache.pulsar.tests.integration.topologies.PulsarCluster.getAnyWorker(PulsarCluster.java:446) at org.apache.pulsar.tests.integration.functions.FunctionsCLITest.checkUpload(FunctionsCLITest.java:55) at org.apache.pulsar.tests.integration.functions.FunctionsCLITest.checkDownload(FunctionsCLITest.java:63) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54) at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #5418: Unable to Configure a JDBC sink
tuteng commented on issue #5418: Unable to Configure a JDBC sink URL: https://github.com/apache/pulsar/issues/5418#issuecomment-544555715 ``` WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. ``` Try adding useSSL parameter to the link. ``` jdbc:mysql://127.0.0.1:3306/test?useUnicode=true=utf-8=false ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jm7647 commented on issue #5418: Unable to Configure a JDBC sink
jm7647 commented on issue #5418: Unable to Configure a JDBC sink
URL: https://github.com/apache/pulsar/issues/5418#issuecomment-544538500
I followed your suggestion and download nar package. Then I followed
tutorial:
https://pulsar.apache.org/docs/en/io-quickstart/#connect-pulsar-to-mysql
Still jdbc connector is not working according to pulsar logs:
14:20:14.877 [function-timer-thread-86-1] INFO
org.apache.pulsar.functions.runtime.ProcessRuntime - ProcessBuilder starting
the process with args java -cp
/pulsar/instances/java-instance.jar:/pulsar/instances/deps/*
-Dpulsar.functions.extra.dependencies.dir=/pulsar/instances/deps
-Dpulsar.functions.instance.classpath=/pulsar/conf:::/pulsar/lib/*:
-Dlog4j.configurationFile=java_instance_log4j2.xml
-Dpulsar.function.log.dir=/pulsar/logs/functions/public/default/pulsar-mysql-jdbc-sink
-Dpulsar.function.log.file=pulsar-mysql-jdbc-sink-0 -Xmx1073741824
org.apache.pulsar.functions.instance.JavaInstanceMain --jar
/tmp/pulsar_functions/public/default/pulsar-mysql-jdbc-sink/0/pulsar-io-jdbc-2.4.1.nar
--instance_id 0 --function_id 993eb03e-f517-45ec-bb99-e4e29a58ae43
--function_version 3491094b-20c9-4117-b80e-ef8e686ebcd1 --function_details
'{"tenant":"public","namespace":"default","name":"pulsar-mysql-jdbc-sink","className":"org.apache.pulsar.functions.api.utils.IdentityFunction","autoAck":true,"parallelism":1,"source":{"typeClassName":"org.apache.pulsar.client.api.schema.GenericRecord","inputSpecs":{"pulsar-mysql-jdbc-sink-topic":{}},"cleanupSubscription":true},"sink":{"className":"org.apache.pulsar.io.jdbc.JdbcAutoSchemaSink","configs":"{\"userName\":\"root\",\"password\":\"jdbc\",\"jdbcUrl\":\"jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink\",\"tableName\":\"pulsar_mysql_jdbc_sink\"}","typeClassName":"org.apache.pulsar.client.api.schema.GenericRecord"},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"componentType":"SINK"}'
--pulsar_serviceurl pulsar://mtn21cvaas04.mt.att.com:6650
--max_buffered_tuples 1024 --port 34596 --metrics_port 46333
--state_storage_serviceurl bk://127.0.0.1:4181 --expected_healthcheck_interval
30 --secrets_provider
org.apache.pulsar.functions.secretsprovider.ClearTextSecretsProvider
--cluster_name standalone
14:20:14.880 [function-timer-thread-86-1] INFO
org.apache.pulsar.functions.runtime.ProcessRuntime - Started process
successfully
Using function root classloader: sun.misc.Launcher$AppClassLoader@7852e922
Using function instance classloader: java.net.URLClassLoader@5c647e05
Starting function instance...
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver
class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via
the SPI and manual loading of the driver class is generally unnecessary.
Mon Oct 21 14:20:17 UTC 2019 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL 5.5.45+,
5.6.26+ and 5.7.6+ requirements SSL connection must be established by default
if explicit option isn't set. For compliance with existing applications not
using SSL the verifyServerCertificate property is set to 'false'. You need
either to explicitly disable SSL by setting useSSL=false, or set useSSL=true
and provide truststore for server certificate verification.
14:20:18.482 [pulsar-io-50-4] INFO
org.apache.pulsar.broker.service.ServerCnx - New connection from
/135.25.67.33:38616
14:20:18.556 [pulsar-io-50-4] INFO
org.apache.pulsar.broker.service.ServerCnx - Closed connection from
/135.25.67.33:38616
14:20:29.003 [pulsar-web-57-8] INFO org.eclipse.jetty.server.RequestLog -
135.25.67.33 - - [21/Oct/2019:14:20:28 +] "GET
/admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 837 "-"
"Pulsar-Java-v2.4.1" 4
14:20:44.876 [function-timer-thread-86-1] ERROR
org.apache.pulsar.functions.runtime.ProcessRuntime - Extracted Process death
exception
java.lang.RuntimeException:
at
org.apache.pulsar.functions.runtime.ProcessRuntime.tryExtractingDeathException(ProcessRuntime.java:380)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
at
org.apache.pulsar.functions.runtime.ProcessRuntime.isAlive(ProcessRuntime.java:367)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
at
org.apache.pulsar.functions.runtime.RuntimeSpawner.lambda$start$0(RuntimeSpawner.java:88)
~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_212]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_212]
at
[GitHub] [pulsar] wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers
wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers URL: https://github.com/apache/pulsar/pull/5389#issuecomment-544503266 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5405: [Docs] Add functions-worker commands for pulsar admin CLI
wolfstudy commented on issue #5405: [Docs] Add functions-worker commands for pulsar admin CLI URL: https://github.com/apache/pulsar/pull/5405#issuecomment-544503199 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] liketic commented on issue #5423: [Test] Migrate deprecated Matchers to ArgumentMatchers
liketic commented on issue #5423: [Test] Migrate deprecated Matchers to ArgumentMatchers URL: https://github.com/apache/pulsar/pull/5423#issuecomment-544488923 run java8 tests run integration tests run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] liketic commented on issue #5424: Replace foreach with System.arraycopy
liketic commented on issue #5424: Replace foreach with System.arraycopy URL: https://github.com/apache/pulsar/pull/5424#issuecomment-544488656 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers
wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers URL: https://github.com/apache/pulsar/pull/5389#issuecomment-55038 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5405: [Docs] Add functions-worker commands for pulsar admin CLI
wolfstudy commented on issue #5405: [Docs] Add functions-worker commands for pulsar admin CLI URL: https://github.com/apache/pulsar/pull/5405#issuecomment-54866 > org.apache.pulsar.tests.integration.functions.PulsarStateTest.pulsar-standalone-suite run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar-client-go] wolfstudy commented on issue #69: [ISSUE #68][feat]add Option config for init (#68)
wolfstudy commented on issue #69: [ISSUE #68][feat]add Option config for init (#68) URL: https://github.com/apache/pulsar-client-go/pull/69#issuecomment-54049 @sijie PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4955: [PIP-39] Introduce system topic and topic policies service
codelipenghui commented on issue #4955: [PIP-39] Introduce system topic and topic policies service URL: https://github.com/apache/pulsar/pull/4955#issuecomment-544432489 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Guchman opened a new pull request #5429: fix: java doc on MessageListener - acknowledgement part
Guchman opened a new pull request #5429: fix: java doc on MessageListener - acknowledgement part URL: https://github.com/apache/pulsar/pull/5429 JavaDoc fix MessageListener's message ack responsibility was not documented This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4955: [PIP-39] Introduce system topic and topic policies service
codelipenghui commented on issue #4955: [PIP-39] Introduce system topic and topic policies service URL: https://github.com/apache/pulsar/pull/4955#issuecomment-544372543 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5405: [Docs] Add functions-worker commands for pulsar admin CLI
wolfstudy commented on issue #5405: [Docs] Add functions-worker commands for pulsar admin CLI URL: https://github.com/apache/pulsar/pull/5405#issuecomment-544371029 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers
wolfstudy commented on issue #5389: [Docs] Add delete-dynamic-config command for brokers URL: https://github.com/apache/pulsar/pull/5389#issuecomment-544370534 run integration tests run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5413: [pulsar-client-cpp] Fix Dockerfile
massakam commented on issue #5413: [pulsar-client-cpp] Fix Dockerfile URL: https://github.com/apache/pulsar/pull/5413#issuecomment-544365566 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #5408: [pulsar-broker] Fix bug that namespace policies does not take effect due to NPE
massakam commented on issue #5408: [pulsar-broker] Fix bug that namespace policies does not take effect due to NPE URL: https://github.com/apache/pulsar/pull/5408#issuecomment-544362743 run Integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
