[pulsar-helm-chart] branch master updated: Update grafana & prometheus docker images (#8)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git The following commit(s) were added to refs/heads/master by this push: new 4009c04 Update grafana & prometheus docker images (#8) 4009c04 is described below commit 4009c04811f97bba9db524ed2104695e22f9a097 Author: Oscar Espitia AuthorDate: Wed Apr 29 04:25:32 2020 -0400 Update grafana & prometheus docker images (#8) ### Motivation As seen below, there is a fix for one of the Grafana dashboards that are currently broken in this project (available since version 0.0.5): - [The Pulsar-topics metrics can't load in Grafana](https://github.com/streamnative/charts/issues/49) Additionally, upgrading Prometheus to the latest version improves performance as seen here: https://prometheus.io/blog/2017/11/08/announcing-prometheus-2-0 ### Modifications Bring Docker images to their most up-to-date version (streamnative/apache-pulsar-grafana-dashboard-k8s:0.0.6, prom/prometheus:v2.17.2) to fix the following issues: - https://github.com/streamnative/charts/issues/49 <- fixes Pulsar-topics metrics failure to load - https://github.com/prometheus/prometheus/pull/2859 <- prevent escalation vulnerabilities by defaulting to the ```nobody``` user **Note**: upgrading to the latest version of Prometheus (currently v2.17.2) caused the pod to fail with the following error: ```open /prometheus/queries.active: permission denied```. In order to fix this issue I followed the instructions from these 2 comments: - [Permission denied UID/GID solution](https://github.com/prometheus/prometheus/issues/5976#issuecomment-532942295) - [Unable to create mmap-ed active query log securityContext fix](https://github.com/aws/eks-charts/issues/21#issuecomment-607031756) ### Verifying this change - [x] Make sure that the change passes the CI checks. --- charts/pulsar/templates/prometheus-deployment.yaml | 5 + charts/pulsar/values.yaml | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/charts/pulsar/templates/prometheus-deployment.yaml b/charts/pulsar/templates/prometheus-deployment.yaml index 75bae72..b51fd7a 100644 --- a/charts/pulsar/templates/prometheus-deployment.yaml +++ b/charts/pulsar/templates/prometheus-deployment.yaml @@ -68,6 +68,11 @@ spec: mountPath: /etc/prometheus - name: "{{ template "pulsar.fullname" . }}-{{ .Values.prometheus.component }}-{{ .Values.prometheus.volumes.data.name }}" mountPath: /prometheus + securityContext: +fsGroup: 65534 +runAsGroup: 65534 +runAsNonRoot: true +runAsUser: 65534 volumes: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.prometheus.component }}-config" configMap: diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml index 5c1582d..a441690 100644 --- a/charts/pulsar/values.yaml +++ b/charts/pulsar/values.yaml @@ -149,11 +149,11 @@ images: tag: 2.5.0 prometheus: repository: prom/prometheus -tag: v1.6.3 +tag: v2.17.2 pullPolicy: IfNotPresent grafana: repository: streamnative/apache-pulsar-grafana-dashboard-k8s -tag: 0.0.4 +tag: 0.0.6 pullPolicy: IfNotPresent pulsar_manager: repository: apachepulsar/pulsar-manager
[pulsar-helm-chart] branch master updated: Decouple credentials from key secrets generation (#7)
This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git The following commit(s) were added to refs/heads/master by this push: new 06652d7 Decouple credentials from key secrets generation (#7) 06652d7 is described below commit 06652d7e8b118d13345cd076bde6dd8ca7a0b058 Author: Oscar Espitia AuthorDate: Wed Apr 29 04:27:16 2020 -0400 Decouple credentials from key secrets generation (#7) Fixes #6 ### Motivation As suggested here: https://pulsar.apache.org/docs/en/helm-deploy/#prepare-the-helm-release. The ```prepare_helm_release.sh``` script provided with this Helm chart can create a secret credentials resource and > The username and password are used for logging into Grafana dashboard and Pulsar Manager. However, I haven't been able to make use of such a feature for a number of reasons: 1. This secret doesn't seem to affect the ```pulsar-manager-deployment.yaml``` definition. Instead, the ```./templates/pulsar-manager-admin-secret.yaml``` seems to be the one providing the credentials for the pulsar manager (UI) (with the added possibility to overwrite via values.yaml at ```pulsar_manager.admin.user/password```). 2. Using the Pulsar chart as a dependency for an umbrella chart (this is currently my use case), will bring extra hassle that will make it very hard to have all resources follow the same naming structure, thus causing some resources to never be deployed successfully e.g.: ```./templates/grafana-deployment.yaml``` will complain that it couldn't find the secret created by the bash script. Attempting to fix this issue via the ```-k``` flag passed to the script will cause the JWT secret t [...] ### Modifications Decouple grafana credentials from pulsar manager via a new secret resource named ```./charts/pulsar/templates/grafana-admin-secret.yaml```. Add credentials overriding via values.yaml in the same way as pulsar_manager (grafana.admin.user/password) & delete secret resource manipulation from bash scripts (cleaup_helm_release.sh & prepare_helm_release.sh) ### Verifying this change - [x] Make sure that the change passes the CI checks. --- charts/pulsar/templates/grafana-admin-secret.yaml | 35 +++ charts/pulsar/templates/grafana-deployment.yaml | 8 +++--- charts/pulsar/values.yaml | 3 ++ scripts/pulsar/cleanup_helm_release.sh| 3 -- scripts/pulsar/prepare_helm_release.sh| 27 - 5 files changed, 42 insertions(+), 34 deletions(-) diff --git a/charts/pulsar/templates/grafana-admin-secret.yaml b/charts/pulsar/templates/grafana-admin-secret.yaml new file mode 100644 index 000..cca87b1 --- /dev/null +++ b/charts/pulsar/templates/grafana-admin-secret.yaml @@ -0,0 +1,35 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +{{- if or .Values.monitoring.grafana .Values.extra.monitoring }} +apiVersion: v1 +kind: Secret +metadata: + name: "{{ template "pulsar.fullname" . }}-{{ .Values.grafana.component }}-secret" + namespace: {{ .Values.namespace }} + labels: +{{- include "pulsar.standardLabels" . | nindent 4 }} +component: {{ .Values.grafana.component }} +type: Opaque +stringData: + {{- if .Values.grafana.admin}} + GRAFANA_ADMIN_PASSWORD: {{ .Values.grafana.admin.password | default "pulsar" }} + GRAFANA_ADMIN_USER: {{ .Values.grafana.admin.user | default "pulsar" }} + {{- end }} +{{- end }} diff --git a/charts/pulsar/templates/grafana-deployment.yaml b/charts/pulsar/templates/grafana-deployment.yaml index 675d897..434794f 100644 --- a/charts/pulsar/templates/grafana-deployment.yaml +++ b/charts/pulsar/templates/grafana-deployment.yaml @@ -74,11 +74,11 @@ spec: - name: GRAFANA_ADMIN_USER valueFrom: secretKeyRef: - name: "{{ template "pulsar.fullname" . }}-admin-secret" - key: USER + name: "{{ template "pulsar.fullname" . }}-{{ .Values.grafana.component }}-secret" + key: GRAFANA_ADMIN_USER - name: GRAFANA_AD
[GitHub] [pulsar] kjaggann commented on issue #6834: Apache pulsar Encryption failed using ecdsa
kjaggann commented on issue #6834: URL: https://github.com/apache/pulsar/issues/6834#issuecomment-621169665 I used maven to develop a sample producer and consumer class using java. I am able to send the messages with out encryption. But for this topic i have enabled encryption. I followed the same steps from the apache pulsar document, generated a public key and private key, used it as the document said. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #6836: PULSAR_PID_DIR setting does not appear to work
frankjkelly commented on issue #6836: URL: https://github.com/apache/pulsar/issues/6836#issuecomment-621177056 Gotcha - for use cases like Kubernetes health checks or waiting until Pulsar is up to run tests is there some mechanism to check that Pulsar is up-and-running 100% "Server - Started" in the log file? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] laxpio commented on issue #6839: command "pulsar-admin functions list" unavailable in run function-worker separately
laxpio commented on issue #6839: URL: https://github.com/apache/pulsar/issues/6839#issuecomment-621177254 fix it, because org.apache.pulsar.functions.worker.rest.api.v3.FunctionsApiV3Resource.java miss listFunctions function. add the listFunctions to support cmd "functions list". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] frankjkelly edited a comment on issue #6836: PULSAR_PID_DIR setting does not appear to work
frankjkelly edited a comment on issue #6836: URL: https://github.com/apache/pulsar/issues/6836#issuecomment-621177056 @sijie - Thanks - Gotcha - for use cases like Kubernetes health checks or waiting until Pulsar is up to run tests is there some mechanism to check that Pulsar is up-and-running 100% e.g. "Server - Started" in the log file? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] trexinc commented on issue #6834: Apache pulsar Encryption failed using ecdsa
trexinc commented on issue #6834: URL: https://github.com/apache/pulsar/issues/6834#issuecomment-621221348 @sijie I use gradle. If I do not add the pulsar-client-messagecrypto-bc dependency manually it fails at runtime with java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/pulsar/client/impl/crypto/MessageCryptoBc This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] Persi opened a new issue #6842: Pulsar Java client contains shaded Jackson which makes it kind of unusable with Spring Boot
Persi opened a new issue #6842: URL: https://github.com/apache/pulsar/issues/6842 **Is your feature request related to a problem? Please describe.** I am trying to use Pulsar Java client in a Spring Boot application and was able to produce and consume messages. I ran in trouble when sending messages with JSON content which contains fields of type OffsetDateTime. My Spring Boot application already uses the same JSON for it's REST API and can serialize and deserialize OffsetDateTime as "jackson-datatype-jsr310" is in my classpath. But the included version of Jackson in the Pulsar Java client, does not know this extension and instead fails while deserialization with erros like the following: ``` ... Caused by: org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `java.time.OffsetDateTime` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator) ``` **Describe the solution you'd like** Could you provide a non-shaded version of the Pulsar Java client? This would be really helpful for Spring Boot integrations, as most Spring Boot applications already provide a global configured instance of Jackson ObjectMapper, which the Pulsar Java client could use, to use the same mapping settings as the rest of the application. **Describe alternatives you've considered** The JSONSchema class instanciates it's ObjectMapper as internal ThreadLocal which makes it unchangeble as well as unconfigurable. You could add the opportunity to provide an ObjectMapper instance, e.g. on pulsar client instanciation, which could be used by all producers and consumers created. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] kjaggann commented on issue #6834: Apache pulsar Encryption failed using ecdsa
kjaggann commented on issue #6834: URL: https://github.com/apache/pulsar/issues/6834#issuecomment-621248014 do you have any sample python script encryption for producer and consumer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] liudezhi2098 opened a new issue #6843: support setPassword to managedLedger from brokerconfg
liudezhi2098 opened a new issue #6843: URL: https://github.com/apache/pulsar/issues/6843 ## Motivation broker supports ManagedLedger configuration ## Proposed changes org.apache.pulsar.broker.ServiceConfiguration ```java @FieldContext( category = CATEGORY_STORAGE_ML, doc = "Default password to use when writing to BookKeeper. \n\nDefault is ``." ) private String managedLedgerPassword = ""; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] liudezhi2098 opened a new pull request #6844: Support set password to managed ledger from brokerconfg
liudezhi2098 opened a new pull request #6844: URL: https://github.com/apache/pulsar/pull/6844 Master Issue: #6843 ## Motivation broker supports ManagedLedger configuration ## Modifications This change can be supported managedLedgerPassword configuration org.apache.pulsar.broker.ServiceConfiguration ```java @FieldContext( category = CATEGORY_STORAGE_ML, doc = "Default password to use when writing to BookKeeper. \n\nDefault is ``." ) private String managedLedgerPassword = ""; ``` org.apache.pulsar.broker.service.BrokerService ```java managedLedgerConfig.setPassword(serviceConfig.getManagedLedgerPassword()); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] itaross opened a new issue #6845: Subscription to topics with regex not working for non-persisten topics
itaross opened a new issue #6845: URL: https://github.com/apache/pulsar/issues/6845 Messages on non-persistent topics are not received when using a regex subscription. This problem is encountered when using a Pulsar IO custom connector (sink). When using persistent topics the subscription is working as expected **Expected behavior** Records are received when subscribing to regex topics. **Additional context** Issues #2009 https://github.com/apache/pulsar/issues/2009 Issues #2025 https://github.com/apache/pulsar/issues/2025 Does not solve this problem @merlimat @sijie This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui opened a new pull request #6846: Avoid creating partitioned topic for partition name
codelipenghui opened a new pull request #6846: URL: https://github.com/apache/pulsar/pull/6846 Fixes #6840 ### Motivation Avoid creating partitioned topic for partition name ### Verifying this change New unit test added. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: ( no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] digi691 commented on issue #5417: Broker hangs and crashes when listing non-persistent topics
digi691 commented on issue #5417: URL: https://github.com/apache/pulsar/issues/5417#issuecomment-621328591 Know this issue is stale though we are still having this issue on our bare metal clusters. I recently setup Pulsar in Kubernetes and I'm not seeing this behavior. Is there any kind of miss-configuration of the brokers and zookeepers that could cause the broker just to swallow these type of requests, not respond, and not log any errors about it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] complex64 commented on issue #5847: Error at starting apachepulsar/pulsar-dashboard docker container within Kubernetes
complex64 commented on issue #5847: URL: https://github.com/apache/pulsar/issues/5847#issuecomment-621342768 Running into this issue with the image `apachepulsar/pulsar-dashboard:2.5.1` and the following error: ``` Starting Pulsar dasboard + /pulsar/init-postgres.sh + rm -rf '/data/*' + chown -R postgres: /data + chmod 700 /data + sudo -u postgres /usr/lib/postgresql/11/bin/initdb /data/ sudo: /usr/lib/postgresql/11/bin/initdb: command not found ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] complex64 edited a comment on issue #5847: Error at starting apachepulsar/pulsar-dashboard docker container within Kubernetes
complex64 edited a comment on issue #5847: URL: https://github.com/apache/pulsar/issues/5847#issuecomment-621342768 Running into this issue with the image `apachepulsar/pulsar-dashboard:2.5.1` and the following error: ``` Starting Pulsar dasboard + /pulsar/init-postgres.sh + rm -rf '/data/*' + chown -R postgres: /data + chmod 700 /data + sudo -u postgres /usr/lib/postgresql/11/bin/initdb /data/ sudo: /usr/lib/postgresql/11/bin/initdb: command not found ``` Inspecting the docker image, it appears `/usr/lib/postgresql/9.6/bin/initdb` exists but `/usr/lib/postgresql/11/bin/initdb` does not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] kjaggann commented on issue #6834: Apache pulsar Encryption failed using ecdsa
kjaggann commented on issue #6834: URL: https://github.com/apache/pulsar/issues/6834#issuecomment-621346012 I don't see any document in apache pulsar at-least to try python encryption and decryption This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] addisonj commented on pull request #6846: Avoid creating partitioned topic for partition name
addisonj commented on pull request #6846: URL: https://github.com/apache/pulsar/pull/6846#issuecomment-621395527 This looks good to me. I will hotfix this into my clusters at some point in the next few days and will confirm the fix as well. Thanks for the quick turn around! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] cdbartholomew commented on pull request #6767: [function] Function endpoint admin/v3/functions/{tenant}/{namespace} always returns 404
cdbartholomew commented on pull request #6767: URL: https://github.com/apache/pulsar/pull/6767#issuecomment-621407940 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #4857: Error: "Trying to subscribe with incompatible schema" even though isSchemaValidationEnforced=false
devinbost commented on issue #4857: URL: https://github.com/apache/pulsar/issues/4857#issuecomment-621444103 @FFC-SystemAdmin We've noticed that it seems like this issue may occur depending on the order in which subscriptions are made (e.g. from sink to source vs source to sink.) We've had more success going from sink to source (meaning connecting the sink to the upstream function, then connecting that function to the function upstream from it . . . until you connect an upstream function to the source.) I don't know what your flow looks like though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] devinbost commented on issue #4937: Retrieve schema instance from schema info for type 'NONE'
devinbost commented on issue #4937: URL: https://github.com/apache/pulsar/issues/4937#issuecomment-621445978 At a glance, I wonder if this issue is related to this other one: https://github.com/apache/pulsar/issues/4857#issuecomment-517065570 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] devinbost edited a comment on issue #4937: Retrieve schema instance from schema info for type 'NONE'
devinbost edited a comment on issue #4937: URL: https://github.com/apache/pulsar/issues/4937#issuecomment-621445978 At a glance, I wonder if this issue is related to this other one: https://github.com/apache/pulsar/issues/4857 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gfredericks opened a new issue #6847: Reader skips over large block of messages after calling `seek`
gfredericks opened a new issue #6847: URL: https://github.com/apache/pulsar/issues/6847 # Bug Description In some difficult-to-reproduce circumstances, a reader will skip forward to later in the topic with no indication that anything abnormal happened. # To Reproduce 1. Publish to a topic for a while; high volume might help 2. Choose two timestamps, `t1` and `t2`, with a decent window between them, that are both within the `publishTimestamp` history for the topic 3. Do the following dozens of times 1. Create a reader and call `.seek(t1)` 2. Read initial messages until you get one in between `t1` and `t2` (I would think the first message should qualify, but often it doesn't, and I don't know why) 3. Read _and count_ messages until you see one with a `publishTimestamp > t2` 4. Print/return the count 4. Compare the results across runs # Expected behavior The result should always be the same, assuming the publisher is setting `publishTimestamp` monotonically and has already passed `t2` at the beginning of the test. # Additional context I often see that during the run the producer makes at most one jump. Just recently I have noticed that the point it jumps to (the "jump target") increases over time, in a way that I think tracks the messages being published to the topic. I saw a very reliable increase until I stopped the producer, and then the increase stopped. ## Configurations I've Tested - Broker versions 2.4.2, 2.5.1 - Producer version 2.3.0, 2.5.1 - Non-partitioned topics - Batching enabled - Python client, but mostly Java client - Both reader and consumer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] cdbartholomew opened a new pull request #6848: [JDBC Sink] JDBC sink does not handle null in schema
cdbartholomew opened a new pull request #6848: URL: https://github.com/apache/pulsar/pull/6848 ### Motivation JDBC sink does not handle `null` fields. For example, the field `example` can potentially be null. The schema registered in Pulsar allows for it, and the table schema in MySQL has a column of the same name, is configured as double and also allows nulls. When messages are sent to the JDBC sink without that field, an exception like this is seen: ``` 21:08:38.472 [pool-5-thread-1] ERROR org.apache.pulsar.io.jdbc.JdbcAbstractSink - Got exception java.sql.SQLException: Data truncated for column 'example' at row 1 at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:127) ~[mysql-connector-java-8.0.11.jar:8.0.11] at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95) ~[mysql-connector-java-8.0.11.jar:8.0.11] at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.11.jar:8.0.11] at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960) ~[mysql-connector-java-8.0.11.jar:8.0.11] at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:388) ~[mysql-connector-java-8.0.11.jar:8.0.11] at org.apache.pulsar.io.jdbc.JdbcAbstractSink.flush(JdbcAbstractSink.java:202) ~[pulsar-io-jdbc-2.5.0.nar-unpacked/:?] at org.apache.pulsar.io.jdbc.JdbcAbstractSink.lambda$open$0(JdbcAbstractSink.java:108) ~[pulsar-io-jdbc-2.5.0.nar-unpacked/:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_232] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_232] ``` Looking at the code for the JDBC sink, there was no handling of the case where the field was `null`. The PR adds code to handle that case. It also adds unit tests to cover this for both binary and JSON encoding of the schema. ### Modifications When the sink encounters a `null` field value it uses the `setColumnNull` method to properly reflect this in the database row. ### Verifying this change - [X ] Make sure that the change passes the CI checks. This change added tests and can be verified as follows: - *Run unit test that sends null values and validates that they are properly set to null by doing a SQL query on the resulting table* - *Run unit test that cover JSON encoding since `null`s are handled differently depending on the encoding* In addition, this change has been verified by installing the new NAR into an existing cluster, sending messages with `null` values to a MySQL database, and confirming that the resulting rows are properly represented. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shustsud opened a new pull request #6849: [broker] Fixed IllegalArgumentException issue in Key_Shared
shustsud opened a new pull request #6849: URL: https://github.com/apache/pulsar/pull/6849 ### Motivation If subscription type is Key_Shared, IllegalArgumentException occurs. Reproduces when the Replicated subscription is enabled. Log: ``` 14:47:24.297 [bookkeeper-ml-workers-OrderedExecutor-2-0] ERROR o.a.b.common.util.SafeRunnable - Unexpected throwable caught java.lang.IllegalArgumentException: Keys and values must be >= 0 at org.apache.pulsar.common.util.collections.ConcurrentLongPairSet.checkBiggerEqualZero(ConcurrentLongPairSet.java:503) at org.apache.pulsar.common.util.collections.ConcurrentLongPairSet.add(ConcurrentLongPairSet.java:127) at org.apache.pulsar.common.util.collections.ConcurrentSortedLongPairSet.add(ConcurrentSortedLongPairSet.java:83) at org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumers.lambda$sendMessagesToConsumers$2(PersistentStickyKeyDispatcherMultipleConsumers.java:149) at java.util.ArrayList.forEach(ArrayList.java:1257) at org.apache.pulsar.broker.service.persistent.PersistentStickyKeyDispatcherMultipleConsumers.sendMessagesToConsumers(PersistentStickyKeyDispatcherMultipleConsumers.java:148) at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.readEntriesComplete(PersistentDispatcherMultipleConsumers.java:460) at org.apache.bookkeeper.mledger.impl.OpReadEntry.lambda$checkReadCompletion$2(OpReadEntry.java:146) at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) ``` If sublist contains metadata, element will be null. [https://github.com/apache/pulsar/blob/1fd1b2b440af2477f916999a67752f9f532d1620/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L114](https://github.com/apache/pulsar/blob/1fd1b2b440af2477f916999a67752f9f532d1620/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L114) This will not remove element from entriesWithSameKey. [https://github.com/apache/pulsar/blob/1fd1b2b440af2477f916999a67752f9f532d1620/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L122](https://github.com/apache/pulsar/blob/1fd1b2b440af2477f916999a67752f9f532d1620/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L122) ### Modifications * Fixed PersistentStickyKeyDispatcherMultipleConsumers class. ### Verifying this change - [ ] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-go] jiazhai commented on issue #234: Cannot build the project
jiazhai commented on issue #234: URL: https://github.com/apache/pulsar-client-go/issues/234#issuecomment-621570463 @wolfstudy Could we fix this? This is misleading. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Avoid creating partitioned topic for partition name (#6846)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new bb41702 Avoid creating partitioned topic for partition name (#6846) bb41702 is described below commit bb41702501e90a0789a39a1cb921b6b82eb55d07 Author: lipenghui AuthorDate: Thu Apr 30 10:13:49 2020 +0800 Avoid creating partitioned topic for partition name (#6846) Fixes #6840 Motivation Avoid creating partitioned topic for partition name Verifying this change New unit test added. --- .../pulsar/broker/service/BrokerService.java | 1 + .../pulsar/broker/admin/TopicAutoCreationTest.java | 88 ++ 2 files changed, 89 insertions(+) diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java index 7f69792..ed10e1d 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java @@ -1882,6 +1882,7 @@ public class BrokerService implements Closeable, ZooKeeperCacheListenerhttp://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.pulsar.broker.admin; + +import org.apache.pulsar.client.admin.PulsarAdminException; +import org.apache.pulsar.client.api.Producer; +import org.apache.pulsar.client.api.ProducerConsumerBase; +import org.apache.pulsar.client.api.PulsarClientException; +import org.junit.Assert; +import org.testng.annotations.AfterMethod; +import org.testng.annotations.BeforeMethod; +import org.testng.annotations.Test; + +import java.util.List; +import java.util.UUID; + +public class TopicAutoCreationTest extends ProducerConsumerBase { + +@Override +@BeforeMethod +protected void setup() throws Exception { +conf.setAllowAutoTopicCreationType("partitioned"); +conf.setAllowAutoTopicCreation(true); +conf.setDefaultNumPartitions(3); +super.internalSetup(); +super.producerBaseSetup(); +} + +@Override +@AfterMethod +protected void cleanup() throws Exception { +super.internalCleanup(); +} + +@Test +public void testPartitionedTopicAutoCreation() throws PulsarAdminException, PulsarClientException { +final String namespaceName = "my-property/my-ns"; +final String topic = "persistent://" + namespaceName + "/test-partitioned-topi-auto-creation-" ++ UUID.randomUUID().toString(); + +Producer producer = pulsarClient.newProducer() +.topic(topic) +.create(); + +List partitionedTopics = admin.topics().getPartitionedTopicList(namespaceName); +List topics = admin.topics().getList(namespaceName); +Assert.assertEquals(partitionedTopics.size(), 1); +Assert.assertEquals(topics.size(), 3); + +producer.close(); +for (String t : topics) { +admin.topics().delete(t); +} + +admin.topics().deletePartitionedTopic(topic); + + +final String partition = "persistent://" + namespaceName + "/test-partitioned-topi-auto-creation-partition-0"; + +producer = pulsarClient.newProducer() +.topic(partition) +.create(); + +partitionedTopics = admin.topics().getPartitionedTopicList(namespaceName); +topics = admin.topics().getList(namespaceName); +Assert.assertEquals(partitionedTopics.size(), 0); +Assert.assertEquals(topics.size(), 1); + +producer.close(); +} +}
[GitHub] [pulsar] sijie commented on issue #6842: Pulsar Java client contains shaded Jackson which makes it kind of unusable with Spring Boot
sijie commented on issue #6842: URL: https://github.com/apache/pulsar/issues/6842#issuecomment-621574013 @Persi the default `pulsar-client` shades all dependencies. You can use `pulsar-client-original` with unshaded dependencies. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] jiazhai commented on issue #6843: support setPassword to managedLedger from brokerconfg
jiazhai commented on issue #6843: URL: https://github.com/apache/pulsar/issues/6843#issuecomment-621574099 @liudezhi2098 Thanks for the help on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] gh777777 commented on pull request #4946: [pulsar-functions][python-examples] Fixing python function example custom_object_function.py
gh77 commented on pull request #4946: URL: https://github.com/apache/pulsar/pull/4946#issuecomment-621576745 @sijie thank you for making this example works! I'm learning pulsar functions dev in python. About this example (custom_object_function.py), how can I build and send a custom object in producer, so that I can get it back like that from the input in CustomObjectFunction.process(self, input, context)? Do I have to use schema? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[pulsar] branch master updated: Support set password to managed ledger from brokerconfg (#6844)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 2122e27 Support set password to managed ledger from brokerconfg (#6844) 2122e27 is described below commit 2122e271ba5cddeecb2fee411213bbb1c9b672f6 Author: liudezhi <33149602+liudezhi2...@users.noreply.github.com> AuthorDate: Thu Apr 30 10:59:44 2020 +0800 Support set password to managed ledger from brokerconfg (#6844) Master Issue: #6843 ## Motivation broker supports ManagedLedger configuration ## Modifications This change can be supported managedLedgerPassword configuration org.apache.pulsar.broker.ServiceConfiguration ```java @FieldContext( category = CATEGORY_STORAGE_ML, doc = "Default password to use when writing to BookKeeper. \n\nDefault is ``." ) private String managedLedgerPassword = ""; ``` org.apache.pulsar.broker.service.BrokerService ```java managedLedgerConfig.setPassword(serviceConfig.getManagedLedgerPassword()); ``` * support setPassword to managedLedger from brokerconfg * support setPassword to managedLedger from brokerconfg Co-authored-by: dezhiliu --- .../main/java/org/apache/pulsar/broker/ServiceConfiguration.java| 6 ++ .../main/java/org/apache/pulsar/broker/service/BrokerService.java | 1 + 2 files changed, 7 insertions(+) diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java index 2b0b0e8..a50611a 100644 --- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java +++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java @@ -986,6 +986,12 @@ public class ServiceConfiguration implements PulsarConfiguration { private DigestType managedLedgerDigestType = DigestType.CRC32C; @FieldContext( +category = CATEGORY_STORAGE_ML, +doc = "Default password to use when writing to BookKeeper. \n\nDefault is ``." +) +private String managedLedgerPassword = ""; + +@FieldContext( minValue = 1, category = CATEGORY_STORAGE_ML, doc = "Max number of bookies to use when creating a ledger" diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java index ed10e1d..4c935d1 100644 --- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java +++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java @@ -1017,6 +1017,7 @@ public class BrokerService implements Closeable, ZooKeeperCacheListener
[GitHub] [pulsar] tuteng commented on issue #5847: Error at starting apachepulsar/pulsar-dashboard docker container within Kubernetes
tuteng commented on issue #5847: URL: https://github.com/apache/pulsar/issues/5847#issuecomment-621584874 You can try http://pulsar.apache.org/docs/en/administration-pulsar-manager/ first. https://github.com/apache/pulsar-manager is a better management tool. Let's take a look at this problem next. @complex64 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] MarvinCai opened a new issue #6850: Latest pulsar-all docker image might be broken.
MarvinCai opened a new issue #6850: URL: https://github.com/apache/pulsar/issues/6850 **Describe the bug** When following instruction to run pulsar with minikube, I stuck at the step "$ kubectl apply -f bookie.yaml" The pod can not start up and I see error log "Error: Could not find or load main class" After change to version 2.5.0 it seems working fine. So there might be some problem with the latest(2.5.1) pulsar-all image. **To Reproduce** Steps to reproduce the behavior: Follow instruction [here](https://pulsar.apache.org/docs/en/2.5.1/deploy-kubernetes/#deploying-pulsar-components) to run pulsar cluster in Kubernetes. **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. ![image](https://user-images.githubusercontent.com/13480084/80670164-0e9c4200-8a5b-11ea-8f18-0ffb58764552.png) **Desktop (please complete the following information):** - OS: Mac This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shustsud commented on pull request #6849: [broker] Fixed IllegalArgumentException issue in Key_Shared
shustsud commented on pull request #6849: URL: https://github.com/apache/pulsar/pull/6849#issuecomment-621596809 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #6818: pulsar-client vulnerability CVE-2015-5237 (shaded protobuf-java:2.4.1)
sijie commented on issue #6818: URL: https://github.com/apache/pulsar/issues/6818#issuecomment-621601193 @guyv currently pulsar client is using a customized protobuf-2.4 version for processing the requests. It is used for avoiding generating a lot of objects and reducing the frequency of garbage collection. It is pretty hard for us to upgrade. We will look into how to address it in future releases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] rueian commented on issue #6850: Latest pulsar-all docker image might be broken.
rueian commented on issue #6850: URL: https://github.com/apache/pulsar/issues/6850#issuecomment-621609045 I run into the same issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] shustsud commented on pull request #6849: [broker] Fixed IllegalArgumentException issue in Key_Shared
shustsud commented on pull request #6849: URL: https://github.com/apache/pulsar/pull/6849#issuecomment-621618631 /pulsarbot run-failure-checks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
codelipenghui commented on issue #6845: URL: https://github.com/apache/pulsar/issues/6845#issuecomment-621642156 @itaross Can you provide more details about the bug? what version are you using and how to reproduce it. This will help to troubleshoot this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org