[GitHub] [pulsar] willome closed pull request #4660: Add Debezium Postgresql Integration test
willome closed pull request #4660: Add Debezium Postgresql Integration test URL: https://github.com/apache/pulsar/pull/4660 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] willome opened a new pull request #4668: Add Debezium Postgresql integration test
willome opened a new pull request #4668: Add Debezium Postgresql integration test URL: https://github.com/apache/pulsar/pull/4668 Provide new CDC/Debezium Postgresql Integration test. Only Mysql one was present. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4667: Modify file system version
codelipenghui commented on issue #4667: Modify file system version URL: https://github.com/apache/pulsar/pull/4667#issuecomment-507993976 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] liketic commented on issue #4640: [Doc] Fix FAQ broken link
liketic commented on issue #4640: [Doc] Fix FAQ broken link URL: https://github.com/apache/pulsar/pull/4640#issuecomment-507995130 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4621: [PIP-38] Support batch receive in java client.
codelipenghui commented on issue #4621: [PIP-38] Support batch receive in java client. URL: https://github.com/apache/pulsar/pull/4621#issuecomment-508000241 @sijie Oh, sorry i forgot it, i will send a email soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT
congbobo184 commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT URL: https://github.com/apache/pulsar/pull/4667#issuecomment-508039523 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #4668: Add Debezium Postgresql integration test
tuteng commented on issue #4668: Add Debezium Postgresql integration test URL: https://github.com/apache/pulsar/pull/4668#issuecomment-508055318 Can you add a more detailed test? For example, actively insert a piece of data into PostgreSql, then receive it use consumer, and finally compare the fields. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on a change in pull request #4666: Doc release 2.4.0
jiazhai commented on a change in pull request #4666: Doc release 2.4.0 URL: https://github.com/apache/pulsar/pull/4666#discussion_r299912328 ## File path: site2/website/versioned_docs/version-2.4.0/io-cdc-canal.md ## @@ -0,0 +1,175 @@ +--- +id: version-2.4.0-io-cdc-canal +title: CDC Canal Connector +sidebar_label: CDC Canal Connector +original_id: io-cdc-canal +--- + +### Source Configuration Options + +The Configuration is mostly related to Canal task config. + +| Name | Required | Default | Description | +|--|--|-|-| +| `zkServers` | `false` | `127.0.0.1:2181` | `The address and port of the zookeeper . if canal server configured to cluster mode` | +| `batchSize` | `true` | `5120` | `Take 5120 records from the canal server in batches` | +| `username` | `false` | `` | `Canal server account, not MySQL` | +| `password` | `false` | `` | `Canal server password, not MySQL` | +| `cluster` | `false` | `false` | `Decide whether to open cluster mode based on canal server configuration, true: cluster mode, false: standalone mode` | +| `singleHostname` | `false` | `127.0.0.1` | `The address of canal server` | +| `singlePort` | `false` | `1` | `The port of canal server` | + + +### Configuration Example + +Here is a configuration Json example: + +```$json +{ +"zkServers": "127.0.0.1:2181", +"batchSize": "5120", +"destination": "example", +"username": "", +"password": "", +"cluster": false, +"singleHostname": "127.0.0.1", +"singlePort": "1", +} +``` +You could also find the yaml example in this [file](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/resources/canal-mysql-source-config.yaml), which has similar content below: + +```$yaml +configs: +zkServers: "127.0.0.1:2181" +batchSize: "5120" +destination: "example" +username: "" +password: "" +cluster: false +singleHostname: "127.0.0.1" +singlePort: "1" +``` + +### Usage example + +Here is a simple example to store MySQL change data using above example config. + +- Start a MySQL server + +```$bash +docker pull mysql:5.7 +docker run -d -it --rm --name pulsar-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=canal -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw mysql:5.7 +``` +- Modify configuration files mysqld.cnf + +``` +[mysqld] +pid-file= /var/run/mysqld/mysqld.pid +socket = /var/run/mysqld/mysqld.sock +datadir = /var/lib/mysql +#log-error = /var/log/mysql/error.log +# By default we only accept connections from localhost +#bind-address = 127.0.0.1 +# Disabling symbolic-links is recommended to prevent assorted security risks +symbolic-links=0 +log-bin=mysql-bin +binlog-format=ROW +server_id=1 +``` + +- Copy file to mysql server from local and restart mysql server +```$bash +docker cp mysqld.cnf pulsar-mysql:/etc/mysql/mysql.conf.d/ +docker restart pulsar-mysql +``` + +- Create test database in mysql server +```$bash +docker exec -it pulsar-mysql /bin/bash +mysql -h 127.0.0.1 -uroot -pcanal -e 'create database test;' +``` + +- Start canal server and connect mysql server + +``` +docker pull canal/canal-server:v1.1.2 +docker run -d -it --link pulsar-mysql -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=pulsar-mysql:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=canal -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false --name=pulsar-canal-server -p 8000:8000 -p : -p 1:1 -p 2:2 -m 4096m canal/canal-server:v1.1.2 +``` + +- Start pulsar standalone + +```$bash +docker pull apachepulsar/pulsar:2.3.0 Review comment: Seems here 2.3.0 should be replaced? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on a change in pull request #4666: Doc release 2.4.0
jiazhai commented on a change in pull request #4666: Doc release 2.4.0 URL: https://github.com/apache/pulsar/pull/4666#discussion_r299912979 ## File path: site2/website/versioned_docs/version-2.4.0/io-cdc-canal.md ## @@ -0,0 +1,175 @@ +--- +id: version-2.4.0-io-cdc-canal +title: CDC Canal Connector +sidebar_label: CDC Canal Connector +original_id: io-cdc-canal +--- + +### Source Configuration Options + +The Configuration is mostly related to Canal task config. + +| Name | Required | Default | Description | +|--|--|-|-| +| `zkServers` | `false` | `127.0.0.1:2181` | `The address and port of the zookeeper . if canal server configured to cluster mode` | +| `batchSize` | `true` | `5120` | `Take 5120 records from the canal server in batches` | +| `username` | `false` | `` | `Canal server account, not MySQL` | +| `password` | `false` | `` | `Canal server password, not MySQL` | +| `cluster` | `false` | `false` | `Decide whether to open cluster mode based on canal server configuration, true: cluster mode, false: standalone mode` | +| `singleHostname` | `false` | `127.0.0.1` | `The address of canal server` | +| `singlePort` | `false` | `1` | `The port of canal server` | + + +### Configuration Example + +Here is a configuration Json example: + +```$json +{ +"zkServers": "127.0.0.1:2181", +"batchSize": "5120", +"destination": "example", +"username": "", +"password": "", +"cluster": false, +"singleHostname": "127.0.0.1", +"singlePort": "1", +} +``` +You could also find the yaml example in this [file](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/resources/canal-mysql-source-config.yaml), which has similar content below: + +```$yaml +configs: +zkServers: "127.0.0.1:2181" +batchSize: "5120" +destination: "example" +username: "" +password: "" +cluster: false +singleHostname: "127.0.0.1" +singlePort: "1" +``` + +### Usage example + +Here is a simple example to store MySQL change data using above example config. + +- Start a MySQL server + +```$bash +docker pull mysql:5.7 +docker run -d -it --rm --name pulsar-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=canal -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw mysql:5.7 +``` +- Modify configuration files mysqld.cnf + +``` +[mysqld] +pid-file= /var/run/mysqld/mysqld.pid +socket = /var/run/mysqld/mysqld.sock +datadir = /var/lib/mysql +#log-error = /var/log/mysql/error.log +# By default we only accept connections from localhost +#bind-address = 127.0.0.1 +# Disabling symbolic-links is recommended to prevent assorted security risks +symbolic-links=0 +log-bin=mysql-bin +binlog-format=ROW +server_id=1 +``` + +- Copy file to mysql server from local and restart mysql server +```$bash +docker cp mysqld.cnf pulsar-mysql:/etc/mysql/mysql.conf.d/ +docker restart pulsar-mysql +``` + +- Create test database in mysql server +```$bash +docker exec -it pulsar-mysql /bin/bash +mysql -h 127.0.0.1 -uroot -pcanal -e 'create database test;' +``` + +- Start canal server and connect mysql server + +``` +docker pull canal/canal-server:v1.1.2 +docker run -d -it --link pulsar-mysql -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=pulsar-mysql:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=canal -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false --name=pulsar-canal-server -p 8000:8000 -p : -p 1:1 -p 2:2 -m 4096m canal/canal-server:v1.1.2 +``` + +- Start pulsar standalone + +```$bash +docker pull apachepulsar/pulsar:2.3.0 +docker run -d -it --link pulsar-canal-server -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-standalone apachepulsar/pulsar:2.3.0 bin/pulsar standalone +``` + +- Start pulsar-io in standalone + +- Config file canal-mysql-source-config.yaml + +```$yaml +configs: +zkServers: "" +batchSize: "5120" +destination: "test" +username: "" +password: "" +cluster: false +singleHostname: "pulsar-canal-server" +singlePort: "1" +``` +- Consumer file pulsar-client.py for test +``` +import pulsar + +client = pulsar.Client('pulsar://localhost:6650') +consumer = client.subscribe('my-topic', +subscription_name='my-sub') + +while True: +msg = consumer.receive() +print("Received message: '%s'" % msg.data()) +consumer.acknowledge(msg) + +client.close() +``` + +- Copy config file and test file to pulsar server + +```$bash +docker cp canal-mysql-source-config.yaml pulsar-standalone:/pulsar/conf/ +docker cp pulsar-client.py pulsar-standalone:/pulsar/ +``` + +- Download canal connector and start canal connector +```$bash +docker exec -it pulsar-standalone /bin/bash +wget http://apache.01link.hk/pulsar/pulsar-2.3.0/connectors/pulsar-io-canal-2.3.0.nar -P connectors Review comment: also here for 2.
[GitHub] [pulsar] codelipenghui commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT
codelipenghui commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT URL: https://github.com/apache/pulsar/pull/4667#issuecomment-508130394 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] massakam commented on issue #4635: Bookie down causes deadlock in broker
massakam commented on issue #4635: Bookie down causes deadlock in broker URL: https://github.com/apache/pulsar/issues/4635#issuecomment-508150928 However, `pulsar-ordered-OrderedExecutor-2-0-EventThread` seems to block itself until a timeout occurs. ``` "pulsar-ordered-OrderedExecutor-2-0-EventThread" daemon prio=5 tid=23 TIMED_WAITING java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:95) at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.getRack(ZkBookieRackAffinityMapping.java:154) at org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.resolve(ZkBookieRackAffinityMapping.java:146) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl$DNSResolverDecorator.resolve(RackawareEnsemblePlacementPolicyImpl.java:174) at org.apache.bookkeeper.net.NetUtils.resolveNetworkLocation(NetUtils.java:81) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.resolveNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:402) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.createBookieNode(RackawareEnsemblePlacementPolicyImpl.java:259) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.convertBookiesToNodes(RackawareEnsemblePlacementPolicyImpl.java:512) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.newEnsembleInternal(RackawareEnsemblePlacementPolicyImpl.java:575) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.newEnsembleInternal(RackawareEnsemblePlacementPolicyImpl.java:540) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.newEnsemble(RackawareEnsemblePlacementPolicyImpl.java:531) at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.newEnsemble(RackawareEnsemblePlacementPolicy.java:98) at org.apache.bookkeeper.client.BookieWatcherImpl.newEnsemble(BookieWatcherImpl.java:233) at org.apache.bookkeeper.client.LedgerCreateOp.initiate(LedgerCreateOp.java:141) at org.apache.bookkeeper.client.BookKeeper.asyncCreateLedger(BookKeeper.java:831) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncCreateLedger(ManagedLedgerImpl.java:3046) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.ledgerClosed(ManagedLedgerImpl.java:1388) at org.apache.bookkeeper.mledger.impl.OpAddEntry.closeComplete(OpAddEntry.java:192) at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$0(LedgerHandle.java:554) at org.apache.bookkeeper.client.LedgerHandle$5$$Lambda$828/960239181.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.bookkeeper.common.concurrent.FutureUtils.complete(FutureUtils.java:126) at org.apache.bookkeeper.client.LedgerHandle$5.lambda$safeRun$3(LedgerHandle.java:614) at org.apache.bookkeeper.client.LedgerHandle$5$$Lambda$833/911729943.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.bookkeeper.client.MetadataUpdateLoop.lambda$writeLoop$1(MetadataUpdateLoop.java:129) at org.apache.bookkeeper.client.MetadataUpdateLoop$$Lambda$825/2131232665.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[GitHub] [pulsar] ivankelly opened a new pull request #4669: Don't consider deleted schema when checking compatibility
ivankelly opened a new pull request #4669: Don't consider deleted schema when checking compatibility URL: https://github.com/apache/pulsar/pull/4669 feca5bbb changed topic delete logic to delete the schema when the topic is deleted (though this only seems to be enabled for idle topic GC). This exposed a bug in compatibility checking whereby if the a subscription tries to attach to the topic, even if using the same schema as had been used previously, a compatibility exception will be thrown. The root cause was a logic bomb in the compatibility check. !(A || B) && C where it should have been !(A || B) || C. The issue doesn't affect producers because the check was already correct there. I've also updated the check for transitive compatibility to remove the prefix of schemas before the deleted schema. Previously this was throwing an NPE on the broker as it couldn't decode the deleted schema. This issue was discovered by failures in the healthcheck. The check period (5 minutes) was longer than the GC period (60 seconds). I would expect it to hit quite often in other scenarios also. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] ivankelly commented on issue #4669: Don't consider deleted schema when checking compatibility
ivankelly commented on issue #4669: Don't consider deleted schema when checking compatibility URL: https://github.com/apache/pulsar/pull/4669#issuecomment-508166082 CI won't pass until #4667 is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] ivankelly commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT
ivankelly commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT URL: https://github.com/apache/pulsar/pull/4667#issuecomment-508165902 CI won't pass until #4667 is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] ivankelly removed a comment on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT
ivankelly removed a comment on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT URL: https://github.com/apache/pulsar/pull/4667#issuecomment-508165902 CI won't pass until #4667 is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #4577: fix issue when submitting NAR via file url
jerrypeng commented on issue #4577: fix issue when submitting NAR via file url URL: https://github.com/apache/pulsar/pull/4577#issuecomment-508186894 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat merged pull request #4667: Modify file system offload version to 2.5.0-SNAPSHOT
merlimat merged pull request #4667: Modify file system offload version to 2.5.0-SNAPSHOT URL: https://github.com/apache/pulsar/pull/4667 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: A modified version (#4667)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 7afdf6b A modified version (#4667) 7afdf6b is described below commit 7afdf6b077ca41ce55dbfec683c5a263a361f6b3 Author: congbo <39078850+congbobo...@users.noreply.github.com> AuthorDate: Thu Jul 4 01:58:15 2019 +0800 A modified version (#4667) --- tiered-storage/file-system/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tiered-storage/file-system/pom.xml b/tiered-storage/file-system/pom.xml index 9e76101..bc47aae 100644 --- a/tiered-storage/file-system/pom.xml +++ b/tiered-storage/file-system/pom.xml @@ -25,7 +25,7 @@ org.apache.pulsar tiered-storage-parent -2.4.0-SNAPSHOT +2.5.0-SNAPSHOT ..
[pulsar] branch master updated (7afdf6b -> b367206)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 7afdf6b A modified version (#4667) add b367206 [pulsar-broker] avoid retrying deleting namespace when topic is already deleted/fenced (#4665) No new revisions were added by this update. Summary of changes: .../apache/pulsar/broker/service/persistent/PersistentTopic.java | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-)
[GitHub] [pulsar] merlimat merged pull request #4665: [pulsar-broker] avoid retrying deleting namespace when topic is already deleted/fenced
merlimat merged pull request #4665: [pulsar-broker] avoid retrying deleting namespace when topic is already deleted/fenced URL: https://github.com/apache/pulsar/pull/4665 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #4657: [#4569][pulsar-client] C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569)
merlimat commented on issue #4657: [#4569][pulsar-client] C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569) URL: https://github.com/apache/pulsar/pull/4657#issuecomment-508196296 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT
congbobo184 commented on issue #4667: Modify file system offload version to 2.5.0-SNAPSHOT URL: https://github.com/apache/pulsar/pull/4667#issuecomment-508196503 > run java8 tests 👍👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #4669: Don't consider deleted schema when checking compatibility
merlimat commented on issue #4669: Don't consider deleted schema when checking compatibility URL: https://github.com/apache/pulsar/pull/4669#issuecomment-508196608 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #4664: Renamed C++ logger enum names to avoid conflicts with compiler macros
merlimat commented on issue #4664: Renamed C++ logger enum names to avoid conflicts with compiler macros URL: https://github.com/apache/pulsar/pull/4664#issuecomment-508204974 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat closed issue #4569: C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout.
merlimat closed issue #4569: C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. URL: https://github.com/apache/pulsar/issues/4569 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569) (#4657)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 2f68615 C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569) (#4657) 2f68615 is described below commit 2f6861579c0bfc370abd9ecf5409c683eaa96ee7 Author: Easyfan Zheng AuthorDate: Thu Jul 4 03:16:56 2019 +0800 C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569) (#4657) --- pulsar-client-cpp/lib/ProducerImpl.cc | 8 1 file changed, 8 insertions(+) diff --git a/pulsar-client-cpp/lib/ProducerImpl.cc b/pulsar-client-cpp/lib/ProducerImpl.cc index b2e7848..bf9e3ac 100644 --- a/pulsar-client-cpp/lib/ProducerImpl.cc +++ b/pulsar-client-cpp/lib/ProducerImpl.cc @@ -232,6 +232,14 @@ void ProducerImpl::failPendingMessages(Result result) { // without holding producer mutex. for (MessageQueue::const_iterator it = pendingMessagesQueue_.begin(); it != pendingMessagesQueue_.end(); it++) { +// When dealing any failure message, if the current message is a batch one, we should also release +// the reserved spots in the pendingMessageQueue_, for all individual messages inside this batch +// message. See 'ProducerImpl::sendAsync' for more details. +if (it->msg_.impl_->metadata.has_num_messages_in_batch()) { +// batch message - need to release more spots +// -1 since the pushing batch message into the queue already released a spot + pendingMessagesQueue_.release(it->msg_.impl_->metadata.num_messages_in_batch() - 1); +} messagesToFail.push_back(*it); }
[GitHub] [pulsar] merlimat merged pull request #4657: [#4569][pulsar-client] C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569)
merlimat merged pull request #4657: [#4569][pulsar-client] C++ client producer sendAsync() method will be blocked forever, if enough batched messages sent timeout. (#4569) URL: https://github.com/apache/pulsar/pull/4657 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #4623: Fixed the default port for https and http in admin client
merlimat commented on issue #4623: Fixed the default port for https and http in admin client URL: https://github.com/apache/pulsar/pull/4623#issuecomment-508223854 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vzhikserg opened a new pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation
vzhikserg opened a new pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation URL: https://github.com/apache/pulsar/pull/4670 ### Motivation We could not find Java examples which would include Avro and Protobuf schemas, but from the other [documentation page](https://pulsar.apache.org/docs/en/concepts-schema-registry/) it was clear that Pulsar supports them. ### Modification Two code snippets (one for Protobuf and the other one for Avro) were added to the [Schema example](https://pulsar.apache.org/docs/en/client-libraries-java/#schemas) chapter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Guy-Dipietro-Bose commented on issue #4540: Pulsar Client has no support in Mac 10.14.5 (18F132) ?
Guy-Dipietro-Bose commented on issue #4540: Pulsar Client has no support in Mac 10.14.5 (18F132) ? URL: https://github.com/apache/pulsar/issues/4540#issuecomment-508240414 This issue also occurs with 2.4.0. I'm unable to use any version greater than 2.2.1. With 2.3.0 I see when running a test that import pulsar: E ImportError: dlopen(../lib/python2.7/site-packages/_pulsar.so, 2): Symbol not found: __ZN6icu_636LocaleC1ERKS0_ E Referenced from: ../lib/python2.7/site-packages/_pulsar.so E Expected in: flat namespace With 2.3.1 I see no error but pytest fails to run: I see Empty test suite. I don't see any errors, but this impacts pytest in a way that it fails to discover tests. This issue doesn't occur with This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #4669: Don't consider deleted schema when checking compatibility
merlimat commented on issue #4669: Don't consider deleted schema when checking compatibility URL: https://github.com/apache/pulsar/pull/4669#issuecomment-508256587 run cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation
merlimat commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation URL: https://github.com/apache/pulsar/pull/4670#discussion_r300155806 ## File path: site2/docs/client-libraries-java.md ## @@ -602,6 +603,24 @@ The following schema formats are currently available for Java: .create(); ``` +* Protobuf schemas can be generate using the `ProtobufSchema` class. The following example shows how to create the Protobuf schema and use it to instantiate a new producer: + + ```java + Schema protobufSchema = ProtobufSchema.of(MyProtobuf.class); + Producer protobufProducer = client.newProducer(protobufSchema) +.topic("some-protobuf-topic") +.create(); + ``` + +* Avro schemas can be defined with the help of the `AvroSchema` class. The next code snippet demonstrates the creation and usage of the Avro schema: + + ```java + Schema avroSchema = AvroSchema.of(MyAvro.class); + Producer avroProducer = client.newProducer(avroSchema) Review comment: Same here: ```java client.newProducer(Schema.AVRO(MyProtobuf.class)) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation
merlimat commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation URL: https://github.com/apache/pulsar/pull/4670#discussion_r300155752 ## File path: site2/docs/client-libraries-java.md ## @@ -602,6 +603,24 @@ The following schema formats are currently available for Java: .create(); ``` +* Protobuf schemas can be generate using the `ProtobufSchema` class. The following example shows how to create the Protobuf schema and use it to instantiate a new producer: + + ```java + Schema protobufSchema = ProtobufSchema.of(MyProtobuf.class); + Producer protobufProducer = client.newProducer(protobufSchema) Review comment: The preferred form should be: ```java client.newProducer(Schema.PROTOBUF(MyProtobuf.class)) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vzhikserg commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation
vzhikserg commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation URL: https://github.com/apache/pulsar/pull/4670#discussion_r300158086 ## File path: site2/docs/client-libraries-java.md ## @@ -602,6 +603,24 @@ The following schema formats are currently available for Java: .create(); ``` +* Protobuf schemas can be generate using the `ProtobufSchema` class. The following example shows how to create the Protobuf schema and use it to instantiate a new producer: + + ```java + Schema protobufSchema = ProtobufSchema.of(MyProtobuf.class); + Producer protobufProducer = client.newProducer(protobufSchema) Review comment: yes, I will change it. In my project the `Schema.PROTOBUF` form is used, but I wanted to keep it consistent with the sample for the JSON schema. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation
merlimat commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation URL: https://github.com/apache/pulsar/pull/4670#discussion_r300159463 ## File path: site2/docs/client-libraries-java.md ## @@ -602,6 +603,24 @@ The following schema formats are currently available for Java: .create(); ``` +* Protobuf schemas can be generate using the `ProtobufSchema` class. The following example shows how to create the Protobuf schema and use it to instantiate a new producer: + + ```java + Schema protobufSchema = ProtobufSchema.of(MyProtobuf.class); + Producer protobufProducer = client.newProducer(protobufSchema) Review comment: Oh, I see. Do you mind also changing the example for JSON? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] vzhikserg commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation
vzhikserg commented on a change in pull request #4670: [doc] Add code samples for Protobuf and Avro schemas in java documentation URL: https://github.com/apache/pulsar/pull/4670#discussion_r300160119 ## File path: site2/docs/client-libraries-java.md ## @@ -602,6 +603,24 @@ The following schema formats are currently available for Java: .create(); ``` +* Protobuf schemas can be generate using the `ProtobufSchema` class. The following example shows how to create the Protobuf schema and use it to instantiate a new producer: + + ```java + Schema protobufSchema = ProtobufSchema.of(MyProtobuf.class); + Producer protobufProducer = client.newProducer(protobufSchema) Review comment: sure, I will update it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat merged pull request #4577: fix issue when submitting NAR via file url
merlimat merged pull request #4577: fix issue when submitting NAR via file url URL: https://github.com/apache/pulsar/pull/4577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: fix issue when submitting NAR via file url (#4577)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new e8025d5 fix issue when submitting NAR via file url (#4577) e8025d5 is described below commit e8025d50c5d2cf0a632ad1573308b676d0607923 Author: Boyang Jerry Peng AuthorDate: Wed Jul 3 14:31:23 2019 -0700 fix issue when submitting NAR via file url (#4577) * fix issue when submitting NAR via file url * fix unit tests * add more specific errors * fix test --- .../worker/PulsarFunctionLocalRunTest.java | 21 ++- .../org/apache/pulsar/functions/LocalRunner.java | 1 - .../pulsar/functions/utils/SinkConfigUtils.java| 145 + .../pulsar/functions/utils/SourceConfigUtils.java | 122 ++--- .../functions/worker/rest/api/FunctionsImpl.java | 1 - .../functions/worker/rest/api/SinksImpl.java | 1 - .../functions/worker/rest/api/SourcesImpl.java | 1 - .../worker/rest/api/v3/SinkApiV3ResourceTest.java | 17 ++- .../rest/api/v3/SourceApiV3ResourceTest.java | 5 +- 9 files changed, 192 insertions(+), 122 deletions(-) diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java index bdcd27d..db7dbf2 100644 --- a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java +++ b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java @@ -599,12 +599,11 @@ public class PulsarFunctionLocalRunTest { testPulsarSourceLocalRun(null); } -// TODO bug to fix involving submitting a NAR via URI file:///tmp/pulsar-io-twitter-0.0.1.nar -//@Test(timeOut = 2) -//public void testPulsarSourceLocalRunWithFile() throws Exception { -//String jarFilePathUrl = Utils.FILE + ":" + getClass().getClassLoader().getResource("pulsar-io-data-generator.nar").getFile(); -//testPulsarSourceStats(jarFilePathUrl); -//} +@Test(timeOut = 2) +public void testPulsarSourceLocalRunWithFile() throws Exception { +String jarFilePathUrl = Utils.FILE + ":" + getClass().getClassLoader().getResource("pulsar-io-data-generator.nar").getFile(); +testPulsarSourceLocalRun(jarFilePathUrl); +} @Test(timeOut = 4) public void testPulsarSourceLocalRunWithUrl() throws Exception { @@ -705,11 +704,11 @@ public class PulsarFunctionLocalRunTest { testPulsarSinkStats(null); } -//@Test(timeOut = 2) -//public void testPulsarSinkStatsWithFile() throws Exception { -//String jarFilePathUrl = Utils.FILE + ":" + getClass().getClassLoader().getResource("pulsar-io-data-generator.nar").getFile(); -//testPulsarSinkStats(jarFilePathUrl); -//} +@Test(timeOut = 2) +public void testPulsarSinkStatsWithFile() throws Exception { +String jarFilePathUrl = Utils.FILE + ":" + getClass().getClassLoader().getResource("pulsar-io-data-generator.nar").getFile(); +testPulsarSinkStats(jarFilePathUrl); +} @Test(timeOut = 4) public void testPulsarSinkStatsWithUrl() throws Exception { diff --git a/pulsar-functions/localrun/src/main/java/org/apache/pulsar/functions/LocalRunner.java b/pulsar-functions/localrun/src/main/java/org/apache/pulsar/functions/LocalRunner.java index 1540744..f4c6eca 100644 --- a/pulsar-functions/localrun/src/main/java/org/apache/pulsar/functions/LocalRunner.java +++ b/pulsar-functions/localrun/src/main/java/org/apache/pulsar/functions/LocalRunner.java @@ -248,7 +248,6 @@ public class LocalRunner { .loadClass(LocalRunner.class.getName()) .getProtectionDomain().getCodeSource().getLocation().getFile(); } -log.info("userCodeFile: {}", userCodeFile); String builtInSource = isBuiltInSource(userCodeFile); if (builtInSource != null) { diff --git a/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java b/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java index 42eedc3..f0a39b9 100644 --- a/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java +++ b/pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java @@ -42,7 +42,11 @@ import java.io.File; import java.io.IOException; import java.lang.reflect.Type; import java.nio.file.Path; -import java.util.*; +import java.util.Collection; +import java.util.HashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; import static org.apache.commons.lang3.StringUtils.isE
[GitHub] [pulsar] ivankelly commented on issue #4669: Don't consider deleted schema when checking compatibility
ivankelly commented on issue #4669: Don't consider deleted schema when checking compatibility URL: https://github.com/apache/pulsar/pull/4669#issuecomment-508271739 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #4664: Renamed C++ logger enum names to avoid conflicts with compiler macros
merlimat commented on issue #4664: Renamed C++ logger enum names to avoid conflicts with compiler macros URL: https://github.com/apache/pulsar/pull/4664#issuecomment-508274405 run cpp tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat opened a new pull request #4671: Upgrade to Mockito 2.x
merlimat opened a new pull request #4671: Upgrade to Mockito 2.x URL: https://github.com/apache/pulsar/pull/4671 ## Motivation Upgrading to Mockito 2.28 and PowerMock 2.0. This a pre-step to be able to run CI with Java 11 / 12 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 closed pull request #4646: [pulsar-sql]deserialize the avro schema data by correct reader
congbobo184 closed pull request #4646: [pulsar-sql]deserialize the avro schema data by correct reader URL: https://github.com/apache/pulsar/pull/4646 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] congbobo184 commented on issue #4646: [pulsar-sql]deserialize the avro schema data by correct reader
congbobo184 commented on issue #4646: [pulsar-sql]deserialize the avro schema data by correct reader URL: https://github.com/apache/pulsar/pull/4646#issuecomment-508307591 > @congbobo184 okay I will review this PR. I will close this PR and then come up with a new PR similar to #4151 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #4646: [pulsar-sql]deserialize the avro schema data by correct reader
sijie commented on issue #4646: [pulsar-sql]deserialize the avro schema data by correct reader URL: https://github.com/apache/pulsar/pull/4646#issuecomment-508310173 @congbobo184 okay. thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4666: Doc release 2.4.0
codelipenghui commented on issue #4666: Doc release 2.4.0 URL: https://github.com/apache/pulsar/pull/4666#issuecomment-508335003 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4669: Don't consider deleted schema when checking compatibility
codelipenghui commented on issue #4669: Don't consider deleted schema when checking compatibility URL: https://github.com/apache/pulsar/pull/4669#issuecomment-508352696 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #4666: Doc release 2.4.0
codelipenghui commented on issue #4666: Doc release 2.4.0 URL: https://github.com/apache/pulsar/pull/4666#issuecomment-508354136 @jiazhai I have addressed your comments, PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services