2018-11-15 09:26:58 UTC - Samuel Sun: Hi looks like we did not revert the "jackson version" in the pulsar/pom.xml from 2.9.7 to 2.8.11, so right now the build for whole project will fail. ---- 2018-11-15 09:27:27 UTC - Samuel Sun: ``` mvn install -DskipTests [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20.192 s [INFO] Finished at: 2018-11-15T17:20:55+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project pulsar-common: Compilation failure: Compilation failure: [ERROR] error reading /Users/haoli/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.7/jackson-core-2.9.7.jar; zip file is empty [ERROR] error reading /Users/haoli/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.7/jackson-core-2.9.7.jar; zip file is empty ``` ---- 2018-11-15 09:38:27 UTC - Samuel Sun: for more details, please check this pr : #2989 ---- 2018-11-15 09:48:24 UTC - Sijie Guo: @Samuel Sun:
1) The error message seems indicating your local cached 2.9.7 version has problem. 2) I also tried to rebuild the project from master. I can’t reproduce your issue. can you try run `mvn clean install -DskipTests`? 3) the reason we didn’t change root jackson version to 2.8.11 is because 2.8.11 is only required for presto. but the remaining the project can use 2.9.7. ---- 2018-11-15 10:00:09 UTC - Shawn Huang: @Shawn Huang has joined the channel ---- 2018-11-15 10:16:19 UTC - Shawn Huang: How to manage the meta data about the distribution of segments in Pulsar? Is there anybody know that?Or any links? ---- 2018-11-15 10:22:43 UTC - Penghui Li: Ledger meta data? ---- 2018-11-15 10:25:40 UTC - hansika luthra: @hansika luthra has joined the channel ---- 2018-11-15 10:25:49 UTC - Shawn Huang: I don't know how a broker get it's topic-part info in bookie. And if a broker is down, how do pulsar transfer its ownship to another broker? ---- 2018-11-15 10:28:20 UTC - Penghui Li: Topic(partition) meta data maintain in zookeeper ---- 2018-11-15 10:29:55 UTC - Penghui Li: If broker crash, client will find a broker by lookup service, and reconnect to a health broker. ---- 2018-11-15 10:46:32 UTC - Shawn Huang: I see. The same to Kafka.Thank you . ---- 2018-11-15 10:48:27 UTC - Penghui Li: Hmm… Broker crash has a deal of difference with Kafka ---- 2018-11-15 10:50:43 UTC - Penghui Li: <https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works> ---- 2018-11-15 10:51:02 UTC - Penghui Li: At Layer 2 - Pulsar Brokers and Topic Ownership ---- 2018-11-15 11:20:29 UTC - Samuel Sun: let me try it again after I clean the cache, thanks for help ---- 2018-11-15 11:23:25 UTC - Samuel Sun: @Sijie Guo, after I ran the mvn clean, the error still go off, not sure if I missed any config ? ---- 2018-11-15 11:25:36 UTC - Samuel Sun: let me try this on another laptop. ---- 2018-11-15 11:28:38 UTC - Mohit S: @Mohit S has joined the channel ---- 2018-11-15 11:36:48 UTC - Samuel Sun: @Sijie Guo fixed after ran mvn dependency:purge-local-repository to force download the lib again. thanks for help +1 : Sijie Guo, Ali Ahmed, Penghui Li ---- 2018-11-15 12:23:29 UTC - Nicolas Ha: I have a few other questions :slightly_smiling_face: (1) schemas ( <https://github.com/apache/pulsar/blob/701f3ea7e3541f61f064167043de04e8a2db5c0f/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/Schema.java> ) If I want to pass arbitrary/dynamic Json schemas that are not POJOs - is there a way to pass something dynamic there? I would like to create a Json schema that specifies a <https://json-schema.org/> , or maybe a `Map<String, String> properties` All examples I’ve seen are `JSONSchema.of`. Or is possible to describe the new schema over JSON? It seems that yes: <https://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats> but I don’t see any example. (2) can I use the SQL from Java? Does the Java client have support to make such queries? Or the REST api? Or should I use an external tool to connect? (or just wait until it is supported - not in a hurry with this one) ---- 2018-11-15 16:40:14 UTC - Harald Gustafsson: Hi, what is the recommended way of verifying that pulsar service is available. I have a problem were the CI/CD pipeline starts a bunch of containers including pulsar, some of the containers starts communicating with pulsar directly, both admin and messaging clients. ---- 2018-11-15 16:40:44 UTC - Harald Gustafsson: I mean would be good to know if pulsar has died or if it is starting up ---- 2018-11-15 16:44:35 UTC - Sijie Guo: 1) currently it doesn’t support passing in a defined map for schema. But the underlying mechanism is there. Should be straightforward to add it. Can you file a GitHub issue for it? I will take a :eyes: 2) it is presto. It has rest endpoints for taking SQL queries. We can add a documentation on how to use it. ---- 2018-11-15 16:47:05 UTC - Sijie Guo: A simple approach is to curl some rest endpoints eg /admin/v2/clusters, if rest endpoint is ready the pulsar service should be ready for use ---- 2018-11-15 16:49:19 UTC - Harald Gustafsson: ok, assumed as much since I had not seen any dedicated health endpoint ---- 2018-11-15 16:51:08 UTC - Sijie Guo: There is an change in current master added a health endpoint. Once that was released, you can use that endpoint ---- 2018-11-15 16:52:20 UTC - Harald Gustafsson: OK, I'll look for that in next release then ---- 2018-11-15 17:12:08 UTC - Christophe Bornet: Do you have an estimated date for the 2.3 release ? ---- 2018-11-15 19:20:10 UTC - Matteo Merli: Target is end of this month ---- 2018-11-15 19:20:28 UTC - Nicolas Ha: filed <https://github.com/apache/pulsar/issues/2992> (will refine if needed) ---- 2018-11-15 19:20:35 UTC - Nicolas Ha: do you want an issue for (2) as well? ---- 2018-11-15 19:29:31 UTC - Sijie Guo: yeah an issues for (2) is great as well :slightly_smiling_face: ---- 2018-11-15 22:40:18 UTC - Christophe Bornet: :+1: ---- 2018-11-16 03:08:39 UTC - yangyujie: @yangyujie has joined the channel ---- 2018-11-16 06:49:10 UTC - tuteng: java.lang.RuntimeException: User class must be in class path at org.apache.pulsar.functions.utils.Reflections.createInstance(Reflections.java:108) ~[java-instance.jar:2.3.0-SNAPSHOT] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:584) ~[java-instance.jar:2.3.0-SNAPSHOT] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:180) ~[java-instance.jar:2.3.0-SNAPSHOT] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:206) [java-instance.jar:2.3.0-SNAPSHOT] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] Caused by: java.lang.ClassNotFoundException: org.apache.pulsar.io.canal.CanalSource at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_191] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_191] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_191] at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_191] at org.apache.pulsar.functions.utils.Reflections.createInstance(Reflections.java:106) ~[java-instance.jar:2.3.0-SNAPSHOT] ... 4 more ---- 2018-11-16 06:49:13 UTC - tuteng: What is the problem? ---- 2018-11-16 06:49:55 UTC - tuteng: command: /data/pulsar/pulsar/bin/pulsar-admin source localrun --archive file:///data/pulsar/pulsar/pulsar-io/canal/target/pulsar-io-canal-2.3.0-SNAPSHOT.nar --classname org.apache.pulsar.io.canal.CanalSource --tenant public --namespace default --name canal2 --destination-topic-name my-topic --source-config-file /data/pulsar/canal-mysql-source-config.yaml --parallelism 1 ---- 2018-11-16 06:51:14 UTC - tuteng: Is there any change ?user Class search path? ---- 2018-11-16 08:42:28 UTC - Sijie Guo: @tuteng: can you check the nar package to see if this class exists in the nar package? ---- 2018-11-16 08:44:20 UTC - tuteng: pulsar-io-canal-2.3.0-SNAPSHOT.nar is exist in /data/pulsar/pulsar/connectors/ ---- 2018-11-16 08:46:29 UTC - tuteng: The previous code can run, and don't run after pull. ---- 2018-11-16 08:47:37 UTC - Sijie Guo: @tuteng are you using latest master? and did you rebuild the nar package? ---- 2018-11-16 08:49:23 UTC - Sijie Guo: > pulsar-io-canal-2.3.0-SNAPSHOT.nar is exist in /data/pulsar/pulsar/connectors/ sorry. my question was “is the class in the nar package”? ---- 2018-11-16 08:51:26 UTC - tuteng: Yes, it has been rebuilt. /tmp/pulsar-nar/pulsar-io-canal-2.3.0-SNAPSHOT.nar-unpacked/ is right ---- 2018-11-16 08:52:18 UTC - Sijie Guo: can you run `tar -tf <nar-package>` and show me the output? ---- 2018-11-16 08:56:47 UTC - tuteng: [tuteng@tuteng target]$ tar -tf pulsar-io-canal-2.3.0-SNAPSHOT.nar tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with failure status due to previous errors ---- 2018-11-16 08:59:24 UTC - tuteng: [tuteng@tuteng connectors]$ tar -tf pulsar-io-rabbitmq-2.3.0-SNAPSHOT.nar tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with failure status due to previous errors ---- 2018-11-16 08:59:25 UTC - tuteng: The same output ---- 2018-11-16 08:59:49 UTC - Sijie Guo: okay can you use `jar -tf`? ---- 2018-11-16 09:00:19 UTC - Sijie Guo: nar package essentially is a jar/zip file. ---- 2018-11-16 09:03:34 UTC - tuteng: ---- 2018-11-16 09:07:54 UTC - Sijie Guo: hmm the nar package looks to be fine. ---- 2018-11-16 09:08:01 UTC - tuteng: name: canal description: canal source and read data from mysql sourceClass: org.apache.pulsar.io.canal.CanalSource ---- 2018-11-16 09:08:10 UTC - tuteng: pulsar-io.yaml file ----
