2019-05-04 22:03:00 UTC - Nicolas Ha: Hello everyone :slightly_smiling_face: long time no see - I am reading this: <https://pulsar.apache.org/docs/en/sql-getting-started/> Is it accessible from Java too? ---- 2019-05-04 22:04:32 UTC - Ali Ahmed: you can use the presto jdbc client in java to run the queries ---- 2019-05-04 22:05:00 UTC - Nicolas Ha: Oh great news - what should I put for the connection? ---- 2019-05-04 22:05:45 UTC - Nicolas Ha: I mean - is there a default port? Auth? ---- 2019-05-04 22:06:28 UTC - Ali Ahmed: it will be something like this ```jdbc:<presto://192.168.203.141:8080>``` ---- 2019-05-04 22:06:48 UTC - Ali Ahmed: there should presto jbbc doc that you can reference ---- 2019-05-04 22:12:28 UTC - Nicolas Ha: thanks :slightly_smiling_face: ---- 2019-05-04 22:12:52 UTC - Nicolas Ha: (I was looking for something in pulsar - that’s why I couldn’t find it :slightly_smiling_face: ) ---- 2019-05-04 22:34:42 UTC - Nicolas Ha: anyone knows if presto runs be default there: <https://hub.docker.com/r/apachepulsar/pulsar/tags> ? (can’t find the source of that) ---- 2019-05-04 22:35:29 UTC - Nicolas Ha: ah - spoke too soon, looks like <https://github.com/apache/pulsar/blob/master/docker/pulsar/Dockerfile> (which doesn’t seem to have it) ---- 2019-05-04 22:41:07 UTC - Matteo Merli: @Nicolas Ha It’s included in that image, you just need to start it with `bin/pulsar sql-worker start` ---- 2019-05-04 22:41:23 UTC - Matteo Merli: that will start a single node presto instance ---- 2019-05-04 22:43:16 UTC - Nicolas Ha: I am trying to do that:```FROM apachepulsar/pulsar:latest
RUN bin/pulsar sql-worker start ``` =>```docker build . Sending build context to Docker daemon 2.048kB Step 1/2 : FROM apachepulsar/pulsar:latest ---> fd323a11901e Step 2/2 : RUN bin/pulsar sql-worker start ---> Running in c35536078dde -- Invalid command ‘sql-worker’ -- Use ‘bin/pulsar help’ to get a list of valid commands The command ‘/bin/sh -c bin/pulsar sql-worker start’ returned a non-zero code: 1``` ---- 2019-05-04 22:44:00 UTC - Nicolas Ha: oh - I need to do it in the `command` perhaps ---- 2019-05-04 22:44:44 UTC - Matteo Merli: can you try with `apachepulsar/pulsar:2.3.1` ? ---- 2019-05-04 22:45:15 UTC - Nicolas Ha: 2.3.1 worked ---- 2019-05-04 22:46:29 UTC - Nicolas Ha: ```docker build . Sending build context to Docker daemon 2.048kB Step 1/2 : FROM apachepulsar/pulsar:2.3.1 ---> 30fb009acc89 Step 2/2 : RUN bin/pulsar sql-worker start ---> Running in f67d7d9ca1ac Started as 23 Removing intermediate container f67d7d9ca1ac ---> 01f08af2b851 Successfully built 01f08af2b851``` ---- 2019-05-04 23:32:10 UTC - Nicolas Ha: ```presto> show catalogs; ERROR: failed to open pager: Cannot run program "less": error=2, No such file or directory Catalog --------- pulsar system (2 rows) Query 20190504_233044_00001_75ftd, FINISHED, 1 node Splits: 19 total, 19 done (100.00%) 0:03 [0 rows, 0B] [0 rows/s, 0B/s] presto> show schemas in pulsar; Query 20190504_233103_00002_75ftd failed: Failed to get schemas from pulsar: Connection refused ``` ^is the Connection refused expected? ---- 2019-05-04 23:32:51 UTC - Nicolas Ha: (I ran the docker image, `bin/pulsar sql-worker run` in one terminal and `bin/pulsar sql` in the above) ---- 2019-05-04 23:34:15 UTC - Nicolas Ha: I did it so because it seems I need to wait for the broker before starting `bin/pulsar sql-worker run` (not sure about that at the moment) ---- 2019-05-04 23:35:21 UTC - Nicolas Ha: `show schemas in system;` works though, so it may (?) be expected? ---- 2019-05-05 03:53:00 UTC - Jianfeng Qiao: Anyone ever tried valgrind on Pulsar broker? ---- 2019-05-05 04:17:56 UTC - Sijie Guo: you mean running broker on valgrind? ---- 2019-05-05 04:18:33 UTC - Sijie Guo: did you expose the port to outside of the docker container? ---- 2019-05-05 06:10:04 UTC - Jianfeng Qiao: Yes ---- 2019-05-05 06:10:37 UTC - Sijie Guo: I am not aware of anyone doing that yet. ---- 2019-05-05 06:13:58 UTC - Jianfeng Qiao: I'm running openmessaging benchmark tool against pulsar. I find a lot of memory will be used by broker and in the end I get ' java.lang.OutOfMemoryError: Java heap space' error. ---- 2019-05-05 06:14:37 UTC - Jianfeng Qiao: I suspect there might be memory leak somewhere. ---- 2019-05-05 06:21:10 UTC - Sijie Guo: did you check if you give enough memory to broker? ---- 2019-05-05 06:31:49 UTC - Jianfeng Qiao: -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -Dio.netty.leakDetectionLevel=advanced -Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024 ---- 2019-05-05 06:32:45 UTC - Jianfeng Qiao: All these configuration are by default, I do not change them. ---- 2019-05-05 06:36:53 UTC - Sijie Guo: I see. it would be good to check if your bookie write latency. If you have a slow bookie, the add entries might be accumulating at your broker. ---- 2019-05-05 06:44:53 UTC - Jianfeng Qiao: There is no thottling for add entries? ---- 2019-05-05 07:00:53 UTC - Sijie Guo: there are settings at bookie side of throttling the add entries. I guess the throttling doesn’t kick in before broker OOM. I am not sure about openmessaging client and how it configures the client. It might be worth checking what is the maxPendingRequests. Alternatively, I would suggest u using pulsar-perf to get some baseline latency number. ---- 2019-05-05 07:31:37 UTC - chenyongxi: @chenyongxi has joined the channel ----
