2019-04-12 11:43:51 UTC - Kim Christian Gaarder: @Kim Christian Gaarder has joined the channel ---- 2019-04-12 12:13:33 UTC - Kim Christian Gaarder: Pulsar documentation state that ordering guarantees only apply per partition for partitioned topics. I wanted to know whether this applies only to write-ordering, and whether there is still is a read-ordering guarantee here that ensures that messages read by multiple consumers from a partitioned topic will be seen in the same order across partitions, consumers and subscriptions? ---- 2019-04-12 12:13:50 UTC - Kim Christian Gaarder: I am investigating using a partitioned Pulsar topic as a transaction log in a distributed system and wanted to know whether this will ensure consistent read-ordering despite that writes across partitions may get re-ordered. i.e. Will multiple exclusive subscriptions to the partioned topic all see messages in the same order, and also if the subscribe and consume at very different times (assuming unlimted message retention) ---- 2019-04-12 12:31:48 UTC - Adam F: I'm also interested in using Pulsar as a transaction log.
Looked at the docs quickly and saw this: <https://pulsar.apache.org/docs/latest/cookbooks/PartitionedTopics/#Routingmodes-6b001> (that info appears to be identical to another page <https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#Routingmodes-sq0su6>) ---- 2019-04-12 12:33:32 UTC - Adam F: At the link, there is a table that lays out 3 different routing modes, each of which provides a different ordering guarantee, ranging from per-message-key bucket ordering to per-producer ordering to no guarantees ---- 2019-04-12 12:35:42 UTC - Adam F: (note: I have no practical Pulsar experience, just have been reviewing the docs) ---- 2019-04-12 12:39:57 UTC - Adam F: On a related note, this person's blog reports on their experience doing chaos testing with Pulsar and other similar distributed systems, and other posts talk about designing transaction logs, etc; <https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-messages-on-an-apache-pulsar-cluster> ---- 2019-04-12 12:42:25 UTC - Adam F: (spoiler alert: "...regarding the claims of zero message loss and ordering guarantees, Apache Pulsar has come out flawless.") ---- 2019-04-12 12:46:07 UTC - Adam F: I'm interested to hear more about your investigations @Kim Christian Gaarder -- use-case and other components you're taking a look at ---- 2019-04-12 14:33:32 UTC - Roman: @Roman has joined the channel ---- 2019-04-12 14:49:07 UTC - Fredrick P Eisele: I am running Pulsar-Dashboard with ... export SERVICE_URL=<http://ledger2.scope.vu.edu:8080/> ...id does not seem to be connecting to my pulsar cluster. I look in my broker.conf where... webServicePort=8080 ...any ideas on how to proceed in debugging the problem? If I point my browser at SERVICE_URL ... <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /. Reason: <pre> Not Found</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.12.v20180830</a><hr/> </body> </html> ---- 2019-04-12 14:55:05 UTC - David Kjerrumgaard: @Fredrick P Eisele Can you verify that the service is running on the ledger2 server and that port 8080 is open? ---- 2019-04-12 15:24:51 UTC - Fredrick P Eisele: @David Kjerrumgaard Here is the status of the ledger2 service $ sudo systemctl status pulsar.service ● pulsar.service - Pulsar Broker Service Loaded: loaded (/etc/systemd/system/pulsar.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-04-11 22:17:23 UTC; 17h ago Process: 8820 ExecStart=/opt/services/apache-pulsar/latest/bin/pulsar-daemon start broker (code=exited, status=0/SUCCESS) Main PID: 8836 (java) Tasks: 229 (limit: 4915) CGroup: /system.slice/pulsar.service └─8836 /usr/bin/java -cp /opt/services/apache-pulsar/latest/conf:::/opt/services/apache-pulsar/latest/lib/*: -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefPr Apr 11 22:17:20 scope-ledger-2 systemd[1]: Starting Pulsar Broker Service... Apr 11 22:17:20 scope-ledger-2 pulsar-daemon[8820]: doing start broker ... Apr 11 22:17:20 scope-ledger-2 pulsar-daemon[8820]: starting broker, logging to /opt/services/apache-pulsar/latest/logs/pulsar-broker-scope-ledger-2.log Apr 11 22:17:23 scope-ledger-2 systemd[1]: Started Pulsar Broker Service. ---- 2019-04-12 15:26:26 UTC - David Kjerrumgaard: @Fredrick P Eisele Can you run a netstat on that host and confirm that process id 8820 is listening on port 8080? ---- 2019-04-12 15:36:20 UTC - Fredrick P Eisele: @David Kjerrumgaard The firewall says: ALLOW IPv4 8080/tcp from 0.0.0.0/0 so that looks like it is open and the pro $ netstat -tulpn | grep 8836 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8836/java tcp 0 0 0.0.0.0:6650 0.0.0.0:* LISTEN 8836/java $ nmap -Pn ledger2 Starting Nmap 7.70 ( <https://nmap.org> ) at 2019-04-12 10:37 CDT Nmap scan report for ledger2 (129.59.107.170) Host is up (0.0010s latency). Not shown: 985 filtered ports PORT STATE SERVICE 22/tcp open ssh 1720/tcp open h323q931 6000/tcp closed X11 6001/tcp closed X11:1 6002/tcp closed X11:2 6003/tcp closed X11:3 6004/tcp closed X11:4 6005/tcp closed X11:5 6006/tcp closed X11:6 6007/tcp closed X11:7 6009/tcp closed X11:9 6025/tcp closed x11 6059/tcp closed X11:59 8080/tcp open http-proxy 8443/tcp closed https-alt ---- 2019-04-12 15:40:27 UTC - Fredrick P Eisele: @David Kjerrumgaard I am running pulsar-dashboard in a docker. $ docker run -p 80:80 -e SERVICE_URL=$SERVICE_URL apachepulsar/pulsar-dashboard ---- 2019-04-12 18:51:58 UTC - Devin G. Bost: I am attempting to migrate our usage of the Admin CLI to the Java Admin API, and I am confused about how to properly setup the `Map<String, Object> configs` property of the `SourceConfig` class. When creating a source with the Admin CLI, I have a conf file that contains these contents: ```configs: bootstrapServers: obfuscated consumerConfigProperties: auto.offset.reset: latest sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true useTicketCache=false serviceName="kafka" principal="[email protected]" useKeyTab=true keyTab="/pulsar/conf/auth/pulsar_runtime_dev.keytab" client=true; sasl.kerberos.service.name: kafka security.protocol: SASL_PLAINTEXT groupId: obfuscated-log-group topic: obfuscated-log-topic``` I want to load these into the `Map<String, Object> configs` property of the `SourceConfig` class with the Jackson YAML parser. However, it's not clear how I should handle the hierarchy of the configs. What should the Map<String, Object> contain? ---- 2019-04-12 18:55:26 UTC - Devin G. Bost: Perhaps, is there a unit test somewhere with an example? ---- 2019-04-12 19:13:59 UTC - David Kjerrumgaard: @Devin G. Bost <https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/test/resources/kafkaSourceConfig.yaml> ---- 2019-04-12 19:16:15 UTC - David Kjerrumgaard: I don't think you need the leading `configs:` unless are nesting it inside a larger config that specifies the general source configs such as topic, namespace, parallelism, etc. ---- 2019-04-12 19:16:53 UTC - Devin G. Bost: Thanks for the link and guidance! ---- 2019-04-12 19:17:07 UTC - Devin G. Bost: That's perfect. ---- 2019-04-12 19:54:47 UTC - Sree Vaddi: Apache Heron, Fastest Stream Processing Engine : One Year Anniversary celebrations. you are all most welcome to join. <https://www.meetup.com/Apache-Heron-Bay-Area/events/xvmlqqyzgbtb/> ---- 2019-04-12 22:03:25 UTC - Fredrick P Eisele: What should connecting to the Pulsar HTTP connection look like? Mine looks like this... <http://ledger2:8080/> <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /. Reason: <pre> Not Found</pre></p><hr><a href="<http://eclipse.org/jetty>">Powered by Jetty:// 9.4.12.v20180830</a><hr/> </body> </html> That does not seem right to me. ---- 2019-04-12 22:05:59 UTC - David Kjerrumgaard: @Fredrick P Eisele So are pulsar and the dashboard running in separate docker containers? ---- 2019-04-12 22:06:37 UTC - David Kjerrumgaard: @Fredrick P Eisele Is the <http://ledger2.scope.vu.edu|ledger2.scope.vu.edu> server a physical machine? ---- 2019-04-12 22:15:36 UTC - Fredrick P Eisele: No, it is a vm on an openstack cloud. ---- 2019-04-12 22:17:18 UTC - Fredrick P Eisele: @David Kjerrumgaard They are separate; pulsar is running on an openstack vm and pulsar-dashboard is running in a docker container on my desktop machine, (not on the openstack cloud). ---- 2019-04-12 22:22:25 UTC - Fredrick P Eisele: @David Kjerrumgaard What do you get when you point your browser at a Pulsar http port? (I am trying to take pulsar-dashboard out of the equation.) ---- 2019-04-12 22:26:52 UTC - Ali Ahmed: @Fredrick P Eisele it should 404 at the base level try this “/admin/v2/tenants” ---- 2019-04-12 22:39:42 UTC - Fredrick P Eisele: @Ali Ahmed excellent <http://ledger2:8080/admin/v2/tenants> works fine. ---- 2019-04-12 22:40:03 UTC - Ali Ahmed: @Fredrick P Eisele are you running master or 2.3 release ? ---- 2019-04-12 22:42:04 UTC - Fredrick P Eisele: pulsar is the 2.3.0 release but pulsar-dashboard is <https://github.com/apache/pulsar/tree/master> . ---- 2019-04-12 22:43:10 UTC - Ali Ahmed: I would try the 2.3.0 dashboard ---- 2019-04-12 22:43:53 UTC - Ali Ahmed: check the ```docker-compose/standalone-dashboard/docker-compose.yml ``` ---- 2019-04-12 22:44:17 UTC - Ali Ahmed: compose file to see how to use the dashboard container image ---- 2019-04-12 22:53:00 UTC - Fredrick P Eisele: @Ali Ahmed thanks I will give that a try. (tomorrow) ---- 2019-04-13 08:15:37 UTC - Kevin Brown: @Kevin Brown has joined the channel ----
