2019-09-23 16:57:00 UTC - Chris Bartholomew: @Mahesh You cannot directly delete
a globally replicated topic. See here for more details:
<https://pulsar.apache.org/docs/en/administration-geo/#deleting-a-geo-replication-topic>
----
2019-09-23 16:57:48 UTC - Chris Bartholomew: If you need to delete the topic,
you have to first update the cluster list so that the local cluster is the only
one on the list (effectively removing the geo-replication). Then you can delete
it.
----
2019-09-23 17:07:20 UTC - Chris Bartholomew: @Junli Antolovich On the
performance question, I recently wrote a detailed blog post comparing the
latency between Pulsar and Kafka:
<https://kafkaesque.io/performance-comparison-between-apache-pulsar-and-kafka-latency/>
----
2019-09-23 17:38:03 UTC - Zhenhao Li: this page is missing
<https://pulsar.apache.org/docs/en/concepts-overview/concepts-schema-registry.md>
----
2019-09-23 17:38:33 UTC - Zhenhao Li: is there a guide to install the schema
registry?
----
2019-09-23 17:39:36 UTC - Addison Higham:
<https://pulsar.apache.org/docs/en/concepts-schema-registry/> <- new link
----
2019-09-23 17:39:41 UTC - Addison Higham: that link looks to be old doc format
----
2019-09-23 17:40:57 UTC - Addison Higham: @Zhenhao Li and the schema registry
is included in the broker, no need to install anything, just starting using it,
but it is best supported in java right now
----
2019-09-23 17:42:31 UTC - Zhenhao Li: I've used Confluent schema registry
before. do you follow the same design?
----
2019-09-23 17:42:53 UTC - Matteo Merli: No
----
2019-09-23 17:43:24 UTC - Zhenhao Li: what I didn't like about Confluent is
that they use a special binary Avro format which is not compatible with the
Avro specs
----
2019-09-23 17:43:31 UTC - Matteo Merli: The schema is integrated in Pulsar
brokers. It’s not an external component and it will be able to enforce the
schema on the topic
----
2019-09-23 17:43:39 UTC - Zhenhao Li: I hope you guys don't make such mistake
:wink:
----
2019-09-23 17:44:03 UTC - Matteo Merli: We use regular Avro
+1 : Zhenhao Li
----
2019-09-23 17:45:00 UTC - Zhenhao Li: since it is internal to Pulsar. can I use
it outside Pulsar?
----
2019-09-23 17:45:37 UTC - Zhenhao Li: for example, if I am to build an event
sourced backend with Cassandra as event store
----
2019-09-23 17:45:56 UTC - Zhenhao Li: can I still use Pulsar for schema
management?
----
2019-09-23 17:45:57 UTC - Matteo Merli: You could but it won’t be much useful,
in its current form
----
2019-09-23 17:46:14 UTC - Matteo Merli: eg. the validation is only performed on
the topics
----
2019-09-23 17:46:26 UTC - Zhenhao Li: I see
----
2019-09-23 17:46:54 UTC - Zhenhao Li: thanks! is there any future plan to
extend it?
----
2019-09-23 17:51:44 UTC - Matteo Merli: I don’t think it was discussed in the
past, though it might be an interesting are to explore
----
2019-09-23 17:54:10 UTC - Zhenhao Li: Hortonworks has been developing its SR as
a self-standing service <https://github.com/hortonworks/registry>
----
2019-09-23 17:57:15 UTC - Matteo Merli: Yes, though for us it represented few
challenges:
* One external system
* That requires external storage (eg. MySQL)
* Not directly integrated in Pulsar clients
That’s why we preferred doing it internally
----
2019-09-23 17:59:48 UTC - Zhenhao Li: fair points
----
2019-09-23 17:59:50 UTC - Addison Higham: my plan on that: use pulsar schema
registry for pulsar but as we slowly move towards "one schema registry to rule
them all" we will make tooling to support synchronizing the schema into pulsar
+1 : Zhenhao Li, Matteo Merli, Ali Ahmed
----
2019-09-23 18:00:27 UTC - Zhenhao Li: I guess for non-streaming services, I
still need to use the confluent SR because it is easy to setup
----
2019-09-23 18:02:48 UTC - Matteo Merli: what are the API operations you’re
using from that?
----
2019-09-23 18:06:34 UTC - Zhenhao Li: on the high level, to enforce
compatibility of schema evolution
----
2019-09-23 18:07:56 UTC - Matteo Merli: that is enforced when doing a schema
upload operation, though it would be better to expose a checkSchema() operation
----
2019-09-23 18:13:00 UTC - Zhenhao Li: Confluent has good libraries to integrate
their schema registry and Avro (de)serialization. but they kind of vendor
lock-in those who use it because of its special binary encoding
----
2019-09-23 18:14:15 UTC - Zhenhao Li: I think the market is missing a general
purpose schema registry and (de)serialization lib for it but with standard Avro
----
2019-09-23 18:14:38 UTC - Matteo Merli: Got it. Please take a look at the
current APIs of Pulsar schema and check what would be required to make it
useful for other systems
+1 : Zhenhao Li, Ali Ahmed
----
2019-09-23 18:44:08 UTC - Tarek Shaar: How many producer and consumer TCP
connections is good for a 1M messages per day throughput? I know there is no
right or wrong formula but when should I think of increasing the number of TCP
connections?
----
2019-09-23 18:53:06 UTC - Matteo Merli: 1 TCP connection will be enough.
> when should I think of increasing the number of TCP connections?
Having more connection will give more parallelism, as network channels and for
CPUs (eg. each connection is associated with 1 thread from the pool, therefore
with more connections, more threads can share the load).
----
2019-09-23 19:37:14 UTC - Zhenhao Li: where can I configure the parent path of
`data` and `log`?
----
2019-09-23 19:38:30 UTC - Zhenhao Li: on NixOS we got
```
21:32:15.245 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to
start pulsar service.
java.nio.file.FileSystemException:
/nix/store/1kvfy977ymmf28mh0w7c3alzwz27p84g-apache-pulsar-2.4.1/data: Read-only
file system
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
~[?:1.8.0_212]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[?:1.8.0_212]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~[?:1.8.0_212]
at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
~[?:1.8.0_212]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_212]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
~[?:1.8.0_212]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_212]
at
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runZookeeper(LocalBookkeeperEnsemble.java:175)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
at
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:394)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:2.4.1]
```
----
2019-09-23 19:39:07 UTC - Zhenhao Li: so we need to make the parent path
configurable
----
2019-09-23 19:42:35 UTC - Ali Ahmed: @Zhiyong Li ```
journalDirectory=data/bookkeeper/journal
``` in conf/bookkeeper.conf
+1 : Zhenhao Li
----
2019-09-23 19:43:06 UTC - Ali Ahmed: ```# the directory where the snapshot is
stored.
dataDir=data/zookeeper
``` in conf/zookeeper.conf
----
2019-09-23 19:44:36 UTC - Ali Ahmed: @Addison Higham @Zhiyong Li @Matteo Merli
We can just map
<https://github.com/confluentinc/schema-registry/blob/d546a35d27/client/src/main/java/io/confluent/kafka/schemaregistry/client/SchemaRegistryClient.java>
to pulsar schema registry
----
2019-09-23 19:53:15 UTC - Zhenhao Li: @Ali Ahmed what is the config file name
for standalone starter?
----
2019-09-23 20:15:30 UTC - Tarek Shaar: Thanks got it
----
2019-09-23 20:23:27 UTC - David Kjerrumgaard: standalone.conf is the config
file for running Pulsar in standalone mode.
+1 : Zhenhao Li
----
2019-09-24 03:36:39 UTC - Poule: my pulsar_standalone is on ip 192.168.5.55
i want to tell sql-worker to use it
it's not clear from the doc
----
2019-09-24 03:37:49 UTC - Poule: is it --node-config=pulsar.properties
----
2019-09-24 04:52:58 UTC - Poule: k I found out
----
2019-09-24 05:28:37 UTC - Poule: how can I tell SQL to connect to bookkeeper at
192.168.5.55:3181 ?
It tries to connect to 127
```
bookkeeper-io-8-7 org.apache.bookkeeper.proto.PerChannelBookieClient
Could not connect to bookie: [id: 0x176ae5db]/127.0.0.1:3181, current state
CONNECTING :
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:
/127.0.0.1:3181
```
----
2019-09-24 05:55:57 UTC - Sijie Guo: @Poule are you starting a standalone?
----
2019-09-24 06:10:27 UTC - Zhenhao Li: hi, where is
`wal_dir=data/standalone/bookkeeper0/current/ledgers?` configured? I can't find
it
----
2019-09-24 06:12:19 UTC - Sijie Guo: @Zhenhao Li in standalone mode, you can
specify the location via `--bookkeeper-dir`. You can run “bin/pulsar standalone
--help” to get the list of available options.
+1 : Zhenhao Li
----
2019-09-24 06:24:43 UTC - Poule: @Sijie Guo my docker pulsar standalone is
started on 192.168.5.55
my sql-worker is connected to it on 192.168.5.55, I can show tables but when I
try to query, it gives error trying to contact bookie on 127.0.0.1:3181 but it
should instead try 192.168.5.55:3181
----
2019-09-24 06:27:00 UTC - Sijie Guo: in standalone alone mode, the bookie is
advertised its local address which is 127.0.0.1. the presto sql worker access
the bookies directly, which will bypass brokers. with that being said, your
presto worker will have to be in the same network as bookies. you can not have
a presto worker running in a separated network.
----
2019-09-24 06:34:16 UTC - Poule: ok so sql-worker reads the advertised address
127.0.0.1 and use it to connect
----
2019-09-24 06:35:12 UTC - Poule: if i understand correctly
----
2019-09-24 06:37:44 UTC - Poule: would not be possible to have a
pulsar.bookkeeper-uri= property for sql-worker to read at startup?
----
2019-09-24 06:38:05 UTC - Poule: in the file pulsar.properties
----
2019-09-24 06:38:47 UTC - Poule: to overwrite the bookie advertised address
----
2019-09-24 06:57:44 UTC - Poule: For now I forwarded 127.0.0.1:3181 to
192.168.5.55:3181 with netcat
----
2019-09-24 07:11:32 UTC - Sijie Guo: It doesn’t work in that way. Bookies
register themselves in zookeeper, the presto worker discover them from zk.
----
2019-09-24 07:12:38 UTC - Sijie Guo: So you need to let bookies advertise
192.168.5.55. But you can’t simply change the advertise address because broker
needs to access bookies as well.
----
2019-09-24 07:13:17 UTC - Sijie Guo: With that being said, there is no
workaround if you run a stand-alone in docker and presto works outside docker
----
2019-09-24 07:13:58 UTC - Sijie Guo: Feel free to create an issue and we can
look into how to improve this case
hugging_face : Poule
----
2019-09-24 07:35:21 UTC - Poule: thanks for the explanation. No need to improve
this for now I think
----
