2019-08-16 10:12:35 UTC - Sijie Guo: HI
----
2019-08-16 10:12:42 UTC - Sijie Guo: sorry I missed the notification
----
2019-08-16 10:13:04 UTC - Sijie Guo: the producer defines the schema because it
produces messages (modify the state of the topic).
----
2019-08-16 10:15:40 UTC - Alexandre DUVAL: Sure but why a subscription doesn't
do the same thing when a topic does not exist?
----
2019-08-16 10:16:16 UTC - Alexandre DUVAL: "why" is a large question, but you
get my point :stuck_out_tongue:.
----
2019-08-16 10:24:37 UTC - Sijie Guo: subscription also trigger creation of a
topic and define the schema for the topic if it doesn’t exist, no?
----
2019-08-16 10:25:01 UTC - Maxim Shatunov: @Maxim Shatunov has joined the channel
----
2019-08-16 10:27:40 UTC - Alexandre DUVAL: In my case it's triggering the
creation of the topic but let the default schema even if custom one is defined.
So it is possibly an issue on the pulsar4s client now that you told me the
normal behavior. Thx.
----
2019-08-16 10:28:24 UTC - Sijie Guo: you might check the behavior of java client
----
2019-08-16 12:37:08 UTC - Alexandre DUVAL: ok, fixed, the implicit in pulsar4s
was just conflict with another, mb. sorry.
----
2019-08-16 13:24:15 UTC - Aaron: Has anyone encountered the exception
`java.lang.UnsatisfiedLinkError:
org.apache.pulsar.shade.org.xerial.snappy.SnappyNative.maxCompressedLength(I)I`
when trying to use SNAPPY compression?
----
2019-08-16 13:37:34 UTC - Alexandre DUVAL: When I delete a topic there is cache
about its schema?
----
2019-08-16 14:08:08 UTC - Richard Sherman: Does pulsar handle zookeeper session
timeouts gracefully?
----
2019-08-16 14:22:12 UTC - Chris Bartholomew: Yes, Pulsar will automatically
redeliver the message. This is a way to manually force redelivery. There is
some debate about whether it should be public here:
<https://github.com/apache/pulsar/pull/51#discussion_r82476470>. It is used a
lot in the tests to, I assume, speed up test execution (no need to wait for
timer to fire).
+1 : Raman Gupta
----
2019-08-16 14:42:32 UTC - Tarek Shaar: Will a subscriber ever miss a message
after establishing a subscription? In other words, if a subscriber comes online
then disconnects for a long time then come back on line in the future, is there
a time out on the subscription (assuming we are still within the message
retention policies).
----
2019-08-16 15:06:37 UTC - balaji: @balaji has joined the channel
----
2019-08-16 15:26:43 UTC - balaji: can someone please give me some example on
exactly once delivery with pulsar and flink combination. I have also posted the
same question in stack overflow.
<https://stackoverflow.com/questions/57523574/exactly-once-delivery-end-to-end-with-pulsar-and-flink>
----
2019-08-16 15:35:37 UTC - Addison Higham: there are few gaps in pulsar to make
that possible yet, first:
- the flink pulsar consumer isn't exactly once yet. For that to happen,
ideally, you need flink to track the position into the subscription so it can
accurately rewind. As it stands, the existing consumer just uses a pretty
normal subscription. If you look at <#CA678TN9G|flink-connector> there are some
more technical details. That said... if you are willing to implement a de-dupe
operator, that just keeps N number of minutes of message ids, it is fairly easy
to de-dupe right after the consumer
- pulsar doesn't yet have transactions like kafka does, but it is actively
being developed. When pulsar has right now is the producer dedupe, which can
get you pretty far, basically, if you can control the sequence number you send
along with your messages into pulsar, then your flink job fails and it sends
those same sequence numbers again, it won't persist the duplicates, see this
blog post for more detail:
<https://jack-vanlightly.com/blog/2018/10/25/testing-producer-deduplication-in-apache-kafka-and-apache-pulsar>
+1 : balaji
----
2019-08-16 15:39:00 UTC - Addison Higham: so you can get effectively once with
flink right now... it just takes more state tracking in your app by de-duping
on input and using more state to build stable sequence numbers, you can get
that wrong, so it isn't bullet proof. I am sure as transactions and flink
integration improves there will be announcements around it
----
2019-08-16 16:53:57 UTC - Ali Ahmed: @balaji @Addison Higham exactly once is
partially a misnomer here are some docs to read through to better understand
things and what is possible in modern distributed systems.
<https://www.slideshare.net/BigDataSpain/endtoend-exactly-once-with-heron-pulsar-by-ivan-kelly-at-big-data-spain-2017>
<https://streaml.io/blog/exactly-once>
----
2019-08-16 16:54:56 UTC - Addison Higham: I am trying to get in the habit of
saying "effectively once" but yes, "exactly once" in the context of what
flink/kafka would use the term
----
2019-08-16 16:59:01 UTC - Alberto Antenangeli: @Alberto Antenangeli has joined
the channel
----
2019-08-16 17:48:25 UTC - balaji: @Ali Ahmed In the slides you posted there is
no concrete code example on how end to end effectively once delivery was
achieved with heron and pulsar.
----
2019-08-16 18:01:41 UTC - Jerry Peng: @balaji you can also achieve
exactly/effectively once using pulsar functions if only have output to a topi
----
2019-08-16 18:01:44 UTC - Jerry Peng: Topic
----
2019-08-16 18:56:47 UTC - Tarek Shaar: Am using Apache Pulsar 2.3.2. I have
enabled message de-duplication in broker.conf and I am setting the producer
name and seq number on each message sent. When I ask for the last seq number I
am getting -1 even though I request it from the same producer name. Are there
any other settings I need to take care of?
----
2019-08-16 20:22:22 UTC - Martin Svensson: Can I create a subscription that
allows me to consume from the start of an existing topic?
----
2019-08-16 20:30:29 UTC - Ali Ahmed: @Martin Svensson You can use the position
arg in consumer or use the reader class
```
Consumer<UserMessagePojo> consumer =
client.newConsumer(JSONSchema.of(UserMessagePojo.class))
.topic(topic)
.subscriptionInitialPosition(position)
.subscriptionName("schema-tester")
.subscribe()) {
```
----
2019-08-16 20:30:55 UTC - Martin Svensson: thanks!
----
2019-08-16 21:16:56 UTC - Ming Fang: are there any plans/talks of supporting
topic/queue aliasing? similar to this
<https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.adm.doc/q020820_.htm>
----
2019-08-16 21:19:56 UTC - Tarek Shaar: If anyone encounters the issue I
mentioned above where the seq # returned is -1, you need to set the initial seq
id when constructing the producer using the producer builder method,
initialSequenceId()
+1 : Chris Bartholomew
----
2019-08-16 23:11:23 UTC - Ali Ahmed: @Ming Fang there are no current plans for
something like this
----
2019-08-16 23:19:12 UTC - Alexandre DUVAL: @Matteo Merli
----
2019-08-16 23:38:41 UTC - Ming Fang: ok thanks
----
2019-08-17 01:52:15 UTC - marcelomfranca: @marcelomfranca has joined the channel
----
2019-08-17 08:04:09 UTC - Kendall Magesh-Davis: Good morning. I’ve deployed
pulsar 2.4 on kubernetes using the public helm chart. I had to cordon/drain a
node which had a bookkeeper on it, this required deleting the local storage on
that node.
Not surprisingly, that bookie is not able to successfully startup again after
being deployed to a different node. It’s complaining of cookie issues.
```07:53:28.259 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are
directories without a cookie, and this is neither a new environment, nor is
storage expansion enabled. Empty directories are
[data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
07:53:28.259 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer -
Shutting down BookieNettyServer
```
```07:53:28.267 [main] ERROR org.apache.bookkeeper.server.Main - Failed to
build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at
org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:470)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at
org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:691)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at
org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at
org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:106)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at
org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301)
~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221)
[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.main(Main.java:203)
[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:314)
[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]```
What is the best practice for resolving this issue?
----
