2020-09-15 14:17:24 UTC - Brent Evans: @Brent Evans has joined the channel
----
2020-09-15 14:42:46 UTC - Evan Furman: @Tim Haak fyi
----
2020-09-15 14:51:07 UTC - Shivji Kumar Jha: Hi I am following the *python*
pulsar client and in particular following
<https://github.com/merlimat/pulsar/blob/73127e705b63a5cbecaadc11757c7d0aaed7e994/pulsar-client-cpp/python/schema_test.py#L185|this
test>. Is python client aware of the avro schema *version*?
In particular, because avro needs the read and write schema to be same, i want
to get the exact schema version from message and then decode the message with
this schema (not a schema supplied to consumer instance). I can do this in java
client for instance, not in python.
----
2020-09-15 15:41:50 UTC - Brent Evans: Hi, feel free to point me to another
channel if this one isn't appropriate. We're currently using Pulsar as our
message broker and are seeing issues with the Proxy returning 504s
intermittently. I've detailed it below, anyone able to advise?
```Our Setup:
Pulsar Services are on 2.6.0. These are all running on EC2 instances in AWS.
Pulsar Java Client is on 2.6.0. Services using these are running in ECS in
Docker Containers.
1 Pulsar Proxy
2 Pulsar Broker
3 Zookeeper
3 Bookkeeper
- Key Shared subscription across 2 services.
- On a 1 minute schedule we query for topics on a namespace using
pulsarAdminClient.namespaces().getTopics(namespace)
- At times this throws:
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
HTTP 504 Gateway Timeout
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:209)
at
org.apache.pulsar.client.admin.internal.NamespacesImpl$3.failed(NamespacesImpl.java:162)
at
org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:886)
at
org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:873)
at
org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:228)
- At this point a restart of the impacted service is required.
- Resource usage seems fine across Pulsar.
- Pulsar Proxy logs show nothing obvious, Pulsar Broker contains no error
messages.
Seems similar to a previous issue:
<https://github.com/apache/pulsar/issues/3702> ```
----
2020-09-15 15:51:51 UTC - Allen ONeill: We are in the middle of implementing
function state @Devin G. Bost .... ping me in about 10 days and I'll have
comments on it.
+1 : Devin G. Bost
----
2020-09-15 16:01:52 UTC - Addison Higham: @Brent Evans how is your pulsar proxy
discovering your brokers? As you can see in this doc
<https://pulsar.apache.org/docs/en/administration-proxy/#configure-the-proxy>
there are two mechanisms for discovering brokers, there are currently some
issues with zookeeper discovery, which results in the proxy not having the
correct list of brokers
----
2020-09-15 16:07:04 UTC - Brent Evans: We are currently using Zookeeper for
discovery. Are brokerUrls the recommended option?
----
2020-09-15 16:07:26 UTC - Raman Gupta: I finally restarted all the Pulsar
brokers, bookies, and proxies, and then things started working ok -- redelivery
counts incremented, stuff went to DLQ as expected. Weird.
----
2020-09-15 16:08:33 UTC - Addison Higham: @Enrico yes, it does, see
<https://pulsar.apache.org/api/cpp/2.6.0-SNAPSHOT/classpulsar_1_1_consumer_configuration.html#ace6b2f1da486fcffacc4e877e5533268>
and the `ConsumerType` enum
----
2020-09-15 16:09:04 UTC - Addison Higham: `ConsumerKeyShared` should enable key
shared mode :slightly_smiling_face:
----
2020-09-15 16:12:53 UTC - Addison Higham: Not with the existing function API,
the functions always run across the same cluster. What you may want to consider
for this sort of use case is Pulsar's built in replication. This way, your
function can simply write to your same cluster but the topic will be replicated
automatically between clusters.
In the docs, it discusses needing a global zookeeper, but this isn't a strict
requirement, this doc
<https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930> and this
video <https://www.youtube.com/watch?v=0UoCHpdG9E0> discuss how you can make
replication happen without needing a shared zookeeper
+1 : Rattanjot Singh
----
2020-09-15 16:17:32 UTC - Addison Higham: yes, fixing the proxy with zookeeper
discovery should be hopefully fixed soon but it is coming along with some
bigger reworks of zookeeper session handling.
Since you mentioned you are running on AWS, one option is simply to put the
brokers behind a load balancer (perhaps an NLB) and point the proxy at that LB.
The role of the proxy is twofold, one is for discovery, where it can basically
ask any broker for the answer (which sometimes involves redirects). After a
client has discovered the correct broker, it passes this information along when
it connects to the proxy.
What this means in practice is that you can use an LB for discovery but for
known connections to a broker, the proxy will just bypass the LB and talk
directly to the brokers
----
2020-09-15 16:18:14 UTC - Devin G. Bost: Sounds great. Thanks!
----
2020-09-15 16:18:42 UTC - Brent Evans: OK, thanks for this info, I'll see what
we can do and hopefully this resolves it :+1:
----
2020-09-15 16:51:01 UTC - Evan Furman: @Addison Higham the unload kick-started
the consumers again
----
2020-09-15 16:52:53 UTC - Addison Higham: okay... hrm... that is strange, if
you can get logs during the initial period and attach to a github issue, that
would be helpful. That is the first time I have seen that sort of behavior (and
I have taken down bookies many times )
----
2020-09-15 16:53:59 UTC - Evan Furman: ok, I’ll grab broker logs. I also plan
on destroying this cluster, rebuilding it, and trying again.
----
2020-09-15 17:23:38 UTC - Sankararao Routhu: sure @Penghui Li thank you
----
2020-09-15 17:25:25 UTC - Evan Furman: Thanks for the help @Addison Higham. The
issue is here <https://github.com/apache/pulsar/issues/8069>
----
2020-09-15 17:26:52 UTC - Addison Higham: thanks! sorry for the pain! if you
end up repeating the test and can make it consistently happen, let me know. We
could maybe even jump on a call and debug the issues
+1 : Evan Furman
----
2020-09-15 17:40:53 UTC - Sankararao Routhu: Hi, Is it possible to set
different TTL or backlog quota for replicated messages alone? We want to expire
messages that are replicated to remote region after configured time when the
messages are not consumed on remote region.
----
2020-09-15 18:35:44 UTC - Rattanjot Singh: Trying to start pulsar function
worker separately . Getting this. Any idea how to resolve this
```11:25:46.251 [main-SendThread(<ip1>:2281)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
<ip1>:2281. Will not attempt to authenticate using SASL (unknown error)
11:25:46.253 [main-SendThread(<ip1>:2281)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established, initiating
session, client: /10.80.117.93:46614, server: <ip1>:2281```
----
2020-09-15 18:41:31 UTC - Addison Higham: I don't believe that is currently
possible if you are doing replication that shares a global zookeeper, as all
the policies are stored in the global zookeeper.
If you did replication that didn't use zookeeper (see
<https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930> for details on
that) then you could make that happen
----
2020-09-15 18:56:24 UTC - Alan Hoffmeister: Hello folks, can I implement a
request/reply queue pattern with Pulsar? Does someone have some material that I
could read around this subject?
----
2020-09-15 19:09:57 UTC - Addison Higham: There isn't quite as much of a
standard around how to do that in Pulsar (compared to say rabbitmq), nor do I
know of any documentation, but the tools for implementing it are all there.
Messages can have individual metadata in the form of properties, see
<https://pulsar.apache.org/api/client/2.6.0-SNAPSHOT/> for an example in the
java API.
With a property like `ReplyTo` with the fully qualified topic name `RequestId`
as a unique ID, you could implement an RPC like wrapper. You would likely want
to use exclusive subscriptions and depending on your needs, you topic names may
need to include host identifiers.
----
2020-09-15 19:13:49 UTC - Alan Hoffmeister: @Addison Higham thx! Should I go
with one reply topic per host? Not sure if that or one non-persistent topic per
request
----
2020-09-15 19:16:24 UTC - Addison Higham: assuming you want to ensure the same
host that send the messages process the reply, then using a reply topic per
host makes sense
+1 : Alan Hoffmeister
----
2020-09-16 01:35:01 UTC - KK Raccoon: @KK Raccoon has joined the channel
----
2020-09-16 06:28:41 UTC - Shivji Kumar Jha: @Penghui Li? Dont see version info
in python client. Can you confirm on the same?
----
2020-09-16 06:52:13 UTC - Sankararao Routhu: thanks @Addison Higham ,are there
any downsides if we enable replication with out global zookeeper and vice versa?
----
2020-09-16 07:22:50 UTC - Enrico: Ok but if i would assign specific key for
specific consumer it is possible?
----
2020-09-16 07:26:33 UTC - Gilles Barbier: It does not appear stable right now
unfortunately
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1596541103177700>
----
2020-09-16 07:32:54 UTC - Gilles Barbier: Same issues with counters actually.
When the states crash, counters are providing obviously wrong results
----
2020-09-16 07:33:51 UTC - Penghui Li: Sorry, I’m not familiar with python
client, I think you can ask in the general channel.
----
2020-09-16 08:08:31 UTC - Linton: @Linton has joined the channel
----
