2020-08-03 13:29:16 UTC - SamMas: @SamMas has joined the channel
----
2020-08-03 16:13:25 UTC - Addison Higham: you may need to just remove the `\`
and the newline as that is a bash-ism for putting a command across multiple
lines
----
2020-08-03 16:34:10 UTC - Matt Mitchell: Hi. I’m investigating an error from an
application running in Kubernetes, where we have 3 broker and 3 bookkeeper
instances. The app error is: `Failed to create producer: Topic is temporarily
unavailable` - the topic is non-persistent. I’m still in the process of
determining if the Pulsar logs are still available and will share if they are
(if anyone is interested). I’m wondering what causes this issue and how to deal
with it?
----
2020-08-03 16:51:07 UTC - Gilles Barbier: @Jerry Peng I confirm. You have to
“rewind” the ByteBuffer provided by the state to be able to read it
----
2020-08-03 16:58:01 UTC - Matt Mitchell: Ok in the Pulsar broker logs, there’s
this:
```Caused by:
org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException:
Topic is already fenced
at
org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.delete(NonPersistentTopic.java:349)
~[org.apache.pulsar-pulsar-broker-2.5.2.jar:2.5.2]```
----
2020-08-03 16:58:54 UTC - Matt Mitchell: Does Pulsar fence topics that it
considers inactive? And if so, should clients just retry until the topic is
re-created?
----
2020-08-03 17:00:13 UTC - Matt Mitchell: The logging above is actually the
“cause” of this:
```16:55:58.106 [pulsar-inactivity-monitor-24-1] WARN
org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic -
[<non-persistent://robot/_connectors/connectors.plugin-notifications>] Topic is
already being closed or deleted
16:55:58.106 [pulsar-inactivity-monitor-24-1] WARN
org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic -
[<non-persistent://robot/_connectors/connectors.plugin-notifications>] Inactive
topic deletion failed
java.util.concurrent.CompletionException:
org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException:
Topic is already fenced```
----
2020-08-03 17:01:25 UTC - Matt Mitchell: So it looks like a fenced topic can’t
be deleted, but is also unusable, how should an application deal with this
scenario?
----
2020-08-03 17:10:50 UTC - Addison Higham: @Matt Mitchell ~what version of k8s
is this?~ oops i meant pulsar :face_palm:
----
2020-08-03 17:29:12 UTC - Matt Mitchell: Hi @Addison Higham - this is Pulsar
2.5.2
----
2020-08-03 17:32:14 UTC - Addison Higham: Hrm, there were some earlier issues
for that and I *think* most of them should be in 2.6.0, Another thing to look
for in your logs, do you see it trying to unload any bundles/topics (for
re-balancing)? What can happen is that if your topic is unloaded and reloaded
to another broker, that does work a bit different though for non-persistent
topics, so it might be some details there
----
2020-08-03 17:33:05 UTC - Addison Higham: one thing that should fix it is to do
a `namespace unload` as that should force pulsar to move that topic to a
different broker
----
2020-08-03 17:33:49 UTC - Matt Mitchell: > do you see it trying to unload
any bundles/topics (for re-balancing)?
I haven’t seen anything in the logs related to re-balancing, but I’ll let you
know if I do see something.
----
2020-08-03 17:34:16 UTC - Addison Higham: but as far as your question, that
isn't something an application shouldn't have to handle, the client can already
deal with a small period as topics get moved across the cluster, if it doesn't
resolve itself then *usually* that is an issue with the cluster
----
2020-08-03 17:36:27 UTC - Matt Mitchell: Ok good to know. I believe this
happened during a time when several other issues were present on the cluster
too, potentially even zk problems. So it may be that it was a temporary issue
and not something we’ll see in production.
----
2020-08-03 17:37:10 UTC - Matt Mitchell: I’ll see if I can reproduce the
problem though, and also look into `namespace unload`.
----
2020-08-03 17:41:49 UTC - Addison Higham: ah yes, that would be a problem if ZK
were having an issue
----
2020-08-03 17:42:30 UTC - Matt Mitchell: ok, that may be it then. I’ll do some
more research to make sure that was really happening at the time of this error
and let you know.
----
2020-08-03 17:56:16 UTC - Axel Sirota: Hey man! thanks for answering! I think,
howver, this is a bad use case from my side! I see in the config there that I
cannot set the hostname of the bookkeeper, and that makes sense since zookeeper
is for that :slightly_smiling_face:
so i assume the issue I am having is that as my deployment is standalone in a
docker image, zookeper answers, hey my BK is localhost! however as my
pulsar-sql container was another one, it complained because localhost for it is
not the same! But that is an issue that I deployed “standalone” but with SQL
not “standalone” making a weird mix! If the Pulsar deployment was “correct”
(ie: multiple services & containers for each component) then it would work
fine :slightly_smiling_face:
----
2020-08-03 17:57:40 UTC - Axel Sirota: I have, on the other hand, a more “I
don’t know what the hell is going on why is this not working” issue I am going
to write now about regarding the Kafka client adapter :slightly_smiling_face:
I think another contribution to the docs is to recall that pulsar-sql, in
standalone deployment, is meant to run _on the same container._
:slightly_smiling_face:
----
2020-08-03 18:21:37 UTC - Axel Sirota: Hey! I have a last question (I
promise!!) I am trying to use the `pulsar-client-kafka` package to finally
switch a sample kafka application using `lof4j` to use pulsar
:slightly_smiling_face: (this is all educational content which is the reason I
am doing such random questions haha ) and i have the app correctly loggin to
kafka, however when I switched to the dependency:
```<!-- <dependency>-->
<!-- <groupId>org.apache.kafka</groupId>-->
<!-- <artifactId>kafka-clients</artifactId>-->
<!-- <version>2.5.0</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-kafka</artifactId>
<version>2.6.0</version>
</dependency>```
then it logged on console, but never submitted to Pulsar. I sincerelly don’t
have a clue how to even debug this … any idea how the `log4j2.xml` must look
like? I tried this app (`<https://github.com/sijie/kafka-logger-demo>`) but
never got it to work actually :disappointed:
My `log4j2.xml` file looks like the following:
````=<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level
%logger{36} - %msg%n" />
</Console>
<!-- <Kafka name="Kafka" topic="log-test">-->
<!-- <JsonLayout />-->
<!-- <Property
name="bootstrap.servers">localhost:9092</Property>-->
<!-- </Kafka>-->
<Kafka name="Kafka" topic="persistent://public/default/log-test">
<JsonLayout />
<Property
name="bootstrap.servers">localhost:6650</Property>
</Kafka>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="Console" />
</Root>
<Logger name="kafka" level="info">
<AppenderRef ref="Kafka" />
</Logger>
</Loggers>
</Configuration>```
Thanks in advance!! Maybe I need to use an older version??
----
2020-08-03 19:57:30 UTC - Feanil: Hey all, I'm to understand how the pulsar
topics and partition topics work, in particular if I start out with a regular
topic and later decide that it should be partitioned because it needs to scale
better, is it easy to convert the unpartitioned topic to a partitioned topic?
I looked on this page, but it was not clear how or if unpartitioned topics can
become partitioned topics after the fact.
----
2020-08-03 20:01:05 UTC - Addison Higham: currently, there isn't support for an
un-partitioned topic to become a partitioned topic. The suggestion is that if
you think you would ever want it to be partitioned, start with a partitioned
topic with a partition count of 1. There is also support in the latest versions
of pulsar for this to be how auto-created partitions are created (as
partitioned topics with a default count of 1). This may be the direction pulsar
goes by default in the future
----
2020-08-03 20:02:55 UTC - Feanil: Got it, and if I already have an
unpartitioned topic, I'll just need to migrate to a new partitioned topic if I
want that scalability?
----
2020-08-03 20:07:58 UTC - Addison Higham: yes, there isn't support for
converting right now
----
2020-08-03 20:13:03 UTC - Chris Hansen: Also worth noting, if I try
`/pulsar/bin/pulsar-admin functions trigger` I get Request Timed Out pretty
consistently. Might the call to the function be timing out and what might cause
that?
----
2020-08-03 20:13:42 UTC - Caleb Epstein: @Caleb Epstein has joined the channel
----
2020-08-03 20:19:01 UTC - Shivam Arora: Thanks, makes sense. Need to find a way
to set it on K8 :slightly_smiling_face:
----
2020-08-03 20:22:38 UTC - Caleb Epstein: Hey all. Just scratching the surface
of Pulsar for the first time, and am liking what I see. I'm using the C++ API
(didn't see a channel for that) and am mildly confused about some terminology
in the docs vs. that API. Notably, if I'm going to "set the key" on a Message
I send to a topic, do I use the `setPartitionKey` or `setOrderingKey` method on
MessageBuilder?
See
<https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/include/pulsar/MessageBuilder.h#L70>
nad L76 as the Doxygen docs seem to be out of date (filed an Issue for that...)
----
2020-08-03 20:34:09 UTC - Addison Higham: @Shivam Arora easiest way is to use a
privileged initContainer and set them
----
2020-08-03 20:47:21 UTC - Manoj Sonavane: @Manoj Sonavane has joined the channel
----
2020-08-03 21:53:54 UTC - Chris Hansen: Weirdly, triggering the function first
causes it to behave as expected. I’ve filed an issue for this:
<https://github.com/apache/pulsar/issues/7732>
----
2020-08-04 01:35:40 UTC - krishna reddy jaggannolla: Hi, If the messages are in
a counter and there are no subscriptions yet connected. What if the consumers
are connected later how can the subscribers get that messages in the
msgIncounter
----
2020-08-04 02:33:21 UTC - Addison Higham: @Caleb Epstein `partitionkey` is the
more common of the keys, it is used for routing, compaction, etc.
The `orderingKey` is used only in key_shared subscriptions
----
2020-08-04 07:27:57 UTC - Yifan: Hi all. I have a question on retention and
expire policy: I would like to have acknowledged messages to be kept for 3d and
unacknowledged messages to be kept for 7d. Is there a way to do that?
----
2020-08-04 08:28:02 UTC - Ermir Zaimi: Hi all, i am trying to connect mysql
database with debezium connector, but when i start the connector in pulsar IO i
get the following error: 10:16:08.041 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xaed1087f,
L:/<http://127.0.0.1:43114/|127.0.0.1:43114> -
R:localhost/<http://127.0.0.1:6650/|127.0.0.1:6650>]] Connected to server
10:16:08.078 [pulsar-client-io-1-1] WARN
org.apache.pulsar.client.impl.ClientCnx - [id: 0xaed1087f,
L:/<http://127.0.0.1:43114/|127.0.0.1:43114> -
R:localhost/<http://127.0.0.1:6650/|127.0.0.1:6650>] Received error from
server: Unable to authenticate
10:16:08.078 [pulsar-client-io-1-1] WARN
org.apache.pulsar.client.impl.ClientCnx - [id: 0xaed1087f,
L:/<http://127.0.0.1:43114/|127.0.0.1:43114> -
R:localhost/<http://127.0.0.1:6650/|127.0.0.1:6650>] Received unknown request
id from server: -1
10:16:08.080 [pulsar-client-io-1-1] INFO
org.apache.pulsar.client.impl.ClientCnx - [id: 0xaed1087f,
L:/<http://127.0.0.1:43114/|127.0.0.1:43114> !
R:localhost/<http://127.0.0.1:6650/|127.0.0.1:6650>] Disconnected
10:16:08.082 [pulsar-client-io-1-1] WARN
org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xaed1087f,
L:/<http://127.0.0.1:43114/|127.0.0.1:43114> !
R:localhost/<http://127.0.0.1:6650/|127.0.0.1:6650>]] Connection handshake
failed: org.apache.pulsar.client.api.PulsarClientException: Connection already
closed
10:16:08.094 [main] INFO org.apache.pulsar.functions.LocalRunner -
RuntimeSpawner quit because of
java.util.concurrent.CompletionException:
org.apache.pulsar.client.api.PulsarClientException: Connection already closed,
----
2020-08-04 09:02:36 UTC - Vil: What would certainly help is documenting when to
use pulsar functions, and when not. There was a pulsar summit talk (can’t
remember what one) about processing where it was recommended to use Flink. I do
not recall what the recommend was for pulsar functions, but it was not for
“real” processing, which was again Flink to be used
----
