2019-08-08 09:16:15 UTC - divyasree: ok i wont use hereafter..
----
2019-08-08 09:43:05 UTC - Alexandre DUVAL: @Jerry Peng
----
2019-08-08 09:43:21 UTC - Alexandre DUVAL: im interested in the answer
----
2019-08-08 09:48:53 UTC - Alexandre DUVAL: ```PulsarProducer.send() failed:
org.apache.pulsar.client.api.PulsarClientException:
java.lang.IllegalStateException: recycled already
{:exception=>java.util.concurrent.ExecutionException:
org.apache.pulsar.client.
api.PulsarClientException: java.lang.IllegalStateException: recycled already}
``` what means this?
----
2019-08-08 10:07:29 UTC - Alexandre DUVAL: on exception catch my code retried
the send and it did it 180 000times, then my consumer received the same message
180 000 times. I'm currently in ProducerImpl.java to check how it works, but if
you have infos.. :slightly_smiling_face:
----
2019-08-08 13:29:11 UTC - Alexandre DUVAL: other point, is it possible to
unsubscribe automatically when no consumer are active on consumerConfig?
----
2019-08-08 15:58:18 UTC - Grant Wu: Anyone have any idea what to do with
```
503 Failed to create producer: Cannot create producer on topic with backlog
quota exceeded: org.apache.nifi.processor.exception.ProcessException: Failed to
register processor to WebSocket service due to: java.io.IOException: Failed to
connect <censored> due to: java.util.concurrent.ExecutionException:
org.eclipse.jetty.websocket.api.UpgradeException: 503 Failed to create
producer: Cannot create producer on topic with backlog quota exceeded
```
----
2019-08-08 15:59:22 UTC - Yuvaraj Loganathan:
<https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#backlog-quotas>
----
2019-08-08 15:59:35 UTC - Grant Wu: sorry
----
2019-08-08 15:59:38 UTC - Grant Wu: Of note is:
----
2019-08-08 15:59:40 UTC - Grant Wu: ```
root@pulsar-broker-5d47994776-fxs2h:/pulsar# bin/pulsar-admin namespaces
get-backlog-quotas public/solution
{ }
```
----
2019-08-08 15:59:44 UTC - Grant Wu: So there doesn’t appear to be a backlog set
----
2019-08-08 16:00:37 UTC - Yuvaraj Loganathan: Unless you are using pulsar with
this pull request <https://github.com/apache/pulsar/pull/4320> You need to set
a very large backlog
----
2019-08-08 16:00:47 UTC - Yuvaraj Loganathan: by default it is 10gb i think
----
2019-08-08 16:03:04 UTC - Grant Wu: Okay thanks!!!
----
2019-08-08 17:15:25 UTC - Sam Leung: A couple of admin actions were added in
2.4.0 which include deleting subscription, clearing backlog, and peeking at
messages in a subscription.
----
2019-08-08 17:17:23 UTC - Sam Leung: There’s a broker setting
```brokerServicePurgeInactiveFrequencyInSeconds
How often broker checks for inactive topics to be deleted (topics with no
subscriptions and no one connected)```
----
2019-08-08 17:18:22 UTC - Sam Leung: I wonder if there’s something similar for
deleting subscriptions.
----
2019-08-08 17:23:11 UTC - Addison Higham: I created this:
<https://github.com/apache/pulsar/issues/4920>
----
2019-08-08 17:28:10 UTC - Jerry Peng: @Alexandre DUVAL you can not pass in
option as a CLI flag but you can just set it in your function config yaml file
that you submit for your function. Does that make sense?
----
2019-08-08 17:29:54 UTC - Jerry Peng: > isn’t the write quorum across
multiple nodes?
Yes but alternatively instead of increasing the write quorum (i.e. copies of
your data) you can also choose to strip your data across more nodes to increase
read throughput.
----
2019-08-08 17:54:11 UTC - Alexandre DUVAL: it does :slightly_smiling_face:.
thanks.
----
2019-08-08 17:54:36 UTC - Alexandre DUVAL: that's the point
:slightly_smiling_face:
----
2019-08-08 17:54:49 UTC - Addison Higham: and a proposed fix:
<https://github.com/apache/pulsar/pull/4921>
----
2019-08-08 18:00:02 UTC - Aaron: Can anyone help with this?
----
2019-08-08 18:15:27 UTC - Sam Leung: I have a question about `Message has been
dropped for non-persistent topic producer-id ...`. I am producing a message to
a persistent topic that has an undersized bookie cluster (e.g. ensemble/ack
quorum is supposed to be 3, I only have 2 instances). When I see that message,
is there any chance it’s referring to my supposed persistent topic’s message
being dropped, or is there some metadata exchange that’s getting dropped?
Note in the example logs below, I have 3 producers on one topic, and one
consumer on a different topic.
----
2019-08-08 18:19:51 UTC - Sam Leung: Yeah sorry, misread the question at first
:stuck_out_tongue:
----
2019-08-08 20:31:17 UTC - Federico Ponzi: @Federico Ponzi has joined the channel
----
2019-08-08 20:41:42 UTC - Tarek Shaar: We are trying to move to Apache Pulsar
but I am getting a lot of resistance from management due to lack of GUI based
admin and monitoring tooling. Does Pulsar only support Java, REST and CLI tools
for monitoring and administration?
----
2019-08-08 20:43:45 UTC - Jerry Peng: @Tarek Shaar pulsar does have an open
source UI in which you can perform certain actions. For metrics, people usually
use Prometheus and Graphana to monitor the cluster
----
2019-08-08 20:45:43 UTC - Tarek Shaar: What's the open source UI that is used?
What kinds of actions does it support?
----
2019-08-08 20:54:03 UTC - Sam Leung: We have deduplication on, and it seems
like that could generate that message.
----
2019-08-08 21:35:13 UTC - Ali Ahmed: @Tarek Shaar it mostly basic read only
operations , there is another ui <https://github.com/bbonnin/pulsar-express>
which has more features
----
2019-08-08 21:41:12 UTC - Tarek Shaar: Ok I will take a look. We have been
using Sonic MQ here and its admin GUI is very strong (from protocol set up to
cluster administration, topic and queue monitoring and much more) so when I
present my case to the management about Pulsar the question about tooling is
always coming up. Let me check this one out
----
2019-08-08 21:54:08 UTC - Sam Leung: I see the broker in
`MessageDeduplication.java` updates `highestSequencedPushed.put(producerName,
sequenceId);` before persist happens.
If an error in persistence happens, my producer client re-sends the same
message without changing the sequenceId right? And that message would be
dropped at the broker because it thinks it’s a duplicate.
The only time `highestSequencedPushed` is cleared is when dedup is disabled or
after an inactivity timer, not when an error in persistence happens.
----
2019-08-08 22:29:55 UTC - Sam Leung: Looks like it’s a bug in dedupe. I turned
dedupe off and I’m seeing the `Re-Sending 1 messages to server` retry until
timeout, and throwing me a `TimeoutException` from Producer.send() as expected.
Will file a bug tomorrow.
----
2019-08-08 22:46:55 UTC - Victor Li: Thank you! @Sijie Guo
----
2019-08-09 01:33:46 UTC - Yi Tang: @Sam Leung yeah, checked these actions,
thanks
----
2019-08-09 02:34:02 UTC - Luo Tian: Help: I started a function in my pulsar
instance, shows:
```json
{
"numInstances" : 1,
"numRunning" : 0,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : false,
"error" : "",
"numRestarts" : 1,
"numReceived" : 0,
"numSuccessfullyProcessed" : 0,
"numUserExceptions" : 0,
"latestUserExceptions" : [ ],
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"averageLatency" : 0.0,
"lastInvocationTime" : 0,
"workerId" : "c-local-fw-10.0.2.131-8080"
}
} ]
}
```
----
2019-08-09 02:34:48 UTC - Luo Tian: why is `numRunning` 0?
----
2019-08-09 02:35:28 UTC - Luo Tian: BTW, it does not produce any output, and
log, and dead letter (bother log and dead letter topics specified when creating)
----
2019-08-09 02:47:00 UTC - Sijie Guo: FYI. beside pulsar-express, I have also
known:
Yahoo! Japan has developed an internal UI tool. @Nozomi Kurihara will present
it in the pulsar meetup (beijing) next week. He might also be thinking of open
sourcing the tool as well.
@tuteng is also developing a pulsar-manager to manage multiple pulsar clusters.
that one is also to be open sourced soon.
----
2019-08-09 02:47:28 UTC - Sijie Guo: numRestarts: 1
----
2019-08-09 02:47:36 UTC - Sijie Guo: it seems that there are exceptions thrown
----
2019-08-09 02:47:43 UTC - Sijie Guo: might be worth checking the function log
----
2019-08-09 02:48:10 UTC - Luo Tian: thanks
----
2019-08-09 02:56:37 UTC - Luo Tian: It shows a schema incompatible error,
----
2019-08-09 02:57:00 UTC - Luo Tian: The source topic is of json messages, but
does not set any schema.
----
2019-08-09 02:57:21 UTC - Luo Tian: while the function using java native
interface as `Function<String, String>`
----
2019-08-09 02:59:18 UTC - Nozomi Kurihara: Yahoo! JAPAN is now developing an UI
tool for admin operations. We're going to open it early September.
----
2019-08-09 05:08:40 UTC - Sijie Guo: oh I see. I think Function attempts to use
a JSON / String schema to consume a topic without schema. Currently it prevents
a consumer with schema to consume data from a topic without a schema.
----
2019-08-09 05:09:20 UTC - Sijie Guo: We can consider introducing a flag to
allow function user bypass the schema validation.
----
2019-08-09 05:19:01 UTC - Luo Tian:
<https://github.com/apache/pulsar/issues/4857>
----
2019-08-09 05:19:10 UTC - Luo Tian: I think it is the same as this issue
----
2019-08-09 05:41:05 UTC - Luo Tian: can I use `pulsar-admin schemas upload` to
apply a schema to the topic?
----
2019-08-09 05:52:44 UTC - Luo Tian: I figured it out.
----
2019-08-09 05:53:11 UTC - Luo Tian: uploaded the schema you showed in the
comment.
----
2019-08-09 06:10:48 UTC - Sijie Guo: cool
----
2019-08-09 06:14:16 UTC - jia zhai: @Aaron This looks like a bug, Would you
please help open an issue in github?
<https://github.com/apache/pulsar/issues/new?assignees=&labels=type%2Fbug&template=bug_report.md&title=>
----
2019-08-09 06:20:19 UTC - Luo Tian: There seems no documentation for exact
content of a schema.
----
2019-08-09 06:34:26 UTC - Kim Christian Gaarder: I reported a bug
<https://github.com/apache/pulsar/issues/4912>
This is basic functionality in the java client Reader that is not working as
expected.
Do I need to do more in order for the bug to be considered for fixing?
----
2019-08-09 06:35:57 UTC - Sijie Guo: will triage and pick it up
----
2019-08-09 06:38:53 UTC - Kim Christian Gaarder: @Sijie Guo awesome, It should
be pretty easy to create some unit tests in Pulsar directly to reproduce this
and get some regression tests, I have never written any unit-tests in Pulsar
yet, so I opted to use a separate project for reproduction.
+1 : Sijie Guo
----
2019-08-09 06:39:30 UTC - Sijie Guo: yeah.
----
2019-08-09 06:39:32 UTC - Sijie Guo: will check
----
2019-08-09 07:20:58 UTC - ppgoml: @ppgoml has joined the channel
----
2019-08-09 07:29:15 UTC - jinfeng105: @jinfeng105 has joined the channel
----
2019-08-09 07:33:34 UTC - ppgoml: Hello, could pulsar share zookeeper with
other applications?
----
2019-08-09 07:34:37 UTC - Ali Ahmed: @ppgoml yes
----
2019-08-09 07:37:34 UTC - Diego Salvi: @Diego Salvi has joined the channel
----
2019-08-09 07:41:39 UTC - Diego Salvi: Hi! I have a problem with a newly
created topic (non partitioned created from admin api). When I attempt to open
a reader on it i get a long wait and then a failure with "6 lookup request
timedout after ms 30000" and not much more infos
----
2019-08-09 07:41:48 UTC - ppgoml: @Ali Ahmed thanks, where can I set the root
znode dedicated for pulsar?
----
2019-08-09 07:43:39 UTC - Diego Salvi:
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 6 lookup
request timedout after ms 30000
at
org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:247)
at
org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:69)
Unfortunately a I cannot find any more logs explaining why it happened
----
2019-08-09 07:47:30 UTC - Sijie Guo: is your client able to connect to your
brokers?
or can you explain a bit how did you set up the cluster?
----
2019-08-09 07:54:26 UTC - Diego Salvi: acually I have 3 bookie and one pulsar
broker
----
2019-08-09 07:55:48 UTC - Diego Salvi: I created 3 topics with PulsarAdmin
----
2019-08-09 07:56:52 UTC - Sijie Guo: bin/pulsar-admin brokers list
<pulsar-cluster>
----
2019-08-09 07:57:16 UTC - Sijie Guo: can you show me the result of running this
command?
----
2019-08-09 07:57:45 UTC - Diego Salvi: can be run directly from api too?
----
2019-08-09 07:57:57 UTC - Diego Salvi: like /admin/v2/broker-stats/topics?
----
2019-08-09 07:59:50 UTC - Sijie Guo: /admin/v2/<cluster-name>
----
2019-08-09 08:01:57 UTC - Diego Salvi:
<http://localhost:9372/admin/v2/clusters/mycluster>
----
2019-08-09 08:02:01 UTC - Diego Salvi: gave me {}
----
2019-08-09 08:02:28 UTC - Diego Salvi: a... I forgot to add... for one topic I
can create thereader... for another no
----
2019-08-09 08:03:55 UTC - Diego Salvi:
<http://localhost:9372/admin/v2/brokers/mycluster>
----
2019-08-09 08:05:33 UTC - Diego Salvi: ["dev1:9372"]
----
2019-08-09 08:25:26 UTC - Sijie Guo: ```
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-test\
--zookeeper hdp1:2181/kobe \
--configuration-store hdp1:2181/kobe \
--web-service-url <http://hdp1:8080> \
--web-service-url-tls <https://hdp1:8443> \
--broker-service-url <pulsar://hdp1:6650> \
--broker-service-url-tls <pulsar+ssl://hdp1:6651>
```
in conf/bookkeeper.conf:
```
zkServers=hdp1:2181
zkLedgersRootPath=/kobe/ledgers
```
in conf/broker.conf
```
zookeeperServers=zk:2181/kobe
configurationStoreServers=zk:2181/kobe
```
----
2019-08-09 08:25:53 UTC - Sijie Guo: are you able to connect to dev1:9372 ?
----
2019-08-09 08:35:14 UTC - Diego Salvi: directly? checking
----
2019-08-09 08:36:51 UTC - Diego Salvi: Yes I can
----
2019-08-09 08:36:57 UTC - Diego Salvi: checked with telnet
----
2019-08-09 08:39:28 UTC - Sijie Guo: Can you show the example code how do you
configure the reader?
----
2019-08-09 08:45:34 UTC - Diego Salvi: sure... going to get te code
----
2019-08-09 08:48:00 UTC - Diego Salvi: try (Reader<byte[]> pulsarReader =
pulsarClient.newReader()
.startMessageId(lastConsumedMessageId)
.topic(topicName)
.create()) {
/* reading logic */
}
----
2019-08-09 08:48:06 UTC - Diego Salvi: it fails on the create
----
2019-08-09 08:48:36 UTC - Diego Salvi: topicName is the same used for creation,
lastConsumedMessageId is actually == MessageId.earliest
----
2019-08-09 08:48:47 UTC - Diego Salvi: the client itself is created with
----
2019-08-09 08:49:18 UTC - Diego Salvi: PulsarClient pulsarClient =
PulsarClient.builder().serviceUrlProvider(new ServiceUrlProvider() {
@Override
public void initialize(PulsarClient pc) {
}
@Override
public String getServiceUrl() {
return pulsarBrokerURI.get();
}
}).build();
----
2019-08-09 08:49:33 UTC - Diego Salvi: pulsarBrokerURI is just a supplier to
provide the right url
----
2019-08-09 08:50:34 UTC - Sijie Guo: http:// or pulsar:// ?
----
2019-08-09 09:05:38 UTC - Diego Salvi: hum actually is <pulsar://dev1:6650>
----
2019-08-09 09:05:48 UTC - Diego Salvi: which is listening and i can connect to
----
2019-08-09 09:07:16 UTC - Sijie Guo: No broker log?
----
2019-08-09 09:09:55 UTC - Diego Salvi: this is what I found
----
2019-08-09 09:09:57 UTC - Diego Salvi: 19-08-08-14-10-19 [idc7/ns/events]
Failed to get create topic reader
19-08-08-14-10-19 Client closing. URL: <pulsar://dev1:6650>
19-08-08-14-09-19 [idc7/ns/events][reader-69e6697932] Subscribing to
topic on cnx [id: 0x2be67bab, L:/10.168.10.79:52044 - R:dev1/10.168.10.81:6650]
19-08-08-14-10-19 [idc7/ns/events][reader-69e6697932] Failed to subscribe
to topic on dev1/10.168.10.81:6650
19-08-08-14-10-19 [id: 0x2be67bab, L:/10.168.10.79:52044 -
R:dev1/10.168.10.81:6650] request 2 timed out after 30000 ms
19-08-08-14-10-19 [id: 0x28390d63, L:/10.168.10.79:48170 !
R:dev1/10.168.10.79:6650] Disconnected
19-08-08-14-10-19 [id: 0x2be67bab, L:/10.168.10.79:52044 !
R:dev1/10.168.10.81:6650] Disconnected
----
