2019-03-29 11:06:27 UTC - Sébastien de Melo: Hello, we have a Pulsar function,
in Python, and we are using its state storage to manage some counters. However
we are not succeeding in retrieving their value. The get_counter function does
not fail but always returns None.
Moreover the querystate command fails:
bin/pulsar-admin functions querystate --name Router --tenant public --namespace
default --key main_counter
State storage client is not done initializing. Please try again in a little
while.
Reason: HTTP 503 Service Unavailable
command terminated with exit code 1
Does it mean that we have to configure something?
----
2019-03-29 11:16:55 UTC - bhagesharora: @Ali Ahmed I have added above
dependency in my pom.xml file, the I had given mvn clean install but still
error is coming, see below.
----
2019-03-29 11:17:19 UTC - bhagesharora: [INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.686 s
[INFO] Finished at: 2019-03-29T11:14:21+00:00
[INFO] Final Memory: 50M/527M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.4.1:single (default) on
project pulsar_kafka_adaptor: Failed to create assembly: Unable to resolve
dependencies for assembly 'jar-with-dependencies': Failed to resolve
dependencies for assembly: The artifact has no valid ranges
[ERROR] io.netty:netty-codec-http2:jar:4.1.22.Final
[ERROR]
[ERROR] Path to dependency:
[ERROR] 1) pulsar_kafka_adaptor:pulsar_kafka_adaptor:jar:0.0.2-SNAPSHOT
[ERROR] 2) org.apache.pulsar:pulsar-broker:jar:2.3.0
[ERROR] 3) org.apache.pulsar:pulsar-websocket:jar:2.3.0
[ERROR] 4) org.apache.pulsar:pulsar-broker-common:jar:2.3.0
[ERROR] 5) org.apache.pulsar:pulsar-zookeeper-utils:jar:2.3.0
[ERROR] 6) org.apache.bookkeeper:stream-storage-server:jar:4.9.0
[ERROR] 7) org.apache.bookkeeper:stream-storage-java-client:jar:4.9.0
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
----
2019-03-29 11:19:17 UTC - Sijie Guo: I think it is available in standalone. but
it requires some steps to enable in cluster mode.
----
2019-03-29 11:20:12 UTC - Ali Ahmed: it’not a regular dependency it’s a
dependencyManagement block
----
2019-03-29 11:20:22 UTC - Ali Ahmed: they are different things
----
2019-03-29 11:30:37 UTC - bhagesharora: @Ali Ahmed So, How can I resolve this
error ??
----
2019-03-29 11:31:11 UTC - bhagesharora: @Ali Ahmed Is there any way to make a
maven build success then, I can run producer and consumer example ??
----
2019-03-29 12:02:23 UTC - Ezequiel Lovelle: From pulsar 2.3 you cant create a
"single" subscription with multiple consumer because from the way pulsar works
internally this does not make sense, one consumer is multi threaded in the same
way multi instances of the same consumer with the same subscription will
----
2019-03-29 12:05:31 UTC - Ezequiel Lovelle: If you were doing this in order tu
have a "more" threaded consumer just use a single consumer instance and you'll
be fine
----
2019-03-29 12:35:04 UTC - Sébastien de Melo: Oh I see, do these steps happen to
be described in some documentation by any chance? :slightly_smiling_face:
----
2019-03-29 12:47:32 UTC - Ezequiel Lovelle: One alternative to a really more
"threaded consumers" would be to pass a new pulsar client for each consumer,
which pulsar client accepts the `ioThreads` and `listenerThreads` setter.
----
2019-03-29 14:02:59 UTC - Shivji Kumar Jha: Hi @Sijie Guo
``` /**
* Sets the number of messages to receive before acknowledging. This
defaults to 100. This
* value is only used when checkpointing is disabled.
*
* @param size number of messages to receive before acknowledging
* @return this builder
*/
public PulsarSourceBuilder<T> acknowledgementBatchSize(long size) {
if (size > 0 && size <= MAX_ACKNOWLEDGEMENT_BATCH_SIZE) {
acknowledgementBatchSize = size;
}
return this;
}```
I think this method should throw an exception if the input arg is not correct?
Whats your point of view?
Or is this check already done somewhere else?
----
2019-03-29 14:05:50 UTC - Shivji Kumar Jha: This code is from
org.apache.flink.streaming.connectors.pulsar.PulsarSourceBuilder.java
----
2019-03-29 16:30:47 UTC - Emma Pollum: Our zk disks got full. We shut them
down, grew the disks, and booted back up. When we did, there was no
managed-ledger data. Only namespace and tenant info. We have 3 nodes in our ZK
cluster.
----
2019-03-29 16:34:14 UTC - vinay Parekar: Hi Can we any one let me know, if we
can aggregate on pulsar functions ?
----
2019-03-29 16:34:32 UTC - Jerry Peng: @vinay Parekar yes
----
2019-03-29 16:34:49 UTC - vinay Parekar: is there any documentation related to
it ?
----
2019-03-29 16:34:59 UTC - Jerry Peng:
<http://pulsar.apache.org/docs/en/functions-state/#incrcounter>
----
2019-03-29 16:35:02 UTC - vinay Parekar: thanks
----
2019-03-29 16:37:52 UTC - vinay Parekar: is there any time window based
aggregation ?
----
2019-03-29 16:39:48 UTC - Jerry Peng: yes
----
2019-03-29 16:39:50 UTC - Jerry Peng:
<https://github.com/apache/pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/AddWindowFunction.java>
----
2019-03-29 16:40:16 UTC - Jerry Peng: or
<https://github.com/apache/pulsar/blob/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/ContextWindowFunction.java>
----
2019-03-29 16:43:03 UTC - Jerry Peng: @vinay Parekar toward the bottom of this
blog explains some of the windowing options in more detail:
<https://streaml.io/blog/eda-event-processing-design-patterns-with-pulsar-functions>
----
2019-03-29 16:46:24 UTC - Yuvaraj Loganathan: @Emma Pollum By defualt zookeeper
creates snapshots. Can try restoring the snapshots ?
----
2019-03-29 16:46:39 UTC - Yuvaraj Loganathan:
<https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html> search for
snapshots
----
2019-03-29 16:46:46 UTC - Emma Pollum: Awesome, thanks!
----
2019-03-29 16:51:56 UTC - Yuvaraj Loganathan: Also make sure you are purging
the snapshots by configuring the PurgeTxnLog with retention of
----
2019-03-29 16:52:18 UTC - Emma Pollum: is that basically telling zk to remove
old snapshots?
----
2019-03-29 16:52:47 UTC - Yuvaraj Loganathan: Yes! but keep an good number so
snashot backups.
----
2019-03-29 17:00:39 UTC - vinay Parekar: Thanks Jerry.
----
2019-03-29 17:03:06 UTC - Emma Pollum: :thumbsup: thanks for the tips
----
2019-03-29 20:06:01 UTC - Grant Wu: Hrm, I guess the `.value()` vs `.data()`
distinction helps here
----
2019-03-29 20:45:53 UTC - Dave Southwell: I'm searching for a way to determine
if pulsar zookeeper is just taking a long time to startup, or if it's something
else broken. I'm starting them as usual with `bin/pulsar zookeeper` and I see
it listening on port 2184 almost immedately, but I don't see it listening on
either of the other ports: connect port or election port that I have configured
as 2185, and 2186 respectively.
----
2019-03-29 20:46:19 UTC - Dave Southwell: Is there a debug option I could add
to somehow see if it's just taking a long time to look through some files?
----
2019-03-29 20:49:28 UTC - Sijie Guo: @Sébastien de Melo It is on my TODO list.
I will try my best to get one out in a few days :slightly_smiling_face:
----
2019-03-29 20:49:58 UTC - Sijie Guo: ping me if I didn’t do so
----
2019-03-29 20:52:57 UTC - Sijie Guo: I think it tries to cap the max number of
acks around MAX_ACKNOWLEDGEMENT_BATCH_SIZE. that’s probably okay from some
extends. however I agreed with you failing with an explicit error is probably
much clearer for users.
----
2019-03-29 21:17:11 UTC - Shivji Kumar Jha: Right, i have that tiny PR waiting.
Will publish it. Thanks!
----
2019-03-29 21:39:36 UTC - Shivji Kumar Jha: Hi @Sijie Guo! Regarding
<https://github.com/apache/pulsar/pull/3949> I was wondering about the BC
story.. Something on the lines of
<https://github.com/shiv4289/pulsar/commit/4456b74e46f18055918b5c4c6365c99035d065ff>
But I am not sure which of these are exactly public APIs.
----
2019-03-29 21:41:21 UTC - Shivji Kumar Jha: Of course I dont want to pollute
the files with too many constructors if BC does not apply to a file change.
----
2019-03-29 21:44:48 UTC - Sijie Guo: @Shivji Kumar Jha agreed. I think one of
the improvement is to change the PulsarOutputFormat to use a builder pattern
for easy extensions in future. However I think it is probably fine for a pulsar
flink connector, because when people upgrade a pulsar connector version, it
will have to recompile its flink code to general a new jar anyway. so IMO
changing code to add Authentication is probably fine. but if we want to be
better, we can add more constructors or move to use a builder pattern.
----
2019-03-29 21:45:32 UTC - Shivji Kumar Jha: So you think the current PR is good?
----
2019-03-29 21:45:48 UTC - Shivji Kumar Jha: @Sijie Guo ^^
----
2019-03-29 21:46:01 UTC - Shivji Kumar Jha: as far as BC is concerned..
----
2019-03-29 21:46:09 UTC - Sijie Guo: @Shivji Kumar Jha I think it is good
bananadance : Shivji Kumar Jha, Yuvaraj Loganathan
----
2019-03-29 21:46:14 UTC - Shivji Kumar Jha: Nice!
----
2019-03-29 21:46:59 UTC - Shivji Kumar Jha: @Sijie Guo I was also thinking that
we should probably expose loadConf(Map<String, Object>) in pulsar client
builder eventually.
----
2019-03-29 21:47:24 UTC - Shivji Kumar Jha: in flink module that is..
----
2019-03-29 21:47:48 UTC - Shivji Kumar Jha: thoughts?
----
2019-03-29 21:48:02 UTC - Sijie Guo: @Shivji Kumar Jha yes. I commented with a
similar idea in #3936 :slightly_smiling_face:
----
2019-03-29 21:48:54 UTC - Sijie Guo: if you can pick up that item, we probably
don’t need the changes in #3936 :slightly_smiling_face:
----
2019-03-29 21:51:04 UTC - Shivji Kumar Jha: @Sijie Guo I can surely do that
next week provided the current author is not doing it already
:slightly_smiling_face:
+1 : Sijie Guo
----
2019-03-29 21:51:36 UTC - Shivji Kumar Jha: I actually wanted to do that in my
current patch but my team needs the auth one really quickly so i just went with
the fast approach :smile:
----
2019-03-29 21:52:17 UTC - Sijie Guo: I see :slightly_smiling_face:
----
2019-03-29 21:52:26 UTC - Sijie Guo: :+1:
----
2019-03-29 21:53:58 UTC - Shivji Kumar Jha: Cool, i will pick that up then,
thank you for your prompt response to my PR.
You are a great help in us fast-tracking our move from pulsar in stage to prod
:slightly_smiling_face:
slightly_smiling_face : Sijie Guo
----
2019-03-29 21:54:18 UTC - Shivji Kumar Jha: @Sijie Guo ^^
----
2019-03-29 21:56:07 UTC - Sijie Guo: glad to hear you guys are moving to prod
:slightly_smiling_face: it would be really great if you guys blog it sometime
after you are on production :+1:
----
2019-03-29 21:56:25 UTC - Matteo Merli: :+1:
----
2019-03-29 22:05:32 UTC - Shivji Kumar Jha: Yes, i have blogs in my todo list
already. Will surely do that. I have also proposed a couple of talks ina
conference. Hopefully those are accepted too :grinning:
----
2019-03-29 22:39:34 UTC - Abhishek Negi: @Abhishek Negi has joined the channel
----
2019-03-29 23:05:58 UTC - Emma Pollum: What are the advantages of running
zookeeper via the pulsar binary vs a zookeeper binary?
----
2019-03-29 23:06:22 UTC - Matteo Merli: It’s just convenience
----
2019-03-29 23:06:40 UTC - Matteo Merli: You can get the standard ZK distribution
----
2019-03-29 23:09:26 UTC - Sijie Guo: :+1:
----
2019-03-29 23:09:48 UTC - Emma Pollum: :thumbsup:
----
2019-03-29 23:10:24 UTC - Matteo Merli: There just is minimal difference that
we added hooks to report stats in Prometheus format
----
2019-03-30 02:21:51 UTC - Karthik Ramasamy: Nice @Shivji Kumar Jha
slightly_smiling_face : Shivji Kumar Jha
----
