2019-03-08 13:33:18 UTC - Sébastien de Melo: Hi, we are currently deploying
pulsar with helm and the cluster metadata initialization fails because of the
following error:
Exception in thread "main"
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /namespace
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
at
org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:178)
Do you know why?
----
2019-03-08 13:34:28 UTC - Sébastien de Melo: As a consequence the default
namespace is not created
----
2019-03-08 14:29:47 UTC - Darragh: Hey @Matteo Merli, we've removed all our old
instances, created some new better instances, used the st1 and io1 nvme's and
see some improvements but the tail latencies remain. I've opened up the
grafana board temporarily. Could you perhaps take a look and give some more
pointers ?
----
2019-03-08 14:30:38 UTC - Darragh: I did a perf-test produce persistent -r
10000 -b 0
----
2019-03-08 14:30:58 UTC - Darragh: (stopped now, so you might have to go back
in time)
----
2019-03-08 14:52:21 UTC - Darragh: I can pm you the grafana endpoint with a
login if you would be up for that
----
2019-03-08 15:12:32 UTC - Sébastien de Melo: It works when I delay the brokers
launch
----
2019-03-08 16:03:59 UTC - Chris DiGiovanni: So I have two questions...
1) How do I get this bookie out of this 100% pickle? I can't expand the
disk/partition on the bookie as it will invalidate itself. Remove Bookie and
auto-recover?
2) How large should by ledger disks be to avoid needing to stray away from the
default Entry Log Compaction and Disk Utilization Settings?
3) I lied, I have a third question :wink: Docs explain how to invalidate a
bookie by changing port, or name/ip. Since I'm running a StatefulSet in K8s.
Both of these options are less than ideal. Is there a way to get zookeeper to
forget about a bookie if I were to blow it away, allowing it to re-register w/
it's orig name and port?
----
2019-03-08 16:13:12 UTC - Chris DiGiovanni: Maybe the answer to #1 and #3 is to
format the bookie is using `bookie shell bookieformat`
----
2019-03-08 16:19:14 UTC - Guy Feldman: When i deployed using the helm chart the
broker pods restarted a few times after which they were fine, but the zookeeper
metadata job didn't fail
----
2019-03-08 16:19:55 UTC - Guy Feldman: the ordering of the deployment isn't the
best
----
2019-03-08 16:20:24 UTC - Guy Feldman: I was going to work on it, but no time
----
2019-03-08 16:40:56 UTC - Sijie Guo: are you using the helm script in pulsar
repo?
----
2019-03-08 17:46:48 UTC - David Kjerrumgaard: @Vikas I have some updated NARs
that support JWT authentication for the NiFi processors. Which version of Nifi
are you running?
----
2019-03-08 18:24:10 UTC - Vikas: thanks @David Kjerrumgaard, we have NiFi 1.7.1
----
2019-03-08 18:32:48 UTC - David Kjerrumgaard:
----
2019-03-08 18:33:21 UTC - David Kjerrumgaard:
----
2019-03-08 18:34:15 UTC - David Kjerrumgaard: The Pulsar client service now has
a property named AUTHENTICATION_SERVICE, which you have to define.
----
2019-03-08 18:34:51 UTC - David Kjerrumgaard: There are 3 options for the
AUTHENTICATION_SERVICE. Just choose the PulsarClientJwtAuthenticationService
option and configure it accordingly
----
2019-03-08 18:35:14 UTC - Vikas: sure, thanks. Let me give a try with the new
processor.
+1 : David Kjerrumgaard
----
2019-03-08 19:45:51 UTC - Sébastien de Melo: I have added that for the brokers
(in initContainers) and it seems to help:
- name: wait-bookkeeper-ready
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["sh", "-c"]
args:
- >-
until nslookup {{ template "pulsar.fullname" . }}-{{
.Values.bookkeeper.component }}-{{ add (.Values.bookkeeper.replicaCount | int)
-1 }}.{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}.{{
.Values.namespace }}; do
sleep 3;
done;
----
2019-03-08 19:58:11 UTC - Sébastien de Melo: Yes, with small changes. I'll
give more details on Monday
----
2019-03-08 20:41:54 UTC - Ryan Samo: Is it possible to use DistributedLog on
the bookies of a Pulsar cluster? Just thinking it would be handy to reuse my
bookies for it
----
2019-03-08 20:42:25 UTC - Ali Ahmed: yes with the right configs
----
2019-03-08 20:43:26 UTC - Ryan Samo: Cool, yeah I see that dlog, etc. is not
included with the bookies packaged in Pulsar. I assume I’d have to add that and
other things right?
----
2019-03-08 20:43:57 UTC - Ali Ahmed: @Sijie Guo will know better
----
2019-03-08 20:49:50 UTC - chris: @Ryan Samo i think dlog is just a client
library so there is no extra configuration. the client library just needs to be
able to access zookeeper and each bookie directly
----
2019-03-08 20:51:25 UTC - Ryan Samo: Ok thanks. So what I don’t want to do is
break my Pulsar cluster for sure. Trying to figure out how to try
DistributedLog on those bookies if possible. Not even sure it’s the best idea
really.
----
2019-03-08 20:52:23 UTC - chris: it’s possible and should have no effect on
your clusters - pulsar functions actually use it underneath to store binaries
----
2019-03-08 20:54:10 UTC - Ryan Samo: Nice! I saw that DistributedLog is now
baked into Bookkeeper so trying to decipher the docs to see how to start trying
it out. Do you know if it’s ready to hit API wise as it stands or if there is
additional configuration to do to the bookies first?
----
2019-03-08 20:54:44 UTC - chris: no additional configuration should be needed
you should be good to go
+1 : Sijie Guo
----
2019-03-08 20:55:22 UTC - Ryan Samo: Super cool! Thanks @chris for the
knowledge!
----
2019-03-08 21:58:55 UTC - Vikas: hey @David Kjerrumgaard, the new processor
works :grinning:
Thanks a ton for helping me out to publish the messages to the Pulsar topic,
have a wonderful weekend!
----
2019-03-08 21:59:31 UTC - David Kjerrumgaard: Glad to hear that. Thank you for
the confirmation. Enjoy the code!!
party-parrot : Vikas
----
2019-03-08 22:26:01 UTC - Sree Vaddi: please join <#CGUU96J22|job-board>
channel for jobs/work. thank you. :slightly_smiling_face:
----
2019-03-08 22:54:00 UTC - Sree Vaddi: There is a speaking opportunity on March
25th evening at Walmart in Sunnyvale, CA. If anyone interested, please let me
know by this weekend. ~100 attends. The scheduled speaker is out on emergency
and ACM is looking for a replacement, hence short notice.
----
2019-03-09 00:24:36 UTC - vinay Parekar: Hi Everyone , I am facing a wierd
issue when creating function
----
2019-03-09 00:25:12 UTC - vinay Parekar: if i try to create this function , i
get an error : Reason: javax.ws.rs.ProcessingException: Java heap space
----
2019-03-09 00:25:28 UTC - vinay Parekar: Has any one seen this kind of behavior
?
----
2019-03-09 00:26:38 UTC - Ali Ahmed: @vinay Parekar you need to increase the
heap space for the admin
+1 : Thor Sigurjonsson
----
