2020-05-02 09:11:36 UTC - Franck Schmidlin: Can a *pulsar function* listen to
messages without consuming them? I'd like to listen to topics and *copy* some
messages to a separate, additional, topic.
+1 : Franck Schmidlin
----
2020-05-02 09:27:26 UTC - JG: ok! maybe I should not interact with it and leave
it to Pulsar usage only.
----
2020-05-02 09:36:31 UTC - JG: it will be good to know if it can makes problems
or if we are free to use it
----
2020-05-02 09:44:47 UTC - Franck Schmidlin: The opposite. Sounded like it can
be shared. I'll try and dig out an article
----
2020-05-02 10:56:04 UTC - Guillaume Audic: Mount volume to /tmp didn't avoit
high memory cache :disappointed:
I debug the startup of broker to find that it's really at the startup there is
a lot of file created under /tmp.
So actually a the minimal memory to startup the booker is the size of the /tmp
directory and some memory for the booker process.
I don't know if it's an issue or not but it's really not the good way to work
with containers. In container we have to avoid creation of lot of file during
startup. If file are required for startup it can be include in the docker base
image
----
2020-05-02 11:35:49 UTC - Vladimir Shchur: You shouldn't worry about that,
function subscription is independent, it doesn't affect any other subscription
----
2020-05-02 12:00:09 UTC - Guillaume Audic: Using the image pulsar instead
pulsar-all is better there is not all default connectors
----
2020-05-02 12:04:09 UTC - Franck Schmidlin: If inactive topics get deleted,
what happens if i recreate the topic later. Will I be able to read messages
published on the original topic?
----
2020-05-02 12:59:22 UTC - JG: I just tested the version 2.5.1 and pulsar
manager and it seems the SQL problem remains when I try to build a new tenant:
----
2020-05-02 12:59:23 UTC - JG: postgres_1 | 2020-05-02 12:58:00.468 UTC [79]
ERROR: LIMIT #,# syntax is not supported at character 38
postgres_1 | 2020-05-02 12:58:00.468 UTC [79] HINT: Use separate LIMIT and
OFFSET clauses.
postgres_1 | 2020-05-02 12:58:00.468 UTC [79] STATEMENT: SELECT name,broker
FROM environments LIMIT $1, $2
----
2020-05-02 15:33:05 UTC - Nikita Muravyov: @Nikita Muravyov has joined the
channel
----
2020-05-02 15:47:28 UTC - Allen ONeill: Hi all - myself and @Nikita Muravyov
have deployed Pulsar in K8s ... 3 x instances. It works well when we produce
2..3 messages in a short burst.... however, when we push it with any volume,
the deployment borks and our producers/subscriber clients lose connectivity,
The pods themselves dont seem to be going down, and the clients then reconnect
after some 10/20 re-try time .... the difference seems to be the number of
messages we are sending into the producer .... are we missing something? ...
should we be creating a new producer for each message we want to send, or leave
one producer open and send through it all the time?
----
2020-05-02 16:18:20 UTC - JG: Nobody has an exemple of Pulsar Function in java
? I mean the FunctionConfig
----
2020-05-02 16:18:48 UTC - JG: for the method:
functionConfig.setTopicsPattern(); does he need a regex or a normal topic name
?
----
2020-05-02 18:31:09 UTC - Ruian: Does anyone have tried the Kubernetes
CustomRuntimeOptions of Pulsar Function?
I have already:
1. set
`runtimeCustomerClassName=org.apache.pulsar.functions.runtime.kubernetes.BasicKubernetesManifestCustomizer`
in the `functions-worker.yaml`
2. and created the function with the command `bin/pulsar-admin functions create
--custom-runtime-options '{"resourceRequirements":{"limits":{"memory":"64M"}}}'`
but the created function pod is still using the default resource requirements
instead of the one specified in the CustomRuntimeOptions.
----
2020-05-02 20:21:33 UTC - Vladimir Shchur: No
----
2020-05-02 20:23:23 UTC - Vladimir Shchur: You should leave producer open
----
2020-05-02 20:26:19 UTC - Vladimir Shchur:
<http://pulsar.apache.org/docs/en/concepts-messaging/#multi-topic-subscriptions|http://pulsar.apache.org/docs/en/concepts-messaging/#multi-topic-subscriptions>
----
2020-05-02 20:57:27 UTC - JG: thx !
----
2020-05-02 20:59:18 UTC - JG: A quick question regarding pulsar functions, when
its uploaded and started, is it directly effective ? Necause regarding my
topics I had to send 2 or 3 messages with my producers to see the function
working.
----
2020-05-02 21:26:23 UTC - Vladimir Shchur: If you are using regex, then you
have to wait some time until scheduler reruns regex check and subscribe to the
new topics
----
2020-05-02 22:01:56 UTC - JG: I was not using regex but only 1 topic and it
didnt work directlty, do you know the exact amount fo time before the funtion
will be effective ? I don't see anything in the documentation.
----
2020-05-02 22:31:14 UTC - David Kjerrumgaard: The data that you need to backup
is the Zookeeper data which keeps track of where the ledgers are written. It is
analogous to the NameNode in a Hadoop cluster. However, if you want to have a
copy of all the data in BookKeeper then your best bet is to stand up a second
cluster and implement geo-replication.
----
2020-05-02 22:33:16 UTC - David Kjerrumgaard: No, the "old" topic data
metadata, e.g. where the ledgers are stored will be overwritten by the "new"
version of the topic.
+1 : Franck Schmidlin
----
2020-05-03 00:51:10 UTC - VanderChen: @VanderChen has joined the channel
----
2020-05-03 01:03:56 UTC - guoyejun: @guoyejun has joined the channel
----
2020-05-03 01:04:51 UTC - VanderChen: Hi all, we have some design trouble as
following:
1. How to broadcast a message in a group of consumers which subscribe to the
same topic?
2. Is there any performance test about _Number of topic in a namespace_
----
2020-05-03 02:02:14 UTC - tree1123: @tree1123 has joined the channel
----
2020-05-03 03:16:46 UTC - Ruian: In my case with using kubernetes runtime, it
is very quick to see that the function pod is running after uploading it
through pulsar-admin cli.
However it seems that the broker has some io problem to instruct the function
pod to do real work. I can only see the `UNAVAILABLE: io exception` from the
`pulsar-admin functions status`.
And it will have to wait about ~10min for the broker to resolve the issue
automatically.
Then the function pod will finally download the function source and start
consuming messages.
----
2020-05-03 04:18:58 UTC - Franck Schmidlin: Ok. So i need to turn off auto
deletion for long lived, seldom used topics. Thanks
----
2020-05-03 09:02:41 UTC - JG: so it seems its not stable yet
----
