2020-08-11 09:57:39 UTC - Allen ONeill: *FUNCTIONS - STATE*
Hi all - I am trying to use STATE with python ... set and get.
My simple example (below) is to set a counter, set state, then read them back.
I am doing something wrong as it seems to be neither setting or getting the
data ... here is my code:
```from pulsar import Function
class State(Function):
def process(self, input, context):
properties = context.get_message_properties()
run_id = properties.get("run_id", "none")
if run_id != "none":
context.incr_counter(run_id, 1)
context.put_state("run_id", run_id)
print(f'Counter result: {run_id} = {context.get_counter(run_id)}')
print(f'State result: {run_id} = {context.get_state(run_id)}')```
Any ideas?
Thanks, Allen.
----
2020-08-11 10:45:26 UTC - vikash: i am also not able to understand how
to enable jmx port in standalone pulsar
----
2020-08-11 11:57:56 UTC - Raghav: Hi all,
Does anyone know how to mark a ledger directory from read-only to read-write?
chmod is not working.
----
2020-08-11 12:05:31 UTC - Alexandre DUVAL:
----
2020-08-11 12:51:50 UTC - Seun: Hi everyone,
Is anyone here aware of any light-weight version of helm charts deployment of
Apache pulsar? Something that doesn't consume many CPUs and memories of compute
machines
----
2020-08-11 12:52:10 UTC - Caleb Epstein: ns = dns? I'd think this would be in
your operating system configs (i.e. /etc/resolv.conf, /etc/nsswitch.conf on
Linux)
----
2020-08-11 12:53:14 UTC - Caleb Epstein: What are helm charts? Does this help?
<https://github.com/streamnative/apache-pulsar-grafana-dashboard>
----
2020-08-11 12:54:55 UTC - Seun: Many thanks @Caleb Epstein Helm charts are k8s
deployment and management tool. Simplify control of k8s clusters. I will check
that out.
----
2020-08-11 12:58:32 UTC - Boluwatife Fasugba: @Boluwatife Fasugba has joined
the channel
----
2020-08-11 13:11:34 UTC - Emanuel: Hi all, is there any pessimistic lock on a
event enters in bookkeeper ?
----
2020-08-11 13:13:57 UTC - Thomas O'Neill: You can use the helm chart and change
the replica counts if you want fewer pods. You can also change the resource
requests too, but you will want to leave enough that the application can still
work. All of this can be set in a values file that you use when installing and
updating the helm chart. You can also disable the proxy, toolset and
pulsar_manager, and the monitoring (grafana and prometheus) if you are not
using them.
----
2020-08-11 13:17:28 UTC - Seun: I did all of these except deleting
pulsar_manager. I setup GKE on gcp and i keep running into CPUs and memory
insufficiency and Google has refused to raise compute CPus quotas. Shame! I
will remove pulsar_manager and see what happens.
Many thanks @Thomas O'Neill
----
2020-08-11 13:38:12 UTC - Meyappan Ramasamy: hi team, i was unable to subscribe
to future topics when using pulsar version 2.5.0. please let me know if future
subscription will work in pulsar version 2.6.0
----
2020-08-11 13:39:37 UTC - Meyappan Ramasamy: hi team, sorry i missed your
response. this error was showing up when the gradle build and unit test was
happening in jenkins server, maybe some compatibility issue exists with other
platforms, please let me know how to resolve.
----
2020-08-11 17:03:44 UTC - Rich Adams: Yeah, I was wondering about that too. But
not sure if the shaded jar would fix it. I think that as long as you get the
classloader from the main process the dependency injection will grab the
classes that presto has loaded. But I’m not an expert in this stuff, so I may
be wrong.
----
2020-08-11 18:04:24 UTC - Addison Higham: Hrm... as mentioned, that demo code
is probably pretty out of date. If you could provide some logs of what log4j
says it is doing (which you can get by setting ` log4j2.debug` as a system
prop) then that would probably be helpful
@Axel Sirota however, your best bet is probably to just try and use this
instead: <https://github.com/apache/pulsar/tree/master/pulsar-log4j2-appender>
It as a log4j2 append that goes direct to Pulsar and doesn't use the kafka
appender
----
2020-08-11 18:04:47 UTC - Addison Higham: here is the deps:
<https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-log4j2-appender>
----
2020-08-11 18:11:46 UTC - Addison Higham: @Alexandre DUVAL AFAICT, I don't see
any mechanism to change the default NS name or the creation of that namespace.
The values are defined here:
<https://github.com/apache/pulsar/blob/8aee353d5b000c80be5cc281e6b574bd62ba5531/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java#L42>
And here is where the namespace is created:
<https://github.com/apache/pulsar/blob/5421a4c36671823e70750995bfb1cef763ea48c1/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java#L233>
----
2020-08-11 18:12:04 UTC - Addison Higham: that said, you should be able to
delete it and change it's policies pretty easy
----
2020-08-11 18:22:22 UTC - Yarden Arane: @Yarden Arane has joined the channel
----
2020-08-11 18:43:11 UTC - Yarden Arane: *FUNCTIONS: heartbeat*
Hi all, is there any built-in heartbeat functionality for pulsar functions?
Essentially I want a thread to trigger a function every second/minute/whatever.
A few possibilities would be:
1) a cron job to trigger / produce message on specific topic
2) a custom io-connector with a never ending thread triggering a function
Has anyone implemented a heartbeat before? Is there a better, more idiomatic
way of achieving this functionality.
Thanks in advance.
----
2020-08-11 18:46:16 UTC - Addison Higham: @Seun you can use very small resource
requests, like 256MB of heap can still allow pulsar to function, but you will
have to worry about heap exceptions as you push the cluster
----
2020-08-11 18:48:09 UTC - Aaron: What type of avro encoding does Pulsar use?
Binary, Single Object?
----
2020-08-11 18:52:27 UTC - Allen ONeill: I have the same use-case (almost) and
havn't found a timer/cron function for Pulsar functions. My solution therefore
is to make an external cron service that will send a message into a specific
topic (call it the pulse/heartbeat topic, whatever), and then have a function
watch for those specific messages/that topic and take that message as a trigger
to do whatever it is I need done at the heartbeat time.
hth.
Allen.
----
2020-08-11 18:57:03 UTC - Allen ONeill: Cant see anything in documentation but
does this indicate single object casting?
```Define Avro schemas with Schema.AVRO. The following code snippet
demonstrates how to create and use Avro schema.
Producer<MyAvro> avroProducer =
client.newProducer(Schema.AVRO(MyAvro.class))
.topic("some-avro-topic")
.create();```
from here:
<https://pulsar.apache.org/docs/en/client-libraries-java/#schemas>
----
2020-08-11 18:58:29 UTC - Yarden Arane: Hi Allen, thanks so much for your
reply. Will likely go down the external cron service path as well.
----
2020-08-11 19:01:28 UTC - Aaron: Interesting. When forming a GenericAvroRecord
it looks like Pulsar is using the binary encoding. (see line 99
<https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroReader.java>)
----
2020-08-11 19:48:30 UTC - Aaron: To clarify, I am wondering what the encoding
is when using AUTO_PRODUCE to a topic with an avro schema
----
2020-08-11 22:01:39 UTC - Jay Sun: Hi team, I am wondering how to refresh the
tls cert and key for a long-lived consumer? especially for go client lib, but I
am ok if you show me java code. Thanks
----
2020-08-11 22:13:22 UTC - Addison Higham: will answer in dev-go
:slightly_smiling_face:
----
2020-08-11 22:18:01 UTC - Jay Sun: Thanks. In fact I am waiting there for one
day already.. not sure it is being monitored..
----
2020-08-11 23:03:01 UTC - Guillaume: Hello eveyrone,
I'd like to know if there is any way to know which clients are connected to
Pulsar?
Thanks
----
2020-08-11 23:17:00 UTC - Allen ONeill: if by client you mean message
consumers, you can get this information by querying the admin API ... check
getStats for example
<https://pulsar.apache.org/docs/en/admin-api-persistent-topics/#get-stats>
<https://pulsar.apache.org/admin-rest-api/?version=2.6.0&apiversion=v2#operation/getStats>
@Chris Bartholomew led me down this useful little path :)
----
2020-08-12 07:24:12 UTC - Hiroyuki Yamada: @Penghui Li I’m sorry for pushing,
but can you give me the updates on this ?
<https://github.com/apache/pulsar/issues/7455#issuecomment-671699106>
For example, no clue at all at the moment ? or is someone investigating or
woking on it ?
If no one is doing anything, I need to dig into it by myself or need to explore
different approaches or different platforms.
----
2020-08-12 07:25:34 UTC - Penghui Li: Ok
----
2020-08-12 08:44:10 UTC - Dim: @Dim has joined the channel
----
