2019-04-29 11:11:51 UTC - Poule: I follow the `functions` docs and when I
create a function, server standalone tells me `ImportError: No module named
_pulsar`
----
2019-04-29 11:12:23 UTC - Poule: comes from
`apache-pulsar-2.3.1/instances/python-instance/pulsar/__init__.py", line 99`
----
2019-04-29 11:14:40 UTC - Alexandre DUVAL: Hi, where can I find the last
archive of the master branch, even snapshots versions?
----
2019-04-29 11:26:56 UTC - Poule: ok fixed, I started the standalone in a
`pipenv` env with python 2.7.15 and it works now
----
2019-04-29 11:29:07 UTC - jia zhai: @Alexandre DUVAL
<https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/>
hope this will hep you.
----
2019-04-29 11:30:17 UTC - Alexandre DUVAL: @jia zhai it does, thank you.
----
2019-04-29 11:30:37 UTC - jia zhai: welcome:+1:
----
2019-04-29 11:32:54 UTC - Alexandre DUVAL: @jia zhai oh, nope, it's not the
built one. I need it as bin archive :stuck_out_tongue:.
----
2019-04-29 11:37:58 UTC - jia zhai: @Sijie Guo Is there a binary for nightly
build?
----
2019-04-29 11:40:21 UTC - Sijie Guo: @jia zhai: it used to publish here -
<https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/distribution/2.4.0-SNAPSHOT/>
----
2019-04-29 11:40:36 UTC - Sijie Guo: but it seems that it stopped publishing
----
2019-04-29 11:41:38 UTC - Sijie Guo: I am not sure if it is related to
<https://github.com/apache/pulsar/commit/8365808ae66eafc2bc15c2dfdd4c1e98315c09ec>
----
2019-04-29 11:43:05 UTC - jia zhai: seems in 2.2.0 there is also no binary
left:<https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/distribution/2.2.0-incubating-SNAPSHOT/>
----
2019-04-29 11:44:39 UTC - Sijie Guo:
<https://repository.apache.org/content/repositories/snapshots/org/apache/pulsar/pulsar-server-distribution/2.4.0-SNAPSHOT/>
----
2019-04-29 11:44:40 UTC - Sijie Guo: nvm here you go @Alexandre DUVAL
----
2019-04-29 11:45:09 UTC - Sijie Guo: I checked the wrong location
----
2019-04-29 11:45:21 UTC - Alexandre DUVAL: Cool! Ty.
----
2019-04-29 11:46:32 UTC - jia zhai: seems this parameter
“skipBuildDistribution” is not used
----
2019-04-29 11:48:45 UTC - Sijie Guo: yes because it is not defined. it only
skips deploying the package when `-DskipBuildDistribution` is provided
----
2019-04-29 12:13:11 UTC - Alexandre DUVAL: For HTTP requests on admin rest api,
how the auth is provided on the http request (which headers, which format), I
don't find these informations on <https://pulsar.apache.org/admin-rest-api/>.
More, why there is no HTTP 200 response possibility on
<https://pulsar.apache.org/admin-rest-api/#operation/createTenant>?
----
2019-04-29 12:16:55 UTC - Alexandre DUVAL: (I need to setup a tenant/namespace
creation if they don't exist during context.publish in pulsar function). What's
the better way to do it? I'm trying stuff like this: ``` context.publish(
new StringBuilder("persistent://")
.append(ownerId)
.append("/")
.append(appId)
.append("/")
.append("logs")
.toString(),
input
).exceptionally(exception -> {
if (exception instanceof PreconditionFailedException) { //
tenant/namespace couple does not exists (HTTP 412)
createTenantAndNamespaceFor(ownerId, appId);
// then retry to publish
}
})```
----
2019-04-29 12:25:55 UTC - Alexandre DUVAL: Or maybe the best way is to reuse
pulsarAdmin methods directly from pulsar function?
----
2019-04-29 12:30:13 UTC - Sijie Guo: > how the auth is provided on the http
request (which headers, which format), I don’t find these informations on
<https://pulsar.apache.org/admin-rest-api/>
ah yeah. the documentation around authentication is missing for rest api.
> maybe the best way is to reuse pulsarAdmin methods directly from pulsar
function?
if you are using java to implement a pulsar function, it might be worth just
using the pulsar-admin library.
----
2019-04-29 12:39:07 UTC - Poule: in 2.3.1 is it possible that `pulsar-admin
function trigger`calls the wrong function?
----
2019-04-29 12:41:24 UTC - Poule: i created `reverse` and `exclamation` and
calling `exclamation` gives me the `reverse` result.
----
2019-04-29 12:42:03 UTC - Poule: I delete the `reverse` function then I get the
correct result when calling `exclamation`
----
2019-04-29 12:50:43 UTC - Poule: i will investigate more later
----
2019-04-29 13:41:24 UTC - Shivji Kumar Jha: I am trying to route the prometheus
metrics pulsar exposes at broker-url:8080/metrics to telegraf/ Telegraf errors
out saying prometheus metrics output is not in correct format.
Error:
2019-04-29T13:38:40Z E! Error in plugin [inputs.prometheus]: error reading
metrics for <broker-host>:8080/metrics: reading text format failed: text
format parsing error in line 189: second TYPE line for metric name
“pulsar_subscriptions_count”, or TYPE reported after samples
Here s what the output of /metric line 189 where the error is pointing
# TYPE pulsar_subscriptions_count gauge
pulsar_subscriptions_count{cluster=“stage”,namespace=“beam/sc-realtime”,topic=“<persistent://beam/test-ns/test-topic4%22|persistent://beam/test-ns/test-topic4">}
1.0 1556541781642
----
2019-04-29 13:44:50 UTC - Shivji Kumar Jha: seems like
<https://github.com/apache/pulsar/issues/3112>
----
2019-04-29 13:44:56 UTC - Shivji Kumar Jha: Anyone else faced this? how did you
fix this?
----
2019-04-29 14:05:36 UTC - Alexandre DUVAL: ```# Enable Functions Worker Service
in Broker
functionsWorkerEnabled=false``` this field in broker.conf is related to
activate broker as function worker or it should be enabled for the functions
worker node to work?
----
2019-04-29 14:05:57 UTC - Alexandre DUVAL: Because I have the following issue
```bin/pulsar-admin functions list
Function worker service is not done initializing. Please try again in a little
while.
Reason: HTTP 503 Service Unavailable```
----
2019-04-29 14:06:26 UTC - Alexandre DUVAL: But my functions worker node is
running: ```13:55:10.274 [pulsar-client-io-7-1] INFO
org.apache.pulsar.client.impl.ConsumerImpl -
[<persistent://yo/functions/assignments>][reader-37f12b46a5] Subscribing to
topic on cnx [id: 0xf51b9e1a, L:/10.2.1.4:33364 -
R:<http://c1-pulsar-yo-customers.services.clever-cloud.com/185.42.117.147:2001|c1-pulsar-yo-customers.services.clever-cloud.com/185.42.117.147:2001>]
13:55:10.279 [pulsar-client-io-7-1] INFO
org.apache.pulsar.client.impl.ConsumerImpl -
[<persistent://yo/functions/assignments>][reader-37f12b46a5] Subscribed to
topic on
<http://c1-pulsar-yo-customers.services.clever-cloud.com/123.213.123.123:2001|c1-pulsar-yo-customers.services.clever-cloud.com/123.213.123.123:2001>
-- consumer: 2
13:55:10.280 [main] INFO org.apache.pulsar.client.impl.ConsumerImpl -
[<persistent://yo/functions/assignments>][reader-37f12b46a5] Get topic last
message Id
13:55:10.284 [pulsar-client-io-7-1] INFO
org.apache.pulsar.client.impl.ConsumerImpl -
[<persistent://yo/functions/assignments>][reader-37f12b46a5] Successfully
getLastMessageId 120:-1
13:55:10.293 [main] INFO
org.apache.pulsar.broker.authentication.AuthenticationService - Authentication
is disabled
13:55:10.294 [main] INFO org.apache.pulsar.functions.worker.WorkerService -
Start cluster services...
13:55:10.295 [main] INFO
org.apache.pulsar.functions.worker.FunctionRuntimeManager - /** Starting
Function Runtime Manager **/
13:55:10.296 [main] INFO
org.apache.pulsar.functions.worker.FunctionRuntimeManager - Initialize metrics
sink...
13:55:10.296 [main] INFO
org.apache.pulsar.functions.worker.FunctionRuntimeManager - Starting function
assignment tailer...
13:55:10.296 [main] INFO org.apache.pulsar.functions.utils.io.ConnectorUtils -
Searching for connectors in /opt/apache-pulsar-2.3.1/./connectors
13:55:10.296 [main] WARN org.apache.pulsar.functions.utils.io.ConnectorUtils -
Connectors archive directory not found
13:55:10.310 [main] INFO org.eclipse.jetty.util.log - Logging initialized
@4296ms to org.eclipse.jetty.util.log.Slf4jLog
13:55:10.452 [main] INFO org.eclipse.jetty.server.Server -
jetty-9.4.12.v20180830; built: 2018-08-30T13:59:14.071Z; git:
27208684755d94a92186989f695db2d7b21ebc51; jvm 1.8.0_192-b12
13:55:10.720 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@77ad2767{/admin,null,AVAILABLE}
13:55:10.789 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@e460ca1{/admin/v2,null,AVAILABLE}
13:55:10.908 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@411ae0d7{/admin/v3,null,AVAILABLE}
13:55:10.943 [main] INFO org.eclipse.jetty.server.handler.ContextHandler -
Started o.e.j.s.ServletContextHandler@608c2042{/,null,AVAILABLE}
13:55:10.954 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started
ServerConnector@4f571c0e{HTTP/1.1,[http/1.1]}{0.0.0.0:2000}
13:55:10.958 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started
ServerConnector@3f919230{SSL,[ssl, http/1.1]}{0.0.0.0:2001}
13:55:10.965 [main] INFO org.eclipse.jetty.server.Server - Started @4952ms
13:55:10.965 [main] INFO org.apache.pulsar.functions.worker.rest.WorkerServer
- Worker Server started at <http://192.168.10.16:2000/admin>
13:55:10.965 [main] INFO org.apache.pulsar.functions.worker.Worker - Start
worker server on port 2000...
````
----
2019-04-29 14:06:44 UTC - Alexandre DUVAL: Or it's another problem? :smile:
----
2019-04-29 14:07:02 UTC - Alexandre DUVAL: (@Sijie Guo maybe :))
----
2019-04-29 14:09:32 UTC - Sijie Guo: @Alexandre DUVAL :
`functionsWorkerEnabled=false` is for enabling funcation worker on broker.
if you have run a separated function worker, specifying your function worker
location when you using `bin/pulsar-admin functions`.
`bin/pulsar-admin --admin-url http://<function-worker>:6750 functions`
----
2019-04-29 14:09:51 UTC - Alexandre DUVAL: Oh, ok :slightly_smiling_face:.
----
2019-04-29 14:28:18 UTC - Alexandre DUVAL: Ok, I still have problems on
authentication configuration on functions worker node.
----
2019-04-29 14:28:20 UTC - Alexandre DUVAL:
```############################################
# security settings for worker service
############################################
# Enforce authentication
authenticationEnabled: true
# Enforce authorization on accessing functions api
authorizationEnabled: true
# Set of autentication provider name list, which is a list of class names
authenticationProviders:
- org.apache.pulsar.broker.authentication.AuthenticationProviderToken
# Set of role names that are treated as "super-user", meaning they will be able
to access any admin-api
superUserRoles:
- super-user
tokenPublicKey: file:///pulsar/conf/keys/my-public.key```
----
2019-04-29 14:28:32 UTC - Alexandre DUVAL: I don't know how to write them.
----
2019-04-29 14:28:44 UTC - Alexandre DUVAL: ```14:24:51.797 [main] ERROR
org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker
org.apache.pulsar.broker.PulsarServerException: Failed to load an
authentication provider.
at
org.apache.pulsar.broker.authentication.AuthenticationService.<init>(AuthenticationService.java:64)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
at
org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:179)
[org.apache.pulsar-pulsar-functions-worker-2.3.1.jar:2.3.1]
at org.apache.pulsar.functions.worker.Worker.start(Worker.java:51)
[org.apache.pulsar-pulsar-functions-worker-2.3.1.jar:2.3.1]
at
org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:67)
[org.apache.pulsar-pulsar-functions-worker-2.3.1.jar:2.3.1]
Caused by: java.io.IOException: No secret key was provided for token
authentication
at
org.apache.pulsar.broker.authentication.AuthenticationProviderToken.getValidationKey(AuthenticationProviderToken.java:135)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
at
org.apache.pulsar.broker.authentication.AuthenticationProviderToken.initialize(AuthenticationProviderToken.java:61)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
at
org.apache.pulsar.broker.authentication.AuthenticationService.<init>(AuthenticationService.java:56)
~[org.apache.pulsar-pulsar-broker-common-2.3.1.jar:2.3.1]
... 3 more
```
----
2019-04-29 14:30:03 UTC - Alexandre DUVAL: (@Sijie Guo :p)
----
2019-04-29 15:12:26 UTC - Dan C: @Dan C has joined the channel
----
2019-04-29 15:12:59 UTC - Laurent Chriqui: Hello, everyone, we are experiencing
TimeOut errors when publishing messages to topics using the python client with
async producers.
We have the option batching enabled and block if queue full enabled with a
batching max publish delay at 10 ms.
Has anyone encountered this problem yet ?
----
2019-04-29 15:17:53 UTC - Chris Bartholomew: @Shivji Kumar Jha Yes, I have seen
this. I was able to workaround it by disabling topic-level metrics. You do that
by setting ```exposeTopicLevelMetricsInPrometheus=false```
in broker.conf. I am working on a fix for this. Just doing some final testing.
I should be submitting a PR today.
----
2019-04-29 15:22:38 UTC - Shivji Kumar Jha: Great, i will do that and try it
out. Thank you :blush:
----
2019-04-29 15:31:13 UTC - Alexandre DUVAL: or @David Kjerrumgaard? or @Matteo
Merli?
----
2019-04-29 15:46:35 UTC - David Kjerrumgaard: According to the error message,
"No secret key was provided for token authentication", It looks like you didn't
specify a private key for the token.
----
2019-04-29 15:46:51 UTC - David Kjerrumgaard: And the config file you shared
only listed a public key.
----
2019-04-29 15:50:37 UTC - Alexandre DUVAL: But if I'm using public key as auth,
why do I need secret?
----
2019-04-29 15:55:46 UTC - Alexandre DUVAL: I mean i use token generated from
private key, and i give the public to the node.
----
2019-04-29 16:00:34 UTC - David Kjerrumgaard: You will need a private key on
your broker.
----
2019-04-29 16:00:35 UTC - David Kjerrumgaard:
<http://pulsar.apache.org/docs/en/security-token-admin/>
----
2019-04-29 16:04:01 UTC - Alexandre DUVAL: It's write public or secret on this
doc, no?
----
2019-04-29 16:05:43 UTC - Marc Le Labourier: We don't know yet if the batching
is not working properly, but it seems that our brokers are running at full
capacity (Memory and CPU). Should it not be regulated before it happens ?
----
2019-04-29 16:07:12 UTC - David Kjerrumgaard: When you configure token
authentication, you need to specify the 'tokenSecretKey' property in your
broker config, but it looks like you have 'tokenPublicKey' instead
----
2019-04-29 16:08:10 UTC - Alexandre DUVAL: No, I use asym tokens generated not
using secret. That's my point.
----
2019-04-29 16:08:25 UTC - Alexandre DUVAL: ````# If using secret key
tokenSecretKey=file:///path/to/secret.key
# The key can also be passed inline:
# tokenSecretKey=data:base64,FLFyW0oLJ2Fi22KKCm21J18mbAdztfSHN/lAT5ucEKU=
# If using public/private
# tokenPublicKey=file:///path/to/public.key``````
----
2019-04-29 16:08:27 UTC - Alexandre DUVAL: if*
----
2019-04-29 16:15:32 UTC - David Kjerrumgaard: With JWT Symmetric tokens there
is a single Secret key that is used both to generate and validate. So it has to
be labelled as "secret" in the broker config file AFAIK
----
2019-04-29 16:16:38 UTC - Alexandre DUVAL: Yes, but there are asymetric tokens
that im using :stuck_out_tongue:.
----
2019-04-29 16:17:09 UTC - Alexandre DUVAL: And not symetrics ones
----
2019-04-29 16:19:33 UTC - David Kjerrumgaard: Either way, the broker config
expects a property named "tokenSecretKey=file:///path/to/secret.key" and you
have tokenPublicKey=file:///path/to/public.key```
----
2019-04-29 16:20:14 UTC - David Kjerrumgaard: So the error is complaining that
you configured token based authentication but did NOT provide a required
property, namely "tokenSecretKey"
----
2019-04-29 16:20:45 UTC - David Kjerrumgaard: you provided "tokenPublicKey",
which the broker doesn't recognize
----
2019-04-29 16:21:03 UTC - Alexandre DUVAL: In the documentation it is write:
----
2019-04-29 16:21:11 UTC - Alexandre DUVAL: ``# If using secret key
tokenSecretKey=file:///path/to/secret.key````
----
2019-04-29 16:21:18 UTC - Alexandre DUVAL: and
----
2019-04-29 16:21:20 UTC - Alexandre DUVAL: ``# If using public/private
# tokenPublicKey=file:///path/to/public.key```````
----
2019-04-29 16:21:31 UTC - Alexandre DUVAL: sorry windows char for ` is bad^^^.
----
2019-04-29 16:22:04 UTC - David Kjerrumgaard: "Caused by: java.io.IOException:
No secret key was provided for token authentication
at
org.apache.pulsar.broker.authentication.AuthenticationProviderToken.getValidationKey(AuthenticationProviderToken.java:135)
"
----
2019-04-29 16:22:22 UTC - Alexandre DUVAL: yes, I read this, but it's on
functions worker only.
----
2019-04-29 16:22:36 UTC - Alexandre DUVAL: Broker have publictokenkey only and
it's work perfectly
----
2019-04-29 16:22:40 UTC - Alexandre DUVAL: you get the point?
----
2019-04-29 16:22:50 UTC - David Kjerrumgaard: ^^^ Above is complaining about
the key. Is the path to the key correct? Are file permissions correct nd
allow read access?
----
2019-04-29 16:23:15 UTC - David Kjerrumgaard: So the error is in the function
worker?
----
2019-04-29 16:23:17 UTC - Alexandre DUVAL: That's my point, I don't know what
to write the configuration using yaml, because functions worker conf uses yaml
----
2019-04-29 16:24:21 UTC - Alexandre DUVAL: There are no documentation on the
way to write the configuration field using file in yaml.
----
2019-04-29 16:27:25 UTC - Devin G. Bost: Is there a particular reason that
`FunctionConfigBuilder` is not public?
----
2019-04-29 16:39:55 UTC - Dan C: Hi all. I’m attempting to use the websocket
API (broker embedded) on a local minikube. When running a test consumer, I
receive “Handshake status 404 Not Found”. I’ve confirmed that the API is
enabled on the broker config. I am using the pulsar proxy.
----
2019-04-29 16:41:48 UTC - David Kjerrumgaard: @Dan C Can you confirm that you
have configured port forwarding between the machine where the websocket client
is running and the broker inside the minikube container?
----
2019-04-29 16:44:06 UTC - Dan C: I have the proxy service running and the port
forwarding appears correct (30002 -> 8080 on the container)
----
2019-04-29 16:44:58 UTC - David Kjerrumgaard: So you have completed both of
these steps?
----
2019-04-29 16:46:13 UTC - David Kjerrumgaard: I would try using the standard
auth settings in the broker.conf file FIRST. if that doesn't work, then we can
go to the function worker.yaml file. I suspect that the auth properties are
picked up from the borker.conf file
----
2019-04-29 16:48:15 UTC - David Kjerrumgaard: Can you telnet to that address
and port? (confirm networking, etc first).
----
2019-04-29 17:14:24 UTC - Shivji Kumar Jha: That didnt work out :disappointed:
Because you are already working on this part can you run your test for my
prometheus output maybe? Let me know if you need anything else from my
prometheus output.
----
2019-04-29 17:59:31 UTC - Devin G. Bost: Nevermind. I figured out how to use it.
----
2019-04-29 18:06:10 UTC - Alexandre DUVAL: Okay, but no @Sijie Guo told that
functionworkerenabled=true is only to run function on broker but im running a
separated node for functions only.
----
2019-04-29 18:06:51 UTC - Alexandre DUVAL: And function cluster is correctly
configured yes
----
2019-04-29 18:07:34 UTC - Dan C: I am able to telnet and also reach the admin
REST endpoint. It seems that it is just the websocket endpoint that isn’t
working via the proxy, although it is configured to run on the broker web
service.
----
2019-04-29 18:09:05 UTC - Alexandre DUVAL: And it uses the yaml file cause it's
work when i disable auth.
----
2019-04-29 18:09:50 UTC - Matteo Merli: Each authentication plugin will have a
different way to pass the credentials. eg:
* TLS --> client certificate
* Tokens > `Authorization: Bearer xxxxxxxx`
----
2019-04-29 18:10:43 UTC - Dan C: If I ssh to a broker pod, I can access it
though. So it appears to just not work via the proxy service.
----
2019-04-29 18:11:04 UTC - Matteo Merli: @Shivji Kumar Jha This PR fixes that:
<https://github.com/apache/pulsar/pull/4136>
----
2019-04-29 18:11:22 UTC - Alexandre DUVAL: But without auth, i cant create
function on it.
----
2019-04-29 18:13:38 UTC - Alexandre DUVAL: WDYT @Matteo Merli (sorry there is
no TL; DR :/)
----
2019-04-29 18:14:46 UTC - Alexandre DUVAL: Thanks for the answer
----
2019-04-29 18:40:33 UTC - Dan C: Changing the broker service to a NodePort
resolved this.
----
2019-04-29 18:42:47 UTC - Dan C: I was connecting to the proxy service
webservice as opposed to the broker webservice which has the websocket api
running.
+1 : David Kjerrumgaard
----
2019-04-29 19:08:56 UTC - Alexandre DUVAL: @Matteo Merli here is the
configuration of a separated functions worker
----
2019-04-29 19:09:22 UTC - Alexandre DUVAL: @Matteo Merli and here is the error,
i can't understand why it requires secret because im using asym with publickey
----
2019-04-29 21:36:08 UTC - Ryan Dietrich: @Ryan Dietrich has joined the channel
----
2019-04-29 21:37:23 UTC - Ryan Dietrich: Hi, quick questions using node.js and
pulsar. 1. Is the websocket API the best option right now? 2. Using
websockets, can you subscribe to multiple topics on a single connection, or is
it one connection per topic?
----
2019-04-29 21:38:26 UTC - Ali Ahmed: @Ryan Dietrich you can try the binary
protocol node js client , it’s still under development .
<https://github.com/apache/pulsar-client-node>
----
2019-04-29 21:38:39 UTC - Ryan Dietrich: Woah, cool! I had no idea that was in
the works!
----
2019-04-29 21:39:51 UTC - Guy Feldman: @Ryan Dietrich it's fairly new. There's
a <#CFTUA9AHK|dev-node> for discussion of the library
----
2019-04-29 21:41:13 UTC - Ryan Dietrich: that's really great news.. thank you
both for the quick reply!
----
2019-04-29 21:42:17 UTC - Ryan Dietrich: My place of work is having a bake-off
between Pulsar and Kafka, and after reading about embedded functions and
non-persistent storage I became a fan, just gotta deliver a solid
implementation now.
----
2019-04-29 21:43:15 UTC - Guy Feldman: we chose pulsar over kafka, mostly for
security and multi-tenancy
----
2019-04-29 21:43:40 UTC - Guy Feldman: we're using the websocket api cause the
node client wasn't around yet
----
2019-04-29 21:44:11 UTC - Guy Feldman: and the performance has been good too
----
2019-04-29 21:45:06 UTC - Ryan Dietrich: do you happen to know the speed
difference between persistent and non-persistent storage?
----
2019-04-29 21:45:11 UTC - Ali Ahmed: @Ryan Dietrich I am guessing you are in
finance , you should know pulsar offers much stronger durability guarantees
then kafka, that alone should a reason to switch for mission critical
applications
----
2019-04-29 21:45:28 UTC - Guy Feldman: i do not. we haven't used non-persistent
yet
----
2019-04-29 21:45:29 UTC - Ryan Dietrich: @Ali Ahmed uh oh, you looked me up
didn't you :wink:
----
2019-04-29 21:45:48 UTC - Guy Feldman: it's been fast enough with persistent
storage
----
2019-04-29 21:45:49 UTC - Ryan Dietrich: I just have always had a thing for
non-critical messages not having to go to disk.
----
2019-04-29 21:46:21 UTC - Ryan Dietrich: The idea that "not all messages are
created equal" is something I think is valuable.
----
2019-04-29 21:46:43 UTC - Ryan Dietrich: just my opinion
----
2019-04-29 21:46:46 UTC - Guy Feldman: personally what sold me is the s3
offloading.
+1 : Matteo Merli
----
2019-04-29 21:47:04 UTC - Guy Feldman: you can always put those messages in a
short lived topic and offload it
----
2019-04-29 21:47:12 UTC - Ali Ahmed: I don’t think anyone has benchmarked
non-persistent topics. so don’t know how big is the perf delta.
----
2019-04-29 21:47:54 UTC - Ryan Dietrich: well, I will, in places where we don't
send "deltas" the latest messages is only the most important, so non-persistent
would be better in that case
----
2019-04-29 21:48:01 UTC - Ryan Dietrich: also, I'm ignorant of what s3
offloading does?
----
2019-04-29 21:48:34 UTC - Guy Feldman: you can basically set a size or time
where messages past a certain time are stored in s3
----
2019-04-29 21:48:37 UTC - Guy Feldman: rather than disk
----
2019-04-29 21:48:51 UTC - Guy Feldman: so you have infinite retention if you
want but only recent messages are on disk
----
2019-04-29 21:49:33 UTC - Ryan Dietrich: I assume you treat it as a
cold-storage, as the access time is a tad slower, right?
----
2019-04-29 21:49:37 UTC - Guy Feldman: yeah
----
2019-04-29 21:49:39 UTC - Ali Ahmed: it’s a transparent tiered storage layer so
you can build a cost efficient event store
----
2019-04-29 21:50:03 UTC - Guy Feldman: yeah the benefit is you can connect
tools that use pulsar and still query the data
----
2019-04-29 21:50:04 UTC - Guy Feldman: like presto
----
2019-04-29 21:50:34 UTC - Ryan Dietrich: Our devops team has complained that
dealing with zookeeper was their biggest fear for either project, has that been
an issue for you?
----
2019-04-29 21:51:09 UTC - Guy Feldman: not for us
+1 : Ryan Dietrich
----
2019-04-29 21:52:56 UTC - Guy Feldman: we're using zookeeper for other things
though too
----
2019-04-29 21:53:00 UTC - Ali Ahmed: pulsar depends upon but doesn’t
excessively put load on zk, it use bookkeeper for a lot heavy lifting so zk
does not become a bottleneck. It can therefore support millions of topics
----
2019-04-29 21:53:36 UTC - Guy Feldman: we're using zookeeper for flink ha too
----
2019-04-29 21:55:01 UTC - Ryan Dietrich: last question (you guys have been
super awesome btw). can you give me any details on how replication between
nodes works? If you have a cluster of pulsars, does it end up being having
multiple points of failure, or do you get increased performance and stability?
(that question may sound weird, but please trust me I am asking it for the
right reasons).
----
2019-04-29 21:56:10 UTC - Ali Ahmed: pulsar has two layers stateless brokers
for serving data and distributed storage layer bookies for persistence.
----
2019-04-29 21:56:38 UTC - Ali Ahmed: bookies transparent keep the required
number of copies distributed among them in custer
----
2019-04-29 21:57:10 UTC - Ryan Dietrich: at a technical level, does it use Akka
by chance?
----
2019-04-29 21:57:16 UTC - Ali Ahmed: you can scale brokers and bookies
independently of each other depending on the workload
----
2019-04-29 21:57:44 UTC - Ali Ahmed: no akka is not used anywhere
----
2019-04-29 21:58:54 UTC - Ryan Dietrich: Phew.. Ok.. I had some incredibly bad
experiences with OpenDaylight recently (which uses Akka, and was hoping to
avoid debugging issues in there again)
----
2019-04-29 23:08:41 UTC - Devin G. Bost: How much of a change would it be for
us to be able to get a hash (MD5 or SHA1) of the Jar for a given component
(e.g. Function) in Pulsar? It would be helpful for determining if the Jar
version matches the version we have in our build repo.
----
2019-04-30 02:44:02 UTC - Ali Ahmed: @Devin G. Bost I think it should not be
that hard but the wider issue is how to manage function versioning
----
2019-04-30 03:56:11 UTC - Matt Grosso: @Matt Grosso has joined the channel
----
2019-04-30 05:57:11 UTC - Shivji Kumar Jha: thanks @Matteo Merli I will cherry
pick and this and give it a shot.
----
2019-04-30 08:01:42 UTC - Shivji Kumar Jha: @Chris Bartholomew disabling topic
level metrics works. telegraf can parse the prometheus output in that case.
----
