2019-06-28 15:31:45 UTC - Zhenhao Li: hi guys, I am new to Pulsar but very interested in trying it out ---- 2019-06-28 15:32:45 UTC - Zhenhao Li: is there a plan to provide a Scala API for Pulsar function? ---- 2019-06-28 15:37:32 UTC - Matteo Merli: Not that I know of, thought it shouldn’t be too difficult to wrap the existing Java API ---- 2019-06-28 15:49:17 UTC - Zhenhao Li: indeed not ---- 2019-06-28 15:53:38 UTC - Addison Higham: how open would ya'll be towards support for adding annotations to pods in the kubernetes functions runtime? I imagine that should be some new top level config item that could be a blob of keys/values that the runtime could interpret ---- 2019-06-28 16:25:57 UTC - Zhenhao Li: I find this part a bit odd when reading <https://streaml.io/blog/pulsar-streaming-queuing> ---- 2019-06-28 16:26:44 UTC - Zhenhao Li: it seems to be talking about different things. ---- 2019-06-28 16:27:07 UTC - Zhenhao Li: shouldn't it be mentioned where Pulsar stores offsets? ---- 2019-06-28 16:29:39 UTC - Matteo Merli: Offsets (or more properly: subscription state) is stored in BookKeeper too ---- 2019-06-28 16:46:54 UTC - Addison Higham: also... along those lines... trying to think what the right way is to achieve what I am going for with k8s functions runtime and tenants:
Background: I have multiple tenants, one of those tenants should be privileged to run their pulsar functions in a k8s namespace that has a privileged pool of compute with network access to DBs to do CDC. Ideally, I would be able to have other tenants use a different pool of compute for their pulsar functions. The natural way to do this would be with being able to map a tenant to k8s namespace, but because the namespace you configure for the k8s runtime is static for the cluster, all tenants share a namespace. That leaves two options: run a "virtual" broker, which could probably just re-use the existing BK/ZK of the "main" cluster and would be a tenant just mapped to that cluster for these privileged functions. The other option, which seems more scalable, would be able to define a namespace for each tenant. Ideally, only the super-user would be able to set that ---- 2019-06-28 16:53:52 UTC - Zhenhao Li: thanks. does BookKeeper needs ZK? or Pulsar needs both BookKeeper and ZK? ---- 2019-06-28 16:54:34 UTC - Matteo Merli: Both Pulsar and BookKeeper use ZK for metadata ---- 2019-06-28 17:52:18 UTC - Jerry Peng: @Addison Higham you can currently attach custom labels to all function pods, we can also add attaching annotations but they kind of serve the same purpose ---- 2019-06-28 17:54:03 UTC - Addison Higham: certain things, like using kiam on AWS, AFAIK, requires the annotation as opposed to the label ---- 2019-06-28 17:54:33 UTC - Addison Higham: same with prometheus metrics annotation (but it looks like it adds that one :slightly_smiling_face: ) ---- 2019-06-28 18:27:19 UTC - chris: @Addison Higham have you looked into mutating webhooks? this would allow you to change anything in the function spec before it’s sent to the kubernetes api server. <https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74> ---- 2019-06-28 18:29:26 UTC - Addison Higham: I had considered it, but it seemed a bit heavy handed for adding the annotation but could maybe work for isolating... ---- 2019-06-28 18:32:13 UTC - chris: yes, but it could also be used to ensure the namespace is there per tenant and it could allow you to isolate functions into multiple kubernetes namespaces ---- 2019-06-28 18:34:27 UTC - Addison Higham: yeah, that is what I was thinking, I would need some metadata, ideally, added by the broker, that would let me know which tenant and function it is so I could whitelist those ---- 2019-06-28 20:44:31 UTC - Aaron: @Matteo Merli Can you follow up that this works with a standalone? ---- 2019-06-28 22:08:30 UTC - Zhenhao Li: cool. thank you! ---- 2019-06-28 23:10:02 UTC - Gilberto Muñoz Hernández: Hi, am trying to use the hdfs2 sink connector over a byte schema topic, but am getting a io error ---- 2019-06-28 23:11:00 UTC - Gilberto Muñoz Hernández: Is it a bug? Or can't be used with a schema different from string? ---- 2019-06-28 23:12:49 UTC - Gilberto Muñoz Hernández: I noticed there is an encode option for utf8 and ascii in the sink properties but i know that my messages cant be encoded with string formats like those ---- 2019-06-28 23:13:38 UTC - Ali Ahmed: do you have logs ? ---- 2019-06-28 23:14:09 UTC - Gilberto Muñoz Hernández: Sorry i am in my phone right now ---- 2019-06-28 23:14:10 UTC - David Kjerrumgaard: @Gilberto Muñoz Hernández Can you submit an issue for that in Github? I wrote that connector before Pulsar had a schema registry ---- 2019-06-28 23:14:56 UTC - Gilberto Muñoz Hernández: Yes but i dont want to use pulsar schemas ---- 2019-06-28 23:15:16 UTC - Gilberto Muñoz Hernández: Those have more bugs than a bee hive ---- 2019-06-28 23:15:32 UTC - Gilberto Muñoz Hernández: Just want to sink raw bytes ---- 2019-06-28 23:15:39 UTC - Gilberto Muñoz Hernández: Can't i? ---- 2019-06-28 23:15:47 UTC - David Kjerrumgaard: @Gilberto Muñoz Hernández That fine, just enough info to reproduce the issue and I will work on a fix ---- 2019-06-28 23:16:51 UTC - Gilberto Muñoz Hernández: Ok thanks, i will submit the bug first thing in the morning with more info +1 : David Kjerrumgaard ---- 2019-06-28 23:17:02 UTC - Gilberto Muñoz Hernández: Thanks again @David Kjerrumgaard ---- 2019-06-28 23:18:04 UTC - David Kjerrumgaard: @Gilberto Muñoz Hernández It looks like I only wrote one to support Strings, i.e. `HdfsStringSink`. ---- 2019-06-28 23:18:35 UTC - David Kjerrumgaard: I can add one that supports `byte[]` as well ---- 2019-06-29 01:18:36 UTC - Sijie Guo: I think the blog post has some problems around the offset part which might need to be fixed. ----
