2018-09-25 15:36:56 UTC - Grant Wu: Are full Pulsar topics URLs? ---- 2018-09-25 15:37:08 UTC - Grant Wu: Is it appropriate to use a URL library to deal with them? ---- 2018-09-25 15:46:31 UTC - Grant Wu: yikes @Matteo Merli @Sijie Guo I cannot download the binary tarball ---- 2018-09-25 15:46:44 UTC - Grant Wu: The link on the download page is going to <http://apache.cs.utah.edu/incubator/pulsar/pulsar-2.1.1-incubating/apache-pulsar-2.1.1-incubating-bin.tar.gz> ---- 2018-09-25 15:46:49 UTC - Grant Wu: and that’s 404'ing ---- 2018-09-25 15:48:11 UTC - Sijie Guo: @Grant Wu ---- 2018-09-25 15:48:20 UTC - Sijie Guo: it is because pulsar graduated :slightly_smiling_face: ---- 2018-09-25 15:48:27 UTC - Sijie Guo: remove incubator path ---- 2018-09-25 15:48:31 UTC - Grant Wu: oh ---- 2018-09-25 15:48:33 UTC - Grant Wu: when’d that happen? ---- 2018-09-25 15:48:38 UTC - Sijie Guo: <http://apache.cs.utah.edu/pulsar/pulsar-2.1.1-incubating/apache-pulsar-2.1.1-incubating-bin.tar.gz> ---- 2018-09-25 15:48:40 UTC - Sijie Guo: here you go ---- 2018-09-25 15:49:22 UTC - Sijie Guo: I think there are bunch of PRs are merging to update the website ---- 2018-09-25 15:49:35 UTC - Grant Wu: Well, congratulations :tada: ---- 2018-09-25 15:49:43 UTC - Sijie Guo: <https://s.apache.org/DLL1> ---- 2018-09-25 15:49:56 UTC - Sijie Guo: here is the apache blog post about Pulsar going TLP tada : Matteo Merli, Ivan Kelly, Grant Wu, Steven King, Karthik Ramasamy, Sijie Guo, Ali Ahmed, jia zhai +1 : jia zhai bananadance : jia zhai thumbsup_all : jia zhai ---- 2018-09-25 15:51:02 UTC - Sijie Guo: @Sijie Guo set the channel topic: Pulsar graduated as a Top-Level project :tada: - Pulsar release 2.1.1-incubating - <https://pulsar.incubator.apache.org/release-notes/#2.1.1-incubating> ---- 2018-09-25 15:54:11 UTC - Grant Wu: wait so why are we still `-incubating` ---- 2018-09-25 15:55:59 UTC - Matteo Merli: Old releases are still “incubating” ---- 2018-09-25 15:56:03 UTC - Grant Wu: ah, okay ---- 2018-09-25 16:06:02 UTC - Matteo Merli: We’ll have few issues during the migration. Please report any problem with website, etc. ---- 2018-09-25 16:06:15 UTC - Matteo Merli: I’ll take a look at the above link ---- 2018-09-25 16:21:59 UTC - Guillem: @Guillem has joined the channel ---- 2018-09-25 16:25:06 UTC - Guillem: hi everybody wave : Matteo Merli ---- 2018-09-25 16:51:46 UTC - Dave Southwell: The latest pulsar release link is no good. <http://apache.osuosl.org/incubator/pulsar/pulsar-2.1.1-incubating/apache-pulsar-2.1.1-incubating-bin.tar.gz> Anyone know where I can get the 2.1.1 bits? ---- 2018-09-25 16:53:56 UTC - Matteo Merli: Sorry, the existing releases have been moved out of the incubator folder. I just opened a PR to fix the download page: <https://github.com/apache/pulsar/pull/2654>
In meantime, just go one folder up: <http://apache.osuosl.org/pulsar/pulsar-2.1.1-incubating/apache-pulsar-2.1.1-incubating-bin.tar.gz> +1 : Dave Southwell ---- 2018-09-25 16:54:25 UTC - Guillem: got a broad question that has probably been asked, but can't find any specific answers ---- 2018-09-25 16:54:44 UTC - Guillem: i'm interested in being able to develop pulsar functions with NodeJS ---- 2018-09-25 16:54:55 UTC - Guillem: but would prefer an approach not based on using the WS ---- 2018-09-25 16:55:09 UTC - Guillem: what would be involved in being able to do so? ---- 2018-09-25 16:55:42 UTC - Guillem: i guess having a runtime for NodeJS that uses some of the existing librares and or writing a SDK for NodeJS? ---- 2018-09-25 16:57:07 UTC - Sijie Guo: @Guillem pulsar has a c++ client. if you are familiar with nodejs, probably you can wrap it into a nodejs client :slightly_smiling_face: ---- 2018-09-25 16:58:02 UTC - Guillem: ok, that's great ---- 2018-09-25 16:58:24 UTC - Guillem: but then i guess i need to do something with the container environment where my function is gonna be running? ---- 2018-09-25 16:59:28 UTC - Guillem: so it can run nodejs code ---- 2018-09-25 17:01:54 UTC - Sijie Guo: :slightly_smiling_face: once there is a nodejs client, it would be pretty straightforward to add a runtime to support nodejs. we can help with that. we have talked about supporting nodejs before, however it was postponed due to the lack of nodejs client. if you are willing to help with a nodejs client, we can work together to add the support for running functions with nodejs ---- 2018-09-25 17:03:33 UTC - Guillem: that's great ---- 2018-09-25 17:03:57 UTC - Guillem: so as long as we can have a nodejs client, then everything else should be fairly simple ---- 2018-09-25 17:04:53 UTC - Sijie Guo: yes ---- 2018-09-25 17:06:01 UTC - Grant Wu: I am curious - why don’t you want to use WS? ---- 2018-09-25 17:07:15 UTC - Guillem: there's a couple of reasons ---- 2018-09-25 17:07:44 UTC - Guillem: one is performance, although i guess this is debatable. I presume a native approach would be faster than WS? ---- 2018-09-25 17:09:16 UTC - Guillem: i'm also concerned on whether i can get ack's and 'once' only processing with WS ---- 2018-09-25 17:09:32 UTC - Guillem: and also, thinking about having it as a clean approach to code ---- 2018-09-25 17:09:35 UTC - Sijie Guo: (@Guillem: this was the ticket for nodejs client - <https://github.com/apache/pulsar/issues/1070>) ---- 2018-09-25 17:09:40 UTC - Grant Wu: a clean approach to code? ---- 2018-09-25 17:10:42 UTC - Guillem: yeah, so when you write pulsar functions in pyhton you essentially just code (afaik) something like this: ---- 2018-09-25 17:10:48 UTC - Guillem: ```def process(input): return "{0}!".format(input)``` ---- 2018-09-25 17:10:57 UTC - Guillem: i'd rather have the equivalent in node ---- 2018-09-25 17:11:12 UTC - Grant Wu: What does this have to do with WebSockets vs the proto protocol ---- 2018-09-25 17:11:45 UTC - Guillem: i understand if i use websocket ---- 2018-09-25 17:11:52 UTC - Guillem: i will need to use a websockets library ---- 2018-09-25 17:12:00 UTC - Guillem: and have some additional code in every function ---- 2018-09-25 17:12:06 UTC - Guillem: to do the websockets bit ---- 2018-09-25 17:12:16 UTC - Grant Wu: And if you use the proto protocol, you will need to use a protocol buffers library and write code to do the protocol buffers bit ---- 2018-09-25 17:12:56 UTC - Guillem: maybe i'm missing some bits on the architecture here ---- 2018-09-25 17:13:03 UTC - Guillem: but i thought that when you had a native runtime ---- 2018-09-25 17:13:10 UTC - Guillem: you could just code the function itself ---- 2018-09-25 17:13:17 UTC - Guillem: and the runtime would take care of unpacking and packing ---- 2018-09-25 17:13:21 UTC - Guillem: in protocols buffers ---- 2018-09-25 17:13:29 UTC - Grant Wu: I guess I’m not very familiar with Pulsar functions - wait, why do they need to talk the protocol at all ---- 2018-09-25 17:13:37 UTC - Grant Wu: But you could presumably implement the same thing with WebSockets ---- 2018-09-25 17:14:54 UTC - Guillem: I understand the runtime (so what gets deployed as a docker container for the pulsar functions) include everything to get messages and post messages to the pulsar bus ---- 2018-09-25 17:15:12 UTC - Guillem: so your function essentially just receives the input data as a parameter ---- 2018-09-25 17:15:49 UTC - Grant Wu: Okay, yes ---- 2018-09-25 17:15:59 UTC - Grant Wu: How did you envision using websockets with Pulsar functions? ---- 2018-09-25 17:16:06 UTC - Grant Wu: I think that may have been a red herring that confused me… ---- 2018-09-25 17:17:16 UTC - Guillem: there's a sample ---- 2018-09-25 17:17:17 UTC - Guillem: gimme a sec ---- 2018-09-25 17:17:22 UTC - Guillem: example i should say ---- 2018-09-25 17:17:51 UTC - Guillem: <http://pulsar.apache.org/docs/latest/clients/WebSocket/#nodejs> ---- 2018-09-25 17:18:03 UTC - Grant Wu: Yes but that is not a Pulsar function ---- 2018-09-25 17:18:25 UTC - Grant Wu: That is just a regular WebSocket client, speaking the WebSocket protocol ---- 2018-09-25 17:19:18 UTC - Grant Wu: As you said, the Pulsar function interface is different - you just provide a stream processing function ---- 2018-09-25 17:19:31 UTC - Grant Wu: Using any of the “regular” clients will require managing connections ---- 2018-09-25 17:20:13 UTC - Grant Wu: So for example, there’s a Go official client ---- 2018-09-25 17:20:42 UTC - Grant Wu: But there’s no Go Pulsar Functions support yet ---- 2018-09-25 17:20:45 UTC - Guillem: mhhh, i know what you mean, yeah ---- 2018-09-25 17:21:29 UTC - Grant Wu: So I’m not sure what is meant when they say that part of the reason why we don’t have JS support is that there’s no official client - it seems that perhaps the Pulsar function framework uses the client under the hood ---- 2018-09-25 17:21:33 UTC - Grant Wu: @Sijie Guo is that accurate? ---- 2018-09-25 17:23:23 UTC - Grant Wu: Anyways, yeah, I do agree that if you’re going to create an official client it should probably be using the proto based protocol ---- 2018-09-25 17:24:07 UTC - Grant Wu: I wrote a very tiny WebSockets based NodeJS bare-bones client that implements only the most basic functionality of producing and consuming, and also calling resetCursor since we need that for our use case ---- 2018-09-25 17:25:48 UTC - Grant Wu: (Which is why I am interested in any efforts to get an official client) ---- 2018-09-25 17:25:53 UTC - Sijie Guo: for each language runtime in pulsar functions it is using the client written in it. so currently we have go, but we haven’t support go functions yet. it is in the roadmap though. I think for supporting nodejs function, a nodejs client would be preferrable that ws, since if the nodejs wrapper is wrapping over existing c++ client, it will get all the features in order to support nodejs functions. ---- 2018-09-25 17:26:21 UTC - Grant Wu: Right, that makes sense. ---- 2018-09-25 17:54:25 UTC - Grant Wu: anyways @Guillem I’m interested if you’re working on a client ---- 2018-09-25 17:57:07 UTC - Matteo Merli: @Grant Wu BTW we do have Go client library, although it’s not yet integrated in Pulsar Functions ---- 2018-09-25 17:57:11 UTC - Grant Wu: yes I know ---- 2018-09-25 17:57:21 UTC - Grant Wu: oops I misstated that in one place ---- 2018-09-25 17:57:56 UTC - Matteo Merli: NP ---- 2018-09-25 17:58:10 UTC - Grant Wu: I should know, because I’m writing a tiny helper library for it for internal usage >_> ok_hand : Matteo Merli ---- 2018-09-25 17:58:52 UTC - Guillem: sorry, got caught on a phone call! ---- 2018-09-25 18:00:51 UTC - Guillem: so the idea would be to develop a nodejs client library ---- 2018-09-25 18:01:06 UTC - Guillem: and then use that for the runtime to do the functions bit? ---- 2018-09-25 18:01:13 UTC - Guillem: would that be the preferred approach? ---- 2018-09-25 18:01:28 UTC - Grant Wu: I think so ---- 2018-09-25 18:01:32 UTC - Matteo Merli: ...wrapper on top of C++ lib, that would be the easier option ---- 2018-09-25 18:01:46 UTC - Guillem: yeah, i think the pyhton is a wrapper on the c++ lib, right? ---- 2018-09-25 18:02:18 UTC - Guillem: if that is the case, maybe i could have a look at the current python library to see how it has been handled ---- 2018-09-25 18:02:25 UTC - Guillem: and try and create something similar ---- 2018-09-25 18:03:32 UTC - Matteo Merli: Python uses BoostPython which simplifies a lot of things in creating a wrapper. For other languages it depends on that language C extension API ---- 2018-09-25 18:03:54 UTC - Guillem: ok ---- 2018-09-25 18:11:37 UTC - Grant Wu: I feel like looking at examples of how other C++ based libraries are wrapped for Node.js is going to be more useful ---- 2018-09-25 18:11:46 UTC - Grant Wu: <https://nodejs.org/api/addons.html> appears to be the right thing to look at ---- 2018-09-25 18:13:26 UTC - Sijie Guo: there was one example in the issue I attached above ---- 2018-09-25 18:13:40 UTC - Sijie Guo: but I think that was based on a very old version of v8 ---- 2018-09-25 18:13:53 UTC - Sijie Guo: not sure how v8 has evolved :slightly_smiling_face: ---- 2018-09-25 18:14:27 UTC - Grant Wu: <https://nodejs.org/api/addons.html#addons_n_api> appears to describe a more stable interface ---- 2018-09-25 18:17:11 UTC - Guillem: what about context. is that something that comes from the client library also for pulsar functions? ---- 2018-09-25 18:17:30 UTC - Matteo Merli: It’s nice that at least it support C++ directly.. for Go we had to build C++ -> C -> Go :confused: ---- 2018-09-25 18:17:58 UTC - Guillem: was just checking that site @Grant Wu just before you posted it :slightly_smiling_face: ---- 2018-09-25 18:22:17 UTC - Guillem: btw, there's a pulsar python client ---- 2018-09-25 18:22:22 UTC - Guillem: as well as a go ---- 2018-09-25 18:22:24 UTC - Guillem: but on github ---- 2018-09-25 18:22:36 UTC - Guillem: i can see a pulsar-client-go ---- 2018-09-25 18:22:41 UTC - Guillem: but not a pulsar-client-python ---- 2018-09-25 18:23:05 UTC - Matteo Merli: <https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python> +1 : Guillem ---- 2018-09-25 18:23:09 UTC - Grant Wu: I think `pulsar-client-go` is third party ---- 2018-09-25 18:23:32 UTC - Matteo Merli: nope :slightly_smiling_face: <https://github.com/apache/pulsar/tree/master/pulsar-client-go> ---- 2018-09-25 18:23:48 UTC - Grant Wu: huh. ---- 2018-09-25 18:23:53 UTC - Grant Wu: I thought <https://github.com/Comcast/pulsar-client-go> was being referred to ---- 2018-09-25 18:24:13 UTC - Matteo Merli: That’s a client that doesn’t depend on the C++ library ---- 2018-09-25 18:24:30 UTC - Grant Wu: right, ok ---- 2018-09-25 18:34:23 UTC - Guillem: what about context. is that something that comes from the client library also for pulsar functions? ---- 2018-09-25 18:36:36 UTC - Grant Wu: Well there’s <https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/python/pulsar/functions/context.py> ---- 2018-09-25 18:37:11 UTC - Grant Wu: and <https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/python/contextimpl.py> ---- 2018-09-25 18:38:03 UTC - Grant Wu: I think the code for the context stuff is implemented in the non-C++ language ---- 2018-09-25 18:40:15 UTC - Grant Wu: and <https://github.com/apache/pulsar/blob/master/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java> vs <https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java> ---- 2018-09-25 18:40:38 UTC - Grant Wu: the code organization isn’t completely obvious to me, as someone not very familiar with the directory conventions of all the different languages being used ---- 2018-09-25 18:40:44 UTC - Grant Wu: so it does seem to require some digging ---- 2018-09-25 18:45:17 UTC - Guillem: agree ---- 2018-09-25 18:45:25 UTC - Guillem: i'll try and spend some time on it ---- 2018-09-25 18:45:31 UTC - Guillem: see if i can figure something out ---- 2018-09-25 19:01:34 UTC - Sanjeev Kulkarni: as @Sijie Guo said, there are two components here. 1) Client. Currently we have java and cpp, and python and go wrapper clients. Reason we went with wrapper style for go and python is that its very hard to develop and maintain a fully featured client in multiple languages, thus it is best to just make two full fledged language clients(Java and C++) and have wrapper based clients for other languages. ---- 2018-09-25 19:03:15 UTC - Guillem: sounds reasonable, yes ---- 2018-09-25 19:03:17 UTC - Sanjeev Kulkarni: 2) Pulsar Functions Runtime. Currently we have Python and Java runtimes written in Python and Java respectively. Internally they use the python and java client respectively. The go runtime is in the roadmap. ---- 2018-09-25 19:03:42 UTC - Sanjeev Kulkarni: for javascript, we first need a javascript client. ---- 2018-09-25 19:03:53 UTC - Sanjeev Kulkarni: and then we can add a javascript runtime for functions ---- 2018-09-25 19:05:24 UTC - Guillem: are the runtimes the code in this folder? ---- 2018-09-25 19:05:25 UTC - Guillem: <https://github.com/apache/pulsar/tree/master/pulsar-functions> ---- 2018-09-25 19:06:50 UTC - Sanjeev Kulkarni: They are called language instances in the code and can be found at <https://github.com/apache/pulsar/tree/master/pulsar-functions/instance/src/main> ---- 2018-09-25 19:07:17 UTC - Grant Wu: oh, I was wondering why they were called “instance” ---- 2018-09-25 19:07:46 UTC - Grant Wu: I have a question about the official go client ---- 2018-09-25 19:07:52 UTC - Sanjeev Kulkarni: because the word ‘runtime’ was already taken. See <https://github.com/apache/pulsar/tree/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime> ---- 2018-09-25 19:08:02 UTC - Guillem: and this is then built into a docker image @Sanjeev Kulkarni? ---- 2018-09-25 19:08:15 UTC - Sanjeev Kulkarni: yes @Guillem ---- 2018-09-25 19:08:47 UTC - Grant Wu: I had ``` import ( "<http://github.com/apache/incubator-pulsar/pulsar-client-go/pulsar|github.com/apache/incubator-pulsar/pulsar-client-go/pulsar>" ) ``` and I seem to have gotten candidate-1 ``` grant.wu@MBP00245s-MacBook-Pro:~/petuum/petuum-pulsar-utils/go/src/example-client ⇒ dep init Using ^2.1.1-incubating-candidate-1 as constraint for direct dep <http://github.com/apache/incubator-pulsar|github.com/apache/incubator-pulsar> Locking in v2.1.1-incubating-candidate-1 (68faf85) for direct dep <http://github.com/apache/incubator-pulsar|github.com/apache/incubator-pulsar> ``` ---- 2018-09-25 19:08:50 UTC - Grant Wu: does anyone know why ---- 2018-09-25 19:10:09 UTC - Sanjeev Kulkarni: not from the top of my head, but since iirc there was only one rc for 2.1.1, it should be ok ---- 2018-09-25 19:10:32 UTC - Sijie Guo: @Grant Wu are you using any dep management in go? ---- 2018-09-25 19:10:34 UTC - Guillem: is the base image /files for that docker or a script to build that docker image also on github? if so, could you point me in the right direction? thanks ---- 2018-09-25 19:11:00 UTC - Grant Wu: @Sijie Guo uh… what do you mean. ---- 2018-09-25 19:11:08 UTC - Sijie Guo: I think 2.1.1-incubating-cadidnate-1 and 2.1.1-incubating are referring same git commit ---- 2018-09-25 19:11:12 UTC - Grant Wu: right, ok ---- 2018-09-25 19:11:28 UTC - Sijie Guo: are you using ‘dep’? ---- 2018-09-25 19:11:33 UTC - Grant Wu: yes ---- 2018-09-25 19:11:41 UTC - Grant Wu: I mean the command was `dep init` ---- 2018-09-25 19:12:51 UTC - Guillem: @Sanjeev Kulkarni do you know where the images / files for the docker images or the script to build the docker images for the runtimes are on github? ---- 2018-09-25 19:14:14 UTC - Sanjeev Kulkarni: <https://github.com/apache/pulsar/tree/master/docker> ---- 2018-09-25 19:14:31 UTC - Sijie Guo: @Grant Wu that’s probably dep resolving the same gitsha to candidate-1, because they are the same. you probably can update your Gopkg.toml to pin the version to 2.1.1-incubating ---- 2018-09-25 19:14:33 UTC - Sanjeev Kulkarni: has a bunch of scripts for different kinds of docker images for pulsar ---- 2018-09-25 19:14:51 UTC - Grant Wu: makes sense. ---- 2018-09-25 19:15:46 UTC - Guillem: those include the ones for pulsar functions? I did take a look at that folder, but coulnd't find anything that did seem to be related to pulsar functions ---- 2018-09-25 19:18:34 UTC - Sanjeev Kulkarni: Currently we don’t make any official pulsar docker images for just running functions, the current ones bundle quite a bit apart from functions stuff, but they can be used to run functions as well ---- 2018-09-25 19:19:21 UTC - Guillem: ok. so when you're using pulsar functions, the docker image includes quite a bit more than the bare minimum to run functions? ---- 2018-09-25 19:19:38 UTC - Guillem: so which one are functions using of those docker images? ---- 2018-09-25 19:19:56 UTC - Guillem: the one in pulsar folder directly? ---- 2018-09-25 19:21:41 UTC - Grant Wu: I’m not sure why Docker comes into the equation ---- 2018-09-25 19:21:56 UTC - Grant Wu: Based on <https://pulsar.apache.org/docs/en/functions-overview/#cluster-run-mode>, the code is directly uploaded to the cluster. ---- 2018-09-25 19:22:11 UTC - Grant Wu: The cluster may be running in Docker, but no Docker involved for Pulsar functions specifically ---- 2018-09-25 19:22:47 UTC - Guillem: ok, that's right ---- 2018-09-25 19:22:52 UTC - Guillem: i did missread that ---- 2018-09-25 19:23:04 UTC - Guillem: i was thinking on the 'traditional' serverless approach ---- 2018-09-25 19:23:19 UTC - Guillem: and i thought pulsar was creating containers for the functions ---- 2018-09-25 19:24:38 UTC - Eugene Gordon: @Eugene Gordon has joined the channel ---- 2018-09-25 19:24:40 UTC - Guillem: so it's more the 'breaker' that i was referring to ---- 2018-09-25 19:24:45 UTC - Grant Wu: Is the following something I should be worried about? ---- 2018-09-25 19:25:12 UTC - Grant Wu: ---- 2018-09-25 19:25:28 UTC - Grant Wu: (wtf, I can’t create a code snippet in a thread?) ---- 2018-09-25 19:28:16 UTC - Grant Wu: breaker? ---- 2018-09-25 19:29:40 UTC - Sanjeev Kulkarni: Pulsar Functions currently can be run either locally(via just invoking the function), submitted to a pulsar cluster that will manage them. In these scenarios, the docker doesnt come into play ---- 2018-09-25 19:30:27 UTC - Sanjeev Kulkarni: However there is work going on to run pulsar functions inside kubernetes. A slimmer docker image containing only relevant stuff might be more efficient in that context ---- 2018-09-25 21:10:47 UTC - Ali Ahmed: @Grant Wu did the artifacts get generated ? ---- 2018-09-25 21:16:31 UTC - Grant Wu: How do I check? ---- 2018-09-25 21:20:36 UTC - Ali Ahmed: check for *.whl file being generated for your platform ---- 2018-09-25 21:20:54 UTC - Grant Wu: Do you know in which directory those would be in? ---- 2018-09-25 21:27:38 UTC - Grant Wu: <https://blog.streaml.io/pulsar-effectively-once/> appears to be down ---- 2018-09-25 21:28:35 UTC - Grant Wu: oh, need to delete the blog ---- 2018-09-25 21:33:27 UTC - Ali Ahmed: don’t remember just run find at the repo base ---- 2018-09-25 21:49:50 UTC - Jon Bock: That link should be <https://streaml.io/blog/pulsar-effectively-once/> ---- 2018-09-25 22:02:21 UTC - Grant Wu: Okay, sorry. I found an old link on Google that hadn’t had its link updated ---- 2018-09-25 22:02:44 UTC - Grant Wu: I do not have the Pulsar source code checked out ---- 2018-09-25 22:03:46 UTC - Ali Ahmed: so how are you compiling ? ---- 2018-09-25 22:04:01 UTC - Grant Wu: ``` brew install <https://raw.githubusercontent.com/apache/incubator-pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb> ``` ---- 2018-09-25 22:06:26 UTC - Ali Ahmed: can you post the whole log ---- 2018-09-25 22:08:00 UTC - Ali Ahmed: this is what I got ``` brew install <https://raw.githubusercontent.com/apache/incubator-pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb> Updating Homebrew... ==> Auto-updated Homebrew! Updated 4 taps (homebrew/cask-versions, homebrew/core, homebrew/cask, homebrew/cask-drivers). ==> Updated Formulae awscli ✔ digdag erlang file-roller gedit grakn ice imagemagick imagemagick@6 jenkins libdazzle tgui unp64 vala vte3 yelp-tools ==> Deleted Formulae submarine ######################################################################## 100.0% ==> Installing dependencies for libpulsar: pkg-config ==> Installing libpulsar dependency: pkg-config ==> Downloading <https://homebrew.bintray.com/bottles/pkg-config-0.29.2.high_sierra.bottle.tar.gz> ######################################################################## 100.0% ==> Pouring pkg-config-0.29.2.high_sierra.bottle.tar.gz 🍺 /usr/local/Cellar/pkg-config/0.29.2: 11 files, 627.2KB ==> Installing libpulsar ==> Downloading <https://s3-us-west-2.amazonaws.com/pulsar-preview/apache-pulsar-2.1.0-incubating-SNAPSHOT-src.tar.gz> ######################################################################## 100.0% ==> cmake . -DBUILD_TESTS=OFF -DLINK_STATIC=ON ==> make pulsarShared pulsarStatic 🍺 /usr/local/Cellar/libpulsar/2.1.0-incubating-SNAPSHOT: 46 files, 13.1MB, built in 1 minute 46 seconds ``` ---- 2018-09-25 22:10:58 UTC - Grant Wu: <https://apache-pulsar.slack.com/files/UBHR9CH5E/FD04CB5Q8/-.pl> ---- 2018-09-25 22:11:02 UTC - Grant Wu: is this not the full log? ---- 2018-09-25 22:14:20 UTC - Ali Ahmed: try ``` brew install <https://raw.githubusercontent.com/apache/incubator-pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb> --with-python3 ``` ---- 2018-09-25 22:28:00 UTC - Grant Wu: Thanks, that worked! ---- 2018-09-25 22:28:07 UTC - Grant Wu: Should we document this on the website? ---- 2018-09-25 22:31:31 UTC - Ali Ahmed: best create an issue it will be taken up ---- 2018-09-25 22:34:58 UTC - Grant Wu: wait… it’s already documented there, lol. I just didn’t quite read far enough. ---- 2018-09-25 22:35:16 UTC - Grant Wu: sorry, multitasking + stressed at work with fires ---- 2018-09-26 00:13:47 UTC - Grant Wu: Are pulsar subscriptions garbage collected at all? ---- 2018-09-26 00:22:01 UTC - Guillem: yeah, the 'breaker' in pulsar is what executes the pulsar functions' code (as well as doing a number of other things i guess) ---- 2018-09-26 02:09:00 UTC - Grant Wu: Also, we have hit the 10k lines cap ---- 2018-09-26 02:09:04 UTC - Grant Wu: Should we consider moving off of Slack? ---- 2018-09-26 06:51:48 UTC - Nathanial Murphy: @Nathanial Murphy has joined the channel ---- 2018-09-26 06:53:50 UTC - Nathanial Murphy: Hey guys! Is there any way to have the tiered storage write to a bare-metal server that's not owned by a cloud provider? ---- 2018-09-26 07:57:48 UTC - Sijie Guo: @Nathanial Murphy all the tiered storage related operations are abstracted in an interface called offloader. you can implement your own offloader to write to your own storage. there is actually an inprogress effort on providing hdfs offloader. ---- 2018-09-26 08:02:29 UTC - Ali Ahmed: @Nathanial Murphy also you can make do with s3 api compatible deployment like minio ---- 2018-09-26 08:03:04 UTC - Ali Ahmed: @Grant Wu suggestions are welcome ----
