2020-01-16 09:44:30 UTC - testinglab89: @testinglab89 has joined the channel ---- 2020-01-16 10:39:33 UTC - Fernando: It seems it’s not possible to create topics via prestos’s `CREATE TABLE` directive. Is this the case or am I missing something? I keep getting ```Table 'pulsar.public/default.orders' already exists``` even though it clearly doesn’t. ---- 2020-01-16 11:08:25 UTC - testinglab89: ---- 2020-01-16 13:39:34 UTC - Naby: I didn’t do anything this time. I didn’t need to define schema at all, neither at source nor server. It worked. Maybe adding some instruction into each built-in connector webpage on how to setup and create connection and manage the connection helps. For example, there’s not enough information here: <https://pulsar.apache.org/docs/en/io-influxdb/> On the other hand, I found some info on this page: <https://pulsar.apache.org/docs/en/next/io-influxdb-sink/#configuration> But, I had to figure out how to write the configuration file, by reading so many pages, some even contradicted each other. Finally, I tuned my config file after a couple of errors and trials. Maybe I did it all wrong. Perhaps after adding enough information on influxdb connector page, I’ll be able to find out what I did wrong. ---- 2020-01-16 14:34:27 UTC - Julius.b: I got a fileending issue... There is always an \n added in the function code... ```Invalid topicname <persistent://public/default/cleaned_data>\n", "msSinceEpoch": "1579185122370"``` anyone experienced such issues... I already tried fileendings for windows and linux, doesnt change anything ---- 2020-01-16 14:46:58 UTC - Ryan Samo: Hey guys, is there a good example on the web for Spark producing messages into Pulsar? I can’t seem to locate one in the GitHub or docs, only the streaming receiver ---- 2020-01-16 16:02:37 UTC - Naby: How can I specify a regular expression on input topics when I create a function? Can I use `--topic-pattern` for this purpose? If so, how does it work? ---- 2020-01-16 16:03:52 UTC - Matteo Merli: In the `pulsar` catalog, tables are mapped to Pulsar topics. All operations through Presto are read-only with respect to Pulsar. ---- 2020-01-16 16:05:08 UTC - Matteo Merli: A single Pulsar client instance can be used to create multiple producers and consumers. The underlying TCP connections and threads will be pooled across all of them. ---- 2020-01-16 16:06:02 UTC - Fernando: ok so just for my understanding, the role or presto right now is for aggregation/analysis but not to be used for ETL like flink ---- 2020-01-16 16:06:20 UTC - Matteo Merli: Yes, you can pass `--topic-pattern 'public/default/my-.*'` ---- 2020-01-16 16:06:48 UTC - Fernando: I was thinking of using it to create my data pipeline instead of pulsar functions but since I can’t write topics from presto then it’s not possible ---- 2020-01-16 16:07:11 UTC - Matteo Merli: Presto itself it's designed for interactive querying ---- 2020-01-16 16:08:19 UTC - Matteo Merli: In Pulsar context, you can use it to query historical data together with event just published on the topic.
Though it's *not* streaming SQL ---- 2020-01-16 16:09:12 UTC - Fernando: ok I understand, for streaming I would need to hook pulsar to some other system ---- 2020-01-16 16:28:45 UTC - Naby: :+1: ---- 2020-01-16 17:01:41 UTC - Addison Higham: hrm... really strange issue today with ZK 3.5.5. wonder if anyone has seen anything like it before: • a few days earlier, we rotated out all our global ZK hosts (in k8s), everything appeared to be working, had quorum and clients were able to send messages in Pulsar • two days ago, we started have an issue where we were getting exceptions (as well as a NPE in an exceptional handling branch) due to being unable to communicate with ZK. Looking at the ZK logs, it was actively rejecting connections with an error like `Refusing session request for client as it has seen zxid 0x16000000aa our last zxid is 0x11 client must try another server` • We were also unable to connect with just the zookeeper shell • We restarted all the nodes in the ZK cluster and 4 of 5 came back with the expected state and began serving fine • One of the 5 nodes had an empty state, but it said it was still a follower. There were snapshot and log files that looked like a reasonable size, but it still somehow appeared to be out of sync with the quorum. We deleted the data directory and restarted and it synced and was fine The two main open questions are, what could happen to cause us to get in the state where connections were rejected due to (apparently) going back in time? And secondary (and perhaps more concerning) how could we have a member who appeared to be healthy and serving connections be out of sync with quorum with the corrective action being to delete the data dir ---- 2020-01-16 17:19:28 UTC - Pedro Cardoso: Sounds like a zookeeper only question. Suggest you ask this to the zookeeper community. ---- 2020-01-16 17:20:52 UTC - Addison Higham: going to x-post there, but a fair amount of them are also here :slightly_smiling_face: ---- 2020-01-16 17:23:02 UTC - Matteo Merli: did any disk got full on any node? ---- 2020-01-16 17:24:59 UTC - Addison Higham: no, this is just the global config store... so it is like 5k of JSON ---- 2020-01-16 19:02:15 UTC - Addison Higham: :thinking_face: it looks like the 2.4.2 links to the pulsar distribution (from the downloads, page, this link <https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=pulsar/pulsar-2.4.2/apache-pulsar-2.4.2-bin.tar.gz>) as well as to the libpulsar dev packages are failing (from <https://pulsar.apache.org/docs/en/client-libraries-cpp/> the <https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=pulsar/pulsar-2.4.2/DEB/apache-pulsar-client.deb> link). It looks like the apache mirrors only have 2.5.0 ---- 2020-01-16 19:07:27 UTC - Michael Jasper: @Michael Jasper has joined the channel ---- 2020-01-16 19:13:00 UTC - Guilherme Perinazzo: The `.deb` packages links are also broken ---- 2020-01-16 19:14:21 UTC - Ming: This works <https://archive.apache.org/dist/pulsar> with most DEB lib I need ---- 2020-01-16 19:14:41 UTC - Guilherme Perinazzo: Oh nice, thanks! ---- 2020-01-16 19:15:36 UTC - Matteo Merli: The mirror link should not considered "stable" links. Only the latest release is going to be present there. Older releases will be always available from the Apache archive. ---- 2020-01-16 19:18:32 UTC - Guilherme Perinazzo: Back when I was looking for it, I didn't find the apache archive, but the 2.4.1 mirror was still working until today. It may be a good idea to link the archive in the official documentation! ---- 2020-01-16 19:19:34 UTC - Matteo Merli: It's really not super-clear, though if you look at the "Older releases" section on <https://pulsar.apache.org/en/download/>, all the links are to the archive. ---- 2020-01-16 19:22:52 UTC - Guilherme Perinazzo: I was looking specifically for the .deb files for the C++ client, those aren't listed on that page. Maybe having a link on <https://pulsar.apache.org/docs/en/client-libraries-cpp/> to the archives would help a lot. The cgo client doesn't seem to like newer versions ---- 2020-01-16 19:23:33 UTC - Matteo Merli: Ouch, got it. ---- 2020-01-16 19:23:54 UTC - Matteo Merli: If you're starting out in Go, I'd suggest to go directly to <https://github.com/apache/pulsar-client-go> ---- 2020-01-16 19:24:49 UTC - Guilherme Perinazzo: I started the project before that lib became usable! But will be looking to switch. Is that one ready for production usage? ---- 2020-01-16 19:40:17 UTC - Matteo Merli: Is getting close to it :slightly_smiling_face: ---- 2020-01-16 19:40:29 UTC - Matteo Merli: we're about to get to an official release ---- 2020-01-16 21:00:42 UTC - Bo Han: I am trying to build pulsar from source but failed because access denied by <http://maven.restlet.org/org/restlet/jee/org.restlet/2.3.0/org.restlet-2.3.0.pom|http://maven.restlet.org/org/restlet/jee/org.restlet/2.3.0/org.restlet-2.3.0.pom>. Is there a solution? ---- 2020-01-16 21:21:10 UTC - Matteo Merli: Is that dependency coming from Pulsar? We shouldn't be using that. Which module are you compiling? ---- 2020-01-16 21:24:58 UTC - Matt Mitchell: Curious about Pulsar security/access and recommended approach, where Pulsar is running inside of Kubernetes, but external client access (for specific topics) is desired. Is it possible to prevent access to all but a few topics? ---- 2020-01-16 21:26:58 UTC - Matteo Merli: You can do that with the regular AuthZ mechanism. Use one authentication method (eg: JWT tokens) and grant permission to the principal associated with a token to publish on a particular topic (or namespace) ---- 2020-01-16 21:28:29 UTC - Matt Mitchell: Perfect ---- 2020-01-16 21:28:53 UTC - Matt Mitchell: I’ll read the docs and try to get something working ---- 2020-01-16 21:28:54 UTC - Matt Mitchell: thanks! ---- 2020-01-16 21:38:37 UTC - Matteo Merli: Take a look at <https://pulsar.apache.org/docs/en/security-token-admin/> ---- 2020-01-16 21:44:28 UTC - Naby: I am using topics pattern for my function. When new topics being generated, it takes some time to be recognized. Is there a way to accelerate auto discovery of pattern? ---- 2020-01-16 21:44:59 UTC - Bo Han: I am trying to compile the code by just typing `mvn packages.` ---- 2020-01-16 21:46:19 UTC - Bo Han: What should I do If I want to have the standalone pulsar binary? ---- 2020-01-16 21:47:57 UTC - Roman Popenov: Anyone else hitting: ```Templating init failed Datasource named ${DS_default} was not found``` ---- 2020-01-16 21:48:06 UTC - Roman Popenov: When loading Topics dashboard in grafana? ---- 2020-01-16 22:29:03 UTC - Roman Popenov: Does anyone have steps how to make Grafana work? It appears there are no data displayed ---- 2020-01-16 22:29:05 UTC - Kirill Merkushev: Hello, is there a native/quick way of copying one topic to another with filtering of some keys without much coding? ---- 2020-01-16 22:29:43 UTC - Matteo Merli: That sounds a good fit for Pulsar functions +1 : Kirill Merkushev, Roman Popenov, xiaolong.ran ---- 2020-01-16 22:48:57 UTC - Shane Head: @Shane Head has joined the channel ---- 2020-01-17 02:32:52 UTC - Sijie Guo: I think you need to configure the datasource. ---- 2020-01-17 02:33:27 UTC - Sijie Guo: You can try this as well: <https://github.com/streamnative/apache-pulsar-grafana-dashboard> +1 : xiaolong.ran ---- 2020-01-17 02:34:20 UTC - Sijie Guo: Solr changed its maven repo: <https://github.com/apache/lucene-solr/pull/1144> ---- 2020-01-17 02:34:48 UTC - Sijie Guo: <https://github.com/apache/pulsar/pull/6068> is the fix ---- 2020-01-17 02:34:57 UTC - Sijie Guo: waiting for CI to merge this change ---- 2020-01-17 02:40:09 UTC - Sijie Guo: FYI. For people who is building Pulsar from master and encountered following issue: ```access denied by <http://maven.restlet.org/org/restlet/jee/org.restlet/2.3.0/org.restlet-2.3.0.pom>.``` --- It is because spring changed its maven repo url. (<https://github.com/apache/lucene-solr/pull/1144> )which causes dependency not found. A fix is out for fixing <https://github.com/apache/pulsar/pull/6068> but waiting for migrating CI (or passing Jenkins CI, whichever comes first) to merge the PR. If you want to mitigate the build issue, add the following repo in the `repositories` section at the root pom.xml file. ``` <repository> <id>spring-plugins-release</id> <url><https://repo.spring.io/plugins-release/></url> </repository>``` +1 : Fernando ---- 2020-01-17 05:38:22 UTC - DashWang: @DashWang has joined the channel ---- 2020-01-17 09:08:33 UTC - Alessandro Luccaroni: @Alessandro Luccaroni has joined the channel ----
