2019-01-23 14:50:45 UTC - Pratik Narode: Hello, I am using pulsar go client. I
am trying to create a pulsar subscriber using regex for topic name. Ex. I have
created a topic `testtopic` and am trying to subscribe to it using regex `t.*`.
The producer and consumer are created successfully but I don't receive any msg
on the consumer, that is produced on the `testtopic`. could anyone please help
me understand what am I doing wrong. My producer and consumer code are as
follow:
----
2019-01-23 14:51:17 UTC - Pratik Narode:
----
2019-01-23 14:51:51 UTC - Pratik Narode:
----
2019-01-23 15:01:05 UTC - jia zhai: If not support, would you please open an
issue for it?
----
2019-01-23 15:03:40 UTC - jia zhai: @Pratik Narode I am not familiar with go
client. But in cpp/java, we provide a topic pattern like this:
```
String patternString = "<persistent://my-property/my-ns/pattern-topic.*>";
```
----
2019-01-23 15:04:12 UTC - jia zhai: ```
std::string pattern =
"<persistent://public/default/patternMultiTopicsConsumer.*>";
```
----
2019-01-23 15:05:33 UTC - Pratik Narode: @jia zhai: passing above topic
actually gets parsed as same what you showed i.e
`<persistent://public/default/t.*>`
----
2019-01-23 15:06:03 UTC - Pratik Narode: Also I tried passing it as
`<persistent://my-property/my-ns/t.*>`
----
2019-01-23 15:06:13 UTC - Pratik Narode: but no luck still same result
----
2019-01-23 15:07:23 UTC - jia zhai: oh
----
2019-01-23 15:07:54 UTC - jia zhai: @Pratik Narode checked the code, seems it
is still not supported in go client.
----
2019-01-23 15:08:25 UTC - jia zhai: I remember that some one is trying to do
this.
----
2019-01-23 15:08:37 UTC - jia zhai: It should be soon.
----
2019-01-23 15:10:54 UTC - Pratik Narode: ok Thanks
----
2019-01-23 15:11:12 UTC - Pratik Narode: @jia zhai: Can you please tell me in
what all client this feature is available ?
----
2019-01-23 15:13:21 UTC - jia zhai: cpp and java
----
2019-01-23 15:13:29 UTC - jia zhai: @Pratik Narode
----
2019-01-23 15:13:52 UTC - Pratik Narode: @jia zhai: Thanks for you quick help
----
2019-01-23 15:14:35 UTC - jia zhai: welcome
----
2019-01-23 15:26:27 UTC - Brian: anyone have an idea of when 2.3 will be
released?
----
2019-01-23 15:31:10 UTC - jia zhai: @Brian From what I know, it is depends on
bookkeeper 4.9 release, which is in voting status. After bk 4.9, it should be
soon.
@Sijie Guo and @Matteo Merli may have more information.
+1 : Brian
----
2019-01-23 16:14:15 UTC - Matteo Merli: @jia zhai @Pratik Narode The feature is
already supported in 2.2 release.
Take a look at
<https://godoc.org/github.com/apache/pulsar/pulsar-client-go/pulsar#ConsumerOptions>
There is `TopicsPattern` field that needs to be specified in alternative to
`Topic` (or `Topics`)
----
2019-01-23 16:15:52 UTC - Matteo Merli: The reason is a different option is
that topic names are themselves allowed to contain special characters
+1 : jia zhai
----
2019-01-23 16:55:00 UTC - Matteo Merli: @jia zhai @bossbaby The reason
cumulative ack it’s not supported across partitions is that a particular
`MessageId` is relative to only a partition. Therefore each message would have
to carry information of the message ids from other partitions (as received by a
particular consumer).
For that reason it’s not currently supported (in any of the client libraries).
It’s not impossible to support, it would just require to keep track of the
overall ordering of messages received across partitions.
Regarding seek(), it’s actually supported in C++:
<http://pulsar.apache.org/api/cpp/classpulsar_1_1_consumer.html#a949eda4ed6de3ecc2e4f793cce1dc06c>
----
2019-01-23 16:57:07 UTC - Matteo Merli: @Brian we’re shooting for ASAP.
Realistically it would be a 1-2 week timeframe. We’re waiting for bk-4.9 to be
out and then we’ll do few adjustments in Pulsar (as enabled by new features in
bk-4.9) and it will be ready.
----
2019-01-23 17:01:31 UTC - Brian: awesome, looking forward to it
----
2019-01-23 17:02:11 UTC - Grant Wu: @Jerry Peng
----
2019-01-23 18:56:22 UTC - Grant Wu: I’m seeing errors of this sort:
----
2019-01-23 18:56:32 UTC - Grant Wu:
----
2019-01-23 18:56:49 UTC - Grant Wu: And this is causing all sorts of fun like
this:
----
2019-01-23 18:56:53 UTC - Grant Wu: ```
18:48:25.442 [prometheus-stats-42-1] WARN
org.apache.pulsar.functions.worker.FunctionsStatsGenerator - Failed to collect
metrics for function instance public/grantwu/adminNotifications:0
```
----
2019-01-23 18:57:08 UTC - Grant Wu: Is this related to the `className` vs
`classname` change
----
2019-01-23 19:51:58 UTC - Grant Wu: Okay, no, it doesn’t seem related to that
----
2019-01-23 19:52:10 UTC - Prasanna kumar Krishnamurthy: @Prasanna kumar
Krishnamurthy has joined the channel
----
2019-01-23 21:59:29 UTC - Sanjeev Kulkarni: @Grant Wu context.publish is
essentially inside the function logic. So if the function is being run in
atmost_once mode, you may or may not see the effects of context.publish on
every incming message. In atleast_once mode, you will see >=1 effects of
publish
----
2019-01-23 22:00:36 UTC - Grant Wu: Is there any difference between
`AT_LEAST_ONCE` and `EFFECTIVELY_ONCE` for the special case of a function which
uses `context.publish` exclusively and does not publish to an output topic?
----
2019-01-23 22:09:13 UTC - Sanjeev Kulkarni: context.publish is a ‘side-effect’
of the function for the case of effectively once function.
----
2019-01-23 22:09:26 UTC - Sanjeev Kulkarni: One way to have effectigely once
and publish inside function would be to use sequence ids
----
2019-01-23 22:11:41 UTC - Sanjeev Kulkarni: We might have to enhance the
publish interface to allow that capability
----
2019-01-23 22:12:10 UTC - Grant Wu: So the answer is “no” :stuck_out_tongue: ?
----
2019-01-23 22:12:16 UTC - Grant Wu: Because it’s a side-effect, essentially
----
2019-01-23 22:12:25 UTC - Grant Wu: And the model assumes pure functions?
----
2019-01-23 22:12:33 UTC - Grant Wu: It is alright if the answer is “no”
----
2019-01-23 22:12:54 UTC - Grant Wu: It would be pretty magical if your system
could somehow rewind side effects of Pulsar exceptions
----
2019-01-23 22:13:35 UTC - Sanjeev Kulkarni: So currently its a no
----
2019-01-23 22:13:40 UTC - Grant Wu: Okay, got it!
----
2019-01-23 22:14:24 UTC - Sanjeev Kulkarni: But its a capability that we could
easily add
----
2019-01-23 22:14:33 UTC - Sanjeev Kulkarni: maybe create a github issue?
----
2019-01-23 22:14:51 UTC - Grant Wu: Okay, I can do that.
----
2019-01-23 22:15:55 UTC - Grant Wu: Personally, getting this documented is
higher on my list of wants.
----
2019-01-23 22:16:49 UTC - Grant Wu: Appreciate that amount of time all of you
spend answering questions for free here for sure :slightly_smiling_face:, but
it’s not very scalable
----
2019-01-23 22:16:53 UTC - Grant Wu: Maybe I can write the docs
----
2019-01-23 23:12:46 UTC - Grant Wu: Hey, @Sanjeev Kulkarni, I saw you touched
the Python import logic for Pulsar functions recently
----
2019-01-23 23:13:53 UTC - Grant Wu: I keep on running into this line
<https://github.com/apache/pulsar/blob/63a0491ec86a4123062eb2835d6f385bf79e2552/pulsar-functions/instance/src/main/python/python_instance.py#L171>
----
2019-01-23 23:15:57 UTC - Grant Wu: ~My current working assumption is that
something is wrong with what I’m putting into the `classname` parameter~
----
2019-01-23 23:17:53 UTC - Grant Wu: ~Can you explain what exactly is supposed
to go there?~
----
2019-01-24 00:50:57 UTC - Grant Wu: Okay, I figured it out.
----
2019-01-24 00:51:15 UTC - Grant Wu: It turns out that I was using a Python 3
only library (pathlib) in my Pulsar function, causing an exception to be thrown
when attempting to import them
----
2019-01-24 00:53:43 UTC - Grant Wu: Where are the logging messages, in, for
example,
<https://github.com/apache/pulsar/blob/fa212bc8bb0cbd81729b498b6041cde794be9684/pulsar-functions/instance/src/main/python/util.py>
actually supposed to show up?
----
2019-01-24 00:53:59 UTC - Grant Wu: They don’t appear to show up in the stdout
logs in pulsar standalone, nor in the Pulsar broker logs when we deploy Pulsar
in k8s
----
2019-01-24 00:55:17 UTC - Grant Wu: If I had gotten access to those logs
earlier - I eventually got access to them by inserting a
`logging.basicConfig(level=logging.DEBUG)` statement at the top of `log.py` - I
would have saved many many hours of debugging
----
2019-01-24 01:00:47 UTC - Sanjeev Kulkarni: depending on where the function is
running. If running in pulsar cluster, they will appear on one of the function
worker machines. If running in k8, you can use kubectl logs to fetch them
----
2019-01-24 01:01:39 UTC - Grant Wu: Fairly sure they are not showing up
----
2019-01-24 01:03:07 UTC - Grant Wu: If you want I can try adding the
logging.basicConfig statement into `log.py` and then running my standalone
cluster again
----
2019-01-24 01:03:19 UTC - jia zhai: Thanks @Matteo Merli :+1:
----
2019-01-24 01:03:24 UTC - Grant Wu: To see if that makes a difference
----
2019-01-24 01:12:37 UTC - Grant Wu: That doesn’t really seem to make a
difference
----
2019-01-24 01:19:09 UTC - Grant Wu: I don’t have much to go on, but I can say I
have never seen the logging output in `util.py` ever show up in the stdout I
see in `bin/pulsar standalone`
----
2019-01-24 01:27:45 UTC - Grant Wu: I can file a bug tomorrow.
----
2019-01-24 02:02:56 UTC - bossbaby: I mean here is seek earliest
----
2019-01-24 02:18:26 UTC - Matteo Merli: You can pass `MessageId::earliest()`
<http://pulsar.apache.org/api/cpp/classpulsar_1_1_message_id.html#a6c5231ab632b49863c4a227e9cff353d>
----
2019-01-24 03:02:19 UTC - bossbaby: but i was testing and its not work, maybe
partition not support earliest
----
2019-01-24 03:06:39 UTC - bossbaby: i have a question that "message send by
producer will go to broker and broker will write it on bookie" or " broker will
say message that you should write in bookie"?
----
2019-01-24 05:37:26 UTC - Matteo Merli: seek is not supported on partitioned
topics
----
2019-01-24 05:37:58 UTC - Matteo Merli: (Because it could be an arbitrary
message id)
+1 : bossbaby
----
2019-01-24 06:29:32 UTC - Simon: @Simon has joined the channel
----
2019-01-24 07:01:13 UTC - Sijie Guo: sorry I am not very clear about your
question here?
----
2019-01-24 07:14:50 UTC - bossbaby: Does broker transport messages from
producer to bookie?
----
2019-01-24 07:15:11 UTC - Sijie Guo: yes
----
2019-01-24 07:22:10 UTC - bossbaby:
----
2019-01-24 07:23:03 UTC - bossbaby: why performance in pulsar is slower than I
expected with 4 broker and 4 bookie
----
