2019-06-03 10:43:46 UTC - pradeep: Does pulsar provide handler/listener to
notify the application if any new topic added/deleted from the pulsar cluster?
----
2019-06-03 16:10:54 UTC - Sudhakar Namasi: @Sudhakar Namasi has joined the
channel
----
2019-06-03 18:41:07 UTC - Ben Hood: What's the idiomatic format to store
message ids so that a `Reader` can store a recovery point in a DB? One option
is to store the string representation of the 4 component fields of a message
id. Another is to take the HEX encoding of the serialized protobuf binary, or
even store the raw binary serialization. It feels like the string
representation would be most stable option to put a descending index on. Am I
overthinking the binary format - does it preserve order?
----
2019-06-03 18:42:12 UTC - Matteo Merli: `MessageId.serialize()` will give you a
`byte[]` that you can store. That will be guaranteed to work across releases
and such
----
2019-06-03 18:43:10 UTC - Matteo Merli: Sorry, `MessageId.toByteArray()` and
`fromByteArray()`
----
2019-06-03 18:45:22 UTC - Ben Hood: Ah, I hadn't come accross that API, until
now I've only been looking at the C API
----
2019-06-03 18:46:49 UTC - Ben Hood: And the resultant byte array orders
naturally across releases?
----
2019-06-03 18:50:15 UTC - Matteo Merli: Yes, that’s part of the “offical” API
so we guarantee `toByteArray()` and `fromByteArray()` will always work
----
2019-06-03 18:53:50 UTC - Ben Hood: OK, cool, I’ve just checked out the
implementation of the Java API `toByteArray()` which delegates to protobuf
serialization, which is what I’ve seen in the C++ API as well
----
2019-06-03 19:16:55 UTC - Grant Wu:
<https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Topics.html#resetCursor-java.lang.String-java.lang.String-long->
does this give you the message that’s “closest” in terms of time delta, or
just the latest message with timestamp before the provided parameter?
----
2019-06-03 19:20:27 UTC - Grant Wu: I’m seeing the latter behavior in some of
my tests
----
2019-06-03 19:32:05 UTC - Addison Higham: open question: Is anyone doing
backups of bookkeeper? I *think* I should just be able to take a snapshot of
the underlying block device (EBS volume) and have it be consistent, at least
for the journal, but am curious if there is something I am not thinking of
there..
----
2019-06-03 19:32:09 UTC - Ben Hood: Judging from the message id specification
```message MessageIdData {
required uint64 ledgerId = 1;
required uint64 entryId = 2;
optional int32 partition = 3 [default = -1];
optional int32 batch_index = 4 [default = -1];
}```
Therefore I’m guessing that global ordering is determined by the numeric
ordering of the concatenated `uint64` and `int32` fields.
Just by looking at it I wasn’t sure, because of the signed 32-bit `partition`
and `batch_index` fields. Plus I wasn’t sure whether protobuf would inject any
frame markers anywhere (I guess it uses the compiled stubs to avoid having to
do this?).
So many thanks @Matteo Merli for confirming the ordering guarantee
:slightly_smiling_face:
----
2019-06-03 19:34:43 UTC - Sijie Guo: the combination of `ledgerId`, `entryId`
and `batch_index` determines the order of messages within a topic (partition -
specified by partition field)
----
2019-06-03 20:21:35 UTC - Ben Hood: Thanks for the clarification @Sijie Guo
----
2019-06-03 20:53:46 UTC - Grant Wu: @Matteo Merli ping
----
2019-06-03 20:54:05 UTC - Grant Wu: I feel like I’ve asked this question before
but wasn’t able to find it in search
----
2019-06-03 21:15:21 UTC - Matteo Merli: I think it should be the 1st message
with publishTime >= timestamp
----
2019-06-03 21:15:35 UTC - Grant Wu: I’m getting the other behavior :confused:
----
2019-06-03 21:15:49 UTC - Matteo Merli: Which are you getting?
----
2019-06-03 21:15:58 UTC - Grant Wu: The last message with publishTime <=
timestamp
----
2019-06-03 21:18:50 UTC - Matteo Merli: I see. that actually make sense
----
2019-06-03 21:19:17 UTC - Matteo Merli: In the sense that it’s better to replay
more messages than skipping them
----
2019-06-03 22:17:23 UTC - Devin G. Bost: Does anyone have suggestions on how I
could mock or emulate Pulsar for integration tests that use the Pulsar Admin
API? I have tests in my deployment process that test whether or not Pulsar
Admin operations (e.g. create, update, delete) do what I'm expecting when given
certain configurations, but these tests currently require a locally running
Pulsar environment, which makes them brittle.
----
2019-06-03 22:19:47 UTC - Ali Ahmed: you can use an embedded pulsar process
----
2019-06-03 22:20:16 UTC - Devin G. Bost: That sounds great. Do you have an
example anywhere?
----
2019-06-03 22:21:25 UTC - Ali Ahmed:
<https://github.com/streamlio/pulsar-embedded-tutorial>
----
2019-06-03 22:22:12 UTC - Devin G. Bost: Thanks!
----
2019-06-03 22:30:15 UTC - Jerry Peng: @Devin G. Bost also take a look at these
integration tests we have
<https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java>
----
2019-06-03 22:31:33 UTC - Devin G. Bost: Hi @Jerry Peng, thanks for the
example! :slightly_smiling_face:
----
2019-06-03 23:26:03 UTC - Devin G. Bost: @Jerry Peng Is there a POM dependency
that I can add to include the PortManager class from `import
org.apache.bookkeeper.test.PortManager;`?
I added the dependencies that I found, but that one is still missing.
`import org.apache.pulsar.functions.utils.FunctionCommon;`
and
`import static org.apache.pulsar.broker.auth.`
are also missing.
----
2019-06-03 23:49:14 UTC - Jerry Peng: > import
org.apache.bookkeeper.test.PortManager;
```
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>managed-ledger-original</artifactId>
<version>${project.version}</version>
</dependency>
```
----
2019-06-03 23:49:51 UTC - Jerry Peng: > import
org.apache.pulsar.functions.utils.FunctionCommon
```
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-functions-utils</artifactId>
<version>${project.version}</version>
</dependency>
```
----
2019-06-03 23:52:35 UTC - Jerry Peng: > import static
org.apache.pulsar.broker.auth
----
2019-06-03 23:53:03 UTC - Jerry Peng: ```
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-broker</artifactId>
<version>${project.version}</version>
<classifier>tests</classifier>
</dependency>
```
----
2019-06-03 23:53:32 UTC - Jerry Peng: @Devin G. Bost ^^^
----
2019-06-03 23:55:47 UTC - Devin G. Bost: Thanks!
----
2019-06-03 23:59:53 UTC - Matteo Merli: The main problem is the consistency of
metadata (in ZK) with respect to the data in bookies.
If the metadata is gone, restoring a backup for a bookie won’t be much helpful.
----
2019-06-04 00:03:25 UTC - Devin G. Bost: @Jerry Peng That fixed my issue with:
> import static org.apache.pulsar.broker.auth
but I'm still getting `java: package org.apache.bookkeeper.test does not exist`
and:
```java: cannot find symbol
symbol: class FunctionCommon
location: package org.apache.pulsar.functions.utils```
----
2019-06-04 00:03:37 UTC - Devin G. Bost: Any ideas?
----
2019-06-04 00:04:08 UTC - Jerry Peng: did you add the previous two dependencies
I listed above
----
2019-06-04 00:05:16 UTC - Devin G. Bost: Yes.
----
2019-06-04 00:06:05 UTC - Jerry Peng: oh sorry should be this:
```
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>managed-ledger-original</artifactId>
<version>${project.version}</version>
<classifier>tests</classifier>
</dependency>
```
----
2019-06-04 00:06:19 UTC - Devin G. Bost: Ah, I figured something was missing.
I'll try that. Thanks!
----
2019-06-04 00:06:51 UTC - Devin G. Bost: That solves the PortMapper issue.
----
2019-06-04 00:07:28 UTC - Jerry Peng: oh what version of pulsar are you trying
to pull in?
----
2019-06-04 00:07:53 UTC - Devin G. Bost: 2.3.2. It was previously 2.3.1, and I
changed it while debugging.
----
2019-06-04 00:08:03 UTC - Devin G. Bost: FunctionCommon is still missing.
----
2019-06-04 00:09:44 UTC - Jerry Peng: you don’t really need that
----
2019-06-04 00:09:49 UTC - Jerry Peng: you can just remove that import
----
2019-06-04 00:09:57 UTC - Devin G. Bost: Oh, you're right haha.
----
2019-06-04 00:10:00 UTC - Devin G. Bost: Thanks!
----
2019-06-04 00:10:10 UTC - Jerry Peng: np
----
2019-06-04 00:33:58 UTC - Thor Sigurjonsson: Question: Can function state be
deleted (I noticed get/set semantics in the API)?
----
2019-06-04 00:37:03 UTC - Jerry Peng: @Thor Sigurjonsson currently there is not
a delete API that allows you to delete a key, however I just looked at the code
and it should be trivial to add
----
2019-06-04 00:37:26 UTC - Jerry Peng: @Thor Sigurjonsson if you would like this
feature can you create an issue?
----
2019-06-04 00:37:59 UTC - Thor Sigurjonsson: @Jerry Peng Sure, I can do that.
----
2019-06-04 00:38:46 UTC - Thor Sigurjonsson: Any reason we could not (or should
not) have gigabytes of storage for function state in BookKeeper?
----
2019-06-04 00:40:51 UTC - Jerry Peng: nope just make sure your function have
enough memory allocated to it
----
2019-06-04 00:41:29 UTC - Thor Sigurjonsson: Does the full state end up in
memory?
----
2019-06-04 00:41:38 UTC - Jerry Peng: no just what you read
----
2019-06-04 00:43:36 UTC - Thor Sigurjonsson: So I can do `put`/`get` in a
function with some `key` and a small `value` but have gobs and gobs of
gigamabæts because I have millions of `keys`? (And I just need the small read
values in memory).
----
2019-06-04 00:45:22 UTC - Jerry Peng: yes in theory that should all work
----
2019-06-04 00:45:46 UTC - Thor Sigurjonsson: Perfect! We may end up testing
that in practice then :slightly_smiling_face:
----
2019-06-04 00:46:18 UTC - Thor Sigurjonsson: So, when I delete a function, will
the storage for the state get deleted?
----
2019-06-04 00:46:26 UTC - Jerry Peng: no
----
2019-06-04 00:47:13 UTC - Jerry Peng: maybe @Sijie Guo can comment further
regarding the scalability of table service in BK since he is more familiar with
that than I am
+1 : Thor Sigurjonsson
----
2019-06-04 01:55:33 UTC - Ryan Samo: Hey guys, is there a way to get the number
of messages consumed by a subscription since it’s inception?
----
