2019-09-11 09:56:33 UTC - wlkid: Hi guys, just want to know what's the order of
the messages sent to same topic from multiple producers concurrently, from the
point view of consumer?
----
2019-09-11 09:59:05 UTC - wlkid: is the order determined by the timestamp
specified by the producers, or by the sequence the broker received the messages?
----
2019-09-11 11:47:41 UTC - Kirill Merkushev: so basically we are serializing
messageId as long
```
public static MessageId fromOffset(long offset) {
return new MessageIdImpl(offset >>> 28, offset &
0x0F_FF_FF_FFL, -1);
}
public static long toOffset(MessageId messageId) {
MessageIdImpl msgId = (MessageIdImpl) messageId;
// Combine ledger id and entry id to form offset
// Use less than 32 bits to represent entry id since it will get
// rolled over way before overflowing the max int range
return (msgId.getLedgerId() << 28) | msgId.getEntryId();
}
```
and keep that separately from the Pulsar, and after some time seeking that
gives us exception about wrong seek
We discussed that in the
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1567085799031200>
----
2019-09-11 11:48:36 UTC - Kirill Merkushev: but we didn’t get what’re the
possible workarounds
----
2019-09-11 11:52:51 UTC - Kirill Merkushev: @Sijie Guo here is the rationale
about ledger id question as well
----
2019-09-11 12:57:33 UTC - el akroudi abdessamad: I have a question
----
2019-09-11 12:58:22 UTC - el akroudi abdessamad: Any work to integrate pulsar
with cloudeea stack, it will push pulsar in the top level and gain popularity
----
2019-09-11 12:58:27 UTC - el akroudi abdessamad: ??
----
2019-09-11 13:15:05 UTC - Nick Marchessault: Can someone clarify what
ReceivedFailed and AcksFailed mean in the getStats method
(<https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Consumer.html#getStats-->)?
Is there a one to one correlation between AcksFailed and unacknowledged
messages?
----
2019-09-11 13:32:49 UTC - Tarek Shaar: Thanks Yuva, my understanding was that
ZK is an in memory store and it periodically stores the data in the transaction
log file. So are you saying that each message that the producer sends to the
broker gets stored in Book Keeper ledger log file and in the ZK disk before the
ack is sent back? Can you please share the flow with me? This is critical for
our company to understand
----
2019-09-11 13:34:27 UTC - Yuvaraj Loganathan: Zk stores only metadata. But
metadata is crtical. Actual flow of data does go through boookeeper and broker.
----
2019-09-11 13:35:21 UTC - Yuvaraj Loganathan:
<https://pulsar.apache.org/docs/en/concepts-architecture-overview/>
----
2019-09-11 13:38:39 UTC - Tarek Shaar: Thanks Yuva. I have been through this
document and it talks about how Book Keeper drives the persistence not ZK. I
guess am trying to understand what information is stored in ZK for pulsar and
at which stage etc...
----
2019-09-11 14:35:21 UTC - Matteo Merli: The order is determined by how they are
received by broker
----
2019-09-11 14:49:28 UTC - Sandeep Kotagiri: @Mahesh two primary settings I have
used are as what I wrote above. In bookkeeper.conf file add
extraServerComponents=
org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent and in
functions_worker.yml add stateStorageServiceUrl: "<bk://localhost:4181>"
----
2019-09-11 14:51:08 UTC - Sandeep Kotagiri: For errors such as yours, you might
find some different error in the bookkeeper log files. That might give a clue
as of what is happening.
----
2019-09-11 15:40:12 UTC - Matteo Merli: Not directly. The unacked messages,
from broker’s perspective, are all the messages that were pushed to one
consumer and not acked yet.
----
2019-09-11 18:11:24 UTC - Subramaniyam: Hi All, I am trying to install
pulsar-client in my npm (nodejs)
But i am getting the below error..
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp WARN Tried to download(404):
<https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.1.0-rc.0-node-v64-win32-x64.tar.gz>
node-pre-gyp WARN Pre-built binaries not found for [email protected] and
[email protected] (node-v64 ABI, unknown) (falling back to source compile with
node-gyp)
gyp WARN install got an error, rolling back install
gyp ERR! configure error
gyp ERR! stack Error: tunneling socket could not be established, statusCode=403
Looks like the above library is not there in the particular URL
Its roll backing total installation.
----
2019-09-11 19:50:44 UTC - Areeb Ahmed: @Areeb Ahmed has joined the channel
----
2019-09-11 20:25:16 UTC - Tarek Shaar: I am running internal stats on my pulsar
topic. Here is snippet of what I get
----
2019-09-11 20:25:17 UTC - Tarek Shaar: "lastConfirmedEntry" : "11:7",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 7,
"entries" : 5,
"size" : 290,
"offloaded" : false
}, {
"ledgerId" : 11,
"entries" : 0,
"size" : 0,
"offloaded" : false
} ],
----
2019-09-11 20:26:34 UTC - Tarek Shaar: Question is Ledger 11 shows no entries
at all but last confirmed entry is 11:7. In other words I expect ledger 11 to
have some entries but there are none. Am i missing something?
----
2019-09-12 03:54:52 UTC - Ali Ahmed: @Ali Ahmed set the channel topic: Pulsar
graduated as a Top-Level project :tada: - Pulsar release 2.4.1 -
<http://pulsar.apache.org/release-notes/#2.4.1>
----
2019-09-12 05:13:19 UTC - Vinuja: @Vinuja has joined the channel
----
2019-09-12 05:47:21 UTC - divyasree: Can you guide me to proceed further?
----
2019-09-12 06:09:44 UTC - Shishir Pandey: @Shishir Pandey has joined the channel
----
2019-09-12 06:45:13 UTC - Vinuja: I have deployed Pulsar on Azure Kubernetes
service. I am testing Failover scenarios for Pulsar. As a scenario, I deleted
the zookeeper and its pvc, Now I created a new zookeeper and a pvc and
initialized cluster metadata. Now, All the bookie, broker are all up. But the
Cluster is not initialized properly. I am not able to see the cluster itself.
Can you please help me in resolving this issue.
----
2019-09-12 06:47:40 UTC - Jerry Peng: @Tarek Shaar that is expected. The stats
will show up when the ledger is closed. So always expect the latest ledger to
have empty
----
2019-09-12 06:48:51 UTC - wlkid: does it imply that there's no guarantee of
FIFO? or, for a simpler case, consuming mysql binlog events needs strictly
FIFO, will pulsar suitable for it?
----
2019-09-12 07:04:22 UTC - Walter: i have deployed pulsar in kubernetes with 1
broker pod(2cpu , 8Gb), 3 bookkeeper pod (each 2cpu, 8Gb), 3 zookeeper(1cpu,
2Gb)
i have given more load with pulsar-perf tool but it's reached memory limit of
broker means (7.9Gb). Then I waited for 2days to cool down broker pods memory.
so why broker pod memory is not reducing?
----
2019-09-12 07:11:42 UTC - Zhenhao Li: looking at
<https://www.slideshare.net/Odin_cool/fsharp-goodness-for-everyday-work>
----
2019-09-12 07:16:25 UTC - Zhenhao Li: I wonder what made the F# client is 1.5x
as fast as Java
----
2019-09-12 08:09:40 UTC - Dennis Sehalic:
<https://jack-vanlightly.com/blog/2019/9/4/a-look-at-multi-topic-subscriptions-with-apache-pulsar>
----
2019-09-12 08:50:16 UTC - Maksim Bazhenov: @Maksim Bazhenov has joined the
channel
----
2019-09-12 08:53:32 UTC - Shishir Pandey: Hi all,
I am trying to solve the following problem - I want to place the messages into
the queue sorted by the message key the following manner { message_key:
{string: name, integer: id, message_created time}, message_value: { <xml>
} } this is kind of to ensure that messages for the same name and id are next
to each other, I want to do this because I need to process messages for a
particular name and id that arrive within a specific interval (this is
reasonably large - up to 7 days) together and I do not want to traverse the
messages one at a time. I've been reading the documentation for Pulsar and it
is not entirely clear to me if this can be achieved? If I understand it
correctly the custom partitioning scheme would only put the message into a
specific partition, and the documentation further says "The ordering of
messages is related to MessageRoutingMode and Message Key. Usually, user would
want an ordering of Per-key-partition guarantee" so it is not entirely clear to
me if I can achieve what I'm looking to do.
----
