2019-09-10 10:33:08 UTC - vikash: @Vladimir Shchur is nuget also support
.net framework
----
2019-09-10 10:33:09 UTC - vikash:
<https://www.nuget.org/packages/Pulsar.Client/0.1.4>
----
2019-09-10 10:33:35 UTC - vikash: i am getting below Error
----
2019-09-10 10:33:41 UTC - vikash:
----
2019-09-10 10:34:13 UTC - vikash: but above code working fine in .net
core
----
2019-09-10 10:34:42 UTC - vikash: just query because our c# application
is in .net framework
----
2019-09-10 11:03:25 UTC - Vladimir Shchur: Hi! Thanks for report, will check if
it possible to fix and update you
----
2019-09-10 13:20:36 UTC - Vladimir Shchur: @vikash Two binding redirects are
needed to make it work:
`<dependentAssembly>`
` <assemblyIdentity name="System.Memory"
publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />`
` <bindingRedirect oldVersion="0.0.0.0-4.0.1.1" newVersion="4.0.1.0"
/>`
` </dependentAssembly>`
` <dependentAssembly>`
` <assemblyIdentity name="System.Threading.Tasks.Extensions"
publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />`
` <bindingRedirect oldVersion="0.0.0.0-4.2.0.1" newVersion="4.2.0.0"
/>`
` </dependentAssembly>`
----
2019-09-10 13:28:28 UTC - vikash: @Vladimir Shchur Thanks ,yes it works
for me after changes above config
+1 : Vladimir Shchur
----
2019-09-10 14:31:36 UTC - Sijie Guo: Not just this one. I think the function
worker needs some additional configurations as well.
----
2019-09-10 14:31:44 UTC - Sijie Guo: I think the documentation about this part
is missing.
----
2019-09-10 14:31:54 UTC - Sijie Guo: We will try to enhance this part
----
2019-09-10 14:48:43 UTC - Kirill Merkushev: Is there any way for java api to
know that the ledgerId is closed (or about to be closed) so that with manual
seek we don’t get an exception for the last message in the closed ledger and
for the current one we don’t go one position earlier than we actually have? I’m
talking about this part
<https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2729>
and the `seekAsync(MessageId)` method
----
2019-09-10 16:01:26 UTC - Kirill Merkushev: found the
`org.apache.pulsar.client.impl.ConsumerImpl#getLastMessageIdAsync` but that’s a
package-private. Is that safe to assume that ledgerId within the messageId it
is the latest/active ledger?
----
2019-09-10 18:48:35 UTC - Tarek Shaar: It is recommended to co-locate ZooKeeper
with the Pulsar brokers on the same VMs (possibly with Book Keeper too)? We
process roughly 1M messages per business day.
----
2019-09-10 19:51:33 UTC - Sijie Guo: I think that method will be exposed in 2.5.
----
2019-09-10 19:51:51 UTC - Sijie Guo: I wouldn’t suggest making that assumption
----
2019-09-10 19:52:07 UTC - Sijie Guo: LedgerId is kind of internal to Pulsar so
far
----
2019-09-10 19:56:38 UTC - Sijie Guo: What was the exception you are hitting?
Can you show a code snippet?
----
2019-09-10 19:58:59 UTC - Sijie Guo: That’s probably fine. Although make sure
you don’t share the disks for zk and bk. Also keep in mind with collocation,
you might loose flexibility of scaling individual layers
----
2019-09-10 20:01:07 UTC - Tarek Shaar: I am trying to get the last sequence ID
from my producer but I am always getting -1. I am using 2.4.1 on both client
and broker sides
----
2019-09-10 20:04:34 UTC - Tarek Shaar: On a separate note, is the sequence as
far as writing to disk is book keeper node stores the messages in the journal
disk then acks back to the producer? Or does another disk write happen in ZK
Journal as well before acking back to the producer?
----
2019-09-10 20:22:33 UTC - Sijie Guo: > is the sequence as far as writing to
disk is book keeper node stores the messages in the journal disk then acks back
to the producer?
yes
> Or does another disk write happen in ZK Journal as well before acking back
to the producer?
no. pulsar didn’t talk to zk that much. onlly about the metadata part.
----
2019-09-10 20:24:53 UTC - Sijie Guo: can you explain more with a code example?
----
2019-09-10 20:26:13 UTC - Tarek Shaar: Thanks but I got a little confused when
you said keep ZK disks and BK disks separate. So at which point does ZK use
disk? I see there is a log file in the data directory that's relatively large.
Do you have any documentation on when Pulsar uses ZK?
----
2019-09-10 20:27:38 UTC - Tarek Shaar: Here is the producer code..
Producer<byte[]> producer =
client.newProducer().enableBatching(true).producerName("my-dev").batchingMaxMessages(2000).blockIfQueueFull(true).compressionType(CompressionType.LZ4).topic(topic).create();
----
2019-09-10 20:27:57 UTC - Tarek Shaar: Then I set the sequence ID on the
producer like this
----
2019-09-10 20:27:59 UTC - Tarek Shaar:
producer.newMessage().sequenceId(100).key("key1").value("test".getBytes()).sendAsync();
----
2019-09-10 20:28:11 UTC - Tarek Shaar: Then I shut down the producer and start
it again
----
2019-09-10 20:28:29 UTC - Tarek Shaar: and get the Sequence ID like this
----
2019-09-10 20:28:30 UTC - Tarek Shaar: long last = producer.getLastSequenceId();
----
2019-09-10 20:28:38 UTC - Tarek Shaar: But is is always -1
----
2019-09-10 20:30:39 UTC - Ming: @Ming has joined the channel
----
2019-09-10 20:39:39 UTC - Sandeep Kotagiri: @Sandeep Kotagiri has joined the
channel
----
2019-09-10 20:57:28 UTC - Sandeep Kotagiri: I was working with functions and
state in a Kubernetes environment. Encountred the same issue as above. With
some older posts from @Sijie Guo I was able to get this working. In Kubernetes
environment, I had to set 'extraServerComponents' parameter in bookkeeper
configuration file and the 'stateStorageServiceUrl' parameter in the
functions_worker.yml configuration file. Kubernetes HELM chart has lower JVM
settings that would also need to be bumped up to get this working properly.
----
2019-09-10 20:59:03 UTC - Sijie Guo: Did you enable message deduplication at
the broker or the namespace? If not, Pulsar won’t record the information.
----
2019-09-10 21:01:14 UTC - Sandeep Kotagiri: @Sijie Guo I have a question on the
very topic. I have three bookies running in my Kubernetes cluster. At the
moment, I am configuring 'stateStorageServiceUrl' to be
<bk://pulsar-bookkeeper-0:4181/>. Here 'pulsar-bookkeeper-0' is the first
bookie in my statefulset. How do we configure to utilize the reminder of the
bookies (pulsar-bookkeeper-1, pulsar-bookkeeper-2) for storing state?
----
2019-09-10 21:16:02 UTC - Tarek Shaar: Ok I'll try and let you know if it
fails. Thanks
----
2019-09-10 22:19:41 UTC - BigSam: @BigSam has joined the channel
----
2019-09-10 23:09:39 UTC - Addison Higham: oh hrm... so for pulsar functions on
a geo-replication cluster, I probably want to use a different function
namespace for each region? it looks like my function is getting sent randomly
around my cluster
----
2019-09-10 23:09:47 UTC - Addison Higham: and then unable to find the jar I
uploaded
----
2019-09-10 23:09:49 UTC - Addison Higham: is that correct?
----
2019-09-10 23:09:55 UTC - Matteo Merli: Correct
----
2019-09-10 23:10:29 UTC - Addison Higham: makes sense, but that was surprising
:stuck_out_tongue: good thing worker IDs have the hostname in it otherwise I
would have been really confused
----
2019-09-10 23:50:33 UTC - Ali Ahmed:
<https://twitter.com/Lanayx/status/1171508406072401926>
----
2019-09-11 07:15:25 UTC - Yuvaraj Loganathan: @Tarek Shaar ZK persists every
write to disk before acknowledging it. Also as a good practice i would
recommend to launch separate VM for zookeeper so that you will have better
& superior fault tolerance level.
----
2019-09-11 08:35:49 UTC - divyasree: @Sijie Guo I am trying out topic
compaction in pulsar. Is there any configuration need to be enabled for topic
compaction or we can make the producer to produce data with specific key and
read from the consumer with "readCompacted(true)?
----
2019-09-11 08:36:53 UTC - divyasree: If changes from the client side is enough
for the compaction to work, when will the compaction get triggered. What is the
default threshold for it to trigger?
----
2019-09-11 08:57:26 UTC - Mahesh: @Sijie Guo After configuring as suggested by
Sandeep, I get this error now
01:50:35.109 [public/default/word-count-0] WARN
org.apache.pulsar.functions.instance.JavaInstanceRunnable - Encountered
internal server on opening table 'word-count', re-attempt in 100 milliseconds :
Encountered internal server exception on stream 1026 : code =
INTERNAL_SERVER_ERROR
----
2019-09-11 08:57:52 UTC - Mahesh: Can you please share the full list of config
entries to be added to use table service in functions
----
2019-09-11 09:10:39 UTC - Vladimir Ontikov: Hi,
How many cores should be on a machine running Pulsar?
----
