2018-10-15 12:56:16 UTC - Rodrigo Malacarne: @Ali Ahmed, some information about
the 'public/default' namespace...
./bin/pulsar-admin namespaces get-persistence public/default
retrieves
{
"bookkeeperEnsemble" : 1,
"bookkeeperWriteQuorum" : 1,
"bookkeeperAckQuorum" : 1,
"managedLedgerMaxMarkDeleteRate" : 0.0
}
----
2018-10-15 12:56:56 UTC - Rodrigo Malacarne: ./bin/pulsar-admin namespaces
get-backlog-quotas public/default
retrieves
{ }
----
2018-10-15 12:57:25 UTC - Rodrigo Malacarne: ./bin/pulsar-admin namespaces
get-retention public/default
retrieves
{
"retentionTimeInMinutes" : 0,
"retentionSizeInMB" : 0
}
----
2018-10-15 12:58:55 UTC - Rodrigo Malacarne: ./bin/pulsar-admin topics
get-partitioned-topic-metadata <persistent://public/default/test>
retrieves
{
"partitions" : 0
}
----
2018-10-15 12:59:33 UTC - Rodrigo Malacarne: I believe there should be at least
1 partition in the 'test' topic ...
----
2018-10-15 13:09:58 UTC - Rodrigo Malacarne: I've just run "./bin/pulsar-admin
topics create-partitioned-topic <persistent://public/default/test> --partitions
3" and now I get for "./bin/pulsar-admin topics get-partitioned-topic-metadata
<persistent://public/default/test>"
{
"partitions" : 3
}
----
2018-10-15 13:11:29 UTC - Rodrigo Malacarne: But this still seems not to be the
problem, as I'm still getting a "Not enough non-faulty bookies available" when
trying to produce a message to this topic :disappointed:
----
2018-10-15 13:30:03 UTC - Aswani Karteek Yadavilli: @Aswani Karteek Yadavilli
has joined the channel
----
2018-10-15 14:17:57 UTC - Sijie Guo: @Rodrigo Malacarne sorry for late
response. if ‘bookkeeper shell listbookies’ returns ‘no bookies’, it means that
your bookies are not registered in the cluster. if your bookies are started
correctly, did you check bookies’s logs?
----
2018-10-15 14:18:46 UTC - Rodrigo Malacarne: @Sijie Guo I've sent the bookie
logs here yesterday.
----
2018-10-15 14:19:21 UTC - Rodrigo Malacarne:
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1539552825000100>
----
2018-10-15 14:19:44 UTC - Rodrigo Malacarne:
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1539552851000100>
----
2018-10-15 14:20:01 UTC - Rodrigo Malacarne:
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1539552291000100>
----
2018-10-15 14:29:50 UTC - Sijie Guo: @Rodrigo Malacarne it seems that bookies
haven’t completed startup.
----
2018-10-15 14:30:11 UTC - Sijie Guo: did you see the process is running?
----
2018-10-15 14:31:24 UTC - Rodrigo Malacarne: @Sijie Guo checking ...
----
2018-10-15 14:33:01 UTC - Rodrigo Malacarne: When I run "ps aux | grep -i
bookkeeper"
----
2018-10-15 14:33:12 UTC - Rodrigo Malacarne: I get
"delta 3185 0.2 8.9 7165548 359988 ? Sl Oct14 2:40
/usr/lib/jvm/java-8-oracle/bin/java -cp
/home/delta/pulsar/conf:::/home/delta/pulsar/lib/*:
-Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760
-Djava.net.preferIPv4Stack=true -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g
-XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled
-XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis
-XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50
-XX:+DisableExplicitGC -XX:-ResizePLAB -Dio.netty.leakDetectionLevel=disabled
-Dio.netty.recycler.maxCapacity.default=1000
-Dio.netty.recycler.linkCapacity=1024 -Dpulsar.log.appender=RollingFile
-Dpulsar.log.dir=/home/delta/pulsar/logs -Dpulsar.log.level=info
-Dpulsar.routing.appender.default=Console
-Dpulsar.functions.process.container.log.dir=/home/delta/pulsar/logs
-Dpulsar.functions.java.instance.jar=/home/delta/pulsar/instances/java-instance.jar
-Dpulsar.functions.python.instance.file=/home/delta/pulsar/instances/python-instance/python_instance_main.py
-Dpulsar.log.file=pulsar-bookie-pulsar_1.log
org.apache.bookkeeper.proto.BookieServer --conf
/home/delta/pulsar/conf/bookkeeper.conf
----
2018-10-15 14:34:42 UTC - Rodrigo Malacarne: And when I check the port using
"nc -vz pulsar1 3181" I get "Connection to pulsar1 3181 port [tcp/*] succeeded!"
----
2018-10-15 14:34:56 UTC - Rodrigo Malacarne: So at least the bookkeeper port is
active and answering ...
----
2018-10-15 14:36:56 UTC - Sijie Guo: okay can you check
/data/bookkeeper/ledgers/current/VERSION?
----
2018-10-15 14:37:21 UTC - Sijie Guo: and let me know what is the content there
----
2018-10-15 14:37:48 UTC - Rodrigo Malacarne: @Sijie Guo, a "cat VERSION"
retrieves
4
bookieHost: "pulsar1:3181"
journalDir: "/data/bookkeeper/journal"
ledgerDirs: "1\t/data/bookkeeper/ledgers"
instanceId: "806ec278-a198-4d58-a78d-8a27104a9c2a"
----
2018-10-15 14:39:43 UTC - Sijie Guo: okay that seems to be fine
----
2018-10-15 14:42:40 UTC - Sijie Guo: can you try to take a jstack on the bookie
process?
----
2018-10-15 14:46:21 UTC - Rodrigo Malacarne: @Sijie Guo can you please advise
me on how can I do that?
----
2018-10-15 14:47:47 UTC - Rodrigo Malacarne: "jstack -l JAVA_PID >
jstack.out" ?
----
2018-10-15 14:48:11 UTC - Sijie Guo: yes
----
2018-10-15 14:48:49 UTC - Rodrigo Malacarne:
----
2018-10-15 15:01:20 UTC - Sijie Guo: hmm looks normal.
----
2018-10-15 15:01:30 UTC - Sijie Guo: can you run “bin/pulsar zookeeper-shell
-server <zkservers> ls /ledgers”
----
2018-10-15 16:06:24 UTC - Martin Svensson: hey guys, bit of sanity check
here…I’m offloading to s3 and that seems to work. what I do find is that the
bookie disk space is ever growing. Shouldn’t the offloading process manage that
or do I need to do something manually?
eyes : Aaron Langford
----
2018-10-15 16:52:47 UTC - Haijia Zhou: @Haijia Zhou has joined the channel
----
2018-10-15 16:57:08 UTC - Rodrigo Malacarne: @Sijie Guo, sorry for the delay ...
"./bin/pulsar zookeeper-shell -server pulsar1:2181,pulsar2:2181,pulsar3:2181 ls
/ledgers"
retrieves the following
----
2018-10-15 16:57:11 UTC - Rodrigo Malacarne: 16:56:21.945 [main] INFO
org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.name=Linux
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=amd64
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=4.15.0-36-generic
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.name=delta
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/home/delta
16:56:21.945 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/home/delta/pulsar
16:56:21.947 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
connection, connectString=pulsar1:2181,pulsar2:2181,pulsar3:2181
sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@407a7f2a
16:56:22.014 [main-SendThread(pulsar1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
pulsar1/192.168.15.243:2181. Will not attempt to authenticate using SASL
(unknown error)
16:56:22.038 [main-SendThread(pulsar1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
pulsar1/192.168.15.243:2181, initiating session
16:56:22.059 [main-SendThread(pulsar1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
pulsar1/192.168.15.243:2181, sessionid = 0x1000413b7a90000, negotiated timeout
= 30000
[idgen, LAYOUT, available, underreplication, INSTANCEID]
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
----
2018-10-15 17:35:51 UTC - Jeremiah Cutting: @Jeremiah Cutting has joined the
channel
----
2018-10-15 17:42:09 UTC - Jagjeet Singh: @Jagjeet Singh has joined the channel
----
2018-10-15 20:57:31 UTC - Rodrigo Malacarne: Hello, suppose I want to send
messages to 5 different topics as a producer. In the (c++) send( ) function I
don't have the topic as a function argument... in this case, should I
instantiate 5 different producers, and use each one to send messages to a
different topic?
----
2018-10-15 20:59:06 UTC - Sanjeev Kulkarni: @Rodrigo Malacarne yes, you need to
create 5 different producers
----
2018-10-15 20:59:28 UTC - Rodrigo Malacarne: @Sanjeev Kulkarni, thank you very
much!
----
2018-10-15 21:01:38 UTC - Ian Raphael Francisco Bezerra: @Ian Raphael Francisco
Bezerra has joined the channel
----
2018-10-15 22:45:05 UTC - Srikala Munamala: @Srikala Munamala has joined the
channel
----
2018-10-15 23:44:29 UTC - Sijie Guo: Can you do “bin/pulsar zookeeper-shell
-server <zkservers> ls /ledgers/available”
----
2018-10-16 00:25:58 UTC - Sijie Guo: # Delay between a ledger being
successfully offloaded to long term storage
# and the ledger being deleted from bookkeeper (default is 4 hours)
managedLedgerOffloadDeletionLagMs=14400000
----
2018-10-16 00:26:17 UTC - Sijie Guo: there is a setting
`managedLedgerOffloadDeletionLagMs` controlling when the ledger is actually
deleted
----
2018-10-16 00:27:49 UTC - Sijie Guo: you can reduce that so the ledger can be
removed after it is offloaded
----
2018-10-16 00:56:04 UTC - Rodrigo Malacarne: 00:55:48.189 [main] INFO
org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
00:55:48.189 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
00:55:48.189 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
00:55:48.189 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.name=Linux
00:55:48.189 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=amd64
00:55:48.189 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=4.15.0-36-generic
00:55:48.189 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.name=delta
00:55:48.190 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/home/delta
00:55:48.190 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/home/delta/pulsar
00:55:48.192 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client
connection, connectString=pulsar1:2181,pulsar2:2181,pulsar3:2181
sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@407a7f2a
00:55:48.263 [main-SendThread(pulsar3:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
pulsar3/192.168.15.194:2181. Will not attempt to authenticate using SASL
(unknown error)
00:55:48.284 [main-SendThread(pulsar3:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
pulsar3/192.168.15.194:2181, initiating session
00:55:48.301 [main-SendThread(pulsar3:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
pulsar3/192.168.15.194:2181, sessionid = 0x3000415b0c20001, negotiated timeout
= 30000
[readonly]
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
----
2018-10-16 00:56:43 UTC - Rodrigo Malacarne: @Sijie Guo, that's the result ...
----
2018-10-16 01:35:32 UTC - Rodrigo Malacarne: Hello, in Java it's possible to
subscribe to multiple topics
(<https://pulsar.apache.org/docs/en/client-libraries-java/#multi-topic-subscriptions>).
Does anybody know if this is already possible in c++ (version
2.1.1-incubating)?
----
2018-10-16 01:40:21 UTC - Ali Ahmed: @Rodrigo Malacarne yes we can, pulsar has
wild card topic support
----
2018-10-16 01:41:33 UTC - Ali Ahmed: @Rodrigo Malacarne is your cluster up and
running
----
2018-10-16 01:43:48 UTC - Rodrigo Malacarne: @Ali Ahmed unfortunately not yet
... It's working perfectly for non-persistent topics ... but it somehow doesn't
work for persitent topics ... :disappointed:
----
2018-10-16 01:44:30 UTC - Matteo Merli:
<http://pulsar.apache.org/api/cpp/classpulsar_1_1_client.html#aea2d7918286ecd127751d06e191a5471>
----
2018-10-16 01:45:22 UTC - Rodrigo Malacarne: @Matteo Merli, thank you! I was
wondering if I could use a <vector> of topics in the consumer ... Great
to know !!!
----
2018-10-16 01:45:54 UTC - Ali Ahmed: @Rodrigo Malacarne are you using java or
C++
----
2018-10-16 01:46:10 UTC - Rodrigo Malacarne: @Ali Ahmed, only C++
----
2018-10-16 01:47:46 UTC - Rodrigo Malacarne: @Ali Ahmed, BTW, I'm using right
now a docker-based standalone pulsar for development purposes ... I hope to
solve the question of persistent topics later ...
----
2018-10-16 02:05:49 UTC - Rodrigo Malacarne: @Matteo Merli c++ multi-topic
subscription works like a charm ... thank you!
----
2018-10-16 02:06:12 UTC - Matteo Merli: :tada:
+1 : Rodrigo Malacarne, Karthik Ramasamy, jia zhai
----
2018-10-16 02:06:39 UTC - Matteo Merli: cc/ @jia zhai
----
2018-10-16 02:27:15 UTC - Rodrigo Malacarne: Hello again, in order to produce
to compact topics I need to send key/value pairs in the message. According to
the documentation, inside the MessageBuilder class there are two functions
called setProperties ad setProperty, where one can set "name" and "value"
string values. Are these functions related to key/values, as needed for compat
topics?
----
2018-10-16 02:27:38 UTC - Rodrigo Malacarne: I suppose in setProperties it's
possible to send multiple key/value pairs at once?
----
2018-10-16 02:36:13 UTC - jia zhai: Glad multi-topics
helps:slightly_smiling_face:
+1 : Rodrigo Malacarne
clap : Rodrigo Malacarne
----
2018-10-16 02:37:00 UTC - Rodrigo Malacarne: @jia zhai, ABSOLUTELY !!!
----
2018-10-16 02:40:19 UTC - jia zhai: @Rodrigo Malacarne The Key for compat
should use setKey()
----
2018-10-16 02:40:51 UTC - jia zhai: it is different with other properties
----
2018-10-16 02:43:11 UTC - Rodrigo Malacarne: @jia zhai, good to know... thank
you! But those other two functions setProperties and set Property should be
used to send key/value string pairs for non-compact topics?
----
2018-10-16 02:46:18 UTC - jia zhai: Key is a separate property, it is the Key
for a Message. While other properties setting by setProperties and setProperty
is a kind of internal metadata.
When do compact, we use the Key, not the properties. messages with same Key
will be compacted.
----
2018-10-16 02:46:38 UTC - Rodrigo Malacarne: @jia zhai, BTW, there is no
"setKey()" function in the MessageBuilder class... instead, there is
"setPartitionKey( )"...
----
2018-10-16 02:46:49 UTC - jia zhai: Yes.
+1 : Rodrigo Malacarne
----
2018-10-16 02:46:50 UTC - Rodrigo Malacarne: Maybe this is the correct one?
----
2018-10-16 02:46:56 UTC - jia zhai: It is setPartitionKey
+1 : Rodrigo Malacarne
----
2018-10-16 02:47:01 UTC - jia zhai: :+1:
----
2018-10-16 02:47:56 UTC - Rodrigo Malacarne: @jia zhai, I suppose then the
setContent( ) is naturally used to set the value, right?
----
2018-10-16 02:48:04 UTC - jia zhai: right
----
2018-10-16 02:48:16 UTC - Rodrigo Malacarne: @jia zhai, finally, a more
theoretical question ...
----
2018-10-16 02:48:53 UTC - Rodrigo Malacarne: I've seen in Kafka two methods for
topic compaction... the first one is "time based" and the second one is kind of
"size based" ...
----
2018-10-16 02:49:09 UTC - Rodrigo Malacarne: How does it work in Pulsar?
----
2018-10-16 02:49:45 UTC - Rodrigo Malacarne: Would it be possible to simply
guarantee the very last value for each key to be stored ??
----
2018-10-16 02:50:38 UTC - jia zhai: Pulsar have a time based method
----
2018-10-16 02:51:14 UTC - jia zhai: Yes, the last value is always there.
----
2018-10-16 02:51:36 UTC - Rodrigo Malacarne: I know... but many times together
with many other messages ...
----
2018-10-16 02:52:08 UTC - jia zhai: the very last value is always there, It may
be not compated yet
----
2018-10-16 02:52:10 UTC - Matteo Merli: I think the compaction can be also
triggered on size. In any case you can manually force the compaction to run on
a specific topic
+1 : jia zhai, Rodrigo Malacarne
----
2018-10-16 02:54:02 UTC - Rodrigo Malacarne: @Matteo Merli the manual
enforcement is not what I intend to do ... but that's good to know ... I think
it's more a question of how to configure the topic compaction ...
----
2018-10-16 02:54:31 UTC - Rodrigo Malacarne: Anyway all "basic" options exist
in Pulsar ... and that's amazing ... :+1:
----
2018-10-16 02:55:00 UTC - jia zhai: ```
# Interval between checks to see if topics with compaction policies need to be
compacted
brokerServiceCompactionMonitorIntervalInSeconds=60
```
----
2018-10-16 02:55:05 UTC - jia zhai: It is in broker.conf
pray : Rodrigo Malacarne
----
2018-10-16 03:09:16 UTC - Rodrigo Malacarne: @jia zhai, @Matteo Merli,
key/value string pairs work like a charm ... :clap::clap::clap::clap:
I'll soon be testing topic compaction as well ...
----
2018-10-16 03:09:45 UTC - jia zhai: :+1::+1:
----
2018-10-16 07:53:53 UTC - Martin Svensson: awesome! thanks again
----
