2018-10-14 18:29:59 UTC - Matteo Merli: @Julien Nioche I forgot that localrun
is actually using the process based functions runtime, so it spins up a new
process to actually run the function code. I don’t remember why it wasn’t using
the thread runner instead, which would simplify the debugging.
----
2018-10-14 18:53:50 UTC - Julien Nioche: Thanks @Matteo Merli, is there a way
we can pass options to the process running the function code?
----
2018-10-14 18:55:50 UTC - Julien Nioche: I am running a function in local mode
and am getting loads of exception messages such as
{
"exceptionString": "State is not enabled.",
"msSinceEpoch": "1539543083557"
}
Is there anything specific I need to do to be able to use the state?
----
2018-10-14 18:56:35 UTC - Julien Nioche: The input is a simple, non-distributed
topic
----
2018-10-14 20:38:39 UTC - Rodrigo Malacarne: @Rodrigo Malacarne has joined the
channel
----
2018-10-14 21:02:25 UTC - Rodrigo Malacarne: Hi, I have set up a Pulsar cluster
on 3 virtual machines under VMWare Workstation. After a fresh reboot, what I do
to start all parts involved is:
1) "./bin/pulsar-daemon start zookeeper" on all machines
2) "./bin/pulsar-daemon start configuration-store" on all machines
3) "./bin/pulsar-daemon start bookie" on all machines
4) "./bin/pulsar-daemon start broker" on all machines
So far so good... I have developed a c++ producer an a c++ customer and
everything works abolutely fine for non-persistent topics, with a very good
latency (90% below 1ms).
However, I'm stuck on how to create and use *persistent* topics. As far as I
know, to use persistent topics I need to have enough bookies up and running on
each machine, in order for them to store all the necessary topic data. But
after almost 10 days stuck and trying many different things, I decided to look
for help here :slightly_smiling_face:
I'm using the most recent Pulsar version (2.1.1-incubating), and I believe the
problem is somehow related to bookkeeper, as I'm not able to successfully
perform sanity tests on bookkeeper.
I have checked ports 2181, 2184, 3181 and 6650 on all machines, and all of them
are responding. However, a simple "bookiesanity" test fails returning the
message *"Not enough non-faulty bookies available"*.
Can you guys please guide me on this issue?
----
2018-10-14 21:04:28 UTC - Rodrigo Malacarne:
----
2018-10-14 21:05:10 UTC - Rodrigo Malacarne: I'm also sending the log for the
sanitytest command. Thanks in advance for your help!
----
2018-10-14 21:05:48 UTC - Ali Ahmed: @Rodrigo Malacarne are you using default
configs ?
----
2018-10-14 21:06:22 UTC - Rodrigo Malacarne: @Ali Ahmed no... If needed I can
also post here all the .conf files.
----
2018-10-14 21:07:15 UTC - Ali Ahmed: you have threes bookies as you say what is
your replication count ?
----
2018-10-14 21:09:00 UTC - Rodrigo Malacarne: I have changed the default
replication to just 1, as I've read on Pulsar's Github that this could at least
start one bookie for development purposes ...
----
2018-10-14 21:09:09 UTC - Rodrigo Malacarne: Didn't work btw ...
----
2018-10-14 21:10:46 UTC - Ali Ahmed: seems strange, can you post the bookie logs
----
2018-10-14 21:12:00 UTC - Rodrigo Malacarne: @Ali Ahmed, which log,
specifically?
----
2018-10-14 21:12:09 UTC - Rodrigo Malacarne: Please advise me, and I'll post it
here ...
----
2018-10-14 21:12:38 UTC - Ali Ahmed: logs being generated by the bookie process
```/bin/pulsar-daemon start bookie```
----
2018-10-14 21:12:50 UTC - Rodrigo Malacarne: Under pulsar/logs?
----
2018-10-14 21:13:15 UTC - Rodrigo Malacarne: 1 moment please ...
----
2018-10-14 21:14:25 UTC - Ali Ahmed: the command should output where the logs
go when started
----
2018-10-14 21:14:46 UTC - Rodrigo Malacarne:
----
2018-10-14 21:17:46 UTC - Ali Ahmed: you have zookeeper running on all three
nodes ?
----
2018-10-14 21:18:08 UTC - Rodrigo Malacarne: Yes
----
2018-10-14 21:18:43 UTC - Ali Ahmed: is zk healthy ?
----
2018-10-14 21:19:08 UTC - Rodrigo Malacarne: How do I check this?
----
2018-10-14 21:19:15 UTC - Rodrigo Malacarne:
----
2018-10-14 21:19:43 UTC - Ali Ahmed: ```echo ruok | nc ZkHost 2181```
----
2018-10-14 21:20:01 UTC - Rodrigo Malacarne: Yes ...
----
2018-10-14 21:20:09 UTC - Rodrigo Malacarne: All 3 zk's are ok
----
2018-10-14 21:20:28 UTC - Rodrigo Malacarne: Just got "imok" for all 3 zk's
----
2018-10-14 21:22:20 UTC - Ali Ahmed: did you initialize the cluster metadata ?
```
Initializing cluster metadata
Once you've deployed ZooKeeper for your cluster, there is some metadata that
needs to be written to ZooKeeper for each cluster in your instance. It only
needs to be written once.
You can initialize this metadata using the initialize-cluster-metadata command
of the pulsar CLI tool. This command can be run on any machine in your
ZooKeeper cluster. Here's an example:
$ bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper
<http://zk1.us-west.example.com:2181|zk1.us-west.example.com:2181> \
--configuration-store
<http://zk1.us-west.example.com:2181|zk1.us-west.example.com:2181> \
--web-service-url <http://pulsar.us-west.example.com:8080> \
--web-service-url-tls <https://pulsar.us-west.example.com:8443> \
--broker-service-url <pulsar://pulsar.us-west.example.com:6650> \
--broker-service-url-tls <pulsar+ssl://pulsar.us-west.example.com:6651>
```
----
2018-10-14 21:22:39 UTC - Rodrigo Malacarne: Yes, I did
----
2018-10-14 21:23:07 UTC - Ali Ahmed: ok well’s try and see the brokers logs the
bookies look okay
----
2018-10-14 21:23:39 UTC - Rodrigo Malacarne: Should I post the broker's logs
here as well?
----
2018-10-14 21:23:51 UTC - Ali Ahmed: sure also try this
```
bin/pulsar-client produce \
<persistent://public/default/test> \
-n 1 \
-m "Hello, Pulsar"
```
----
2018-10-14 21:24:51 UTC - Rodrigo Malacarne:
----
2018-10-14 21:26:35 UTC - Rodrigo Malacarne: @Ali Ahmed, I've just run the
pruduce command you posted above ..
----
2018-10-14 21:26:39 UTC - Rodrigo Malacarne: Here are the logs ...
----
2018-10-14 21:27:14 UTC - Rodrigo Malacarne:
----
2018-10-14 21:29:11 UTC - Rodrigo Malacarne: Same error ...
"(...)ManagedLedgerException: Not enough non-faulty bookies available"
----
2018-10-14 21:30:16 UTC - Ali Ahmed: don’t know the bookie log you posted looks
okay maybe the other nodes have errors
----
2018-10-14 21:31:57 UTC - Rodrigo Malacarne: I've just run the pulsar-client
produce command on the other 2 machines ...
----
2018-10-14 21:32:23 UTC - Rodrigo Malacarne: Same error on them
----
2018-10-14 21:32:39 UTC - Ali Ahmed: the problem is bookie can you get the
bookie logs from other nodes
----
2018-10-14 21:32:54 UTC - Rodrigo Malacarne: Sure ...
----
2018-10-14 21:32:57 UTC - Rodrigo Malacarne: 1 moment please ..
----
2018-10-14 21:33:45 UTC - Rodrigo Malacarne:
----
2018-10-14 21:34:11 UTC - Rodrigo Malacarne:
----
2018-10-14 21:37:06 UTC - Ali Ahmed: don’t see the issue best bring in @Matteo
Merli and @Sijie Guo they can provide more guidance
----
2018-10-14 21:38:55 UTC - Ali Ahmed: in summary you have a three node cluster,
each running an instance of zk, bk and broker, setup looks okay to me and
services are up but cluster is unable to write to enough replicas to satisfy
quorum requirements.
----
2018-10-14 21:39:16 UTC - Ali Ahmed: possibly something is wrong with the setup
but am not sure what exactly
----
2018-10-14 21:40:48 UTC - Rodrigo Malacarne: @Ali Ahmed, I appreciate your
help... I'm sure it is related to configuration ...
----
2018-10-14 21:41:16 UTC - Rodrigo Malacarne: I'll post here my .conf files as
well ... maybe I'm not able to see what's wrong with them...
----
2018-10-14 21:41:49 UTC - Rodrigo Malacarne:
----
2018-10-14 21:41:53 UTC - Rodrigo Malacarne:
----
2018-10-14 21:42:12 UTC - Rodrigo Malacarne:
----
2018-10-14 21:42:32 UTC - Rodrigo Malacarne:
----
2018-10-14 21:43:33 UTC - Rodrigo Malacarne: These are the .conf files I'm
using for the first machine ... and I'm replicating the same .conf files for
all other machines, with the necessary machine-related modifications, of course
...
----
2018-10-15 00:30:02 UTC - Sijie Guo: @Rodrigo Malacarne
`advertisedAddress=pulsar1` - I see you are using `pulsar1` for the
advertisedAddress, is `pulsar1` a reachable dns name?
Also can you run `bin/bookkeeper shell listbookies -rw` in a broker node?
----
2018-10-15 01:02:13 UTC - Rodrigo Malacarne: @Sijie Guo, yes, pulsar1, pulsar2
and pulsar3 are all reachable dns names, registered at /etc/hosts in all
machines.
----
2018-10-15 01:10:03 UTC - Rodrigo Malacarne: @Sijie Guo, regarding the
listbookies command, I get "No bookie exists!" for all 3 machines
----
2018-10-15 01:11:57 UTC - Rodrigo Malacarne:
----
2018-10-15 01:12:44 UTC - Rodrigo Malacarne: BTW, the same for -ro (readonly)
----
2018-10-15 01:23:17 UTC - Rodrigo Malacarne: @Sijie Guo, BTW, when I leave
'advertisedAddress=' (blank) the same error occurs and no bookies can be found
----
2018-10-15 05:39:51 UTC - sriram: @sriram has joined the channel
----
2018-10-15 08:22:21 UTC - yaoshuai: @yaoshuai has joined the channel
----
