2018-12-08 11:14:30 UTC - Christophe Bornet: Not sure how to make it work with
regexp. But at least this seems to work : `--secondary "pulsar1-us,pulsar2-us"
--primary "pulsar1-eu,pulsar2-eu"`
----
2018-12-08 16:19:33 UTC - Mike Card: Now Matteo what exactly is this telling
me? That this test ran 4 threads in parallel to get 1000 messages/sec and each
thread was running at about 250 Hz, hence the 4 ms latency? At 4 ms latency a
thread couldn't get 1,000 messages/sec
----
2018-12-08 16:20:33 UTC - Matteo Merli: To publish at high rate, you need to
use `sendAsync()` to pipeline more messages
----
2018-12-08 16:21:03 UTC - Matteo Merli: `send()` waits while message is being
published, so 1 thread throughput is bound by the publish latency
----
2018-12-08 16:21:28 UTC - Mike Card: OK so is this test telling me the publish
latency is 4ms?
----
2018-12-08 16:21:51 UTC - Matteo Merli: Yes, avg latency
----
2018-12-08 16:23:33 UTC - Matteo Merli: Recommendation is to use sendAsync
(which gives you a future to track the publish completion) and to set
`blockIfQueueFull=true` on the producer builder (to simplify the backpressure)
----
2018-12-08 16:23:47 UTC - Matteo Merli: With that you can have much higher
throughput.
----
2018-12-08 16:24:05 UTC - Matteo Merli: that’s what the `pulsar-perf produce`
tool does
----
2018-12-08 16:24:37 UTC - Matteo Merli: You can play with that (eg: increasing
the `--rate` value to check what’s the max publish rate)
----
2018-12-08 16:26:53 UTC - Mike Card: OK. Now in my test code I was just doing a
send() but they were at a very high rate, much greater than 250 Hz. It seemed
like the sent messages were being truncated to 64 bytes
----
2018-12-08 16:27:13 UTC - Mike Card: This did not happen at low message rates
----
2018-12-08 17:08:19 UTC - Matteo Merli: @Christophe Bornet have you tried with
double quotes and used `.` ? eg: `--secondary "pulsar.*-us"`
----
2018-12-08 17:11:49 UTC - legendtkl: hello，it seems the rpm and deb link not
work in this url:
<https://pulsar.apache.org/docs/en/client-libraries-cpp/#compilation>
----
2018-12-08 17:16:21 UTC - Mike Card: What happens on an asynchronous send if
the queue fills up but blockIfQueueFull is not set to true?
----
2018-12-08 17:16:53 UTC - Matteo Merli: You get queue full error
----
2018-12-08 17:18:35 UTC - Matteo Merli: @legendtkl you’re right :/
----
2018-12-08 17:18:54 UTC - Matteo Merli: The link is missing a `-1` in the name
----
2018-12-08 17:19:16 UTC - Mike Card: OK. Now if I am calling send() way beyond
250 Hz could messages get truncated to 64 bytes?
----
2018-12-08 17:30:00 UTC - legendtkl: @Matteo Merli yeah, got it. Need pr for
this ?
----
2018-12-08 17:30:25 UTC - Matteo Merli: Please
----
2018-12-08 17:32:15 UTC - legendtkl: OK，I will do it later. Now it’s time to
have a sleep :sleeping:
----
2018-12-08 18:26:38 UTC - Ravi: our container went down and ever since, it is
not coming up with issues - 2018-12-08 18:22:24.803 INFO ConnectionPool:63 |
Created connection for <pulsar://192.168.1.33:6650>
2018-12-08 18:22:24.804 ERROR ClientConnection:329 | [<none> ->
<pulsar://192.168.1.33:6650>] Failed to establish connection: Connection refused
2018-12-08 18:22:24.804 INFO ClientConnection:1182 | [<none> ->
<pulsar://192.168.1.33:6650>] Connection closed
2018-12-08 18:22:24.804 ERROR ClientImpl:158 | Error Checking/Getting Partition
Metadata while creating producer on <persistent://public/default/sentences> -- 5
2018-12-08 18:22:24.804 INFO ClientConnection:195 | [<none> ->
<pulsar://192.168.1.33:6650>] Destroyed connection
----
2018-12-08 18:27:12 UTC - Ravi: the compose restart starts off the container
but connectioons get dropped
----
2018-12-08 18:28:29 UTC - Karthik Ramasamy: @Ravi - how did you build the
container?
----
2018-12-08 18:29:02 UTC - Karthik Ramasamy: which framework was it using pulsar
functions vs pulsar io
----
2018-12-08 18:29:04 UTC - Ravi: version: '3'
services:
pulsar:
image: "apachepulsar/pulsar:2.1.0-incubating"
expose:
- 8001
- 8010
- 6650
ports:
- "192.168.1.33:6650:6650"
- "192.168.1.33:8010:8080"
- "192.168.1.33:8001:8000"
restart: always
command: >
/bin/bash -c
"bin/apply-config-from-env.py conf/standalone.conf
&& bin/pulsar standalone"
networks:
- streamsnet
networks:
streamsnet:
driver: bridge
----
2018-12-08 18:29:15 UTC - Ravi: that has no changes
----
2018-12-08 18:29:27 UTC - Ravi: from last week it worked as is
----
2018-12-08 18:29:59 UTC - Ravi: Here is the container start log
----
2018-12-08 18:30:00 UTC - Ravi: Recreating streamlio_pulsar_1 ... done
Attaching to streamlio_pulsar_1
pulsar_1 | [AppClassLoader@37a71e93] info AspectJ Weaver Version 1.9.1 built
on Friday Apr 20, 2018 at 16:47:33 GMT
pulsar_1 | [AppClassLoader@37a71e93] info register classloader
sun.misc.Launcher$AppClassLoader@37a71e93
pulsar_1 | [AppClassLoader@37a71e93] info using configuration
file:/pulsar/lib/org.apache.pulsar-pulsar-zookeeper-2.1.0-incubating.jar!/META-INF/aop.xml
pulsar_1 | [AppClassLoader@37a71e93] info using configuration
file:/pulsar/lib/org.apache.pulsar-pulsar-broker-2.1.0-incubating.jar!/META-INF/aop.xml
pulsar_1 | [AppClassLoader@37a71e93] info register aspect
org.apache.pulsar.zookeeper.FinalRequestProcessorAspect
pulsar_1 | [AppClassLoader@37a71e93] info register aspect
org.apache.pulsar.zookeeper.ZooKeeperServerAspect
pulsar_1 | [AppClassLoader@37a71e93] info register aspect
org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
pulsar_1 | 18:21:31.074 [main] INFO
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Running 1 bookie(s) and
advertised them at 5d80eaf59462.
pulsar_1 | 18:21:31.471 [main] INFO
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Starting ZK server
pulsar_1 | [AppClassLoader@37a71e93] info processing reweavable type
org.apache.zookeeper.server.ZooKeeperServer:
org/apache/zookeeper/server/ZooKeeperServer.java
pulsar_1 | [AppClassLoader@37a71e93] info successfully verified type
org.apache.pulsar.zookeeper.ZooKeeperServerAspect exists. Originates from
org/apache/pulsar/zookeeper/ZooKeeperServerAspect.java
pulsar_1 | [AppClassLoader@37a71e93] info processing reweavable type
org.apache.zookeeper.server.FinalRequestProcessor:
org/apache/zookeeper/server/FinalRequestProcessor.java
pulsar_1 | [AppClassLoader@37a71e93] info successfully verified type
org.apache.pulsar.zookeeper.FinalRequestProcessorAspect exists. Originates
from org/apache/pulsar/zookeeper/FinalRequestProcessorAspect.java
pulsar_1 | 18:23:32.731 [main] INFO
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Server UP
pulsar_1 | 18:23:32.749 [main] INFO
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - ZooKeeper server up: true
pulsar_1 | 18:23:32.749 [main] INFO
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Instantiate ZK Client
pulsar_1 | 18:23:33.468 [main] INFO
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble - Starting Bookie(s)
----
2018-12-08 18:30:26 UTC - Karthik Ramasamy: essentially you bring up pulsar
container using docker compose
----
2018-12-08 18:33:19 UTC - Ravi: yes
----
2018-12-08 18:33:33 UTC - Ravi: pulsar container comes up
----
2018-12-08 18:33:42 UTC - Ravi: only connections fail since 2 hrs
----
2018-12-08 18:33:45 UTC - Ravi: no idea how
----
2018-12-08 18:34:08 UTC - Karthik Ramasamy: can you get the pulsar container
log while connecting?
----
2018-12-08 18:34:11 UTC - Ravi: going to prune all volumes and restart
container to see how it work
----
2018-12-08 18:34:24 UTC - Karthik Ramasamy: ok
----
2018-12-08 18:38:17 UTC - Ravi: rebuilding now, will post once done
----
2018-12-08 18:41:33 UTC - Ravi: docker prune seems to have made it to work
----
2018-12-08 18:42:08 UTC - Ravi: looks like some socket was locked on rebuilds,
so had to clear the full stash again. this seems like a docker issue than
streamlio issue !
----
2018-12-08 21:27:19 UTC - Matteo Merli: @Mike Card I’ve never seen such
behavior. How did you get that suspicion?
----
2018-12-08 21:27:44 UTC - Matteo Merli: Do you have a consumer getting these
messages?
----
2018-12-08 23:33:30 UTC - Beast in Black: @Christophe Bornet You are a
lifesaver! I had figured out the `min_available` bit and that the params needed
`min_limit` by looking at the code but was stuck on the
`--auto-failover-policy-params` bit because I couldn't figure out what the
format was in which the command expected the input. I ran the following command
with regexes for namespace and brokers it worked without errors (shell return
value 0):
```
bin/pulsar-admin ns-isolation-policy set <MY_PULSAR_CLUSTER> mypolicy
--namespaces ['*'] --primary ['my-pulsar-broker-*'] --auto-failover-policy-type
min_available --auto-failover-policy-params min_limit=0,usage_threshold=100
```
@David Kjerrumgaard I have not yet checked whether this command solves the
policy errors on the broker and the subscription issues - I'll run a couple of
tests and get back to you on this. Is there a roadmap item to expose setting
`ns-isolation-policy` via the REST API? cc @Matteo Merli
----
2018-12-08 23:37:05 UTC - Beast in Black: @Christophe Bornet BTW, since I am
running this on K8s on AWS, if you are interested the full command I ran is:
```
kubectl --kubeconfig=<MY_K8s_KUBECFG> -n <MY_K8s_NAMESPACE> exec
-it <MY_PULSAR_ADMIN_PODNAME> -- bin/pulsar-admin ns-isolation-policy set
<MY_PULSAR_CLUSTER> mypolicy --namespaces ['*'] --primary
['my-pulsar-broker-*'] --auto-failover-policy-type min_available
--auto-failover-policy-params min_limit=0,usage_threshold=100
```
----
2018-12-08 23:47:46 UTC - Beast in Black: @Christophe Bornet I removed the `[]`
from the regexes and that seemed to work too - at least, the command executed
and returned 0 without spitting out errors, although I'm not sure what effect
it had on the actual policy settings vis-a-vis the previous command:
```
pulsar-admin ns-isolation-policy set <MY_PULSAR_CLUSTER> mypolicy
--namespaces '*' --primary 'my-pulsar-broker-*' --auto-failover-policy-type
min_available --auto-failover-policy-params min_limit=0,usage_threshold=100
```
----
2018-12-08 23:51:13 UTC - Matteo Merli: @Beast in Black the namespace isolation
is already configured through REST API (that’s what the CLI tool uses anyway).
Definitely is not (ehem) very well documented:
<http://pulsar.apache.org/en/admin-rest-api/#operation/setNamespaceIsolationPolicy>
----
2018-12-08 23:51:45 UTC - Matteo Merli: For Java admin API:
<http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Clusters.html#createNamespaceIsolationPolicy-java.lang.String-java.lang.String-org.apache.pulsar.common.policies.data.NamespaceIsolationData->
----
2018-12-08 23:55:20 UTC - Beast in Black: @Matteo Merli oh, excellent, thank
you! David K. was not sure whether it was exposed via REST.
----
2018-12-08 23:59:54 UTC - Beast in Black: @Christophe Bornet a followup - it
seems that neither of my commands mentioned above (one with `[]` in the regex,
one without) to set the policy actually worked even though they appeared to
succeed. If I check the `stats` and `stats-internal` command on a topic in one
of my namespaces it still says that no policy was found :disappointed:
```
pulsar-admin topics stats-internal
<MYTENANT>/<MY_NAMESPACE1>/<MY_TOPIC1>
Policies not found for <MYTENANT>/<MY_NAMESPACE1> namespace
Reason: Policies not found for <MYTENANT>/<MY_NAMESPACE1> namespace
command terminated with exit code 1
```
I'll dink around a little more with the command and see what needs to be done
for getting it to report a successful policy - I'm wondering if I need to put
the command in a loop and execute it explicitly for each namespace rather than
using a regex.
----
2018-12-09 03:44:35 UTC - Samuel Sun: what could be the reason if the
webservice-url is not up when cluster is running properly ? Thanks
----
2018-12-09 04:48:40 UTC - Samuel Sun: fixed manually
----
2018-12-09 08:48:23 UTC - Christophe Bornet: No. But I will :smiley:
----
