2020-05-13 10:46:53 UTC - Ruian: And one more question. Does the
`bytesInCounter` from the `pulsar-admin topics stats` count as the compressed
message size or the original size?
grey_question : Julius S
----
2020-05-13 14:04:48 UTC - Vincent: Anyone has any idea how to improve Pulsar
SQL performance? I'm using standalone, on 2.4.2, what I see it that it only
able to run at 9k rows /s , as the topic grow, it's getting longer to retrieve
the data.
----
2020-05-13 15:01:56 UTC - Alexandre DUVAL: Hi Vincent, I opened an issue
related to your question about indexes:
<https://github.com/apache/pulsar/issues/6930>
----
2020-05-13 15:35:15 UTC - dionjansen: Hi, does anyone have any experience with
migrating data from one cluster to another (new fresh one)? It seems to me that
both <https://pulsar.apache.org/docs/en/concepts-replication/|geo replication>
and <https://pulsar.apache.org/docs/en/concepts-tiered-storage/|tiered storage>
are not build for this use case, but I could be misunderstanding the docs. My
current suggestion would be to simply read out my persistent topics and push it
to an intermediate datastore and from there push it into the same topics in the
new cluster. But this seems tedious and is a lot of work.
----
2020-05-13 16:15:33 UTC - David Kjerrumgaard: These settings work for me. I did
notice in your last stack trace that the TLS handshake was failing on port
6650, and not 6651. My guess is that if you change the corresponding config to
use the proper port things should work.
----
2020-05-13 16:17:42 UTC - Jeff Schneller: Thanks. I will double check things a
bit later.
----
2020-05-13 17:06:13 UTC - Sijie Guo: I think it is compressed size
----
2020-05-13 17:06:39 UTC - Sijie Guo: Because broker doesn’t attempt to
decompress the data.
----
2020-05-13 17:41:43 UTC - Patrik Kleindl: @Sijie Guo Just to verify, this means
that the broker does not check if the message actually complies to the defined
schema, correct?
If I understood correctly it only verifies if a producer tries to register with
a valid schema but would not protect against a bad/malicious producer sending
invalid data?
----
2020-05-13 17:49:27 UTC - Kirill Merkushev: we do exactly this. Not that much
work actually - could fit into 300 lines including configs.
----
2020-05-13 17:52:56 UTC - Sijie Guo: it verifies the schema when a producer or
consumer connects with the schema.
----
2020-05-13 17:53:47 UTC - Sijie Guo: It doesn’t verify the schema for
individual messages. Assuming the clients serialize the data using the schema.
----
2020-05-13 17:54:41 UTC - Patrik Kleindl: Thanks, that’s what I meant. It is a
good layer of protection but can not provide more without sacrificing
performance.
----
2020-05-13 17:55:08 UTC - Sijie Guo: geo-replication is effectively a
multi-cluster replication mechanism. You can use it for replicating data from
one pulsar cluster to the other cluster.
----
2020-05-13 17:56:16 UTC - Sijie Guo: Alternatively, you can submit an identity
function (using Pulsar Functions) for replicating messages between topics.
----
2020-05-13 17:56:54 UTC - Sijie Guo: Correct.
----
2020-05-13 18:51:36 UTC - Rattanjot Singh: How to build a custom docker image
for pulsar-manager?
----
2020-05-13 18:52:57 UTC - Evan Furman: You should be able to just base your
image on `apachepulsar/pulsar-manager:v0.1.0`
<https://pulsar.apache.org/docs/en/administration-pulsar-manager/>
----
2020-05-13 21:16:25 UTC - Rounak Jaggi: @Rounak Jaggi has joined the channel
----
2020-05-13 22:09:26 UTC - Alpesh Vaghela: @Alpesh Vaghela has joined the channel
----
2020-05-13 22:13:48 UTC - Alan Broddle: We are attempting to configure the TLS
between the Bookie and Broker. We have completed the steps as outlined in the
following location:
<http://bookkeeper.apache.org/docs/latest/security/tls/#bookie-keystore>
*When we try to execute the test command to see the Bookie cert configuration,
we are not seeing the certificate:*
*openssl s_client -debug -connect localhost:3191 -tls*
What is the format of the tlsKeyStorePasswordPath file supposed to be?
We have tried the following:
• pass=password
• password
Note: We are using port 3191
openssl s_client -debug -connect localhost:3191 -tls1_2
CONNECTED(00000003)
write to 0xe4c830 [0xe56363] (289 bytes => 289 (0x121))
0000 - 16 03 01 01 1c 01 00 01-18 03 03 c1 dd e8 8f 8d ................
0010 - b4 ce 15 84 6a b1 f3 2f-d8 c6 c0 ea fd 5f 56 69 ....j../....._Vi
0020 - 60 c6 f4 25 6e 62 7e 85-52 e1 d2 00 00 ac c0 30 `..%nb~.R......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.’.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........
00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................
00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C..
00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................
00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0120 - 01 .
read from 0xe4c830 [0xe51e13] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1589407785
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
Bookkeeper.conf….
# TLS Provider (JDK or OpenSSL).
tlsProvider=OpenSSL
# The path to the class that provides security.
# tlsProviderFactoryClass=org.apache.bookkeeper.security.SSLContextFactory
# Type of security used by server.
# tlsClientAuthentication=true
# Bookie Keystore type.
tlsKeyStoreType=JKS
# Bookie Keystore location (path).
tlsKeyStore=/<path>/bookie.keystore.jks
# Bookie Keystore password path, if the keystore is protected by a password.
tlsKeyStorePasswordPath=/<path>/bookie.keystore.passwd
# Bookie Truststore type.
tlsTrustStoreType=JKS
# Bookie Truststore location (path).
tlsTrustStore=/<path>/bookie.truststore.jks
# Bookie Truststore password path, if the trust store is protected by a
password.
tlsTrustStorePasswordPath=/<path>/bookie.truststore.passwd
# TLS protocols that you are going to accept from clients
tlsEnabledProtocols=TLSv1.2
----
2020-05-13 23:11:06 UTC - JG: Hey guys, I tested the pulsar functions with
non-persisted topcis and apperently it does not but it works well with same
topic name BUT persistent.
Nobody has the problem ? Using last version 2.5.1
----
2020-05-13 23:47:22 UTC - David Kjerrumgaard: @Alan Broddle I think it is
`tlsTrustStorePasswordPath=/var/private/tls/bookie.truststore.passwd`
----
2020-05-13 23:59:09 UTC - Alan Broddle: @David Kjerrumgaard Thanks David, we
have the path. I replaced the <path> to obscure or server information
We are wanting to know what the ‘contents’ of the password file should be. We
tried just put the password in the file. The openssl command does not give a
valid certificate back.
----
2020-05-14 00:07:42 UTC - David Kjerrumgaard: Never tried that, maybe
<https://stackoverflow.com/questions/4294689/how-to-generate-an-openssl-key-using-a-passphrase-from-the-command-line>
can help? Sorry
----
2020-05-14 00:46:55 UTC - Alan Broddle: Can you share a ‘working’ copy of a
bookkeeper.conf and the .password file you are using? Please ‘change’ the
actual password, but wanted to see how you are actually configuring the .conf
file and the format of the password file. We have followed the steps in the
BookKeeper cert, truststore, keystore process.
We are not seeing the openssl command working correctly after we configure the
bookkeeper.conf and restart the bookie
openssl s_client -debug -connect localhost:3191 -tls1_2
We have tested the stores with the following commands, and they seem to be OK.
keytool -v -list -keystore bookie.keystore.jk
<password>
We see the
Alias: caroot
Alias: localhost
keytool -v -list -keystore bookie.truststore.jks
<password>
We see the
Alias: caroot
----
2020-05-14 01:13:03 UTC - Steve Kim: Can someone point me to a detailed
specification of the bookkeeper ledger format? I am looking for something
similar to this: <https://kafka.apache.org/documentation/#implementation>
----
2020-05-14 01:49:10 UTC - VanderChen: Hi, all. There is a question about
*Key-shared Mode* (_Pulsar Version: 2.5.1_). When specifying a key for
consumers, the relationship between the key string and the key hash is not
clear.
We set key for a message in Producer like this
```producer.newMessage().key("key-1").value("message-1-1".getBytes()).send();```
And specify the key for consumer accroding to
<http://pulsar.apache.org/api/client/2.5.0-SNAPSHOT/org/apache/pulsar/client/api/KeySharedPolicy.html|KeySharedPolicy>
```.keySharedPolicy(KeySharedPolicy.stickyHashRange().ranges(Range.of(0,10)))```
But we don't know the relationship between `.key("key-1")` and
`stickyHashRange().ranges(Range.of(0,10))` .
----
2020-05-14 04:33:34 UTC - Piyaphong: @Piyaphong has joined the channel
----
2020-05-14 04:45:55 UTC - Manikandan: @Manikandan has joined the channel
----
2020-05-14 04:56:54 UTC - Adelina Brask: I understand that, but any thoughts on
why should that happen and how to fix it? :slightly_smiling_face:
----
2020-05-14 05:13:56 UTC - Adelina Brask: I can seem to use ElasticSearch SINK
with json data comming from Logstash via NEtty. Anyone see this before? :
`{"error":{"root_cause":[{"type":"not_x_content_exception","reason":"Compressor
detection can only be called on some xcontent bytes or compressed xcontent
bytes"}],"type":"not_x_content_exception","reason":"Compressor detection can
only be called on some xcontent bytes or compressed xcontent
bytes"},"status":500}`
----
2020-05-14 05:56:45 UTC - Sijie Guo: @Penghui Li ^
----
2020-05-14 06:05:20 UTC - Sijie Guo: the content in the password file is the
password. It is expected to be a UTF-8 string.
Alternatively, you can use the same approach used for configuring broker tls
certs to configure bookkeeper tls certs (without keystore and truststore). See
<https://github.com/streamnative/charts/blob/master/charts/pulsar/templates/bookkeeper/_bookkeeper.tpl#L118>
----
2020-05-14 06:06:08 UTC - Sijie Guo: The link you pasted seems to indicate that
you are interested in message format?
----
2020-05-14 06:07:52 UTC - Sijie Guo: I think ElasticSearch requires the
outermost document to be a JSON object, not a JSON array or scalar value. It
might be worth checking the JSON data you sent.
----
2020-05-14 06:09:55 UTC - Sijie Guo: Ideally it should be working with
non-persistent topics. Although we didn’t actually cover that in the
implementation. What issue did you observe?
----
2020-05-14 06:15:18 UTC - Rattanjot Singh: How can we find the response times.
Can we get from logs? Or from broker stats?
----
2020-05-14 06:20:06 UTC - Sijie Guo: What response time are your looking for?
You mean produce time?
----
2020-05-14 06:20:44 UTC - Rattanjot Singh: time taken by a producer to receive
an acknowledgment for a message
----
2020-05-14 06:22:04 UTC - Penghui Li: The `Murmur3_32Hash` is used to calculate
the key hash. If you want to find the relationship between the key and the hash
range, you can use ``Murmur3_32Hash.make(key) mod 65536`` to get the hash slot.
----
2020-05-14 06:23:31 UTC - Sijie Guo: In Producer client, you can get
ProducerStats and find the send latency.
----
2020-05-14 06:23:51 UTC - Sijie Guo: Or you can measure the latency in your
applications as well.
----
2020-05-14 06:50:27 UTC - Rattanjot Singh: Basically I want the time when
broker receives the request from client till the time it send the
acknowledgement to the client that it has written the message successfully.
----
2020-05-14 07:36:52 UTC - Adelina Brask: thanks for the tip
----
2020-05-14 07:38:55 UTC - VanderChen: Thanks a lot!
----
2020-05-14 08:17:27 UTC - Sijie Guo: pulsar_storage_write_latency
----
2020-05-14 08:17:41 UTC - Sijie Guo: it is available from broker metrics
----
2020-05-14 08:19:52 UTC - Sijie Guo: @Sijie Guo set the channel topic: - Pulsar
Summit Virtual Conference 2020 will happen on June 17-18:
<https://pulsar-summit.org/registration>
- Pulsar 2.5.1 released
<https://pulsar.apache.org/blog/2020/04/23/Apache-Pulsar-2-5-1/>
- 2020 Pulsar User Survey Report: <https://bit.ly/3d1KsGG>
----
2020-05-14 08:35:05 UTC - Rattanjot Singh: PUBLISHING LATENCY ==
pulsar_storage_write_latency right?
----
2020-05-14 08:53:15 UTC - Sijie Guo: Yes
----
