2020-07-16 09:19:59 UTC - Shivji Kumar Jha: What class name in below broker
conf can i set to use tls auth? Don't see an implementation of
BookieAuthProvider.java for this in broker/bookie code..
```bookkeeperClientAuthenticationPlugin```
----
2020-07-16 11:15:03 UTC - Shivji Kumar Jha: @Sijie Guo trying to do what you
describe
<https://youtu.be/sTISVpyq73o?list=PLqRma1oIkcWhWAhKgImEeRiQi5vMlqTc-&t=2578|here>.
----
2020-07-16 12:12:38 UTC - Joshua Decosta: @Sijie Guo That doesn’t seem to be
the case in my situation. The first authentication provider fails and then it
causes an exception which leads to the cluster crashing.
Edit: i think it might be cause I wasn’t throwing an AuthenticionException. I’m
going to test that and see if it’s the issue
----
2020-07-16 15:18:53 UTC - Addison Higham: good point, will see about getting
some more docs there to give some ideas of why you might use that call
+1 : charles
----
2020-07-16 15:31:53 UTC - Joshua Eric: I thought about the return type, but
because it was just a string representing bytes that forms Avro, felt like it
wouldn't help. Almost seems like exception raising logic in the decorator makes
the most sense so you can effectively validate the input and output of the
function to match what the producer created.
----
2020-07-16 15:32:15 UTC - Joshua Eric: I am not sure what you mean by the
encode feedback.
----
2020-07-16 15:38:58 UTC - Addison Higham: @Shivji Kumar Jha Ifor TLS auth you
doesn't use that property, you instead use
`bookkeeperTLSClientAuthentication=true` on brokers and then in your bookkeeper
settings you set `tlsClientAuthentication=true`
----
2020-07-16 17:20:27 UTC - Yezen: Hey @Joe Francis thanks for jumping in,
So I have a service that is a producer of messages.
This service needs to send messages for many different tenants. You can
imagine that it is a tenant management service.
Anytime there is a change in a tenant initiated by this service it will need to
broadcast a message using pulsar.
The service will need to encrypt any messages using an encryption key
specifically for that tenant and send the message out. Our service will handle
the fetching of the correct key for the tenant.
*Let me give an example:*
Pretend this service handles tenanted data regarding `Apache` and
`StreamNative` . You use our service to add a new employee to the `Apache`
tenant. Our service will then publish the message of said employee to pulsar
encrypting the message using a key that belongs to `Apache`. If there was a
modification to the `StreamNative` tenant the tenant management service would
need to encrypt its messages using a different key.
^
Basically I want to make sure that this can be supported through Pulsar. It
seems like it can in where you can use as many encryption keys as you want and
then use the necessary one depending on the tenant for every message you
publish.
----
2020-07-16 17:34:43 UTC - Shivji Kumar Jha: Hi @Addison Higham I have got tls
working but as soon as I turn these two configurations to true, i start getting
tls errors. Setting to false, again everything starts working. Here is my
bookie conf
```#############################################################################
## TLS settings
#############################################################################
# TLS Provider (JDK or OpenSSL).
tlsProvider=OpenSSL
# The path to the class that provides security.
tlsProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory
# Type of security used by server.
tlsClientAuthentication=true
# Bookie Keystore type.
tlsKeyStoreType=JKS
# Bookie Keystore location (path).
tlsKeyStore=/certs/<filename>.jks
# Bookie Keystore password path, if the keystore is protected by a password.
tlsKeyStorePasswordPath=/certs/<passwd-filename>.passwd
# Bookie Truststore type.
tlsTrustStoreType=JKS
# Bookie Truststore location (path).
tlsTrustStore=/usr/local/share/ca-certificates/pulsar-devgov-pki-ca.jks
# Bookie Truststore password path, if the trust store is protected by a
password.
tlsTrustStorePasswordPath=/certs/<filename>.passwd```
broker.conf below
```# Set the client security provider factory class name.
# Default: org.apache.bookkeeper.tls.TLSContextFactory
bookkeeperTLSProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory
# Enable tls authentication with bookie
bookkeeperTLSClientAuthentication=true
# Supported type: PEM, JKS, PKCS12. Default value: PEM
bookkeeperTLSKeyFileType=PEM
#Supported type: PEM, JKS, PKCS12. Default value: PEM
bookkeeperTLSTrustCertTypes=PEM
# Path to file containing keystore password, if the client keystore is password
protected.
bookkeeperTLSKeyStorePasswordPath=
# Path to file containing truststore password, if the client truststore is
password protected.
bookkeeperTLSTrustStorePasswordPath=
# Path for the TLS private key file
bookkeeperTLSKeyFilePath=/certs/<filename>.key
# Path for the TLS certificate file
bookkeeperTLSCertificateFilePath=/certs/<filename>.crt
# Path for the trusted TLS certificate file
bookkeeperTLSTrustCertsFilePath=/certs/<filename>.pem```
----
2020-07-16 17:45:58 UTC - Shivji Kumar Jha: bookie crash
```Caused by: javax.net.ssl.SSLHandshakeException: error:100000c0:SSL
routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE
at
io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1264)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1225)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1296)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1339)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
~[io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
~[io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
... 15 more```
```17:43:50.461 [bookkeeper-io-19-1] ERROR
org.apache.bookkeeper.proto.PerChannelBookieClient - TLS failure on: [id:
0x1ff2b1f2, L:/10.160.6.240:46952 !
R:<http://pulsar-node2.beamdevgov.com/10.160.6.172:3181|pulsar-node2.beamdevgov.com/10.160.6.172:3181>]```
----
2020-07-16 18:10:17 UTC - Rahul Vashishth: What are the use cases for which
folks are using message retention for acknowledged messages?
----
2020-07-16 18:56:44 UTC - Joshua Decosta: Ok it was the lack of Throwing an
AuthenticationException that caused the issue
----
2020-07-16 20:48:31 UTC - Joshua Decosta: Is there a recommended way to package
custom auth classes into the default docker image?
----
2020-07-16 21:00:42 UTC - Addison Higham: The easiest way is simply to do a
small docker image like:
```FROM apachepulsar/pulsar:<version>
COPY your-local-jar /pulsar/lib/```
`/pulsar/lib` is on the classpath, so that is all you really need to do
From there, just push to a docker registry of your choice.
If you want to use docker to also build your jar, then using multi-stage build
is nice
----
2020-07-16 21:08:26 UTC - Addison Higham: One way to think about Pulsar:
A subscription is a "view" on a topic, it represents what you have and haven't
seen yet. When you acknowledge a message, you acknowledge it *per
subscription*. so if you have multiple subscriptions, acknowledgements only
have an effect on a single subscription.
Beyond subscriptions, topics themselves just have retention, but you can always
create a new subscription at any point in the retained messages.
Given that background, here are a few use cases that are common:
- Application data: Setting long (sometimes infinite) retention and using
Pulsar as the store for some events
- Server logs: Setting retention of a few days/weeks and using a topic per
server to store logs, you can easily build tooling to snag logs for that period
of time
- Stream processing: systems like Flink need to have more control over cursor
position, they can use "Reader" API instead of consumer API and more accurately
control how data is read
----
2020-07-16 21:08:28 UTC - Joshua Decosta: @Addison Higham So if i was building
it within the current pulsar-broker-common. Would i simply add that built jar
to that lib folder?
----
2020-07-16 21:09:24 UTC - Devin G. Bost: What would cause Pulsar’s
unackedMessage value to be almost a 1000 times larger than the msgBacklog value
for a function with parallelism of 1?
----
2020-07-16 21:10:37 UTC - Devin G. Bost: It definitely looks like whatever disk
is available to Pulsar is showing as too small.
----
2020-07-16 21:10:55 UTC - Devin G. Bost: Have you checked the mount location?
----
2020-07-16 21:11:21 UTC - Devin G. Bost: Maybe the location you’re expecting
isn’t using your mounted volumes.
----
2020-07-16 21:13:47 UTC - Addison Higham: I think that would work, but in which
case, you will need to replace
`org.apache.pulsar-pulsar-broker-common-<version>.jar` with your jar
----
2020-07-16 21:14:03 UTC - Addison Higham: you might find it easier to put in
it's own package in the long run
+1 : Joshua Decosta, Rahul Vashishth
----
2020-07-16 21:14:05 UTC - Addison Higham: but it would work
----
2020-07-16 21:21:08 UTC - krishna reddy jaggannolla: @krishna reddy jaggannolla
has joined the channel
----
2020-07-16 22:32:15 UTC - Yezen: Pretend my company moves forward with using
Pulsar. What type of support does Pulsar provide? Who would I reach out if we
have issues or find bugs with Pulsar? What are the best ways to ask questions
related to Pulsar as we use it?
----
2020-07-16 22:41:32 UTC - Sijie Guo: Slack, Github and Mailing lists are good
for community support. We (StreamNative) provide the managed services and
enterprise support for Pulsar.
+1 : Yezen, Shivji Kumar Jha
----
2020-07-16 22:58:16 UTC - Greg Methvin: This is a great forum to ask questions.
Just remember that like all OSS projects, you’re not guaranteed any kind of
support, so be patient and polite, and if you need stronger support guarantees
consider purchasing support services from StreamNative or a similar company. I
know <https://kesque.com/> also offers managed Pulsar services.
+1 : Yezen
----
2020-07-16 22:59:12 UTC - Greg Methvin: If you find bugs, you can also report
them on github, but you might want to ask here first if you’re not sure what
you’ve found is a bug
----
2020-07-17 00:05:23 UTC - Foong Fook Won: Appears that this command is not able
to show or proof that TLS is configured. Managed to find another way to verify
TLS in bookie.
----
2020-07-17 02:02:43 UTC - SoulEater: @SoulEater has joined the channel
----
2020-07-17 05:33:38 UTC - Rahul Vashishth: > but you can always create a new
subscription at any point in the retained messages.
Thanks, this clears a lot. I was under impression that a new subscription can
only be applied to backlog quota.
When we create a subscription for the retained message, does these messages get
marked as unacknowledged?
----
2020-07-17 06:11:29 UTC - Rahul Vashishth: @Addison Higham How do we create a
subscription for retained messages.
When we create Subscription with `SubscriptionInitialPosition` set to
`Earliest` does it create the cursor in *Retained Storage* or in *Backlog
Storage* ?
----
2020-07-17 07:41:36 UTC - Arushi Sharma: Hi, I need to make message encryption
configurable(optional) in my application on producer end. However, I could not
find any property for EncryptionKey and CryptoKeyReader for loadConf function.
Can someone suggest a way I can make it optional.
PS- I could not run the producer if encryptionKey is null.
----
