Re: Setting Memory Bound RocksDB Config

Hi Gray, Answering your question below: > If we set TOTAL_OFF_HEAP_MEMORY, to, say, 1GB, does that mean the app should only use 1GB for the whole application, or is it per store (so 3GB total)? Or perhaps per store and per task (3GB * num tasks)? The TOTAL_OFF_HEAP_MEMORY setting limits the

Re: behavior when PID expires before transactional id expires

Sorry for the typo in your name. It gor autocorrected. Corrected it now as below. Regards, Neeraj > On 19 Nov 2021, at 7:03 am, Neeraj Vaidya > wrote: > > Hi Jiahui, > Have a look at this ticket https://issues.apache.org/jira/browse/KAFKA-13292 > > Regards, > Neeraj > >> On 19 Nov 2021, at

Re: Accessing TLS certs and keys from Vault into Kafka

all JRE will look in the truststore for your certificates and keys Generating a KeyStore and TrustStore (Configuring Java CAPS for SSL Support) (oracle.com) You will need to lookup specific "create truststore" capability for

Setting Memory Bound RocksDB Config

Hello! We are currently trying our best to limit the amount of memory RocksDB uses. We have implemented the config setter prescribed in the RocksDB section of this page: https://kafka.apache.org/30/documentation/streams/developer-guide/memory-mgmt.html#id3, but are still hitting OOMs. My

Re: Stream to KTable internals

I'm wondering if the kafka architects have plans to redesign/enhance this behavior. Having to guess the idle time isn't the most satisfying solution. No matter what time I put in there it seems possible that I will miss a join. Respectfully, Chad On Fri, Nov 5, 2021 at 3:07 PM Matthias J. Sax

Re: behavior when PID expires before transactional id expires

Hi Joshua, Have a look at this ticket https://issues.apache.org/jira/browse/KAFKA-13292 Regards, Neeraj > On 19 Nov 2021, at 2:41 am, Jiahui Jiang wrote: > > Hello Kafka! > > I'm using Flink with a Kafka and seeing an edge case that I'm hoping to get > some help to find the source code that

Re: KTable updates by non-key field

Hi Claudia, Looking at your code you have a cyclic relationship. In other words you are using the same topic for your input and output. If your goal is to always get the most up to date BoxInfo it will not work correctly. The KStream to KTable join matches records from the stream with records

behavior when PID expires before transactional id expires

Hello Kafka! I'm using Flink with a Kafka and seeing an edge case that I'm hoping to get some help to find the source code that would explain! In the Flink setup, I have a transactional producer that's regularly committing transactions with a consistent transactional ID. But all these

Accessing TLS certs and keys from Vault into Kafka

Hello, I have a use case where I am using a vault like ansible vault to encrypt and store my SSL related files (certs and Keys) for other existing applications. And I would like to know if it's possible to use the same vault with Kafka SSL without creating jks, pkcs12 or pem files outside the

ACL permissions bypass

Hi all, While debugging an issue in our dev environment, I came to the realization that we are not setting the DESCRIBE ACL permission for a given topic/service-account mapping, yet the consumer is able to fetch messages from that topic without any issues. The documentation says this topic ACL

Re: Re: Accessing TLS certs and keys from Vault into Kafka

Hello, > can you provide any existing implementations for reference? You can check the `FileConfigProvider` in clients/src/main/java/org/apache/kafka/common/config/provider/FileConfigProvider.java. Or check the repo: https://github.com/jcustenborder/kafka-config-provider-vault Hope this helps.

RE: Re: Accessing TLS certs and keys from Vault into Kafka

Okay. Thank you very much for the information. If it's not too much trouble can you provide any existing implementations for reference? Also, I would like to ask about the LDAP implementation of kafka. Is it true that we have to write our own ldaphandler in order to use LDAP with apache kafka?

Re: Accessing TLS certs and keys from Vault into Kafka

You can add a Vault provider for externalized configs by implementing a ` org.apache.kafka.common.config.provider.ConfigProvider`.Details are in https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations and

Accessing TLS certs and keys from Vault into Kafka

Hello, I have a use case where I am using a vault like ansible vault to encrypt and store my SSL related files (certs and Keys) for other existing applications. And I would like to know if it's possible to use the same vault with Kafka SSL without creating jks, pkcs12 or pem files outside the