Re: Question on Kafka Streams

Hello Mark, That's a very general question, and the answer depends on your hardware, your computational logic etc. Could you elaborate a bit more on your use case? Guozhang On Mon, Apr 1, 2019 at 11:16 AM Mark Fursht wrote: > Hello, > > I would like to know, how many opened windows Kafka

Re: leader none, with only one replicat end no ISR

Hi Adrien, What happens when you try running the preferred replica leader election script? Does this restore leadership for those partitions you listed? https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool Harper On Sat, Mar

Question on Kafka Streams

Hello, I would like to know, how many opened windows Kafka streams can hold? Sincerely, Mark [cid:image001.png@01D3B960.F400ABC0] This message has been scanned for malware by Websense. www.websense.com

Strange KafkaConsumer IllegalStateException

Hi list, I've a question regarding a stack trace I see with the 2.2.0 consumer java.lang.IllegalStateException: No entry found for connection 0| at org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:339)| at

Re: [External] Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

If somebody insists on using Kafka as a database you might be able to do the following: 1. Create a "compacted topic". The key for the topic should be the point of sales id. 2. Create a webservice which takes a point of sale id and can read or update the topic 3. Have the point of sale apps

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Thanks, Vincent! Yes, i totally agree about coupling/decoupling. The need for the producer to send messages to a specific partition and the consumer to consume from a specific partition creates high coupling between them. So in the future changing the topic partitioning scheme will be impossible

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Thanks, Dmitry. It seems your solution is the the most appropriate for me, because the reasons why consumers (POS terminals) will be added/removed are different from the reasons why partitions will be added/removed. I think that topic division to partitions and POS terminals division to logical

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Maybe you can find some litterature about messaging patterns ? Usually, a single kafka topic is used to do PubSub pattern, i.e decoupling producers and consumers. In your case, it seems that the situation is quite coupled, i.e you need to generate and send 20k price lists to 20k specific

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Going off of what Hans mentioned, I don't see any reason for 200,000 partitions...you don't need one partition per POS. You can have all of your pos listening to one partition and each pos agent having a unique group id. The POS agent only processes the messages that are relevant to him, and

Re: Offsets of deleted consumer groups do not get deleted correctly

Hi Claudia Just a sidenote, there is a combined policy for "compact, delete" which deletes messages older than retention.ms and compacts newer ones if I remember correctly. It's still not really in the docs as it seems https://kafka.apache.org/documentation/#topicconfigs best regards Patrik On

plaintext connection attempts to SSL secured broker

Hi, We have our brokers secured with these standard properties listeners=SSL://a.b.c:9030 ssl.truststore.location=... ssl.truststore.password=... ssl.keystore.location=... ssl.keystore.password=... ssl.key.password=... ssl.client.auth=required ssl.enabled.protocols=TLSv1.2 It's a bit surprising

Re: Kafka Streams upgrade.from config while upgrading to 2.1.0

Thanks a lot, Guozhang! That clarifies it to a great extent. I'll try to figure out who the leader of the group is. On Wed, Mar 27, 2019 at 9:59 PM Guozhang Wang wrote: > Hello Anirudh, > > The config `upgrade.from` is recommended for safe and smooth upgrade. In > your case it is possible that

AW: Offsets of deleted consumer groups do not get deleted correctly

Hi, thanks for your reply. Also the groups were deleted month ago, there are still valid values there. So I guess deleting the group did not produce the tombstone record correctly. Your explanation made it clearer for me. I know I should keep cleanup policy compact in general. I decided to

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Yes but you have more than 1 POS terminal per location so you still don't need 20,000 partitions. Just one per location. How many locations do you have? In doesn’t matter anyway since you can build a Kafka cluster with up to 200,000 partitions if you use the latest versions of Kafka.

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Thanks, Hans! We use location specific SKU pricing and send specific price lists to the specific POS terminal. пн, 1 апр. 2019 г., 3:01 Hans Jespersen : > Doesn’t every one of the 20,000 POS terminals want to get the same price > list messages? If so then there is no need for 20,000 partitions.