Re: Partition assignment in kafka streams

2019-08-01 Thread Navneeth Krishnan
Hi Guozhang, Thanks for the clarification. What I want to achieve is use of localized data. We have much larger state which has to be used at a per instance context. So if I can get the assignment upfront I can basically write data to partitions in such a way that all data goes to that specific no

Re: Partition assignment in kafka streams

2019-08-01 Thread Guozhang Wang
Hello Navneeth, I may be misunderstanding your intent from the previous emails here, so just a quick summary: 1) if you just want to "know" which partitions are assigned to which instance, this can be retrieved in multiple ways (e.g. the one mentioned by Matthias, and also one can get this info f

Re: Kafka S3 Connector

2019-08-01 Thread lsroudi abdel
I guess LinkedIn open source a solution aimed to resolve this kind of problem, it called Brooklin Le jeu. 1 août 2019 à 17:23, a écrit : > Hi Team, > > I handle operations on Big Data Platform. We have a kafka cluster, which > has to be connected to S3 buckets from on-prem servers. > While searc

Kafka S3 Connector

2019-08-01 Thread himansu . panigrahy
Hi Team, I handle operations on Big Data Platform. We have a kafka cluster, which has to be connected to S3 buckets from on-prem servers. While searching, I did find only Confluent Kafka has S3 Connectors. So I was wondering if there is any solution, apache has in place for this. Kindly suggest