Re: [VOTE] 1.1.0 RC2

[Satish Duggana]

Hi Damian,
Thanks for starting vote thread for 1.1.0 release.

There may be a typo on the tag to be voted upon for this release candidate.
I guess it should be https://github.com/apache/kafka/tree/1.1.0-rc2 instead
of https://github.com/apache/kafka/tree/1.1.0-rc.

On Wed, Mar 14, 2018 at 8:27 AM, Satish Duggana 
wrote:

> Hi Damian,
> Given release plan link in earlier mail is about 1.0 release. You may want
> to replace that with 1.1.0 release plan link[1].
>
> 1 - https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=75957546
>
> Thanks,
> Satish.
>
> On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the third candidate for release of Apache Kafka 1.1.0.
>>
>> This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
>> Please see the release plan for more details:
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913
>>
>> A few highlights:
>>
>> * Significant Controller improvements (much faster and session expiration
>> edge cases fixed)
>> * Data balancing across log directories (JBOD)
>> * More efficient replication when the number of partitions is large
>> * Dynamic Broker Configs
>> * Delegation tokens (KIP-48)
>> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>>
>> Release notes for the 1.1.0 release:
>> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Friday, March 16, 1pm PDT>
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
>>
>> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
>> https://github.com/apache/kafka/tree/1.1.0-rc
>> 2
>>
>>
>> * Documentation:
>> http://kafka.apache.org/11/documentation.html
>> 
>>
>> * Protocol:
>> http://kafka.apache.org/11/protocol.html
>> 
>>
>> * Successful Jenkins builds for the 1.1 branch:
>> Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
>> System tests: https://jenkins.confluent.io/j
>> ob/system-test-kafka/job/1.1/38/
>>
>> /**
>>
>> Thanks,
>> Damian
>>
>>
>> *
>>
>
>

Re: Solution for clients with long-lived sustained SSL connections using JKS

[Kaufman Ng]

I agree with Sönke that kerberos is a better choice here. SSL and JKS is
probably NOT the right choice of this short lived (48 hours) scenario.
Typically certs are valid in order of years.

On Tue, Mar 13, 2018 at 7:21 PM, Sönke Liebau <
soenke.lie...@opencore.com.invalid> wrote:

> Hi Alexander,
>
> I don't (yet) have any real suggestions for what you are trying to
> achieve, but I'd be interested to understand if you looked at the
> alternative forms of authentication instead of SSL. Namely Kerberos
> which is already available and delegation tokens which will be
> available in 1.1
> From reading your message it seems like Kerberos would be a near
> perfect fit for what you are trying to achieve, whereas I can't shake
> the impression that you are trying to "bend" SSL  a little to
> accomodate your specific use case.
>
> Best regards,
> Sönke
>
> On Mon, Mar 12, 2018 at 8:46 PM, Alexander Maniates 
> wrote:
> > Our set up:
> >
> > Brokers on 0.10.1
> > Clients on 0.9
> >
> > -On startup, clients are dynamically issued a signed certificate that is
> > vaild for 48 hours. A JKS is created using this certificate.
> > -All brokers have a signed certificate in their JKS that is valid for
> some
> > years.
> >
> > The issue:
> >
> > Clients only load their JKS once on startup. After 48 hours when the
> > certificate expires, if a broker then restarts, clients are not able to
> > make a new SSL connection with the JKS and certificate that was loaded on
> > startup.
> >
> > We have thousands of clients running at any given time, and do not want
> to
> > need to restart every service each time the certificates expire. We could
> > also make our client certificates last longer but that seems like a
> > possible security flaw.
> >
> > Our first proposed solution was to just rewrite the underlying JKS with a
> > new certificate every hour or so. However, as I mentioned, the JKS is
> only
> > loaded once at startup, so clients will never load this new JKS with a
> new
> > vaild certificate.
> >
> > In the context of a producer, the solution we are thinking of is to
> develop
> > a wrapper that is essentially a rolling client. Every so often, you
> rewrite
> > the JKS with a new valid certificate, create a new client which will load
> > the new JKS, swap the main client with the old client, then close the
> > original client and repeat the process.
> >
> > Has anybody else run into this problem and found a good solution? I'm
> > interested to hear any other solutions for tearing down and rebuilding
> SSL
> > connections on the fly.
> >
> >
> > Thanks,
> > Alex
>
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>



-- 
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io

Re: [VOTE] 1.1.0 RC2

[Satish Duggana]

Hi Damian,
Given release plan link in earlier mail is about 1.0 release. You may want
to replace that with 1.1.0 release plan link[1].

1 -
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546

Thanks,
Satish.

On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 1.1.0.
>
> This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
> Please see the release plan for more details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913
>
> A few highlights:
>
> * Significant Controller improvements (much faster and session expiration
> edge cases fixed)
> * Data balancing across log directories (JBOD)
> * More efficient replication when the number of partitions is large
> * Dynamic Broker Configs
> * Delegation tokens (KIP-48)
> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>
> Release notes for the 1.1.0 release:
> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, March 16, 1pm PDT>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> https://github.com/apache/kafka/tree/1.1.0-rc
> 2
>
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
> 
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
> 
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
> System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/
> 38/
>
> /**
>
> Thanks,
> Damian
>
>
> *
>

Re: Solution for clients with long-lived sustained SSL connections using JKS

[Sönke Liebau]

Hi Alexander,

I don't (yet) have any real suggestions for what you are trying to
achieve, but I'd be interested to understand if you looked at the
alternative forms of authentication instead of SSL. Namely Kerberos
which is already available and delegation tokens which will be
available in 1.1
>From reading your message it seems like Kerberos would be a near
perfect fit for what you are trying to achieve, whereas I can't shake
the impression that you are trying to "bend" SSL  a little to
accomodate your specific use case.

Best regards,
Sönke

On Mon, Mar 12, 2018 at 8:46 PM, Alexander Maniates  wrote:
> Our set up:
>
> Brokers on 0.10.1
> Clients on 0.9
>
> -On startup, clients are dynamically issued a signed certificate that is
> vaild for 48 hours. A JKS is created using this certificate.
> -All brokers have a signed certificate in their JKS that is valid for some
> years.
>
> The issue:
>
> Clients only load their JKS once on startup. After 48 hours when the
> certificate expires, if a broker then restarts, clients are not able to
> make a new SSL connection with the JKS and certificate that was loaded on
> startup.
>
> We have thousands of clients running at any given time, and do not want to
> need to restart every service each time the certificates expire. We could
> also make our client certificates last longer but that seems like a
> possible security flaw.
>
> Our first proposed solution was to just rewrite the underlying JKS with a
> new certificate every hour or so. However, as I mentioned, the JKS is only
> loaded once at startup, so clients will never load this new JKS with a new
> vaild certificate.
>
> In the context of a producer, the solution we are thinking of is to develop
> a wrapper that is essentially a rolling client. Every so often, you rewrite
> the JKS with a new valid certificate, create a new client which will load
> the new JKS, swap the main client with the old client, then close the
> original client and repeat the process.
>
> Has anybody else run into this problem and found a good solution? I'm
> interested to hear any other solutions for tearing down and rebuilding SSL
> connections on the fly.
>
>
> Thanks,
> Alex



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Is Kafka Streams right for me ?

[Matthias J. Sax]

Beside state and DSL Kafka Stream also allow for fully fault-tolerant,
salable, and elastic deployment.


-Matthias

On 3/13/18 10:55 AM, Hans Jespersen wrote:
> "If your system is stateless and the transformations are not interdependent"
> then I would just look at using Kafka Connect's Single Message Transform
> (SMT) feature.
> 
> -hans
> 
> /**
>  * Hans Jespersen, Director Systems Engineering, Confluent Inc.
>  * h...@confluent.io (650)924-2670
>  */
> 
> On Tue, Mar 13, 2018 at 9:18 AM, Jacob Sheck  wrote:
> 
>> If you are augmenting streaming data with dimensional data, or if you can
>> transform your data with a map, filter or join operation Streams will be a
>> good option.  If your system is stateless and the transformations are not
>> interdependent, you may want to look into using one of the queue
>> technologies like AcitveMQ.
>>
>> On Tue, Mar 13, 2018 at 10:00 AM Sameer Rahmani 
>> wrote:
>>
>>> Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline
>>> is a several components which connected together via a stream
>> abstraction.
>>> each component has a input and output stream. Basically the source of
>>> this pipeline can be anything and the output can be anything to.
>>>
>>> On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck  wrote:
>>>
 Sameer when you say that you need to "consume from and produce to a
>>> topic"
 to me that seems like a good fit for Kafka Streams.  Streaming your
>> data
 out of Kafka for a transform and back in has some fundamental costs and
 operational challenges involved.  Are the events in your stream
>>> stateless?
 If it isn't stateless streams will ensure a consistent playback of
>> events
 if needed.  Without knowing more about your pipeline it is hard to make
 recommendations.  Are you possibly using Kafka as a distributed queue?

 On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani 
 wrote:

> Hi folks,
> I need to consume from and produce to a topic. I have my own data
 pipeline
> to process the data.
> So I was wondering beside the stores and StreamDSL what does Kafka
 Streams
> brings to the table
> that might be useful to me ?
>

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

[Andrew Otto]

​Hm, the hardware should be mostly beefier in this new cluster, but there
are a couple of differences: Mainly we are now using RAID instead of JBOD,
and​ our high volume producers (that 150K / second I mentioned) use TLS.

Also, other producers seem have no problems, it is only MirrorMaker.  Even
stranger, this problem has gone away since yesterday.  Several bounces of
MirrorMaker have happened between then and now (as we tried a few different
MirrorMaker settings).  So far, this is a mystery.

Thanks for the reply Andras! If it happens again I’ll look into your
suggestion.


On Tue, Mar 13, 2018 at 4:15 PM, Andras Beni 
wrote:

> Hi Andrew,
>
> It seems the throughput of the new cluster is smaller than that of the old
> cluster. And for this reason MirrorMaker cannot send messages fast enough
> (i.e. they expire). I recommend comparing the configurations.
> For the hanging MirrorMaker instances, I think looking at stack dumps would
> help you get closer to the root cause.
>
> Best regards,
> Andras
>
> On Mon, Mar 12, 2018 at 7:56 PM, Andrew Otto  wrote:
>
> > Hi all,
> >
> > I’m troubleshooting a MirrorMaker issue, and am not quite sure yet why
> this
> > is happening, so I’d thought I’d ask here in case anyone else has seen
> this
> > before.
> >
> > We’ve been running a Kafka 1.0 cluster for a few months now, replicating
> > data from a Kafka 0.9.0.1 cluster using 0.9.0.1 MirrorMaker.  This had
> > mostly been working fine, until last week when we finished migrating a
> set
> > of high volume producers to this new Kafka 1.0 cluster.  Before last
> week,
> > the new 1.0 cluster was handling around 50-70K messages per second, as of
> > last week it is up to around 150K messages per second.
> >
> > Everything in the new 1.0 cluster is working totally fine.  However,
> since
> > we increased traffic to the new 1.0 cluster, our previously operational
> > MirrorMaker instances started dying with a lot of
> >
> > ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback
> -
> > Error when sending message to topic  with key: null, value: 964 bytes
> > with error: Batch Expired
> >
> > and
> >
> > ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback
> -
> > Error when sending message to topic  with key: null, value: 964 bytes
> > with error: Producer is closed forcefully.
> >
> > and
> >
> > INFO  kafka.tools.MirrorMaker$  - Exiting on send failure, skip
> committing
> > offsets.
> >
> >
> > This log is printed for some of the higher volume (not really, these are
> > around 1K messages per second max) topics that MirrorMaker is
> replicating.
> > This happens a few minutes after the MirrorMaker instance starts.  Until
> > this happens, it is able to produce fine.  Once it happens, the instance
> > dies and a rebalance and is triggered. I’m haven’t been able to
> > consistently reproduce what happens next, but it seems that after a short
> > series of instance flapping + rebalancing happens, the MirrorMaker
> > instances all totally get stuck.  They continue owning partitions, but
> > don’t produce anything, and don’t log any more errors.  The MirrorMaker
> > consumers start lagging.
> >
> > A full restart of all instances seems to reset the thing, but eventually
> > they get stuck again.
> >
> > Perhaps there’s some problem I’ve missed with older MirrorMaker producing
> > to new Kafka clusters?  Could more load on the brokers cause MirrorMaker
> > produce requests to expire like this?  We’re were running the same load
> on
> > our old 0.9.0.1 cluster before this migration, with the same MirrorMaker
> > setup with no problems.
> >
> > Our MirrorMaker and 1.0 broker configuration is here:
> > https://gist.github.com/ottomata/5324fc3becdd20e9a678d5d37c2db872
> >
> > Any help is appreciated, thanks!
> >
> > -Andrew Otto
> >  Senior Systems Engineer
> >  Wikimedia Foundation
> >
>

Re: Kafka 0.9 MirrorMaker failing with Batch Expired when producing to Kafka 1.0 cluster

[Andras Beni]

Hi Andrew,

It seems the throughput of the new cluster is smaller than that of the old
cluster. And for this reason MirrorMaker cannot send messages fast enough
(i.e. they expire). I recommend comparing the configurations.
For the hanging MirrorMaker instances, I think looking at stack dumps would
help you get closer to the root cause.

Best regards,
Andras

On Mon, Mar 12, 2018 at 7:56 PM, Andrew Otto  wrote:

> Hi all,
>
> I’m troubleshooting a MirrorMaker issue, and am not quite sure yet why this
> is happening, so I’d thought I’d ask here in case anyone else has seen this
> before.
>
> We’ve been running a Kafka 1.0 cluster for a few months now, replicating
> data from a Kafka 0.9.0.1 cluster using 0.9.0.1 MirrorMaker.  This had
> mostly been working fine, until last week when we finished migrating a set
> of high volume producers to this new Kafka 1.0 cluster.  Before last week,
> the new 1.0 cluster was handling around 50-70K messages per second, as of
> last week it is up to around 150K messages per second.
>
> Everything in the new 1.0 cluster is working totally fine.  However, since
> we increased traffic to the new 1.0 cluster, our previously operational
> MirrorMaker instances started dying with a lot of
>
> ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback  -
> Error when sending message to topic  with key: null, value: 964 bytes
> with error: Batch Expired
>
> and
>
> ERROR org.apache.kafka.clients.producer.internals.ErrorLoggingCallback  -
> Error when sending message to topic  with key: null, value: 964 bytes
> with error: Producer is closed forcefully.
>
> and
>
> INFO  kafka.tools.MirrorMaker$  - Exiting on send failure, skip committing
> offsets.
>
>
> This log is printed for some of the higher volume (not really, these are
> around 1K messages per second max) topics that MirrorMaker is replicating.
> This happens a few minutes after the MirrorMaker instance starts.  Until
> this happens, it is able to produce fine.  Once it happens, the instance
> dies and a rebalance and is triggered. I’m haven’t been able to
> consistently reproduce what happens next, but it seems that after a short
> series of instance flapping + rebalancing happens, the MirrorMaker
> instances all totally get stuck.  They continue owning partitions, but
> don’t produce anything, and don’t log any more errors.  The MirrorMaker
> consumers start lagging.
>
> A full restart of all instances seems to reset the thing, but eventually
> they get stuck again.
>
> Perhaps there’s some problem I’ve missed with older MirrorMaker producing
> to new Kafka clusters?  Could more load on the brokers cause MirrorMaker
> produce requests to expire like this?  We’re were running the same load on
> our old 0.9.0.1 cluster before this migration, with the same MirrorMaker
> setup with no problems.
>
> Our MirrorMaker and 1.0 broker configuration is here:
> https://gist.github.com/ottomata/5324fc3becdd20e9a678d5d37c2db872
>
> Any help is appreciated, thanks!
>
> -Andrew Otto
>  Senior Systems Engineer
>  Wikimedia Foundation
>

[VOTE] 1.1.0 RC2

[Damian Guy]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 1.1.0.

This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
Please see the release plan for more details:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71764913

A few highlights:

* Significant Controller improvements (much faster and session expiration
edge cases fixed)
* Data balancing across log directories (JBOD)
* More efficient replication when the number of partitions is large
* Dynamic Broker Configs
* Delegation tokens (KIP-48)
* Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)

Release notes for the 1.1.0 release:
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Friday, March 16, 1pm PDT>

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~damianguy/kafka-1.1.0-rc2/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
https://github.com/apache/kafka/tree/1.1.0-rc
2


* Documentation:
http://kafka.apache.org/11/documentation.html


* Protocol:
http://kafka.apache.org/11/protocol.html


* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.1-jdk7/78
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.1/38/

/**

Thanks,
Damian


*

Re: Is Kafka Streams right for me ?

[Hans Jespersen]

"If your system is stateless and the transformations are not interdependent"
then I would just look at using Kafka Connect's Single Message Transform
(SMT) feature.

-hans

/**
 * Hans Jespersen, Director Systems Engineering, Confluent Inc.
 * h...@confluent.io (650)924-2670
 */

On Tue, Mar 13, 2018 at 9:18 AM, Jacob Sheck  wrote:

> If you are augmenting streaming data with dimensional data, or if you can
> transform your data with a map, filter or join operation Streams will be a
> good option.  If your system is stateless and the transformations are not
> interdependent, you may want to look into using one of the queue
> technologies like AcitveMQ.
>
> On Tue, Mar 13, 2018 at 10:00 AM Sameer Rahmani 
> wrote:
>
> > Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline
> > is a several components which connected together via a stream
> abstraction.
> > each component has a input and output stream. Basically the source of
> > this pipeline can be anything and the output can be anything to.
> >
> > On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck  wrote:
> >
> > > Sameer when you say that you need to "consume from and produce to a
> > topic"
> > > to me that seems like a good fit for Kafka Streams.  Streaming your
> data
> > > out of Kafka for a transform and back in has some fundamental costs and
> > > operational challenges involved.  Are the events in your stream
> > stateless?
> > > If it isn't stateless streams will ensure a consistent playback of
> events
> > > if needed.  Without knowing more about your pipeline it is hard to make
> > > recommendations.  Are you possibly using Kafka as a distributed queue?
> > >
> > > On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani 
> > > wrote:
> > >
> > > > Hi folks,
> > > > I need to consume from and produce to a topic. I have my own data
> > > pipeline
> > > > to process the data.
> > > > So I was wondering beside the stores and StreamDSL what does Kafka
> > > Streams
> > > > brings to the table
> > > > that might be useful to me ?
> > > >
> > >
> >
>

Re: Is Kafka Streams right for me ?

[Jacob Sheck]

If you are augmenting streaming data with dimensional data, or if you can
transform your data with a map, filter or join operation Streams will be a
good option.  If your system is stateless and the transformations are not
interdependent, you may want to look into using one of the queue
technologies like AcitveMQ.

On Tue, Mar 13, 2018 at 10:00 AM Sameer Rahmani  wrote:

> Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline
> is a several components which connected together via a stream abstraction.
> each component has a input and output stream. Basically the source of
> this pipeline can be anything and the output can be anything to.
>
> On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck  wrote:
>
> > Sameer when you say that you need to "consume from and produce to a
> topic"
> > to me that seems like a good fit for Kafka Streams.  Streaming your data
> > out of Kafka for a transform and back in has some fundamental costs and
> > operational challenges involved.  Are the events in your stream
> stateless?
> > If it isn't stateless streams will ensure a consistent playback of events
> > if needed.  Without knowing more about your pipeline it is hard to make
> > recommendations.  Are you possibly using Kafka as a distributed queue?
> >
> > On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani 
> > wrote:
> >
> > > Hi folks,
> > > I need to consume from and produce to a topic. I have my own data
> > pipeline
> > > to process the data.
> > > So I was wondering beside the stores and StreamDSL what does Kafka
> > Streams
> > > brings to the table
> > > that might be useful to me ?
> > >
> >
>

Re: Is Kafka Streams right for me ?

[Sameer Rahmani]

Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline
is a several components which connected together via a stream abstraction.
each component has a input and output stream. Basically the source of
this pipeline can be anything and the output can be anything to.

On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck  wrote:

> Sameer when you say that you need to "consume from and produce to a topic"
> to me that seems like a good fit for Kafka Streams.  Streaming your data
> out of Kafka for a transform and back in has some fundamental costs and
> operational challenges involved.  Are the events in your stream stateless?
> If it isn't stateless streams will ensure a consistent playback of events
> if needed.  Without knowing more about your pipeline it is hard to make
> recommendations.  Are you possibly using Kafka as a distributed queue?
>
> On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani 
> wrote:
>
> > Hi folks,
> > I need to consume from and produce to a topic. I have my own data
> pipeline
> > to process the data.
> > So I was wondering beside the stores and StreamDSL what does Kafka
> Streams
> > brings to the table
> > that might be useful to me ?
> >
>

Re: Is Kafka Streams right for me ?

[Jacob Sheck]

Sameer when you say that you need to "consume from and produce to a topic"
to me that seems like a good fit for Kafka Streams.  Streaming your data
out of Kafka for a transform and back in has some fundamental costs and
operational challenges involved.  Are the events in your stream stateless?
If it isn't stateless streams will ensure a consistent playback of events
if needed.  Without knowing more about your pipeline it is hard to make
recommendations.  Are you possibly using Kafka as a distributed queue?

On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani  wrote:

> Hi folks,
> I need to consume from and produce to a topic. I have my own data pipeline
> to process the data.
> So I was wondering beside the stores and StreamDSL what does Kafka Streams
> brings to the table
> that might be useful to me ?
>

Is Kafka Streams right for me ?

[Sameer Rahmani]

Hi folks,
I need to consume from and produce to a topic. I have my own data pipeline
to process the data.
So I was wondering beside the stores and StreamDSL what does Kafka Streams
brings to the table
that might be useful to me ?