On 2016-11-28 13:47 (-0500), "Matthias J. Sax" <[email protected]> wrote:
>
> I want to start a discussion about KIP-95:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams
>
> Looking forward to your feedback.
Hi Matthias,
I’d just like to share some feedback on this proposal, from the perspective
of a Kafka user rather than developer:
* Overall, makes a ton of sense and I think it’ll be very useful and will
open up a bunch of additional use cases to Kafka Streams
* Two use cases that are of particular interest to me:
* Just last week I created a simple Kafka Streams app (I called it a
“script”) to copy certain records from one topic over to another,
with filtering. I ran the app/script until it reached the end of
the topic, then I manually shut it down with ctrl-c. This worked
fine, but it would have been an even better UX to have specified
the config value `autostop.at` as `eol` and have the process stop
itself at the desired point. That would have required less manual
monitoring on my part.
* With this new mode I might be able to run Streams apps on AWS Lambda.
I’ve been super-excited about Lambda and similar FaaS services since
their early days, and I’ve been itching to run Kafka Streams apps on
Lambda for since I started using Streams in April or May. Unfortunately,
Lambda functions are limited to 5 minutes per invocation — after 5
minutes they’re killed. I’m not sure, but I wonder if perhaps this new
autostop.at feature could make it more practical to run a Streams
app on Lambda - the feature seems like it could potentially be adapted
to enable a Streams app to be more generally resilient to being
frequently stopped and started.
I look forward to seeing progress on this enhancement!
Thanks,
Avi
————
Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/
