Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

ated the KIP to try to make this more clear, but let me know if it's > still not clear. > > Best regards, > > Randall > > On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > mathieu.fenn...@replicon.com> wrote: > >> Hi Randall, >> >> Awesome

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi Randall, Awesome, very much looking forward to this. It isn't 100% clear from the KIP how multiple config-based rules would be applied; it looks like the first configuration rule whose regex matches the topic name will be used, and no other rules will be applied. Is that correct? (I wasn't s

Re: KTable aggregate corruption on poorly timed exception

ce > processing, too. Exactly-once is not required to avoid the bug. > > Can you open a Jira? I think, I know already how to fix it. > > > -Matthias > > > On 5/19/17 8:53 AM, Mathieu Fenniak wrote: >> Whoops, I said I'd put the specific exception at the bottom

[jira] [Created] (KAFKA-5315) Streams exception w/ partially processed record corrupts state store

Mathieu Fenniak created KAFKA-5315: -- Summary: Streams exception w/ partially processed record corrupts state store Key: KAFKA-5315 URL: https://issues.apache.org/jira/browse/KAFKA-5315 Project

KTable aggregate corruption on poorly timed exception

Hi Kafka devs, This morning I observed a specific Kafka Streams aggregation that ended up with an incorrect computed output after a Kafka Streams thread crashed with an unhandled exception. The topology is pretty simple -- a single KTable source, group by a field in the value, aggregate that adds

Re: KTable aggregate corruption on poorly timed exception

at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361) Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 6 ms. On Fri, May 19, 2017 at 9:47 AM, Mathieu Fenniak wrote: > Hi Kafka devs, > > This morning I observed a specific Kafka Streams aggregation that

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

; wondering, if it would make sense to enlarge the scope of this KIP by > >>> that? WDYT? > >>> > >>> > >>> > >>> -Matthias > >>> > >>> > >>> On 5/3/17 2:08 AM, Jeyhun Karimov wrote: > >>>

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

Hi Konstantine, As KIP-151 is currently adding the connector class information to the Kafka Connect REST API, it might be a nice small detail to augment this KIP to add the module name to the same API. T

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

key (accidentally -- or users are just not aware that they are not > > allowed to modify the provided key object) and thus break the > > application. (This was the original motivation to not provide the key in > > the first place -- it's guards against modification.) > >

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Hi Jeyhun, I just want to add my voice that, I too, have wished for access to the record key during a mapValues or similar operation. On the other hand, the number of compile failures that would need to be fixed from this change is unfortunate. :-) But at least it would all be a pretty clear and

Re: [VOTE] 0.10.2.1 RC1

+1 (non-binding) Upgraded KS & KC applications to 0.10.2.1 RC1, successfully ran application-level acceptance tests. Mathieu On Wed, Apr 12, 2017 at 6:25 PM, Gwen Shapira wrote: > Hello Kafka users, developers, client-developers, friends, romans, > citizens, etc, > > This is the second candid

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

+1 (non-binding) On Wed, Apr 5, 2017 at 10:45 AM, Mayuresh Gharat wrote: > Bumping up this thread. > > On Mon, Apr 3, 2017 at 4:22 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> > wrote: > > > Hi All, > > > > It seems that there is no further concern with the KIP-135. At this point > > we

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 (non-binding) This will be very helpful for me, looking forward to it! :-) On Thu, Mar 30, 2017 at 4:46 AM, Damian Guy wrote: > Hi All, > > I'd like to start the voting thread on KIP-134: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 134%3A+Delay+initial+consumer+group+rebalance

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

On Wed, Feb 15, 2017 at 5:04 PM, Matthias J. Sax wrote: > - We also removed method #topologyBuilder() from KStreamBuilder because > we think #transform() should provide all functionality you need to > mix-an-match Processor API and DSL. If there is any further concern > about this, please let us

Re: Cannot start up Kafka Server within Java

Hi Stefan, The mailing list suppressed your attachment, so, it's hard to offer any advice without the error. I'd suggest trying to post the error in a GitHub Gist, or some similar format. I dug up this mailing list archive about starting a Kafka server in-process that might help: http://mail-arc

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

Hi Jeyhun, This KIP might not be the appropriate time, but my first thought reading it is that it might make sense to introduce a builder-style API rather than adding a mix of new method overloads with independent optional parameters. :-) eg. stream(), table(), globalTable(), addSource(), could a

Request Wiki edit permission for mfenniak

Hello, I'd like to contribute a KIP, and so I am requesting access to edit the Kafka Wiki for the user "mfenniak". Thanks, Mathieu

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

On Tue, Feb 14, 2017 at 9:37 AM, Damian Guy wrote: > > And about printing the topology for debuggability: I agrees this is a > > > potential drawback, and I'd suggest maintain some functionality to > build > > a > > > "dry topology" as Mathieu suggested; the difficulty is that, internally > > we

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

On Tue, Feb 14, 2017 at 1:14 AM, Guozhang Wang wrote: > Some thoughts on the mixture usage of DSL / PAPI: > > There were some suggestions on mixing the usage of DSL and PAPI: > https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking it a > bit more carefully, I'd rather not recommend

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

On Mon, Feb 6, 2017 at 2:35 PM, Matthias J. Sax wrote: > - adding KStreamBuilder#topologyBuilder() seems like be a good idea to > address any concern with limited access to TopologyBuilder and DSL/PAPI > mix-and-match approach. However, we should try to cover as much as > possible with #process()

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Hi Matthias, I use a few of the methods that you're pointing out that will be deprecated and don't have an apparent alternative, so I wanted to just let you know what they are and what my use-cases are for them. First of all, I use a combination of DSL and PAPI in the same application very happil

Re: [VOTE] 0.10.2.0 RC0

+1 (non-binding) Upgraded a KS app, custom KC connectors, and brokers, ran an end-to-end test suite. Looks like a great release to me. :-) Mathieu On Wed, Feb 1, 2017 at 4:44 PM, Ewen Cheslack-Postava wrote: > Hello Kafka users, developers and client-developers, > > This is the first candida

Re: Suppressing redundant KTable forwards

o Thereska wrote: > Hi Mathieu, > > Thanks for the suggestion. Wouldn't the cache introduced in KIP-63 do some > of this for you, in that it dedups records with the same key and prevents > them from being forwarded downstream? > > Eno > > On 4 Dec 2016, at 04:13, Mathie

Extending KStream DSL in applications

Hello devs, I'd like to be able to create my own KTableProcessorSupplier implementation in a Kafka Streams application, but it's currently not possible without the use of reflection due to KTableImpl's valueGetterSupplier & enableSendingOldValues methods being package-local. Would there be any op

Suppressing redundant KTable forwards

Hey all, I'd like to contribute a new KTable API that would allow for the suppression of redundant KTable forwards, and I'd like to solicit feedback before I put together a patch. A typical use-case of this API would be that you're using mapValues to pluck a subset of data out of a topic, but you