Reduce function Null checks

Hi community, When using, reduce(Reducer,Reducer, KeyValueMapper,String) function in KTable, the NullPointerExeption is thrown. In specified function, below call is made: return reduce(adder, subtractor, selector, null, null, name); and afterwards, in reduce(Reducer,Reducer,

Re: Operator order

d to predict. However, the > > filter operators will run in the same thread (it's more or less just > > another chained method call), thus, it should not be too large. > > Furthermore, it should never the required to write tagged record to > > Kafka -- thus, it would only

Re: Operator order

DAG at runtime would help > you? Do you mean you would dynamically change the edge between A->B and > A->sink ? I guess, this would be a very special pattern and I doubt that > any library or system can offer this. > > -Matthias > > On 06/18/2016 05:33 PM, Jeyhun Karimov wrote:

Operator order

Hi community, Is there a way in Kafka Streams to change the order of operators in runtime? For example, I have operators Source->A->B->C->D->E->Sink and I want to forward some tuples from A to E, from B to Sink and etc. As far as I know, the stream execution graph is computed in compile time

Re: Parallelisation factor in kafka streams

u mean by "restart the cluster"? For sure, you do not > need to restart the Kafka Brokers. And for a Streams application, there > is no cluster. Streams applications are regular Java applications and > can run anywhere (not necessarily on the same machines as Kafka Brokers). > > -Ma

Re: Parallelisation factor in kafka streams

via > parameter "num.stream.threads" (default value is 1). > > See > > http://docs.confluent.io/3.0.0/streams/developer-guide.html#optional-configuration-parameters > > > -Matthias > > On 07/06/2016 06:11 PM, Jeyhun Karimov wrote: > > Hi commu

Parallelisation factor in kafka streams

Hi community, How can I set parallelisation factor in kafka streams? Is it related with the number of partitions within topics? Cheers Jeyhun -- -Cheers Jeyhun

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

verloads with independent optional > parameters. > > :-) > > > > eg. stream(), table(), globalTable(), addSource(), could all accept a > > "TopicReference" parameter that can be built like: > > > TopicReference("my-topic").keySerde(...).value

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

uot;TopicReference" parameter that can be built like: > >> > TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).timestampExtractor(...).build(). > >> > >> Mathieu > >> > >> > >> On Tue, Feb 14, 2017

[DISCUSS] KIP-123: Allow per stream/table timestamp extractor

Dear community, I want to share the KIP-123 [1] which is based on issue KAFKA-4144 [2]. You can check the PR in [3]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788 [2] https://issues.apache.org/jira/browse/KAFKA-4144 [3]

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

time vs. wallclock time) in their > processing inadvertently. Before this KIP, all the relevant topics have one > time stamp extractor so that issue does not come up. > > What will be the behavior if times mismatch, e.g., for joins? > > Thanks > Eno > > > On 22 Feb 2017,

[VOTE] KIP-123: Allow per stream/table timestamp extractor

Dear community, I'd like to start the vote for KIP-123: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788 Cheers, Jeyhun -- -Cheers Jeyhun

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

. If the join is not allowed, how do you prevent that join from > happening? Do you throw an exception? > > Thanks > Eno > > > > On 28 Feb 2017, at 10:04, Jeyhun Karimov <je.kari...@gmail.com> wrote: > > > > Hi Eno, > > > > Thanks for feedback. I thin

Streams support for Serdes

Hi community, When using kafka-streams with POJO data types we write our own de/serializers. However I think if we have built-in Serdes support for Tuple-n data types (ex:Serdes.Tuple2) we may easily use Tuples and built-in Serdes can help to reduce the development cycle. Please

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

sides the minor comments mentioned above. > > > Guozhang > > > On Mon, Mar 6, 2017 at 3:14 PM, Jeyhun Karimov <je.kari...@gmail.com> > wrote: > > > Sorry I was late. I will update javadocs in related methods to emphasize > > that TimestampExtractor is statele

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

> wrote: > > > > Thanks Jeyhun. > > > > Can you also update the KIP accordingly. It must contain all changes to > > public API. Thus, list all parameters that get deprecated and newly > > added. And add a sentence about backward compatibility. > > >

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

gt; > On Wed, Mar 1, 2017 at 6:09 AM, Michael Noll <mich...@confluent.io> > > wrote: > > > > > >> +1 (non-binding) > > >> > > >> Thanks for the KIP! > > >> > > >> On Wed, Mar 1, 2017 at 1:49 PM, Bill Bejeck <bbej...@gmail.com&g

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

text() method from > > > InternalProcessorContext to ProcessorContext. > > > > > > In the KIP you have an example showing: > > > richMapper.init((RecordContext) processorContext); > > > But the interface is: > > > public interface RichValueMa

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

<matth...@confluent.io> wrote: > I would not block this KIP with regard to DSL refactoring. IMHO, we can > just finish this one and the DSL refactoring will help later on to > reduce the number of overloads. > > -Matthias > > On 7/7/17 5:28 AM, Jeyhun Karimov wrote: >

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Eno Thereska <eno.there...@gmail.com> > > wrote: > > > > > > > +1 (non-binding) Thanks. > > > > > > > > Eno > > > > > On 6 Jul 2017, at 21:49, Gwen Shapira <g...@confluent.io> wrote: > > > > > > > > >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

for ProcessorContext. This will > > break compatibility. > > > > I think, we should just have two independent interfaces. Our own > > ProcessorContextImpl class would implement both. This allows us to cast > > it to `RecordContext` and thus limit the visible sc

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

> >> upgrade guidance. > >> > >> Regarding the scope I'm still trying to solicit opinions regarding > >> ReducerWithKey and InitializerWithKey; to me they are not necessarily > to be > >> included. > >> > >> > >> Guozhang > >> &g

[VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Dear all, I would like to start the vote on KIP-149 [1]. Cheers, Jeyhun [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner -- -Cheers Jeyhun

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

to address this issue as well or continue as it is? Cheers, Jeyhun On Wed, Jun 14, 2017 at 11:29 PM Guozhang Wang <wangg...@gmail.com> wrote: > LGTM. Thanks! > > > Guozhang > > On Tue, Jun 13, 2017 at 2:20 PM, Jeyhun Karimov <je.kari...@gmail.com> > wrote: >

Re: Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

ated() or similar? So that users don't have to read the docs > to know it isn't the creation timestamp for instance. > Cheers, > Michał > > > On 04/06/17 01:24, Jeyhun Karimov wrote: > > Hi Matthias, > > Thanks for comments. > > - why do you only consider get()

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; > > > > > > > I have one concern with regard to backward compatibility. You suggest > > to > > > > use RecrodContext as base interface for ProcessorContext. This will > > > > break compatibility. > > > > > > > > I think, we s

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

gt; > On Tue, 2017-02-28 at 08:59 +, Jeyhun Karimov wrote: > > Dear community, > > > > I'd like to start the vote for KIP-123: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6871 > > 4788 > > > > > > Cheers, > >

Re: Splitting tasks in streams?

PM Eno Thereska <eno.there...@gmail.com> wrote: > Hi Jeyhun, > > You make a good observation and I think a discussion/contribution around > this would be very much appreciated by the community. Are you thinking of a > KIP perhaps? > > Eno > > > On 28 Apr 201

Splitting tasks in streams?

Hi community, I have a question regarding with streams library. Currently, in kafka-streams we run the whole topology in one instance and there can be several topologies or tasks in a single node. However, there can be use-cases with very complex topologies with costly operators. So, when we

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

d the new > overlaod. > > And yes, we need to do one check -- but this happens when building the > topology. At runtime (I mean after KafkaStream#start() we don't need any > check as we will always use `ValueMapperWithKey`) > > > -Matthias > > > On 5/9/17 2:55 AM, Je

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

One correction: and in runtime (inside processor) we still have to check it is ValueMapper > or ValueMapperWithKey before wrapping it into the rich function. this will be in compile time in API level. Cheers, Jeyhun On Tue, May 9, 2017 at 11:55 AM Jeyhun Karimov <je.kari...@gma

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

chFunctions). > > > > This approach should not effect lambdas (or do I miss something?) and > > might be cleaner, as we could have one more top level interface > > `RichFunction` with methods `init()` and `close()` and also interfaces > > for `RichValueMapper` etc. (thu

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

hat about KStream-KTable joins? You don't have overlaods added for > > >> them. Why? (Even if I still hope that we don't need to add any new > > >> overloads) > > >> > > >> - Why do we need `AbstractRichFunction`? > > >> > &

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

s to get Lambdas for `withKey` interfaces too much because we have > already so many overlaods. On the other hand, I do see value in > supporting Lambdas for `withKey`. > > Depending on what we want to support, it might make sense to > include/exclude RichFunctions from this KIP -- and th

Re: Handling 2 to 3 Million Events before Kafka

Hi, With kafka you can increase overall throughput by increasing the number of nodes in a cluster. I had a similar issue, where we needed to ingest vast amounts of data to streaming system. In our case, kafka was a bottleneck, because of disk I/O. To solve it, we implemented (simple) distributed

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Also, i don't like the abstract classes: RichValueMapper, RichValueJoiner, > RichInitializer etc. Why can't they not just be interfaces? Generally we > should provide users with Intefaces to code against, not classes. > > Thanks, > Damian > > On Fri, 19 May 2017 at 00:50 Jeyhun Karim

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

do this even in parallel) -- we > don't want to slow you down :) But it make the discussion and code > review easier, if we separate both IMHO. > > > -Matthias > > > On 5/19/17 2:25 AM, Jeyhun Karimov wrote: > > Hi Damian, > > > > Thanks for your comments. I t

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

Noll <mich...@confluent.io> > wrote: > > > Thanks for your work and for driving this, Jeyhun! :-) > > > > -Michael > > > > > > On Tue, Apr 25, 2017 at 11:11 PM, Jeyhun Karimov <je.kari...@gmail.com> > > wrote: > > > > > Dear a

[DISCUSS]: KIP-159: Introducing Rich functions to Streams

Dear community, As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate KIP for rich functions (interfaces) [2]. I would like to get your comments. [1]

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

, so I'm asking all this to better > understand the rationale for init() and close(). > > Thanks, > Michał > > On 20/05/17 17:05, Jeyhun Karimov wrote: > > Dear community, > > As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate > KIP for rich func

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

>> function. Sounds more like the Processor/Transformer kind of thing >> semantically, rather than a function. >> >> The KIP says there are multiple use-cases for this but doesn't enumerate >> any - I think some examples would be useful, otherwise that section sounds >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

would allow us to drop "init()". This might even > allow to use Lambdas and we could keep the name RichFunction as we > preserve the nature of being a function. > > > -Matthias > > On 5/24/17 12:13 PM, Jeyhun Karimov wrote: > > Hi Michal, > > > > Thank

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

t/complete API, it might be a good idea. Any thoughts? > > > -Matthias > > > On 5/24/17 3:47 PM, Matthias J. Sax wrote: > > Jeyhun, > > > > I was just wondering if you did look into the key-deep-copy idea we > > discussed. I am curious to see what the impact might be. &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

te and also allows to register > punctuations. Both those features will not be available via withKey() > interfaces. > > -Matthias > > > On 5/27/17 1:25 PM, Jeyhun Karimov wrote: > > Hi Matthias, > > > > Thanks for your comments. > > > > I t

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

if we duplicate > functionality. > > For this reason, it seems to be better to got with the > `#valueMapper(ValueMapper mapper, RecordContext context)` approach. > > WDYT? > > > > -Matthias > > On 5/27/17 11:00 AM, Jeyhun Karimov wrote: > > Hi, > >

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

ut we can't do this for ValueTransformer (and similar). > > Btw: This reminds me about KIP-159: with regard to the RichFunction we > might need a supplier pattern, too. (I'll comment on the other thread, > too.) > > > -Matthias > > On 5/28/17 5:45 AM, Jeyhun Karimov wrote: > &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

> > > Overall, I like the KIP a lot! > > > -Matthias > > > On 5/16/17 7:03 AM, Jeyhun Karimov wrote: > > Hi, > > > > Thanks for your comments. > > > > I think supporting Lambdas for `withKey` and `AbstractRichFunction` > >> don't go togeth

[DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

Dear community, I want to share KIP-165 [1] based on issue KAFKA-4304 [2]. I would like to get your comments. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-165%3A+Extend+Interactive+Queries+for+return+latest+update+timestamp+per+key [2]

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

. Cheers, Jeyhun On Fri, Jun 2, 2017 at 2:27 AM Guozhang Wang <wangg...@gmail.com> wrote: > Does this KIP subsume this ticket as well? > https://issues.apache.org/jira/browse/KAFKA-4125 > > On Sat, May 20, 2017 at 9:05 AM, Jeyhun Karimov <je.kari...@gmail.com> >

Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

le new > method returning KeyContext not be sufficient? > - for backward compatibility, we will also need a new interface and > cannot just extend the existing one > > > > -Matthias > > On 5/29/17 4:55 PM, Jeyhun Karimov wrote: > > Dear community, > > > > I

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

6/6/17 5:15 PM, Guozhang Wang wrote: > > Thanks for the comprehensive summary! > > > > Personally I'd prefer the option of passing RecordContext as an > additional > > parameter into he overloaded function. But I'm also open to other > arguments > > if there are

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

formerCommon: > > > > I think changing the interface to extend from a new interface is not > binary > > compatible though source compatible, i.e. users still need to recompile > > their code though no need to make code changes. We may need to mention > that > > in the

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

like that would require subclassing RichFunction? That's a > bit of an inconvenience IMO. > > Cheers, > > Michal > > On 07/05/17 01:29, Jeyhun Karimov wrote: > > Hi, > > Thanks for comments. I extended PR and KIP to include rich functions. I > will still have to evalu

[DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Dear community, I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726 [3], KAFKA-3745 [4]. The related PR can be found at [5]. I would like to get your comments. [1]

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

gned. If the key type is a mutable object (eg. byte[]), > it can still be mutated. (eg. key[0] = 0). But I'm not really sure there's > much that can be done about that. > > Mathieu > > > On Mon, May 1, 2017 at 5:39 PM, Jeyhun Karimov <je.kari...@gmail.com> > wrote: > &

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

terface that > > would allow users to access record metadata (like timestamp, offset, > > partition etc) within DSL. This would be a similar concept. Thus, I am > > wondering, if it would make sense to enlarge the scope of this KIP by > > that? WDYT? > > > > &g

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

gt; > We do have a performance system test. Maybe it's worth for you to apply > > the deep copy strategy and run the test. It's very basic performance > > test only, but might give some insight. If you want to do this, look > > into folder "tests" for general test se

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

t; > I just want to add my voice that, I too, have wished for access to the > > record key during a mapValues or similar operation. > > > > On the other hand, the number of compile failures that would need to be > > fixed from this change is unfortunate. :-) But at least it

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

ly a little bit undecided what the better option might be. > >> Curious to hear what other think about this trade off. > >> > >> > >> -Matthias > >> > >> > >> On 6/1/17 6:13 PM, Jeyhun Karimov wrote: > >>> Hi Guozhang, >

Re: KIP-160 - Augment KStream.print() to allow users pass in extra parameters in the printed string

I agree with Matthias's comment. Constructing RecordContext with more metadata seems more feasible for me. Cheers, Jeyun On Mon, Jun 5, 2017 at 7:47 AM Matthias J. Sax wrote: > Not with the scope of the current discussion. > > So far, we discuss to add `RecordContext`,

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; lines of option 2 where we collapse the Rich and ValueWithKey etc > interfaces into 1 interface that has all of the arguments. I think we then > only need to add one additional overload for each operator? > > Thanks, > Damian > > On Wed, 13 Sep 2017 at 10:59 Jeyhun K

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

public RecordContext recordContext() { > return this.recordContext(); > > Can you explain a bit about the above implementation ? > >void commit () { > throw new UnsupportedOperationException("commit() is not supported in > this context"); > > Is the exc

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; Damian > > On Thu, 21 Sep 2017 at 15:23 Jeyhun Karimov <je.kari...@gmail.com> wrote: > > > Hi all, > > > > Thanks a lot for your comments. For the single interface (RichXXX and > > XXXWithKey) solution, I have already submitted a PR but probably it is &

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

;bbej...@gmail.com> > wrote: > > > > > > > +1 for me on collapsing the Rich and ValueWithKey interfaces > > > into 1 > > > > interface. > > > > > > > > Thanks, > > > > Bill > > > > > > > > On

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

Dear community, I updated the related KIP [1]. Please feel free to comment. Cheers, Jeyhun [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams On Fri, Sep 22, 2017 at 12:20 AM Jeyhun Karimov <je.kari...@gmail.com> wrote: >

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

.processor.internals to > o.a.k.processor. > > 6. I'm not clear why we want to move `commit()` from ProcessorContext to > RecordContext? Conceptually I think it would better staying in the > ProcessorContext. Do you find this not doable in the internal > implementations? > > &

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

lease feel free to comment on this. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=73637757 Cheers, Jeyhun On Fri, Jul 21, 2017 at 2:00 AM Jeyhun Karimov <je.kari...@gmail.com> wrote: > Hi Matthias, Damian, all, > > Thanks for your comments and sorry for su

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

r context not a record context > logically: when you call that function, it means we would commit the state > of the whole task up to this processed record, not only that single record > itself. > > > Guozhang > > On Mon, Oct 16, 2017 at 9:19 AM, Jeyhun Karimov &l

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

l I am not even sure if we should fix KAFKA-3907 at all. > Manual commits are something DSL users should not worry about -- and if > one really needs this, an advanced user can still insert a dummy > `transform` to request a commit from there. > > -Matthias > > > On 10/18/1

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

gt;>>>>>> Even then its difficult already: I never ran a topology with > caching > >>>>>>>> but I > >>>>>>>> am not even 100% sure what the record Context means behind > >>>>>>>> a materialized KTable

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

bout it at all, but only specify num.partitions. > >> > >> 3. Details: in Produced we do not have specs for specifying the > >> num.partitions or should we repartition or not. So it is still not > >> clear to > >> me how we would make use of that to achi

[VOTE] KIP-159: Introducing Rich functions to Streams

Dear community, It seems the discussion for KIP-159 [1] converged finally. I would like to initiate voting for the particular KIP. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams Cheers, Jeyhun

[DISCUSS] KIP-221: Repartition Topic Hints in Streams

Dear community, I would like to initiate discussion on KIP-221 [1] based on issue [2]. Please feel free to comment. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Repartition+Topic+Hints+in+Streams [2] https://issues.apache.org/jira/browse/KAFKA-6037 Cheers, Jeyhun

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

> > Maybe, we could use `through()` to create a topics in the back ground > > and picky back all to `Produced`. > > > > > > -Matthias > > > > On 11/4/17 4:16 PM, Ted Yu wrote: > > > API is given without much javadoc on the role / meaning of method >

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

ser don't want to manage topic manually, thus, it's still an > >> internal topic and Streams create the topic name automatically as for > >> all other internal topics. However, users gets some more control about > >> topic parameters like number of partitions (we should

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

7 10:19 PM, Jan Filipiak wrote: > >> -1 non binding > >> > >> I don't get the motivation. > >> In 80% of my DSL processors there is no such thing as a reasonable > >> RecordContext. > >> After a join the record I am processing belongs to at least 2 top

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

gt; wrote: > > > >> Hi Jeyhun, thanks, looks good. > >> Do we need to remove the line that says: > >> > >>- on-demand commit() feature > >> > >> Cheers, > >> Damian > >> > >> On Tue, 31 Oct 2017 at 23:07 J

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

p it that way until #6. In addition, the > >>>> RecordContext > >>>> fields (topic, offset, etc) are really orthogonal to the key-value > >>>> payloads > >>>> themselves, so I think separating them into this object is a > >>>> clean

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

re saying, but I don't agree that > > > but also we need a commit() method > > I would just not provide `commit()` at DSL level and close the > corresponding Jira as "not a problem" or similar. > > > -Matthias > > On 10/27/17 3:42 PM, Jeyhun Karimov wrote:

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

f `through()` and `to()`, can you add > >> the different behavior using different overloads? It's not clear from > >> the KIP what the semantics are. > >> > >> > >> -Matthias > >> > >> On 11/17/17 3:27 PM, Jeyhun Karimov wrote: > >>>

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

; On Mon, Oct 23, 2017 at 1:40 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > Fair point. This is a long discussion and I totally forgot that we > > discussed this. > > > > Seems I changed my opinion about including KAFKA-3907... > > > > Happ

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

` is for a specific record. > > To me, this does not seem to be a sound API design if we follow this path. > > > -Matthias > > > > On 10/26/17 10:41 PM, Jeyhun Karimov wrote: > > Hi, > > > > Thanks for your suggestions. > > > > I have some co

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

gt;> interested to drive this further. So we will just "reassign" it to them. >> >> Thanks for letting us know. >> >> >> -Matthias >> >> On 6/20/18 2:51 PM, Jeyhun Karimov wrote: >> > Hi Matthias, all, >> > >> > Curr

[jira] [Work started] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3839 started by Jeyhun Karimov. - > Handling null keys in KTable.grou

[jira] [Commented] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330865#comment-15330865 ] Jeyhun Karimov commented on KAFKA-3839: --- I can take this task as a starter. > Handling null k

[jira] [Assigned] (KAFKA-3836) KStreamReduce and KTableReduce should not pass nulls to Deserializers

[ https://issues.apache.org/jira/browse/KAFKA-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3836: - Assignee: Jeyhun Karimov > KStreamReduce and KTableReduce should not pass nu

[jira] [Assigned] (KAFKA-3825) Allow users to specify different types of state stores in Streams DSL

[ https://issues.apache.org/jira/browse/KAFKA-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3825: - Assignee: Jeyhun Karimov > Allow users to specify different types of state sto

[jira] [Commented] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336310#comment-15336310 ] Jeyhun Karimov commented on KAFKA-3839: --- It seems that groupby operator is gone from KTable

[jira] [Commented] (KAFKA-3825) Allow users to specify different types of state stores in Streams DSL

[ https://issues.apache.org/jira/browse/KAFKA-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350110#comment-15350110 ] Jeyhun Karimov commented on KAFKA-3825: --- [~guozhang] I am starting on this issue and I would to get

[jira] [Assigned] (KAFKA-3856) Move inner classes accessible only functions in TopologyBuilder out of public APIs

[ https://issues.apache.org/jira/browse/KAFKA-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3856: - Assignee: Jeyhun Karimov > Move inner classes accessible only functi

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833708#comment-15833708 ] Jeyhun Karimov commented on KAFKA-4144: --- [~elevy] I am not sure I got your point. Please correct me

[jira] [Assigned] (KAFKA-4613) Treat null-key records the same way for joins and aggreations

[ https://issues.apache.org/jira/browse/KAFKA-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4613: - Assignee: Jeyhun Karimov > Treat null-key records the same way for joins and aggreati

[jira] [Commented] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[ https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833607#comment-15833607 ] Jeyhun Karimov commented on KAFKA-4304: --- [~mjsax] Can we generalize this feature to {{top-k

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833131#comment-15833131 ] Jeyhun Karimov commented on KAFKA-4144: --- [~elevy] Shouldn't it be per partitioned stream/table? So

[jira] [Comment Edited] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833131#comment-15833131 ] Jeyhun Karimov edited comment on KAFKA-4144 at 1/22/17 3:05 PM: [~elevy

[jira] [Comment Edited] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833131#comment-15833131 ] Jeyhun Karimov edited comment on KAFKA-4144 at 1/22/17 3:06 PM: [~elevy

[jira] [Comment Edited] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836328#comment-15836328 ] Jeyhun Karimov edited comment on KAFKA-4144 at 1/24/17 8:58 PM: [~mjsax

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837350#comment-15837350 ] Jeyhun Karimov commented on KAFKA-4144: --- Thanks for clarification. I got the point. > Allow

[jira] [Assigned] (KAFKA-3835) Streams is creating two ProducerRecords for each send via RecordCollector

[ https://issues.apache.org/jira/browse/KAFKA-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3835: - Assignee: Jeyhun Karimov > Streams is creating two ProducerRecords for each send

[jira] [Assigned] (KAFKA-4144) Allow per stream/table timestamp extractor

[ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-4144: - Assignee: Jeyhun Karimov > Allow per stream/table timestamp extrac

  1   2   >