You can do a conditional branching by using KStream.branch(Predicate...).
You can then merge multiple streams using KStreamBuilder.merge(KStream...).
-Yasuhiro
On Mon, Jun 20, 2016 at 4:45 AM, Jeyhun Karimov
wrote:
> Hi Guozhang,
>
> Thank you for your reply. Yes, it is correct. Your solution i
It may not be ideal, but there is a way to prioritize particular topics. It
is to set the record timestamps to zero. This can be done by using a custom
TimestampExtractor. Kafka Streams tries to synchronize multiple streams
using the extracted timestamps. So, records with the timestamp 0 have
great
Yasuhiro Matsuda created KAFKA-3395:
---
Summary: prefix job id to internal topic names
Key: KAFKA-3395
URL: https://issues.apache.org/jira/browse/KAFKA-3395
Project: Kafka
Issue Type: Sub
Yasuhiro Matsuda created KAFKA-3271:
---
Summary: Notification upon unclean leader election
Key: KAFKA-3271
URL: https://issues.apache.org/jira/browse/KAFKA-3271
Project: Kafka
Issue Type
[
https://issues.apache.org/jira/browse/KAFKA-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-3262:
Description:
Current KafkaStreams polls records in the same thread as the data processing
Yasuhiro Matsuda created KAFKA-3262:
---
Summary: Make KafkaStreams debugging friendly
Key: KAFKA-3262
URL: https://issues.apache.org/jira/browse/KAFKA-3262
Project: Kafka
Issue Type: Sub
Yasuhiro Matsuda created KAFKA-3245:
---
Summary: need a way to specify the number of replicas for change
log topics
Key: KAFKA-3245
URL: https://issues.apache.org/jira/browse/KAFKA-3245
Project
Yasuhiro Matsuda created KAFKA-3153:
---
Summary: Serializer/Deserializer Registration and Type inference
Key: KAFKA-3153
URL: https://issues.apache.org/jira/browse/KAFKA-3153
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-3108:
Priority: Minor (was: Major)
> KStream custom StreamPartitioner for windowed
[
https://issues.apache.org/jira/browse/KAFKA-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-3108:
Summary: KStream custom StreamPartitioner for windowed key (was: KStream
custom
Yasuhiro Matsuda created KAFKA-3108:
---
Summary: KStream custom Partitioner for windowed key
Key: KAFKA-3108
URL: https://issues.apache.org/jira/browse/KAFKA-3108
Project: Kafka
Issue Type
Yasuhiro Matsuda created KAFKA-3060:
---
Summary: Refactor MeteredXXStore
Key: KAFKA-3060
URL: https://issues.apache.org/jira/browse/KAFKA-3060
Project: Kafka
Issue Type: Sub-task
Yasuhiro Matsuda created KAFKA-3016:
---
Summary: Add KStream-KStream window joins
Key: KAFKA-3016
URL: https://issues.apache.org/jira/browse/KAFKA-3016
Project: Kafka
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/KAFKA-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2984:
Summary: KTable should send old values along with new values to downstreams
(was: KTable
Yasuhiro Matsuda created KAFKA-2984:
---
Summary: KTable should send old values to downstreams
Key: KAFKA-2984
URL: https://issues.apache.org/jira/browse/KAFKA-2984
Project: Kafka
Issue Type
[
https://issues.apache.org/jira/browse/KAFKA-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2962:
Description: Stream-Table and Table-Table joins
> Add Simple Join
[
https://issues.apache.org/jira/browse/KAFKA-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2962:
Summary: Add Simple Join API (was: Add Join API)
> Add Simple Join
Yasuhiro Matsuda created KAFKA-2962:
---
Summary: Add Join API
Key: KAFKA-2962
URL: https://issues.apache.org/jira/browse/KAFKA-2962
Project: Kafka
Issue Type: Sub-task
Components
The group id is removed from the restore consumer config because the
restore consumer should not participate in the specified consumer group. I
don't know why it is failing.
On Fri, Nov 27, 2015 at 12:37 PM, Guozhang Wang wrote:
> Hello Bill,
>
> Thanks for reporting it, this is a valid issue, c
Yasuhiro Matsuda created KAFKA-2856:
---
Summary: add KTable
Key: KAFKA-2856
URL: https://issues.apache.org/jira/browse/KAFKA-2856
Project: Kafka
Issue Type: Sub-task
Components
[
https://issues.apache.org/jira/browse/KAFKA-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda reassigned KAFKA-2811:
---
Assignee: Yasuhiro Matsuda
> Add standby ta
Yasuhiro Matsuda created KAFKA-2811:
---
Summary: Add standby tasks
Key: KAFKA-2811
URL: https://issues.apache.org/jira/browse/KAFKA-2811
Project: Kafka
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/KAFKA-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda reassigned KAFKA-2763:
---
Assignee: Yasuhiro Matsuda
> Reduce stream task migrations and initialization co
Yasuhiro Matsuda created KAFKA-2763:
---
Summary: Reduce stream task migrations and initialization costs
Key: KAFKA-2763
URL: https://issues.apache.org/jira/browse/KAFKA-2763
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2727:
Description:
Currently each streaming task initializes the entire topology regardless of
Yasuhiro Matsuda created KAFKA-2727:
---
Summary: initialize only the part of the topology relevant to the
task
Key: KAFKA-2727
URL: https://issues.apache.org/jira/browse/KAFKA-2727
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2707:
Description: Currently KStream processor names are generated from
AtomicInteger static
Yasuhiro Matsuda created KAFKA-2707:
---
Summary: Make KStream processor names deteministic
Key: KAFKA-2707
URL: https://issues.apache.org/jira/browse/KAFKA-2707
Project: Kafka
Issue Type
[
https://issues.apache.org/jira/browse/KAFKA-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2707:
Summary: Make KStream processor names deterministic (was: Make KStream
processor names
Yasuhiro Matsuda created KAFKA-2706:
---
Summary: Make state stores first class citizens in the processor
DAG
Key: KAFKA-2706
URL: https://issues.apache.org/jira/browse/KAFKA-2706
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda resolved KAFKA-2592.
-
Resolution: Won't Fix
> Stop Writing the Change-log in store.put() / delete()
[
https://issues.apache.org/jira/browse/KAFKA-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2694:
Summary: Make a task id be a composite id of a topic group id and a
partition id (was
[
https://issues.apache.org/jira/browse/KAFKA-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda reassigned KAFKA-2694:
---
Assignee: Yasuhiro Matsuda
> Make a task id be a composite id of a topic group
[
https://issues.apache.org/jira/browse/KAFKA-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2694:
Issue Type: Sub-task (was: Task)
Parent: KAFKA-2590
> Make a task id b
Yasuhiro Matsuda created KAFKA-2694:
---
Summary: Make a task id be a composite id of a task group id and a
partition id
Key: KAFKA-2694
URL: https://issues.apache.org/jira/browse/KAFKA-2694
Project
t; The bottom line is that *different* topics will likely need to be
> partitioned differently.
>
> On October 14, 2015 at 12:57:37 PM, Yasuhiro Matsuda (
> yasuhiro.mats...@gmail.com) wrote:
>
> A partitioning scheme should be a cluster wide thing. Letting each sink
> have a d
A partitioning scheme should be a cluster wide thing. Letting each sink
have a different partitioning scheme does not make sense to me. A
partitioning scheme is not specific to a stream job, each task or a sink. I
think specifying it at sink level is more error prone.
If a user wants to customize
t; > >> >> > For 1, yes, when there is a transient leader change,
> it's
> > > >> > > guaranteed
> > > >> > > >> >> that a
> > > >> > > >> >> > prefix of the messages in a request will be co
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640958#comment-14640958
]
Yasuhiro Matsuda commented on KAFKA-2350:
-
Throwing exceptions makes sense
Jay, I understand that. Context can provide more information without
breaking the compatibility if needed. Also I am not sure ConsumerRecord is
the right abstraction of data for stream processing. After transformation
or join, what is the topic and the offset? It is odd to use ConsumerRecord.
We ca
The goal of this KIP is to provide a lightweight/embeddable streaming
framework, and allows Kafka users to start using stream processing easily. DSL
is not covered in this KIP. But, DSL is a very attractive option to have.
> In the proposed KafkaProcessor API, there is no interface like Collector
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639460#comment-14639460
]
Yasuhiro Matsuda commented on KAFKA-2350:
-
Suppose we are using auto assign
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637219#comment-14637219
]
Yasuhiro Matsuda commented on KAFKA-2350:
-
I think overloading subsc
n general, being able to get the offsets from the producer interface
> > does sound convenient.
> >
> > > We plan to have a KIP discussion meeting tomorrow at 11am PST. Perhaps
> > you
> > > can describe this KIP a bit then?
> >
> > Sure, happy to join
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634322#comment-14634322
]
Yasuhiro Matsuda commented on KAFKA-2350:
-
To me the state preservation is n
[
https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634229#comment-14634229
]
Yasuhiro Matsuda commented on KAFKA-2350:
-
Can we have TopicPartition rather
[
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632259#comment-14632259
]
Yasuhiro Matsuda edited comment on KAFKA-2260 at 7/18/15 4:1
[
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632259#comment-14632259
]
Yasuhiro Matsuda commented on KAFKA-2260:
-
Here is the outline of the variant
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2226:
Attachment: KAFKA-2226_2015-05-29_15:10:24.patch
> NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565538#comment-14565538
]
Yasuhiro Matsuda commented on KAFKA-2226:
-
Updated reviewboard h
/r/34734/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
/r/34734/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2226:
Attachment: KAFKA-2226_2015-05-29_15:04:35.patch
> NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565522#comment-14565522
]
Yasuhiro Matsuda commented on KAFKA-2226:
-
Updated reviewboard h
led" is a legitimate spelling in American
English.
- Yasuhiro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34734/#review85764
---
ms that we can add
> > the entry either when it's created for the first time or when it's removed
> > from the current list and needs to be added to a new list during reinsert.
> > In both cases, the list in the entry will be null and there is no need to
> > remove
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565133#comment-14565133
]
Yasuhiro Matsuda commented on KAFKA-2226:
-
Updated reviewboard h
; remove the entry from the list.
I will remove this.
- Yasuhiro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34734/#review85596
---------
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2226:
Attachment: KAFKA-2226_2015-05-29_10:49:34.patch
> NullPointerException
/r/34734/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34734/#review85665
---
On May 29, 2015, 12:19 a.m., Yasuhiro Matsuda wrote:
>
> --
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563965#comment-14563965
]
Yasuhiro Matsuda commented on KAFKA-2226:
-
Updated reviewboard h
/r/34734/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2226:
Attachment: KAFKA-2226_2015-05-28_17:18:55.patch
> NullPointerException
> > 134 is called and the entry is not removed since list is now null, (4) line
> > 133 is tested again and since list is now null, we quit the loop, (5) the
> > reinsert process adds the entry to a new list.
> >
> > At this point, a completed entry still exi
> > 134 is called and the entry is not removed since list is now null, (4) line
> > 133 is tested again and since list is now null, we quit the loop, (5) the
> > reinsert process adds the entry to a new list.
> >
> > At this point, a completed entry still exi
e-mail. To reply, visit:
https://reviews.apache.org/r/34734/#review85596
---
On May 27, 2015, 9 p.m., Yasuhiro Matsuda wrote:
>
> ---
> This is an automatically generated
apache.org/r/34734/#review85530
---
On May 27, 2015, 9 p.m., Yasuhiro Matsuda wrote:
>
> ---
> This is an automatically generated e-mail. To reply, vis
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2226:
Status: Patch Available (was: Open)
> NullPointerException in TestPurgatoryPerforma
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561724#comment-14561724
]
Yasuhiro Matsuda commented on KAFKA-2226:
-
Created reviewboard h
,
Yasuhiro Matsuda
[
https://issues.apache.org/jira/browse/KAFKA-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2226:
Attachment: KAFKA-2226.patch
> NullPointerException in TestPurgatoryPerforma
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561470#comment-14561470
]
Yasuhiro Matsuda commented on KAFKA-1989:
-
Thanks. I am looking into it. W
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33071/#review80342
---
Cool! It looks good to me.
- Yasuhiro Matsuda
On April 10, 2015
[
https://issues.apache.org/jira/browse/KAFKA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487830#comment-14487830
]
Yasuhiro Matsuda commented on KAFKA-2112:
-
Created reviewboard h
[
https://issues.apache.org/jira/browse/KAFKA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2112:
Attachment: KAFKA-2112.patch
> make overflowWheel volat
[
https://issues.apache.org/jira/browse/KAFKA-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2112:
Assignee: Yasuhiro Matsuda (was: Joel Koshy)
Status: Patch Available (was: Open
/33028/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
Yasuhiro Matsuda created KAFKA-2112:
---
Summary: make overflowWheel volatile
Key: KAFKA-2112
URL: https://issues.apache.org/jira/browse/KAFKA-2112
Project: Kafka
Issue Type: Bug
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486099#comment-14486099
]
Yasuhiro Matsuda commented on KAFKA-1989:
-
Updated reviewboard h
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-1989:
Attachment: KAFKA-1989_2015-04-08_14:29:51.patch
> New purgatory des
-CREATION
Diff: https://reviews.apache.org/r/31568/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-1989:
Attachment: KAFKA-1989_2015-04-08_13:27:59.patch
> New purgatory des
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485950#comment-14485950
]
Yasuhiro Matsuda commented on KAFKA-1989:
-
Updated reviewboard h
-CREATION
Diff: https://reviews.apache.org/r/31568/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
multiple executor threads in the pool.
I agree!
- Yasuhiro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31568/#review79408
-------
-
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31568/#review79269
---
On April 7, 2015, 9:59 p.m., Yasuhiro Matsuda wrote:
>
>
only if it returns
> > true, run onExpiration().
>
> Yasuhiro Matsuda wrote:
> This came from the original ExpiredOperationReaper.expireNext(). Also the
> comment on onExpiration says, "Call-back to execute when a delayed operation
> expires, but before completion.
single null compare (no method
call, no synchronization), which is the cheapest.
- Yasuhiro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31568/#review79257
------
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484170#comment-14484170
]
Yasuhiro Matsuda commented on KAFKA-1989:
-
Updated reviewboard h
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-1989:
Attachment: KAFKA-1989_2015-04-07_14:59:33.patch
> New purgatory des
-CREATION
Diff: https://reviews.apache.org/r/31568/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
n why that is better? It will trigger a lot more purge calls.
And the frequency of calls depends on how many keys each request has. When the
average number of keys per operation is large, it is possible to have a case
that the total number of watchers exceeds the threshold, but there are only a
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-1989:
Attachment: KAFKA-1989_2015-04-01_13:49:58.patch
> New purgatory des
[
https://issues.apache.org/jira/browse/KAFKA-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391422#comment-14391422
]
Yasuhiro Matsuda commented on KAFKA-1989:
-
Updated reviewboard h
-CREATION
Diff: https://reviews.apache.org/r/31568/diff/
Testing
---
Thanks,
Yasuhiro Matsuda
gt;
> >
> > Not quite sure what this is testing. It's not clear to me why the
> > sharedCounter won't increase after add. Perhaps, we can add some comments.
It is testing that reinserting the existing tasks doesn't change the task
count. I will add comments.
enerated e-mail. To reply, visit:
https://reviews.apache.org/r/31893/#review76975
---
On April 1, 2015, 12:31 a.m., Yasuhiro Matsuda wrote:
>
> ---
> This is an automatically generated e-mail.
[
https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yasuhiro Matsuda updated KAFKA-2013:
Attachment: KAFKA-2013_2015-03-31_17:30:56.patch
> benchmark test for the purgat
[
https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389764#comment-14389764
]
Yasuhiro Matsuda commented on KAFKA-2013:
-
Updated reviewboard h
1 - 100 of 168 matches
Mail list logo
