You can figure out the names via
builder.build().describe()
The naming pattern is
--[repartition|changelog]
You will also need to create those topic with the correct number of
partitions (usually the number of input topic partitions).
With this information, you can pre-create the topics and Kaf
> I wanted to set *retention bytes* and change *cleanup policy* to *delete*
> to prevent the storage being full. I set following configs in kafka
> streams code:
Are you sure that you use Kafka Streams correctly? Seems like a miss
configuration to switch from compaction to deletion policy. Also n
No.
The cleanup interval configures when old state, that is not longer used,
will be deleted. This does not imply a TTL feature. It's about tasks
that got assigned to a different KafkaStreams instance.
State would only grow unbounded if your program increases the state
unbounded. For example, if
I would assume, that you refer to "commit markers". Each time you call
commitTransaction(), a special message called commit marker is written
to the log to indicate a successful transaction (there are also "abort
markers" if a transaction gets aborted).
Those markers "eat up" one offset, but wont'
That makes sense to me.
Please file a Jira. Thanks a lot for reporting this issue!
-Matthias
On 8/21/18 3:29 AM, Patrik Kleindl wrote:
> Hello
>
> Yesterday we had the following exception:
>
> Exception thrown when sending a message with key='null' and payload='...'
> to topic sometopic:: org
Thanks for reporting and for creating the ticket!
-Matthias
On 8/20/18 5:17 PM, Ted Yu wrote:
> I was able to reproduce what you saw with modification
> to StreamToTableJoinScalaIntegrationTestImplicitSerdes.scala
> I have logged KAFKA-7316 and am looking for a fix.
>
> FYI
>
> On Mon, Aug 20,
I cannot spot any obvious reasons.
As you consume from the result topic for verification, we should verify
that the latency spikes original on write and not on read: you might
want to have a look into Kafka Streams JMX metric to see if processing
latency spikes or throughput drops.
Also watch for
I think, at command line tool level, you need to use
--add-config log.cleanup.policy=[compact,delete]
ie, you need to add square bracket to mark the config as a list.
This is different to Java code for which you would use
props.put("log.cleanup.policy", "compact,delete");
The config should be
You might hit a RockDB issues as reported here:
https://issues.apache.org/jira/browse/KAFKA-6327
-Matthias
On 8/6/18 11:07 PM, Siva Ram wrote:
> Hi,
>
> We switched to a virtual machine (from physical node) and we are observing
> the following exception occurs and all our the stream application
Messages don't contain any information to what transaction they belong.
Note, that `transaction markers`, as special messages that only indicate
a commit or abort. There is no `transaction marker` _in_ a message.
-Matthias
On 8/2/18 12:04 PM, ma...@kafkatool.com wrote:
>
>
> This would
t. But it doesn't work either.
>
> Pulkit
>
> On Tue, Jul 31, 2018 at 9:22 PM, Matthias J. Sax
> wrote:
>
>> Is `delete.topic.enable` set to `true`? It's a broker configuration.
>>
>>
>> -Matthias
>>
>> On 7/31/18 8:57 AM, Pulkit
Is `delete.topic.enable` set to `true`? It's a broker configuration.
-Matthias
On 7/31/18 8:57 AM, Pulkit Manchanda wrote:
> HI All,
>
> I am want to create and delete Kafka topics on runtime in my Application.
> I followed few projects on GitHub like
> https://github.com/simplesteph/kafka-0.11
No. Transaction markers are not exposed.
Why would you need them? They are considered implementation details.
-Matthias
On 7/27/18 5:58 AM, ma...@kafkatool.com wrote:
> Is there any way for a KafkaConsumer to view/get the transactional
> marker messages?
>
> --
> Best regards,
> Mark
31y8>
> groups.google.com
> Posted 11/13/17 2:50 AM, 8 messages
>
>
>
> From: Matthias J. Sax
> Sent: Wednesday, July 18, 2018 10:20:02 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka Streams: Share state store across processors
&
Thanks Dong.
I am a little late, but +1, too.
- verified signatures
- build from sources
- run unit test suite
- run streams quickstart
Thanks for running the release!
-Matthias
On 7/18/18 10:24 AM, Dong Lin wrote:
> Thank you all for taking time to certify and vote for the release!
>
>
You can connect both stores to both processor for this.
-Matthias
On 7/17/18 11:12 PM, Druhin Sagar Goel wrote:
> Hi,
>
> I am new to the Kafka Streams framework. I have the following streams use
> case:
>
> State store A
> State store B
>
> Processor A
> Processor B
>
> State store A is onl
h cases (not a general case)
> can be improved.
> On Tue, Jul 17, 2018 at 1:48 AM Matthias J. Sax wrote:
>>
>> It is not possible to use a single message, because both messages may go
>> to different partitions and may be processed by different applications
>> instan
>
> Would this change make sense?
> On Mon, Jul 16, 2018 at 10:34 PM Matthias J. Sax
> wrote:
>>
>> Vasily,
>>
>> yes, it can happen. As you noticed, both messages might be processed on
>> different machines. Thus, Kafka Streams provides 'eventual cons
Vasily,
yes, it can happen. As you noticed, both messages might be processed on
different machines. Thus, Kafka Streams provides 'eventual consistency'
guarantees.
-Matthias
On 7/16/18 6:51 AM, Vasily Sulatskov wrote:
> Hi John,
>
> Thanks a lot for you explanation. It does make much more sens
York Times, Uber, Yelp, and Zalando, among others.
A big thank you for the following 32 contributors to this release!
Matthias J. Sax, Rajini Sivaram, Anna Povzner, Jason Gustafson, Ewen
Cheslack-Postava, Guozhang Wang, Dong Lin, huxi, John Roesler, Ismael
Juma, Jun Rao, Manikumar Reddy O, Max
> After executing the first command to start zookeeper, do i have to open a
> Terminal to run the Kafka Server?
Yes. What is the problem with this? "but run into problem in Step 2"
If you follow the quickstart, you download the binaries and start
multiple processes (Zookeeper, Kafka Broker, Ka
To understand joins better, you might want to check out:
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
KSQL uses the same join semantics as Kafka Streams.
-Matthias
On 7/11/18 8:01 AM, Guozhang Wang wrote:
> Hello Jonathan,
>
> At the very high-level, KSQL statements is c
d Kafka Streams on version 1.1.0 so
> I wonder why this issue is still occurring?
>
> -David
>
>> On Jul 10, 2018, at 9:38 AM, Matthias J. Sax wrote:
>>
>> Can it be, that you hit: https://issues.apache.org/jira/browse/KAFKA-6634
>>
>> -Matthias
>
Try to remove the space after the comma.
-Matthias
On 7/10/18 10:43 AM, Jayaraman, AshokKumar (CCI-Atlanta-CON) wrote:
> Hi,
>
> When we try to use the same (square brackets), the internal topics are
> failing to get created. Any suggestions?
>
> changelogConfig.put("cleanup.policy", "[compac
w (instead of two), which gives
> incorrect result.
>
> But when I look at the API document, I had the impression that it would
> behave like sending individual metrics.
>
> Hopefully I gave a better explanation of what I try to achieve.
>
> Thanks,
> Sicheng
>
>
Can it be, that you hit: https://issues.apache.org/jira/browse/KAFKA-6634
-Matthias
On 7/9/18 7:58 PM, David Chu wrote:
> I have a Kafka Streams application which is currently failing to start due to
> the following ProducerFencedException:
>
> "Caused by: org.apache.kafka.common.errors.Produce
Not sure what you mean by "does not reset recordContext".
Note, that the "contract" for `flatMapValues` is that the output records
inherit the timestamp of the input record.
Not sure what behavior you expect? Maybe you can elaborate?
-Matthias
On 7/9/18 7:27 PM, Sicheng Liu wrote:
> Hi,
>
> I
gt; Harsha
>
> On Mon, Jul 2nd, 2018 at 11:57 AM, Jun Rao <mailto:j...@confluent.io>> wrote:
>
> >
> >
> >
> > Hi, Matthias,
> >
> > Thanks for the running the release. Verified quickstart on scala 2.12
>> wrote:
>
> >
> >
> >
> > Hi, Matthias,
> >
> > Thanks for the running the release. Verified quickstart on scala 2.12
> > binary. +1
> >
> > Jun
> >
> > On Fri, Jun 29, 2018 at 10:
, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.
A big thank you for the following 30 contributors to this release!
Ewen Cheslack-Postava, Matthias J. Sax, Randall Hauch, Eno Thereska,
Damian Guy, Rajini Sivaram, Colin P. Mccabe, Kelvin Rutt, Kyle
Winkelman, Max Zheng
, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.
A big thank you for the following 26 contributors to this release!
Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
Adireddi, Jarek Rudzinski
votes
-1 votes
* No votes
-Matthias
On 7/2/18 9:54 AM, Matthias J. Sax wrote:
> This vote passes with 8 +1 votes (3 bindings) and no 0 or -1 votes.
>
> +1 votes PMC Members:
> * Jun
> * Rajini
> * Ismael
>
> Committers:
> * Matthias
>
> Community:
> * Va
This vote passes with 8 +1 votes (3 bindings) and no 0 or -1 votes.
+1 votes PMC Members:
* Jun
* Rajini
* Ismael
Committers:
* Matthias
Community:
* Vahid
* Manikumar
* Ted
* Harash
0 votes
* No votes
-1 votes
* No votes
Vote thread:
http://search-hadoop.com/m/Kafka/uyzND1yVsGqxIVN5?subj=+VO
se!
>
> Ismael
>
> On Fri, Jun 22, 2018 at 3:14 PM Matthias J. Sax
> mailto:matth...@confluent.io>> wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka
This vote passes with 5 +1 votes (3 bindings) and no 0 or -1 votes.
+1 votes PMC Members:
* Jason
* Guozhang
* Jun
Committers:
* Matthias
Community:
* Ted
0 votes
* No votes
-1 votes
* No votes
Vote thread:
http://search-hadoop.com/m/Kafka/uyzND1Wt6721GMzOE1?subj=+VOTE+0+10+2+2+RC1
I'll cont
+1
-Matthias
On 6/29/18 2:46 PM, Jun Rao wrote:
> Hi, Matthias,
>
> Thanks for running the release. Verified quickstart on scala 2.12 binary. +1
>
> Jun
>
> On Fri, Jun 22, 2018 at 6:43 PM, Matthias J. Sax <mailto:matth...@confluent.io>> wrote:
>
>
It's self-service. Unsubscribe with your old email and subscribe with
your new one: https://kafka.apache.org/contact
On 7/1/18 9:19 PM, Malik, Shibha (GE Renewable Energy, consultant) wrote:
> Hi,
>
> I want to change my email id for subscription. Is this the right group to
> email to ?
>
> Tha
Hi Dong,
it seems that the kafka-streams-quickstart artifacts are missing. Is it
just me or is the RC incomplete?
-Matthias
On 6/29/18 4:07 PM, Rajini Sivaram wrote:
> Hi Dong,
>
> +1 (binding)
>
> Verified binary using quick start, ran tests from source, checked
> release notes.
>
> Thanks
Hello Kafka users, developers and client-developers,
This is the second candidate for release of Apache Kafka 1.0.2.
This is a bug fix release addressing 27 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2
Release notes for the 1.0.2 release:
http://home.apache.org/~
:16 AM Matthias J. Sax
> wrote:
>> You cannot suppress those records, because both are required for
>> correctness. Note, that each event might go to a different instance in
>> the downstream aggregation -- that's why both records are required.
>>
>> Not sure
Hi,
You cannot suppress those records, because both are required for
correctness. Note, that each event might go to a different instance in
the downstream aggregation -- that's why both records are required.
Not sure what the problem for your business logic is. Note, that Kafka
Streams provides e
t;
>>>> Checked signatures.
>>>>
>>>> On Fri, Jun 22, 2018 at 11:42 AM, Vahid S Hashemian <
>>>> vahidhashem...@us.ibm.com> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> Built from s
Yes.
If you do this, the writes of both producers with interleave and there
are no ordering guarantees between records written by different producers.
-Matthias
On 6/27/18 11:26 AM, Malik, Shibha (GE Renewable Energy, consultant) wrote:
> Hi,
>
> Can multiple producers write to the same partit
gt;
>> On Jun 24, 2018, at 8:03 PM, Matthias J. Sax wrote:
>>
>> Michael,
>>
>> It depends on the semantics you want to get. About retries in general,
>> as long as a producer retries internally, you would not even notice.
>> Only after retries are exhaus
Michael,
It depends on the semantics you want to get. About retries in general,
as long as a producer retries internally, you would not even notice.
Only after retries are exhausted, an exception is thrown.
Kafka Streams allows you to implement a handler for this (cf
https://kafka.apache.org/11/d
Jozsef,
Your question is a little unclear to me.
> To detect lost messages
For what topology?
>> KTable inputTable = builder.table("inputTopic",
>> Consumed.with(...).filter(...));
The code you show contains a `filter()` that can remove record? Could
this be the issue?
It's also unclear to m
Sam,
Thanks for your email. This is a very interesting find. I did not double
check the code but your reasoning makes sense to me. Note, that caching
was _not_ introduced to reduce the writes to RocksDB, but to reduce the
write the the changelog topic and to reduce the number of records send
downs
Hello Kafka users, developers and client-developers,
This is the second candidate for release of Apache Kafka 0.10.2.2.
Note, that RC0 was created before the upgrade to Gradle 4.8.1 and thus,
we discarded it in favor of RC1 (without sending out a email for RC0).
This is a bug fix release closing
Hello Kafka users, developers and client-developers,
This is the first candidate for release of Apache Kafka 0.11.0.3.
This is a bug fix release closing 27 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.3
Release notes for the 0.11.0.3 release:
http://home.apache.
Hello Kafka users, developers and client-developers,
This is the first candidate for release of Apache Kafka 1.0.2.
This is a bug fix release closing 26 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2
Release notes for the 1.0.2 release:
http://home.apache.org/~mjsa
ivered, for example,
> backfill.). If you use event-time for retention, these old metrics could be
> dropped and won't be aggregated. If we use process-time, at least it will
> stay in state-store for some time for aggregation.
>
> On Thu, Jun 21, 2018 at 1:24 PM, Matthias J. Sa
I don't understand why event-time retention time cannot be used? Cannot
elaborate?
-Matthias
On 6/21/18 10:59 AM, Sicheng Liu wrote:
> Hi All,
>
> We have a use case that we aggregate some metrics with its event-time
> (timestamp on the metric itself) using the simplest tumbling window. The
> wi
Created a Jira for each:
- https://issues.apache.org/jira/browse/KAFKA-6998
- https://issues.apache.org/jira/browse/KAFKA-6999
-Matthias
On 5/11/18 10:06 AM, Guozhang Wang wrote:
> Hello Steven, thanks for pointing it out. I think both of the mentioned
> issues worth be improving:
>
> 1. The
Sorry for late reply.
> The source stream contains millions of messages produced over several
months.
What is the retention time of the output topic? If it is smaller than
the message timestamp (that I expect to be multiple month old), on write
the data would be delete quitckly, because it's olde
It is an know issue. You can increase the retention time for stored
offsets via configs thought.
There is already an open PR to fix this issue:
https://issues.apache.org/jira/browse/KAFKA-4682
-Matthias
On 6/1/18 2:00 AM, Dinesh Subramanian wrote:
> Hi M. Manna,
>
> Am planning to store outsid
As a workaround, you can specify the config just as a string directly:
props.put("default.deserialization.exception.handler", ...)
-Matthias
On 5/31/18 7:48 AM, Guozhang Wang wrote:
> Hello Sumit,
>
> We are going to release 2.0 soon which should contain this fix:
> https://issues.apache.org/
You can also pass in a custom partitioner instead of using the default
partitioner.
-Matthias
On 5/31/18 7:39 AM, Hans Jespersen wrote:
> Why don’t to just put the metadata in the header and leave the key null so it
> defaults to round robin?
>
> -hans
>
>> On May 31, 2018, at 6:54 AM, M. Man
About the docs:
Config `cleanup.policy` states:
> A string that is either "delete" or "compact".
> This string designates the retention policy to
> use on old log segments. The default policy> ("delete") will discard old
> segments when their
> retention time or size limit has been reached.> The
s some
> issues. If it is mentioned clearly then everyone will be aware. Could you
> please point in right direction about reading timestamp of log message? I
> will see about implementing that solution in code.
>
> On Tue, May 29, 2018 at 11:37 AM Matthias J. Sax
> wrote:
>
>>
Retention time is a lower bound for how long it is guaranteed that data
will be stored. This guarantee work "one way" only. There is no
guarantee when data will be deleted after the bound passed.
However, client side, you can always check the record timestamp and just
drop older data that is still
To subscribe, please follow instructions here:
https://kafka.apache.org/contact
On 5/24/18 8:16 PM, wrote:
> subscribe mail list
>
signature.asc
Description: OpenPGP digital signature
Your understanding is correct.
Unfortunately, a regression slipped into 1.0 release such that the
described optimization is not done... It's fixed in upcoming 2.0 release.
-Matthias
On 5/24/18 4:52 PM, Todd Hughes wrote:
> From what I've read, a Ktable directly sourced from a compacted topic is
Question cross-posted at SO:
https://stackoverflow.com/questions/50492491/customize-window-store-implementation-in-kstream-kstream-join
I did put an answer there.
-Matthias
On 5/23/18 8:56 AM, Edmondo Porcu wrote:
> We need to perform a Kstream - Kstream join with a very large window, where
> a
Assuming, that there are no duplicates in your input topic, as long as
no failure occurs, the consumer will read every message exactly-once by
default.
Only in case of failure, when the consumer "falls back" to an older
offset, you might see some duplicates. You will need to write custom
code to h
he best
> practice to write to same topic on different broker? Is there one? I should
> be able to get a list of brokers programmatically from zk path /brokers/ids
> ?
>
> On Sun, May 20, 2018, 3:21 PM Matthias J. Sax wrote:
>
>> You can register a callback for each sent
You can register a callback for each sent record to learn about
successful write or fail:
> producer.send(record, callback);
For replication, you don't need to send twice. If the replication factor
is configured broker side, the broker take care of replication
automatically.
You can also configu
f I try to run again my job (after the failure) without cleaning my
> environment (so on the same topics), my GlobalKTable is going to read the
> records as expected (so not-null records).
>
>
>
>
> 2018-05-18 0:04 GMT+02:00 Matthias J. Sax :
>
>>>> So, are you impl
at runtime by the same job,
> but it needs instead a pre-populated topic either some other application
> populating it?
> If so, I can not execute the next left join because globalKTable records
> must be consumed at runtime.
>
> Is there any possibility of using another method to ob
Should be ok to read the topic. I cannot spot any error in your
configs/program either.
However, I am not entirely sure, if I understand the problem correctly.
>> The problem is in the first run, where GlobalKTable reads null records (I
>> have a json serializer and it reads a record with null v
ht?
>
> Best,
> Claudia
>
> -Ursprüngliche Nachricht-
> Von: Matthias J. Sax
> Gesendet: Dienstag, 15. Mai 2018 22:58
> An: users@kafka.apache.org
> Betreff: Re: Exception stopps data processing (Kafka Streams)
>
> Claudia,
>
> I leader change is a
Thanks for reporting this @David!
@Guozhang: I actually think this is two different issues. This is also
exposed in a current PR:
https://github.com/apache/kafka/pull/4912/files#r188179256
I created https://issues.apache.org/jira/browse/KAFKA-6906 for this issue.
-Matthias
On 5/11/18 10:39 A
Claudia,
I leader change is a retryable error. What is your producer config for
`retries`? You might want to increase it such that the producer does not
throw the exception immediately but retries couple of times -- you might
also want to adjust `retry.backoff.ms` that sets the time to wait until
It depends on your version. The behavior is known and we put one
improvement into 1.1 release: https://github.com/apache/kafka/pull/4410
Thus, it's "by design" (for 1.0 and older) but we we want to improve it.
Cf: https://issues.apache.org/jira/browse/KAFKA-4969
-Matthias
On 5/13/18 7:52 PM, Lia
This might be interesting:
https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
Not sure what you mean by "streams" exactly but for brokers the number
of partitions are the dominating factor, not the number of topics.
-Matthias
On 5/11/18 2:01 AM, Sathi
`max.poll.records` only configures how many records are returned from
poll(). Internally, the consumer buffers a batch or records and only if
this batch is empty, if will do a new fetch request within poll().
-Matthias
On 5/10/18 10:46 PM, Mads Tandrup wrote:
> Hi
>
> I forgot to metion that I
You might want to look into Kafka Streams. In particular KTable and
Interactive Queries (IQ).
A `put` would be a write to the table source topic, while a `get` can be
implemented via IQ.
For subscribe to particular key, you would consume the whole source
topic and filter for the key you are inter
Hard to say. Might be a Spark issue though...
On 5/9/18 3:42 AM, Pena Quijada Alexander wrote:
> Hi all,
>
> We're facing some problems with ours Spark Streaming jobs, from yesterday we
> have got the following error into our logs when the jobs fail:
>
> java.lang.AssertionError: assertion fail
artRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.ju
You are hitting: https://issues.apache.org/jira/browse/KAFKA-6499
Was fixed in 1.1 release.
Thus, you can just ignore the checkpoint file. There should be no issue
with running on Kubernetes.
Also, if there is no store (independent of disk based or in-memory)
there will be no changelog topic.
As long as no repartitioning happens yes. If one of both sub-streams is
repartitioned, there is no guarantee.
-Matthias
On 4/20/18 4:31 PM, Botuck, Jacob (STL) - contr wrote:
> If I call branch on a kStream, and I send record A into the trunkStream
> followed by record B. If records A and B go
never be deleted and will stay in the KTable. Is this correct?
>
> Thanks,
> Mihaela Stoycheva
>
> On Thu, Apr 19, 2018 at 3:12 PM, Matthias J. Sax
> wrote:
>
>> Not sure what you mean by "old state that is not longer needed" ?
>>
>> key-
Not sure what you mean by "old state that is not longer needed" ?
key-value entries are kept forever, and there is no TTL. If you want to
delete something from the store, you can return `null` as aggregation
result though.
-Matthias
On 4/19/18 2:28 PM, adrien ruffie wrote:
> Hi Mihaela,
>
>
>
Make totally sense and is a known issue:
https://issues.apache.org/jira/browse/KAFKA-6711
Please follow up the ticket and/or PR -- not sure what the current
status is.
-Matthias
On 4/17/18 11:16 PM, David Chu wrote:
> I have a custom in-memory state store which I’d like to configure as a global
9, 2018 at 1:44 AM, Matthias J. Sax
> wrote:
>
>> It depend on the concrete program you write... Most likely it's 4, but
>> hard to say without more information.
>>
>> If you read all 4 topics as a single stream (i.e. pattern subscription)
>> it should be 4
em: we added 250 hits to remaining, but actually we had to add only
> 150 hits. We have to subtract previous count and it means we need to keep
> them all somewhere. That's where we hope access to KV store can help.
>
>
>
>
>
>
>
>
>
>
> On
imestamps why not, is it possible with windowing ?
>
>
> Thank Matthias
>
> Adrien
>
>
> De : Matthias J. Sax
> Envoyé : dimanche 8 avril 2018 23:04:24
> À : users@kafka.apache.org
> Objet : Re: join 2 topic streams --> to another topic
>
Check out this blog post that explain how the different joins work:
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
It's hard to give a general answer -- it depends on the context of your
application. Are keys unique? Do you want to get exactly one result or
should a single stoc
It depend on the concrete program you write... Most likely it's 4, but
hard to say without more information.
If you read all 4 topics as a single stream (i.e. pattern subscription)
it should be 4. If you repartitions data in your applications later, it
might be more thought.
-Matthias
On 4/8/1
>> ok, question then - is it possible to use state store with .aggregate()?
Not sure what you exactly mean by this. An aggregations always uses a
store; it's a stateful operation and cannot be computed without a store.
For TopN, if you get the hit-count as input, you can use a
`.aggregate()` oper
mbstones records will be deleted after "delete.retention.ms”, right?
> Which defaults to 24 hours - meaning that the internal topic should only
> contain data for 24 hours + window Size? Is this somehow right?
>
> Again, thank you very much for taking the time to answer these ques
Björn,
broker configs are default config but can be overwritten when a topic is
created. And this happens when Kafka Streams creates internal topics.
Thus, you need to change the setting Kafka Streams applies when creating
topics.
Also note: if cleanup.policy = compact, the setting of `log.retent
KGroupedStream and TimeWindowedKStream are only logical representations
at DSL level. They don't really "do" anything.
Thus, you can mimic them as follows:
builder.addStore(...)
in.selectKey().through(...).transform(..., "storeName").
selectKey() set's the new key for the grouping and the throug
Hi,
multiple answers to this question:
1) it depends of you send messages sync or async to the brokers.
Producers do buffer messages in-memory for more efficient writes to the
brokers. If messages are successfully sent to the brokers, you can get
an acknowledgment back the you can check on the pr
Check out the upgrade notes: https://kafka.apache.org/documentation/#upgrade
You should consider all notes for 0.11, 1.0 and 1.1 releases.
And yes, 1.1 is absolutely ready for production.
-Matthias
On 4/5/18 11:57 AM, Raghav wrote:
> Hi
>
> Are there anything that needs to be taken care for
Thanks Fred!
On 4/5/18 3:41 AM, Frederic Arno wrote:
> That's what I'm doing now, I override the config from within my tests.
>
> Reported here: https://issues.apache.org/jira/browse/KAFKA-6749
>
> Thanks, Fred
>
> On 04/04/2018 10:56 PM, Matthias J. Sax wro
ps://github.com/apache/kafka/pull/4826
>>
>> I will fill in JIRA Id once Frederic creates the JIRA.
>>
>> Cheers
>>
>> On Wed, Apr 4, 2018 at 4:29 PM, Matthias J. Sax
>> wrote:
>>
>>> Yes. That looks promising to me. Feel free to open an PR afte
mbie) {
> +if (!isZombie && transactionInFlight) {
> producer.abortTransaction();
> }
> transactionInFlight = false;
>
> On Wed, Apr 4, 2018 at 2:02 PM, Matthias J. Sax
> wro
Thanks for reporting this.
It's indeed a bug in Kafka Streams. It's related to this fix:
https://issues.apache.org/jira/browse/KAFKA-6634 -- the corresponding PR
introduces the issue.
Because, we initialize TX delayed, for your case, we never initialize TX
and thus aborting the TX fails.
Please
Just a side remark. As a workaround it should be fine to remove the
config. TopologyTestDriver will not produce duplicates anyway and is
also not suitable to test EOS.
-Matthias
On 4/4/18 1:26 PM, Guozhang Wang wrote:
> Thanks Frederic for reporting the issue, I think it is indeed a missing
> pie
What broker and Streams version do you use? Can you share the log/error
message/stacktrace?
-Matthias
On 4/3/18 10:58 AM, Ariel Debernardi wrote:
> Hi,
>
> I have a services with kafka with three brokers, and other with kafka
> streams.
>
> In kafka we have a topic "dep-stream", whit 12 partiti
501 - 600 of 1199 matches
Mail list logo
