[jira] [Created] (KAFKA-6236) stream not picking data from topic - after rebalancing

[DHRUV BANSAL (JIRA)]

DHRUV BANSAL created KAFKA-6236:
---

 Summary: stream not picking data from topic - after rebalancing 
 Key: KAFKA-6236
 URL: https://issues.apache.org/jira/browse/KAFKA-6236
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
Reporter: DHRUV BANSAL
Priority: Critical


Kafka stream is not polling new messages from the topic. 

On enquiring the consumer group it is showing in rebalancing state

Command output:
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group 
 --describe

Warning: Consumer group 'name' is rebalancing.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Kafka 0.9.0.1 partitions shrink and expand frequently after restart the broker

[Json Tu]

someone can help to analysis it？

> 在 2017年11月10日，上午11:08，Json Tu  写道：
> 
> I‘m so sorry for my poor english.
> 
> what I really means is my broker machine is configured as 8 core 16G. but my 
> jvm configure is as below.
> java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC 
> -Djava.awt.headless=true -Xloggc:/xx/yy/kafkaServer-gc.log -verbose:gc 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=10M 
> -XX:+HeapDumpOnOutOfMemoryError.
> 
> we have 30+ clusters with this jvm configure, and are deployed on the machine 
> which configured as 8 core 16G. compare to other clusters, the current 
> cluster have more than 5 times partitions than other clusters.
> when we restart other clusters,  there is no such phenomenon.
> 
> may be some metrics or logs can leads to find root cause of this phenomenon.
> Looking forward to more suggestions.
> 
> 
>> 在 2017年11月9日，下午9:59，John Yost  写道：
>> 
>> I've seen this before and it was due to long GC pauses due in large part to
>> a memory heap > 8 GB.
>> 
>> --John
>> 
>> On Thu, Nov 9, 2017 at 8:17 AM, Json Tu  wrote:
>> 
>>> Hi,
>>>   we have a kafka cluster which is made of 6 brokers,  with 8 cpu and
>>> 16G memory on each broker’s machine, and we have about 1600 topics in the
>>> cluster，about 1700 partitions’ leader and 1600 partitions' replica on each
>>> broker.
>>>   when we restart a normal broke,  we find that there are 500+
>>> partitions shrink and expand frequently when restart the broker，
>>> there are many logs as below.
>>> 
>>>  [2017-11-09 17:05:51,173] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:22,047] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:28,634] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:44,658] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:06:47,611] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:07:19,703] INFO Partition [Yelp,5] on broker 4759726:
>>> Shrinking ISR for partition [Yelp,5] from 4759726,4759750 to 4759726
>>> (kafka.cluster.Partition)
>>> [2017-11-09 17:07:26,811] INFO Partition [Yelp,5] on broker 4759726:
>>> Expanding ISR for partition [Yelp,5] from 4759726 to 4759726,4759750
>>> (kafka.cluster.Partition)
>>> …
>>> 
>>> 
>>>   and repeat shrink and expand after 30 minutes which is the default
>>> value of leader.imbalance.check.interval.seconds, and at that time
>>> we can find the log of controller’s auto rebalance，which can leads some
>>> partition’s leader change to this restarted broker.
>>>   we have no shrink and expand when our cluster is running except when
>>> we restart it，so replica.fetch.thread.num is 1，and it seems enough.
>>> 
>>>   we can reproduce it at each restart，can someone give some suggestions.
>>> thanks before.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
> 

答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

[Hu Xi]

Jun,


Thanks for the comments. Do you think it'd better to add topic/partition tags 
for those metrics as well as keep the prefix? If those prefixes should really 
be removed, does this KIP need to do the same thing for `lag` ones?


发件人: Jun Rao 
发送时间: 2017年11月18日 8:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition 
lead metrics to KafkaConsumer

Hi, Charly,

Thanks for the input. It makes sense.

Hi, Hu,

Perhaps we can keep the per partition records-lead-min and records-lead-avg
as you had before, but just add the topic and the partition as the tags
instead of prefix of the metric name.

Thanks,

Jun



On Wed, Nov 15, 2017 at 4:58 AM, charly molter 
wrote:

> Hi Jun, Jiangle,
>
> I'd just like to clarify that KIP-225 seems to be using per partition
> metric the same way as KIP-223 seems to be doing.
>
> I believe avg and max are still necessary because the MetricsReporter
> doesn't work in a "push" manner and the "Value" measurableStat will only
> keep the last recorded entry.
> Therefore a MetricsReporter usually polls to grab a current view with Value
> this view is incomplete so it becomes not possible to compute the
> Max/Min/Avg.
> Max/Min/Avg uses SampledStats which work with a rolling window of samples
> and therefore periodic polling would work.
>
> This is why I believe it's necessary to keep Avg, Min and Max for these
> metrics as otherwise we wouldn't be able to recompute it in an external
> monitoring system.
>
> Am I wrong thinking this?
>
> Thanks,
> Charly
>
>
> On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao  wrote:
>
> > Hi, Charly,
> >
> > Thanks for KIP-225. Your proposal looks reasonable.
> >
> > Hi, Jiangjie,
> >
> > Do you think the approach that KIP-225 proposes is better for exposing
> the
> > per partition metric? Also, do we really need the per partition
> > record-lag-avg
> > and record-lag-max? It seems that an external monitoring system can
> always
> > derive that from the per partition record-lag.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 14, 2017 at 6:57 AM, charly molter 
> > wrote:
> >
> > > Hi Jun, Hu,
> > >
> > > I have KIP-225 open for adding tags to records-lag:
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=74686649
> > >
> > > I have a patch more or less ready so I could probably get the fix
> checked
> > > in (after the vote) and you could build on top of it. Otherwise we
> could
> > > merge both KIPs if you want but they do sound different to me.
> > >
> > > Thanks!
> > > Charly
> > >
> > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi  wrote:
> > >
> > > > Jun,
> > > >
> > > >
> > > > Let me double confirm with your comments:
> > > >
> > > > 1 remove partition-level records-lead-avg and records-lead-min since
> > they
> > > > both can be deduced by external monitoring system.
> > > >
> > > > 2 Tag partition-level records-lead with topic&partition info
> > > >
> > > >
> > > > If they are the case you expect, do we need to do the same thing for
> > > those
> > > > `lag` metrics? Seems partition-level records-lag metrics are not
> tagged
> > > > with topic&partition information  which might deserve a bug.
> > > >
> > > >
> > > > huxihx
> > > >
> > > >
> > > > 
> > > > 发件人: Jun Rao 
> > > > 发送时间: 2017年11月14日 12:44
> > > > 收件人: dev@kafka.apache.org
> > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition
> > > > lead metrics to KafkaConsumer
> > > >
> > > > Hi, Hu,
> > > >
> > > > Currently, records-lag-max is an attribute for the mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}".
> > > > So, it probably makes sense for records-lead-min to be an attribute
> > under
> > > > the same mbean.
> > > >
> > > > The partition level records-lead can probably be an attribute for the
> > > mbean
> > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > id="{client-id}",topic=topic1,partition=0,
> > > > where topic and partition are the tags. This matches the topic level
> > > mbeans
> > > > that we have in the consumer. I am not sure what the per partition
> > level
> > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> the
> > > lead
> > > > since the consumer is started? I am not sure we need those since an
> > > > external monitoring system can always derive them from records-lead.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi  wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for the feedback. Some things need to make sure. Currently,
> > > these
> > > > > new-added metrics follow the exact naming convention with those
> 'lag'
> > > > > counterparts, as shown below:
> > > > >
> > > > >
> > > > > Consumer-level metric:
> > > > >
> > > > > records-lag-max ==> records-lead-min
> > > > >
> > > > >
> > > > > Partition-level metrics:
> > > > >
> 

[GitHub] kafka pull request #4238: KAFKA-6234: Increased timeout value for lowWaterma...

[soenkeliebau]

GitHub user soenkeliebau opened a pull request:

https://github.com/apache/kafka/pull/4238

KAFKA-6234: Increased timeout value for lowWatermark response to avoid test 
failing occasionally

Increase timeout to fix flaky integration test testLogStartOffsetCheckpoint.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/soenkeliebau/kafka KAFKA-6234

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4238.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4238


commit 06eb877538ba01a2a285ef266592f72d124ebee9
Author: Soenke Liebau 
Date:   2017-11-20T06:26:03Z

KAFKA-6234: Increased timeout value for lowWatermark response to avoid test 
failing occasionally.




---

[GitHub] kafka pull request #4237: KAFKA-6207 : Include start of record when RecordIs...

[jawalesumit]

GitHub user jawalesumit opened a pull request:

https://github.com/apache/kafka/pull/4237

KAFKA-6207 : Include start of record when RecordIsTooLarge

When a message is too large to be sent (at 
org.apache.kafka.clients.producer.KafkaProducer#doSend), the 
RecordTooLargeException should carry the start of the record (for example, the 
first 1KB) so that the calling application can debug which message caused the 
error.

To resolve this, added functionality to log the first 1Kb of message, if 
RecordIsTooLarge exception is thrown.

Thanks,
Sumit Jawale

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jawalesumit/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4237.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4237


commit f2fbecfa87ac8152d5ae5d2c73fff3fcdc20e861
Author: jawalesumit 
Date:   2017-11-19T21:19:56Z

KAFKA-6207 : Include start of record when RecordIsTooLarge




---

Re: KIP-199 - Could offset management be part of the Connect REST API?

[Sönke Liebau]

Hi Gunnar,

I agree with you that the goal should be to implement offset management
capabilities in the rest interface. I believe Randall's intention with this
KIP was to lay the groundwork for accessing and changing offsets in Connect
and then build on that functionality once the changes necessary for that
groundwork are implemented.

That being said, I think the major changes for this to become possible will
be in the OffsetBackingStore and implementing classes, since as far as I
can tell these currently don't support retrieving all offsets for a
connector, but require a partition key to be passed in, which we do not
necessarily have available in all scenarios.
Adding in the rest endpoints and necessary functionality once these basic
changes are in place should be a fairly simple exercise and we could
consider adding them to the KIP to avoid unnecessary overhead in creating a
dedicated KIP for just the rest functionality.


Kind regards,
Sönke

On Fri, Nov 17, 2017 at 9:32 PM, Gunnar Morling 
wrote:

> Hi,
>
> I was reading KIP-199 [1] for adding a tool for Kafka Connect offset
> management. This would be a very useful functionality for users of the
> Debezium CDC connectors, too.
>
> What I was wondering, instead of having a separate tool for this, has it
> been considered to expose offset management via the REST API of Connect?
> There could be a resource /connectors//offsets for read
> and
> write access. In line with the current KIP, write access would only be
> allowed if the connector is stopped. Exposing this in the REST API might
> allow for a consistent experience with the other connector management
> functionalities.
>
> I'm not sure whether there are any guidelines on when to have some
> functionality as a separate tool vs. in the API, but I thought I'd bring up
> the idea and see what others think.
>
> Thanks,
>
> --Gunnar
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 199%3A+Add+Kafka+Connect+offset+tool
>



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

[GitHub] kafka pull request #4236: testing...

[jawalesumit]

Github user jawalesumit closed the pull request at:

https://github.com/apache/kafka/pull/4236


---

Jenkins build is back to normal : kafka-trunk-jdk7 #2983

[Apache Jenkins Server]

See 

Re: SessionKeySchema#segmentsToSearch()

[Ted Yu]

For `getMinSegmentGreaterThanEqualToTimestamp` , the email was indeed meant
for #4162.

Pardon.

On Sun, Nov 19, 2017 at 11:55 AM, Guozhang Wang  wrote:

> For `SessionKeySchema#segmentsToSearch`: for session store, multiple
> sessions may merge together when receiving late arrived records. When I
> looked at the code, it seems that we have merged the sessions during
> aggregations to effectively move the sessions between segments. So I'm not
> 100% certain why we still need to enforce MAX_VALUE. @Damian?
>
> For `getMinSegmentGreaterThanEqualToTimestamp` and `
> getMaxSegmentLessThanEqualToTimestamp`: I think you meant to leave it as a
> comment on https://github.com/apache/kafka/pull/4162? This is only added
> in
> that PR.
>
>
> Guozhang
>
>
> On Sat, Nov 18, 2017 at 11:16 AM, Ted Yu  wrote:
> >
> > This code:
> >
> > final Segment minSegment = segments
> > .getMinSegmentGreaterThanEqualToTimestamp(timeFrom);
> >
> > final Segment maxSegment = segments
> > .getMaxSegmentLessThanEqualToTimestamp(timeTo);
> >
> > Can be replaced with:
> >
> > final List searchSpace = keySchema.segmentsToSearch(
> > segments, from, to);
> >
> > The minSegment would be first in List and maxSegment would be last in
> List.
> >
> > On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu  wrote:
> >
> > > Hi,
> > > I was reading code for SessionKeySchema#segmentsToSearch() where:
> > >
> > > public List segmentsToSearch(final Segments segments,
> final
> > > long from, final long to) {
> > > return segments.segments(from, Long.MAX_VALUE);
> > >
> > > I wonder why the parameter to is ignored.
> > > WindowKeySchema#segmentsToSearch() passes parameter to
> > > to segments.segments().
> > >
> > > Cheers
> > >
>
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

[Guozhang Wang]

Jan: which approach are you referring to as "the approach that is on the
table would be perfect"?

Note that in today's PAPI layer we are already effectively exposing the
record context which has the issues that we have been discussing right now,
and its semantics is always referring to the "processing record" at hand.
More specifically, we can think of processing a record a bit different:

1) the record traversed the topology from source to sink, it may be
transformed into new object or even generate multiple new objects (think:
branch) along the traversal. And the record context is referring to this
processing record. Here the "lifetime" of the record lasts for the entire
topology traversal and any new records of this traversal is treated as
different transformed values of this record (this applies to join and
aggregations as well).

2) the record being processed is wiped out in the first operator after the
source, and NEW records are forwarded to downstream operators. I.e. each
record only lives between two adjacent operators, once it reached the new
operator it's lifetime has ended and new records are generated.

I think in the past we have talked about Streams under both context, and we
do not have a clear agreement. I agree that 2) is logically more
understandable for users as it does not leak any internal implementation
details (e.g. for stream-table joins, table record's traversal ends at the
join operator as it is only be materialized, while stream record's
traversal goes through the join operator to further down until sinks).
However if we are going to interpret following 2) above then even for
non-stateful operators we would not inherit record context. What we're
discussing now, seems to infer a third semantics:

3) a record would traverse "through" one-to-one (non-stateful) operators,
will "replicate" at one-to-many (non-stateful) operators (think: "mapValues"
 ) and will "end" at many-to-one (stateful) operators where NEW records
will be generated and forwarded to the downstream operators.

Just wanted to lay the ground for discussions so we are all on the same
page before chatting more.


Guozhang


On Sat, Nov 18, 2017 at 3:10 AM, Jan Filipiak 
wrote:

> Hi,
>
>  not an issue at all. IMO
> the approach that is on the table would be perfect
>
>
> On 18.11.2017 10:58, Jeyhun Karimov wrote:
>
>> Hi,
>>
>> I did not expected that Context will be this much an issue. Instead of
>> applying different semantics for different operators, I think we should
>> remove this feature completely.
>>
>>
>> Cheers,
>> Jeyhun
>> On Sat 18. Nov 2017 at 07:49, Jan Filipiak 
>> wrote:
>>
>> Yes, the mail said only join so I wanted to clarify.
>>>
>>>
>>>
>>> On 17.11.2017 19:05, Matthias J. Sax wrote:
>>>
 Yes. But I think an aggregation is an many-to-one operation, too.

 For the stripping off part: internally, we can just keep some record
 context, but just do not allow users to access it (because the context
 context does not make sense for them) by hiding the corresponding APIs.


 -Matthias

 On 11/16/17 10:05 PM, Guozhang Wang wrote:

> Matthias,
>
> For this idea, are your proposing that for any many-to-one mapping
> operations (for now only Join operators), we will strip off the record
> context in the resulted records and claim "we cannot infer its traced
> context anymore"?
>
>
> Guozhang
>
>
> On Thu, Nov 16, 2017 at 1:03 PM, Matthias J. Sax <
> matth...@confluent.io
> wrote:
>
> Any thoughts about my latest proposal?
>>
>> -Matthias
>>
>> On 11/10/17 10:02 PM, Jan Filipiak wrote:
>>
>>> Hi,
>>>
>>> i think this is the better way. Naming is always tricky Source is
>>>
>> kinda
>>>
 taken
>>> I had TopicBackedK[Source|Table] in mind
>>> but for the user its way better already IMHO
>>>
>>> Thank you for reconsideration
>>>
>>> Best Jan
>>>
>>>
>>> On 10.11.2017 22:48, Matthias J. Sax wrote:
>>>
 I was thinking about the source stream/table idea once more and it

>>> seems
>>>
 it would not be too hard to implement:

 We add two new classes

  SourceKStream extends KStream

 and

  SourceKTable extend KTable

 and return both from StreamsBuilder#stream and StreamsBuilder#table

 As both are sub-classes, this change is backward compatible. We

>>> change
>>>
 the return type for any single-record transform to this new types,

>>> too,
>>>
 and use KStream/KTable as return type for any multi-record operation.

 The new RecordContext API is added to both new classes. For old

>>> classes,
>>>
 we only implement KIP-149 to get access to the key.


 WDYT?


 -Matthias

 O

Re: SessionKeySchema#segmentsToSearch()

[Guozhang Wang]

For `SessionKeySchema#segmentsToSearch`: for session store, multiple
sessions may merge together when receiving late arrived records. When I
looked at the code, it seems that we have merged the sessions during
aggregations to effectively move the sessions between segments. So I'm not
100% certain why we still need to enforce MAX_VALUE. @Damian?

For `getMinSegmentGreaterThanEqualToTimestamp` and `
getMaxSegmentLessThanEqualToTimestamp`: I think you meant to leave it as a
comment on https://github.com/apache/kafka/pull/4162? This is only added in
that PR.


Guozhang


On Sat, Nov 18, 2017 at 11:16 AM, Ted Yu  wrote:
>
> This code:
>
> final Segment minSegment = segments
> .getMinSegmentGreaterThanEqualToTimestamp(timeFrom);
>
> final Segment maxSegment = segments
> .getMaxSegmentLessThanEqualToTimestamp(timeTo);
>
> Can be replaced with:
>
> final List searchSpace = keySchema.segmentsToSearch(
> segments, from, to);
>
> The minSegment would be first in List and maxSegment would be last in
List.
>
> On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu  wrote:
>
> > Hi,
> > I was reading code for SessionKeySchema#segmentsToSearch() where:
> >
> > public List segmentsToSearch(final Segments segments, final
> > long from, final long to) {
> > return segments.segments(from, Long.MAX_VALUE);
> >
> > I wonder why the parameter to is ignored.
> > WindowKeySchema#segmentsToSearch() passes parameter to
> > to segments.segments().
> >
> > Cheers
> >




--
-- Guozhang

[jira] [Created] (KAFKA-6235) Kafka should have an emergency retention setting for max disk used

[Antony Stubbs (JIRA)]

Antony Stubbs created KAFKA-6235:


 Summary: Kafka should have an emergency retention setting for max 
disk used
 Key: KAFKA-6235
 URL: https://issues.apache.org/jira/browse/KAFKA-6235
 Project: Kafka
  Issue Type: New Feature
Reporter: Antony Stubbs


Kafka should have an emergency retention setting for max disk used to prevent 
the broker running out of disk and partitions going off line. When this max is 
reached, Kafka could perhaps delete segments from the largest topics.. Would 
have to be used with care as current behaviour is to preserve data at the cost 
of availability. This would favour availability over data retention.

At the moment it's quite hard to reason about disk usage and Kafka as the max 
byte settings are all per partition, and the math can get complicated when you 
have lots of topics of different use cases and sizes..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Guozhang Wang]

Hello Jan,

I think I get your point about the cumbersome that CombinedKey would
introduce for serialization and tooling based on serdes. What I'm still
wondering is the underlying of joinPrefixFakers mapper: from your latest
comment it seems this mapper will be a one-time mapper: we use this to map
the original resulted KTable, V0> to KTable and
then that mapper can be thrown away and be forgotten. Is that true? My
original thought is that you propose to carry this mapper all the way along
the rest of the topology to "abstract" the underlying combined keys.

If it is the other way (i.e. the former approach), then the diagram of
these two approaches would be different: for the less intrusive approach we
would add one more step in this diagram to always do a mapping after the
"task perform join" block.

Also another minor comment on the internal topic: I think many readers may
not get the schema of this topic, so it is better to indicate that what
would be the key of this internal topic used for compaction, and what would
be used as the partition-key.

Guozhang


On Sat, Nov 18, 2017 at 2:30 PM, Jan Filipiak 
wrote:

> -> it think the relationships between the different used types, K0,K1,KO
> should be explains explicitly (all information is there implicitly, but
> one need to think hard to figure it out)
>
>
> I'm probably blind for this. can you help me here? how would you formulate
> this?
>
> Thanks,
>
> Jan
>
>
> On 16.11.2017 23:18, Matthias J. Sax wrote:
>
>> Hi,
>>
>> I am just catching up on this discussion and did re-read the KIP and
>> discussion thread.
>>
>> In contrast to you, I prefer the second approach with CombinedKey as
>> return type for the following reasons:
>>
>>   1) the oneToManyJoin() method had less parameter
>>   2) those parameters are easy to understand
>>   3) we hide implementation details (joinPrefixFaker, leftKeyExtractor,
>> and the return type KO leaks internal implementation details from my
>> point of view)
>>   4) user can get their own KO type by extending CombinedKey interface
>> (this would also address the nesting issue Trevor pointed out)
>>
>> That's unclear to me is, why you care about JSON serdes? What is the
>> problem with regard to prefix? It seems I am missing something here.
>>
>> I also don't understand the argument about "the user can stick with his
>> default serde or his standard way of serializing"? If we have
>> `CombinedKey` as output, the use just provide the serdes for both input
>> combined-key types individually, and we can reuse both internally to do
>> the rest. This seems to be a way simpler API. With the KO output type
>> approach, users need to write an entirely new serde for KO in contrast.
>>
>> Finally, @Jan, there are still some open comments you did not address
>> and the KIP wiki page needs some updates. Would be great if you could do
>> this.
>>
>> Can you also explicitly describe the data layout of the store that is
>> used to do the range scans?
>>
>> Additionally:
>>
>> -> some arrows in the algorithm diagram are missing
>> -> was are those XXX in the diagram
>> -> can you finish the "Step by Step" example
>> -> it think the relationships between the different used types, K0,K1,KO
>> should be explains explicitly (all information is there implicitly, but
>> one need to think hard to figure it out)
>>
>>
>> Last but not least:
>>
>> But noone is really interested.
>>>
>> Don't understand this statement...
>>
>>
>>
>> -Matthias
>>
>>
>> On 11/16/17 9:05 AM, Jan Filipiak wrote:
>>
>>> We are running this perfectly fine. for us the smaller table changes
>>> rather infrequent say. only a few times per day. The performance of the
>>> flush is way lower than the computing power you need to bring to the
>>> table to account for all the records beeing emmited after the one single
>>> update.
>>>
>>> On 16.11.2017 18:02, Trevor Huey wrote:
>>>
 Ah, I think I see the problem now. Thanks for the explanation. That is
 tricky. As you said, it seems the easiest solution would just be to
 flush the cache. I wonder how big of a performance hit that'd be...

 On Thu, Nov 16, 2017 at 9:07 AM Jan Filipiak >>> > wrote:

  Hi Trevor,

  I am leaning towards the less intrusive approach myself. Infact
  that is how we implemented our Internal API for this and how we
  run it in production.
  getting more voices towards this solution makes me really happy.
  The reason its a problem for Prefix and not for Range is the
  following. Imagine the intrusive approach. They key of the RockDB
  would be CombinedKey and the prefix scan would take an A, and
  the range scan would take an CombinedKey still. As you can
  see with the intrusive approach the keys are actually different
  types for different queries. With the less intrusive apporach we
  use the same type and rely on Serde Invariances. For us this w

Request to add to contributor's list...

[Alex Ott]

Hello

I have some free time & want to try to contribute to Kafka via resolving 
issues, and contributing to documentation.
Please give me access rights for work with issues in JIRA & wiki edit rights

My JIRA name is alexott

Thank you

-- 
With best wishes, Alex Ott
http://alexott.blogspot.com/http://alexott.net/
http://alexott-ru.blogspot.com/
Skype: alex.ott

[jira] [Created] (KAFKA-6234) Transient failure in kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpoint

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6234:


 Summary: Transient failure in 
kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpoint
 Key: KAFKA-6234
 URL: https://issues.apache.org/jira/browse/KAFKA-6234
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang


Saw this once: 
https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2669/testReport/junit/kafka.api/AdminClientIntegrationTest/testLogStartOffsetCheckpoint/

{code}
Stacktrace

java.util.concurrent.TimeoutException
at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:225)
at 
kafka.api.AdminClientIntegrationTest.$anonfun$testLogStartOffsetCheckpoint$3(AdminClientIntegrationTest.scala:762)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:858)
at 
kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpoint(AdminClientIntegrationTest.scala:756)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:844)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4232: KAFKA-6233 :Removed unnecessary null check

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4232


---

[jira] [Resolved] (KAFKA-6233) Removed unnecessary null check

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6233.
--
   Resolution: Fixed
Fix Version/s: 1.1.0

Issue resolved by pull request 4232
[https://github.com/apache/kafka/pull/4232]

> Removed unnecessary null check
> --
>
> Key: KAFKA-6233
> URL: https://issues.apache.org/jira/browse/KAFKA-6233
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.1.1, 0.10.2.1, 1.0.0, 0.11.0.2
>Reporter: sagar sukhadev chavan
>Priority: Trivial
> Fix For: 1.1.0
>
>
> Removed unnecessary null check
> if (encodingValue != null && encodingValue instanceof String)
> null instanceof String returns false hence replaced the check with
> if (encodingValue instanceof String)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4236: testing...

[jawalesumit]

GitHub user jawalesumit opened a pull request:

https://github.com/apache/kafka/pull/4236

testing...

*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jawalesumit/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4236.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4236


commit b9a862445f01dba1e47edb9350057c2a5d40d90e
Author: jawalesumit 
Date:   2017-11-19T10:28:39Z

testing...




---

[GitHub] kafka pull request #4235: KAFKA-6207 : Include start of record when RecordIs...

[jawalesumit]

Github user jawalesumit closed the pull request at:

https://github.com/apache/kafka/pull/4235


---

[jira] [Resolved] (KAFKA-6223) Please delete old releases from mirroring system

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6223.

Resolution: Fixed

> Please delete old releases from mirroring system
> 
>
> Key: KAFKA-6223
> URL: https://issues.apache.org/jira/browse/KAFKA-6223
> Project: Kafka
>  Issue Type: Bug
> Environment: https://dist.apache.org/repos/dist/release/kafka/
>Reporter: Sebb
>Assignee: Rajini Sivaram
>
> To reduce the load on the ASF mirrors, projects are required to delete old 
> releases [1]
> Please can you remove all non-current releases?
> It's unfair to expect the 3rd party mirrors to carry old releases.
> Note that older releases can still be linked from the download page, but such 
> links should use the archive server at:
> https://archive.apache.org/dist/kafka/
> A suggested process is:
> + Change the download page to use archive.a.o for old releases
> + Delete the corresponding directories from 
> {{https://dist.apache.org/repos/dist/release/kafka/}}
> e.g. {{svn delete https://dist.apache.org/repos/dist/release/kafka/0.8.0}}
> Thanks!
> [1] http://www.apache.org/dev/release.html#when-to-archive



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] Apache Kafka 0.11.0.2 Released

[Ismael Juma]

Thanks for running the release, Rajini! And with a single RC. :)

Ismael

On Fri, Nov 17, 2017 at 7:15 PM, Rajini Sivaram  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka
> 0.11.0.2.
>
>
> This is a bug fix release and it includes fixes and improvements from 16
> JIRAs,
> including a few critical bugs.
>
>
> All of the changes in this release can be found in the release notes:
>
>
> https://dist.apache.org/repos/dist/release/kafka/0.11.0.2/
> RELEASE_NOTES.html
>
>
>
> You can download the source release from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 2/kafka-0.11.0.2-src.tgz
>
>
> and binary releases from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 2/kafka_2.11-0.11.0.2.tgz
> (Scala 2.11)
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 2/kafka_2.12-0.11.0.2.tgz
> (Scala 2.12)
>
>
> 
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
>
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
>
> ** Building real-time streaming applications that transform or react to the
> streams of data.
>
>
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital
> One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target,
> The New York Times, Uber, Yelp, and Zalando, among others.
>
>
>
> A big thank you for the following 20 contributors to this release!
>
>
> Alex Good, Apurva Mehta, bartdevylder, Colin P. Mccabe, Damian Guy, Erkan
> Unal, Ewen Cheslack-Postava, Guozhang Wang, Hugo Louro, Jason Gustafson,
> Konstantine Karantasis, Manikumar Reddy, manjuapu, Mickael Maison, oleg,
> Onur Karaman, Rajini Sivaram, siva santhalingam, Xavier Léauté, Xin Li
>
>
> We welcome your help and feedback. For more information on how to
> report problems,
> and to get involved, visit the project website at http://kafka.apache.org/
>
>
> Thank you!
>
>
> Regards,
>
>
> Rajini
>

[GitHub] kafka pull request #4235: KAFKA-6207 : Include start of record when RecordIs...

[jawalesumit]

GitHub user jawalesumit opened a pull request:

https://github.com/apache/kafka/pull/4235

KAFKA-6207 : Include start of record when RecordIsTooLarge

When a message is too large to be sent (at 
org.apache.kafka.clients.producer.KafkaProducer#doSend), the 
RecordTooLargeException should carry the start of the record (for example, the 
first 1KB) so that the calling application can debug which message caused the 
error.
To resolve this, added functionality to log the first 1Kb of message, if 
RecordIsTooLarge exception is thrown.

Thanks,
Sumit Jawale

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jawalesumit/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4235.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4235


commit 6236c3f10076ab6a8e576f9de9caf8cb2ad3200b
Author: jawalesumit 
Date:   2017-11-19T09:57:41Z

KAFKA-6207 : Include start of record when RecordIsTooLarge




---

[GitHub] kafka pull request #4234: KAFKA-6207 : Include start of record when RecordIs...

[jawalesumit]

Github user jawalesumit closed the pull request at:

https://github.com/apache/kafka/pull/4234


---