Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Chia-Ping Tsai]

hi Ismael

> I think we should be thinking about
> https://issues.apache.org/jira/browse/KAFKA-6923 at the same time.

you are right. KAFKA-6923 and KAFKA-6161 are the keys to complete the 
Serializer and Deserializer. Let us add the default implementations first 
(KAFKA-6161). And then integrate the ExtendedSerializer/ExtendedDeSerializer to 
Serializer/Deserialize (KAFKA-6923)

--
Chia-Ping

On 2018/07/06 15:33:18, Ismael Juma  wrote: 
> Thanks for the KIP. I think we should be thinking about
> https://issues.apache.org/jira/browse/KAFKA-6923 at the same time.
> 
> Ismael
> 
> On Thu, 5 Jul 2018, 07:45 Chia-Ping Tsai,  wrote:
> 
> > hi all,
> >
> > I would like to start voting on "KIP-331 Add default implementation to
> > close() and configure() for Serializer, Deserializer and Serde"
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >
> > Cheers,
> > Chia-Ping
> >
> 

Re: [kafka-clients] [VOTE] 1.1.1 RC2

[Dong Lin]

Hi all,

We found a deadlock that affects metrics reporters that use synchronization
(https://issues.apache.org/jira/browse/KAFKA-7136). Since it may introduce
new issue in the bug fix release, I will create another RC that includes
the fix.

Thank you all for testing and voting for this release! Please help test RC3 if
you have time.

Regards,
Dong

On Thu, Jul 5, 2018 at 2:57 PM, Ismael Juma  wrote:

> We need to decide if KAFKA-7136 is a blocker or not. At the moment, it's
> not clear to me.
>
> Ismael
>
>
> On Thu, 5 Jul 2018, 14:41 Dong Lin,  wrote:
>
>> Thanks much for the information Ismael. It seems that the the
>> streams-quickstart files (link
>> )
>> are uploaded on July 5 whereas the binary files (link
>> )
>> are uploaded on Jun 28.
>>
>> Maybe this RC is good to go? If so, can we get another +1 from PMC?
>>
>> Thanks!
>> Dong
>>
>> On Thu, Jul 5, 2018 at 10:38 AM, Ismael Juma  wrote:
>>
> Thanks Dong. There are a couple of quickstart links below:
>>>
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>
>>> Ismael
>>>
>>> On Thu, Jul 5, 2018 at 10:32 AM Dong Lin  wrote:
>>>
 Hey everyone,

 Thanks for the vote! Sorry for late reply. I have been trying to
 understand
 whether streams-quickstart is required for release and how to fix the
 issue. I have limited availability to fix this issue in the past few
 days.

 It seems that all contents to be verified (e.g. source file, binary
 file,
 quick start) and verification steps (e.g. signature,
 https://kafka.apache.org/11/documentation/streams/quickstart) as shown
 in
 the first email are not affected directly by the
 kafka-streams-quickstart
 artifacts. So I think this RC can go ahead after
 kafka-streams-quickstart
 artifacts is uploaded. Hopefully community does not have to re-test and
 re-vote for this release again :) Please correct me if this is wrong.

 Hey Matthias,

 Thanks much for helping me with the steps to upload streams-quickstart!
 I
 have successfully run the "mvn deploy" command for the
 streams-quickstart.
 Do you know where to check if this has been successfully uploaded?

 Thanks,
 Dong


 On Thu, Jul 5, 2018 at 2:06 AM, Skrzypek, Jonathan <
 jonathan.skrzy...@gs.com
 > wrote:

 > Hi,
 >
 > Will this RC go ahead or should a RC3 be put together ?
 >
 >
 > -Original Message-
 > From: Matthias J. Sax [mailto:matth...@confluent.io]
 > Sent: 30 June 2018 06:13
 > To: Rajini Sivaram; Users
 > Cc: Dong Lin; dev; kafka-clients
 > Subject: Re: [kafka-clients] [VOTE] 1.1.1 RC2
 >
 > Hi Dong,
 >
 > it seems that the kafka-streams-quickstart artifacts are missing. Is
 it
 > just me or is the RC incomplete?
 >
 >
 > -Matthias
 >
 >
 > On 6/29/18 4:07 PM, Rajini Sivaram wrote:
 > > Hi Dong,
 > >
 > > +1 (binding)
 > >
 > > Verified binary using quick start, ran tests from source, checked
 > > release notes.
 > >
 > > Thanks for running the release!
 > >
 > > Regards,
 > >
 > > Rajini
 > >
 > > On Fri, Jun 29, 2018 at 11:11 PM, Jun Rao >>> > > > wrote:
 > >
 > > Hi, Dong,
 > >
 > > Thanks for running the release. Verified quickstart on scala
 2.12
 > > binary. +1
 > >
 > > Jun
 > >
 > > On Thu, Jun 28, 2018 at 6:12 PM, Dong Lin >>> > > > wrote:
 > >
 > > > Hello Kafka users, developers and client-developers,
 > > >
 > > > This is the second candidate for release of Apache Kafka
 1.1.1.
 > > >
 > > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch
 that
 > > was first
 > > > released with 1.1.0 about 3 months ago. We have fixed about 25
 > > issues since
 > > > that release. A few of the more significant fixes include:
 > > >
 > > > KAFKA-6925  > > - Fix
 > > > memory leak in StreamsMetricsThreadImpl
 > > > KAFKA-6937  > > - In-sync
 > > > replica delayed during fetch if replica throttle is exceeded
 > > > KAFKA-6917  > > - Process
 > > > txn completion asynchronously to avoid deadlock
 > > > KAFKA-6893 

[jira] [Resolved] (KAFKA-7136) PushHttpMetricsReporter may deadlock when processing metrics changes

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-7136.
--
   Resolution: Fixed
Fix Version/s: 1.1.1

> PushHttpMetricsReporter may deadlock when processing metrics changes
> 
>
> Key: KAFKA-7136
> URL: https://issues.apache.org/jira/browse/KAFKA-7136
> Project: Kafka
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Blocker
> Fix For: 2.0.0, 1.1.1
>
>
> We noticed a deadlock in {{PushHttpMetricsReporter}}. Locking for metrics was 
> changed under KAFKA-6765 to avoid {{NullPointerException}} in metrics 
> reporters due to concurrent read and updates. {{PushHttpMetricsReporter}} 
> requires a lock to process metrics registration that is invoked while holding 
> the sensor lock. It also reads metrics attempting to acquire sensor lock 
> while holding its lock (inverse order). This resulted in the deadlock below.
> {quote}Found one Java-level deadlock:
>  Java stack information for the threads listed above:
>  ===
>  "StreamThread-7":
>  at 
> org.apache.kafka.tools.PushHttpMetricsReporter.metricChange(PushHttpMetricsReporter.java:144)
>  - waiting to lock <0x000655a54310> (a java.lang.Object)
>  at org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:563)
>  - locked <0x000655a44a28> (a org.apache.kafka.common.metrics.Metrics)
>  at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:236)
>  - locked <0x00065629c170> (a org.apache.kafka.common.metrics.Sensor)
>  at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:217)
>  at 
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:1016)
>  at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:462)
>  at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
>  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
>  at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
>  at 
> org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:274)
>  at 
> org.apache.kafka.clients.consumer.internals.Fetcher.getAllTopicMetadata(Fetcher.java:254)
>  at 
> org.apache.kafka.clients.consumer.KafkaConsumer.listTopics(KafkaConsumer.java:1820)
>  at 
> org.apache.kafka.clients.consumer.KafkaConsumer.listTopics(KafkaConsumer.java:1798)
>  at 
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.refreshChangelogInfo(StoreChangelogReader.java:224)
>  at 
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.initialize(StoreChangelogReader.java:121)
>  at 
> org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:74)
>  at 
> org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:317)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:824)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
>  "pool-17-thread-1":
>  at 
> org.apache.kafka.common.metrics.KafkaMetric.measurableValue(KafkaMetric.java:82)
>  - waiting to lock <0x00065629c170> (a 
> org.apache.kafka.common.metrics.Sensor)
>  at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:58)
>  at 
> org.apache.kafka.tools.PushHttpMetricsReporter$HttpReporter.run(PushHttpMetricsReporter.java:177)
>  - locked <0x000655a54310> (a java.lang.Object)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> Found 1 deadlock.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-1.1-jdk7 #162

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-1.0-jdk7 #217

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #2794

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Use FetchRequest v8 and ListOffsetRequest v3 in

--
[...truncated 2.48 MB...]

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyNoOptimizationWhenNotExplicitlyAddedToConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyNoOptimizationWhenNotExplicitlyAddedToConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionWhenOptimizationConfigNotValueInRange STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionWhenOptimizationConfigNotValueInRange PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfKeySerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfKeySerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfBootstrapServersIsNotSet STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfBootstrapServersIsNotSet PASSED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustUseAdminClientConfigForRetries STARTED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustUseAdminClientConfigForRetries PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetMainConsumerConfigsWithMainConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetMainConsumerConfigsWithMainConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfApplicationIdIsNotSet STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfApplicationIdIsNotSet PASSED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustContainStreamPartitionAssignorConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustContainStreamPartitionAssignorConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectKeySerdeClassOnError STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectKeySerdeClassOnError PASSED

org.apache.kafka.streams.StreamsBuilderTest > 

[jira] [Resolved] (KAFKA-7130) EOFException after rolling log segment

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-7130.

Resolution: Duplicate

> EOFException after rolling log segment
> --
>
> Key: KAFKA-7130
> URL: https://issues.apache.org/jira/browse/KAFKA-7130
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 1.1.0
>Reporter: Karsten Schnitter
>Priority: Major
> Attachments: dump-001311940075.index.bz2, 
> dump-001311940075.log.bz2
>
>
> When rolling a log segment one of our Kafka cluster got an immediate read 
> error on the same partition. This lead to a flood of log messages containing 
> the corresponding stacktraces. Data was still appended to the partition but 
> consumers were unable to read from that partition. Reason for the exception 
> is unclear.
> {noformat}
> [2018-07-02 23:53:32,732] INFO [Log partition=ingestion-3, 
> dir=/var/vcap/store/kafka] Rolled new log segment at offset 971865991 in 1 
> ms. (kafka.log.Log)
> [2018-07-02 23:53:32,739] INFO [ProducerStateManager partition=ingestion-3] 
> Writing producer snapshot at offset 971865991 (kafka.log.ProducerStateManager)
> [2018-07-02 23:53:32,739] INFO [Log partition=ingestion-3, 
> dir=/var/vcap/store/kafka] Rolled new log segment at offset 971865991 in 1 
> ms. (kafka.log.Log)
> [2018-07-02 23:53:32,750] ERROR [ReplicaManager broker=1] Error processing 
> fetch operation on partition ingestion-3, offset 971865977 
> (kafka.server.ReplicaManager)
> Caused by: java.io.EOFException: Failed to read `log header` from file 
> channel `sun.nio.ch.FileChannelImpl@2e0e8810`. Expected to read 17 bytes, but 
> reached end of file after reading 0 bytes. Started read from position 
> 2147483643.
> {noformat}
> We mitigated the issue by stopping the affected node and deleting the 
> corresponding directory. Once the partition was recreated for the replica (we 
> use replication-factor 2) the other replica experienced the same problem. We 
> mitigated likewise.
> To us it is unclear, what caused this issue. Can you help us in finding the 
> root cause of this problem?
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

[Guozhang Wang]

I think I agree with Matthias for having dedicated APIs for windowed
operation final output scenario, PLUS separating the window close which the
"final output" would rely on, from the window retention time itself
(admittedly it would make this KIP effort larger, but if we believe we need
to do this separation anyways we could just do it now).

And then we can have the `KTable#suppress()` for intermediate-suppression
only, not for late-record-suppression, until we've seen that becomes a
common feature request because our current design still allows to be
extended for that purpose.


Guozhang

On Wed, Jul 4, 2018 at 12:53 PM, Matthias J. Sax 
wrote:

> Thanks for the discussion. I am just catching up.
>
> In general, I think we have different uses cases and non-windowed and
> windowed is quite different. For the non-windowed case, suppress() has
> no (useful) close or retention time, no final semantics, and also no
> business logic impact.
>
> On the other hand, for windowed aggregations, close time and final
> result do have a meaning. IMHO, `close()` is part of business logic
> while retention time is not. Also, suppression of intermediate result is
> not a business rule and there might be use case for which either "early
> intermediate" (before window end time) are suppressed only, or all
> intermediates are suppressed (maybe also something in the middle, ie,
> just reduce the load of intermediate updates). Thus, window-suppression
> is much richer.
>
> IMHO, a generic `suppress()` operator that can be inserted into the data
> flow at any point is useful. Maybe we should keep is as generic as
> possible. However, it might be difficult to use with regard to
> windowing, as the mental effort to use it is high.
>
> With regard to Guozhang's comment:
>
> > we will actually
> > process data as old as 30 days as well, while most of the late updates
> > beyond 5 minutes would be discarded anyways.
>
> If we use `suppress()` as a standalone operator, this is correct and
> intended IMHO. To address the issue if the behavior is unwanted, I would
> suggest to add a "suppress option" directly to
> `count()/reduce()/aggregate()` window operator similar to
> `Materialized`. This would be an "embedded suppress" and avoid the
> issue. It would also address the issue about mental effort for "single
> final window result" use case.
>
> I also think that a shorter close-time than retention time is useful for
> window aggregation. If we add close() to the window definition and
> until() to `Materialized`, we can separate both correctly IMHO.
>
> About setting `close = min(close,retention)` I am not sure. We might
> rather throw an exception than reducing the close time automatically.
> Otherwise, I see many user question about "I set close to X but it does
> not get updated for some data that is with delay of X".
>
> The tricky question might be to design the API in a backward compatible
> way though.
>
>
>
> -Matthias
>
> On 7/3/18 5:38 AM, John Roesler wrote:
> > Hi Guozhang,
> >
> > I see. It seems like if we want to decouple 1) and 2), we need to alter
> the
> > definition of the window. Do you think it would close the gap if we
> added a
> > "window close" time to the window definition?
> >
> > Such as:
> >
> > builder.stream("input")
> > .groupByKey()
> > .windowedBy(
> >   TimeWindows
> > .of(60_000)
> > .closeAfter(10 * 60)
> > .until(30L * 24 * 60 * 60 * 1000)
> > )
> > .count()
> > .suppress(Suppression.finalResultsOnly());
> >
> > Possibly called "finalResultsAtWindowClose" or something?
> >
> > Thanks,
> > -John
> >
> > On Mon, Jul 2, 2018 at 6:50 PM Guozhang Wang  wrote:
> >
> >> Hey John,
> >>
> >> Obviously I'm too lazy on email replying diligence compared with you :)
> >> Will try to reply them separately:
> >>
> >>
> >> 
> -
> >>
> >> To reply your email on "Mon, Jul 2, 2018 at 8:23 AM":
> >>
> >> I'm aware of this use case, but again, the concern is that, in this
> setting
> >> in order to let the window be queryable for 30 days, we will actually
> >> process data as old as 30 days as well, while most of the late updates
> >> beyond 5 minutes would be discarded anyways. Personally I think for the
> >> final update scenario, the ideal situation users would want is that "do
> not
> >> process any data that is less than 5 minutes, and of course no update
> >> records to the downstream later than 5 minutes either; but retain the
> >> window to be queryable for 30 days". And by doing that the final window
> >> snapshot would also be aligned with the update stream as well. In other
> >> words, among these three periods:
> >>
> >> 1) the retention length of the window / table.
> >> 2) the late records acceptance for updating the window.
> >> 3) the late records update to be sent downstream.
> >>
> >> Final update use cases would naturally want 2) = 3), while 1) may be
> >> different and larger, while what we provide now is 

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jun Rao]

Hi, Jason,

Thanks for the KIP. Looks good overall. Just a few minor comments below.

1. "As the consumer is fetching from a partition, it will keep a small
cache of the recent epochs that were fetched for each partition. " Do we
need to cache more than one leader epoch? Also, during consumer failover,
initially, only the the last epoch will be available.

2. "This KIP has implications for the consumer's offset reset policy, which
defines what the consumer should do if its fetch offset becomes out of
range. With this KIP, the only case in which this is possible is if the
consumer fetches from an offset earlier than the log start offset. ". If
the fetch epoch matches that in the leader, but the offset is larger than
the leader's HW, should we still treat it as offset out of range?

3. "We propose in this KIP to change the behavior for both the "earliest"
and "latest" reset modes to do this automatically as long as the message
format supports lookup by leader epoch.  ". It will probably be useful to
indicate to the user that a reset has happened. So, it's probably useful to
at least log this in the client.

4. "If the user ignores the exception, we will continue fetching from the
current offset, but we will drop the last fetched offset metadata from the
new FetchRequest so that we do not get the same log truncation error." Is
it better to do this or keep failing?

5. LogTruncationException:  Should we add an error code for that?

6. "We have added fields for the leader epoch and the timestamp." It seems
that we only added the leader epoch?

Jun


On Mon, Jun 25, 2018 at 9:17 AM, Jason Gustafson  wrote:

> Hey All,
>
> I wrote up a KIP to handle one more edge case in the replication protocol
> and to support better handling of truncation in the consumer when unclean
> leader election is enabled. Let me know what you think.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 320%3A+Allow+fetchers+to+detect+and+handle+log+truncation
>
> Thanks to Anna Povzner and Dong Lin for initial feedback.
>
> Thanks,
> Jason
>

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

[Florian Hussonnois]

Hi,

The option #3 seems to be a good alternative and I find the API more
elegant (thanks John).

But, we still have the need to overload some methods either because they do
not accept an action instance or because they are translated to multiple
processors.

For example, this is the case for methods branch() and merge(). We could
introduce a new interface Named (or maybe a different name ?) with a method
name(). All action interfaces could extend this one to implement the option
3).
This would result by having the following overloads  :

Stream merge(final Named name, final KStream stream);
KStream[] branch(final Named name, final Predicate... predicates)

N.B : The list above is  not exhaustive

-
user's code will become :

KStream stream = builder.stream("test");
KStream[] branches =
stream.branch(Named.with("BRANCH-STREAM-ON-VALUE"),
Predicate.named("STREAM-PAIR-VALUE", (k, v) -> v % 2 == 0),
Predicate.named("STREAM-IMPAIR-VALUE", (k, v) -> v % 2 !=
0));

branches[0].to("pair");
branches[1].to("impair");
-

This is a mix of the options 3) and 1)

Le ven. 6 juil. 2018 à 22:58, Guozhang Wang  a écrit :

> Hi folks, just to summarize the options we have so far:
>
> 1) Add a new "as" for KTable / KStream, plus adding new fields for
> operators-returns-void control objects (the current wiki's proposal).
>
> Pros: no more overloads.
> Cons: a bit departing with the current high-level API design of the DSL,
> plus, the inconsistency between operators-returns-void and
> operators-not-return-voids.
>
> 2) Add overloaded functions for all operators, that accepts a new control
> object "Described".
>
> Pros: consistent with current APIs.
> Cons: lots of overloaded functions to add.
>
> 3) Add another default function in the interface (thank you J8!) as John
> proposed.
>
> Pros: no overloaded functions, no "Described".
> Cons: do we lose lambda functions really (seems not if we provide a "named"
> for each func)? Plus "Described" may be more extensible than a single
> `String`.
>
>
> My principle of considering which one is better depends primarily on "how
> to make advanced users easily use the additional API, while keeping it
> hidden from normal users who do not care at all". For that purpose I think
> 3) > 1) > 2).
>
> One caveat though, is that changing the interface would not be
> binary-compatible though source-compatible, right? I.e. users need to
> recompile their code though no changes needed.
>
>
>
> Another note: for 3), if we really want to keep extensibility of Described
> we could do sth. like:
>
> -
>
> public interface Predicate {
> // existing method
> boolean test(final K key, final V value);
>
> // new default method adds the ability to name the predicate
> default Described described() {
> return new Described(null);
> }
> }
>
> --
>
> where user's code becomes:
>
> stream.filter(named("key", (k, v) -> true));   // note `named` now just
> sets a Described("key") in "described()".
>
> stream.filter(described(Described.as("key", /* any other fancy parameters
> in the future*/), (k, v) -> true));
> --
>
>
> I feel it is not much likely that we'd need to extend it further in the
> future, so just a `String` would be good enough. But just listing all
> possibilities here.
>
>
>
> Guozhang
>
>
>
>
>
>
> On Fri, Jul 6, 2018 at 8:19 AM, John Roesler  wrote:
>
> > Hi Florian,
> >
> > Sorry I'm late to the party, but I missed the message originally.
> >
> > Regarding the names, it's probably a good idea to stick to the same
> > character set we're currently using: letters, numbers, and hyphens. The
> > names are used in Kafka topics, files and folders, and RocksDB databases,
> > and we also need them to work with the file systems of Windows, Linux,
> and
> > MacOS. My opinion is that with a situation like that, it's better to be
> > conservative. It might also be a good idea to impose an upper limit on
> name
> > length to avoid running afoul of any of those systems.
> >
> > ---
> >
> > It seems like there's a small debate between 1) adding a new method to
> > KStream (and maybe KTable) to modify its name after the fact, or 2)
> > piggy-backing on the config objects where they exist and adding one where
> > they don't. To me, #2 is the better alternative even though it produces
> > more overloads and may be a bit awkward in places.
> >
> > The reason is simply that #1 is a high-level departure from the
> > graph-building paradigm we're using in the DSL. Consider:
> >
> > Graph.node1(config).node2(config)
> >
> > vs
> >
> > Graph.node1().config().node2().config()
> >
> > We could have done either, but we picked the former. I think it's
> probably
> > a good goal to try and stick to it so that developers can develop and
> rely
> > on their instincts for how the DSL will behave.
> >
> > I do want to present one alternative to adding new config objects: we can
> > just add a 

Re: [DISCUSS] KIP-326: Schedulable KTable as Graph source

[Guozhang Wang]

That's a lot of email exchanges for me to catch up :)

My original proposed alternative solution is indeed relying on
pre-aggregate before sending to the single-partition topic, so that the
traffic on that single-partition topic would not be huge (I called it
partial-aggregate but the intent was the same).

What I was thinking is that, given such a scenario could be common, if
we've decided to go down this route should we provide a new API that wrap's
John's proposed topology (right now with KIP-328 users still need to
leverage this trick manually):


--

final KStream siteEvents = builder.stream("/site-events");

final KStream keyedByPartition = siteEvents.transform(/*
generate KeyValue(key, 1) for the pre-aggregate*/);

final KTable countsByPartition =
keyedByPartition.groupByKey().count();   /* pre-aggregate */

final KGroupedTable singlePartition =
countsByPartition.groupBy((key, value) -> new KeyValue<>("ALL", value));
 /* sent the suppressed pre-aggregate values to the single partition topic
*/

final KTable totalCount = singlePartition.reduce((l, r) -> l +
r, (l, r) -> l - r);   /* read from the single partition topic, do reduce
on the data*/

--

Note that if we wrap them all into a new operator, users would need to
provide two functions, for the aggregate and for the final "reduce" (in my
previous email I called it merger function, but for the same intent).



Guozhang



On Thu, Jul 5, 2018 at 3:38 PM, John Roesler  wrote:

> Ok, I didn't get quite as far as I hoped, and several things are far from
> ready, but here's what I have so far:
> https://github.com/apache/kafka/pull/5337
>
> The "unit" test works, and is a good example of how you should expect it to
> behave:
> https://github.com/apache/kafka/pull/5337/files#diff-
> 2fdec52b9cc3d0e564f0c12a199bed77
>
> I have one working integration test, but it's slow going getting the timing
> right, so no promises of any kind ;)
>
> Let me know what you think!
>
> Thanks,
> -John
>
> On Thu, Jul 5, 2018 at 8:39 AM John Roesler  wrote:
>
> > Hey Flávio,
> >
> > Thanks! I haven't got anything usable yet, but I'm working on it now. I'm
> > hoping to push up my branch by the end of the day.
> >
> > I don't know if you've seen it but Streams actually already has something
> > like this, in the form of caching on materialized stores. If you pass in
> a
> > "Materialized.withCachingEnabled()", you should be able to get a POC
> > working by setting the max cache size pretty high and setting the commit
> > interval for your desired rate:
> > https://docs.confluent.io/current/streams/developer-
> guide/memory-mgmt.html#streams-developer-guide-memory-management
> > .
> >
> > There are a couple of cases in joins and whatnot where it doesn't work,
> > but for the aggregations we discussed, it should. The reason for KIP-328
> is
> > to provide finer control and hopefully a more straightforward API.
> >
> > Let me know if that works, and I'll drop a message in here when I create
> > the draft PR for KIP-328. I'd really appreciate your feedback.
> >
> > Thanks,
> > -John
> >
> > On Wed, Jul 4, 2018 at 10:17 PM flaviost...@gmail.com <
> > flaviost...@gmail.com> wrote:
> >
> >> John, that was fantastic, man!
> >> Have you built any custom implementation of your KIP in your machine so
> >> that I could test it out here? I wish I could test it out.
> >> If you need any help implementing this feature, please tell me.
> >>
> >> Thanks.
> >>
> >> -Flávio Stutz
> >>
> >>
> >>
> >>
> >> On 2018/07/03 18:04:52, John Roesler  wrote:
> >> > Hi Flávio,
> >> > Thanks! I think that we can actually do this, but the API could be
> >> better.
> >> > I've included Java code below, but I'll copy and modify your example
> so
> >> > we're on the same page.
> >> >
> >> > EXERCISE 1:
> >> >   - The case is "total counting of events for a huge website"
> >> >   - Tasks from Application A will have something like:
> >> >  .stream(/site-events)
> >> >  .transform( re-key s.t. the new key is the partition id)
> >> >  .groupByKey() // you have to do this before count
> >> >  .count()
> >> >   // you explicitly published to a one-partition topic here,
> but
> >> > it's actually sufficient just
> >> >   // to re-group onto one key. You could name and pre-create
> the
> >> > intermediate topic here,
> >> >   // but you don't need a separate application for the final
> >> > aggregation.
> >> >  .groupBy((partitionId, partialCount) -> new KeyValue("ALL",
> >> > partialCount))
> >> >  .aggregate(sum up the partialCounts)
> >> >  .publish(/counter-total)
> >> >
> >> > I've left out the suppressions, but they would go right after the
> >> count()
> >> > and the aggregate().
> >> >
> >> > With this program, you don't have to worry about the
> double-aggregation
> >> you
> >> > mentioned in the last email. The KTable produced by the first count()
> >> will
> >> > maintain the correct count per partition. If 

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

[Guozhang Wang]

Hi folks, just to summarize the options we have so far:

1) Add a new "as" for KTable / KStream, plus adding new fields for
operators-returns-void control objects (the current wiki's proposal).

Pros: no more overloads.
Cons: a bit departing with the current high-level API design of the DSL,
plus, the inconsistency between operators-returns-void and
operators-not-return-voids.

2) Add overloaded functions for all operators, that accepts a new control
object "Described".

Pros: consistent with current APIs.
Cons: lots of overloaded functions to add.

3) Add another default function in the interface (thank you J8!) as John
proposed.

Pros: no overloaded functions, no "Described".
Cons: do we lose lambda functions really (seems not if we provide a "named"
for each func)? Plus "Described" may be more extensible than a single
`String`.


My principle of considering which one is better depends primarily on "how
to make advanced users easily use the additional API, while keeping it
hidden from normal users who do not care at all". For that purpose I think
3) > 1) > 2).

One caveat though, is that changing the interface would not be
binary-compatible though source-compatible, right? I.e. users need to
recompile their code though no changes needed.



Another note: for 3), if we really want to keep extensibility of Described
we could do sth. like:

-

public interface Predicate {
// existing method
boolean test(final K key, final V value);

// new default method adds the ability to name the predicate
default Described described() {
return new Described(null);
}
}

--

where user's code becomes:

stream.filter(named("key", (k, v) -> true));   // note `named` now just
sets a Described("key") in "described()".

stream.filter(described(Described.as("key", /* any other fancy parameters
in the future*/), (k, v) -> true));
--


I feel it is not much likely that we'd need to extend it further in the
future, so just a `String` would be good enough. But just listing all
possibilities here.



Guozhang






On Fri, Jul 6, 2018 at 8:19 AM, John Roesler  wrote:

> Hi Florian,
>
> Sorry I'm late to the party, but I missed the message originally.
>
> Regarding the names, it's probably a good idea to stick to the same
> character set we're currently using: letters, numbers, and hyphens. The
> names are used in Kafka topics, files and folders, and RocksDB databases,
> and we also need them to work with the file systems of Windows, Linux, and
> MacOS. My opinion is that with a situation like that, it's better to be
> conservative. It might also be a good idea to impose an upper limit on name
> length to avoid running afoul of any of those systems.
>
> ---
>
> It seems like there's a small debate between 1) adding a new method to
> KStream (and maybe KTable) to modify its name after the fact, or 2)
> piggy-backing on the config objects where they exist and adding one where
> they don't. To me, #2 is the better alternative even though it produces
> more overloads and may be a bit awkward in places.
>
> The reason is simply that #1 is a high-level departure from the
> graph-building paradigm we're using in the DSL. Consider:
>
> Graph.node1(config).node2(config)
>
> vs
>
> Graph.node1().config().node2().config()
>
> We could have done either, but we picked the former. I think it's probably
> a good goal to try and stick to it so that developers can develop and rely
> on their instincts for how the DSL will behave.
>
> I do want to present one alternative to adding new config objects: we can
> just add a "name()" method to all our "action" interfaces. For example,
> I'll demonstrate how we can add a "name" to Predicate and then use it to
> name a "KStream#filter" DSL operator:
>
> public interface Predicate {
> // existing method
> boolean test(final K key, final V value);
>
> // new default method adds the ability to name the predicate
> default String name() {
> return null;
> }
>
> // new static factory method adds the ability to wrap lambda predicates
> with a named predicate
> static  Predicate named(final String name, final
> Predicate predicate) {
> return new Predicate() {
> @Override
> public boolean test(final K key, final V value) {
> return predicate.test(key, value);
> }
>
> @Override
> public String name() {
> return name;
> }
> };
> }
> }
>
> Then, here's how it would look to use it:
>
> // Anonymous predicates continue to work just fine
> stream.filter((k, v) -> true);
>
> // Devs can swap in a Predicate that implements the name() method.
> stream.filter(new Predicate() {
> @Override
> public boolean test(final Object key, final Object value) {
> return true;
> }
>
> @Override
> public String name() {
> return "hey";
> }
> });
>
> // Or they can wrap their existing lambda 

Re: [VOTE] KIP-280: Enhanced log compaction

[Jun Rao]

Hi, Guozhang,

For #4, what you suggested could make sense for timestamp based de-dup, but
not sure how general it is since the KIP also supports de-dup based on
header.

Thanks,

Jun

On Fri, Jul 6, 2018 at 1:12 PM, Guozhang Wang  wrote:

> Hello Jun,
> Thanks for your feedbacks. I'd agree on #3 that it's worth adding a special
> check to not delete the last message, since although unlikely, it is still
> possible that a new active segment gets rolled out but contains no data
> yet, and hence the actual last message in this case would be in a
> "compact-able" segment.
>
> For the second part of #4 you raised, maybe we could educate users to set "
> message.timestamp.difference.max.ms" to be no larger than "
> log.cleaner.delete.retention.ms" (its default value is Long.MAX_VALUE)? A
> more aggressive approach would be changing the default value of the former
> to be the value of the latter if:
>
> 1. cleanup.policy = compact OR compact,delete
> 2. log.cleaner.compaction.strategy != offset
>
> Because in this case I think it makes sense to really allow users send any
> data longer than "log.cleaner.delete.retention.ms", WDYT?
>
>
> Guozhang
>
>
> On Fri, Jul 6, 2018 at 11:51 AM, Jun Rao  wrote:
>
> > Hi, Luis,
> >
> > 1. The cleaning policy is configurable at both global and topic level.
> The
> > global one has the name log.cleanup.policy and the topic level has the
> name
> > cleanup.policy by just stripping the log prefix. We can probably do the
> > same for the new configs.
> >
> > 2. Since this KIP may require an admin to configure a larger dedup buffer
> > size, it would be useful to document this impact in the wiki and the
> > release notes.
> >
> > 3. Yes, it's unlikely for the last message to be removed in the current
> > implementation since we never clean the active segment. However, in
> theory,
> > this can happen. So it would be useful to guard this explicitly.
> >
> > 4. Just thought about another issue. We probably want to be a bit careful
> > with key deletion. Currently, one can delete a key by sending a message
> > with a delete tombstone (a null payload). To prevent a reader from
> missing
> > a deletion if it's removed too quickly, we depend on a configuration
> > log.cleaner.delete.retention.ms (defaults to 1 day). The delete
> tombstone
> > will only be physically removed from the log after that amount of time.
> The
> > expectation is that a reader should finish reading to the end of the log
> > after consuming a message within that configured time. With the new
> > strategy, we have similar, but slightly different problems. The first
> > problem is that the delete tombstone may be delivered earlier than an
> > outdated record in offset order to a consumer. In order for the consumer
> > not to take the outdated record, the consumer should cache the deletion
> > tombstone for some configured amount of time. We ca probably piggyback
> this
> > on log.cleaner.delete.retention.ms, but we need to document this. The
> > second problem is that once the delete tombstone is physically removed
> from
> > the log, how can we prevent outdated records to be added (otherwise, they
> > will never be garbage collected)? Not sure what's the best way to do
> this.
> > One possible way is to push this back to the application and require the
> > user not to publish outdated records after log.cleaner.delete.retention.
> ms
> > .
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jul 4, 2018 at 11:11 AM, Luís Cabral
>  > >
> > wrote:
> >
> > > Hi Jun,
> > >
> > > -:  1. I guess both new configurations will be at the topic level?
> > >
> > > They will exist in the global configuration, at the very least.
> > > I would like to have them on the topic level as well, but there is an
> > > inconsistency between the cleanup/compaction properties that exist
> “only
> > > globally” vs “globally + per topic”.
> > > I haven’t gotten around to investigating why, and if that reason would
> > > then also impact the properties I’m suggesting. At first glance they
> seem
> > > to belong with the properties that are "only globally” configured, but
> > > Guozhang has written earlier with a suggestion of a compaction property
> > > that works for both (though I haven’t had the time to look into it yet,
> > > unfortunately).
> > >
> > > -:  2. Since the log cleaner now needs to keep both the offset and
> > another
> > > long (say timestamp) in the de-dup map, it reduces the number of keys
> > that
> > > we can keep in the map and thus may require more rounds of cleaning.
> This
> > > is probably not a big issue, but it will be useful to document this
> > impact
> > > in the KIP.
> > >
> > > As a reader, I tend to prefer brief documentation on new features (they
> > > tend to be too many for me to find the willpower to read a 200-page
> essay
> > > about each one), so that influences me to avoid writing about every
> > > micro-impact that may exist, and simply leave it inferred (as you have
> > just
> > > done).
> > > But I also 

Build failed in Jenkins: kafka-1.1-jdk7 #161

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-7136: Avoid deadlocks in synchronized metrics reporters (#5341)

--
[...truncated 422.58 KB...]

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing PASSED

kafka.log.ProducerStateManagerTest > testTruncate STARTED

kafka.log.ProducerStateManagerTest > testTruncate PASSED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload STARTED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload PASSED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump STARTED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
PASSED

kafka.log.ProducerStateManagerTest > testTakeSnapshot STARTED

kafka.log.ProducerStateManagerTest > testTakeSnapshot PASSED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore STARTED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore PASSED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached STARTED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction PASSED

kafka.log.ProducerStateManagerTest > testNoValidationOnFirstEntryWhenLoadingLog 
STARTED

kafka.log.ProducerStateManagerTest > testNoValidationOnFirstEntryWhenLoadingLog 
PASSED

kafka.log.ProducerStateManagerTest > testLoadFromEmptySnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromEmptySnapshotFile PASSED

kafka.log.ProducerStateManagerTest > 
testProducersWithOngoingTransactionsDontExpire STARTED

kafka.log.ProducerStateManagerTest > 
testProducersWithOngoingTransactionsDontExpire PASSED

kafka.log.ProducerStateManagerTest > testBasicIdMapping STARTED

kafka.log.ProducerStateManagerTest > testBasicIdMapping PASSED

kafka.log.ProducerStateManagerTest > updateProducerTransactionState STARTED

kafka.log.ProducerStateManagerTest > updateProducerTransactionState PASSED

kafka.log.ProducerStateManagerTest > testRecoverFromSnapshot STARTED

kafka.log.ProducerStateManagerTest > testRecoverFromSnapshot PASSED

kafka.log.ProducerStateManagerTest > testPrepareUpdateDoesNotMutate STARTED

kafka.log.ProducerStateManagerTest > testPrepareUpdateDoesNotMutate PASSED

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic STARTED

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic PASSED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers STARTED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset STARTED

kafka.log.ProducerStateManagerTest > 

Re: [VOTE] KIP-280: Enhanced log compaction

[Ismael Juma]

Thanks for the KIP, Luis. A brief comment below.

On Wed, Jul 4, 2018 at 11:11 AM Luís Cabral 
wrote:

> As a reader, I tend to prefer brief documentation on new features (they
> tend to be too many for me to find the willpower to read a 200-page essay
> about each one), so that influences me to avoid writing about every
> micro-impact that may exist, and simply leave it inferred (as you have just
> done).
> But I also don’t feel strongly enough about it to argue either way. So,
> after reading my argument, if you still insist, I’ll happily add this there.
>

KIPs are not your typical user level documentation. We strive to document
details like the one Jun pointed out as they're beneficial during review,
but also from understanding the operations impact.

Ismael

Re: [VOTE] KIP-280: Enhanced log compaction

[Guozhang Wang]

Hello Jun,
Thanks for your feedbacks. I'd agree on #3 that it's worth adding a special
check to not delete the last message, since although unlikely, it is still
possible that a new active segment gets rolled out but contains no data
yet, and hence the actual last message in this case would be in a
"compact-able" segment.

For the second part of #4 you raised, maybe we could educate users to set "
message.timestamp.difference.max.ms" to be no larger than "
log.cleaner.delete.retention.ms" (its default value is Long.MAX_VALUE)? A
more aggressive approach would be changing the default value of the former
to be the value of the latter if:

1. cleanup.policy = compact OR compact,delete
2. log.cleaner.compaction.strategy != offset

Because in this case I think it makes sense to really allow users send any
data longer than "log.cleaner.delete.retention.ms", WDYT?


Guozhang


On Fri, Jul 6, 2018 at 11:51 AM, Jun Rao  wrote:

> Hi, Luis,
>
> 1. The cleaning policy is configurable at both global and topic level. The
> global one has the name log.cleanup.policy and the topic level has the name
> cleanup.policy by just stripping the log prefix. We can probably do the
> same for the new configs.
>
> 2. Since this KIP may require an admin to configure a larger dedup buffer
> size, it would be useful to document this impact in the wiki and the
> release notes.
>
> 3. Yes, it's unlikely for the last message to be removed in the current
> implementation since we never clean the active segment. However, in theory,
> this can happen. So it would be useful to guard this explicitly.
>
> 4. Just thought about another issue. We probably want to be a bit careful
> with key deletion. Currently, one can delete a key by sending a message
> with a delete tombstone (a null payload). To prevent a reader from missing
> a deletion if it's removed too quickly, we depend on a configuration
> log.cleaner.delete.retention.ms (defaults to 1 day). The delete tombstone
> will only be physically removed from the log after that amount of time. The
> expectation is that a reader should finish reading to the end of the log
> after consuming a message within that configured time. With the new
> strategy, we have similar, but slightly different problems. The first
> problem is that the delete tombstone may be delivered earlier than an
> outdated record in offset order to a consumer. In order for the consumer
> not to take the outdated record, the consumer should cache the deletion
> tombstone for some configured amount of time. We ca probably piggyback this
> on log.cleaner.delete.retention.ms, but we need to document this. The
> second problem is that once the delete tombstone is physically removed from
> the log, how can we prevent outdated records to be added (otherwise, they
> will never be garbage collected)? Not sure what's the best way to do this.
> One possible way is to push this back to the application and require the
> user not to publish outdated records after log.cleaner.delete.retention.ms
> .
>
> Thanks,
>
> Jun
>
> On Wed, Jul 4, 2018 at 11:11 AM, Luís Cabral  >
> wrote:
>
> > Hi Jun,
> >
> > -:  1. I guess both new configurations will be at the topic level?
> >
> > They will exist in the global configuration, at the very least.
> > I would like to have them on the topic level as well, but there is an
> > inconsistency between the cleanup/compaction properties that exist “only
> > globally” vs “globally + per topic”.
> > I haven’t gotten around to investigating why, and if that reason would
> > then also impact the properties I’m suggesting. At first glance they seem
> > to belong with the properties that are "only globally” configured, but
> > Guozhang has written earlier with a suggestion of a compaction property
> > that works for both (though I haven’t had the time to look into it yet,
> > unfortunately).
> >
> > -:  2. Since the log cleaner now needs to keep both the offset and
> another
> > long (say timestamp) in the de-dup map, it reduces the number of keys
> that
> > we can keep in the map and thus may require more rounds of cleaning. This
> > is probably not a big issue, but it will be useful to document this
> impact
> > in the KIP.
> >
> > As a reader, I tend to prefer brief documentation on new features (they
> > tend to be too many for me to find the willpower to read a 200-page essay
> > about each one), so that influences me to avoid writing about every
> > micro-impact that may exist, and simply leave it inferred (as you have
> just
> > done).
> > But I also don’t feel strongly enough about it to argue either way. So,
> > after reading my argument, if you still insist, I’ll happily add this
> there.
> >
> > -: 3. With the new cleaning strategy, we want to be a bit careful with
> > removing the last message in a partition (which is possible now). We need
> > to preserve the offset of the last message so that we don't reuse the
> > offset for a different message. One way to simply never remove the last
> > message. 

Re: [VOTE] 2.0.0 RC1

[Rajini Sivaram]

Hi all,

We found a deadlock that affects metrics reporters that use synchronisation
(https://issues.apache.org/jira/browse/KAFKA-7136). A fix has been merged.
I will create another RC that includes the fix for this as well as the PR
to update request versions in ReplicaFetcherThread (
https://github.com/apache/kafka/pull/5342/files). Will wait for the system
test run over the weekend and create a new RC on Monday.

Many thanks to everyone who tested and voted for RC1. Please help test RC2
next week if you have time.

Thank you,

Rajini

On Fri, Jul 6, 2018 at 12:19 PM, Matthias Wessendorf 
wrote:

> +1
> used RC1 in little app, that I have - worked fine
>
> On Wed, Jul 4, 2018 at 1:58 PM Mickael Maison 
> wrote:
>
> > +1 (non-binding)
> > Ran tests and quickstart using kafka_2.12-2.0.0.tgz with Java 8
> >
> > Thanks
> >
> > On Wed, Jul 4, 2018 at 10:24 AM, Manikumar 
> > wrote:
> > > +1 (non-binding)  Verified the release notes, src, binary artifacts,
> Ran
> > > the test suite,
> > > Verified quick start, Ran producer/consumer perf test, log compaction
> > tests
> > >
> > > Thanks
> > >
> > >
> > > On Wed, Jul 4, 2018 at 8:33 AM Brett Rann 
> > wrote:
> > >
> > >> +1 tentative
> > >> rolling upgrade of tiny shared staging multitenacy (200+ consumer
> > groups)
> > >> cluster from 1.1 to 2.0.0-rc1. cluster looks healthy. Will monitor.
> > >>
> > >> On Tue, Jul 3, 2018 at 8:18 AM Harsha  wrote:
> > >>
> > >> > +1.
> > >> >
> > >> > 1) Ran unit tests
> > >> > 2) 3 node cluster , tested basic operations.
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" <
> > >> > vahidhashem...@us.ibm.com> wrote:
> > >> >
> > >> > >
> > >> > >
> > >> > >
> > >> > > +1 (non-binding)
> > >> > >
> > >> > > Built from source and ran quickstart successfully on Ubuntu (with
> > Java
> > >> > 8).
> > >> > >
> > >> > >
> > >> > > Minor: It seems this doc update PR is not included in the RC:
> > >> > > https://github.com/apache/kafka/pull/5280
> > >> > 
> > >> > > Guozhang seems to have wanted to cherry-pick it to 2.0.
> > >> > >
> > >> > > Thanks Rajini!
> > >> > > --Vahid
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > From: Rajini Sivaram < rajinisiva...@gmail.com >
> > >> > > To: dev < dev@kafka.apache.org >, Users < us...@kafka.apache.org
> >,
> > >> > > kafka-clients < kafka-clie...@googlegroups.com >
> > >> > > Date: 06/29/2018 11:36 AM
> > >> > > Subject: [VOTE] 2.0.0 RC1
> > >> > >
> > >> > >
> > >> > >
> > >> > > Hello Kafka users, developers and client-developers,
> > >> > >
> > >> > >
> > >> > > This is the second candidate for release of Apache Kafka 2.0.0.
> > >> > >
> > >> > >
> > >> > > This is a major version release of Apache Kafka. It includes 40
> new
> > >> KIPs
> > >> > > and
> > >> > >
> > >> > > several critical bug fixes. Please see the 2.0.0 release plan for
> > more
> > >> > > details:
> > >> > >
> > >> > >
> > >> >
> > >>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pa
> geId=80448820
> > >> > <
> > >>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pa
> geId=80448820>
> > >> > >
> > >> > >
> > >> > >
> > >> > > A few notable highlights:
> > >> > >
> > >> > > - Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
> > CreateTopics
> > >> > > (KIP-277)
> > >> > > - SASL/OAUTHBEARER implementation (KIP-255)
> > >> > > - Improved quota communication and customization of quotas
> (KIP-219,
> > >> > > KIP-257)
> > >> > > - Efficient memory usage for down conversion (KIP-283)
> > >> > > - Fix log divergence between leader and follower during fast
> leader
> > >> > > failover (KIP-279)
> > >> > > - Drop support for Java 7 and remove deprecated code including old
> > >> > > scala
> > >> > > clients
> > >> > > - Connect REST extension plugin, support for externalizing secrets
> > and
> > >> > > improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> > >> > > - Scala API for Kafka Streams and other Streams API improvements
> > >> > > (KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> > >> > >
> > >> > > Release notes for the 2.0.0 release:
> > >> > >
> > >> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/RELEASE_NOT
> ES.html
> > >> >  TES.html>
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > *** Please download, test and vote by Tuesday, July 3rd, 4pm PT
> > >> > >
> > >> > >
> > >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > >
> > >> > > http://kafka.apache.org/KEYS
> > >> > 
> > >> > >
> > >> > >
> > >> > >
> > >> > > * Release artifacts to be voted upon (source and binary):
> > >> > >
> > >> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/
> > >> > 
> > >> > >
> > >> > >
> > >> > >
> > >> > > * Maven artifacts to be voted upon:
> > >> > >
> > >> > > 

Re: [VOTE] KIP-280: Enhanced log compaction

[Jun Rao]

Hi, Luis,

1. The cleaning policy is configurable at both global and topic level. The
global one has the name log.cleanup.policy and the topic level has the name
cleanup.policy by just stripping the log prefix. We can probably do the
same for the new configs.

2. Since this KIP may require an admin to configure a larger dedup buffer
size, it would be useful to document this impact in the wiki and the
release notes.

3. Yes, it's unlikely for the last message to be removed in the current
implementation since we never clean the active segment. However, in theory,
this can happen. So it would be useful to guard this explicitly.

4. Just thought about another issue. We probably want to be a bit careful
with key deletion. Currently, one can delete a key by sending a message
with a delete tombstone (a null payload). To prevent a reader from missing
a deletion if it's removed too quickly, we depend on a configuration
log.cleaner.delete.retention.ms (defaults to 1 day). The delete tombstone
will only be physically removed from the log after that amount of time. The
expectation is that a reader should finish reading to the end of the log
after consuming a message within that configured time. With the new
strategy, we have similar, but slightly different problems. The first
problem is that the delete tombstone may be delivered earlier than an
outdated record in offset order to a consumer. In order for the consumer
not to take the outdated record, the consumer should cache the deletion
tombstone for some configured amount of time. We ca probably piggyback this
on log.cleaner.delete.retention.ms, but we need to document this. The
second problem is that once the delete tombstone is physically removed from
the log, how can we prevent outdated records to be added (otherwise, they
will never be garbage collected)? Not sure what's the best way to do this.
One possible way is to push this back to the application and require the
user not to publish outdated records after log.cleaner.delete.retention.ms.

Thanks,

Jun

On Wed, Jul 4, 2018 at 11:11 AM, Luís Cabral 
wrote:

> Hi Jun,
>
> -:  1. I guess both new configurations will be at the topic level?
>
> They will exist in the global configuration, at the very least.
> I would like to have them on the topic level as well, but there is an
> inconsistency between the cleanup/compaction properties that exist “only
> globally” vs “globally + per topic”.
> I haven’t gotten around to investigating why, and if that reason would
> then also impact the properties I’m suggesting. At first glance they seem
> to belong with the properties that are "only globally” configured, but
> Guozhang has written earlier with a suggestion of a compaction property
> that works for both (though I haven’t had the time to look into it yet,
> unfortunately).
>
> -:  2. Since the log cleaner now needs to keep both the offset and another
> long (say timestamp) in the de-dup map, it reduces the number of keys that
> we can keep in the map and thus may require more rounds of cleaning. This
> is probably not a big issue, but it will be useful to document this impact
> in the KIP.
>
> As a reader, I tend to prefer brief documentation on new features (they
> tend to be too many for me to find the willpower to read a 200-page essay
> about each one), so that influences me to avoid writing about every
> micro-impact that may exist, and simply leave it inferred (as you have just
> done).
> But I also don’t feel strongly enough about it to argue either way. So,
> after reading my argument, if you still insist, I’ll happily add this there.
>
> -: 3. With the new cleaning strategy, we want to be a bit careful with
> removing the last message in a partition (which is possible now). We need
> to preserve the offset of the last message so that we don't reuse the
> offset for a different message. One way to simply never remove the last
> message. Another way (suggested by Jason) is to create an empty message
> batch.
>
> That is a good point, but isn’t the last message always kept regardless?
> In all of my tests with this approach, never have I seen it being removed.
> This is not because I made it so while changing the code, it was simply
> like this before, which made me smile!
> Given these results, I just *assumed* (oops) that these scenarios
> represented the reality, so the compaction would only affected the “tail”,
> while the “head” remained untouched. Now that you say its possible that the
> last message actually gets overwritten somehow, I guess a new bullet point
> will have to be added to the KIP for this (after I’ve found the time to
> review the portion of the code that enacts this behaviour).
>
> Kind Regards,
> Luís Cabral
>
> From: Jun Rao
> Sent: 03 July 2018 23:58
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> Hi, Luis,
>
> Thanks for the KIP. Overall, this seems a useful KIP. A few comments below.
>
> 1. I guess both new configurations will be at the topic level?
> 2. Since 

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

[Harsha]

+1.

Thanks,
Harsha

On Fri, Jun 1st, 2018 at 10:21 AM, "Vahid S Hashemian" 
 wrote:

> 
> 
> 
> I'm bumping this vote thread up as the KIP requires only one binding +1 to
> 
> pass.
> The KIP is very similar in nature to the recently approved KIP-277 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API
> 
> ) and proposes a small improvement to make APIs' minimum required
> permissions more consistent.
> 
> Thanks.
> --Vahid
> 
> 
> 
> 
> From: Vahid S Hashemian/Silicon Valley/IBM
> To: dev < dev@kafka.apache.org >
> Date: 12/19/2017 11:30 AM
> Subject: [VOTE] KIP-231: Improve the Required ACL of ListGroups API
> 
> 
> I believe the concerns on this KIP have been addressed so far.
> Therefore, I'd like to start a vote.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-231%3A+Improve+the+Required+ACL+of+ListGroups+API
> 
> 
> Thanks.
> --Vahid
> 
> 
> 
> 
> 
> 
>

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

[Vahid S Hashemian]

Bump!




From:   "Vahid S Hashemian" 
To: dev 
Date:   06/01/2018 10:21 AM
Subject:Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups 
API



I'm bumping this vote thread up as the KIP requires only one binding +1 to 

pass.
The KIP is very similar in nature to the recently approved KIP-277 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API

) and proposes a small improvement to make APIs' minimum required 
permissions more consistent.

Thanks.
--Vahid




From:   Vahid S Hashemian/Silicon Valley/IBM
To: dev 
Date:   12/19/2017 11:30 AM
Subject:[VOTE] KIP-231: Improve the Required ACL of ListGroups API


I believe the concerns on this KIP have been addressed so far.
Therefore, I'd like to start a vote.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-231%3A+Improve+the+Required+ACL+of+ListGroups+API


Thanks.
--Vahid

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Ismael Juma]

Thanks for the KIP. I think we should be thinking about
https://issues.apache.org/jira/browse/KAFKA-6923 at the same time.

Ismael

On Thu, 5 Jul 2018, 07:45 Chia-Ping Tsai,  wrote:

> hi all,
>
> I would like to start voting on "KIP-331 Add default implementation to
> close() and configure() for Serializer, Deserializer and Serde"
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
>
> Cheers,
> Chia-Ping
>

Re: [VOTE] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[John Roesler]

Thank you!

I'm +1 (non-binding)

On Thu, Jul 5, 2018 at 10:24 PM Richard Yu 
wrote:

> Nice KIP!
> +1 (non-binding)
> -Richard
>
> On Friday, July 6, 2018, 9:10:43 AM GMT+8, Matthias J. Sax <
> matth...@confluent.io> wrote:
>
>  Thanks for the KIP!
>
> +1 (binding)
>
>
> -Matthias
>
> On 7/5/18 7:45 AM, Chia-Ping Tsai wrote:
> > hi all,
> >
> > I would like to start voting on "KIP-331 Add default implementation to
> close() and configure() for Serializer, Deserializer and Serde"
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> >
> > Cheers,
> > Chia-Ping
> >
>

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

[John Roesler]

Hi Florian,

Sorry I'm late to the party, but I missed the message originally.

Regarding the names, it's probably a good idea to stick to the same
character set we're currently using: letters, numbers, and hyphens. The
names are used in Kafka topics, files and folders, and RocksDB databases,
and we also need them to work with the file systems of Windows, Linux, and
MacOS. My opinion is that with a situation like that, it's better to be
conservative. It might also be a good idea to impose an upper limit on name
length to avoid running afoul of any of those systems.

---

It seems like there's a small debate between 1) adding a new method to
KStream (and maybe KTable) to modify its name after the fact, or 2)
piggy-backing on the config objects where they exist and adding one where
they don't. To me, #2 is the better alternative even though it produces
more overloads and may be a bit awkward in places.

The reason is simply that #1 is a high-level departure from the
graph-building paradigm we're using in the DSL. Consider:

Graph.node1(config).node2(config)

vs

Graph.node1().config().node2().config()

We could have done either, but we picked the former. I think it's probably
a good goal to try and stick to it so that developers can develop and rely
on their instincts for how the DSL will behave.

I do want to present one alternative to adding new config objects: we can
just add a "name()" method to all our "action" interfaces. For example,
I'll demonstrate how we can add a "name" to Predicate and then use it to
name a "KStream#filter" DSL operator:

public interface Predicate {
// existing method
boolean test(final K key, final V value);

// new default method adds the ability to name the predicate
default String name() {
return null;
}

// new static factory method adds the ability to wrap lambda predicates
with a named predicate
static  Predicate named(final String name, final
Predicate predicate) {
return new Predicate() {
@Override
public boolean test(final K key, final V value) {
return predicate.test(key, value);
}

@Override
public String name() {
return name;
}
};
}
}

Then, here's how it would look to use it:

// Anonymous predicates continue to work just fine
stream.filter((k, v) -> true);

// Devs can swap in a Predicate that implements the name() method.
stream.filter(new Predicate() {
@Override
public boolean test(final Object key, final Object value) {
return true;
}

@Override
public String name() {
return "hey";
}
});

// Or they can wrap their existing lambda using the static factory method
stream.filter(named("key", (k, v) -> true));

Just a thought.

Overall, I think it's really valuable to be able to name the processors,
for all the reasons you mentioned in the KIP. So thank you for introducing
this!

Thanks,
-John

On Thu, Jul 5, 2018 at 4:53 PM Florian Hussonnois 
wrote:

> Hi, thank you very much for all you suggestions. I've started to update the
> KIP (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL
> ).
> Also, I propose to rename the Processed class into Described - this will be
> more meaningful (but this is just a detail).
>
> I'm OK to not enforcing uppercase for specific names but should we allow
> arbitrary names with whitespaces for example ? Currently, I can't tell if
> this can lead to some side effects ?
>
> Le lun. 11 juin 2018 à 01:31, Matthias J. Sax  a
> écrit :
>
> > Just catching up on this thread.
> >
> > I like the general idea. Couple of comments:
> >
> >  - I think that adding `Processed` (or maybe a different name?) is a
> > valid proposal for stateless operators that only have a single overload
> > atm. It would align with the overall API design.
> >
> >  - for all methods with multiple existing overloads, we can consider to
> > extend `Consumed`, `Produced`, `Materialized` etc to take an additional
> > processor name (not sure atm how elegant this is; we would need to
> > "play" with the API a little bit; the advantage would be, that we do not
> > add more overloads what seems to be key for this KIP)
> >
> >  - operators return void: while I agree that the "name first" chaining
> > idea is not very intuitive, it might still work, if we name the method
> > correctly (again, we would need to "play" with the API a little bit to
> see)
> >
> >  - for DSL operators that are translated to multiple nodes: it might
> > make sense to use the specified operator name as prefix and add
> > reasonable suffixes. For example, a join translates into 5 operators
> > that could be name "name-left-store-processor",
> > "name-left-join-processor", "name-right-store-processor",
> > "name-right-join-processor", and "name-join-merge-processor" (or
> > similar). Maybe just using numbers might also work.
> >
> >  

Re: [VOTE] 2.0.0 RC1

[Matthias Wessendorf]

+1
used RC1 in little app, that I have - worked fine

On Wed, Jul 4, 2018 at 1:58 PM Mickael Maison 
wrote:

> +1 (non-binding)
> Ran tests and quickstart using kafka_2.12-2.0.0.tgz with Java 8
>
> Thanks
>
> On Wed, Jul 4, 2018 at 10:24 AM, Manikumar 
> wrote:
> > +1 (non-binding)  Verified the release notes, src, binary artifacts,  Ran
> > the test suite,
> > Verified quick start, Ran producer/consumer perf test, log compaction
> tests
> >
> > Thanks
> >
> >
> > On Wed, Jul 4, 2018 at 8:33 AM Brett Rann 
> wrote:
> >
> >> +1 tentative
> >> rolling upgrade of tiny shared staging multitenacy (200+ consumer
> groups)
> >> cluster from 1.1 to 2.0.0-rc1. cluster looks healthy. Will monitor.
> >>
> >> On Tue, Jul 3, 2018 at 8:18 AM Harsha  wrote:
> >>
> >> > +1.
> >> >
> >> > 1) Ran unit tests
> >> > 2) 3 node cluster , tested basic operations.
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" <
> >> > vahidhashem...@us.ibm.com> wrote:
> >> >
> >> > >
> >> > >
> >> > >
> >> > > +1 (non-binding)
> >> > >
> >> > > Built from source and ran quickstart successfully on Ubuntu (with
> Java
> >> > 8).
> >> > >
> >> > >
> >> > > Minor: It seems this doc update PR is not included in the RC:
> >> > > https://github.com/apache/kafka/pull/5280
> >> > 
> >> > > Guozhang seems to have wanted to cherry-pick it to 2.0.
> >> > >
> >> > > Thanks Rajini!
> >> > > --Vahid
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > From: Rajini Sivaram < rajinisiva...@gmail.com >
> >> > > To: dev < dev@kafka.apache.org >, Users < us...@kafka.apache.org >,
> >> > > kafka-clients < kafka-clie...@googlegroups.com >
> >> > > Date: 06/29/2018 11:36 AM
> >> > > Subject: [VOTE] 2.0.0 RC1
> >> > >
> >> > >
> >> > >
> >> > > Hello Kafka users, developers and client-developers,
> >> > >
> >> > >
> >> > > This is the second candidate for release of Apache Kafka 2.0.0.
> >> > >
> >> > >
> >> > > This is a major version release of Apache Kafka. It includes 40 new
> >> KIPs
> >> > > and
> >> > >
> >> > > several critical bug fixes. Please see the 2.0.0 release plan for
> more
> >> > > details:
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
> >> > <
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820>
> >> > >
> >> > >
> >> > >
> >> > > A few notable highlights:
> >> > >
> >> > > - Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
> CreateTopics
> >> > > (KIP-277)
> >> > > - SASL/OAUTHBEARER implementation (KIP-255)
> >> > > - Improved quota communication and customization of quotas (KIP-219,
> >> > > KIP-257)
> >> > > - Efficient memory usage for down conversion (KIP-283)
> >> > > - Fix log divergence between leader and follower during fast leader
> >> > > failover (KIP-279)
> >> > > - Drop support for Java 7 and remove deprecated code including old
> >> > > scala
> >> > > clients
> >> > > - Connect REST extension plugin, support for externalizing secrets
> and
> >> > > improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> >> > > - Scala API for Kafka Streams and other Streams API improvements
> >> > > (KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> >> > >
> >> > > Release notes for the 2.0.0 release:
> >> > >
> >> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/RELEASE_NOTES.html
> >> > 
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > *** Please download, test and vote by Tuesday, July 3rd, 4pm PT
> >> > >
> >> > >
> >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> >> > >
> >> > > http://kafka.apache.org/KEYS
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Release artifacts to be voted upon (source and binary):
> >> > >
> >> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Maven artifacts to be voted upon:
> >> > >
> >> > > https://repository.apache.org/content/groups/staging/
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Javadoc:
> >> > >
> >> > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc1/javadoc/
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
> >> > >
> >> > > https://github.com/apache/kafka/tree/2.0.0-rc1
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Documentation:
> >> > >
> >> > > http://kafka.apache.org/20/documentation.html
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Protocol:
> >> > >
> >> > > http://kafka.apache.org/20/protocol.html
> >> > 
> >> > >
> >> > >
> >> > >
> >> > > * Successful Jenkins builds for the 2.0 branch:
> >> 

[jira] [Created] (KAFKA-7138) Kafka Connect - Make errors.deadletterqueue.topic.replication.factor default consistent

[Robin Moffatt (JIRA)]

Robin Moffatt created KAFKA-7138:


 Summary: Kafka Connect - Make 
errors.deadletterqueue.topic.replication.factor default consistent
 Key: KAFKA-7138
 URL: https://issues.apache.org/jira/browse/KAFKA-7138
 Project: Kafka
  Issue Type: Bug
Reporter: Robin Moffatt


{{errors.deadletterqueue.topic.replication.factor}} defaults to RF 3

The standard out of the box config files override the RF for 
{{offset.storage.replication.factor}} (and {{config}} and {{status}}) to 1

To make the experience consistent for users (especially new users, running a 
single-node dev environment), the default RF in effect for 
{{errors.deadletterqueue.topic.replication.factor}} should also be 1. 

It would make it easier for devs getting started on single-node setups.

For prod people should be actively configuring this stuff anyway, this would 
get included in that.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7137) ability to trigger compaction for tombstoning and GDPR

[Brett Rann (JIRA)]

Brett Rann created KAFKA-7137:
-

 Summary: ability to trigger compaction for tombstoning and GDPR
 Key: KAFKA-7137
 URL: https://issues.apache.org/jira/browse/KAFKA-7137
 Project: Kafka
  Issue Type: Wish
Reporter: Brett Rann


Just spent some time wrapping my head around the inner workings of compaction 
and tombstoning, with a view to providing guarantees for deleting previous 
values of tombstoned keys from kafka within a desired time.

There's a couple of good posts that touch on this:
https://www.confluent.io/blog/handling-gdpr-log-forget/
http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/

Basically, log.cleaner.min.cleanable.ratio or min.cleanable.dirty.ratio is 
hijacked to force aggressive compaction (by setting it to 0, or 0.1 
depending on what you read), and along with segment.ms can provide timing 
guarantees that a tombstone will result in any other values for the key will be 
deleted within a desired time.

But that sacrifices the utility of min.cleanable.dirty.ratio (and to a lesser 
extent, control over segment sizes).  On any duplicate key and a new segment 
roll it will run compaction, when otherwise it might be preferrable to allow a 
more generous dirty.ratio in the case of plain old duplicates.

It would be useful to have control over triggering a compaction without losing 
the utility of the dirty.ratio setting.

The pure need here is to specify a minimum time for the log cleaner to run on a 
topic that has keys replaced by a tombstone message that are past the minimum 
retention times provided by min.compaction.lag.ms

Something like a log.cleaner.max.delay.ms, and an API to trigger compaction, 
with some nuances to be fleshed out.

Does this make sense, and sound like it's worth a KIP? I'd be happy to write 
something up.

In the mean time, this can be worked around with some duct tape:

* make sure any values you want deleted by a tombstone have passed min 
retention configs
* set global log.cleaner.io.max.bytes.per.second to what you want for the 
compaction task
* set topic min.cleanable.dirty.ratio=0 for the topic
* set a small segment.ms
* wait for a new segment to roll (ms + a message coming in) and wait for 
compaction to kick in. GDPR met!
* undo the hacks




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)