Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Jun Rao]

Hi, Jan,

Thanks for the discussion. Great points.

Let me try to summarize the approach that you are proposing. On the broker
side, we reshuffle the existing data in a topic from current partitions to
the new partitions. Once the reshuffle fully catches up, switch the
consumers to start consuming from the new partitions. If a consumer needs
to rebuild its local state (due to partition changes), let the consumer
rebuild its state by reading all existing data from the new partitions.
Once all consumers have switches over, cut over the producer to the new
partitions.

The pros for this approach are that :
1. There is just one way to rebuild the local state, which is simpler.

The cons for this approach are:
1. Need to copy existing data.
2. The cutover of the producer is a bit complicated since it needs to
coordinate with all consumer groups.
3. The rebuilding of the state in the consumer is from the input topic,
which can be more expensive than rebuilding from the existing state.
4. The broker potentially has to know the partitioning function. If this
needs to be customized at the topic level, it can be a bit messy.

Here is an alternative approach by applying your idea not in the broker,
but in the consumer. When new partitions are added, we don't move existing
data. In KStreams, we first reshuffle the new input data to a new topic T1
with the old number of partitions and feed T1's data to the rest of the
pipeline. In the meantime, KStreams reshuffles all existing data of the
change capture topic to another topic C1 with the new number of partitions.
We can then build the state of the new tasks from C1. Once the new states
have been fully built, we can cut over the consumption to the input topic
and delete T1. This approach works with compacted topic too. If an
application reads from the beginning of a compacted topic, the consumer
will reshuffle the portion of the input when the number of partitions
doesn't match the number of tasks.

The pros of this approach are:
1. No need to copy existing data.
2. Each consumer group can cut over to the new partitions independently.
3. The state is rebuilt from the change capture topic, which is cheaper
than rebuilding from the input topic.
4. Only the KStreams job needs to know the partitioning function.

The cons of this approach are:
1. Potentially the same input topic needs to be reshuffled more than once
in different consumer groups during the transition phase.

What do you think?

Thanks,

Jun



On Thu, Mar 15, 2018 at 1:04 AM, Jan Filipiak 
wrote:

> Hi Jun,
>
> thank you for following me on these thoughts. It was important to me to
> feel that kind of understanding for my arguments.
>
> What I was hoping for (I mentioned this earlier) is that we can model the
> case where we do not want to copy the data the exact same way as the case
> when we do copy the data. Maybe you can peek into the mails before to see
> more details for this.
>
> This means we have the same mechanism to transfer consumer groups to
> switch topic. The offset mapping that would be generated would even be
> simpler End Offset of the Old topic => offset 0 off all the partitions of
> the new topic. Then we could model the transition of a non-copy expansion
> the exact same way as a copy-expansion.
>
> I know this only works when topic growth by a factor. But the benefits of
> only growing by a factor are to strong anyways. See Clemens's hint and
> remember that state reshuffling is entirely not needed if one doesn't want
> to grow processing power.
>
> I think these benefits should be clear, and that there is basically no
> downside to what is currently at hand but just makes everything easy.
>
> One thing you need to know is. that if you do not offer rebuilding a log
> compacted topic like i suggest that even if you have consumer state
> reshuffling. The topic is broken and can not be used to bootstrap new
> consumers. They don't know if they need to apply a key from and old
> partition or not. This is a horrible downside I haven't seen a solution for
> in the email conversation.
>
> I argue to:
>
> Only grow topic by a factor always.
> Have the "no copy consumer" transition as the trivial case of the "copy
> consumer transition".
> If processors needs to be scaled, let them rebuild from the new topic and
> leave the old running in the mean time.
> Do not implement key shuffling in streams.
>
> I hope I can convince you especially with the fact how I want to handle
> consumer transition. I think
> you didn't quite understood me there before. I think the term "new topic"
> intimidated you a little.
> How we solve this on disc doesn't really matter, If the data goes into the
> same Dir or a different Dir or anything. I do think that it needs to
> involve at least rolling a new segment for the existing partitions.
> But most of the transitions should work without restarting consumers.
> (newer consumers with support for this). But with new topic i just meant
> the topic that now has a differe

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

[Panuwat Anawatmongkhon]

I couldn’t find this thread in mailing list. this reply is just to trigger
it up so I can include it in KIP

On Thu, 28 Dec 2560 at 03:07 Matthias J. Sax  wrote:

> @Matthias: just wanted to follow up on your question:
>
>  I wanted to double check. If I understand the proposal, it would
> replace
>  the explicit name with a name that is dynamically generated using the
>  AtomicInteger index. Would this affect the naming of any internally
>  generated topics?
>
> I think it would -- note, that the old API will not be removed but
> deprecated -- thus, you can still update without any issues staying with
> the old API -- only if you start to use the new API, it could impact an
> application.
>
> It should still be possible to upgrade to the new API if you invest the
> time to rename the corresponding topics correctly -- this will only work
> if you use a new application id or take your application offline though.
>
> -Matthias
>
>
> On 12/16/17 10:17 AM, Panuwat Anawatmongkhon wrote:
> > Hi all,
> > I would like to start the vote thread tomorrow, feel free to ask if there
> > is any concern.
> > Thank you
> >
> > On Thu, 7 Dec 2560 at 19:22 Panuwat Anawatmongkhon <
> > panuwat.anawatmongk...@gmail.com> wrote:
> >
> >>
> >> Yes, Matthias.
> >> The object will be used togerther with function table and function
> stream.
> >> I didn’t see how this will affect other part but if you do, please
> explain
> >> more on how this will affect generated topic name.
> >> Thank you
> >> Panuwat
> >>
> >>
> >> On Thu, 7 Dec 2560 at 00:01 Matthias Margush <
> matthias.marg...@gmail.com>
> >> wrote:
> >>
> >>> Hi.
> >>>
> >>> I wanted to double check. If I understand the proposal, it would
> replace
> >>> the explicit name with a name that is dynamically generated using the
> >>> AtomicInteger index. Would this affect the naming of any internally
> >>> generated topics?
> >>>
> >>> On Wed, Dec 6, 2017 at 7:59 AM Panuwat Anawatmongkhon <
> >>> panuwat.anawatmongk...@gmail.com> wrote:
> >>>
>  Thanks Bill.
> 
>  I can't think of reason to keep the old method too so if there is no
>  further discussion by tomorrow, I would like to start the vote thread.
> 
>  On Tue, Dec 5, 2017 at 10:38 PM, Bill Bejeck 
> wrote:
> 
> > Hi Panuwat,
> >
> > Thanks for the KIP, overall looks good to me.
> >
> > I want to play the devil's advocate for a second and ask do we want
> to
>  keep
> > the older method with the extra parameters vs. deprecation?
> >
> > Although ATM I can't think of a good reason to keep the old method
> >>> with
>  the
> > extra parameters.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Dec 5, 2017 at 5:48 AM, Ted Yu  wrote:
> >
> >> Fine by me.
> >>
> >> On Tue, Dec 5, 2017 at 2:45 AM, Panuwat Anawatmongkhon <
> >> panuwat.anawatmongk...@gmail.com> wrote:
> >>
> >>> Thank you, Matthias.
> >>>
> >>> Ted,
> >>> How about this.
> >>>
> >>> String globalTopicName = "testGlobalTopic";
> >>> String globalStoreName = "testAddGlobalStore";
> >>> final StreamsBuilder builder = new StreamsBuilder();
> >>> final KeyValueStoreBuilder globalStoreBuilder =
> >>> EasyMock.createNiceMock(KeyValueStoreBuilder.class);
> >>>
>  EasyMock.expect(globalStoreBuilder.name()).andReturn(globalStoreName).
> >>> anyTimes();
> >>> EasyMock.replay(globalStoreBuilder);
> >>> builder.addGlobalStore(globalStoreBuilder,globalTopicName,new
> >>> ConsumedInternal(),new MockProcessorSupplier());
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Dec 5, 2017 at 4:58 AM, Matthias J. Sax <
>  matth...@confluent.io
> >>
> >>> wrote:
> >>>
>  Panuwat,
> 
>  Thanks a lot for the KIP!
> 
>  Just one nit: `does not follow provide a good` -> spelling:
> >>> remove
>  `follow` ?
> 
>  Otherwise, looks good to me.
> 
> 
>  -Matthias
> 
> 
> 
>  On 12/4/17 10:49 AM, Ted Yu wrote:
> > Looks like you're implying logic similar to this:
> >
> > public synchronized  GlobalKTable
>  globalTable(final
>  String
> > topic,
> >
> >
>   final
> > Consumed consumed) {
> >
> >
> > StreamsBuilder is returned instead of GlobalKTable.
> >
> >
> > Can you add code snippet showing how the new API is used ?
> >
> >
> > On Mon, Dec 4, 2017 at 10:09 AM, Panuwat Anawatmongkhon <
> > panuwat.anawatmongk...@gmail.com> wrote:
> >
> >> What i am thinking right now is using the same approach as
> >> org.apache.kafka.streams.kstream.internals.
> > InternalStreamsBuilder#
> >> globalTable
> >>
> >> On Mon, 4 Dec 2560 at 23:10 Ted Yu 
> >>> wrote:
> >>
> >>>

[jira] [Resolved] (KAFKA-6486) TimeWindows causes unordered calls to windowed aggregation functions

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6486.

Resolution: Fixed

> TimeWindows causes unordered calls to windowed aggregation functions
> 
>
> Key: KAFKA-6486
> URL: https://issues.apache.org/jira/browse/KAFKA-6486
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Valentino Proietti
>Assignee: Asutosh Pandya
>Priority: Minor
> Fix For: 1.2.0
>
> Attachments: KAFKA-6486.patch
>
>
> This is not a real bug but it causes some weird behaviour, at least in my 
> opinion.
> The TimeWindows has a method called windowsFor() that uses and returns an 
> HashMap:
>     @Override
>     *public* Map windowsFor(*final* *long* timestamp) {
>         *long* windowStart = (Math._max_(0, timestamp - sizeMs + advanceMs) / 
> advanceMs) * advanceMs;
>         *final* Map windows = *new* HashMap<>();
>         
> the HashMap does not preserve the order of insertion and this ends up later 
> in calls to any streams windowed aggregation functions that are not ordered 
> by window time as I would expect.
> A simple solution is to replace the HashMap with a LinkedHashMap and that's 
> what I did.
> Anyway replacing it directly in your code can save hours of debugging to 
> understand what's happening.
> Thank you 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2488

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Use log start offset as high watermark if current value is out of

--
[...truncated 3.53 MB...]
kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > testIdleConnection STARTED

kafka.network.SocketServerTest > testIdleConnection PASSED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed STARTED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > configureNewConnectionException STARTED

kafka.network.SocketServerTest > configureNewConnectionException PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > processNewResponseException STA

Build failed in Jenkins: kafka-1.1-jdk7 #89

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Use log start offset as high watermark if current value is out of

--
[...truncated 1.89 MB...]
org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldProcessDataFromStoresWithLoggingDisabled STARTED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldProcessDataFromStoresWithLoggingDisabled PASSED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldRestoreState STARTED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldRestoreState PASSED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldSuccessfullyStartWhenLoggingDisabled STARTED

org.apache.kafka.streams.integration.RestoreIntegrationTest > 
shouldSuccessfullyStartWhenLoggingDisabled PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldAllowToQueryAfterThreadDied STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldAllowToQueryAfterThreadDied PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithZeroSizedCache STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithZeroSizedCache PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesAfterFilterState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesAfterFilterState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryFilterState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryFilterState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithNonZeroSizedCache STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithNonZeroSizedCache PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduce STARTED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey STARTED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldGroupByKey PASSED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduceWindowed STARTED

org.apache.kafka.streams.integration.KStreamAggregationDedupIntegrationTest > 
shouldReduceWindowed PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest
 STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest
 PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingPattern STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingPattern PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingTopic STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldThrowExceptionOverlappingTopic PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldOnlyReadRecordsWhereEarliestSpecifiedWithInvalidCommittedOffsets 
STARTED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > shouldOnlyReadRecordsWhereEarliestSpecifiedWithInvalidCommittedOffsets PASSED

org.apache.kafka.streams.integration.KStreamsFineGrainedAutoResetIntegrationTest
 > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithDefaultGlobalAutoOffsetResetEarliest
 

Build failed in Jenkins: kafka-trunk-jdk9 #490

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Use log start offset as high watermark if current value is out of

--
[...truncated 1.48 MB...]

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithNoOffsetReset PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetAndMatchingFromBeginning 
PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldResetUnConsumedOffsetsBeforeExitForNewConsumer STARTED

kafka.tools.ConsoleConsumerTest > 
shouldResetUnConsumedOffsetsBeforeExitForNewConsumer PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.message.MessageCompressionTest > testCompressSize STARTED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress STARTED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageTest > testChecksum STARTED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp STARTED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable STARTED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination STARTED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping STARTED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues STARTED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte STARTED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality STARTED

kafka.message.MessageTest > testEquality PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq 
STARTED

kafka.message.ByteBu

Build failed in Jenkins: kafka-1.0-jdk7 #173

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Use log start offset as high watermark if current value is out of

--
[...truncated 373.48 KB...]
kafka.log.BrokerCompressionTest > testBrokerSideCompression[15] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing PASSED

kafka.log.ProducerStateManagerTest > testTruncate STARTED

kafka.log.ProducerStateManagerTest > testTruncate PASSED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload STARTED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload PASSED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump STARTED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
PASSED

kafka.log.ProducerStateManagerTest > testTakeSnapshot STARTED

kafka.log.ProducerStateManagerTest > testTakeSnapshot PASSED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore STARTED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore PASSED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached STARTED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction 
STARTED

Build failed in Jenkins: kafka-trunk-jdk7 #3269

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Use log start offset as high watermark if current value is out of

--
[...truncated 417.77 KB...]
kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing PASSED

kafka.log.ProducerStateManagerTest > testTruncate STARTED

kafka.log.ProducerStateManagerTest > testTruncate PASSED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload STARTED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload PASSED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump STARTED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
PASSED

kafka.log.ProducerStateManagerTest > testTakeSnapshot STARTED

kafka.log.ProducerStateManagerTest > testTakeSnapshot PASSED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore STARTED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore PASSED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached STARTED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction PASSED

kafka.log.ProducerStateManagerTest > testNoValidation

[jira] [Created] (KAFKA-6679) Random corruption (CRC validation issues)

[Ari Uka (JIRA)]

Ari Uka created KAFKA-6679:
--

 Summary: Random corruption (CRC validation issues) 
 Key: KAFKA-6679
 URL: https://issues.apache.org/jira/browse/KAFKA-6679
 Project: Kafka
  Issue Type: Bug
  Components: consumer, replication
Affects Versions: 1.0.1, 0.10.2.0
 Environment: FreeBSD 11.0-RELEASE-p8
Reporter: Ari Uka






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6678) Upgrade dependencies with later release versions

[Ted Yu (JIRA)]

Ted Yu created KAFKA-6678:
-

 Summary: Upgrade dependencies with later release versions
 Key: KAFKA-6678
 URL: https://issues.apache.org/jira/browse/KAFKA-6678
 Project: Kafka
  Issue Type: Improvement
Reporter: Ted Yu


{code}
The following dependencies have later release versions:
 - net.sourceforge.argparse4j:argparse4j [0.7.0 -> 0.8.1]
 - org.bouncycastle:bcpkix-jdk15on [1.58 -> 1.59]
 - com.puppycrawl.tools:checkstyle [6.19 -> 8.8]
 - org.owasp:dependency-check-gradle [3.0.2 -> 3.1.1]
 - org.ajoberstar:grgit [1.9.3 -> 2.1.1]
 - org.glassfish.jersey.containers:jersey-container-servlet [2.25.1 -> 2.26]
 - org.eclipse.jetty:jetty-client [9.2.24.v20180105 -> 9.4.8.v20171121]
 - org.eclipse.jetty:jetty-server [9.2.24.v20180105 -> 9.4.8.v20171121]
 - org.eclipse.jetty:jetty-servlet [9.2.24.v20180105 -> 9.4.8.v20171121]
 - org.eclipse.jetty:jetty-servlets [9.2.24.v20180105 -> 9.4.8.v20171121]
 - org.openjdk.jmh:jmh-core [1.19 -> 1.20]
 - org.openjdk.jmh:jmh-core-benchmarks [1.19 -> 1.20]
 - org.openjdk.jmh:jmh-generator-annprocess [1.19 -> 1.20]
 - org.lz4:lz4-java [1.4 -> 1.4.1]
 - org.apache.maven:maven-artifact [3.5.2 -> 3.5.3]
 - org.jacoco:org.jacoco.agent [0.7.9 -> 0.8.0]
 - org.jacoco:org.jacoco.ant [0.7.9 -> 0.8.0]
 - org.rocksdb:rocksdbjni [5.7.3 -> 5.11.3]
 - org.scala-lang:scala-library [2.11.12 -> 2.12.4]
 - com.typesafe.scala-logging:scala-logging_2.11 [3.7.2 -> 3.8.0]
 - org.scala-lang:scala-reflect [2.11.12 -> 2.12.4]
 - org.scalatest:scalatest_2.11 [3.0.4 -> 3.0.5]
{code}
Looks like we can consider upgrading scalatest, jmh-core and checkstyle



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Subject: [VOTE] 1.1.0 RC3

[Damian Guy]

We have a green system test build -
https://jenkins.confluent.io/job/system-test-kafka/job/1.1/43/


Thanks,
Damian

On Fri, 16 Mar 2018 at 22:10 Vahid S Hashemian 
wrote:

> Hi Damian,
>
> Thanks for running the release.
>
> I tried building from source and running the quick start on Linux &
> Windows with both Java 8 & 9.
> Here's the result:
>
> +-+-+-+
> | |  Linux  | Windows |
> + +-+-+
> | | J8 | J9 | J8 | J9 |
> +-+++++
> |  Build  |  + |  + |  + |  + |
> +-+++++
> |  Single broker  |  + |  + |  + |  + |
> | produce/consume |||||
> +-+++++
> | Connect |  + |  ? |  - |  - |
> +-+++++
> | Streams |  + |  + |  + |  + |
> +-+++++
>
> ?: Connect quickstart on Linux with Java 9 runs but the connect tool
> throws a bunch of exceptions (https://www.codepile.net/pile/yVg8XJB8)
> -: Connect quickstart on Windows fails (Java 8:
> https://www.codepile.net/pile/xJGra6BP, Java 9:
> https://www.codepile.net/pile/oREYeORK)
>
> Given that Windows is not an officially supported platform, and the
> exceptions with Linux/Java 9 are not breaking the functionality, my vote
> is a +1 (non-binding).
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Damian Guy 
> To: dev@kafka.apache.org, us...@kafka.apache.org,
> kafka-clie...@googlegroups.com
> Date:   03/15/2018 07:55 AM
> Subject:Subject: [VOTE] 1.1.0 RC3
>
>
>
> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.0.
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D75957546&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=cKgJtQXXRauZ3HSAoSbsC9SLVTAkO-pbLdPrOCBuJzE&e=
>
>
> A few highlights:
>
> * Significant Controller improvements (much faster and session expiration
> edge cases fixed)
> * Data balancing across log directories (JBOD)
> * More efficient replication when the number of partitions is large
> * Dynamic Broker Configs
> * Delegation tokens (KIP-48)
> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>
>
> Release notes for the 1.1.0 release:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_RELEASE-5FNOTES.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=26FgbzRhKImhxyEkB4KzDPG-l8W_Y99xU6LykOAgpFI&e=
>
>
> *** Please download, test and vote by Monday, March 19, 9am PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=xlnrfgxVFMRCKk8pTOhujyC-Um4ogtsxK6Xwks6mc3U&e=
>
>
> * Release artifacts to be voted upon (source and binary):
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=ulHUeYnWIp28Gsn4VV1NK3FrGV4Jn5rUpuU6tvgekME&e=
>
>
> * Maven artifacts to be voted upon:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_groups_staging_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=G9o4hXVXF0bjL_a3Wocod9GUEfy9WBBgoGa2u6yFKQw&e=
>
>
> * Javadoc:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_javadoc_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=2auaI4IIJhEORGYm1Kdpxt5TDHh0PzSvtK77lC3SJVY&e=
>
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_tree_1.1.0-2Drc3&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=h7G8XPD8vAWl_gqySi2Iocag5NnP32IT_PyirPC3Lss&e=
>
>
>
> * Documentation:
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_11_documentation.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=KcPsL867-tPQxKPC4ufl5tqg9RSL3lxwsgKhOxeA7t0&e=
>
> <
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_1_documentation.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2G

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

[Debasish Ghosh]

>
> > Should this be 1.2  (maybe it's even better to not put any version at
> all)


Actually wanted to emphasize that the support is from 1.0.0 onwards ..
Should we make that explicit ? Like ..

kafka-streams-scala only depends on the Scala standard library and Kafka
> Streams 1.0.0+.


 In 1.1 release, we add a new module `kafka-streams-test-utils` to simplify
> testing for Kafka Streams applications. Are those test utils suitable for
> Scala users or should we add Scala wrappers for those, too?


I will check up and let you know ..

Also I am not clear about the decision on renaming of Scala abstractions.
Can we have a consensus on this ? Here's the summary ..

*Option 1:* Keep names separate (KStream for Java class, KStreamS for
Scala). No renaming of imports required.
*Option 2:* Unify names (KStream for Java and Scala class names). No
conflict since they will reside in different packages. But if we need to
use both abstractions, renaming of imports are required. But again, this
may not be a too frequent use case.

Suggestions ?

regards.

On Sat, Mar 17, 2018 at 3:07 AM, Matthias J. Sax 
wrote:

> Thanks a lot for the KIP! Two questions:
>
> 1) the KIP states:
>
> > kafka-streams-scala only depends on the Scala standard library and Kafka
> Streams 1.0.0.
>
> Should this be 1.2  (maybe it's even better to not put any version at all)
>
>
> 2) In 1.1 release, we add a new module `kafka-streams-test-utils` to
> simplify testing for Kafka Streams applications. Are those test utils
> suitable for Scala users or should we add Scala wrappers for those, too?
>
>
> -Matthias
>
>
> On 3/16/18 11:10 AM, Ted Yu wrote:
> > Import renames seem to be fine.
> >
> > The class names with trailing 'S' look clean.
> >
> > Cheers
> >
> > On Fri, Mar 16, 2018 at 11:04 AM, Ismael Juma  wrote:
> >
> >> If this is rare (as it sounds), relying on import renames seems fine to
> me.
> >> Let's see what others think.
> >>
> >> Ismael
> >>
> >> On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
> >> debasish.gh...@lightbend.com> wrote:
> >>
> >>> I am not sure if this is practical or not. But theoretically a user may
> >>> want to extract the unsafe Java abstraction from the Scala ones and use
> >>> Java APIs on them .. e.g.
> >>>
> >>> val userClicksStream: KStreamS[String, Long] =
> >>> builder.stream(userClicksTopic) // Scala abstraction
> >>>
> >>> val jStream: KStream[String, Long] = userClicksStream.inner //
> publishes
> >>> the underlying Java abstraction
> >>>
> >>> //.. work with Java, may be pass to some function written in Java
> >>>
> >>> I do realize this is somewhat of a convoluted use case and may not be
> >>> practically useful ..
> >>>
> >>> Otherwise we can very well work on the suggested approach of unifying
> the
> >>> names ..
> >>>
> >>> regards.
> >>>
> >>>
> >>>
> >>> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma 
> wrote:
> >>>
>  What does "mixed mode application" mean? What are the cases where a
> >> user
>  would want to use both APIs? I think that would help understand the
>  reasoning.
> 
>  Thanks,
>  Ismael
> 
>  On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
>  debasish.gh...@lightbend.com> wrote:
> 
> > Hi Damian -
> >
> > We could. But in case the user wants to use both Scala and Java APIs
> >>> (may
> > be for some mixed mode application), won't that be confusing ? She
> >> will
> > have to do something like ..
> >
> > import o.a.k.s.scala.{KStream => KStreamS}
> >
> > to rename Scala imports or the other way round for imported Java
> >>> classes.
> >
> > regards.
> >
> >
> >
> > On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy 
>  wrote:
> >
> >> Hi Debasish,
> >>
> >> Thanks for the KIP - will be a great addition to streams. I've only
>  had a
> >> quick scan, but seeing as the Scala classes are going to be in
> >> their
>  own
> >> package could we drop the S at the end of the class names?
> >>
> >> Thanks,
> >> Damian
> >>
> >>
> >> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > debasish.gh...@lightbend.com>
> >> wrote:
> >>
> >>> Hi -
> >>>
> >>> A new KIP, KIP-270 is up for discussion:
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >>>
> >>> The relevant JIRA issue: https://issues.apache.org/
> >> jira/browse/KAFKA-6670
> >>>
> >>> The library as proposed in the KIP has been implemented at
> >>> https://github.com/lightbend/kafka-streams-scala and the current
> > release
> >>> is
> >>> 0.2.0 (
> >>> https://github.com/lightbend/kafka-streams-scala/releases/
> >>> tag/v0.2.0
>  ).
> >>> We at Lightbend has been using it since quite some time now.
> >>>
> >>> regards.
> >>>
> >>> --
> >>> Debasish Ghosh
> >>> Principal Engineer
> >>>
> >>> Twitt

Build failed in Jenkins: kafka-trunk-jdk8 #2487

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: fix flaky Streams EOS system test (#4728)

--
[...truncated 3.53 MB...]

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > processNewResponseException STARTED

kafka.network.SocketServerTest > processNewResponseException PASSED

kafka.network.SocketServerTest > processCompletedSendException STARTED

kafka.network.SocketServerTest > processCompletedSendException PASSED

kafka.network.SocketServerTest > processDisconnectedException STARTED

kafka.network.SocketServerTest > processDisconnectedException PASSED

kafka.network.SocketServerTest > sendCancelledKeyException STARTED

kafka.network.SocketServerTest > sendCancelledKeyException PASSED

kafka.network.SocketServerTest > processCompletedReceiveException STARTED

kafka.network.SocketServerTest > processCompletedReceiveException PASSED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown STARTED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown PASSED

kafka.network.SocketServerTest > pollException STARTED

kafka.network.SocketServerTest > pollException PASSED

kafka.network.SocketServerTest > testSslSocketServer STARTED

kafka.network.SocketServerTest > testSslSocketServer PASSED

kafka.network.SocketServerTest > tooBigRequestIsRejected STARTED

kafka.network.SocketServerTest > tooBigRequestIsRejected PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdInvalidGroupId STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdInvalidGroupId PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteNonEmptyGroup STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteNonEmptyGroup PASSED

unit.kafka.admin.DeleteConsumerGroupTest > 
testDeleteCmdWithMixOfSuccessAndError STARTED

unit.kafka.admin.DeleteConsumerGroupTest > 
testDeleteCmdWithMixOfSuccessAndError PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteWithShortInitialization 
STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteWithShortInitialization 
PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdWithShortInitialization 
STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdWithShortInitialization 
PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteWithTopicOption STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteWithTopicOption PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteNonExistingGroup STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteNonExistingGroup PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdNonEmptyGroup STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdNonEmptyGroup PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdNonExistingGroup STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdNonExistingGroup PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteEmptyGroup STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteEmptyGroup PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteWithMixOfSuccessAndError 
STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteWithMixOfSuccessAndError 
PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdEmptyGroup STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteCmdEmptyGroup PASSED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteInvalidGroupId STARTED

unit.kafka.admin.DeleteConsumerGroupTest > testDeleteInvalidGroupId PASSED
:core:copyDependantLibs
:core:jar
:examples:compileJavawarning: [options] bootstrap class path not set in 
conjunction with -source 1.7
:19:
 warning: [deprecation] FetchRequest in kafka.api has been deprecated
import kafka.api.FetchRequest;
^
:20:
 warning: [deprecation] FetchRequestBuilder in kafka.api has been deprecated
import kafka.api.FetchRequestBuilder;
^
:22:
 warning: [deprecation] SimpleConsumer in kafka.javaapi.consumer has been 
deprecated
import kafka.javaapi.consumer.SimpleConsumer;
 ^
4 warnings

:examples:processResources NO-SOURCE
:examples:classes
:examples:checkstyleMain
:examples:compileTestJava NO-SOURCE
:examples:processTestResources NO-SOURCE
:examples:testClasses UP-TO-DATE
:examples:checkstyleTest NO-SOURCE
:examples:findbugsMain
:examples:test NO-SOURCE
:connect:api:compileJavawarning: [options] bootstrap class path not set in 
conjunction with -source 1.7
1 warning

:connect