[UPDATE] KIP-341: Update Sticky Assignor's User Data Protocol

[Vahid Hashemian]

Just a heads up to the community that the implementation of this KIP is
almost complete. I'd like to just mention that there was a slight deviation
in implementation from the approved KIP. I have updated the KIP to keep it
consistent with the final implementation.

To check what has changes please see this version comparison
.
Please let me know within the next couple of days if there is any objection
to this update. Otherwise, the corresponding PR will be merged to trunk in
its current form.

Thank you!
--Vahid

[jira] [Created] (KAFKA-8238) Log how many bytes and messages were read from __consumer_offsets

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-8238:
--

 Summary: Log how many bytes and messages were read from 
__consumer_offsets
 Key: KAFKA-8238
 URL: https://issues.apache.org/jira/browse/KAFKA-8238
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin P. McCabe


We should log how many bytes and messages were read from __consumer_offsets.  
Currently we only log how long it took.  Example: 
{code}
[GroupMetadataManager brokerId=2] Finished loading offsets and group metadata 
from __consumer_offsets-22 in 23131 milliseconds.
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSSION] KIP-421: Automatically resolve external configurations.

[Tejal Adsul]

Hi All,

I have updated the KIP to address the comments in the discussion. I have added 
the flow as to how  dynamic config values will be  resolved. Please could 
you’ll review the updated changes and let me know your feedback.

Thanks,
Tejal

On 2019/03/21 20:38:54, Tejal Adsul  wrote: 
> I have addressed the comments 1 and 2 in the KIP.> 
> 3. The example is a bit misleading with the password in it. I have modified 
> it. We basically wanted to show that you cam pass any additional parameters 
> required by the config provider> 
> 4. Yes  all the public config classes (ProducerConfig, ConsumerConfig, 
> ConnectorConfig etc.) will> > 
> >be extended to optionally use the new AbstractConfig constructors?>> 
> 
> 
> On 2019/03/14 11:49:46, Rajini Sivaram  wrote: > 
> > Hi Tejal,> > 
> > > 
> > Thanks for the updates. A few comments:> > 
> > > 
> > > 
> >1. In the standard KIP template, we have two sections `Public> > 
> >Interfaces` and `Proposed Changes`. Can you split the section 
> > `Proposal`> > 
> >into two so that public interface changes are more obvious?> > 
> >2. Under `Public Interfaces`, can you separate out interface changes 
> > and> > 
> >new configurations since the config changes are sort of lost in the 
> > text?> > 
> >In particular, I think this KIP is proposing to reserve the config name> 
> > > 
> >`config.providers` as well as all config names starting with> > 
> >`config.providers.` to resolve configs.> > 
> >3. The example looks a bit odd to me. It looks like we are removing> > 
> >local passwords like truststore password from a client config and 
> > instead> > 
> >adding a master password like vault password in cleartext into the 
> > file.> > 
> >Perhaps the intention is that the vault password won't be in the file 
> > for a> > 
> >vault provider?> > 
> >4. The example instantiates AbstractConfig. I am not familiar with the> 
> > > 
> >usage of this class in Connect, but is the intention that all the 
> > public> > 
> >config classes (ProducerConfig, ConsumerConfig, ConnectorConfig etc.) 
> > will> > 
> >be extended to optionally use the new AbstractConfig constructors?> > 
> > > 
> > Regards,> > 
> > > 
> > Rajini> > 
> > > 
> > > 
> > On Mon, Mar 11, 2019 at 5:49 PM Tejal Adsul  wrote:> > 
> > > 
> > > Hi Folks,> > 
> > >> > 
> > > I have accommodated most of the review comments for> > 
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-421%3A+Support+resolving+externalized+secrets+in+AbstractConfig>
> > >  > 
> > > . Reopening the thread for further discussion. Please let me know your> > 
> > > thoughts on it.> > 
> > >> > 
> > > Thanks,> > 
> > > Tejal> > 
> > >> > 
> > > On 2019/01/25 19:11:07, "Colin McCabe"  wrote:> > 
> > > > On Fri, Jan 25, 2019, at 09:12, Andy Coates wrote:>> > 
> > > > > > Further, if we're worried about confusion about how to)>> > 
> > > > > load the two files, we could have a constructor that does that> > 
> > > default>> > 
> > > > > pattern for you.>> > 
> > > > > >> > 
> > > > > Yeah, I don't really see the need for this two step / two file> > 
> > > approach. I>> > 
> > > > > think the config providers should be listed in the main property 
> > > > > file,> > 
> > > not>> > 
> > > > > some secondary file, and we should avoid backwards compatibility> > 
> > > issues by,>> > 
> > > > > as Ewan says, having a new constructor, (deprecating the old), that> 
> > > > > > 
> > > allows>> > 
> > > > > the functionality to be turned on/off.>> > 
> > > >> > 
> > > > +1.  In the case of the Kafka broker, it really seems like we should 
> > > > put> > 
> > > the config providers in the main config file. >> > 
> > > >  It's more complex to have multiple configuration files, and it 
> > > > doesn't> > 
> > > seem to add any value.>> > 
> > > >> > 
> > > > In the case of other components like Connect, I don't have a strong> > 
> > > opinion.  We can discuss this on a component-by-component basis.  
> > > Clearly> > 
> > > not all components manage configuration exactly the same way, and that> > 
> > > difference might motivate different strategies here.>> > 
> > > >> > 
> > > > > >> > 
> > > > > I suggest we also consider adding a new method to AbstractConfig to 
> > > > > >> > 
> > > > > allow>> > 
> > > > > applications to get the unresolved raw value, e.g. String>> > 
> > > > > getRawValue(String key).  Given a config entry like ">> > 
> > > > > config.providers.vault.password=$>> > 
> > > > > <> > 
> > > https://cwiki.apache.org/confluence/display/KAFKA/config.providers.vault.password=$>>>
> > >  > 
> > >> > 
> > > > > {file:/path/to/secrets.properties:vault.secret.password}" then >> > 
> > > > > getRawValue>> > 
> > > > > would always return "$>> > 
> > > > > <> > 
> > > https://cwiki.apache.org/confluence/display/KAFKA/config.providers.vault.password=$>>>
> > >  > 
> > >> > 
> > > > > {file:/path/to/secrets.properties:vault.secret.password}". I can see 
> > > > > >> > 
> > 

Request KIP Permissions

[Jukka Karvanen]

Hi,

Could you please grant me write access to KIP proposals?
I am planning to make KIP for KAFKA-8233: Helper class to make it simpler
to write test logic with TopologyTestDriver
Wiki ID: jkarvanen

Best regards,
Jukka Karvanen

Build failed in Jenkins: kafka-trunk-jdk11 #438

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-7778: Add KTable.suppress to Scala API (#6314)

[vahid.hashemian] KAFKA-7471: Multiple Consumer Group Management Feature (#5726)

--
[...truncated 2.37 MB...]
org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideNonPrefixedCustomConfigsWithPrefixedConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideNonPrefixedCustomConfigsWithPrefixedConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfNotAtLestOnceOrExactlyOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfNotAtLestOnceOrExactlyOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingConsumerIsolationLevelIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingConsumerIsolationLevelIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyOptimizationWhenNotExplicitlyAddedToConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyOptimizationWhenNotExplicitlyAddedToConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetGlobalConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetRestoreConsumerConfigsWithRestoreConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetRestoreConsumerConfigsWithRestoreConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionIfMaxInFlightRequestsPerConnectionIsInvalidStringIfEosEnabled
 STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionIfMaxInFlightRequestsPerConnectionIsInvalidStringIfEosEnabled
 PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 

[DISCUSS] KIP-421: Automatically resolve external configurations.

[TEJAL ADSUL]

Hi All,

I have updated the KIP to address the comments in the discussion. I have added 
the flow as to how  dynamic config values will be  resolved. Please could 
you’ll review the updated changes and let me know your feedback.

Thanks,
Tejal

On 2019/03/21 20:36:24, Tejal Adsul  wrote: 
> @Colin we won’t be supporting the subscriber mode currently and it will be 
> added as a future work
> 2. By disabling the feature the constructor will work as it earlier. If we 
> know the configs don’t have any indirect values or we want the indirect 
> values to remain unresolved we will can just do so by using the enable flag. 
> We won’t be using this in broker as its dynamic and we have added it in 
> future section for now. 
> 
> On 2019/03/14 16:36:42, "Colin McCabe"  wrote: 
> > Hi Tejal,> 
> > 
> > Thanks for the update.> 
> > 
> > One of the critical parts of the ConfigProvider interface is the ability to 
> > monitor changes to a configuration key through ConfigProvider#subscribe and 
> > ConfigProvider#unsubscribe, etc.  I don't see how the proposed API supports 
> > this.  Can you clarify?> 
> > 
> > Also, it's not clear to me when you would want to enable KIP-421 
> > functionality and when you would want to disable it.  What is the purpose 
> > of making it possible to disable this?  Do you have examples of cases where 
> > we would use it and cases where we would not?  Would the broker use this 
> > functionality?> 
> > 
> > best,> 
> > Colin> 
> > 
> > 
> > On Mon, Mar 11, 2019, at 10:49, Tejal Adsul wrote:> 
> > > Hi Folks,> 
> > > > 
> > > I have accommodated most of the review comments for > 
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-421%3A+Support+resolving+externalized+secrets+in+AbstractConfig
> > >  . Reopening the thread for further discussion. Please let me know your 
> > > thoughts on it.> 
> > > > 
> > > Thanks,> 
> > > Tejal> 
> > > > 
> > > On 2019/01/25 19:11:07, "Colin McCabe"  wrote: > 
> > > > On Fri, Jan 25, 2019, at 09:12, Andy Coates wrote:> > 
> > > > > > Further, if we're worried about confusion about how to)> > 
> > > > > load the two files, we could have a constructor that does that 
> > > > > default> > 
> > > > > pattern for you.> > 
> > > > > > > 
> > > > > Yeah, I don't really see the need for this two step / two file 
> > > > > approach. I> > 
> > > > > think the config providers should be listed in the main property 
> > > > > file, not> > 
> > > > > some secondary file, and we should avoid backwards compatibility 
> > > > > issues by,> > 
> > > > > as Ewan says, having a new constructor, (deprecating the old), that 
> > > > > allows> > 
> > > > > the functionality to be turned on/off.> > 
> > > > > 
> > > > +1.  In the case of the Kafka broker, it really seems like we should 
> > > > put the config providers in the main config file. > > 
> > > >  It's more complex to have multiple configuration files, and it doesn't 
> > > > seem to add any value.> > 
> > > > > 
> > > > In the case of other components like Connect, I don't have a strong 
> > > > opinion.  We can discuss this on a component-by-component basis.  
> > > > Clearly not all components manage configuration exactly the same way, 
> > > > and that difference might motivate different strategies here.> > 
> > > > > 
> > > > > > > 
> > > > > I suggest we also consider adding a new method to AbstractConfig to > 
> > > > > > 
> > > > > allow> > 
> > > > > applications to get the unresolved raw value, e.g. String> > 
> > > > > getRawValue(String key).  Given a config entry like "> > 
> > > > > config.providers.vault.password=$> > 
> > > > > >
> > > > >  > 
> > > > > {file:/path/to/secrets.properties:vault.secret.password}" then > > 
> > > > > getRawValue> > 
> > > > > would always return "$> > 
> > > > > >
> > > > >  > 
> > > > > {file:/path/to/secrets.properties:vault.secret.password}". I can see 
> > > > > > > 
> > > > > this> > 
> > > > > being useful.> > 
> > > > > 
> > > > I think one of the problems with the interface proposed in KIP-421 is 
> > > > that it doesn't give brokers any way to listen for changes to the 
> > > > configuration.  We've done a lot of work to make certain configuration 
> > > > keys dynamic, but we're basically saying if you use external secrets, 
> > > > you can't make use of that at all-- you have to restart the broker to 
> > > > change configuration.> > 
> > > > > 
> > > > Unfortunately, the AbstractConfig interface isn't well suited to 
> > > > listening for config changes.  In order to do that, you probably need 
> > > > to use the KIP-297 interface directly.  Which means that maybe we 
> > > > should go back to the drawing board here, unfortunately. :(> > 
> > > > > 
> > > > best,> > 
> > > > Colin> > 
> > > > > 
> > > > > > > 
> > > > > With regards to on-change subscription: surely all we'd need 

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

[Colin McCabe]

link: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment

C.

On Mon, Apr 15, 2019, at 18:07, Colin McCabe wrote:
> Hi all,
> 
> We've been having discussions on a few different KIPs (KIP-236, 
> KIP-435, etc.) about what the Admin Client replica reassignment API 
> should look like. The current API is really hard to extend and 
> maintain, which is a big source of problems. I think it makes sense to 
> have a KIP that establishes a clean API that we can use and extend 
> going forward, so I posted KIP-455. Take a look. :)
> 
> best,
> Colin
>

Build failed in Jenkins: kafka-trunk-jdk8 #3556

[Apache Jenkins Server]

See 

--
[...truncated 73 B...]
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read 81a23220014b719ff39bcb3d422ed489e461fe78
error: missing object referenced by 'refs/tags/2.2.0'
error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read bc745af6a7a640360a092d8bbde5f68b6ad637be
remote: Enumerating objects: 4038, done.
remote: Counting objects:   0% (1/4038)   remote: Counting objects:   
1% (41/4038)   remote: Counting objects:   2% (81/4038)   
remote: Counting objects:   3% (122/4038)   remote: Counting objects:   
4% (162/4038)   remote: Counting objects:   5% (202/4038)   
remote: Counting objects:   6% (243/4038)   remote: Counting objects:   
7% (283/4038)   remote: Counting objects:   8% (324/4038)   
remote: Counting objects:   9% (364/4038)   remote: Counting objects:  
10% (404/4038)   remote: Counting objects:  11% (445/4038)   
remote: Counting objects:  12% (485/4038)   remote: Counting objects:  
13% (525/4038)   remote: Counting objects:  14% (566/4038)   
remote: Counting objects:  15% (606/4038)   remote: Counting objects:  
16% (647/4038)   remote: Counting objects:  17% (687/4038)   
remote: Counting objects:  18% (727/4038)   remote: Counting objects:  
19% (768/4038)   remote: Counting objects:  20% (808/4038)   
remote: Counting objects:  21% (848/4038)   remote: Counting objects:  
22% (889/4038)   remote: Counting objects:  23% (929/4038)   
remote: Counting objects:  24% (970/4038)   remote: Counting objects:  
25% (1010/4038)   remote: Counting objects:  26% (1050/4038)   
remote: Counting objects:  27% (1091/4038)   remote: Counting objects:  
28% (1131/4038)   remote: Counting objects:  29% (1172/4038)   
remote: Counting objects:  30% (1212/4038)   remote: Counting objects:  
31% (1252/4038)   remote: Counting objects:  32% (1293/4038)   
remote: Counting objects:  33% (1333/4038)   remote: Counting objects:  
34% (1373/4038)   remote: Counting objects:  35% (1414/4038)   
remote: Counting objects:  36% (1454/4038)   remote: Counting objects:  
37% (1495/4038)   remote: Counting objects:  38% (1535/4038)   
remote: Counting objects:  39% (1575/4038)   remote: Counting objects:  
40% (1616/4038)   remote: Counting objects:  41% (1656/4038)   
remote: Counting objects:  42% (1696/4038)   remote: Counting objects:  
43% (1737/4038)   remote: Counting objects:  44% (1777/4038)   
remote: Counting objects:  45% (1818/4038)   remote: Counting objects:  
46% (1858/4038)   remote: Counting objects:  47% (1898/4038)   
remote: Counting objects:  48% (1939/4038)   remote: Counting objects:  
49% (1979/4038)   remote: Counting objects:  50% (2019/4038)   
remote: Counting objects:  51% (2060/4038)   remote: Counting objects:  
52% (2100/4038)   remote: Counting objects:  53% (2141/4038)   
remote: Counting objects:  54% 

Build failed in Jenkins: kafka-trunk-jdk11 #437

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6635; Producer close awaits pending transactions (#5971)

--
[...truncated 2.38 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

Build failed in Jenkins: kafka-trunk-jdk8 #3555

[Apache Jenkins Server]

See 

--
[...truncated 73 B...]
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read 81a23220014b719ff39bcb3d422ed489e461fe78
error: missing object referenced by 'refs/tags/2.2.0'
error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read bc745af6a7a640360a092d8bbde5f68b6ad637be
remote: Enumerating objects: 4038, done.
remote: Counting objects:   0% (1/4038)   remote: Counting objects:   
1% (41/4038)   remote: Counting objects:   2% (81/4038)   
remote: Counting objects:   3% (122/4038)   remote: Counting objects:   
4% (162/4038)   remote: Counting objects:   5% (202/4038)   
remote: Counting objects:   6% (243/4038)   remote: Counting objects:   
7% (283/4038)   remote: Counting objects:   8% (324/4038)   
remote: Counting objects:   9% (364/4038)   remote: Counting objects:  
10% (404/4038)   remote: Counting objects:  11% (445/4038)   
remote: Counting objects:  12% (485/4038)   remote: Counting objects:  
13% (525/4038)   remote: Counting objects:  14% (566/4038)   
remote: Counting objects:  15% (606/4038)   remote: Counting objects:  
16% (647/4038)   remote: Counting objects:  17% (687/4038)   
remote: Counting objects:  18% (727/4038)   remote: Counting objects:  
19% (768/4038)   remote: Counting objects:  20% (808/4038)   
remote: Counting objects:  21% (848/4038)   remote: Counting objects:  
22% (889/4038)   remote: Counting objects:  23% (929/4038)   
remote: Counting objects:  24% (970/4038)   remote: Counting objects:  
25% (1010/4038)   remote: Counting objects:  26% (1050/4038)   
remote: Counting objects:  27% (1091/4038)   remote: Counting objects:  
28% (1131/4038)   remote: Counting objects:  29% (1172/4038)   
remote: Counting objects:  30% (1212/4038)   remote: Counting objects:  
31% (1252/4038)   remote: Counting objects:  32% (1293/4038)   
remote: Counting objects:  33% (1333/4038)   remote: Counting objects:  
34% (1373/4038)   remote: Counting objects:  35% (1414/4038)   
remote: Counting objects:  36% (1454/4038)   remote: Counting objects:  
37% (1495/4038)   remote: Counting objects:  38% (1535/4038)   
remote: Counting objects:  39% (1575/4038)   remote: Counting objects:  
40% (1616/4038)   remote: Counting objects:  41% (1656/4038)   
remote: Counting objects:  42% (1696/4038)   remote: Counting objects:  
43% (1737/4038)   remote: Counting objects:  44% (1777/4038)   
remote: Counting objects:  45% (1818/4038)   remote: Counting objects:  
46% (1858/4038)   remote: Counting objects:  47% (1898/4038)   
remote: Counting objects:  48% (1939/4038)   remote: Counting objects:  
49% (1979/4038)   remote: Counting objects:  50% (2019/4038)   
remote: Counting objects:  51% (2060/4038)   remote: Counting objects:  
52% (2100/4038)   remote: Counting objects:  53% (2141/4038)   
remote: Counting objects:  54% 

Build failed in Jenkins: kafka-trunk-jdk8 #3554

[Apache Jenkins Server]

See 

--
[...truncated 570 B...]
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read 81a23220014b719ff39bcb3d422ed489e461fe78
error: missing object referenced by 'refs/tags/2.2.0'
error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read bc745af6a7a640360a092d8bbde5f68b6ad637be
remote: Enumerating objects: 4076, done.
remote: Counting objects:   0% (1/4076)   remote: Counting objects:   
1% (41/4076)   remote: Counting objects:   2% (82/4076)   
remote: Counting objects:   3% (123/4076)   remote: Counting objects:   
4% (164/4076)   remote: Counting objects:   5% (204/4076)   
remote: Counting objects:   6% (245/4076)   remote: Counting objects:   
7% (286/4076)   remote: Counting objects:   8% (327/4076)   
remote: Counting objects:   9% (367/4076)   remote: Counting objects:  
10% (408/4076)   remote: Counting objects:  11% (449/4076)   
remote: Counting objects:  12% (490/4076)   remote: Counting objects:  
13% (530/4076)   remote: Counting objects:  14% (571/4076)   
remote: Counting objects:  15% (612/4076)   remote: Counting objects:  
16% (653/4076)   remote: Counting objects:  17% (693/4076)   
remote: Counting objects:  18% (734/4076)   remote: Counting objects:  
19% (775/4076)   remote: Counting objects:  20% (816/4076)   
remote: Counting objects:  21% (856/4076)   remote: Counting objects:  
22% (897/4076)   remote: Counting objects:  23% (938/4076)   
remote: Counting objects:  24% (979/4076)   remote: Counting objects:  
25% (1019/4076)   remote: Counting objects:  26% (1060/4076)   
remote: Counting objects:  27% (1101/4076)   remote: Counting objects:  
28% (1142/4076)   remote: Counting objects:  29% (1183/4076)   
remote: Counting objects:  30% (1223/4076)   remote: Counting objects:  
31% (1264/4076)   remote: Counting objects:  32% (1305/4076)   
remote: Counting objects:  33% (1346/4076)   remote: Counting objects:  
34% (1386/4076)   remote: Counting objects:  35% (1427/4076)   
remote: Counting objects:  36% (1468/4076)   remote: Counting objects:  
37% (1509/4076)   remote: Counting objects:  38% (1549/4076)   
remote: Counting objects:  39% (1590/4076)   remote: Counting objects:  
40% (1631/4076)   remote: Counting objects:  41% (1672/4076)   
remote: Counting objects:  42% (1712/4076)   remote: Counting objects:  
43% (1753/4076)   remote: Counting objects:  44% (1794/4076)   
remote: Counting objects:  45% (1835/4076)   remote: Counting objects:  
46% (1875/4076)   remote: Counting objects:  47% (1916/4076)   
remote: Counting objects:  48% (1957/4076)   remote: Counting objects:  
49% (1998/4076)   remote: Counting objects:  50% (2038/4076)   
remote: Counting objects:  51% (2079/4076)   remote: Counting objects:  
52% (2120/4076)   remote: Counting objects:  53% (2161/4076)   
remote: Counting objects:  54% (2202/4076)   remote: Counting objects:  
55% (2242/4076)   remote: Counting objects:  56% (2283/4076)   

Build failed in Jenkins: kafka-trunk-jdk8 #3553

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6635; Producer close awaits pending transactions (#5971)

--
[...truncated 2.36 MB...]

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > testStringHeaderToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > testStringHeaderToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToConnectOptional 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > decimalToConnectOptional 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > structSchemaIdentical STARTED

org.apache.kafka.connect.json.JsonConverterTest > structSchemaIdentical PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnectWithDefaultValue 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnectWithDefaultValue 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
dateToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
dateToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnectOptional STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnectOptional PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
timeToConnectOptionalWithDefaultValue STARTED

[jira] [Resolved] (KAFKA-8228) Exactly once semantics break during server restart for kafka-streams application

[Boquan Tang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boquan Tang resolved KAFKA-8228.

Resolution: Duplicate

This might duplicate KAFKA-7866, close for now and watch that ticket.

> Exactly once semantics break during server restart for kafka-streams 
> application
> 
>
> Key: KAFKA-8228
> URL: https://issues.apache.org/jira/browse/KAFKA-8228
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.2.0
>Reporter: Boquan Tang
>Priority: Major
>
> We are using 2.2.0 for kafka-streams client and 2.0.1 for server.
> We have a simple kafka-streams application that has the following topology:
> {code:java}
> Source: KSTREAM-SOURCE-04 (topics: [deduped-adclick]) 
> --> KSTREAM-TRANSFORM-05 
> Processor: KSTREAM-TRANSFORM-05 (stores: [uid-offset-store]) 
> --> KSTREAM-TRANSFORM-06 
> <-- KSTREAM-SOURCE-04 
> Source: KSTREAM-SOURCE-00 (topics: [advertiser-budget]) 
> --> KTABLE-SOURCE-01 
> Source: KSTREAM-SOURCE-02 (topics: [advertisement-budget]) 
> --> KTABLE-SOURCE-03 
> Processor: KSTREAM-TRANSFORM-06 (stores: [advertiser-budget-store, 
> advertisement-budget-store]) 
> --> KSTREAM-SINK-07 
> <-- KSTREAM-TRANSFORM-05 
> Sink: KSTREAM-SINK-07 (topic: budget-adclick) 
> <-- KSTREAM-TRANSFORM-06 
> Processor: KTABLE-SOURCE-01 (stores: [advertiser-budget-store]) 
> --> none 
> <-- KSTREAM-SOURCE-00 
> Processor: KTABLE-SOURCE-03 (stores: [advertisement-budget-store]) 
> --> none 
> <-- KSTREAM-SOURCE-02{code}
> The *Processor: KSTREAM-TRANSFORM-05 (stores: [uid-offset-store])* is 
> added additionally to investigate this EOS broken issue, and its transform() 
> is like this (specific K V class name is removed):
> {code:java}
> public void init(final ProcessorContext context) {
> uidStore = (WindowStore) 
> context.getStateStore(uidStoreName);
> this.context = context;
> }
> public KeyValue transform(final K key, final V value) {
> final long offset = context.offset();
> final String uid = value.getUid();
> final long beginningOfHour = 
> Instant.ofEpochMilli(clickTimestamp).atZone(ZoneId.systemDefault()).withMinute(0).withSecond(0).toEpochSecond()
>  * 1000;
> final Long maybeLastOffset = uidStore.fetch(uid, beginningOfHour);
> final boolean dupe = null != maybeLastOffset && offset == maybeLastOffset;
> uidStore.put(uid, offset, beginningOfHour);
> if (dupe) {
> LOGGER.warn("Find duplication in partition {}, uid is {}, current 
> offset is {}, last offset is {}",
> context.partition(),
> uid,
> offset,
> maybeLastOffset);
> statsEmitter.count("duplication");
> }
> return dupe ? null : new KeyValue<>(key, value);
> }
> {code}
> Although not 100% reproduce-able, we found that after we restart one or more 
> server on the cluster side, the duplication would be found:
> {code:java}
> 2019-04-12T07:12:58Z WARN [org.apache.kafka.clients.NetworkClient] 
> [kafka-producer-network-thread | 
> adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1-0_9-producer]
>  [Producer 
> clientId=adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1-0_9-producer,
>  transactionalId=adclick-budget-decorator-streams-0_9] Connection to node 2 
> (*:9092) could not be established. Broker may not be available.
> 2019-04-12T07:12:58Z WARN [org.apache.kafka.clients.NetworkClient] 
> [kafka-producer-network-thread | 
> adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1-0_9-producer]
>  [Producer 
> clientId=adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1-0_9-producer,
>  transactionalId=adclick-budget-decorator-streams-0_9] Connection to node 2 
> (*:9092) could not be established. Broker may not be available.
> 2019-04-12T07:14:02Z WARN [org.apache.kafka.clients.NetworkClient] 
> [kafka-producer-network-thread | 
> adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1-0_12-producer]
>  [Producer 
> clientId=adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1-0_12-producer,
>  transactionalId=adclick-budget-decorator-streams-0_12] Connection to node 2 
> (*:9092) could not be established. Broker may not be available.
> 2019-04-12T07:27:39Z WARN 
> [org.apache.kafka.streams.processor.internals.StreamThread] 
> [adclick-budget-decorator-streams-e70f7538-4125-4e8f-aeee-0c8717d663bb-StreamThread-1]
>  stream-thread 
> 

[DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

[Colin McCabe]

Hi all,

We've been having discussions on a few different KIPs (KIP-236, KIP-435, etc.) 
about what the Admin Client replica reassignment API should look like.  The 
current API is really hard to extend and maintain, which is a big source of 
problems.  I think it makes sense to have a KIP that establishes a clean API 
that we can use and extend going forward, so I posted KIP-455.  Take a look.  :)

best,
Colin

Re: [DISCUSS] KIP-449: Add connector contexts to Connect worker logs

[Randall Hauch]

Thanks, Konstantine. I'm not sure it's within the scope of this PR to
address the `worker.id` MDC parameter, especially because the behavior is
not yet well defined as to what happens when the `client.id` is set to a
non-unique value. I guess I'd prefer to limit the scope to just the
additional connector-specific contexts.

Best regards,

Randall

On Mon, Apr 15, 2019 at 2:03 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Thanks for the quick updates on the KIP Randall!
>
> 3. Indeed. I thought I remembered some ambiguity in the numbering of tasks,
> but looking closer I think we are fine, especially now that you mention
> explicitly 0-based indexing for task ids.
> 4. Fine with the example as is.
> 5. Sounds good.
> 6. "client.id" is set by default. It's just that it's not unique across
> the
> connect cluster by default. I think it's ok to use it as a default value
> for "worker.id" in MDC. Regarding placement, I'd consider adding it
> wherever there's the least redundancy in logging and also make it
> optional/configurable based on the logger's conversion pattern.
> Up to you if you think there's room for inclusion in this KIP.
>
> Konstantine
>
>
>
> On Fri, Apr 12, 2019 at 3:24 PM Randall Hauch  wrote:
>
> > Thanks for the review and feedback, Konstantine.
> >
> > 1. Great suggestion. I've updated the KIP to hopefully make it more clear
> > that the uncommented line is unchanged from the existing Log4J
> > configuration file.
> >
> > 2. Regarding including a `-` before the task number is acceptable if it
> > makes it easier to, read and filter. I've updated the KIP and PR to
> > incorporate your suggestion.
> >
> > 3. Task numbers do start at 0, as seen by the DistributedHerder code that
> > creates the tasks (
> >
> >
> https://github.com/apache/kafka/blob/02221bd907a23041c95ce6446986bff631652b3a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L608-L611
> > ).
> > I've updated the KIP to highlight that task numbers are 0-based. As you
> can
> > see in the sample log included in the KIP, "task-0" corresponds to the
> > first task, not to the connector. In fact, the following are examples of
> > the log messages with the "worker" scope where the Connector
> implementation
> > is called:
> >
> > [2019-04-02 17:01:38,315] INFO [local-file-source|worker] Creating
> > connector local-file-source of type FileStreamSource
> > (org.apache.kafka.connect.runtime.Worker:227)
> > [2019-04-02 17:01:38,317] INFO [local-file-source|worker] Instantiated
> > connector local-file-source with version 2.3.0-SNAPSHOT of type class
> > org.apache.kafka.connect.file.FileStreamSourceConnector
> > (org.apache.kafka.connect.runtime.Worker:230)
> > ...
> >
> > [2019-04-02 17:11:46,408] INFO [local-file-sink|worker] Stopping
> connector
> > local-file-sink (org.apache.kafka.connect.runtime.Worker:334)
> > [2019-04-02 17:11:46,408] INFO [local-file-sink|worker] Stopped connector
> > local-file-sink (org.apache.kafka.connect.runtime.Worker:350)
> >
> >
> > The FileStreamSourceConnector class does not actually log any messages
> > itself, but if it did  those would be with the
> "[local-file-source|worker]"
> > context and would include the name of the class and line number in
> > parentheses (instead of e.g.,
> > "org.apache.kafka.connect.runtime.Worker:230").
> >
> > 4. I thought about doing a more complex example, but IMO the extra log
> > lines made the *sample* in the KIP quite a bit longer harder to
> understand.
> > I thought it important to keep the KIP a bit more readable while showing
> > which scope appear for the different log messages. A second task would
> > essentially have the nearly the same log messages, just with a task
> number
> > in the scope.
> >
> > 5. The PR already changes the
> `runtime/src/test/resources/log4j.properties`
> > to include the connector context MDC parameter. Because that's not a
> public
> > API, I've not mentioned it in the KIP.
> >
> > 6. I guess I understand your goal - it's not always clear which worker is
> > being referenced. However, I'm not sure whether it's that valuable to
> > include the "client.id" (if set) just in the "worker" scope. That means
> > that it's maybe more useful to introduce a second MDC parameter (e.g.,
> "%{
> > worker.id}") and optionally include that on all log messages. We'd have
> to
> > set the MDC context in the code in each thread, which isn't too much
> > effort. The other advantage of this approach is that it doesn't have to
> be
> > configurable: you can control it via your own logging configuration file
> > (rather than optionally including it in the "worker" scope on some of the
> > log messages). Thoughts? What would the "%{worker.id}" MDC value be set
> to
> > if "client.id" is not set?
> >
> > Final notes: Removed the placeholders, and corrected the typos and
> grammar.
> >
> > Thanks again for the detailed review!
> >
> > On Fri, Apr 12, 2019 at 2:05 PM Konstantine Karantasis <

[jira] [Created] (KAFKA-8237) Untangle TopicDeletionManager and add test cases

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-8237:
--

 Summary: Untangle TopicDeletionManager and add test cases
 Key: KAFKA-8237
 URL: https://issues.apache.org/jira/browse/KAFKA-8237
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson


There are a few circular dependencies involving `TopicDeletionManager`. For 
example, both `PartitionStateMachine` and `ReplicaStateMachine` depend on 
`TopicDeletionManager` while it also depends on them. This makes testing 
difficult and so there are no unit tests. We should fix this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

[Sophie Blee-Goldman]

Thanks for the feedback Matthias and Bill. After discussing offline we
realized the type of windows I originally had in mind were quite different,
and I agree now that the semantics outlined by Matthias are the direction
to go in here. I will update the KIP accordingly with the new semantics
(and corresponding design) and restart the discussion from there.

In the meantime, to respond to some other points:

1) API:

I propose adding only the one class -- public class SlidingWindows extends
Windows {} --  so I do not believe we need any new Serdes? It
will still be a fixed size TimeWindow, but handled a bit differently. I've
updated the KIP to state explicitly all of the classes/methods being added

2) Zero grace period

The "zero grace period" was essentially just consequence of my original
definition for sliding windows; with the new semantics we can (and should)
allow for a nonzero grace period

3) Wall-clock time

Hm, I had not considered this yet but it may be a good idea to keep in mind
while rethinking the design. To clarify, we don't support wall-clock based
aggregations with hopping or tumbling windows though (yet?)

4) Commutative vs associative vs invertible aggregations

I agree that it's reasonable to assume commutativity and associativity, but
that's not the same as being subtractable -- that requires invertibility,
which is broken by a lot of very simple functions and is not, I think, ok
to assume. However we could consider adding a separate API which also takes
a subtractor and corresponds to a completely different implementation. We
could also consider an API that takes a function that aggregates two
aggregates together in addition to the existing aggregator (which
aggregates a single value with an existing aggregate) WDYT?




On Thu, Apr 11, 2019 at 1:13 AM Matthias J. Sax 
wrote:

> Thanks for the KIP Sophie. Couple of comments:
>
> It's a little unclear to me, what public API you propose. It seems you
> want to add
>
> > public class SlidingWindow extends TimeWindow {}
>
> and
>
> > public class SlidingWindows extends TimeWindows {} // or maybe `extends
> Windows`
>
> If yes, should we add corresponding public Serdes classes?
>
> Also, can you list all newly added classes/methods explicitly in the wiki?
>
>
> About the semantics of the operator.
>
> > "Only one single window is defined at a time,"
>
> Should this be "one window per key" instead?
>
> I agree that both window boundaries should be inclusive. However, I am
> not sure about:
>
> > At most one record is forwarded when new data arrives
>
> (1) For what case, no output would be produced?
>
> (2) I think, if we advance in time, it can also happen that we emit
> multiple records. If a window "slides" (not "hops"), we cannot just
> advance it to the current record stream time but would need to emit more
> result if records expire before the current input record is added. For
> example, consider a window with size 5ms, and the following ts (all
> records have the same key):
>
> 1 2 3 10 11
>
> This should result in windows:
>
> [1]
> [1,2]
> [1,2,3]
> [2,3]
> [3]
> [10]
> [10,11]
>
> Ie, when the record with ts=10 is processed, it will trigger the
> computation of [2,3], [3] and [10].
>
>
> About out-of-order handling: I am wondering, if the current design that
> does not allow any grace period is too restrictive. Can you elaborate
> more on the motivation for this suggestions?
>
>
> Can you give more details about the "simple design"? Atm, it's not clear
> to me how it works. I though we always need to store all raw values. If
> we only store the current aggregate, would we end up with the same
> inefficient solution as using a hopping window with advance 1ms?
>
>
> For the O(sqrt(N)) proposal: can you maybe add an example with concrete
> bucket sizes, window size etc. The current proposal is a little unclear
> to me, atm.
>
>
> How are windows advance? Do you propose to advance all windows over all
> keys at the same time, or would each window (per key) advance
> independent from all other windows? What would be the pros/cons for both
> approaches?
>
>
> To add to Guozhang's comment: atm, DSL operators assume that aggregate
> functions are commutative and associative. Hence, it seems ok to make
> the same assumption for sliding window. Addressing holistic and
> non-subtractable aggregations should be supported out of the box at some
> point, too, but this would be a different KIP adding this to all
> existing aggregations.
>
>
> -Matthias
>
>
>
> On 4/9/19 4:38 PM, Guozhang Wang wrote:
> > Hi Sophie,
> >
> > Thanks for the proposed KIP. I've made a pass over it and here are some
> > thoughts:
> >
> > 1. "The window size is effectively the grace and retention period". The
> > grace time is defined as "the time to admit late-arriving events after
> the
> > end of the window." hence it is the additional time beyond the window
> size.
> > I guess your were trying to say it should be zero?
> >
> > Also for retention period, it is not 

Re: [DISCUSSION] KIP-422: Add support for user/client configuration in the Kafka Admin Client

[Colin McCabe]

Hi all,

In KIP-133: Describe and Alter Configs Admin APIs, there is "future work" 
section that explains:

> Future Work
> ...
>
 > 2. Support for reading and updating client, user and replication quotas. We 
 > initially included that in the KIP, but it subsequently became apparent 
 > that a separate protocol and AdminClient API would be more appropriate. 
 > The reason is that client/user quotas can be applied on a client id, user 
 > or (client id, user) tuple. In the future, the hierarchy may get even more 
 > complicated. So, it makes sense to keeping the API simple for the simple 
 > cases while introducing a more sophisticated API for the more complex case.

In other words, we deliberately didn't implement quotas through AlterConfigs 
because we felt like it the AlterConfigs API wasn't really complex enough to 
handle quotas well.

I think that the original discussion was correct here -- we should have a 
special API for quotas, rather than trying to shoehorn them into the 
AlterConfigs API.

For example, we could have an API like this:

>
> SetQuotasResults setQuotas(Map quotas, 
> SetQuotasOptions options)
>
> interface QuotaTarget {
> }
>
> class ClientQuotaTarget implements QuotaTarget {
>   String clientId;
> }
>
> class PrincipalQuotaTarget implements QuotaTarget {
>   String principal;
> }
>
> class ClientAndPrincipalQuotaTarget implements QuotaTarget {
>   String clientId;
>   String principal;
> }
>
> class QuotaLimit {
>long bytesWrittenPerSec;
>long bytesReadPerSec;
> }
>
> DescribeQuotasResults describeQuotas(QuotaTarget target, 
> DescribeQuotasOptions options);
>
> ListQuotasResults listQuotas(ListQuotasOptions options);
>

This would avoid the need to parse text strings.  It would also make it really 
clear how to use and extend the API.

best,
Colin

On Mon, Apr 8, 2019, at 05:29, Rajini Sivaram wrote:
> Hi Jun, Yaodong,
> 
> 21. The proposed approach sounds very hacky. User principals can contain
> arbitrary characters. So we can't simply split `user1/clients/clientA` into
> tokens using '/' as delimiter.  Internally, we sanitize names before
> storing in ZK, but the names provided by the user are actual principals and
> client-ids. I think we want to have entity names that explicitly specify
> (type, name) as in the CLI kafka-configs.sh and allow multiple entities to
> be specified together for (user, client-id). That will also enable us to
> configure defaults in a consistent way.
> 
> 22. Yes, that sounds reasonable.
> 
> On Fri, Apr 5, 2019 at 11:13 PM Jun Rao  wrote:
> 
> > Hi, Yaodong,
> >
> > Yes, what you proposed makes sense. A couple of more comments.
> >
> > 21.  Could you add those examples to the KIP wiki? It would also be useful
> > to know how to set the ConfigEntry value for quotas at
> > DefaultClientInUser, DefaultClientDefaultUser and DefaultUser level.
> >
> > 22. To support KIP-257, I guess we can just pass in some special string
> > value in ConfigEntry value through alterConfig and let the customized
> > implementation of ClientQuotaCallback parse it. It would be useful to
> > document this. Does that sound reasonable, Rajini?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Apr 5, 2019 at 2:03 PM Yaodong Yang 
> > wrote:
> >
> >> Hi Jun,
> >>
> >> The proposal we have right now is to directly set the quota through
> >> existing admin client APIs. See following examples:
> >>
> >> Example 1: set a user quota
> >>
> >> AdminClient adminClient = mock(AdminClient.class);
> >> Map configs = new HashMap();
> >> Config config = new Config(Arrays.asList(new ConfigEntry("user1",
> >> "producer_byte_rate=1024")));
> >> configs.put(singletonMap(ConfigResource.USER, config));
> >> adminClient.alterConfigs(configs);
> >>
> >>
> >> Example 2: set a client id quota
> >>
> >> AdminClient adminClient = mock(AdminClient.class);
> >> Map configs = new HashMap();
> >> Config config = new Config(Arrays.asList(new ConfigEntry("client1",
> >> "producer_byte_rate=1024")));
> >> configs.put(singletonMap(ConfigResource.CLIENT, config));
> >> adminClient.alterConfigs(configs);
> >>
> >>
> >> Example 3: set a  quota
> >>
> >> AdminClient adminClient = mock(AdminClient.class);
> >> Map configs = new HashMap();
> >> Config config = new Config(Arrays.asList(new
> >> ConfigEntry("user1/clients/client2", "producer_byte_rate=1024")));
> >> configs.put(singletonMap(ConfigResource.USER, config));
> >> adminClient.alterConfigs(configs);
> >>
> >>
> >> The current KIP is orthogonal to KIP 257. It only adds a new way to store
> >> the quotas in ZK through AdminClient. For customizable quotas, they will
> >> also be a property in User resources or Client resources.
> >>
> >> Quote from *CustomQuotaCallbackTest.scala::GroupedUserQuotaCallback* in
> >> the
> >> codebase, “Group quotas are configured in ZooKeeper as user quotas with
> >> the
> >> entity name "${group}_". Default group quotas are configured in ZooKeeper
> >> as user quotas with the entity name "_".”
> >> In this example, they are 

Re: [DISCUSS] KIP-449: Add connector contexts to Connect worker logs

[Konstantine Karantasis]

Thanks for the quick updates on the KIP Randall!

3. Indeed. I thought I remembered some ambiguity in the numbering of tasks,
but looking closer I think we are fine, especially now that you mention
explicitly 0-based indexing for task ids.
4. Fine with the example as is.
5. Sounds good.
6. "client.id" is set by default. It's just that it's not unique across the
connect cluster by default. I think it's ok to use it as a default value
for "worker.id" in MDC. Regarding placement, I'd consider adding it
wherever there's the least redundancy in logging and also make it
optional/configurable based on the logger's conversion pattern.
Up to you if you think there's room for inclusion in this KIP.

Konstantine



On Fri, Apr 12, 2019 at 3:24 PM Randall Hauch  wrote:

> Thanks for the review and feedback, Konstantine.
>
> 1. Great suggestion. I've updated the KIP to hopefully make it more clear
> that the uncommented line is unchanged from the existing Log4J
> configuration file.
>
> 2. Regarding including a `-` before the task number is acceptable if it
> makes it easier to, read and filter. I've updated the KIP and PR to
> incorporate your suggestion.
>
> 3. Task numbers do start at 0, as seen by the DistributedHerder code that
> creates the tasks (
>
> https://github.com/apache/kafka/blob/02221bd907a23041c95ce6446986bff631652b3a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L608-L611
> ).
> I've updated the KIP to highlight that task numbers are 0-based. As you can
> see in the sample log included in the KIP, "task-0" corresponds to the
> first task, not to the connector. In fact, the following are examples of
> the log messages with the "worker" scope where the Connector implementation
> is called:
>
> [2019-04-02 17:01:38,315] INFO [local-file-source|worker] Creating
> connector local-file-source of type FileStreamSource
> (org.apache.kafka.connect.runtime.Worker:227)
> [2019-04-02 17:01:38,317] INFO [local-file-source|worker] Instantiated
> connector local-file-source with version 2.3.0-SNAPSHOT of type class
> org.apache.kafka.connect.file.FileStreamSourceConnector
> (org.apache.kafka.connect.runtime.Worker:230)
> ...
>
> [2019-04-02 17:11:46,408] INFO [local-file-sink|worker] Stopping connector
> local-file-sink (org.apache.kafka.connect.runtime.Worker:334)
> [2019-04-02 17:11:46,408] INFO [local-file-sink|worker] Stopped connector
> local-file-sink (org.apache.kafka.connect.runtime.Worker:350)
>
>
> The FileStreamSourceConnector class does not actually log any messages
> itself, but if it did  those would be with the "[local-file-source|worker]"
> context and would include the name of the class and line number in
> parentheses (instead of e.g.,
> "org.apache.kafka.connect.runtime.Worker:230").
>
> 4. I thought about doing a more complex example, but IMO the extra log
> lines made the *sample* in the KIP quite a bit longer harder to understand.
> I thought it important to keep the KIP a bit more readable while showing
> which scope appear for the different log messages. A second task would
> essentially have the nearly the same log messages, just with a task number
> in the scope.
>
> 5. The PR already changes the `runtime/src/test/resources/log4j.properties`
> to include the connector context MDC parameter. Because that's not a public
> API, I've not mentioned it in the KIP.
>
> 6. I guess I understand your goal - it's not always clear which worker is
> being referenced. However, I'm not sure whether it's that valuable to
> include the "client.id" (if set) just in the "worker" scope. That means
> that it's maybe more useful to introduce a second MDC parameter (e.g., "%{
> worker.id}") and optionally include that on all log messages. We'd have to
> set the MDC context in the code in each thread, which isn't too much
> effort. The other advantage of this approach is that it doesn't have to be
> configurable: you can control it via your own logging configuration file
> (rather than optionally including it in the "worker" scope on some of the
> log messages). Thoughts? What would the "%{worker.id}" MDC value be set to
> if "client.id" is not set?
>
> Final notes: Removed the placeholders, and corrected the typos and grammar.
>
> Thanks again for the detailed review!
>
> On Fri, Apr 12, 2019 at 2:05 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Thanks for the KIP Randall.
> >
> > It might not be obvious right away, but this is a great improvement when
> > running Connect with multiple connectors or when debugging Connect, for
> > instance in integration tests or system tests. KIP looks good to me
> > overall, I have just a few comments below:
> >
> > 1. In the snippet of the config, by including an uncommented line, maybe
> > it's not immediately clear that this line is an existing line in
> > connect-log4j.properties and not an addition. Should this be mentioned
> in a
> > separate code block or in a different way?
> >
> > 2. Currently 

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

[Colin McCabe]

On Mon, Apr 8, 2019, at 06:38, Sönke Liebau wrote:
> Hi Colin,
> 
> quick summary up front: I totally agree, and always have! I think we
> misunderstood each other a little :)
> I was never really opposed the idea of restricting which ACL features can
> be used, I was just opposed to doing it specifically for this change, while
> not introducing real versioning. But when I suggested a few things around
> how to implement versioning back in January [1] and got no replies on those
> suggestions I figured no one was really interested in that, so I fell back
> to the minimal idea of not doing anything about it. I am absolutely happy
> to pick this up.
> 
> I've done some more digging through the code and feel plenty stupid at the
> moment, because there actually already is an ACL version [2]. Then again,
> good news, we can just use this. Currently loading ACLs from Zookeeper
> fails if the hard-coded version in the authorizer differs from what is
> stored in Zookeeper [3].

Hi Sonke,

Thanks for doing the research!  I remember that we went back and forth on 
whether we should move this over to JSON when implementing the KIP-290 work.  I 
didn't remember where we landed.  I am certainly glad we did land on JSON in 
the end!  Now the last thing to confirm is that all ACLs are created with the 
"extended" format, even the non-prefix ones.  I remember this being a point of 
discussion, don't remember where we landed on it.

> Following your idea of using the inter-broker protocol, we probably need to
> create a mapping of IBP -> ACL version somewhere, so we would have
> something like ACL_2 -> KAFKA_2_3 which defines that this version requires
> the IBP to be at least the stated version. When we make a change that
> requires a new ACL version, we simply add a new mapping to the appropriate
> API version and change the hard-coded version value in the authorizer.
> One potential issue that I see with this is that I am unsure if the
> Authorizer will actually have access to the configured IBP version when it
> is running from the kafka-acls.sh tool. I'll look for a way, but I think
> this might actually not be possible, in "legacy" mode, when writing ACLs
> directly to Zookeeper.

Yeah, I don't think it's possible.  In any case, there are no compatibility 
guarantees when writing directly to ZK.  This is one of the reasons why we're 
phasing out direct ZK access (KIP-4 has many more)  kafka-acls.sh is kind of 
unusual in still providing direct ZK access as an option.  We will have to 
phase that out eventually-- perhaps in the next major release.

> 
> The biggest take away that I have from this is, that we will probably need
> to change the zk path after all. If we do not do this, older brokers using
> the ACLAuthorizer will simply fail to start if someone creates an ACL with
> a newer version. And I am not sure we can actually prohibit them from doing
> that, as mentioned above.

I think the current behavior is OK.  In general, once you bump the IBP, 
downgrade is not possible any more.  A tool to downgrade clusters is an 
interesting idea... which doesn't currently exist.  It would be pretty tricky 
to write because you would need to do things like rewrite logs written in newer 
message formats, and so on.  Or maybe just delete that data?

best,
Colin

> 
> Maybe an alternative would be to keep the current logic as is, i.e. fail on
> presence of an ACL version != current version and add a "migrate" command
> to the acl commandline tool - at the cost of not being able to downgrade
> again, unless we make the migrate tool bi-directional..
> 
> I am unsure of the best way to be honest, will do some more looking through
> the code and thinking. But thought I'd share this in case you have some
> thoughts.
> 
> Best regards,
> Sönke
> 
> [1]
> https://lists.apache.org/thread.html/4936d5205852d0db19653d17aa9821d4ba30f077aee528bb90955ce4@%3Cdev.kafka.apache.org%3E
> [2]
> https://github.com/apache/kafka/blob/b4fa87cc51f0a7d640dc6ae2cc8f89f006aae652/core/src/main/scala/kafka/security/auth/Acl.scala#L37
> [3]
> https://github.com/apache/kafka/blob/b4fa87cc51f0a7d640dc6ae2cc8f89f006aae652/core/src/main/scala/kafka/security/auth/Acl.scala#L66
> 
> 
> On Fri, Apr 5, 2019 at 8:09 AM Colin McCabe  wrote:
> 
> > On Thu, Apr 4, 2019, at 01:52, Sönke Liebau wrote:
> > > Hi Colin,
> > >
> > > I agree, we need to have a way of failing incorrect ranges server-side,
> > > I'll amend the KIP and look into that. I think INVALID_REQUEST should fit
> > > fine, afaik we can send a message along with that code, so that could
> > > explain the actual reason.
> >
> > Hi Sonke,
> >
> > Sounds good.
> >
> > >
> > > Regarding prohibiting these ACLs from being created before the
> > inter-broker
> > > protocol is updated, I am a bit hesitant about this for two reasons.
> > >
> > > 1. I can't help but feel that we are mixing things in with each other
> > here
> > > that don't really belong together. The broker protocol and ACL versions
> > and
> 

Re: [DISCUSS] KIP-449: Add connector contexts to Connect worker logs

[Chris Egerton]

Hi Randall,

Thanks for the KIP. Debugging Connect workers definitely becomes harder as
the number of connectors and tasks increases, and your proposal would
simplify the process of sifting through logs and finding relevant
information faster and more accurately.

I have a couple small comments:

First--I believe the example snippet in your KIP under the "Public
Interfaces" header is inaccurate:
`[my-connector|worker]` - used on log messages where the Connect worker is
validating the configuration for or starting/stopping the
"local-file-source" connector via the SourceConnector / SinkConnector
implementation methods.
`[my-connector|task-0]` - used on log messages where the Connect worker is
executing task 0 of the "local-file-source" connector, including calling
any of the SourceTask / SinkTask implementation methods, processing the
messages for/from the task, and calling the task's producer/consumer.
`[my-connector|task-0|offsets]` - used on log messages where the Connect
worker is committing source offsets for task 0 of the "local-file-source"
connector.
The sample contexts mention "my-connector" but their explanations say that
they correspond to "local-file-source"; shouldn't the two align?

Second--I'm unclear on whether we expect (or want to encourage) developers
to manipulate the "connector.context" MDC key themselves, from with
connectors, transforms, etc. If we want to encourage this (in order to make
debugging even easier, which would align with the intent behind this KIP),
we may want to expose the LoggingContext class in the Connect API package
and expand on it so that users can set the context themselves. This would
be especially helpful in connectors with multithreaded logic. However, if
that would expand the scope of this KIP too much I think we could afford to
address that later.

Cheers,

Chris

[jira] [Created] (KAFKA-8236) Incorporate version control for Kafka Streams Application Reset

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-8236:
--

 Summary: Incorporate version control for Kafka Streams Application 
Reset
 Key: KAFKA-8236
 URL: https://issues.apache.org/jira/browse/KAFKA-8236
 Project: Kafka
  Issue Type: Improvement
Reporter: Boyang Chen


Inspired by Spark mlflow which supports versioning log, we should be 
considering expose a special versioning tag for KStream applications to easy 
rollback bad code deploy. The naive approach is to store the versioning info in 
consumer offset topic so that when we perform rollback, we know where to read 
from the input, and where to cleanup the changelog topic. Essentially, this is 
an extension to our current application reset tool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk11 #436

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

[Viktor Somogyi-Vass]

A follow up on the batching topic to clarify my points above.

Generally I think that batching should be a core feature as Colin said the
controller should possess all information that are related.
Also Cruise Control (or really any 3rd party admin system) might build upon
this to give more holistic approach to balance brokers. We may cater them
with APIs that act like building blocks to make their life easier like
incrementalization, batching, cancellation and rollback but I think the
more advanced we go we'll need more advanced control surface and Kafka's
basic tooling might not be suitable for that.

Best,
Viktor


On Mon, 15 Apr 2019, 18:22 Viktor Somogyi-Vass, 
wrote:

> Hey Guys,
>
> I'll reply to you all in this email:
>
> @Jun:
> 1. yes, it'd be a good idea to add this feature, I'll write this into the
> KIP. I was actually thinking about introducing a dynamic config called
> reassignment.parallel.partition.count and
> reassignment.parallel.replica.count. The first property would control how
> many partition reassignment can we do concurrently. The second would go one
> level in granularity and would control how many replicas do we want to move
> for a given partition. Also one more thing that'd be useful to fix is that
> a given list of partition -> replica list would be executed in the same
> order (from first to last) so it's overall predictable and the user would
> have some control over the order of reassignments should be specified as
> the JSON is still assembled by the user.
> 2. the /kafka/brokers/topics/{topic} znode to be specific. I'll update the
> KIP to contain this.
>
> @Jason:
> I think building this functionality into Kafka would definitely benefit
> all the users and that CC as well as it'd simplify their software as you
> said. As I understand the main advantage of CC and other similar softwares
> are to give high level features for automatic load balancing. Reliability,
> stability and predictability of the reassignment should be a core feature
> of Kafka. I think the incrementalization feature would make it more stable.
> I would consider cancellation too as a core feature and we can leave the
> gate open for external tools to feed in their reassignment json as they
> want. I was also thinking about what are the set of features we can provide
> for Kafka but I think the more advanced we go the more need there is for an
> administrative UI component.
> Regarding KIP-352: Thanks for pointing this out, I didn't see this
> although lately I was also thinking about the throttling aspect of it.
> Would be a nice add-on to Kafka since though the above configs provide some
> level of control, it'd be nice to put an upper cap on the bandwidth and
> make it monitorable.
>
> Viktor
>
> On Wed, Apr 10, 2019 at 2:57 AM Jason Gustafson 
> wrote:
>
>> Hi Colin,
>>
>> On a related note, what do you think about the idea of storing the
>> > reassigning replicas in
>> > /brokers/topics/[topic]/partitions/[partitionId]/state, rather than in
>> the
>> > reassignment znode?  I don't think this requires a major change to the
>> > proposal-- when the controller becomes aware that it should do a
>> > reassignment, the controller could make the changes.  This also helps
>> keep
>> > the reassignment znode from getting larger, which has been a problem.
>>
>>
>> Yeah, I think it's a good idea to store the reassignment state at a finer
>> level. I'm not sure the LeaderAndIsr znode is the right one though.
>> Another
>> option is /brokers/topics/{topic}. That is where we currently store the
>> replica assignment. I think we basically want to represent both the
>> current
>> state and the desired state. This would also open the door to a cleaner
>> way
>> to update a reassignment while it is still in progress.
>>
>> -Jason
>>
>>
>>
>>
>> On Mon, Apr 8, 2019 at 11:14 PM George Li > .invalid>
>> wrote:
>>
>> >  Hi Colin / Jason,
>> >
>> > Reassignment should really be doing a batches.  I am not too worried
>> about
>> > reassignment znode getting larger.  In a real production environment,
>> too
>> > many concurrent reassignment and too frequent submission of
>> reassignments
>> > seemed to cause latency spikes of kafka cluster.  So
>> > batching/staggering/throttling of submitting reassignments is
>> recommended.
>> >
>> > In KIP-236,  The "originalReplicas" are only kept for the current
>> > reassigning partitions (small #), and kept in memory of the controller
>> > context partitionsBeingReassigned as well as in the znode
>> > /admin/reassign_partitions,  I think below "setting in the RPC like
>> null =
>> > no replicas are reassigning" is a good idea.
>> >
>> > There seems to be some issues with the Mail archive server of this
>> mailing
>> > list?  I didn't receive email after April 7th, and the archive for April
>> > 2019 has only 50 messages (
>> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201904.mbox/thread)
>> ?
>> >
>> > Thanks,
>> > George
>> >
>> >on, 08 Apr 2019 17:54:48 GMT  Colin 

Re: [VOTE] KIP-422: Use the default value of max.poll.interval in Streams

[Jun Rao]

The voting thread seems to be referring to the wrong KIP number. It should
be KIP-442 instead of KIP-422.

Thanks,

Jun

On Wed, Apr 3, 2019 at 7:03 PM John Roesler  wrote:

> Thanks all. The KIP-442 vote has passed with 3 binding votes (Guozhang,
> Bill, and Damian) and one non-binding vote (me) in favor and none against.
>
> I'll update the KIP page.
>
> -John
>
> On Fri, Mar 29, 2019 at 10:29 AM Damian Guy  wrote:
>
> > +1
> >
> > On Wed, 27 Mar 2019 at 21:38, John Roesler  wrote:
> >
> > > Ah, good point, Guozhang. I'll remove that mention from the KIP.
> > >
> > > On Wed, Mar 27, 2019 at 3:30 PM Bill Bejeck  wrote:
> > >
> > > > +1 for me,
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > > > On Wed, Mar 27, 2019 at 4:13 PM Guozhang Wang 
> > > wrote:
> > > >
> > > > > +1 from me.
> > > > >
> > > > > Though note that we cannot make such changes in older versions
> since
> > > even
> > > > > if we release new versions out of those branches they are
> considered
> > > > > bug-fix only and hence should not have any interface impacting
> > changes.
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Mar 27, 2019 at 12:55 PM John Roesler 
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Since the KIP is so small, I'm going to optimistically start the
> > vote
> > > > for
> > > > > > KIP-422 to remove our "max int" default max.poll.interval.ms in
> > > > Streams
> > > > > > and
> > > > > > fall back to the Consumer default of five minutes.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-442%3A+Return+to+default+max+poll+interval+in+Streams
> > > > > >
> > > > > > Permalink: https://cwiki.apache.org/confluence/x/1COGBg
> > > > > >
> > > > > > See also: https://issues.apache.org/jira/browse/KAFKA-6399
> > > > > >
> > > > > > Please let me know if you have any objections and wish to return
> to
> > > the
> > > > > > discussion phase!
> > > > > >
> > > > > > Thanks,
> > > > > > -John
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

Jenkins build is back to normal : kafka-trunk-jdk8 #3551

[Apache Jenkins Server]

See 

Re: Kafka Source Code Debugging

[Kartik Kalaghatgi]

Checkout :
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup


Thanks,
Kartik

On Mon, 15 Apr 2019 at 22:03, Shubham Singh 
wrote:

> Hi Team,
>
> I wanted to debug Kafka github (Git hub link
> ) source code on my eclipse.
> Please let me know how to set up and make kafka run in debug mode.
>
> Thanks,
> Shubham
>

Kafka Source Code Debugging

[Shubham Singh]

Hi Team,

I wanted to debug Kafka github (Git hub link
) source code on my eclipse.
Please let me know how to set up and make kafka run in debug mode.

Thanks,
Shubham

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

[Viktor Somogyi-Vass]

Hey Guys,

I'll reply to you all in this email:

@Jun:
1. yes, it'd be a good idea to add this feature, I'll write this into the
KIP. I was actually thinking about introducing a dynamic config called
reassignment.parallel.partition.count and
reassignment.parallel.replica.count. The first property would control how
many partition reassignment can we do concurrently. The second would go one
level in granularity and would control how many replicas do we want to move
for a given partition. Also one more thing that'd be useful to fix is that
a given list of partition -> replica list would be executed in the same
order (from first to last) so it's overall predictable and the user would
have some control over the order of reassignments should be specified as
the JSON is still assembled by the user.
2. the /kafka/brokers/topics/{topic} znode to be specific. I'll update the
KIP to contain this.

@Jason:
I think building this functionality into Kafka would definitely benefit all
the users and that CC as well as it'd simplify their software as you said.
As I understand the main advantage of CC and other similar softwares are to
give high level features for automatic load balancing. Reliability,
stability and predictability of the reassignment should be a core feature
of Kafka. I think the incrementalization feature would make it more stable.
I would consider cancellation too as a core feature and we can leave the
gate open for external tools to feed in their reassignment json as they
want. I was also thinking about what are the set of features we can provide
for Kafka but I think the more advanced we go the more need there is for an
administrative UI component.
Regarding KIP-352: Thanks for pointing this out, I didn't see this although
lately I was also thinking about the throttling aspect of it. Would be a
nice add-on to Kafka since though the above configs provide some level of
control, it'd be nice to put an upper cap on the bandwidth and make it
monitorable.

Viktor

On Wed, Apr 10, 2019 at 2:57 AM Jason Gustafson  wrote:

> Hi Colin,
>
> On a related note, what do you think about the idea of storing the
> > reassigning replicas in
> > /brokers/topics/[topic]/partitions/[partitionId]/state, rather than in
> the
> > reassignment znode?  I don't think this requires a major change to the
> > proposal-- when the controller becomes aware that it should do a
> > reassignment, the controller could make the changes.  This also helps
> keep
> > the reassignment znode from getting larger, which has been a problem.
>
>
> Yeah, I think it's a good idea to store the reassignment state at a finer
> level. I'm not sure the LeaderAndIsr znode is the right one though. Another
> option is /brokers/topics/{topic}. That is where we currently store the
> replica assignment. I think we basically want to represent both the current
> state and the desired state. This would also open the door to a cleaner way
> to update a reassignment while it is still in progress.
>
> -Jason
>
>
>
>
> On Mon, Apr 8, 2019 at 11:14 PM George Li  .invalid>
> wrote:
>
> >  Hi Colin / Jason,
> >
> > Reassignment should really be doing a batches.  I am not too worried
> about
> > reassignment znode getting larger.  In a real production environment,
> too
> > many concurrent reassignment and too frequent submission of reassignments
> > seemed to cause latency spikes of kafka cluster.  So
> > batching/staggering/throttling of submitting reassignments is
> recommended.
> >
> > In KIP-236,  The "originalReplicas" are only kept for the current
> > reassigning partitions (small #), and kept in memory of the controller
> > context partitionsBeingReassigned as well as in the znode
> > /admin/reassign_partitions,  I think below "setting in the RPC like null
> =
> > no replicas are reassigning" is a good idea.
> >
> > There seems to be some issues with the Mail archive server of this
> mailing
> > list?  I didn't receive email after April 7th, and the archive for April
> > 2019 has only 50 messages (
> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201904.mbox/thread) ?
> >
> > Thanks,
> > George
> >
> >on, 08 Apr 2019 17:54:48 GMT  Colin McCabe wrote:
> >
> >   Yeah, I think adding this information to LeaderAndIsr makes sense.  It
> > would be better to track
> > "reassigningReplicas" than "originalReplicas", I think.  Tracking
> > "originalReplicas" is going
> > to involve sending a lot more data, since most replicas in the system are
> > not reassigning
> > at any given point.  Or we would need a hack in the RPC like null = no
> > replicas are reassigning.
> >
> > On a related note, what do you think about the idea of storing the
> > reassigning replicas in
> >  /brokers/topics/[topic]/partitions/[partitionId]/state, rather than in
> > the reassignment znode?
> >  I don't think this requires a major change to the proposal-- when the
> > controller becomes
> > aware that it should do a reassignment, the controller could make the
> > changes.  This also
> 

Jenkins build is back to normal : kafka-2.2-jdk8 #80

[Apache Jenkins Server]

See 

KAFKA-7471: Multiple Consumer Group Management PR merging with trunk

[Alex D]

Hello guys,

Any comments/updates on Multiple Consumer Group Management PR:
https://github.com/apache/kafka/pull/5726 ?

Best Regards,
Alex Dunayevsky

Build failed in Jenkins: kafka-trunk-jdk11 #435

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8232; Test topic delete completion rather than intermediate state

--
[...truncated 2.36 MB...]

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithIntValuesWithoutWhitespaceAsMap PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringListWithExtraDelimitersAndReturnString STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringListWithExtraDelimitersAndReturnString PASSED

org.apache.kafka.connect.data.ValuesTest > shouldFailToParseStringOfEmptyMap 
STARTED

org.apache.kafka.connect.data.ValuesTest > shouldFailToParseStringOfEmptyMap 
PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToConvertToListFromStringWithExtraDelimiters STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToConvertToListFromStringWithExtraDelimiters PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithShortValuesWithWhitespaceAsMap STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithShortValuesWithWhitespaceAsMap PASSED

org.apache.kafka.connect.data.ValuesTest > shouldParseStringsWithoutDelimiters 
STARTED

org.apache.kafka.connect.data.ValuesTest > shouldParseStringsWithoutDelimiters 
PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseStringOfMapWithIntValuesWithBlankEntry STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseStringOfMapWithIntValuesWithBlankEntry PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseStringOfMapWithIntValuesWithBlankEntries STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseStringOfMapWithIntValuesWithBlankEntries PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldConvertMapWithStringKeysAndIntegerValues STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldConvertMapWithStringKeysAndIntegerValues PASSED

org.apache.kafka.connect.data.ValuesTest > shouldConvertDateValues STARTED

org.apache.kafka.connect.data.ValuesTest > shouldConvertDateValues PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseInvalidBooleanValueString STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseInvalidBooleanValueString PASSED

org.apache.kafka.connect.data.ValuesTest > shouldConvertTimestampValues STARTED

org.apache.kafka.connect.data.ValuesTest > shouldConvertTimestampValues PASSED

org.apache.kafka.connect.data.ValuesTest > shouldConvertNullValue STARTED

org.apache.kafka.connect.data.ValuesTest > shouldConvertNullValue PASSED

org.apache.kafka.connect.data.ValuesTest > shouldConvertTimeValues STARTED

org.apache.kafka.connect.data.ValuesTest > shouldConvertTimeValues PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseStringOfMalformedMap STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldFailToParseStringOfMalformedMap PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithShortValuesWithoutWhitespaceAsMap STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithShortValuesWithoutWhitespaceAsMap PASSED

org.apache.kafka.connect.data.ValuesTest > shouldConvertListWithStringValues 
STARTED

org.apache.kafka.connect.data.ValuesTest > shouldConvertListWithStringValues 
PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringsWithEscapedDelimiters STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringsWithEscapedDelimiters PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldConvertStringOfListWithOnlyNumericElementTypesIntoListOfLargestNumericType
 STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldConvertStringOfListWithOnlyNumericElementTypesIntoListOfLargestNumericType
 PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithIntValuesWithWhitespaceAsMap STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringOfMapWithIntValuesWithWhitespaceAsMap PASSED

org.apache.kafka.connect.data.ValuesTest > shouldConvertListWithIntegerValues 
STARTED

org.apache.kafka.connect.data.ValuesTest > shouldConvertListWithIntegerValues 
PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldConvertStringOfListWithMixedElementTypesIntoListWithDifferentElementTypes 
STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldConvertStringOfListWithMixedElementTypesIntoListWithDifferentElementTypes 
PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldEscapeStringsWithEmbeddedQuotesAndBackslashes STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldEscapeStringsWithEmbeddedQuotesAndBackslashes PASSED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringListWithMultipleElementTypesAndReturnListWithNoSchema STARTED

org.apache.kafka.connect.data.ValuesTest > 
shouldParseStringListWithMultipleElementTypesAndReturnListWithNoSchema PASSED

org.apache.kafka.connect.data.ValuesTest > 

Re: [VOTE] KIP-419: Safely notify Kafka Connect SourceTask is stopped

[Edoardo Comar]

Thanks Andrew.

+1 (non-binding)

--

Edoardo Comar

IBM Event Streams
IBM UK Ltd, Hursley Park, SO21 2JN




From:   Mickael Maison 
To: dev 
Date:   10/04/2019 10:14
Subject:Re: [VOTE] KIP-419: Safely notify Kafka Connect SourceTask 
is stopped



+1 (non-binding)
Thanks for the KIP!

On Mon, Apr 8, 2019 at 8:07 PM Andrew Schofield
 wrote:
>
> Hi,
> I’d like to begin the voting thread for KIP-419. This is a minor KIP to 
add a new stopped() method to the SourceTask interface in Kafka Connect. 
Its purpose is to give the task a safe opportunity to clean up its 
resources, in the knowledge that this is the final call to the task.
>
> KIP: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D419-253A-2BSafely-2Bnotify-2BKafka-2BConnect-2BSourceTask-2Bis-2Bstopped=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=vBvXztcRTgKwMpQ54ziN_GoOo0_fHSvTEMoXwQABvfs=ParyN6mWVuOGJR7kA84NOshRJA2LAK6htiD2gqf-h_M=

> PR: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_pull_6551=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=vBvXztcRTgKwMpQ54ziN_GoOo0_fHSvTEMoXwQABvfs=R_udYap1tpd83ISv1Rh0TY6ttH6RuEIwQ0KwOFMB3zU=

> JIRA: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_KAFKA-2D7841=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ=vBvXztcRTgKwMpQ54ziN_GoOo0_fHSvTEMoXwQABvfs=5WqDQPU2J8yAxRXsjOgydtzJSE8yQCoB7qX0TtQyHA0=

>
> Thanks,
> Andrew Schofield
> IBM




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[Jira][Created](KAFKA-8100)kafka consumer not refresh metadata for dynamic topic deletion

[Shengnan YU]

Hi everyone:
I found that KafkaConsumer will not refresh its topic metadata when some unused 
topics are deleted, which leads to continuous UNKNOWN_TOPIC_PARTITION warning. 
In source code KafkaProducer will remove unused topic after a expire time but 
KafkaConsumer not. I know it may be a design that because 
UNKNOWN_TOPIC_PARTITION may not assert the non-existence of topic but is it 
possible to make the topic expirable configurable for KafkaConsumer? I think 
even the topic info is removed by mistake, it can still be discovered soon. 
Thank you very much!

Yu Shengnan

[jira] [Created] (KAFKA-8235) NoSuchElementException when restoring state after a clean shutdown of a Kafka Streams application

[Andrew Klopper (JIRA)]

Andrew Klopper created KAFKA-8235:
-

 Summary: NoSuchElementException when restoring state after a clean 
shutdown of a Kafka Streams application
 Key: KAFKA-8235
 URL: https://issues.apache.org/jira/browse/KAFKA-8235
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.3.0
 Environment: Linux, CentOS 7, Java 1.8.0_191, 50 partitions per topic, 
replication factor 3
Reporter: Andrew Klopper


While performing a larger scale test of a new Kafka Streams application that 
performs aggregation and suppression, we have discovered that we are unable to 
restart the application after a clean shutdown. The error that is logged is:
{code:java}
[rollupd-5ee1997b-8302-4e88-b3db-a74c8b805a6c-StreamThread-1] ERROR 
org.apache.kafka.streams.processor.internals.StreamThread - stream-thread 
[rollupd-5ee1997b-8302-4e88-b3db-a74c8b805a6c-StreamThread-1] Encountered the 
following error during processing:
java.util.NoSuchElementException
at java.util.TreeMap.key(TreeMap.java:1327)
at java.util.TreeMap.firstKey(TreeMap.java:290)
at 
org.apache.kafka.streams.state.internals.InMemoryTimeOrderedKeyValueBuffer.restoreBatch(InMemoryTimeOrderedKeyValueBuffer.java:288)
at 
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
at 
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:92)
at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:317)
at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92)
at 
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:328)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:866)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:804)
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:773)
{code}
The issue doesn't seem to occur for small amounts of data, but it doesn't take 
a particularly large amount of data to trigger the problem either.

Any assistance would be greatly appreciated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #3550

[Apache Jenkins Server]

See 

--
[...truncated 593 B...]
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1810)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read 81a23220014b719ff39bcb3d422ed489e461fe78
error: missing object referenced by 'refs/tags/2.2.0'
error: Could not read 79db30a8d7e3e85641c3c838f48b286cc7500814
error: Could not read bc745af6a7a640360a092d8bbde5f68b6ad637be
remote: Enumerating objects: 4002, done.
remote: Counting objects:   0% (1/4002)   remote: Counting objects:   
1% (41/4002)   remote: Counting objects:   2% (81/4002)   
remote: Counting objects:   3% (121/4002)   remote: Counting objects:   
4% (161/4002)   remote: Counting objects:   5% (201/4002)   
remote: Counting objects:   6% (241/4002)   remote: Counting objects:   
7% (281/4002)   remote: Counting objects:   8% (321/4002)   
remote: Counting objects:   9% (361/4002)   remote: Counting objects:  
10% (401/4002)   remote: Counting objects:  11% (441/4002)   
remote: Counting objects:  12% (481/4002)   remote: Counting objects:  
13% (521/4002)   remote: Counting objects:  14% (561/4002)   
remote: Counting objects:  15% (601/4002)   remote: Counting objects:  
16% (641/4002)   remote: Counting objects:  17% (681/4002)   
remote: Counting objects:  18% (721/4002)   remote: Counting objects:  
19% (761/4002)   remote: Counting objects:  20% (801/4002)   
remote: Counting objects:  21% (841/4002)   remote: Counting objects:  
22% (881/4002)   remote: Counting objects:  23% (921/4002)   
remote: Counting objects:  24% (961/4002)   remote: Counting objects:  
25% (1001/4002)   remote: Counting objects:  26% (1041/4002)   
remote: Counting objects:  27% (1081/4002)   remote: Counting objects:  
28% (1121/4002)   remote: Counting objects:  29% (1161/4002)   
remote: Counting objects:  30% (1201/4002)   remote: Counting objects:  
31% (1241/4002)   remote: Counting objects:  32% (1281/4002)   
remote: Counting objects:  33% (1321/4002)   remote: Counting objects:  
34% (1361/4002)   remote: Counting objects:  35% (1401/4002)   
remote: Counting objects:  36% (1441/4002)   remote: Counting objects:  
37% (1481/4002)   remote: Counting objects:  38% (1521/4002)   
remote: Counting objects:  39% (1561/4002)   remote: Counting objects:  
40% (1601/4002)   remote: Counting objects:  41% (1641/4002)   
remote: Counting objects:  42% (1681/4002)   remote: Counting objects:  
43% (1721/4002)   remote: Counting objects:  44% (1761/4002)   
remote: Counting objects:  45% (1801/4002)   remote: Counting objects:  
46% (1841/4002)   remote: Counting objects:  47% (1881/4002)   
remote: Counting objects:  48% (1921/4002)   remote: Counting objects:  
49% (1961/4002)   remote: Counting objects:  50% (2001/4002)   
remote: Counting objects:  51% (2042/4002)   remote: Counting objects:  
52% (2082/4002)   remote: Counting objects:  53% (2122/4002)   
remote: Counting objects:  54% (2162/4002)   remote: Counting objects:  
55% (2202/4002)   remote: Counting objects:  56% (2242/4002)   
remote: Counting objects:  57% (2282/4002)   remote: Counting objects:  

[jira] [Created] (KAFKA-8234) Multi-module support for JAAS config property

[Gabor Somogyi (JIRA)]

Gabor Somogyi created KAFKA-8234:


 Summary: Multi-module support for JAAS config property
 Key: KAFKA-8234
 URL: https://issues.apache.org/jira/browse/KAFKA-8234
 Project: Kafka
  Issue Type: Improvement
Reporter: Gabor Somogyi


I've tried to add multi-modules to JAAS config property but its not supported 
at the moment:
{code:java}
Exception in thread "main" org.apache.kafka.common.KafkaException: Failed 
create new KafkaAdminClient
at 
org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:370)
at 
org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:52)
at 
com.kafka.delegationtoken.consumer.SecureKafkaConsumer$.main(SecureKafkaConsumer.scala:96)
at 
com.kafka.delegationtoken.consumer.SecureKafkaConsumer.main(SecureKafkaConsumer.scala)
Caused by: java.lang.IllegalArgumentException: JAAS config property contains 2 
login modules, should be 1 module
at 
org.apache.kafka.common.security.JaasContext.load(JaasContext.java:95)
at 
org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
at 
org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:119)
at 
org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65)
at 
org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88)
at 
org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:346)
... 3 more
{code}
I wanted to implement a fallback scenario with sufficient LoginModule flag but 
the missing multi-module support makes in impossible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8232) Flaky test kafka.admin.TopicCommandWithAdminClientTest.testTopicDeletion

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-8232.
---
   Resolution: Fixed
 Reviewer: Manikumar
Fix Version/s: 2.2.1

> Flaky test kafka.admin.TopicCommandWithAdminClientTest.testTopicDeletion
> 
>
> Key: KAFKA-8232
> URL: https://issues.apache.org/jira/browse/KAFKA-8232
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.3.0, 2.2.1
>
>
> {code:java}
> java.lang.AssertionError: Delete path for topic should exist after deletion.
>   at org.junit.Assert.fail(Assert.java:89)
>   at org.junit.Assert.assertTrue(Assert.java:42)
>   at 
> kafka.admin.TopicCommandWithAdminClientTest.testTopicDeletion(TopicCommandWithAdminClientTest.scala:471){code}
> The verification doesn't look safe since the delete path could have been 
> removed before the test verifies it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8233) Helper class to make it simpler to write test logic TopologyTestDriver

[Jukka Karvanen (JIRA)]

Jukka Karvanen created KAFKA-8233:
-

 Summary: Helper class to make it simpler to write test logic 
TopologyTestDriver
 Key: KAFKA-8233
 URL: https://issues.apache.org/jira/browse/KAFKA-8233
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Jukka Karvanen


When using TopologyTestDriver you need to call ConsumerRecordFactory to create 
ConsumerRecord passed into pipeInput method to write to topic. Also when 
calling readOutput to consume from topic, you need to provide correct 
Deserializers each time.

You easily end up writing helper methods in your test classed, but this can be 
avoided when adding generic input and output topic classes.

This improvement adds TestInputTopic class which wraps TopologyTestDriver  and 
ConsumerRecordFactory methods as one class to be used to write to Input Topics 
and TestOutputTopic class which collects TopologyTestDriver  reading methods 
and provide typesafe read methods.

 

 Example of how Stream test looks after using this classes:

[https://github.com/jukkakarvanen/kafka-streams-examples/blob/InputOutputTopic/src/test/java/io/confluent/examples/streams/WordCountLambdaExampleTest.java]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] KIP-437: Custom replacement for MaskField SMT

[Valeria Vasylieva]

Hi all,

Since there are no more objections/proposals I would like to start the
vote on KIP-437.


See: KIP-437 

and related PR 

I will be grateful to hear your opinion!

Valeria