Topic Creation programatically

[Zishan Ali Saiyed]

Hi Team,

I am using kafka integrated with java client. I have a question " What is
the best practice to create topic using programmatically or using CLI topic
creation command?"


Thanks,
Zishan Ali

Can't start zookeeper and Kafka server

[Amose Cd]

Hi kafka,

 I placed my kafka source in "*C:\kafka_2.9.1-0.8.2.2*" ,Open
command prompt as Administrator . moved to "
*C:\kafka_2.9.1-0.8.2.2\bin\windows*" . now execute command "*
zookeeper-server-start.bat ..\..\config\zookeeper.properties*" im getting
following error. *The system cannot find the path specified.*

 [image: Inline image 1]

-- 
*Amose C.D*
Technology
Mobile: +91 9677892850

*LYNK* | Logistics Simplified
Website  | Download the Lynk App!
 |
Facebook 

[GitHub] kafka pull request #2844: HOTFIX: HTML formatting error in upgrade docs from...

[gwenshap]

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/2844

HOTFIX: HTML formatting error in upgrade docs from pr-2824

Already fixed in the website github

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka docs-hotfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2844.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2844


commit feef5406ebbc7c4e43cbb141059d5bb1aedfc094
Author: Gwen Shapira 
Date:   2017-04-12T03:03:37Z

HOTFIX: HTML formatting error in upgrade docs from pr-2824




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #1422

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-5057) "Big Message Log"

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-5057:
---

 Summary: "Big Message Log"
 Key: KAFKA-5057
 URL: https://issues.apache.org/jira/browse/KAFKA-5057
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


Really large requests can cause significant GC pauses which can cause quite a 
few other symptoms on a broker. Will be nice to be able to catch them.

Lets add the option to log details (client id, topic, partition) for every 
produce request that is larger than a configurable threshold.

/cc [~apurva]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Wrong link :)
http://kafka.apache.org/documentation/#upgrade
and
http://kafka.apache.org/documentation/streams#streams_api_changes_0102

On Tue, Apr 11, 2017 at 5:57 PM, Gwen Shapira  wrote:
> FYI: I just updated the upgrade notes with Streams changes:
> http://kafka.apache.org/documentation/#gettingStarted
>
> On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira  wrote:
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
>> is a bug fix release and it includes fixes and improvements from 24 JIRAs
>> (including a few critical bugs). See the release notes for more details:
>>
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Thursday, 13 April, 8am PT ***
>>
>> Your help in validating this bugfix release is super valuable, so
>> please take the time to test and vote!
>>
>> Few notes:
>> 1. There are missing "Notable Changes" in the docs:
>> https://github.com/apache/kafka/pull/2824
>> I will review, merge and update the docs by Monday.
>> 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
>> yet. We may need another RC if system tests fail tonight.
>>
>> Suggested tests:
>>  * Grab the source archive and make sure it compiles
>>  * Grab one of the binary distros and run the quickstarts against them
>>  * Extract and verify one of the site docs jars
>>  * Build a sample against jars in the staging repo
>>  * Validate GPG signatures on at least one file
>>  * Validate the javadocs look ok
>>
>> *
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging
>>
>> * Javadoc:
>> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d
>>
>> * Documentation:
>> http://kafka.apache.org/0102/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0102/protocol.html
>>
>> Thanks,
>> Gwen Shapira
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

FYI: I just updated the upgrade notes with Streams changes:
http://kafka.apache.org/documentation/#gettingStarted

On Fri, Apr 7, 2017 at 5:12 PM, Gwen Shapira  wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
> is a bug fix release and it includes fixes and improvements from 24 JIRAs
> (including a few critical bugs). See the release notes for more details:
>
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, 13 April, 8am PT ***
>
> Your help in validating this bugfix release is super valuable, so
> please take the time to test and vote!
>
> Few notes:
> 1. There are missing "Notable Changes" in the docs:
> https://github.com/apache/kafka/pull/2824
> I will review, merge and update the docs by Monday.
> 2. The last commit (KAFKA-4943 chery-pick) did not pass system tests
> yet. We may need another RC if system tests fail tonight.
>
> Suggested tests:
>  * Grab the source archive and make sure it compiles
>  * Grab one of the binary distros and run the quickstarts against them
>  * Extract and verify one of the site docs jars
>  * Build a sample against jars in the staging repo
>  * Validate GPG signatures on at least one file
>  * Validate the javadocs look ok
>
> *
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging
>
> * Javadoc:
> http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc0/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d08115f05da0e39c7f75b45e05d6d14ad5baf71d
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> Thanks,
> Gwen Shapira



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Updated] (KAFKA-5056) Shuffling of partitions in old consumer fetch requests removed

[Todd Palino (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Palino updated KAFKA-5056:
---
Summary: Shuffling of partitions in old consumer fetch requests removed  
(was: Shuffling of partitions in old consume fetch requests removed)

> Shuffling of partitions in old consumer fetch requests removed
> --
>
> Key: KAFKA-5056
> URL: https://issues.apache.org/jira/browse/KAFKA-5056
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Todd Palino
>Assignee: Todd Palino
>
> [KIP-74|https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes]
>  deprecated the constructor to {{FetchRequest}} which shuffles the 
> {{requestInfo}} parameter, in favor of round robin reordering logic added to 
> the replica fetcher and the consumer API. However, this was not added to the 
> old consumer {{ConsumerFetcherThread}}, which has resulted in unfair 
> partition fetching since 0.10.1.
> In order to maintain the old consumer, we need to add the removed shuffle to 
> {{buildFetchRequest}} as the topic-partition list for each {{FetchRequest}} 
> is composed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2843: KAFKA-5056: Add shuffling of FetchRequest requestI...

[toddpalino]

GitHub user toddpalino opened a pull request:

https://github.com/apache/kafka/pull/2843

KAFKA-5056: Add shuffling of FetchRequest requestInfo back to old consumer

KIP-74 removed the shuffle of requestInfo from the FetchRequest 
constructor, moving the logic to the replica fetcher and new consumer API. This 
adds a shuffle back to the old consumer that uses the same logic as the 
deprecated constructor.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/toddpalino/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2843


commit c05916e4ab97765e9d00dd2bbe31c8b9e3a7e748
Author: tpalino 
Date:   2017-04-12T00:26:31Z

Add shuffling of FetchRequest requestInfo back to old consumer

KIP-74 removed the shuffle of requestInfo from the FetchRequest 
constructor, moving the logic to the replica fetcher and new consumer API. This 
adds a shuffle back to the old consumer that uses the same logic as the 
deprecated constructor.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5056) Shuffling of partitions in old consume fetch requests removed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965198#comment-15965198
 ] 

ASF GitHub Bot commented on KAFKA-5056:
---

GitHub user toddpalino opened a pull request:

https://github.com/apache/kafka/pull/2843

KAFKA-5056: Add shuffling of FetchRequest requestInfo back to old consumer

KIP-74 removed the shuffle of requestInfo from the FetchRequest 
constructor, moving the logic to the replica fetcher and new consumer API. This 
adds a shuffle back to the old consumer that uses the same logic as the 
deprecated constructor.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/toddpalino/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2843


commit c05916e4ab97765e9d00dd2bbe31c8b9e3a7e748
Author: tpalino 
Date:   2017-04-12T00:26:31Z

Add shuffling of FetchRequest requestInfo back to old consumer

KIP-74 removed the shuffle of requestInfo from the FetchRequest 
constructor, moving the logic to the replica fetcher and new consumer API. This 
adds a shuffle back to the old consumer that uses the same logic as the 
deprecated constructor.




> Shuffling of partitions in old consume fetch requests removed
> -
>
> Key: KAFKA-5056
> URL: https://issues.apache.org/jira/browse/KAFKA-5056
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Todd Palino
>Assignee: Todd Palino
>
> [KIP-74|https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes]
>  deprecated the constructor to {{FetchRequest}} which shuffles the 
> {{requestInfo}} parameter, in favor of round robin reordering logic added to 
> the replica fetcher and the consumer API. However, this was not added to the 
> old consumer {{ConsumerFetcherThread}}, which has resulted in unfair 
> partition fetching since 0.10.1.
> In order to maintain the old consumer, we need to add the removed shuffle to 
> {{buildFetchRequest}} as the topic-partition list for each {{FetchRequest}} 
> is composed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5056) Shuffling of partitions in old consume fetch requests removed

[Todd Palino (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Palino updated KAFKA-5056:
---
Reviewer: Joel Koshy
  Status: Patch Available  (was: In Progress)

> Shuffling of partitions in old consume fetch requests removed
> -
>
> Key: KAFKA-5056
> URL: https://issues.apache.org/jira/browse/KAFKA-5056
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Todd Palino
>Assignee: Todd Palino
>
> [KIP-74|https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes]
>  deprecated the constructor to {{FetchRequest}} which shuffles the 
> {{requestInfo}} parameter, in favor of round robin reordering logic added to 
> the replica fetcher and the consumer API. However, this was not added to the 
> old consumer {{ConsumerFetcherThread}}, which has resulted in unfair 
> partition fetching since 0.10.1.
> In order to maintain the old consumer, we need to add the removed shuffle to 
> {{buildFetchRequest}} as the topic-partition list for each {{FetchRequest}} 
> is composed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5056) Shuffling of partitions in old consume fetch requests removed

[Todd Palino (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5056 started by Todd Palino.
--
> Shuffling of partitions in old consume fetch requests removed
> -
>
> Key: KAFKA-5056
> URL: https://issues.apache.org/jira/browse/KAFKA-5056
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Todd Palino
>Assignee: Todd Palino
>
> [KIP-74|https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes]
>  deprecated the constructor to {{FetchRequest}} which shuffles the 
> {{requestInfo}} parameter, in favor of round robin reordering logic added to 
> the replica fetcher and the consumer API. However, this was not added to the 
> old consumer {{ConsumerFetcherThread}}, which has resulted in unfair 
> partition fetching since 0.10.1.
> In order to maintain the old consumer, we need to add the removed shuffle to 
> {{buildFetchRequest}} as the topic-partition list for each {{FetchRequest}} 
> is composed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5056) Shuffling of partitions in old consume fetch requests removed

[Todd Palino (JIRA)]

Todd Palino created KAFKA-5056:
--

 Summary: Shuffling of partitions in old consume fetch requests 
removed
 Key: KAFKA-5056
 URL: https://issues.apache.org/jira/browse/KAFKA-5056
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.1.0
Reporter: Todd Palino
Assignee: Todd Palino


[KIP-74|https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes]
 deprecated the constructor to {{FetchRequest}} which shuffles the 
{{requestInfo}} parameter, in favor of round robin reordering logic added to 
the replica fetcher and the consumer API. However, this was not added to the 
old consumer {{ConsumerFetcherThread}}, which has resulted in unfair partition 
fetching since 0.10.1.

In order to maintain the old consumer, we need to add the removed shuffle to 
{{buildFetchRequest}} as the topic-partition list for each {{FetchRequest}} is 
composed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-4997) Issue with running kafka-acls.sh when using SASL between Kafka and ZK

[Shrikant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shrikant resolved KAFKA-4997.
-
Resolution: Not A Problem

> Issue with running kafka-acls.sh when using SASL between Kafka and ZK
> -
>
> Key: KAFKA-4997
> URL: https://issues.apache.org/jira/browse/KAFKA-4997
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.1.1
> Environment: Redhat Enterprise Edition Linux, 
>Reporter: Shrikant
>Priority: Critical
>
> Hi All, 
> We are using SASL for Authentication between Kafka and ZK. Followed - 
> https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/
> We have 3 Kafka nodes, on each node, we have 
> principal="kafka/server_no.xxx@xxx.com. So 
> On first node in kafka_server_jaas.conf, principal is set to 
> principal="kafka/server1.xxx@xxx.com"
> On second node in kafka_server_jaas.conf, principal is set to 
> principal="kafka/server2.xxx@xxx.com"
> On third node in kafka_server_jaas.conf, principal is set to 
> principal="kafka/server3.xxx@xxx.com"
> When I run the kafka-acls.sh command from node 1, its successful. It all 
> works, but after that I cannot run kafka-acls.sh from the other 2 nodes. On 
> the other 2 nodes it fails, with error 
> [2017-03-31 18:44:38,629] ERROR Conditional update of path 
> /kafka-acl/Topic/shri-topic with data 
> {"version":1,"acls":[{"principal":"User:CN=xxx,OU=,O=,L=x,ST=xx,C=xx","permissionType":"Allow","operation":"Describe","host":"*"},{"principal":"User:CN=xx,OU=,O=,L=x,ST=xx,C=xx","permissionType":"Allow","operation":"Write","host":"*"}]}
>  and expected version 0 failed due to 
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = 
> NoAuth for /kafka-acl/Topic/shri-topic (kafka.utils.ZkUtils)
> When I look at zookeeper-shell.sh for the kafka-acl node, that node only has 
> permission for principal of first node. I believe this is the reason it fails 
> to run  kafka-acls.sh from the other 2 nodes, even though those nodes have 
> valid key tabs.  
> getAcl /kafka-acl
> 'world,'anyone
> : r
> 'sasl,'kafka/server1.xxx@xxx.com
> : cdrwa
> Is it this bug ?? or am I doing something wrong here.   
> Thanks,
> Shri



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4997) Issue with running kafka-acls.sh when using SASL between Kafka and ZK

[Shrikant (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965186#comment-15965186
 ] 

Shrikant commented on KAFKA-4997:
-

Rajini, Thanks for response. 

Figure out the issue, all the kafka node need to use the same principal name. 
We made this change it working now.

Thanks
Shri

> Issue with running kafka-acls.sh when using SASL between Kafka and ZK
> -
>
> Key: KAFKA-4997
> URL: https://issues.apache.org/jira/browse/KAFKA-4997
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.1.1
> Environment: Redhat Enterprise Edition Linux, 
>Reporter: Shrikant
>Priority: Critical
>
> Hi All, 
> We are using SASL for Authentication between Kafka and ZK. Followed - 
> https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/
> We have 3 Kafka nodes, on each node, we have 
> principal="kafka/server_no.xxx@xxx.com. So 
> On first node in kafka_server_jaas.conf, principal is set to 
> principal="kafka/server1.xxx@xxx.com"
> On second node in kafka_server_jaas.conf, principal is set to 
> principal="kafka/server2.xxx@xxx.com"
> On third node in kafka_server_jaas.conf, principal is set to 
> principal="kafka/server3.xxx@xxx.com"
> When I run the kafka-acls.sh command from node 1, its successful. It all 
> works, but after that I cannot run kafka-acls.sh from the other 2 nodes. On 
> the other 2 nodes it fails, with error 
> [2017-03-31 18:44:38,629] ERROR Conditional update of path 
> /kafka-acl/Topic/shri-topic with data 
> {"version":1,"acls":[{"principal":"User:CN=xxx,OU=,O=,L=x,ST=xx,C=xx","permissionType":"Allow","operation":"Describe","host":"*"},{"principal":"User:CN=xx,OU=,O=,L=x,ST=xx,C=xx","permissionType":"Allow","operation":"Write","host":"*"}]}
>  and expected version 0 failed due to 
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = 
> NoAuth for /kafka-acl/Topic/shri-topic (kafka.utils.ZkUtils)
> When I look at zookeeper-shell.sh for the kafka-acl node, that node only has 
> permission for principal of first node. I believe this is the reason it fails 
> to run  kafka-acls.sh from the other 2 nodes, even though those nodes have 
> valid key tabs.  
> getAcl /kafka-acl
> 'world,'anyone
> : r
> 'sasl,'kafka/server1.xxx@xxx.com
> : cdrwa
> Is it this bug ?? or am I doing something wrong here.   
> Thanks,
> Shri



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2824: MINOR: Added changes in 0.10.2.1

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2824


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] 0.10.2.1 RC0

[Gwen Shapira]

Thanks for the feedback.

I'm not super familiar with the inner workings of Apache's Maven
repos, so I can't explain why we do things the way we do. I followed
the same process on all Apache projects I was on (Kafka, Sqoop,
Flume). Do you know projects that do things the way you suggested?

Either way, may be worthwhile to start a different discussion thread
about RC releases in Maven. Perhaps more knowledgable people will see
it and jump in.

Gwen

On Tue, Apr 11, 2017 at 4:31 PM, Steven Schlansker
 wrote:
>
>> On Apr 7, 2017, at 5:12 PM, Gwen Shapira  wrote:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
>> is a bug fix release and it includes fixes and improvements from 24 JIRAs
>> (including a few critical bugs). See the release notes for more details:
>
> Hi Gwen,
>
> I downloaded and tested the RC with a small Kafka Streams app and the upgrade
> seems to have gone smoothly.  (I did not upgrade any brokers though).
>
> One question about the RC process -- currently it seems that the RC is 
> uploaded
> to a staging repo with the final release version.
>
> Would it not be easier for the community if instead the RC is uploaded to the
> main repo with a "-rc" version?
>
>
> Currently, you have to convince Maven to get "0.10.2.1" from the staging repo,
> and then when the final version hits Maven would never update in case there 
> were
> any post-RC changes.
>
> Additionally, if there are further RCs, it is quite easy to confuse yourself
> and not be sure exactly which RC jar you are running at any given time, and 
> the
> problem compounds itself when multiple developers or build boxes are involved.
>
> Many other projects instead would create a "0.10.2.1-rc0" version and publish
> that to the normal Maven Central -- that way it is publicly downloadable and
> strongly tagged / versioned as the RC.
>
> Has the Kafka project given any thought to this sort of a proposal?
> As a tester / outside user it would make the process a little easier.
>
> Either way, excited for the 0.10.2.1 release, and thanks for all the work!
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[GitHub] kafka pull request #2731: MINOR: Make LeaderAndIsr immutable case class.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2731


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2842: MINOR: findbugs should generate XML reports

[cmccabe]

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2842

MINOR: findbugs should generate XML reports



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka findbugs-xml

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2842.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2842


commit dcdb54a4b598918d9c5dd0f1df9b5577c18d8da0
Author: Colin P. Mccabe 
Date:   2017-04-11T23:53:55Z

MINOR: findbugs should generate XML reports




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5013) Fail the build when findbugs fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965154#comment-15965154
 ] 

ASF GitHub Bot commented on KAFKA-5013:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2805


> Fail the build when findbugs fails
> --
>
> Key: KAFKA-5013
> URL: https://issues.apache.org/jira/browse/KAFKA-5013
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> We should fail the build when findbugs fails, so that new findbugs warnings 
> do not creep in.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2805: KAFKA-5013. Fail the build when findbugs fails

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2805


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5013) Fail the build when findbugs fails

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5013.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2805
[https://github.com/apache/kafka/pull/2805]

> Fail the build when findbugs fails
> --
>
> Key: KAFKA-5013
> URL: https://issues.apache.org/jira/browse/KAFKA-5013
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> We should fail the build when findbugs fails, so that new findbugs warnings 
> do not creep in.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] 0.10.2.1 RC0

[Steven Schlansker]

> On Apr 7, 2017, at 5:12 PM, Gwen Shapira  wrote:
> 
> Hello Kafka users, developers and client-developers,
> 
> This is the first candidate for the release of Apache Kafka 0.10.2.1. This
> is a bug fix release and it includes fixes and improvements from 24 JIRAs
> (including a few critical bugs). See the release notes for more details:

Hi Gwen,

I downloaded and tested the RC with a small Kafka Streams app and the upgrade
seems to have gone smoothly.  (I did not upgrade any brokers though).

One question about the RC process -- currently it seems that the RC is uploaded
to a staging repo with the final release version.

Would it not be easier for the community if instead the RC is uploaded to the
main repo with a "-rc" version?


Currently, you have to convince Maven to get "0.10.2.1" from the staging repo,
and then when the final version hits Maven would never update in case there were
any post-RC changes.

Additionally, if there are further RCs, it is quite easy to confuse yourself
and not be sure exactly which RC jar you are running at any given time, and the
problem compounds itself when multiple developers or build boxes are involved.

Many other projects instead would create a "0.10.2.1-rc0" version and publish
that to the normal Maven Central -- that way it is publicly downloadable and
strongly tagged / versioned as the RC.

Has the Kafka project given any thought to this sort of a proposal?
As a tester / outside user it would make the process a little easier.

Either way, excited for the 0.10.2.1 release, and thanks for all the work!



signature.asc
Description: Message signed with OpenPGP using GPGMail

[jira] [Commented] (KAFKA-5011) Replica fetchers may need to down-convert messages during a selective message format upgrade

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964994#comment-15964994
 ] 

Joel Koshy commented on KAFKA-5011:
---

Yes that is correct - it is rare. I think it's reasonable to close this as 
won't fix. Not sure if we need to mention it in docs given that it is extremely 
rare.

> Replica fetchers may need to down-convert messages during a selective message 
> format upgrade
> 
>
> Key: KAFKA-5011
> URL: https://issues.apache.org/jira/browse/KAFKA-5011
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5038) running multiple kafka streams instances causes one or more instance to get into file contention

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5038 started by Eno Thereska.
---
> running multiple kafka streams instances causes one or more instance to get 
> into file contention
> 
>
> Key: KAFKA-5038
> URL: https://issues.apache.org/jira/browse/KAFKA-5038
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
> Environment: 3 Kafka broker machines and 3 kafka streams machines.
> Each machine is Linux 64 bit, CentOS 6.5 with 64GB memory, 8 vCPUs running in 
> AWS
> 31GB java heap space allocated to each KafkaStreams instance and 4GB 
> allocated to each Kafka broker.
>Reporter: Bharad Tirumala
>Assignee: Eno Thereska
>
> Having multiple kafka streams application instances causes one or more 
> instances to get get into file lock contention and the instance(s) become 
> unresponsive with uncaught exception.
> The exception is below:
> 22:14:37.621 [StreamThread-7] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.621 [StreamThread-13] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.623 [StreamThread-18] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.625 [StreamThread-7] ERROR n.a.a.k.t.KStreamTopologyBase - Uncaught 
> Exception:org.apache.kafka.streams.errors.ProcessorStateException: task 
> directory [/data/kafka-streams/rtp-kstreams-metrics/0_119] doesn't exist and 
> couldn't be created
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> This happens within couple of minutes after the instances are up and there is 
> NO data being sent to the broker yet and the streams app is started with 
> auto.offset.reset set to "latest".
> Please note that there are no permissions or capacity issues. This may have 
> nothing to do with number of instances, but I could easily reproduce it when 
> I've 3 stream instances running. This is similar to the (and may be the same) 
> bug as [KAFKA-3758]
> Here are some relevant configuration info:
> 3 kafka brokers have one topic with 128 partitions and 1 replication
> 3 kafka streams applications (running on 3 machines) have a single processor 
> topology and this processor is not doing anything (the process() method just 
> returns and the punctuate method just commits)
> There is no data flowing yet, so the process() and puctuate() methods are not 
> even called yet.
> The 3 kafka stream instances have 43, 43 and 42 threads each respectively 
> (totally making up to 128 threads, so one task per thread distributed across 
> three streams instances on 3 machines).
> Here are the configurations that I'd played around with:
> session.timeout.ms=30
> heartbeat.interval.ms=6
> max.poll.records=100
> num.standby.replicas=1
> commit.interval.ms=1
> poll.ms=100
> When punctuate is scheduled to be called every 1000ms or 3000ms, the problem 
> happens every time. If punctuate is scheduled for 5000ms, I didn't see the 
> problem in my test scenario (described above), but it happened in my real 
> application. But this may have nothing to do with the issue, since punctuate 
> is not even called as there are no messages streaming through yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5038) running multiple kafka streams instances causes one or more instance to get into file contention

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska reassigned KAFKA-5038:
---

Assignee: Eno Thereska

> running multiple kafka streams instances causes one or more instance to get 
> into file contention
> 
>
> Key: KAFKA-5038
> URL: https://issues.apache.org/jira/browse/KAFKA-5038
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
> Environment: 3 Kafka broker machines and 3 kafka streams machines.
> Each machine is Linux 64 bit, CentOS 6.5 with 64GB memory, 8 vCPUs running in 
> AWS
> 31GB java heap space allocated to each KafkaStreams instance and 4GB 
> allocated to each Kafka broker.
>Reporter: Bharad Tirumala
>Assignee: Eno Thereska
>
> Having multiple kafka streams application instances causes one or more 
> instances to get get into file lock contention and the instance(s) become 
> unresponsive with uncaught exception.
> The exception is below:
> 22:14:37.621 [StreamThread-7] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.621 [StreamThread-13] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.623 [StreamThread-18] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.625 [StreamThread-7] ERROR n.a.a.k.t.KStreamTopologyBase - Uncaught 
> Exception:org.apache.kafka.streams.errors.ProcessorStateException: task 
> directory [/data/kafka-streams/rtp-kstreams-metrics/0_119] doesn't exist and 
> couldn't be created
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> This happens within couple of minutes after the instances are up and there is 
> NO data being sent to the broker yet and the streams app is started with 
> auto.offset.reset set to "latest".
> Please note that there are no permissions or capacity issues. This may have 
> nothing to do with number of instances, but I could easily reproduce it when 
> I've 3 stream instances running. This is similar to the (and may be the same) 
> bug as [KAFKA-3758]
> Here are some relevant configuration info:
> 3 kafka brokers have one topic with 128 partitions and 1 replication
> 3 kafka streams applications (running on 3 machines) have a single processor 
> topology and this processor is not doing anything (the process() method just 
> returns and the punctuate method just commits)
> There is no data flowing yet, so the process() and puctuate() methods are not 
> even called yet.
> The 3 kafka stream instances have 43, 43 and 42 threads each respectively 
> (totally making up to 128 threads, so one task per thread distributed across 
> three streams instances on 3 machines).
> Here are the configurations that I'd played around with:
> session.timeout.ms=30
> heartbeat.interval.ms=6
> max.poll.records=100
> num.standby.replicas=1
> commit.interval.ms=1
> poll.ms=100
> When punctuate is scheduled to be called every 1000ms or 3000ms, the problem 
> happens every time. If punctuate is scheduled for 5000ms, I didn't see the 
> problem in my test scenario (described above), but it happened in my real 
> application. But this may have nothing to do with the issue, since punctuate 
> is not even called as there are no messages streaming through yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4930) Connect Rest API allows creating connectors with an empty name

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964886#comment-15964886
 ] 

Sönke Liebau commented on KAFKA-4930:
-

[~gwenshap]: Did you have a chance yet to have a look at my PR? I'm happy to 
add tests, but would like to wait for a +1 in principle to the approach chosen 
before doing so :)

> Connect Rest API allows creating connectors with an empty name
> --
>
> Key: KAFKA-4930
> URL: https://issues.apache.org/jira/browse/KAFKA-4930
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Sönke Liebau
>Priority: Minor
>
> The Connect Rest API allows to deploy connectors with an empty name field, 
> which then cannot be removed through the api.
> Sending the following request:
> {code}
> {
> "name": "",
> "config": {
> "connector.class": 
> "org.apache.kafka.connect.tools.MockSourceConnector",
> "tasks.max": "1",
> "topics": "test-topic"
> 
> }
> }
> {code}
> Results in a connector being deployed which can be seen in the list of 
> connectors:
> {code}
> [
>   "",
>   "testconnector"
> ]{code}
> But cannot be removed via a DELETE call, as the api thinks we are trying to 
> delete the /connectors endpoint and declines the request.
> I don't think there is a valid case for the connector name to be empty so 
> perhaps we should add a check for this. I am happy to work on this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5038) running multiple kafka streams instances causes one or more instance to get into file contention

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964883#comment-15964883
 ] 

Eno Thereska commented on KAFKA-5038:
-

[~btirumala] thank you for posting. Looks like it could be a bug. I've opened 
at PR: https://github.com/apache/kafka/pull/2841. Would you mind trying to see 
if it fixes the problem? Thanks.


> running multiple kafka streams instances causes one or more instance to get 
> into file contention
> 
>
> Key: KAFKA-5038
> URL: https://issues.apache.org/jira/browse/KAFKA-5038
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
> Environment: 3 Kafka broker machines and 3 kafka streams machines.
> Each machine is Linux 64 bit, CentOS 6.5 with 64GB memory, 8 vCPUs running in 
> AWS
> 31GB java heap space allocated to each KafkaStreams instance and 4GB 
> allocated to each Kafka broker.
>Reporter: Bharad Tirumala
>
> Having multiple kafka streams application instances causes one or more 
> instances to get get into file lock contention and the instance(s) become 
> unresponsive with uncaught exception.
> The exception is below:
> 22:14:37.621 [StreamThread-7] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.621 [StreamThread-13] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.623 [StreamThread-18] WARN  o.a.k.s.p.internals.StreamThread - 
> Unexpected state transition from RUNNING to NOT_RUNNING
> 22:14:37.625 [StreamThread-7] ERROR n.a.a.k.t.KStreamTopologyBase - Uncaught 
> Exception:org.apache.kafka.streams.errors.ProcessorStateException: task 
> directory [/data/kafka-streams/rtp-kstreams-metrics/0_119] doesn't exist and 
> couldn't be created
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> This happens within couple of minutes after the instances are up and there is 
> NO data being sent to the broker yet and the streams app is started with 
> auto.offset.reset set to "latest".
> Please note that there are no permissions or capacity issues. This may have 
> nothing to do with number of instances, but I could easily reproduce it when 
> I've 3 stream instances running. This is similar to the (and may be the same) 
> bug as [KAFKA-3758]
> Here are some relevant configuration info:
> 3 kafka brokers have one topic with 128 partitions and 1 replication
> 3 kafka streams applications (running on 3 machines) have a single processor 
> topology and this processor is not doing anything (the process() method just 
> returns and the punctuate method just commits)
> There is no data flowing yet, so the process() and puctuate() methods are not 
> even called yet.
> The 3 kafka stream instances have 43, 43 and 42 threads each respectively 
> (totally making up to 128 threads, so one task per thread distributed across 
> three streams instances on 3 machines).
> Here are the configurations that I'd played around with:
> session.timeout.ms=30
> heartbeat.interval.ms=6
> max.poll.records=100
> num.standby.replicas=1
> commit.interval.ms=1
> poll.ms=100
> When punctuate is scheduled to be called every 1000ms or 3000ms, the problem 
> happens every time. If punctuate is scheduled for 5000ms, I didn't see the 
> problem in my test scenario (described above), but it happened in my real 
> application. But this may have nothing to do with the issue, since punctuate 
> is not even called as there are no messages streaming through yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2841: KAFKA-5038: Catch exception

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2841

KAFKA-5038: Catch exception



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka KAFKA-5038

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2841.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2841


commit b484904a74458dd20d3b7b0deb26f822feca9140
Author: Eno Thereska 
Date:   2017-04-11T19:49:28Z

Catch exception




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-4988) JVM crash when running on Alpine Linux

[Murad M (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964809#comment-15964809
 ] 

Murad M edited comment on KAFKA-4988 at 4/11/17 6:55 PM:
-

Just faced same problem here. Running application in fully blown ubuntu 
environment has no problems. When deployed in official 
https://hub.docker.com/_/openjdk/ image using {{FROM openjdk:8-alpine}} it 
crashed in exactly same way. Quick peek on what is going on shows that:
{quote}
/ # ldd /tmp/librocksdbjni2324596304249162547.so
ldd (0x55ffc73ea000)
libpthread.so.0 => ldd (0x55ffc73ea000)
librt.so.1 => ldd (0x55ffc73ea000)
Error loading shared library libstdc++.so.6: No such file or directory (needed 
by /tmp/librocksdbjni2324596304249162547.so)
libm.so.6 => ldd (0x55ffc73ea000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fe1d1ba3000)
libc.so.6 => ldd (0x55ffc73ea000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory 
(needed by /tmp/librocksdbjni2324596304249162547.so)
Error relocating /tmp/librocksdbjni2324596304249162547.so: _Znam: symbol not 
found
Error relocating /tmp/librocksdbjni2324596304249162547.so: _ZNSo3putEc: symbol 
not found
Error relocating /tmp/librocksdbjni2324596304249162547.so: 
_ZSt18uncaught_exceptionv: symbol not found
Error relocating /tmp/librocksdbjni2324596304249162547.so: 
_ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_: symbol not 
found
{quote}
I suppose that librocksdbjni comes from rocksdb dependency, and compiled 
against libstdc, but {{openjdk:8-alpine}} is packaged with {{libc.musl}}.
Switching to {{openjdk:8}} which is based on debian didn't help me. While above 
problem is solved, other problems added in other parts of application also, in 
kafka:
{quote}
org.apache.kafka.streams.errors.ProcessorStateException: task directory 
[/tmp/kafka-streams//0_15] doesn't exist and couldn't be created
at 
org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
at 
org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
at 
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
at 
org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
{quote}
I suppose that this issue is neither related to Kafka or RocksDB, but to 
container environment.

Long story short, I found that https://hub.docker.com/r/wurstmeister/kafka/ 
image is based on https://hub.docker.com/r/anapsix/alpine-java/ which works 
pretty well on server side. So I solved this issue by switching from 
{{openjdk}} image to {{anapsix/alpine-java}}.


was (Author: muradm):
Just faced same problem here. Running application in fully blown ubuntu 
environment has no problems. When deployed in official 
https://hub.docker.com/_/openjdk/ image using FROM openjdk:8-alpine it crashed 
in exactly same way. Quick peek on what is going on shows that:
{quote}
/ # ldd /tmp/librocksdbjni2324596304249162547.so
ldd (0x55ffc73ea000)
libpthread.so.0 => ldd (0x55ffc73ea000)
librt.so.1 => ldd (0x55ffc73ea000)
Error loading shared library libstdc++.so.6: No such file or directory (needed 
by /tmp/librocksdbjni2324596304249162547.so)
libm.so.6 => ldd (0x55ffc73ea000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fe1d1ba3000)
libc.so.6 => ldd (0x55ffc73ea000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory 
(needed by /tmp/librocksdbjni2324596304249162547.so)
Error relocating /tmp/librocksdbjni2324596304249162547.so: _Znam: symbol not 
found
Error relocating /tmp/librocksdbjni2324596304249162547.so: _ZNSo3putEc: symbol 
not found
Error relocating /tmp/librocksdbjni2324596304249162547.so: 
_ZSt18uncaught_exceptionv: symbol not found
Error relocating /tmp/librocksdbjni2324596304249162547.so: 
_ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_: symbol not 
found
{quote}
I suppose that librocksdbjni comes from rocksdb dependency, and compiled 
against libstdc, but {{openjdk:8-alpine}} is packaged with {{libc.musl}}.
Switching to {{openjdk:8}} which is based on debian didn't help me. While above 
problem is solved, other problems added in other parts of application also, in 
kafka:
{quote}
org.apache.kafka.streams.errors.ProcessorStateException: task directory 
[/tmp/kafka-streams//0_15] doesn't exist and couldn't be created
at 
org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
at 

[jira] [Commented] (KAFKA-4988) JVM crash when running on Alpine Linux

[Murad M (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964809#comment-15964809
 ] 

Murad M commented on KAFKA-4988:


Just faced same problem here. Running application in fully blown ubuntu 
environment has no problems. When deployed in official 
https://hub.docker.com/_/openjdk/ image using FROM openjdk:8-alpine it crashed 
in exactly same way. Quick peek on what is going on shows that:
{quote}
/ # ldd /tmp/librocksdbjni2324596304249162547.so
ldd (0x55ffc73ea000)
libpthread.so.0 => ldd (0x55ffc73ea000)
librt.so.1 => ldd (0x55ffc73ea000)
Error loading shared library libstdc++.so.6: No such file or directory (needed 
by /tmp/librocksdbjni2324596304249162547.so)
libm.so.6 => ldd (0x55ffc73ea000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fe1d1ba3000)
libc.so.6 => ldd (0x55ffc73ea000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory 
(needed by /tmp/librocksdbjni2324596304249162547.so)
Error relocating /tmp/librocksdbjni2324596304249162547.so: _Znam: symbol not 
found
Error relocating /tmp/librocksdbjni2324596304249162547.so: _ZNSo3putEc: symbol 
not found
Error relocating /tmp/librocksdbjni2324596304249162547.so: 
_ZSt18uncaught_exceptionv: symbol not found
Error relocating /tmp/librocksdbjni2324596304249162547.so: 
_ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_: symbol not 
found
{quote}
I suppose that librocksdbjni comes from rocksdb dependency, and compiled 
against libstdc, but {{openjdk:8-alpine}} is packaged with {{libc.musl}}.
Switching to {{openjdk:8}} which is based on debian didn't help me. While above 
problem is solved, other problems added in other parts of application also, in 
kafka:
{quote}
org.apache.kafka.streams.errors.ProcessorStateException: task directory 
[/tmp/kafka-streams//0_15] doesn't exist and couldn't be created
at 
org.apache.kafka.streams.processor.internals.StateDirectory.directoryForTask(StateDirectory.java:75)
at 
org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:102)
at 
org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:205)
at 
org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:753)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:664)
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
{quote}
I suppose that this issue is neither related to Kafka or RocksDB, but to 
container environment.

Long story short, I found that https://hub.docker.com/r/wurstmeister/kafka/ 
image is based on https://hub.docker.com/r/anapsix/alpine-java/ which works 
pretty well on server side. So I solved this issue by switching from 
{{openjdk}} image to {{anapsix/alpine-java}}.

> JVM crash when running on Alpine Linux
> --
>
> Key: KAFKA-4988
> URL: https://issues.apache.org/jira/browse/KAFKA-4988
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Vincent Rischmann
>Priority: Minor
>
> I'm developing my Kafka Streams application using Docker and I run my jars 
> using the official openjdk:8-jre-alpine image.
> I'm just starting to use windowing and now the JVM crashes because of an 
> issue with RocksDB I think.
> It's trivial to fix on my part, just use the debian jessie based image. 
> However, it would be cool if alpine was supported too since its docker images 
> are quite a bit less heavy
> {quote}
> Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError: 
> /tmp/librocksdbjni3285995384052305662.so: Error loading shared library 
> ld-linux-x86-64.so.2: No such file or directory (needed by 
> /tmp/librocksdbjni3285995384052305662.so)
>   at java.lang.ClassLoader$NativeLibrary.load(Native Method)
>   at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
>   at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
>   at java.lang.Runtime.load0(Runtime.java:809)
>   at java.lang.System.load(System.java:1086)
>   at 
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
>   at 
> org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
>   at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
>   at org.rocksdb.RocksDB.(RocksDB.java:35)
>   at org.rocksdb.Options.(Options.java:22)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:115)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:148)
>   at 
> 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Matthias J. Sax]

+1

On 4/11/17 10:34 AM, Eno Thereska wrote:
> Hi Matthias,
> 
> 
>> On 11 Apr 2017, at 09:41, Matthias J. Sax  wrote:
>>
>> Not sure, if we are on the same page already?
>>
>>> "A __store__ can be queryable whether is't materialized or not"
>>
>> This does not make sense -- there is nothing like a non-materialized
>> store -- only non-materialized KTables.
> 
> Yes, there are stores that are simple views, i.e., non-materialized. Damian 
> has such a prototype for Global Tables (it didn't go into trunk). 
> It's still a store, e.g., a KeyValueStore, but when you do a get() it 
> recomputes the result on the fly (e.g., it applies a filter).
> 
> Eno
> 
>>
>>> "Yes, there is nothing that will prevent users from querying
>> internally generated stores, but they cannot assume a store will
>> necessarily be queryable."
>>
>> That is what I disagree on. Stores should be queryable all the time.
>>
>> Furthermore, we should have all non-materialized KTables to be
>> queryable, too.
>>
>>
>> Or maybe there is just some missunderstand going as, and there is some
>> mix-up between "store" and "KTable"
>>
>>
>>
>> -Matthias
>>
>>
>> On 4/11/17 9:34 AM, Eno Thereska wrote:
>>> Hi Matthias,
>>>
>>> See my note: "A store can be queryable whether it's materialized or not". I 
>>> think we're on the same page. Stores with an internal name are also 
>>> queryable. 
>>>
>>> I'm just pointing out that. although that is the case today and with this 
>>> KIP, I don't think we have an obligation to make stores with internal names 
>>> queryable in the future. However, that is a discussion for a future point.
>>>
>>> Eno
>>>
>>>
>>>
>>>
 On 11 Apr 2017, at 08:56, Matthias J. Sax  wrote:

 +1 on including GlobalKTable

 But I am not sure about the materialization / queryable question. For
 full consistency, all KTables should be queryable nevertheless if they
 are materialized or not. -- Maybe this is a second step though (even if
 I would like to get this done right away)

 If we don't want all KTables to be queryable, ie, only those KTables
 that are materialized, then we should have a clear definition about
 this, and only allow to query stores, the user did specify a name for.
 This will simply the reasoning for users, what stores are queryable and
 what not. Otherwise, we still end up confusing user.


 -Matthias

 On 4/11/17 8:23 AM, Damian Guy wrote:
> Eno, re: GlobalKTable - yeah that seems fine.
>
> On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:
>
>> About GlobalKTables, I suppose there is no reason why they cannot also 
>> use
>> this KIP for consistency, e.g., today you have:
>>
>> public  GlobalKTable globalTable(final Serde keySerde,
>>final Serde valSerde,
>>final String topic,
>>final String storeName)
>>
>> For consistency with the KIP you could also have an overload without the
>> store name, for people who want to construct a global ktable, but don't
>> care about querying it directly:
>>
>> public  GlobalKTable globalTable(final Serde keySerde,
>>final Serde valSerde,
>>final String topic)
>>
>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
>> Michael for bringing it up.
>>
>> Eno
>>
>>
>>
>>> On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
>>>
>>> Hi Michael, comments inline:
>>>
 On 11 Apr 2017, at 03:25, Michael Noll  wrote:

 Thanks for the updates, Eno!

 In addition to what has already been said:  We should also explicitly
 mention that this KIP is not touching GlobalKTable.  I'm sure that some
 users will throw KTable and GlobalKTable into one conceptual "it's all
 tables!" bucket and then wonder how the KIP might affect global tables.
>>>
>>> Good point, I'll add.
>>>
>>>

 Damian wrote:
> I think if no store name is provided users would still be able to 
> query
 the
> store, just the store name would be some internally generated name.
>> They
> would be able to discover those names via the IQ API.

 I, too, think that users should be able to query a store even if its
>> name
 was internally generated.  After all, the data is already there /
 materialized.
>>>
>>> Yes, there is nothing that will prevent users from querying internally
>> generated stores, but they cannot
>>> assume a store will necessarily be queryable. So if it's 

[jira] [Commented] (KAFKA-4818) Implement transactional clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964765#comment-15964765
 ] 

ASF GitHub Bot commented on KAFKA-4818:
---

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/2840

KAFKA-4818: Exactly once transactional clients



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
exactly-once-transactional-clients

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2840.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2840


commit 8679628642cd0227a1ff1c81a316a11766230e1b
Author: Apurva Mehta 
Date:   2017-03-30T22:13:11Z

CPKAFKA-465 : Implement READ_COMMITTED mode in the KafkaConsumer (#145)

commit 477f45c560e681cf65ecc72e3e61eba073ba971b
Author: Apurva Mehta 
Date:   2017-04-11T18:12:26Z

Complete implementation of the transactional producer.




> Implement transactional clients
> ---
>
> Key: KAFKA-4818
> URL: https://issues.apache.org/jira/browse/KAFKA-4818
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>
> This covers the implementation of the producer and consumer to support 
> transactions, as described in KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2840: KAFKA-4818: Exactly once transactional clients

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/2840

KAFKA-4818: Exactly once transactional clients



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
exactly-once-transactional-clients

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2840.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2840


commit 8679628642cd0227a1ff1c81a316a11766230e1b
Author: Apurva Mehta 
Date:   2017-03-30T22:13:11Z

CPKAFKA-465 : Implement READ_COMMITTED mode in the KafkaConsumer (#145)

commit 477f45c560e681cf65ecc72e3e61eba073ba971b
Author: Apurva Mehta 
Date:   2017-04-11T18:12:26Z

Complete implementation of the transactional producer.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4818) Implement transactional clients

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-4818:

Description: This covers the implementation of the producer and consumer to 
support transactions.   (was: This covers the implementation of the transaction 
coordinator and the changes to the producer and consumer to support 
transactions. )
Summary: Implement transactional clients  (was: Implement transactional 
producer)

> Implement transactional clients
> ---
>
> Key: KAFKA-4818
> URL: https://issues.apache.org/jira/browse/KAFKA-4818
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>
> This covers the implementation of the producer and consumer to support 
> transactions. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4818) Implement transactional clients

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-4818:

Description: This covers the implementation of the producer and consumer to 
support transactions, as described in KIP-98: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
  (was: This covers the implementation of the producer and consumer to support 
transactions. )

> Implement transactional clients
> ---
>
> Key: KAFKA-4818
> URL: https://issues.apache.org/jira/browse/KAFKA-4818
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>
> This covers the implementation of the producer and consumer to support 
> transactions, as described in KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4818) Implement transactional producer

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta reassigned KAFKA-4818:
---

Assignee: Apurva Mehta  (was: Guozhang Wang)

> Implement transactional producer
> 
>
> Key: KAFKA-4818
> URL: https://issues.apache.org/jira/browse/KAFKA-4818
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>
> This covers the implementation of the transaction coordinator and the changes 
> to the producer and consumer to support transactions. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4967) java.io.EOFException Error while committing offsets

[Upendra Yadav (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964757#comment-15964757
 ] 

Upendra Yadav commented on KAFKA-4967:
--

I solved this issue by calling 2nd time commitOffset.
actually the problem related to connections.max.idle.ms.
this propery is introduced in latest kafka(broker=10 minutes, consumer=9 
minutes, producer=9 minutes).
due to this whenever my old consumer calling next commit offset after 10 
minutes, I am getting above exception.
with old consumer api there is no way to set this property.
and broker configuration change is not in my control...

here i think commitOffset required another connection(other that iterator) and 
that connection when getting ideal for more than 10 minutes connection getting 
close.
I'm not very sure about this.
but next consecutive call for now its working fine for me... at-least if any 
fail happening on 1st call then 2nd call getting success.
and if 1st one getting success then next one execution will not make any 
problem.
any way, we have very few calls for commit offset.

next we'll try to move to use latest kafka consumer and producer java APIs. 

> java.io.EOFException Error while committing offsets
> ---
>
> Key: KAFKA-4967
> URL: https://issues.apache.org/jira/browse/KAFKA-4967
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.0.1
> Environment: OS : CentOS
>Reporter: Upendra Yadav
>
> kafka server and client : 0.10.0.1
> And consumer and producer side using latest kafka jars as mentioned above but 
> still using old consumer apis in code. 
> kafka server side configuration :
> listeners=PLAINTEXT://:9092
> #below configuration is for old clients, that was exists before. but now 
> every clients are already moved with latest kafka client - 0.10.0.1
> log.message.format.version=0.8.2.1
> broker.id.generation.enable=false
> unclean.leader.election.enable=false
> Some of configurations for kafka consumer :
> auto.commit.enable is overridden to false
> auto.offset.reset is overridden to smallest
> consumer.timeout.ms is overridden to 100
> dual.commit.enabled is overridden to true
> fetch.message.max.bytes is overridden to 209715200
> group.id is overridden to crm_topic1_hadoop_tables
> offsets.storage is overridden to kafka
> rebalance.backoff.ms is overridden to 6000
> zookeeper.session.timeout.ms is overridden to 23000
> zookeeper.sync.time.ms is overridden to 2000
> below exception I'm getting on commit offset.
> Consumer process is still running after this exception..
> but when I'm checking offset position through kafka shell scripts its showing 
> old position(Could not fetch offset from topic1_group1 partition [topic1,0] 
> due to missing offset data in zookeeper). after some time when 2nd commit 
> comes then it get updated.
> because of duel commit enabled, I think kafka side position get update 
> successfully for both time.
> ERROR kafka.consumer.ZookeeperConsumerConnector: [], Error while 
> committing offsets.
> java.io.EOFException
> at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
> at 
> kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120)
> at 
> kafka.consumer.ZookeeperConsumerConnector.liftedTree2$1(ZookeeperConsumerConnector.scala:354)
> at 
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:351)
> at 
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:331)
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:111)
> at 
> com.zoho.mysqlbackup.kafka.consumer.KafkaHLConsumer.commitOffset(KafkaHLConsumer.java:173)
> at 
> com.zoho.mysqlbackup.kafka.consumer.KafkaHLConsumer.run(KafkaHLConsumer.java:271)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Eno Thereska]

Hi Matthias,


> On 11 Apr 2017, at 09:41, Matthias J. Sax  wrote:
> 
> Not sure, if we are on the same page already?
> 
>> "A __store__ can be queryable whether is't materialized or not"
> 
> This does not make sense -- there is nothing like a non-materialized
> store -- only non-materialized KTables.

Yes, there are stores that are simple views, i.e., non-materialized. Damian has 
such a prototype for Global Tables (it didn't go into trunk). 
It's still a store, e.g., a KeyValueStore, but when you do a get() it 
recomputes the result on the fly (e.g., it applies a filter).

Eno

> 
>> "Yes, there is nothing that will prevent users from querying
> internally generated stores, but they cannot assume a store will
> necessarily be queryable."
> 
> That is what I disagree on. Stores should be queryable all the time.
> 
> Furthermore, we should have all non-materialized KTables to be
> queryable, too.
> 
> 
> Or maybe there is just some missunderstand going as, and there is some
> mix-up between "store" and "KTable"
> 
> 
> 
> -Matthias
> 
> 
> On 4/11/17 9:34 AM, Eno Thereska wrote:
>> Hi Matthias,
>> 
>> See my note: "A store can be queryable whether it's materialized or not". I 
>> think we're on the same page. Stores with an internal name are also 
>> queryable. 
>> 
>> I'm just pointing out that. although that is the case today and with this 
>> KIP, I don't think we have an obligation to make stores with internal names 
>> queryable in the future. However, that is a discussion for a future point.
>> 
>> Eno
>> 
>> 
>> 
>> 
>>> On 11 Apr 2017, at 08:56, Matthias J. Sax  wrote:
>>> 
>>> +1 on including GlobalKTable
>>> 
>>> But I am not sure about the materialization / queryable question. For
>>> full consistency, all KTables should be queryable nevertheless if they
>>> are materialized or not. -- Maybe this is a second step though (even if
>>> I would like to get this done right away)
>>> 
>>> If we don't want all KTables to be queryable, ie, only those KTables
>>> that are materialized, then we should have a clear definition about
>>> this, and only allow to query stores, the user did specify a name for.
>>> This will simply the reasoning for users, what stores are queryable and
>>> what not. Otherwise, we still end up confusing user.
>>> 
>>> 
>>> -Matthias
>>> 
>>> On 4/11/17 8:23 AM, Damian Guy wrote:
 Eno, re: GlobalKTable - yeah that seems fine.
 
 On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:
 
> About GlobalKTables, I suppose there is no reason why they cannot also use
> this KIP for consistency, e.g., today you have:
> 
> public  GlobalKTable globalTable(final Serde keySerde,
>final Serde valSerde,
>final String topic,
>final String storeName)
> 
> For consistency with the KIP you could also have an overload without the
> store name, for people who want to construct a global ktable, but don't
> care about querying it directly:
> 
> public  GlobalKTable globalTable(final Serde keySerde,
>final Serde valSerde,
>final String topic)
> 
> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
> Michael for bringing it up.
> 
> Eno
> 
> 
> 
>> On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
>> 
>> Hi Michael, comments inline:
>> 
>>> On 11 Apr 2017, at 03:25, Michael Noll  wrote:
>>> 
>>> Thanks for the updates, Eno!
>>> 
>>> In addition to what has already been said:  We should also explicitly
>>> mention that this KIP is not touching GlobalKTable.  I'm sure that some
>>> users will throw KTable and GlobalKTable into one conceptual "it's all
>>> tables!" bucket and then wonder how the KIP might affect global tables.
>> 
>> Good point, I'll add.
>> 
>> 
>>> 
>>> Damian wrote:
 I think if no store name is provided users would still be able to query
>>> the
 store, just the store name would be some internally generated name.
> They
 would be able to discover those names via the IQ API.
>>> 
>>> I, too, think that users should be able to query a store even if its
> name
>>> was internally generated.  After all, the data is already there /
>>> materialized.
>> 
>> Yes, there is nothing that will prevent users from querying internally
> generated stores, but they cannot
>> assume a store will necessarily be queryable. So if it's there, they can
> query it. If it's not there, and they didn't
>> provide a queryable name, they cannot complain and say "hey, where is my
> store". If 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Matthias J. Sax]

Not sure, if we are on the same page already?

> "A __store__ can be queryable whether is't materialized or not"

This does not make sense -- there is nothing like a non-materialized
store -- only non-materialized KTables.

> "Yes, there is nothing that will prevent users from querying
internally generated stores, but they cannot assume a store will
necessarily be queryable."

That is what I disagree on. Stores should be queryable all the time.

Furthermore, we should have all non-materialized KTables to be
queryable, too.


Or maybe there is just some missunderstand going as, and there is some
mix-up between "store" and "KTable"



-Matthias


On 4/11/17 9:34 AM, Eno Thereska wrote:
> Hi Matthias,
> 
> See my note: "A store can be queryable whether it's materialized or not". I 
> think we're on the same page. Stores with an internal name are also 
> queryable. 
> 
> I'm just pointing out that. although that is the case today and with this 
> KIP, I don't think we have an obligation to make stores with internal names 
> queryable in the future. However, that is a discussion for a future point.
> 
> Eno
> 
> 
> 
> 
>> On 11 Apr 2017, at 08:56, Matthias J. Sax  wrote:
>>
>> +1 on including GlobalKTable
>>
>> But I am not sure about the materialization / queryable question. For
>> full consistency, all KTables should be queryable nevertheless if they
>> are materialized or not. -- Maybe this is a second step though (even if
>> I would like to get this done right away)
>>
>> If we don't want all KTables to be queryable, ie, only those KTables
>> that are materialized, then we should have a clear definition about
>> this, and only allow to query stores, the user did specify a name for.
>> This will simply the reasoning for users, what stores are queryable and
>> what not. Otherwise, we still end up confusing user.
>>
>>
>> -Matthias
>>
>> On 4/11/17 8:23 AM, Damian Guy wrote:
>>> Eno, re: GlobalKTable - yeah that seems fine.
>>>
>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:
>>>
 About GlobalKTables, I suppose there is no reason why they cannot also use
 this KIP for consistency, e.g., today you have:

 public  GlobalKTable globalTable(final Serde keySerde,
 final Serde valSerde,
 final String topic,
 final String storeName)

 For consistency with the KIP you could also have an overload without the
 store name, for people who want to construct a global ktable, but don't
 care about querying it directly:

 public  GlobalKTable globalTable(final Serde keySerde,
 final Serde valSerde,
 final String topic)

 Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
 Michael for bringing it up.

 Eno



> On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
>
> Hi Michael, comments inline:
>
>> On 11 Apr 2017, at 03:25, Michael Noll  wrote:
>>
>> Thanks for the updates, Eno!
>>
>> In addition to what has already been said:  We should also explicitly
>> mention that this KIP is not touching GlobalKTable.  I'm sure that some
>> users will throw KTable and GlobalKTable into one conceptual "it's all
>> tables!" bucket and then wonder how the KIP might affect global tables.
>
> Good point, I'll add.
>
>
>>
>> Damian wrote:
>>> I think if no store name is provided users would still be able to query
>> the
>>> store, just the store name would be some internally generated name.
 They
>>> would be able to discover those names via the IQ API.
>>
>> I, too, think that users should be able to query a store even if its
 name
>> was internally generated.  After all, the data is already there /
>> materialized.
>
> Yes, there is nothing that will prevent users from querying internally
 generated stores, but they cannot
> assume a store will necessarily be queryable. So if it's there, they can
 query it. If it's not there, and they didn't
> provide a queryable name, they cannot complain and say "hey, where is my
 store". If they must absolutely be certain that
> a store is queryable, then they must provide a queryable name.
>
>
>>
>>
>> Damian wrote:
>>> I think for some stores it will make sense to not create a physical
>> store, i.e.,
>>> for thinks like `filter`, as this will save the rocksdb overhead. But i
>> guess that
>>> is more of an implementation detail.
>>
>> I think it would help if the KIP would clarify what we'd do in such a
>> case.  For example, if the user did not specify a store name 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Eno Thereska]

Hi Matthias,

See my note: "A store can be queryable whether it's materialized or not". I 
think we're on the same page. Stores with an internal name are also queryable. 

I'm just pointing out that. although that is the case today and with this KIP, 
I don't think we have an obligation to make stores with internal names 
queryable in the future. However, that is a discussion for a future point.

Eno




> On 11 Apr 2017, at 08:56, Matthias J. Sax  wrote:
> 
> +1 on including GlobalKTable
> 
> But I am not sure about the materialization / queryable question. For
> full consistency, all KTables should be queryable nevertheless if they
> are materialized or not. -- Maybe this is a second step though (even if
> I would like to get this done right away)
> 
> If we don't want all KTables to be queryable, ie, only those KTables
> that are materialized, then we should have a clear definition about
> this, and only allow to query stores, the user did specify a name for.
> This will simply the reasoning for users, what stores are queryable and
> what not. Otherwise, we still end up confusing user.
> 
> 
> -Matthias
> 
> On 4/11/17 8:23 AM, Damian Guy wrote:
>> Eno, re: GlobalKTable - yeah that seems fine.
>> 
>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:
>> 
>>> About GlobalKTables, I suppose there is no reason why they cannot also use
>>> this KIP for consistency, e.g., today you have:
>>> 
>>> public  GlobalKTable globalTable(final Serde keySerde,
>>> final Serde valSerde,
>>> final String topic,
>>> final String storeName)
>>> 
>>> For consistency with the KIP you could also have an overload without the
>>> store name, for people who want to construct a global ktable, but don't
>>> care about querying it directly:
>>> 
>>> public  GlobalKTable globalTable(final Serde keySerde,
>>> final Serde valSerde,
>>> final String topic)
>>> 
>>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
>>> Michael for bringing it up.
>>> 
>>> Eno
>>> 
>>> 
>>> 
 On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
 
 Hi Michael, comments inline:
 
> On 11 Apr 2017, at 03:25, Michael Noll  wrote:
> 
> Thanks for the updates, Eno!
> 
> In addition to what has already been said:  We should also explicitly
> mention that this KIP is not touching GlobalKTable.  I'm sure that some
> users will throw KTable and GlobalKTable into one conceptual "it's all
> tables!" bucket and then wonder how the KIP might affect global tables.
 
 Good point, I'll add.
 
 
> 
> Damian wrote:
>> I think if no store name is provided users would still be able to query
> the
>> store, just the store name would be some internally generated name.
>>> They
>> would be able to discover those names via the IQ API.
> 
> I, too, think that users should be able to query a store even if its
>>> name
> was internally generated.  After all, the data is already there /
> materialized.
 
 Yes, there is nothing that will prevent users from querying internally
>>> generated stores, but they cannot
 assume a store will necessarily be queryable. So if it's there, they can
>>> query it. If it's not there, and they didn't
 provide a queryable name, they cannot complain and say "hey, where is my
>>> store". If they must absolutely be certain that
 a store is queryable, then they must provide a queryable name.
 
 
> 
> 
> Damian wrote:
>> I think for some stores it will make sense to not create a physical
> store, i.e.,
>> for thinks like `filter`, as this will save the rocksdb overhead. But i
> guess that
>> is more of an implementation detail.
> 
> I think it would help if the KIP would clarify what we'd do in such a
> case.  For example, if the user did not specify a store name for
> `KTable#filter` -- would it be queryable?  If so, would this imply we'd
> always materialize the state store, or...?
 
 I'll clarify in the KIP with some more examples. Materialization will be
>>> an internal concept. A store can be queryable whether it's materialized or
>>> not
 (e.g., through advanced implementations that compute the value of a
>>> filter on a fly, rather than materialize the answer).
 
 Thanks,
 Eno
 
 
> 
> -Michael
> 
> 
> 
> 
> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy 
>>> wrote:
> 
>> Hi Eno,
>> 
>> Thanks for the update. I agree with what Matthias said. I wonder if
>>> the KIP
>> should talk less about materialization and more about 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Matthias J. Sax]

+1 on including GlobalKTable

But I am not sure about the materialization / queryable question. For
full consistency, all KTables should be queryable nevertheless if they
are materialized or not. -- Maybe this is a second step though (even if
I would like to get this done right away)

If we don't want all KTables to be queryable, ie, only those KTables
that are materialized, then we should have a clear definition about
this, and only allow to query stores, the user did specify a name for.
This will simply the reasoning for users, what stores are queryable and
what not. Otherwise, we still end up confusing user.


-Matthias

On 4/11/17 8:23 AM, Damian Guy wrote:
> Eno, re: GlobalKTable - yeah that seems fine.
> 
> On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:
> 
>> About GlobalKTables, I suppose there is no reason why they cannot also use
>> this KIP for consistency, e.g., today you have:
>>
>> public  GlobalKTable globalTable(final Serde keySerde,
>>  final Serde valSerde,
>>  final String topic,
>>  final String storeName)
>>
>> For consistency with the KIP you could also have an overload without the
>> store name, for people who want to construct a global ktable, but don't
>> care about querying it directly:
>>
>> public  GlobalKTable globalTable(final Serde keySerde,
>>  final Serde valSerde,
>>  final String topic)
>>
>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
>> Michael for bringing it up.
>>
>> Eno
>>
>>
>>
>>> On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
>>>
>>> Hi Michael, comments inline:
>>>
 On 11 Apr 2017, at 03:25, Michael Noll  wrote:

 Thanks for the updates, Eno!

 In addition to what has already been said:  We should also explicitly
 mention that this KIP is not touching GlobalKTable.  I'm sure that some
 users will throw KTable and GlobalKTable into one conceptual "it's all
 tables!" bucket and then wonder how the KIP might affect global tables.
>>>
>>> Good point, I'll add.
>>>
>>>

 Damian wrote:
> I think if no store name is provided users would still be able to query
 the
> store, just the store name would be some internally generated name.
>> They
> would be able to discover those names via the IQ API.

 I, too, think that users should be able to query a store even if its
>> name
 was internally generated.  After all, the data is already there /
 materialized.
>>>
>>> Yes, there is nothing that will prevent users from querying internally
>> generated stores, but they cannot
>>> assume a store will necessarily be queryable. So if it's there, they can
>> query it. If it's not there, and they didn't
>>> provide a queryable name, they cannot complain and say "hey, where is my
>> store". If they must absolutely be certain that
>>> a store is queryable, then they must provide a queryable name.
>>>
>>>


 Damian wrote:
> I think for some stores it will make sense to not create a physical
 store, i.e.,
> for thinks like `filter`, as this will save the rocksdb overhead. But i
 guess that
> is more of an implementation detail.

 I think it would help if the KIP would clarify what we'd do in such a
 case.  For example, if the user did not specify a store name for
 `KTable#filter` -- would it be queryable?  If so, would this imply we'd
 always materialize the state store, or...?
>>>
>>> I'll clarify in the KIP with some more examples. Materialization will be
>> an internal concept. A store can be queryable whether it's materialized or
>> not
>>> (e.g., through advanced implementations that compute the value of a
>> filter on a fly, rather than materialize the answer).
>>>
>>> Thanks,
>>> Eno
>>>
>>>

 -Michael




 On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy 
>> wrote:

> Hi Eno,
>
> Thanks for the update. I agree with what Matthias said. I wonder if
>> the KIP
> should talk less about materialization and more about querying? After
>> all,
> that is what is being provided from an end-users perspective.
>
> I think if no store name is provided users would still be able to
>> query the
> store, just the store name would be some internally generated name.
>> They
> would be able to discover those names via the IQ API
>
> I think for some stores it will make sense to not create a physical
>> store,
> i.e., for thinks like `filter`, as this will save the rocksdb
>> overhead. But
> i guess that is more of an implementation detail.
>
> Cheers,
> Damian
>
> On Tue, 11 Apr 2017 at 00:36 Eno Thereska 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Eno Thereska]

KIP updated, thank you. 

Eno
> On 11 Apr 2017, at 08:23, Damian Guy  wrote:
> 
> Eno, re: GlobalKTable - yeah that seems fine.
> 
> On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:
> 
>> About GlobalKTables, I suppose there is no reason why they cannot also use
>> this KIP for consistency, e.g., today you have:
>> 
>> public  GlobalKTable globalTable(final Serde keySerde,
>> final Serde valSerde,
>> final String topic,
>> final String storeName)
>> 
>> For consistency with the KIP you could also have an overload without the
>> store name, for people who want to construct a global ktable, but don't
>> care about querying it directly:
>> 
>> public  GlobalKTable globalTable(final Serde keySerde,
>> final Serde valSerde,
>> final String topic)
>> 
>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
>> Michael for bringing it up.
>> 
>> Eno
>> 
>> 
>> 
>>> On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
>>> 
>>> Hi Michael, comments inline:
>>> 
 On 11 Apr 2017, at 03:25, Michael Noll  wrote:
 
 Thanks for the updates, Eno!
 
 In addition to what has already been said:  We should also explicitly
 mention that this KIP is not touching GlobalKTable.  I'm sure that some
 users will throw KTable and GlobalKTable into one conceptual "it's all
 tables!" bucket and then wonder how the KIP might affect global tables.
>>> 
>>> Good point, I'll add.
>>> 
>>> 
 
 Damian wrote:
> I think if no store name is provided users would still be able to query
 the
> store, just the store name would be some internally generated name.
>> They
> would be able to discover those names via the IQ API.
 
 I, too, think that users should be able to query a store even if its
>> name
 was internally generated.  After all, the data is already there /
 materialized.
>>> 
>>> Yes, there is nothing that will prevent users from querying internally
>> generated stores, but they cannot
>>> assume a store will necessarily be queryable. So if it's there, they can
>> query it. If it's not there, and they didn't
>>> provide a queryable name, they cannot complain and say "hey, where is my
>> store". If they must absolutely be certain that
>>> a store is queryable, then they must provide a queryable name.
>>> 
>>> 
 
 
 Damian wrote:
> I think for some stores it will make sense to not create a physical
 store, i.e.,
> for thinks like `filter`, as this will save the rocksdb overhead. But i
 guess that
> is more of an implementation detail.
 
 I think it would help if the KIP would clarify what we'd do in such a
 case.  For example, if the user did not specify a store name for
 `KTable#filter` -- would it be queryable?  If so, would this imply we'd
 always materialize the state store, or...?
>>> 
>>> I'll clarify in the KIP with some more examples. Materialization will be
>> an internal concept. A store can be queryable whether it's materialized or
>> not
>>> (e.g., through advanced implementations that compute the value of a
>> filter on a fly, rather than materialize the answer).
>>> 
>>> Thanks,
>>> Eno
>>> 
>>> 
 
 -Michael
 
 
 
 
 On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy 
>> wrote:
 
> Hi Eno,
> 
> Thanks for the update. I agree with what Matthias said. I wonder if
>> the KIP
> should talk less about materialization and more about querying? After
>> all,
> that is what is being provided from an end-users perspective.
> 
> I think if no store name is provided users would still be able to
>> query the
> store, just the store name would be some internally generated name.
>> They
> would be able to discover those names via the IQ API
> 
> I think for some stores it will make sense to not create a physical
>> store,
> i.e., for thinks like `filter`, as this will save the rocksdb
>> overhead. But
> i guess that is more of an implementation detail.
> 
> Cheers,
> Damian
> 
> On Tue, 11 Apr 2017 at 00:36 Eno Thereska 
>> wrote:
> 
>> Hi Matthias,
>> 
>>> However, this still forces users, to provide a name for store that we
>>> must materialize, even if users are not interested in querying the
>>> stores. Thus, I would like to have overloads for all currently
>> existing
>>> methods having mandatory storeName paremeter, with overloads, that do
>>> not require the storeName parameter.
>> 
>> 
>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
>> clear, I'll 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Damian Guy]

Eno, re: GlobalKTable - yeah that seems fine.

On Tue, 11 Apr 2017 at 14:18 Eno Thereska  wrote:

> About GlobalKTables, I suppose there is no reason why they cannot also use
> this KIP for consistency, e.g., today you have:
>
> public  GlobalKTable globalTable(final Serde keySerde,
>  final Serde valSerde,
>  final String topic,
>  final String storeName)
>
> For consistency with the KIP you could also have an overload without the
> store name, for people who want to construct a global ktable, but don't
> care about querying it directly:
>
> public  GlobalKTable globalTable(final Serde keySerde,
>  final Serde valSerde,
>  final String topic)
>
> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
> Michael for bringing it up.
>
> Eno
>
>
>
> > On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
> >
> > Hi Michael, comments inline:
> >
> >> On 11 Apr 2017, at 03:25, Michael Noll  wrote:
> >>
> >> Thanks for the updates, Eno!
> >>
> >> In addition to what has already been said:  We should also explicitly
> >> mention that this KIP is not touching GlobalKTable.  I'm sure that some
> >> users will throw KTable and GlobalKTable into one conceptual "it's all
> >> tables!" bucket and then wonder how the KIP might affect global tables.
> >
> > Good point, I'll add.
> >
> >
> >>
> >> Damian wrote:
> >>> I think if no store name is provided users would still be able to query
> >> the
> >>> store, just the store name would be some internally generated name.
> They
> >>> would be able to discover those names via the IQ API.
> >>
> >> I, too, think that users should be able to query a store even if its
> name
> >> was internally generated.  After all, the data is already there /
> >> materialized.
> >
> > Yes, there is nothing that will prevent users from querying internally
> generated stores, but they cannot
> > assume a store will necessarily be queryable. So if it's there, they can
> query it. If it's not there, and they didn't
> > provide a queryable name, they cannot complain and say "hey, where is my
> store". If they must absolutely be certain that
> > a store is queryable, then they must provide a queryable name.
> >
> >
> >>
> >>
> >> Damian wrote:
> >>> I think for some stores it will make sense to not create a physical
> >> store, i.e.,
> >>> for thinks like `filter`, as this will save the rocksdb overhead. But i
> >> guess that
> >>> is more of an implementation detail.
> >>
> >> I think it would help if the KIP would clarify what we'd do in such a
> >> case.  For example, if the user did not specify a store name for
> >> `KTable#filter` -- would it be queryable?  If so, would this imply we'd
> >> always materialize the state store, or...?
> >
> > I'll clarify in the KIP with some more examples. Materialization will be
> an internal concept. A store can be queryable whether it's materialized or
> not
> > (e.g., through advanced implementations that compute the value of a
> filter on a fly, rather than materialize the answer).
> >
> > Thanks,
> > Eno
> >
> >
> >>
> >> -Michael
> >>
> >>
> >>
> >>
> >> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy 
> wrote:
> >>
> >>> Hi Eno,
> >>>
> >>> Thanks for the update. I agree with what Matthias said. I wonder if
> the KIP
> >>> should talk less about materialization and more about querying? After
> all,
> >>> that is what is being provided from an end-users perspective.
> >>>
> >>> I think if no store name is provided users would still be able to
> query the
> >>> store, just the store name would be some internally generated name.
> They
> >>> would be able to discover those names via the IQ API
> >>>
> >>> I think for some stores it will make sense to not create a physical
> store,
> >>> i.e., for thinks like `filter`, as this will save the rocksdb
> overhead. But
> >>> i guess that is more of an implementation detail.
> >>>
> >>> Cheers,
> >>> Damian
> >>>
> >>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska 
> wrote:
> >>>
>  Hi Matthias,
> 
> > However, this still forces users, to provide a name for store that we
> > must materialize, even if users are not interested in querying the
> > stores. Thus, I would like to have overloads for all currently
> existing
> > methods having mandatory storeName paremeter, with overloads, that do
> > not require the storeName parameter.
> 
> 
>  Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
>  clear, I'll clarify.
> 
>  Thanks
>  Eno
> 
> 
> > On 10 Apr 2017, at 16:00, Matthias J. Sax 
> >>> wrote:
> >
> > Thanks for pushing this KIP Eno.
> 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Eno Thereska]

About GlobalKTables, I suppose there is no reason why they cannot also use this 
KIP for consistency, e.g., today you have:

public  GlobalKTable globalTable(final Serde keySerde,
 final Serde valSerde,
 final String topic,
 final String storeName)

For consistency with the KIP you could also have an overload without the store 
name, for people who want to construct a global ktable, but don't care about 
querying it directly:

public  GlobalKTable globalTable(final Serde keySerde,
 final Serde valSerde,
 final String topic)

Damian, what do you think? I'm thinking of adding this to KIP. Thanks to 
Michael for bringing it up.

Eno

 

> On 11 Apr 2017, at 06:13, Eno Thereska  wrote:
> 
> Hi Michael, comments inline:
> 
>> On 11 Apr 2017, at 03:25, Michael Noll  wrote:
>> 
>> Thanks for the updates, Eno!
>> 
>> In addition to what has already been said:  We should also explicitly
>> mention that this KIP is not touching GlobalKTable.  I'm sure that some
>> users will throw KTable and GlobalKTable into one conceptual "it's all
>> tables!" bucket and then wonder how the KIP might affect global tables.
> 
> Good point, I'll add.
> 
> 
>> 
>> Damian wrote:
>>> I think if no store name is provided users would still be able to query
>> the
>>> store, just the store name would be some internally generated name. They
>>> would be able to discover those names via the IQ API.
>> 
>> I, too, think that users should be able to query a store even if its name
>> was internally generated.  After all, the data is already there /
>> materialized.
> 
> Yes, there is nothing that will prevent users from querying internally 
> generated stores, but they cannot
> assume a store will necessarily be queryable. So if it's there, they can 
> query it. If it's not there, and they didn't
> provide a queryable name, they cannot complain and say "hey, where is my 
> store". If they must absolutely be certain that
> a store is queryable, then they must provide a queryable name.
> 
> 
>> 
>> 
>> Damian wrote:
>>> I think for some stores it will make sense to not create a physical
>> store, i.e.,
>>> for thinks like `filter`, as this will save the rocksdb overhead. But i
>> guess that
>>> is more of an implementation detail.
>> 
>> I think it would help if the KIP would clarify what we'd do in such a
>> case.  For example, if the user did not specify a store name for
>> `KTable#filter` -- would it be queryable?  If so, would this imply we'd
>> always materialize the state store, or...?
> 
> I'll clarify in the KIP with some more examples. Materialization will be an 
> internal concept. A store can be queryable whether it's materialized or not
> (e.g., through advanced implementations that compute the value of a filter on 
> a fly, rather than materialize the answer). 
> 
> Thanks,
> Eno
> 
> 
>> 
>> -Michael
>> 
>> 
>> 
>> 
>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy  wrote:
>> 
>>> Hi Eno,
>>> 
>>> Thanks for the update. I agree with what Matthias said. I wonder if the KIP
>>> should talk less about materialization and more about querying? After all,
>>> that is what is being provided from an end-users perspective.
>>> 
>>> I think if no store name is provided users would still be able to query the
>>> store, just the store name would be some internally generated name. They
>>> would be able to discover those names via the IQ API
>>> 
>>> I think for some stores it will make sense to not create a physical store,
>>> i.e., for thinks like `filter`, as this will save the rocksdb overhead. But
>>> i guess that is more of an implementation detail.
>>> 
>>> Cheers,
>>> Damian
>>> 
>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska  wrote:
>>> 
 Hi Matthias,
 
> However, this still forces users, to provide a name for store that we
> must materialize, even if users are not interested in querying the
> stores. Thus, I would like to have overloads for all currently existing
> methods having mandatory storeName paremeter, with overloads, that do
> not require the storeName parameter.
 
 
 Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
 clear, I'll clarify.
 
 Thanks
 Eno
 
 
> On 10 Apr 2017, at 16:00, Matthias J. Sax 
>>> wrote:
> 
> Thanks for pushing this KIP Eno.
> 
> The update give a very clear description about the scope, that is super
> helpful for the discussion!
> 
> - To put it into my own words, the KIP focus is on enable to query all
> KTables.
> ** The ability to query a store is determined by providing a name for
> the store.
> ** At the same time, 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Eno Thereska]

Hi Michael, comments inline:

> On 11 Apr 2017, at 03:25, Michael Noll  wrote:
> 
> Thanks for the updates, Eno!
> 
> In addition to what has already been said:  We should also explicitly
> mention that this KIP is not touching GlobalKTable.  I'm sure that some
> users will throw KTable and GlobalKTable into one conceptual "it's all
> tables!" bucket and then wonder how the KIP might affect global tables.

Good point, I'll add.


> 
> Damian wrote:
>> I think if no store name is provided users would still be able to query
> the
>> store, just the store name would be some internally generated name. They
>> would be able to discover those names via the IQ API.
> 
> I, too, think that users should be able to query a store even if its name
> was internally generated.  After all, the data is already there /
> materialized.

Yes, there is nothing that will prevent users from querying internally 
generated stores, but they cannot
assume a store will necessarily be queryable. So if it's there, they can query 
it. If it's not there, and they didn't
provide a queryable name, they cannot complain and say "hey, where is my 
store". If they must absolutely be certain that
a store is queryable, then they must provide a queryable name.


> 
> 
> Damian wrote:
>> I think for some stores it will make sense to not create a physical
> store, i.e.,
>> for thinks like `filter`, as this will save the rocksdb overhead. But i
> guess that
>> is more of an implementation detail.
> 
> I think it would help if the KIP would clarify what we'd do in such a
> case.  For example, if the user did not specify a store name for
> `KTable#filter` -- would it be queryable?  If so, would this imply we'd
> always materialize the state store, or...?

I'll clarify in the KIP with some more examples. Materialization will be an 
internal concept. A store can be queryable whether it's materialized or not
(e.g., through advanced implementations that compute the value of a filter on a 
fly, rather than materialize the answer). 

Thanks,
Eno


> 
> -Michael
> 
> 
> 
> 
> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy  wrote:
> 
>> Hi Eno,
>> 
>> Thanks for the update. I agree with what Matthias said. I wonder if the KIP
>> should talk less about materialization and more about querying? After all,
>> that is what is being provided from an end-users perspective.
>> 
>> I think if no store name is provided users would still be able to query the
>> store, just the store name would be some internally generated name. They
>> would be able to discover those names via the IQ API
>> 
>> I think for some stores it will make sense to not create a physical store,
>> i.e., for thinks like `filter`, as this will save the rocksdb overhead. But
>> i guess that is more of an implementation detail.
>> 
>> Cheers,
>> Damian
>> 
>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska  wrote:
>> 
>>> Hi Matthias,
>>> 
 However, this still forces users, to provide a name for store that we
 must materialize, even if users are not interested in querying the
 stores. Thus, I would like to have overloads for all currently existing
 methods having mandatory storeName paremeter, with overloads, that do
 not require the storeName parameter.
>>> 
>>> 
>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
>>> clear, I'll clarify.
>>> 
>>> Thanks
>>> Eno
>>> 
>>> 
 On 10 Apr 2017, at 16:00, Matthias J. Sax 
>> wrote:
 
 Thanks for pushing this KIP Eno.
 
 The update give a very clear description about the scope, that is super
 helpful for the discussion!
 
 - To put it into my own words, the KIP focus is on enable to query all
 KTables.
  ** The ability to query a store is determined by providing a name for
 the store.
  ** At the same time, providing a name -- and thus making a store
 queryable -- does not say anything about an actual materialization (ie,
 being queryable and being materialized are orthogonal).
 
 
 I like this overall a lot. However, I would go one step further. Right
 now, you suggest to add new overload methods that allow users to
>> specify
 a storeName -- if `null` is provided and the store is not materialized,
 we ignore it completely -- if `null` is provided but the store must be
 materialized we generate a internal name. So far so good.
 
 However, this still forces users, to provide a name for store that we
 must materialize, even if users are not interested in querying the
 stores. Thus, I would like to have overloads for all currently existing
 methods having mandatory storeName paremeter, with overloads, that do
 not require the storeName parameter.
 
 Otherwise, we would still have some methods which optional storeName
 parameter and other method with mandatory storeName parameter -- thus,
 still some 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Eno Thereska]

Hi Damian,

Thanks. I agree, I'll adjust the tone so it's more about querying, while 
materialisation is an internal concept. 
If no store name is provided, the user would still be able to discover the 
store, however we are not making any strong guarantees in that case, since 
after all it i an internal decision on whether the names are discoverable. But 
yes, today you can discover them.

I think for V1 I'll go with actual materialization of the stores. We can then 
add a "viewer" option, i.e., not materialize, just compute on the fly. I can't 
help but think that down the line these two options will be considered by some 
sort of query optimizer that picks the best one. So I think both options will 
be needed, but starting the implementation with the materialization one.

Thanks
Eno

> On 11 Apr 2017, at 00:14, Damian Guy  wrote:
> 
> Hi Eno,
> 
> Thanks for the update. I agree with what Matthias said. I wonder if the KIP
> should talk less about materialization and more about querying? After all,
> that is what is being provided from an end-users perspective.
> 
> I think if no store name is provided users would still be able to query the
> store, just the store name would be some internally generated name. They
> would be able to discover those names via the IQ API
> 
> I think for some stores it will make sense to not create a physical store,
> i.e., for thinks like `filter`, as this will save the rocksdb overhead. But
> i guess that is more of an implementation detail.
> 
> Cheers,
> Damian
> 
> On Tue, 11 Apr 2017 at 00:36 Eno Thereska  wrote:
> 
>> Hi Matthias,
>> 
>>> However, this still forces users, to provide a name for store that we
>>> must materialize, even if users are not interested in querying the
>>> stores. Thus, I would like to have overloads for all currently existing
>>> methods having mandatory storeName paremeter, with overloads, that do
>>> not require the storeName parameter.
>> 
>> 
>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
>> clear, I'll clarify.
>> 
>> Thanks
>> Eno
>> 
>> 
>>> On 10 Apr 2017, at 16:00, Matthias J. Sax  wrote:
>>> 
>>> Thanks for pushing this KIP Eno.
>>> 
>>> The update give a very clear description about the scope, that is super
>>> helpful for the discussion!
>>> 
>>> - To put it into my own words, the KIP focus is on enable to query all
>>> KTables.
>>>  ** The ability to query a store is determined by providing a name for
>>> the store.
>>>  ** At the same time, providing a name -- and thus making a store
>>> queryable -- does not say anything about an actual materialization (ie,
>>> being queryable and being materialized are orthogonal).
>>> 
>>> 
>>> I like this overall a lot. However, I would go one step further. Right
>>> now, you suggest to add new overload methods that allow users to specify
>>> a storeName -- if `null` is provided and the store is not materialized,
>>> we ignore it completely -- if `null` is provided but the store must be
>>> materialized we generate a internal name. So far so good.
>>> 
>>> However, this still forces users, to provide a name for store that we
>>> must materialize, even if users are not interested in querying the
>>> stores. Thus, I would like to have overloads for all currently existing
>>> methods having mandatory storeName paremeter, with overloads, that do
>>> not require the storeName parameter.
>>> 
>>> Otherwise, we would still have some methods which optional storeName
>>> parameter and other method with mandatory storeName parameter -- thus,
>>> still some inconsistency.
>>> 
>>> 
>>> -Matthias
>>> 
>>> 
>>> On 4/9/17 8:35 AM, Eno Thereska wrote:
 Hi there,
 
 I've now done a V2 of the KIP, that hopefully addresses the feedback in
>> this discussion thread:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-114%3A+KTable+materialization+and+improved+semantics
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-114:+KTable+materialization+and+improved+semantics>.
>> Notable changes:
 
 - clearly outline what is in the scope of the KIP and what is not. We
>> ran into the issue where lots of useful, but somewhat tangential
>> discussions came up on interactive queries, declarative DSL etc. The exact
>> scope of this KIP is spelled out.
 - decided to go with overloaded methods, not .materialize(), to stay
>> within the spirit of the current declarative DSL.
 - clarified the depreciation plan
 - listed part of the discussion we had under rejected alternatives
 
 If you have any further feedback on this, let's continue on this thread.
 
 Thank you
 Eno
 
 
> On 1 Feb 2017, at 09:04, Eno Thereska  wrote:
> 
> Thanks everyone! I think it's time to do a V2 on the KIP so I'll do
>> that and we can see how it looks and continue the discussion from there.
>> Stay tuned.
> 
> Thanks
> Eno
> 

Build failed in Jenkins: kafka-trunk-jdk8 #1421

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-4866; Console consumer `print.value` property is ignored

--
[...truncated 112.75 KB...]
kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero STARTED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero PASSED

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
STARTED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
PASSED

kafka.api.PlaintextProducerSendTest > testSendBeforeAndAfterPartitionExpansion 
STARTED

kafka.api.PlaintextProducerSendTest > testSendBeforeAndAfterPartitionExpansion 
PASSED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic STARTED

kafka.api.FetchRequestTest > testShuffleWithSingleTopic PASSED

kafka.api.FetchRequestTest > testShuffle STARTED

kafka.api.FetchRequestTest > testShuffle PASSED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled STARTED

kafka.api.ClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.ClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.ClientIdQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic STARTED

kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne STARTED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList STARTED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas STARTED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas PASSED

kafka.api.ProducerFailureHandlingTest > 
testResponseTooLargeForReplicationWithAckAll STARTED

kafka.api.ProducerFailureHandlingTest > 
testResponseTooLargeForReplicationWithAckAll PASSED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic STARTED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition STARTED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed STARTED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero STARTED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.api.ProducerFailureHandlingTest > 
testPartitionTooLargeForReplicationWithAckAll STARTED

kafka.api.ProducerFailureHandlingTest > 
testPartitionTooLargeForReplicationWithAckAll PASSED

kafka.api.ProducerFailureHandlingTest > 
testNotEnoughReplicasAfterBrokerShutdown STARTED

kafka.api.ProducerFailureHandlingTest > 
testNotEnoughReplicasAfterBrokerShutdown PASSED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled 
STARTED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.UserClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.UserClientIdQuotaTest > 

[GitHub] kafka pull request #2661: kafka-4866: Kafka console consumer property is ign...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2661


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4866) Kafka console consumer property is ignored

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4866.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2661
[https://github.com/apache/kafka/pull/2661]

> Kafka console consumer property is ignored
> --
>
> Key: KAFKA-4866
> URL: https://issues.apache.org/jira/browse/KAFKA-4866
> Project: Kafka
>  Issue Type: Bug
>  Components: core, tools
>Affects Versions: 0.10.2.0
> Environment: Java 8, Mac
>Reporter: Frank Lyaruu
>Assignee: huxi
>Priority: Trivial
> Fix For: 0.11.0.0
>
>
> I'd like to read a topic using the console consumer, which prints the keys 
> but not the values:
> kafka-console-consumer --bootstrap-server someserver:9092 --from-beginning 
> --property print.key=true --property print.value=false --topic some_topic
> the print.value property seems to be completely ignored (I seems missing in 
> the source), but it is mentioned in the quickstart.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1420

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Fix typo in consumer ACL example

--
[...truncated 116.86 KB...]
kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne STARTED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList STARTED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas STARTED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas PASSED

kafka.api.ProducerFailureHandlingTest > 
testResponseTooLargeForReplicationWithAckAll STARTED

kafka.api.ProducerFailureHandlingTest > 
testResponseTooLargeForReplicationWithAckAll PASSED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic STARTED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition STARTED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed STARTED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero STARTED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.api.ProducerFailureHandlingTest > 
testPartitionTooLargeForReplicationWithAckAll STARTED

kafka.api.ProducerFailureHandlingTest > 
testPartitionTooLargeForReplicationWithAckAll PASSED

kafka.api.ProducerFailureHandlingTest > 
testNotEnoughReplicasAfterBrokerShutdown STARTED

kafka.api.ProducerFailureHandlingTest > 
testNotEnoughReplicasAfterBrokerShutdown PASSED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled 
STARTED

kafka.api.UserClientIdQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.UserClientIdQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.UserClientIdQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.UserClientIdQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
STARTED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
PASSED

kafka.api.SslProducerSendTest > testClose STARTED

kafka.api.SslProducerSendTest > testClose PASSED

kafka.api.SslProducerSendTest > testFlush STARTED

kafka.api.SslProducerSendTest > testFlush PASSED

kafka.api.SslProducerSendTest > testSendToPartition STARTED

kafka.api.SslProducerSendTest > testSendToPartition PASSED

kafka.api.SslProducerSendTest > testSendOffset STARTED

kafka.api.SslProducerSendTest > testSendOffset PASSED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime STARTED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.api.SslProducerSendTest > testSendBeforeAndAfterPartitionExpansion STARTED

kafka.api.SslProducerSendTest > testSendBeforeAndAfterPartitionExpansion PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.EndToEndClusterIdTest > testEndToEnd STARTED

kafka.api.EndToEndClusterIdTest > testEndToEnd PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

Build failed in Jenkins: kafka-trunk-jdk8 #1419

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-5043; Rename GroupCoordinator to FindCoordinator (KIP-98)

[ismael] MINOR: Document ordering contract of iterator for window stores and

--
[...truncated 151.87 KB...]
kafka.admin.AdminTest > testTopicCreationWithCollision STARTED

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK STARTED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.TopicCommandTest > testCreateIfNotExists STARTED

kafka.admin.TopicCommandTest > testCreateIfNotExists PASSED

kafka.admin.TopicCommandTest > testCreateAlterTopicWithRackAware STARTED

kafka.admin.TopicCommandTest > testCreateAlterTopicWithRackAware PASSED

kafka.admin.TopicCommandTest > testTopicDeletion STARTED

kafka.admin.TopicCommandTest > testTopicDeletion PASSED

kafka.admin.TopicCommandTest > testConfigPreservationAcrossPartitionAlteration 
STARTED

kafka.admin.TopicCommandTest > testConfigPreservationAcrossPartitionAlteration 
PASSED

kafka.admin.TopicCommandTest > testAlterIfExists STARTED

kafka.admin.TopicCommandTest > testAlterIfExists PASSED

kafka.admin.TopicCommandTest > testDeleteIfExists STARTED

kafka.admin.TopicCommandTest > testDeleteIfExists PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitionsWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitionsWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithMultiPartitionTopicAndMultipleConsumersWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithMultiPartitionTopicAndMultipleConsumersWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroup PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitions STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeConsumersWithNoAssignedPartitions PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeGroupWithNewConsumerWithShortInitializationTimeout STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeGroupWithNewConsumerWithShortInitializationTimeout PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeNonExistingGroupWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeNonExistingGroupWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNoMembersWithNewConsumer STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeExistingGroupWithNoMembersWithNewConsumer PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeNonExistingGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeNonExistingGroup PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroupWithNoMembers 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroupWithNoMembers 
PASSED

kafka.admin.ReassignPartitionsIntegrationTest > testRackAwareReassign STARTED

kafka.admin.ReassignPartitionsIntegrationTest > testRackAwareReassign PASSED

kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
STARTED

kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
PASSED

kafka.admin.ConfigCommandTest > testScramCredentials STARTED

kafka.admin.ConfigCommandTest > testScramCredentials PASSED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForTopicsEntityType STARTED

kafka.admin.ConfigCommandTest > shouldParseArgumentsForTopicsEntityType PASSED

kafka.admin.ConfigCommandTest > testUserClientQuotaOpts STARTED

kafka.admin.ConfigCommandTest > testUserClientQuotaOpts PASSED

kafka.admin.ConfigCommandTest > shouldAddTopicConfig STARTED

kafka.admin.ConfigCommandTest > shouldAddTopicConfig PASSED

kafka.admin.ConfigCommandTest > shouldAddClientConfig STARTED

kafka.admin.ConfigCommandTest > shouldAddClientConfig PASSED

kafka.admin.ConfigCommandTest > shouldDeleteBrokerConfig STARTED

kafka.admin.ConfigCommandTest > shouldDeleteBrokerConfig PASSED

kafka.admin.ConfigCommandTest > testQuotaConfigEntity STARTED

kafka.admin.ConfigCommandTest > testQuotaConfigEntity PASSED

kafka.admin.ConfigCommandTest > 
shouldNotUpdateBrokerConfigIfMalformedBracketConfig STARTED

kafka.admin.ConfigCommandTest > 
shouldNotUpdateBrokerConfigIfMalformedBracketConfig PASSED

kafka.admin.ConfigCommandTest > shouldFailIfUnrecognisedEntityType STARTED

kafka.admin.ConfigCommandTest > shouldFailIfUnrecognisedEntityType PASSED

kafka.admin.ConfigCommandTest > 
shouldNotUpdateBrokerConfigIfNonExistingConfigIsDeleted STARTED

[GitHub] kafka pull request #2839: Fix typo - Acls Examples, Adding or removing a pri...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2839


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Michael Noll]

Thanks for the updates, Eno!

In addition to what has already been said:  We should also explicitly
mention that this KIP is not touching GlobalKTable.  I'm sure that some
users will throw KTable and GlobalKTable into one conceptual "it's all
tables!" bucket and then wonder how the KIP might affect global tables.

Damian wrote:
> I think if no store name is provided users would still be able to query
the
> store, just the store name would be some internally generated name. They
> would be able to discover those names via the IQ API.

I, too, think that users should be able to query a store even if its name
was internally generated.  After all, the data is already there /
materialized.


Damian wrote:
> I think for some stores it will make sense to not create a physical
store, i.e.,
> for thinks like `filter`, as this will save the rocksdb overhead. But i
guess that
> is more of an implementation detail.

I think it would help if the KIP would clarify what we'd do in such a
case.  For example, if the user did not specify a store name for
`KTable#filter` -- would it be queryable?  If so, would this imply we'd
always materialize the state store, or...?

-Michael




On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy  wrote:

> Hi Eno,
>
> Thanks for the update. I agree with what Matthias said. I wonder if the KIP
> should talk less about materialization and more about querying? After all,
> that is what is being provided from an end-users perspective.
>
> I think if no store name is provided users would still be able to query the
> store, just the store name would be some internally generated name. They
> would be able to discover those names via the IQ API
>
> I think for some stores it will make sense to not create a physical store,
> i.e., for thinks like `filter`, as this will save the rocksdb overhead. But
> i guess that is more of an implementation detail.
>
> Cheers,
> Damian
>
> On Tue, 11 Apr 2017 at 00:36 Eno Thereska  wrote:
>
> > Hi Matthias,
> >
> > > However, this still forces users, to provide a name for store that we
> > > must materialize, even if users are not interested in querying the
> > > stores. Thus, I would like to have overloads for all currently existing
> > > methods having mandatory storeName paremeter, with overloads, that do
> > > not require the storeName parameter.
> >
> >
> > Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
> > clear, I'll clarify.
> >
> > Thanks
> > Eno
> >
> >
> > > On 10 Apr 2017, at 16:00, Matthias J. Sax 
> wrote:
> > >
> > > Thanks for pushing this KIP Eno.
> > >
> > > The update give a very clear description about the scope, that is super
> > > helpful for the discussion!
> > >
> > > - To put it into my own words, the KIP focus is on enable to query all
> > > KTables.
> > >   ** The ability to query a store is determined by providing a name for
> > > the store.
> > >   ** At the same time, providing a name -- and thus making a store
> > > queryable -- does not say anything about an actual materialization (ie,
> > > being queryable and being materialized are orthogonal).
> > >
> > >
> > > I like this overall a lot. However, I would go one step further. Right
> > > now, you suggest to add new overload methods that allow users to
> specify
> > > a storeName -- if `null` is provided and the store is not materialized,
> > > we ignore it completely -- if `null` is provided but the store must be
> > > materialized we generate a internal name. So far so good.
> > >
> > > However, this still forces users, to provide a name for store that we
> > > must materialize, even if users are not interested in querying the
> > > stores. Thus, I would like to have overloads for all currently existing
> > > methods having mandatory storeName paremeter, with overloads, that do
> > > not require the storeName parameter.
> > >
> > > Otherwise, we would still have some methods which optional storeName
> > > parameter and other method with mandatory storeName parameter -- thus,
> > > still some inconsistency.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 4/9/17 8:35 AM, Eno Thereska wrote:
> > >> Hi there,
> > >>
> > >> I've now done a V2 of the KIP, that hopefully addresses the feedback
> in
> > this discussion thread:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 114%3A+KTable+materialization+and+improved+semantics
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 114:+KTable+materialization+and+improved+semantics>.
> > Notable changes:
> > >>
> > >> - clearly outline what is in the scope of the KIP and what is not. We
> > ran into the issue where lots of useful, but somewhat tangential
> > discussions came up on interactive queries, declarative DSL etc. The
> exact
> > scope of this KIP is spelled out.
> > >> - decided to go with overloaded methods, not .materialize(), to stay
> > within the spirit of the current declarative DSL.
> > >> - clarified the 

[GitHub] kafka-site issue #54: Fix typo - Acls Examples, Adding or removing a princip...

[sunnykrGupta]

Github user sunnykrGupta commented on the issue:

https://github.com/apache/kafka-site/pull/54
  
Sure. Did PR https://github.com/apache/kafka/pull/2839 . :)  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2839: Fix typo - Acls Examples, Adding or removing a pri...

[sunnykrGupta]

GitHub user sunnykrGupta opened a pull request:

https://github.com/apache/kafka/pull/2839

Fix typo - Acls Examples, Adding or removing a principal as producer …

…or consumer

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sunnykrGupta/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2839.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2839


commit 3a95fdc22085b70c7b03c8427b1fc56c349bab80
Author: sunnykrgupta 
Date:   2017-04-11T08:58:01Z

Fix typo - Acls Examples, Adding or removing a principal as producer or 
consumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #54: Fix typo - Acls Examples, Adding or removing a princip...

[ijuma]

Github user ijuma commented on the issue:

https://github.com/apache/kafka-site/pull/54
  
Thanks for the PR, can you do a PR to the code repo 
https://github.com/apache/kafka/tree/trunk/docs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #54: Fix typo - Acls Examples, Adding or removing a princip...

[omkreddy]

Github user omkreddy commented on the issue:

https://github.com/apache/kafka-site/pull/54
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5043) Add FindCoordinatorRequest RPC stub and update InitPidRequest for KIP-98

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964010#comment-15964010
 ] 

ASF GitHub Bot commented on KAFKA-5043:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2825


> Add FindCoordinatorRequest RPC stub and update InitPidRequest for KIP-98
> 
>
> Key: KAFKA-5043
> URL: https://issues.apache.org/jira/browse/KAFKA-5043
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2825: KAFKA-5043 : Add FindCoordinatorRPC stub and updat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2825


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5043) Add FindCoordinatorRequest RPC stub and update InitPidRequest for KIP-98

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5043.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2825
[https://github.com/apache/kafka/pull/2825]

> Add FindCoordinatorRequest RPC stub and update InitPidRequest for KIP-98
> 
>
> Key: KAFKA-5043
> URL: https://issues.apache.org/jira/browse/KAFKA-5043
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3355) GetOffsetShell command doesn't work with SASL enabled Kafka

[Mohammed amine GARMES (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963998#comment-15963998
 ] 

Mohammed amine GARMES commented on KAFKA-3355:
--

Hello [~fredji],
this fix is for kafka 0.9.1.0.

Bests

> GetOffsetShell command doesn't work with SASL enabled Kafka
> ---
>
> Key: KAFKA-3355
> URL: https://issues.apache.org/jira/browse/KAFKA-3355
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.1
> Environment: Kafka 0.9.0.1
>Reporter: TAO XIAO
>Assignee: Ashish Singh
>
> I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe 
> this is due to old producer being used in GetOffsetShell.
> Kafka version 0.9.0.1
> Exception
> % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 
> localhost:9092 --topic test --time -1
> [2016-03-04 21:43:56,597] INFO Verifying properties 
> (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property client.id is overridden to 
> GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden to 
> localhost:9092 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to 
> 1000 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,674] INFO Fetching metadata from broker 
> BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s) 
> Set(test) (kafka.client.ClientUtils$)
> [2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing 
> (kafka.producer.SyncProducer)
> [2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)] failed 
> (kafka.client.ClientUtils$)
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:498)
>   at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304)
>   at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36)
>   at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52)
>   at 
> kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
>   at 
> kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.immutable.Range.foreach(Range.scala:166)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28)
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:120)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
>   at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78)
>   at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka-site pull request #54: Fix typo - Acls Examples, Adding or removing a ...

[sunnykrGupta]

GitHub user sunnykrGupta opened a pull request:

https://github.com/apache/kafka-site/pull/54

Fix typo - Acls Examples, Adding or removing a principal as producer …

…or consumer

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sunnykrGupta/kafka-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/54.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #54


commit 0ad74df61fe7bae84c3e825ed97b2d712bb0fb8d
Author: sunnykrgupta 
Date:   2017-04-11T08:00:17Z

Fix typo - Acls Examples, Adding or removing a principal as producer or 
consumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-5054) ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reassigned KAFKA-5054:
-

Assignee: Damian Guy

> ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized
> 
>
> Key: KAFKA-5054
> URL: https://issues.apache.org/jira/browse/KAFKA-5054
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.0
>
>
> {{putIfAbsent}} and {{delete}} should be synchronized as they involve at 
> least 2 operations on the underlying store and may result in inconsistent 
> results if someone were to query via IQ



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5054) ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized

[Damian Guy (JIRA)]

Damian Guy created KAFKA-5054:
-

 Summary: ChangeLoggingKeyValueByteStore delete and putIfAbsent 
should be synchronized
 Key: KAFKA-5054
 URL: https://issues.apache.org/jira/browse/KAFKA-5054
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.0
Reporter: Damian Guy
 Fix For: 0.11.0.0


{{putIfAbsent}} and {{delete}} should be synchronized as they involve at least 
2 operations on the underlying store and may result in inconsistent results if 
someone were to query via IQ



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

[Damian Guy]

Hi Onur,

It was in my previous email. But here it is again.



1. Better rebalance timing. We will try to rebalance only when all the
consumers in a group have joined. The challenge would be someone has to
define what does ALL consumers mean, it could either be a time or number of
consumers, etc.

2. Avoid frequent rebalance. For example, if there are 100 consumers in a
group, today, in the worst case, we may end up with 100 rebalances even if
all the consumers joined the group in a reasonably small amount of time.
Frequent rebalance is also a bad thing for brokers.

Having a client side configuration may solve problem 1 better because each
consumer group can potentially configure their own timing. However, it does
not really prevent frequent rebalance in general because some of the
consumers can be misconfigured. (This may have something to do with KIP-124
as well. But if quota is applied on the JoinGroup/SyncGroup request it may
cause some unwanted cascading effects.)

Having a broker side configuration may result in less flexibility for each
consumer group, but it can prevent frequent rebalance better. I think with
some reasonable design, the rebalance timing issue can be resolved on the
broker side as well. Matthias had a good point on extending the delay when
a new consumer joins a group (we actually did something similar to batch
ISR change propagation). For example, let's say on the broker side, we will
always delay 2 seconds each time we see a new consumer joining a consumer
group. This would probably work for most of the consumer groups and will
also limit the rebalance frequency to protect the brokers.

I am not sure about the streams use case here, but if something like 2
seconds of delay is acceptable for streams, I would prefer adding the
configuration to the broker so that we can address both problems.

On Thu, 6 Apr 2017 at 17:11 Onur Karaman 
wrote:

> Hi Damian.
>
> Can you copy the point Becket made earlier that you say isn't addressed?
>
> On Thu, Apr 6, 2017 at 2:51 AM, Damian Guy  wrote:
>
> > Thanks all, the Vote is now closed and the KIP has been accepted with 9
> +1s
> >
> > 3 binding::
> > Guozhang,
> > Jason,
> > Ismael
> >
> > 6 non-binding:
> > Bill,
> > Eno,
> > Mathieu,
> > Matthias,
> > Dong,
> > Mickael
> >
> > Thanks,
> > Damian
> >
> > On Thu, 6 Apr 2017 at 09:26 Ismael Juma  wrote:
> >
> > > Thanks for the KIP, +1 (binding).
> > >
> > > Ismael
> > >
> > > On Thu, Mar 30, 2017 at 8:55 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > +1 Thanks for the KIP!
> > > >
> > > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang 
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Sorry about the previous email, Gmail seems be collapsing them
> into a
> > > > > single thread on my inbox.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang <
> wangg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Damian, could you create a new thread for the voting process?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck  >
> > > > wrote:
> > > > > >
> > > > > >> +1(non-binding)
> > > > > >>
> > > > > >> On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska <
> > > eno.there...@gmail.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > +1 (non binding)
> > > > > >> >
> > > > > >> > Thanks
> > > > > >> > Eno
> > > > > >> > > On 30 Mar 2017, at 18:01, Matthias J. Sax <
> > > matth...@confluent.io>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > +1
> > > > > >> > >
> > > > > >> > > On 3/30/17 3:46 AM, Damian Guy wrote:
> > > > > >> > >> Hi All,
> > > > > >> > >>
> > > > > >> > >> I'd like to start the voting thread on KIP-134:
> > > > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > 134%3A+Delay+initial+consumer+group+rebalance
> > > > > >> > >>
> > > > > >> > >> Thanks,
> > > > > >> > >> Damian
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

[Damian Guy]

Hi Florian,

Thanks for the updates. The KIP is looking good.

Cheers,
Damian

On Fri, 7 Apr 2017 at 22:41 Matthias J. Sax  wrote:

> What about KafkaStreams#toString() method?
>
> I think, we want to deprecate it as with KIP-120 and the changes of this
> KIP, is gets obsolete.
>
> If we do so, please update the KIP accordingly.
>
>
> -Matthias
>
> On 3/28/17 7:00 PM, Matthias J. Sax wrote:
> > Thanks for updating the KIP!
> >
> > I think it's good as is -- I would not add anything more to TaskMetadata.
> >
> > About subtopologies and tasks. We do have the concept of subtopologies
> > already in KIP-120. It's only missing and ID that allow to link a
> > subtopology to a task.
> >
> > IMHO, adding a simple variable to `Subtopoloy` that provide the id
> > should be sufficient. We can simply document in the JavaDocs how
> > Subtopology and TaskMetadata can be linked to each other.
> >
> > I did update KIP-120 accordingly.
> >
> >
> > -Matthias
> >
> > On 3/28/17 3:45 PM, Florian Hussonnois wrote:
> >> Hi all,
> >>
> >> I've updated the KIP and the PR to reflect your suggestions.
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+130%3A+Expose+states+of+active+tasks+to+KafkaStreams+public+API
> >> https://github.com/apache/kafka/pull/2612
> >>
> >> Also, I've exposed property StreamThread#state as a string through the
> >> new class ThreadMetadata.
> >>
> >> Thanks,
> >>
> >> 2017-03-27 23:40 GMT+02:00 Florian Hussonnois  >> >:
> >>
> >> Hi Guozhang, Matthias,
> >>
> >> It's a great idea to add sub topologies descriptions. This would
> >> help developers to better understand topology concept.
> >>
> >> I agree that is not really user-friendly to check if
> >> `StreamsMetadata#streamThreads` is not returning null.
> >>
> >> The method name localThreadsMetadata looks good. In addition, it's
> >> more simple to build ThreadMetadata instances from the `StreamTask`
> >> class than from `StreamPartitionAssignor` class.
> >>
> >> I will work on modifications. As I understand, I have to add the
> >> property subTopologyId property to the TaskMetadata class - Am I
> right ?
> >>
> >> Thanks,
> >>
> >> 2017-03-26 0:25 GMT+01:00 Guozhang Wang  >> >:
> >>
> >> Re 1): this is a good point. May be we can move
> >> `StreamsMetadata#streamThreads` as
> >> `KafkaStreams#localThreadsMetadata`?
> >>
> >> 3): this is a minor suggestion about function name of
> >> `assignedPartitions`, to `topicPartitions` to be consistent with
> >> `StreamsMetadata`?
> >>
> >>
> >> Guozhang
> >>
> >> On Thu, Mar 23, 2017 at 4:30 PM, Matthias J. Sax
> >> > wrote:
> >>
> >> Thanks for the progress on this KIP. I think we are on the
> >> right path!
> >>
> >> Couple of comments/questions:
> >>
> >> (1) Why do we not consider the "rejected alternative" to add
> >> the method
> >> to KafkaStreams? The comment on #streamThreads() says:
> >>
> >> "Note this method will return null if called on
> >> {@link
> >> StreamsMetadata} which represent a remote application."
> >>
> >> Thus, if we cannot get any remote metadata, it seems not
> >> straight
> >> forward to not add it to KafkaStreams directly -- this would
> >> avoid
> >> invalid calls and `null` return value in the first place.
> >>
> >> I like the idea about exposing sub-topologies.:
> >>
> >> (2a) I would recommend to rename `topicsGroupId` to
> >> `subTopologyId` :)
> >>
> >> (2b) We could add this to KIP-120 already. However, I would
> >> not just
> >> link both via name, but leverage KIP-120 directly, and add a
> >> "Subtopology" member to the TaskMetadata class.
> >>
> >>
> >> Overall, I like the distinction of KIP-120 only exposing
> >> "static"
> >> information that can be determined before the topology get's
> >> started,
> >> while this KIP allow to access runtime information.
> >>
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 3/22/17 12:42 PM, Guozhang Wang wrote:
> >> > Thanks for the updated KIP, and sorry for the late
> replies!
> >> >
> >> > I think a little bit more about KIP-130, and I feel that
> >> if we are going
> >> > to deprecate the `toString` function (it is not explicitly
> >> said in the
> >> > KIP, so I'm not sure if you plan to still keep the
> >> > `KafkaStreams#toString` as is or are going to replace it
> >> with the
> >> 

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Damian Guy]

Hi Eno,

Thanks for the update. I agree with what Matthias said. I wonder if the KIP
should talk less about materialization and more about querying? After all,
that is what is being provided from an end-users perspective.

I think if no store name is provided users would still be able to query the
store, just the store name would be some internally generated name. They
would be able to discover those names via the IQ API

I think for some stores it will make sense to not create a physical store,
i.e., for thinks like `filter`, as this will save the rocksdb overhead. But
i guess that is more of an implementation detail.

Cheers,
Damian

On Tue, 11 Apr 2017 at 00:36 Eno Thereska  wrote:

> Hi Matthias,
>
> > However, this still forces users, to provide a name for store that we
> > must materialize, even if users are not interested in querying the
> > stores. Thus, I would like to have overloads for all currently existing
> > methods having mandatory storeName paremeter, with overloads, that do
> > not require the storeName parameter.
>
>
> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
> clear, I'll clarify.
>
> Thanks
> Eno
>
>
> > On 10 Apr 2017, at 16:00, Matthias J. Sax  wrote:
> >
> > Thanks for pushing this KIP Eno.
> >
> > The update give a very clear description about the scope, that is super
> > helpful for the discussion!
> >
> > - To put it into my own words, the KIP focus is on enable to query all
> > KTables.
> >   ** The ability to query a store is determined by providing a name for
> > the store.
> >   ** At the same time, providing a name -- and thus making a store
> > queryable -- does not say anything about an actual materialization (ie,
> > being queryable and being materialized are orthogonal).
> >
> >
> > I like this overall a lot. However, I would go one step further. Right
> > now, you suggest to add new overload methods that allow users to specify
> > a storeName -- if `null` is provided and the store is not materialized,
> > we ignore it completely -- if `null` is provided but the store must be
> > materialized we generate a internal name. So far so good.
> >
> > However, this still forces users, to provide a name for store that we
> > must materialize, even if users are not interested in querying the
> > stores. Thus, I would like to have overloads for all currently existing
> > methods having mandatory storeName paremeter, with overloads, that do
> > not require the storeName parameter.
> >
> > Otherwise, we would still have some methods which optional storeName
> > parameter and other method with mandatory storeName parameter -- thus,
> > still some inconsistency.
> >
> >
> > -Matthias
> >
> >
> > On 4/9/17 8:35 AM, Eno Thereska wrote:
> >> Hi there,
> >>
> >> I've now done a V2 of the KIP, that hopefully addresses the feedback in
> this discussion thread:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-114%3A+KTable+materialization+and+improved+semantics
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-114:+KTable+materialization+and+improved+semantics>.
> Notable changes:
> >>
> >> - clearly outline what is in the scope of the KIP and what is not. We
> ran into the issue where lots of useful, but somewhat tangential
> discussions came up on interactive queries, declarative DSL etc. The exact
> scope of this KIP is spelled out.
> >> - decided to go with overloaded methods, not .materialize(), to stay
> within the spirit of the current declarative DSL.
> >> - clarified the depreciation plan
> >> - listed part of the discussion we had under rejected alternatives
> >>
> >> If you have any further feedback on this, let's continue on this thread.
> >>
> >> Thank you
> >> Eno
> >>
> >>
> >>> On 1 Feb 2017, at 09:04, Eno Thereska  wrote:
> >>>
> >>> Thanks everyone! I think it's time to do a V2 on the KIP so I'll do
> that and we can see how it looks and continue the discussion from there.
> Stay tuned.
> >>>
> >>> Thanks
> >>> Eno
> >>>
>  On 30 Jan 2017, at 17:23, Matthias J. Sax 
> wrote:
> 
>  Hi,
> 
>  I think Eno's separation is very clear and helpful. In order to
>  streamline this discussion, I would suggest we focus back on point (1)
>  only, as this is the original KIP question.
> 
>  Even if I started to DSL design discussion somehow, because I thought
> it
>  might be helpful to resolve both in a single shot, I feel that we have
>  too many options about DSL design and we should split it up in two
>  steps. This will have the disadvantage that we will change the API
>  twice, but still, I think it will be a more focused discussion.
> 
>  I just had another look at the KIP, an it proposes 3 changes:
> 
>  1. add .materialized() -> IIRC it was suggested to name this
>  .materialize() though (can you maybe update the KIP Eno?)
>  2. remove print(), writeAsText(), and 

Credentials for Confluence

[Stephane Maarek]

Hi,

I’d like to create a KIP on Confluence but according to Matthias Sax I need
credentials.
Can you please provide me some?

Looking forward to redacting my first KIP :)

Regards,
Stephane