[jira] [Resolved] (KAFKA-5900) Create Connect metrics common to source and sink tasks

[ https://issues.apache.org/jira/browse/KAFKA-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5900. -- Resolution: Fixed Reviewer: Ewen Cheslack-Postava Fixed via

[GitHub] kafka pull request #3911: KAFKA-5900: Add task metrics common to both sink a...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3911 ---

[GitHub] kafka pull request #3969: KAFKA-5888: System test to check ordering of messa...

GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/3969 KAFKA-5888: System test to check ordering of messages with transactions and max.in.flight > 1 You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3968: KAFKA-5952 Refactor Consumer Fetcher metrics

GitHub user wushujames opened a pull request: https://github.com/apache/kafka/pull/3968 KAFKA-5952 Refactor Consumer Fetcher metrics This is ready for review. This does not have to be merged into 1.0.0, it can simply go to trunk. You can merge this pull request into a Git

[GitHub] kafka pull request #3897: KAFKA-5929: Save pre-assignment to file to avoid t...

GitHub user uncleGen reopened a pull request: https://github.com/apache/kafka/pull/3897 KAFKA-5929: Save pre-assignment to file to avoid too long text to display when do topic partition reassign When do partition reassign - before pr Pre-assignment will be printed directly.

[GitHub] kafka pull request #3894: KAFKA-5928: Avoid redundant requests to zookeeper ...

GitHub user uncleGen reopened a pull request: https://github.com/apache/kafka/pull/3894 KAFKA-5928: Avoid redundant requests to zookeeper when reassign topic partition We mistakenly request topic level information according to partitions config in the assignment json file. For

Re: [DISCUSS] KIP-190: Handle client-ids consistently between clients and brokers

It's a fair point that it would undo that sanitization. It's possible that for compatibility reasons, doing so would require a bit more work and care (e.g. supporting both sanitized and unsanitized for awhile so users have a chance to migrate). But I guess my point is that I view the location

[GitHub] kafka pull request #3967: KAFKA-5951: Autogenerate Producer RecordAccumulato...

GitHub user wushujames opened a pull request: https://github.com/apache/kafka/pull/3967 KAFKA-5951: Autogenerate Producer RecordAccumulator metrics Don't review this until https://github.com/apache/kafka/pull/3799/files is merged. This PR layers on top of

Running Kafka on docker containers

Hi, Is it a good idea to run kafka as docker containers in the production deployments? Do you guys foresee any blocks with this approach? Please advise. -Anoop P

Jenkins build is back to normal : kafka-0.11.0-jdk7 #313

See

[GitHub] kafka pull request #3966: KAFKA-5980: FailOnInvalidTimestamp does not log er...

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/3966 KAFKA-5980: FailOnInvalidTimestamp does not log error You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka

[jira] [Created] (KAFKA-5980) FailOnInvalidTimestamp does not log error

Michael Rylander created KAFKA-5980: --- Summary: FailOnInvalidTimestamp does not log error Key: KAFKA-5980 URL: https://issues.apache.org/jira/browse/KAFKA-5980 Project: Kafka Issue Type:

Build failed in Jenkins: kafka-0.11.0-jdk7 #312

See Changes: [me] MINOR: Update Jetty to 9.2.22.v20170606 -- [...truncated 2.44 MB...] org.apache.kafka.streams.integration.EosIntegrationTest >

Jenkins build is back to normal : kafka-0.10.1-jdk7 #128

See

Build failed in Jenkins: kafka-0.10.2-jdk7 #198

See Changes: [me] MINOR: Update Jetty to 9.2.22.v20170606 -- [...truncated 159.19 KB...] kafka.server.AbstractFetcherThreadTest > testConsumerLagRemovedWithPartition

[GitHub] kafka pull request #3964: MINOR: Update Jetty to 9.2.22.v20170606

Github user ewencp closed the pull request at: https://github.com/apache/kafka/pull/3964 ---

[jira] [Created] (KAFKA-5979) kip-120 changed internal streams topic naming

dan norwood created KAFKA-5979: -- Summary: kip-120 changed internal streams topic naming Key: KAFKA-5979 URL: https://issues.apache.org/jira/browse/KAFKA-5979 Project: Kafka Issue Type: Bug

[GitHub] kafka pull request #3965: KAFKA-5944: Unit tests for handling SASL authentic...

GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/3965 KAFKA-5944: Unit tests for handling SASL authentication failures in clients You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3964: MINOR: Update Jetty to 9.2.22.v20170606

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/3964 MINOR: Update Jetty to 9.2.22.v20170606 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka bump-jetty-version Alternatively you can

Re: [DISCUSS] KIP-206: Add support for UUID serialization and deserialization

Hi Ted, Thanks. The link to this thread is there now. Regards Jakub On Tue, Sep 26, 2017 at 7:22 PM, Ted Yu wrote: > Please add link to Discussion thread field. > > Looks good overall. > > On Tue, Sep 26, 2017 at 10:18 AM, Jakub Scholz wrote: > > > Hi, >

Re: [DISCUSS] KIP-206: Add support for UUID serialization and deserialization

Please add link to Discussion thread field. Looks good overall. On Tue, Sep 26, 2017 at 10:18 AM, Jakub Scholz wrote: > Hi, > > I'd like to start a discussion for KIP-206. It is about adding serializers > and deserializers for UUIDs. The details can be found on the wiki: >

[DISCUSS] KIP-206: Add support for UUID serialization and deserialization

Hi, I'd like to start a discussion for KIP-206. It is about adding serializers and deserializers for UUIDs. The details can be found on the wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-206%3A+Add+support+for+UUID+serialization+and+deserialization Thanks & Regards Jakub

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Thanks Tom, In my original KIP-170 I mentioned that the method public Map topicsPartitionCount(); was just a starting point for a general purpose ClusterState as it happened to be exactly the info we needed for our policy implementation :-) it definitely doesn't feel general

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Tom, as I said in the KIP-201 discussion I'm ok with having a unique DeleteTopicPolicy but then it could be useful having more information then just the topic name; partitions and offset for messages deletion could be useful for a fine grained use cases. Paolo Patierno Senior Software

[GitHub] kafka pull request #3963: MINOR:Updated Rabobank description

GitHub user manjuapu opened a pull request: https://github.com/apache/kafka/pull/3963 MINOR:Updated Rabobank description @dguy Please review You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka customer-logo-stream

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Edoardo, what about a single method in ClusterState > > interface ClusterState { > public Map topicsState(); > > } > > which could return a read-only snapshot of the cluster metadata ? > Sure that would work too. A concern with that is that we end up

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Paolo, I guess a RecordDeletionPolicy should work at the partition level, whereas the TopicDeletionPolicy should work at the topic level. But then we run into a similar situation as described in the motivation for KIP-201, where the administrator might have to write+configure two policies in

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

If we want to use the same DeleteTopicPolicy for the KIP-170 and for the delete records (in the topic) proposed by KIP-204, I think that we need more information other than topic name. As I mentioned in the KIP-204 discussion, the user could be interested in having more information because the

Jenkins build is back to normal : kafka-trunk-jdk8 #2073

See

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Thanks Tom, what about a single method in ClusterState interface ClusterState { public Map topicsState(); } which could return a read-only snapshot of the cluster metadata ? -- Edoardo Comar IBM Message Hub

[GitHub] kafka pull request #3962: KAFKA-5973 Exit when ShutdownableThread encounters...

GitHub user tedyu opened a pull request: https://github.com/apache/kafka/pull/3962 KAFKA-5973 Exit when ShutdownableThread encounters uncaught exception This PR installs UncaughtExceptionHandler which calls Exit.exit() . According to discussion on KAFKA-5973, exiting seems

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Tom, I think that we could live with the current authorizer based on delete topic (for both, deleting messages and topic as a whole) but then the RecordsDeletePolicy could be even more fine grained giving the possibility to avoid deleting messages for specific partitions (inside the topic)

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Edoardo and Ismael, Thanks for the comments. If we're going to have an new ClusterState parameter via which we can query the cluster state it would make sense to obtain the existing state from the ClusterState than from the RequestMetadata as in the KIP up to now. So I've updated the KIP

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Edoardo and Paolo, On 26 September 2017 at 14:10, Paolo Patierno wrote: > What could be useful use cases for having a RecordsDeletePolicy ? Records > can't be deleted for a topic name ? Starting from a specific offset ? > I can imagine some users wanting to prohibit

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Edoardo, I was referring to the KIP-107 where the delete records operation is coming with the authorizer I mentioned. You are referring to KIP-170 ... same digits, inverse order ! Sorry for that ;) Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Azure & IoT

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Paolo, the link you posted is to KIP-11, and I can't find the text you quoted. Our KIP-170 did indeed suggest a TopicDeletePolicy - but, as said, for our intent an Authorizer implementation will be usable instead, thanks -- Edoardo Comar IBM

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Edoardo, reading the KIP-170 at this point : 3) API Authorization Given the potential damage that can be caused if this API is used by mistake, it is important that we limit its usage to only authorized users. For this matter, we can take advantage of the existing authorization framework

[GitHub] kafka pull request #3668: KAFKA-5679: Add logging for broker termination due...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3668 ---

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Furthermore, the deletion case could be implemented by a custom Authorizer, as discussed in the KIP-204 thread. -- Edoardo Comar IBM Message Hub IBM UK Ltd, Hursley Park, SO21 2JN From: Edoardo Comar To:

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Paolo, our the motivation for suggesting a delete policy as described in KIP-170 was to avoid deletion of topics that have a specific usage (essential to other services). As you say, it would be entirely replaceable by an Authorizer implementation that performs the equivalent check. thanks

[jira] [Created] (KAFKA-5978) Transient failure in SslTransportLayerTest.testNetworkThreadTimeRecorded

Rajini Sivaram created KAFKA-5978: - Summary: Transient failure in SslTransportLayerTest.testNetworkThreadTimeRecorded Key: KAFKA-5978 URL: https://issues.apache.org/jira/browse/KAFKA-5978 Project:

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Tom, the KIP-170 doesn't propose to use a TopicDeletePolicy as policy classes are meant to be. It's referring to the authorization interface (KIP-11) with operation = DELETE and resource = TOPIC. You know

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

If we are introducing another interface, I think it makes sense to complement the request metadata with a context class that exposes non-request information that may be useful. I'll hopefully review the KIP in more detail today or tomorrow. Ismael On Tue, Sep 26, 2017 at 2:01 PM, Edoardo Comar

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Tom, yes it makes sense to have a single KIP for enhancing these policies. As Mickael pointed out, we need that the create/alter topic policy are able to assess the current cluster metadata. KIP-170 suggested a Provider interface with the minimal set of methods that we needed. However

Build failed in Jenkins: kafka-trunk-jdk8 #2072

See Changes: [damian.guy] KAFKA-5765; Move merge() from StreamsBuilder to KStream -- [...truncated 2.93 MB...] org.apache.kafka.common.security.JaasContextTest >

Jenkins build is back to normal : kafka-trunk-jdk8 #2071

See

Re: The way for the tools and the Admin Client API usage

Hi Paolo, The delete records KIP was voted and implemented before the AdminClient KIP. It couldn't use something that didn't exist. :) Regarding which part to take, I would say it depends on the contributor. Both moving the tool to `tools` and removing the ZK dependency are valuable. If someone

Re: [VOTE] KIP-202

More accurately, this KIP has received 3 binding +1 (Ismael, Damian, myself) and two non-binding +1 (Matthias, Bill). And 72 hours has passed without -1 votes, and hence we can conclude this thread as accepted. Guozhang On Mon, Sep 25, 2017 at 9:52 PM, Richard Yu

[jira] [Created] (KAFKA-5977) Upgrade RocksDB dependency to legally acceptable one

Stevo Slavic created KAFKA-5977: --- Summary: Upgrade RocksDB dependency to legally acceptable one Key: KAFKA-5977 URL: https://issues.apache.org/jira/browse/KAFKA-5977 Project: Kafka Issue Type:

The way for the tools and the Admin Client API usage

Hi devs and committers, I'd like to raise a point about Kafka tools and Admin Client API in order to have a sort of agreement on the path we should follow for having consistency in the project. We all know that the bigger part of tools is in Scala with only few of them in Java. At same time

Re: [DISCUSS] KIP-190: Handle client-ids consistently between clients and brokers

Hi Ewen, By consistency, I meant having all fields sanitized the same way we were previously doing for user principal. But re-reading your previous email, I'm guessing you meant to also remove the current user principal sanitization from the metrics (only use that internally for ZK) and have all

[GitHub] kafka pull request #3961: KAFKA-5976: RequestChannel.sendResponse should rec...

GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3961 KAFKA-5976: RequestChannel.sendResponse should record correct size for NetworkSend. When TRACE logging is enabled, RequestChannel.sendResponse records incorrect size for `Send` due to the fact that

[jira] [Created] (KAFKA-5976) RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging

huxihx created KAFKA-5976: - Summary: RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging Key: KAFKA-5976 URL: https://issues.apache.org/jira/browse/KAFKA-5976 Project: Kafka

[GitHub] kafka pull request #3916: KAFKA-5765 Move merge() from StreamsBuilder to KSt...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3916 ---

[jira] [Resolved] (KAFKA-5765) Move merge() from StreamsBuilder to KStream

[ https://issues.apache.org/jira/browse/KAFKA-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5765. --- Resolution: Fixed Fix Version/s: (was: 1.1.0) 1.0.0 Issue resolved by

[GitHub] kafka-site pull request #86: MINOR:Updating Rabobank description and Zalando...

Github user asfgit closed the pull request at: https://github.com/apache/kafka-site/pull/86 ---

Build failed in Jenkins: kafka-trunk-jdk8 #2070

See Changes: [ismael] KAFKA-5695; Test DeleteRecordsRequest in AuthorizerIntegrationTest -- [...truncated 1.89 MB...]

Speeding up Kafka Start cold startup time

Hi, We have our Kafka Brokers in AWS backed by st1 EBS drives. These are optimised for throughput On a kafka restart (warm restart), if some of the appropriate data is in the pagecache, everything goes well and the broker boots in just a few seconds. (1000 partitions) On a cold start (new

[GitHub] kafka-site issue #86: MINOR:Updating Rabobank description and Zalando image ...

Github user dguy commented on the issue: https://github.com/apache/kafka-site/pull/86 Thanks @manjuapu, LGTM. Can you please also file a PR against apache kafka for the change to `streams/index.html'? Otherwise that change will be lost. ---

[jira] [Resolved] (KAFKA-5695) Test DeleteRecordsRequest in AuthorizerIntegrationTest

[ https://issues.apache.org/jira/browse/KAFKA-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5695. Resolution: Fixed Fix Version/s: (was: 1.1.0) 1.0.0 > Test

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

Removals can only happen in major releases. Ismael On Tue, Sep 26, 2017 at 8:37 AM, Paolo Patierno wrote: > Hi devs, > > I know that we are already voting for this (+1 bindings from Ismael Juma > and Guozhang Wang, +1 non binding from Mickael Maison) but I'd like to ask > a

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

Hi devs, I know that we are already voting for this (+1 bindings from Ismael Juma and Guozhang Wang, +1 non binding from Mickael Maison) but I'd like to ask a question about the possible cycle release for this change. We are really closed to the 1.0.0 release which will have the --new-consumer

[GitHub] kafka pull request #3781: MINOR: Improve documentation of AdminClient

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3781 ---

[GitHub] kafka pull request #3614: KAFKA-5695; Test DeleteRecordsRequest in Authorize...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3614 ---

[jira] [Reopened] (KAFKA-5588) ConsoleConsumer : uselss --new-consumer option

[ https://issues.apache.org/jira/browse/KAFKA-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paolo Patierno reopened KAFKA-5588: --- > ConsoleConsumer : uselss --new-consumer option > --