[jira] [Created] (KAFKA-12674) Client failover takes 2-4 seconds on clean broker shutdown
Gwen Shapira created KAFKA-12674:
Summary: Client failover takes 2-4 seconds on clean broker shutdown
Key: KAFKA-12674
URL: https://issues.apache.org/jira/browse/KAFKA-12674
Project: Kafka
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Gwen Shapira
I ran two perf-producer clients against a 4-broker cluster running AWS, behind
ELB. And then did a rolling restart, taking down one broker at a time using
controlled shutdown.
I got the following errors on every broker shutdown:
{{[2021-04-16 01:31:39,846] WARN [Producer clientId=producer-1] Received
invalid metadata error in produce request on partition perf-test-3 due to
org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests
intended only for the leader, this error indicates that the broker is not the
current leader. For requests intended for any replica, this error indicates
that the broker is not a replica of the topic partition.. Going to request
metadata update now (org.apache.kafka.clients.producer.internals.Sender)}}
{{[2021-04-16 01:44:22,691] WARN [Producer clientId=producer-1] Connection to
node 0 (b0-pkc-7yrmj.us-east-2.aws.confluent.cloud/3.140.123.43:9092)
terminated during authentication. This may happen due to any of the following
reasons: (1) Authentication failed due to invalid credentials with brokers
older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow
HTTPS traffic), (3) Transient network issue.
(org.apache.kafka.clients.NetworkClient)}}
The "Connection to node... terminated" error continued for 2-4 seconds.
It looks like the metadata request was repeatedly sent to the node that just
went down. I'd expect it to go on an existing connection to one of the live
nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-7819) Trogdor - Improve RoundTripWorker
[ https://issues.apache.org/jira/browse/KAFKA-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-7819. - Fix Version/s: 2.2.0 Resolution: Fixed Closing since I noticed the PR was merged. > Trogdor - Improve RoundTripWorker > - > > Key: KAFKA-7819 > URL: https://issues.apache.org/jira/browse/KAFKA-7819 > Project: Kafka > Issue Type: Improvement >Reporter: Stanislav Kozlovski >Assignee: Stanislav Kozlovski >Priority: Minor > Fix For: 2.2.0 > > > Trogdor's RoundTripWorker task has a couple of shortcomings: > * Consumer GroupID is hardcoded and consumers use `KafkaConsumer#assign()`: > [https://github.com/apache/kafka/blob/12947f4f944955240fd14ce8b75fab5464ea6808/tools/src/main/java/org/apache/kafka/trogdor/workload/RoundTripWorker.java#L314] > Leaving you unable to run two separate instances of this worker on the same > partition in the same cluster, as the consumers would overwrite each other's > commits. It's probably better to add the task ID to the consumer group > * the task spec's `maxMessages` [is an > integer|https://github.com/apache/kafka/blob/12947f4f944955240fd14ce8b75fab5464ea6808/tools/src/main/java/org/apache/kafka/trogdor/workload/RoundTripWorkloadSpec.java#L39], > leaving you unable to schedule long-winded tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-3034) kafka.api.PlaintextConsumerTest > testSeek FAILED
[ https://issues.apache.org/jira/browse/KAFKA-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3034. - Resolution: Fixed > kafka.api.PlaintextConsumerTest > testSeek FAILED > - > > Key: KAFKA-3034 > URL: https://issues.apache.org/jira/browse/KAFKA-3034 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira >Priority: Major > Labels: transient-unit-test-failure > > See for example: > https://builds.apache.org/job/kafka-trunk-jdk7/921/console > It doesn't fail consistently, but happens fairly often. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-3035) Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset FAILED
[ https://issues.apache.org/jira/browse/KAFKA-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3035. - Resolution: Fixed > Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset FAILED > --- > > Key: KAFKA-3035 > URL: https://issues.apache.org/jira/browse/KAFKA-3035 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira >Assignee: Liquan Pei >Priority: Major > Labels: transient-unit-test-failure > > See: https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/1868/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-5034) Connect workers don't discover new Connector Plugins without Restart
[ https://issues.apache.org/jira/browse/KAFKA-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-5034. - Resolution: Won't Do Assuming this was fixed since there were many iterations on connector loading. > Connect workers don't discover new Connector Plugins without Restart > > > Key: KAFKA-5034 > URL: https://issues.apache.org/jira/browse/KAFKA-5034 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira >Priority: Major > > If I want to add a new Connector Plugin to a running distributed Connect > cluster, I need to copy the JAR to the classpath and then restart all the > workers so they will pick up the new plugin before I can create a connector. > This is both un-intuitive (most modern up can pick up changes dynamically) > and can get difficult when a connect cluster is shared between teams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-5000) Framework should log some progress information regularly to assist in troubleshooting
[
https://issues.apache.org/jira/browse/KAFKA-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira resolved KAFKA-5000.
-
Resolution: Fixed
Metrics seem like a better indicator than logs.
> Framework should log some progress information regularly to assist in
> troubleshooting
> -
>
> Key: KAFKA-5000
> URL: https://issues.apache.org/jira/browse/KAFKA-5000
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Gwen Shapira
>Priority: Major
>
> We get many questions of the type:
> "I started a connector, it doesn't seem to make any progress, I don't know
> what to do"
> I think that periodic "progress reports" on the worker logs may help.
> We have the offset commit message: "INFO
> WorkerSinkTask{id=cassandra-sink-connector-0} Committing offsets
> (org.apache.kafka.connect.runtime.WorkerSinkTask:244)"
> But I think we'd also want to know: topic, partition, offsets, how many rows
> were read from source/kafka and how many were successfully written.
> This will help determine if there is any progress being made and whether some
> partitions are "stuck" for some reason.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-3321) KafkaConfigStorage should never encounter commit without config data
[ https://issues.apache.org/jira/browse/KAFKA-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3321. - Resolution: Won't Do Let's assume this was already resolved or at least the root cause isn't interesting any more. > KafkaConfigStorage should never encounter commit without config data > > > Key: KAFKA-3321 > URL: https://issues.apache.org/jira/browse/KAFKA-3321 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira >Priority: Major > > After fixing KAFKA-2994, KafkaConfigStorage should no longer blow up with > surprise NPEs, but we did not figure out what causes it to see commit > messages before seeing the configurations that are committed. > This JIRA is to track down the root cause of the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9631) MockAdminClient doesn't handle CreateTopics optional fields
Gwen Shapira created KAFKA-9631:
---
Summary: MockAdminClient doesn't handle CreateTopics optional
fields
Key: KAFKA-9631
URL: https://issues.apache.org/jira/browse/KAFKA-9631
Project: Kafka
Issue Type: Bug
Reporter: Gwen Shapira
AdminClient's {{createTopics()}} method has a variant with two optional fields.
So I'd expect the following code to work correctly:
{{admin.createTopics(Collections.singletonList(new NewTopic(TOPIC_NAME,
Optional.empty(), Optional.empty(}}
Indeed it works great, as long as we are using the real KafkaAdminClient.
MockKafkaAdminClient tries to get number of replicas without checking that the
values make sense , and therefore it fails with:
{{java.lang.IllegalArgumentException: Illegal Capacity: -1}}{{at
java.base/java.util.ArrayList.(ArrayList.java:158)}}
{{ at
org.apache.kafka.clients.admin.MockAdminClient.createTopics(MockAdminClient.java:183)}}
{{ at org.apache.kafka.clients.admin.Admin.createTopics(Admin.java:125)}}
Making a mockery of the mock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9328) Move MirrorMaker 2.0 documentation to site-docs
Gwen Shapira created KAFKA-9328: --- Summary: Move MirrorMaker 2.0 documentation to site-docs Key: KAFKA-9328 URL: https://issues.apache.org/jira/browse/KAFKA-9328 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira Currently MirrorMaker 2.0 is documented here: https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md It will be much easier for our users to find and use if we include it in the main docs, side by side with the original MirrorMaker (while clarifying that MirrorMaker 2.0 is the preferred option going forward and the original is on deprecation schedule). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9327) GroupMetadata metrics are not documented
Gwen Shapira created KAFKA-9327: --- Summary: GroupMetadata metrics are not documented Key: KAFKA-9327 URL: https://issues.apache.org/jira/browse/KAFKA-9327 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira GroupMetadataManager includes quite a few gauges that look very useful! NumGroups, NumGroupsPreparingRebalance, NumGroupsStable, NumGroupsDead, NumGroupsEmpty. I couldn't find them in our site-docs, but they seem quite useful and will be nice to include. LastStableOffsetLag is also missing, but I'm not sure if it is useful enough to include. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9325) Document Follower Fetch
Gwen Shapira created KAFKA-9325: --- Summary: Document Follower Fetch Key: KAFKA-9325 URL: https://issues.apache.org/jira/browse/KAFKA-9325 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira Follower Fetch feature (KIP-392) is not really documented. The individual configs appear in the docs via auto-generation, but I think it is appropriate to explain all 3 configs together, and their intended use. Because this affects both client and brokers, it makes sense to document them in our site docs and in the consumer javadoc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8853) Create sustained connections test for Trogdor
[ https://issues.apache.org/jira/browse/KAFKA-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-8853. - Fix Version/s: 2.4.0 Reviewer: Stanislav Kozlovski Resolution: Fixed > Create sustained connections test for Trogdor > - > > Key: KAFKA-8853 > URL: https://issues.apache.org/jira/browse/KAFKA-8853 > Project: Kafka > Issue Type: Improvement > Components: system tests >Reporter: Scott Hendricks >Assignee: Scott Hendricks >Priority: Major > Fix For: 2.4.0 > > > There are currently tests to run a high amount of connects and disconnects, > but there are no tests that create and maintain connections to bring Kafka to > its limit. > My plan is to write a test that will take a desired number of clients > (KafkaConsumer, KafkaProducer, and AdminClient), the keep-alive rate for > these connections, and the number of threads desired to maintain these > connections. > Each worker will spawn the desired number of threads that will find > connections that need to be maintained and act on them. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8820) Use Admin API of Replica Reassignment in CLI tools
Gwen Shapira created KAFKA-8820: --- Summary: Use Admin API of Replica Reassignment in CLI tools Key: KAFKA-8820 URL: https://issues.apache.org/jira/browse/KAFKA-8820 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Steve Rodrigues KIP-455 and KAFKA-8345 add a protocol and AdminAPI that will be used for replica reassignments. We need to update the reassignment tool to use this new API rather than work with ZK directly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8813) Race condition when creating topics and changing their configuration
Gwen Shapira created KAFKA-8813:
---
Summary: Race condition when creating topics and changing their
configuration
Key: KAFKA-8813
URL: https://issues.apache.org/jira/browse/KAFKA-8813
Project: Kafka
Issue Type: Bug
Reporter: Gwen Shapira
In Partition.createLog we do:
{{val config = LogConfig.fromProps(logManager.currentDefaultConfig.originals,
props)val log = logManager.getOrCreateLog(topicPartition, config, isNew,
isFutureReplica)}}
Config changes that arrive after configs are loaded from ZK, but before
LogManager added the partition to `futureLogs` or `currentLogs` where the
dynamic config handlers picks up topics to update their configs, will be lost.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (KAFKA-8792) Default ZK configuration to disable AdminServer
[ https://issues.apache.org/jira/browse/KAFKA-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-8792. - Resolution: Fixed Reviewer: Ismael Juma Fix Version/s: 2.4.0 > Default ZK configuration to disable AdminServer > --- > > Key: KAFKA-8792 > URL: https://issues.apache.org/jira/browse/KAFKA-8792 > Project: Kafka > Issue Type: Improvement >Reporter: Gwen Shapira >Assignee: Gwen Shapira >Priority: Major > Fix For: 2.4.0 > > > Kafka ships with default ZK configuration. With the upgrade to ZK 3.5, our > defaults include running ZK's AdminServer on port 8080. This is an > unfortunate default as it tends to cause conflicts. > I suggest we default to disable ZK's AdminServer in the default ZK configs > that we ship. Users who want to use AdminServer can enable it and set the > port to something that works for them. Realistically, in most production > environments, a different ZK server will be used anyway. So this is mostly to > save new users who are trying Kafka on their own machine from running into > accidental and frustrating port conflicts. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8798) SaslOAuthBearerSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
Gwen Shapira created KAFKA-8798: --- Summary: SaslOAuthBearerSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl Key: KAFKA-8798 URL: https://issues.apache.org/jira/browse/KAFKA-8798 Project: Kafka Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Gwen Shapira https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6937/testReport/junit/kafka.api/SaslOAuthBearerSslEndToEndAuthorizationTest/testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl/ ``` Error Message org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records Stacktrace org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) at org.scalatest.Assertions.fail(Assertions.scala:1091) at org.scalatest.Assertions.fail$(Assertions.scala:1087) at org.scalatest.Assertions$.fail(Assertions.scala:1389) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:822) at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:781) at kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1312) at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1320) at kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:522) at kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:361) ``` -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8792) Default ZK configuration to disable AdminServer
Gwen Shapira created KAFKA-8792: --- Summary: Default ZK configuration to disable AdminServer Key: KAFKA-8792 URL: https://issues.apache.org/jira/browse/KAFKA-8792 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira Kafka ships with default ZK configuration. With the upgrade to ZK 3.5, our defaults include running ZK's AdminServer on port 8080. This is an unfortunate default as it tends to cause conflicts. I suggest we default to disable ZK's AdminServer in the default ZK configs that we ship. Users who want to use AdminServer can enable it and set the port to something that works for them. Realistically, in most production environments, a different ZK server will be used anyway. So this is mostly to save new users who are trying Kafka on their own machine from running into accidental and frustrating port conflicts. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8644) The Kafka protocol generator should allow null defaults for bytes and array fields
[ https://issues.apache.org/jira/browse/KAFKA-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-8644. - Resolution: Fixed Fix Version/s: 2.4.0 > The Kafka protocol generator should allow null defaults for bytes and array > fields > -- > > Key: KAFKA-8644 > URL: https://issues.apache.org/jira/browse/KAFKA-8644 > Project: Kafka > Issue Type: Improvement >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > Fix For: 2.4.0 > > > The Kafka protocol generator should allow null defaults for bytes and array > fields. Currently, null defaults are only allowed for string fields. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8560) The Kafka protocol generator should support common structures
[ https://issues.apache.org/jira/browse/KAFKA-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-8560. - Resolution: Fixed Fix Version/s: 2.4.0 > The Kafka protocol generator should support common structures > - > > Key: KAFKA-8560 > URL: https://issues.apache.org/jira/browse/KAFKA-8560 > Project: Kafka > Issue Type: Improvement >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > Fix For: 2.4.0 > > > The Kafka protocol generator should support common structures. This would > make things simpler in cases where we need to refer to a single structure > from multiple places in a message. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7469) Broker keeps group rebalance after adding FS
[ https://issues.apache.org/jira/browse/KAFKA-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-7469. - Resolution: Fixed > Broker keeps group rebalance after adding FS > - > > Key: KAFKA-7469 > URL: https://issues.apache.org/jira/browse/KAFKA-7469 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.10.1.1 >Reporter: Boaz >Priority: Major > Fix For: 0.8.1.2 > > > Hi, > I'm using a kafka_2.10-0.10.1.1 with 3 brokers cluster. > A few days ago, we started running out of FS and our System Admin allocated > some more Disc space. After the allocation, we started experiencing high lags > on the consumers which kept growing. > On the Consumer side, we saw that no data is being consumed and the following > message keep coming in the log files: > o.a.k.c.c.internals.AbstractCoordinator : (Re-)joining group > AutoPaymentSyncGroup > > On the Broker logs, we keep seeing seeing messages of restabilize group : > [2018-09-27 18:38:16,264] INFO [GroupCoordinator 0]: Preparing to restabilize > group AutoPaymentActivityGroup with old generation 357 > (kafka.coordinator.GroupCoordinator) > [2018-09-27 18:38:16,264] INFO [GroupCoordinator 0]: Preparing to restabilize > group AutoPaymentCreditCardTypeGroup with old generation 278 > (kafka.coordinator.GroupCoordinator) > [2018-09-27 18:38:16,284] INFO [GroupCoordinator 0]: Preparing to restabilize > group AutoPaymentAuthChannelGroup with old generation 349 > (kafka.coordinator.GroupCoordinator) > [2018-09-27 18:38:16,411] INFO [GroupCoordinator 0]: Preparing to restabilize > group AutoPaymentAuthCodeGroup with old generation 284 > (kafka.coordinator.GroupCoordinator) > [2018-09-27 18:38:16,463] INFO [GroupCoordinator 0]: Preparing to restabilize > group AutoPaymentInteractionSyncGroup with old generation 359 > (kafka.coordinator.GroupCoordinator) > [2018-09-27 18:38:16,464] INFO [GroupCoordinator 0]: Preparing to restabilize > group AutoPaymentSyncGroup with old generation 358 > (kafka.coordinator.GroupCoordinator) > > After bouncing the broker, the issue was resolved. > Thanks, > Boaz. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8344) Fix vagrant-up.sh to work with AWS properly
[
https://issues.apache.org/jira/browse/KAFKA-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira resolved KAFKA-8344.
-
Resolution: Fixed
Fix Version/s: 2.3.0
> Fix vagrant-up.sh to work with AWS properly
> ---
>
> Key: KAFKA-8344
> URL: https://issues.apache.org/jira/browse/KAFKA-8344
> Project: Kafka
> Issue Type: Bug
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 2.3.0
>
>
> I tried to run {{vagrant/vagrant-up.sh --aws}} with the following
> Vagrantfile.local.
> {code}
> enable_dns = true
> enable_hostmanager = false
> # EC2
> ec2_access_key = ""
> ec2_secret_key = ""
> ec2_keypair_name = "keypair"
> ec2_keypair_file = "/path/to/keypair/file"
> ec2_region = "ap-northeast-1"
> ec2_ami = "ami-0905ffddadbfd01b7"
> ec2_security_groups = "sg-"
> ec2_subnet_id = "subnet-"
> {code}
> EC2 instances were successfully created, but it failed with the following
> error after that.
> {code}
> $ vagrant/vagrant-up.sh --aws
> (snip)
> An active machine was found with a different provider. Vagrant
> currently allows each machine to be brought up with only a single
> provider at a time. A future version will remove this limitation.
> Until then, please destroy the existing machine to up with a new
> provider.
> Machine name: zk1
> Active provider: aws
> Requested provider: virtualbox
> {code}
> It seems that the {{vagrant hostmanager}} command also requires
> {{--provider=aws}} option, in addition to {{vagrant up}}.
> With that option, it succeeded as follows:
> {code}
> $ git diff
> diff --git a/vagrant/vagrant-up.sh b/vagrant/vagrant-up.sh
> index 6a4ef9564..9210a5357 100755
> --- a/vagrant/vagrant-up.sh
> +++ b/vagrant/vagrant-up.sh
> @@ -220,7 +220,7 @@ function bring_up_aws {
> # We still have to bring up zookeeper/broker nodes serially
> echo "Bringing up zookeeper/broker machines serially"
> vagrant up --provider=aws --no-parallel --no-provision
> $zk_broker_machines $debug
> -vagrant hostmanager
> +vagrant hostmanager --provider=aws
> vagrant provision
> fi
> @@ -231,11 +231,11 @@ function bring_up_aws {
> local vagrant_rsync_temp_dir=$(mktemp -d);
> TMPDIR=$vagrant_rsync_temp_dir vagrant_batch_command "vagrant up
> $debug --provider=aws" "$worker_machines" "$max_parallel"
> rm -rf $vagrant_rsync_temp_dir
> -vagrant hostmanager
> +vagrant hostmanager --provider=aws
> fi
> else
> vagrant up --provider=aws --no-parallel --no-provision $debug
> -vagrant hostmanager
> +vagrant hostmanager --provider=aws
> vagrant provision
> fi
> $ vagrant/vagrant-up.sh --aws
> (snip)
> ==> broker3: Running provisioner: shell...
> broker3: Running: /tmp/vagrant-shell20190509-25399-8f1wgz.sh
> broker3: Killing server
> broker3: No kafka server to stop
> broker3: Starting server
> $ vagrant status
> Current machine states:
> zk1 running (aws)
> broker1 running (aws)
> broker2 running (aws)
> broker3 running (aws)
> This environment represents multiple VMs. The VMs are all listed
> above with their current state. For more information about a specific
> VM, run `vagrant status NAME`.
> $ vagrant ssh broker1
> (snip)
> ubuntu@ip-172-16-0-62:~$ /opt/kafka-dev/bin/kafka-topics.sh
> --bootstrap-server broker1:9092,broker2:9092,broker3:9092 --create
> --partitions 1 --replication-factor 3 --topic sandbox
> (snip)
> ubuntu@ip-172-16-0-62:~$ /opt/kafka-dev/bin/kafka-topics.sh
> --bootstrap-server broker1:9092,broker2:9092,broker3:9092 --list
> (snip)
> sandbox
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8338) Improve consumer offset expiration logic to take subscription into account
Gwen Shapira created KAFKA-8338: --- Summary: Improve consumer offset expiration logic to take subscription into account Key: KAFKA-8338 URL: https://issues.apache.org/jira/browse/KAFKA-8338 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira Currently, we expire consumer offsets for a group after the group is considered gone. There is a case where the consumer group still exists, but is now subscribed to different topics. In that case, the offsets of the old topics will never expire and if lag is monitored, the monitors will show ever-growing lag on those topics. We need to improve the logic to expire the consumer offsets if the consumer group didn't subscribe to specific topics/partitions for enough time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira resolved KAFKA-7965.
-
Resolution: Fixed
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 1.1.1, 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-8168) Add a generated ApiMessageType class
[ https://issues.apache.org/jira/browse/KAFKA-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-8168. - Resolution: Fixed Reviewer: Gwen Shapira Fix Version/s: 2.3.0 > Add a generated ApiMessageType class > > > Key: KAFKA-8168 > URL: https://issues.apache.org/jira/browse/KAFKA-8168 > Project: Kafka > Issue Type: Improvement >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe >Priority: Major > Fix For: 2.3.0 > > > Add a generated ApiMessageType class. This will make it easier to do > operations based on the type of an ApiMessage. > Once all the RPCs are converted to use protocol generation, we can switch to > using this instead of ApiKeys.java (possibly renaming this to ApiKeys.java?) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8180) Deleting large number of topics can block the controller for the time it takes to delete all of them
Gwen Shapira created KAFKA-8180: --- Summary: Deleting large number of topics can block the controller for the time it takes to delete all of them Key: KAFKA-8180 URL: https://issues.apache.org/jira/browse/KAFKA-8180 Project: Kafka Issue Type: Bug Components: controller Reporter: Gwen Shapira Scenario: - Create large number of topics (In my experiment: 400 topics with 12 partitions each ) - Use the admin client to delete all of them in a single batch operation - Try to bounce another broker while this is going on As you can see from the logs and metrics - topic deletion happens synchronously in the controller and it does not do anything else (leader elections for instance) while it is busy deleting (which can take many minutes for large batches). I recommend fixing it by throttling the deletes - no matter how large a batch the client sent, the controller should delete a subset and complete a full cycle before deleting the next subset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7730) Limit total number of active connections in the broker
[ https://issues.apache.org/jira/browse/KAFKA-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-7730. - Resolution: Fixed Reviewer: Gwen Shapira Fix Version/s: 2.3.0 > Limit total number of active connections in the broker > -- > > Key: KAFKA-7730 > URL: https://issues.apache.org/jira/browse/KAFKA-7730 > Project: Kafka > Issue Type: New Feature > Components: network >Affects Versions: 2.1.0 >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.3.0 > > > Add a new listener config `max.connections` to limit the maximum number of > active connections on each listener. See > https://cwiki.apache.org/confluence/display/KAFKA/KIP-402%3A+Improve+fairness+in+SocketServer+processors > for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-1927) Replace requests in kafka.api with requests in org.apache.kafka.common.requests
[ https://issues.apache.org/jira/browse/KAFKA-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-1927. - Resolution: Fixed Since the only remaining item is a refactoring, I think we can consider this done. > Replace requests in kafka.api with requests in > org.apache.kafka.common.requests > --- > > Key: KAFKA-1927 > URL: https://issues.apache.org/jira/browse/KAFKA-1927 > Project: Kafka > Issue Type: Improvement >Reporter: Jay Kreps >Assignee: Gwen Shapira >Priority: Major > Attachments: KAFKA-1927.patch > > > The common package introduced a better way of defining requests using a new > protocol definition DSL and also includes wrapper objects for these. > We should switch KafkaApis over to use these request definitions and consider > the scala classes deprecated (we probably need to retain some of them for a > while for the scala clients). > This will be a big improvement because > 1. We will have each request now defined in only one place (Protocol.java) > 2. We will have built-in support for multi-version requests > 3. We will have much better error messages (no more cryptic underflow errors) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-1705) Add MR layer to Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-1705. - Resolution: Won't Fix If I didn't do it by now... Also, MapReduce is so 2014 :) > Add MR layer to Kafka > - > > Key: KAFKA-1705 > URL: https://issues.apache.org/jira/browse/KAFKA-1705 > Project: Kafka > Issue Type: Improvement >Reporter: Gwen Shapira >Assignee: Gwen Shapira >Priority: Major > > Many NoSQL-type storage systems (HBase, Mongo, > Cassandra) and file formats (Avro, Parquet) provide is a MapReduce > integration layer - usually an InputFormat, OutputFormat and a utility > class. Sometimes there's also an abstract Job and Mapper that do more > setup, which can make things even more convenient. > This is different than the existing Hadoop contrib project or Camus in that > an MR layer will be providing components for use in MR jobs, not an entire > job that ingests data from Kafka to HDFS. > The benefits I see for a MapReduce layer are: > * Developers can create their own jobs, processing the data as it is > ingested - rather than having to process it in two steps. > * There's reusable components for developers looking to integrate with > Kafka, rather than having everyone implement their own solution. > * Hadoop developers expect projects to have this layer. > * Spark reuses Hadoop's InputFormat and OutputFormat - so we get Spark > integration for free. > * There's a layer to plug the delegation token code into and make it > invisible to MapReduce developers. Without this, everyone who writes > MR jobs will need to think about how to implement authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7121) Intermittently, Connectors fail to assign tasks and keep retrying every second forever.
Gwen Shapira created KAFKA-7121:
---
Summary: Intermittently, Connectors fail to assign tasks and keep
retrying every second forever.
Key: KAFKA-7121
URL: https://issues.apache.org/jira/browse/KAFKA-7121
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Gwen Shapira
We started a connector, and even though it is in RUNNING status, tasks are not
getting assigned:
{"name":"prod-xxx-v2","connector":{"state":"RUNNING","worker_id":"[0.0.0.0:8083|http://0.0.0.0:8083/]"},"tasks":[],"type":"sink"}
Other connectors are running without issues.
Attempt to restart the connector returned 409 status.
Logs show the following messages, keep repeating for hours:
```
[2018-06-29 20:23:19,288] ERROR Task reconfiguration for prod-xxx-v2 failed
unexpectedly, this connector will not be properly reconfigured unless manually
triggered. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:956)
[2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +]
"POST /connectors/prod-xxx-v2/tasks?forward=false HTTP/1.1" 409 113 0
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +]
"POST /connectors/prod-xxx-v2/tasks?forward=true HTTP/1.1" 409 113 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2018-06-29 20:23:19,289] INFO 10.200.149.201 - - [29/Jun/2018:20:23:19 +]
"POST /connectors/prod-xxx-v2/tasks HTTP/1.1" 409 113 1
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2018-06-29 20:23:19,289] ERROR Request to leader to reconfigure connector
tasks failed
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1018)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot
complete request because of a conflicting operation (e.g. worker rebalance)
at
org.apache.kafka.connect.runtime.rest.RestServer.httpRequest(RestServer.java:229)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1015)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
Gwen Shapira created KAFKA-7120: --- Summary: When Connect throws CONFLICT error for REST requests, it will help to see more details Key: KAFKA-7120 URL: https://issues.apache.org/jira/browse/KAFKA-7120 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira Right now, we throw: throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), "Cannot complete request because of a conflicting operation (e.g. worker rebalance)"); There's no information about WHICH request can't be completed. It will help to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-5384) KIP-162: Enable topic deletion by default
[ https://issues.apache.org/jira/browse/KAFKA-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-5384. - Resolution: Fixed Fix Version/s: (was: 0.12.0.0) 0.11.1.0 Issue resolved by pull request 3241 [https://github.com/apache/kafka/pull/3241] > KIP-162: Enable topic deletion by default > - > > Key: KAFKA-5384 > URL: https://issues.apache.org/jira/browse/KAFKA-5384 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Gwen Shapira > Fix For: 0.11.1.0 > > > Change default of delete.topic.enable to true > Remove delete.topic.enable config from config/server.properties. > See KIP for details: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5498) Connect validation API stops returning recommendations for some fields after the right sequence of requests
[
https://issues.apache.org/jira/browse/KAFKA-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira resolved KAFKA-5498.
-
Resolution: Fixed
Fix Version/s: 0.11.1.0
Issue resolved by pull request 3412
[https://github.com/apache/kafka/pull/3412]
> Connect validation API stops returning recommendations for some fields after
> the right sequence of requests
> ---
>
> Key: KAFKA-5498
> URL: https://issues.apache.org/jira/browse/KAFKA-5498
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.11.0.0, 0.11.1.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> If you issue the right sequence of requests against this API, it starts
> behaving differently, omitting certain fields (at a minimum recommended
> values, which is how I noticed this). If you start with
> {code}
> $ curl -X PUT -H "Content-Type: application/json" --data '{"connector.class":
> "org.apache.kafka.connect.file.FileStreamSourceConnector", "name": "file",
> "transforms": "foo"}'
> http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate
> | jq
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 100 5845 100 5730 100 115 36642735 --:--:-- --:--:-- --:--:-- 36496
> {
> "name": "org.apache.kafka.connect.file.FileStreamSourceConnector",
> "error_count": 4,
> "groups": [
> "Common",
> "Transforms",
> "Transforms: foo"
> ],
> "configs": [
> {
> "definition": {
> "name": "name",
> "type": "STRING",
> "required": true,
> "default_value": null,
> "importance": "HIGH",
> "documentation": "Globally unique name to use for this connector.",
> "group": "Common",
> "width": "MEDIUM",
> "display_name": "Connector name",
> "dependents": [],
> "order": 1
> },
> "value": {
> "name": "name",
> "value": "file",
> "recommended_values": [],
> "errors": [],
> "visible": true
> }
> },
> {
> "definition": {
> "name": "connector.class",
> "type": "STRING",
> "required": true,
> "default_value": null,
> "importance": "HIGH",
> "documentation": "Name or alias of the class for this connector. Must
> be a subclass of org.apache.kafka.connect.connector.Connector. If the
> connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can
> either specify this full name, or use \"FileStreamSink\" or
> \"FileStreamSinkConnector\" to make the configuration a bit shorter",
> "group": "Common",
> "width": "LONG",
> "display_name": "Connector class",
> "dependents": [],
> "order": 2
> },
> "value": {
> "name": "connector.class",
> "value": "org.apache.kafka.connect.file.FileStreamSourceConnector",
> "recommended_values": [],
> "errors": [],
> "visible": true
> }
> },
> {
> "definition": {
> "name": "tasks.max",
> "type": "INT",
> "required": false,
> "default_value": "1",
> "importance": "HIGH",
> "documentation": "Maximum number of tasks to use for this connector.",
> "group": "Common",
> "width": "SHORT",
> "display_name": "Tasks max",
> "dependents": [],
> "order": 3
> },
> "value": {
> "name": "tasks.max",
> "value": "1",
> "recommended_values": [],
> "errors": [],
> "visible": true
> }
> },
> {
> "definition": {
> "name": "key.converter",
> "type": "CLASS",
> "required": false,
> "default_value": null,
> "importance": "LOW",
> "documentation": "Converter class used to convert between Kafka
> Connect format and the serialized form that is written to Kafka. This
> controls the format of the keys in messages written to or read from Kafka,
> and since this is independent of connectors it allows any connector to work
> with any serialization format. Examples of common formats include JSON and
> Avro.",
> "group": "Common",
> "width": "SHORT",
> "display_name": "Key converter class",
> "dependents": [],
> "order": 4
> },
> "value": {
> "name": "key.converter",
> "value": null,
> "recommended_values": [],
> "errors": [],
> "visible": true
> }
> },
>
[jira] [Resolved] (KAFKA-2611) Document MirrorMaker
[ https://issues.apache.org/jira/browse/KAFKA-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2611. - Resolution: Fixed MirrorMaker is actually documented. > Document MirrorMaker > > > Key: KAFKA-2611 > URL: https://issues.apache.org/jira/browse/KAFKA-2611 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Its been part of our platform for a while, will be nice to add some > documentation around how to use it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KAFKA-5448) TimestampConverter's "type" config conflicts with the basic Transformation "type" config
[ https://issues.apache.org/jira/browse/KAFKA-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-5448. - Resolution: Fixed Fix Version/s: 0.11.1.0 Issue resolved by pull request 3342 [https://github.com/apache/kafka/pull/3342] > TimestampConverter's "type" config conflicts with the basic Transformation > "type" config > > > Key: KAFKA-5448 > URL: https://issues.apache.org/jira/browse/KAFKA-5448 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava >Priority: Blocker > Fix For: 0.11.0.0, 0.11.1.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > [KIP-66|https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect] > defined one of the configs for TimestampConverter to be "type". However, all > transformations are configured with the "type" config specifying the class > that implements them. > We need to modify the naming of the configs so these don't conflict. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5274) Review and improve AdminClient Javadoc for the first release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5274: Resolution: Fixed Fix Version/s: 0.11.1.0 Status: Resolved (was: Patch Available) Issue resolved by pull request 3316 [https://github.com/apache/kafka/pull/3316] > Review and improve AdminClient Javadoc for the first release (KIP-117) > -- > > Key: KAFKA-5274 > URL: https://issues.apache.org/jira/browse/KAFKA-5274 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0, 0.11.1.0 > > > Once all the AdminClient pieces are in, we should take a pass at the Javadoc > and improve it wherever possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-2378) Add Connect embedded API
[ https://issues.apache.org/jira/browse/KAFKA-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2378: Summary: Add Connect embedded API (was: Add Copycat embedded API) > Add Connect embedded API > > > Key: KAFKA-2378 > URL: https://issues.apache.org/jira/browse/KAFKA-2378 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava > Labels: needs-kip > > Much of the required Copycat API will exist from previous patches since any > main() method will need to do very similar operations. However, integrating > with any other Java code may require additional API support. > For example, one of the use cases when integrating with any stream processing > application will require knowing which topics will be written to. We will > need to add APIs to expose the topics a registered connector is writing to so > they can be consumed by a stream processing task -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5411) Generate javadoc for AdminClient and show configs in documentation
[ https://issues.apache.org/jira/browse/KAFKA-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5411: Resolution: Fixed Fix Version/s: 0.11.1.0 Status: Resolved (was: Patch Available) Issue resolved by pull request 3271 [https://github.com/apache/kafka/pull/3271] > Generate javadoc for AdminClient and show configs in documentation > -- > > Key: KAFKA-5411 > URL: https://issues.apache.org/jira/browse/KAFKA-5411 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Blocker > Fix For: 0.11.0.0, 0.11.1.0 > > > Also fix the table of contents. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5317) Update KIP-98 to reflect changes during implementation.
[ https://issues.apache.org/jira/browse/KAFKA-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043541#comment-16043541 ] Gwen Shapira commented on KAFKA-5317: - I understand the importance, but is it really a release blocker? > Update KIP-98 to reflect changes during implementation. > --- > > Key: KAFKA-5317 > URL: https://issues.apache.org/jira/browse/KAFKA-5317 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Apurva Mehta >Priority: Blocker > Labels: documentation, exactly-once > Fix For: 0.11.0.0 > > > While implementing the EOS design, there are some minor (or major?) tweaks we > made as we hands-on the code bases. We will compile all these changes in a > single run at the end of the code-ready and this JIRA is for keeping track of > all the changes we made. > 02/27/2017: collapse the two types of transactional log messages into a > single type with key as transactional id, and value as [pid, epoch, > transaction_timeout, transaction_state, [topic partition ] ]. Also using a > single memory map in cache instead of two on the TC. > 03/01/2017: for pid expiration, we decided to use min(transactional id > expiration timeout, topic retention). For topics enabled for compaction only, > we just use the transactional timeout. If the retention setting is larger > than the transactional id expiration timeout, then the pid will be > "logically" expired (i.e. we will remove it from the cached pid mapping and > ignore it when rebuilding the cache from the log) > 03/20/2017: add a new exception type in `o.a.k.common.errors` for invalid > transaction timeout values. > 03/25/2017: extend WriteTxnMarkerRequest to contain multiple markers for > multiple PIDs with a single request. > 04/20/2017: add transactionStartTime to TransactionMetadata > 04/26/2017: added a new retriable error: Errors.CONCURRENT_TRANSACTIONS > 04/01/2017: We also enforce acks=all on the client when idempotence is > enabled. Without this, we cannot again guarantee idemptoence. > 04/10/2017: WE also don't pass the underlying exception to > `RetriableOffsetCommitFailedException` anymore: > https://issues.apache.org/jira/browse/KAFKA-5052 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-5384) KIP-162: Enable topic deletion by default
Gwen Shapira created KAFKA-5384: --- Summary: KIP-162: Enable topic deletion by default Key: KAFKA-5384 URL: https://issues.apache.org/jira/browse/KAFKA-5384 Project: Kafka Issue Type: Improvement Components: core Reporter: Gwen Shapira Fix For: 0.12.0.0 Change default of delete.topic.enable to true Remove delete.topic.enable config from config/server.properties. See KIP for details: https://cwiki.apache.org/confluence/display/KAFKA/KIP-162+-+Enable+topic+deletion+by+default -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4755) SimpleBenchmark test fails for streams
[ https://issues.apache.org/jira/browse/KAFKA-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979000#comment-15979000 ] Gwen Shapira commented on KAFKA-4755: - removed 0.10.2.1 from fixVersion since it isn't making it into this RC. > SimpleBenchmark test fails for streams > -- > > Key: KAFKA-4755 > URL: https://issues.apache.org/jira/browse/KAFKA-4755 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska >Assignee: Eno Thereska >Priority: Blocker > Fix For: 0.11.0.0 > > > This occurred Feb 10th 2017: > kafkatest.benchmarks.streams.streams_simple_benchmark_test.StreamsSimpleBenchmarkTest.test_simple_benchmark.test=consume.scale=1 > status: FAIL > run time: 7 minutes 36.712 seconds > Streams Test process on ubuntu@worker2 took too long to exit > Traceback (most recent call last): > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", > line 123, in run > data = self.run_test() > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", > line 176, in run_test > return self.test_context.function(self.test) > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py", > line 321, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/benchmarks/streams/streams_simple_benchmark_test.py", > line 86, in test_simple_benchmark > self.driver[num].wait() > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py", > line 102, in wait > self.wait_node(node, timeout_sec) > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py", > line 106, in wait_node > wait_until(lambda: not node.account.alive(pid), timeout_sec=timeout_sec, > err_msg="Streams Test process on " + str(node.account) + " took too long to > exit") > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py", > line 36, in wait_until > raise TimeoutError(err_msg) > TimeoutError: Streams Test process on ubuntu@worker2 took too long to exit -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4755) SimpleBenchmark test fails for streams
[ https://issues.apache.org/jira/browse/KAFKA-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-4755: Fix Version/s: (was: 0.10.2.1) > SimpleBenchmark test fails for streams > -- > > Key: KAFKA-4755 > URL: https://issues.apache.org/jira/browse/KAFKA-4755 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska >Assignee: Eno Thereska >Priority: Blocker > Fix For: 0.11.0.0 > > > This occurred Feb 10th 2017: > kafkatest.benchmarks.streams.streams_simple_benchmark_test.StreamsSimpleBenchmarkTest.test_simple_benchmark.test=consume.scale=1 > status: FAIL > run time: 7 minutes 36.712 seconds > Streams Test process on ubuntu@worker2 took too long to exit > Traceback (most recent call last): > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", > line 123, in run > data = self.run_test() > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", > line 176, in run_test > return self.test_context.function(self.test) > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py", > line 321, in wrapper > return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/benchmarks/streams/streams_simple_benchmark_test.py", > line 86, in test_simple_benchmark > self.driver[num].wait() > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py", > line 102, in wait > self.wait_node(node, timeout_sec) > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py", > line 106, in wait_node > wait_until(lambda: not node.account.alive(pid), timeout_sec=timeout_sec, > err_msg="Streams Test process on " + str(node.account) + " took too long to > exit") > File > "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py", > line 36, in wait_until > raise TimeoutError(err_msg) > TimeoutError: Streams Test process on ubuntu@worker2 took too long to exit -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4667) Connect should create internal topics
[ https://issues.apache.org/jira/browse/KAFKA-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977460#comment-15977460 ] Gwen Shapira commented on KAFKA-4667: - Also, note that validating configuration of existing internal topics is important. > Connect should create internal topics > - > > Key: KAFKA-4667 > URL: https://issues.apache.org/jira/browse/KAFKA-4667 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Emanuele Cesena >Priority: Critical > Fix For: 0.11.0.0 > > > I'm reporting this as an issue but in fact it requires more investigation > (which unfortunately I'm not able to perform at this time). > Repro steps: > - configure Kafka for consistency, for example: > default.replication.factor=3 > min.insync.replicas=2 > unclean.leader.election.enable=false > - run Connect for the first time, which should create its internal topics > I believe these topics are created with the broker's default, in particular: > min.insync.replicas=2 > unclean.leader.election.enable=false > but connect doesn't produce with acks=all, which in turn may cause the > cluster to go in a bad state (see, e.g., > https://issues.apache.org/jira/browse/KAFKA-4666). > Solution would be to force availability mode, i.e. force: > unclean.leader.election.enable=true > when creating the connect topics, or viceversa detect availability vs > consistency mode and turn acks=all if needed. > I assume the same happens with other kafka-based services such as streams. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (KAFKA-4380) Remove CleanShutdownFile as 0.8.2 has been released
[ https://issues.apache.org/jira/browse/KAFKA-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira reassigned KAFKA-4380: --- Assignee: holdenk > Remove CleanShutdownFile as 0.8.2 has been released > --- > > Key: KAFKA-4380 > URL: https://issues.apache.org/jira/browse/KAFKA-4380 > Project: Kafka > Issue Type: Improvement > Components: log >Reporter: holdenk >Assignee: holdenk >Priority: Trivial > > There is a TODO in the code to remove CleanShutdownFile after 0.8.2 is > shipped, which has happened. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5087) Kafka Connect's configuration topics should always be compacted
[ https://issues.apache.org/jira/browse/KAFKA-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974949#comment-15974949 ] Gwen Shapira commented on KAFKA-5087: - The AdminClient pull request, if someone wants an early start :) https://github.com/apache/kafka/pull/2472 > Kafka Connect's configuration topics should always be compacted > --- > > Key: KAFKA-5087 > URL: https://issues.apache.org/jira/browse/KAFKA-5087 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Priority: Critical > > New users frequently lose their connector configuration because they did not > manually set the topic deletion policy to "compact". Not a good first > experience with our system. > 1. If the topics do not exist, Kafka Connect should create them with the > correct configuration. > 2. If the topics do exist, Kafka Connect should check their deletion policy > and refuse to start if it isn't "compact" > I'd love to do it (or have someone else do it) the moment the AdminClient is > merged and to have it in 0.11.0.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KAFKA-5075) Defer exception to the next pollOnce() if consumer's fetch position has already increased
[ https://issues.apache.org/jira/browse/KAFKA-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-5075. - Resolution: Fixed Resolving so it will show up in 0.10.2.1 release notes > Defer exception to the next pollOnce() if consumer's fetch position has > already increased > - > > Key: KAFKA-5075 > URL: https://issues.apache.org/jira/browse/KAFKA-5075 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.10.2.0 >Reporter: Jiangjie Qin >Assignee: Dong Lin > Fix For: 0.11.0.0, 0.10.2.1 > > > In Fetcher.fetchRecords() we iterate over the partition data to collect the > ConsumerRecords, after we collect some consumer records from a partition, we > advance the position of that partition then move on to the next partition. If > the next partition throws exceptions (e.g. OffsetOutOfRangeException), the > messages that have already been read out of the buffer will not be delivered > to the users. Since the positions of the previous partitions have been be > updated, those messages will not be consumed again either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5075) Defer exception to the next pollOnce() if consumer's fetch position has already increased
[ https://issues.apache.org/jira/browse/KAFKA-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5075: Fix Version/s: 0.10.2.1 > Defer exception to the next pollOnce() if consumer's fetch position has > already increased > - > > Key: KAFKA-5075 > URL: https://issues.apache.org/jira/browse/KAFKA-5075 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.10.2.0 >Reporter: Jiangjie Qin >Assignee: Dong Lin > Fix For: 0.11.0.0, 0.10.2.1 > > > In Fetcher.fetchRecords() we iterate over the partition data to collect the > ConsumerRecords, after we collect some consumer records from a partition, we > advance the position of that partition then move on to the next partition. If > the next partition throws exceptions (e.g. OffsetOutOfRangeException), the > messages that have already been read out of the buffer will not be delivered > to the users. Since the positions of the previous partitions have been be > updated, those messages will not be consumed again either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5087) Kafka Connect's configuration topics should always be compacted
[ https://issues.apache.org/jira/browse/KAFKA-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5087: Priority: Critical (was: Major) > Kafka Connect's configuration topics should always be compacted > --- > > Key: KAFKA-5087 > URL: https://issues.apache.org/jira/browse/KAFKA-5087 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Priority: Critical > > New users frequently lose their connector configuration because they did not > manually set the topic deletion policy to "compact". Not a good first > experience with our system. > 1. If the topics do not exist, Kafka Connect should create them with the > correct configuration. > 2. If the topics do exist, Kafka Connect should check their deletion policy > and refuse to start if it isn't "compact" > I'd love to do it (or have someone else do it) the moment the AdminClient is > merged and to have it in 0.11.0.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5087) Kafka Connect's configuration topics should always be compacted
[ https://issues.apache.org/jira/browse/KAFKA-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5087: Component/s: KafkaConnect > Kafka Connect's configuration topics should always be compacted > --- > > Key: KAFKA-5087 > URL: https://issues.apache.org/jira/browse/KAFKA-5087 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira > > New users frequently lose their connector configuration because they did not > manually set the topic deletion policy to "compact". Not a good first > experience with our system. > 1. If the topics do not exist, Kafka Connect should create them with the > correct configuration. > 2. If the topics do exist, Kafka Connect should check their deletion policy > and refuse to start if it isn't "compact" > I'd love to do it (or have someone else do it) the moment the AdminClient is > merged and to have it in 0.11.0.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-5087) Kafka Connect's configuration topics should always be compacted
Gwen Shapira created KAFKA-5087: --- Summary: Kafka Connect's configuration topics should always be compacted Key: KAFKA-5087 URL: https://issues.apache.org/jira/browse/KAFKA-5087 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira New users frequently lose their connector configuration because they did not manually set the topic deletion policy to "compact". Not a good first experience with our system. 1. If the topics do not exist, Kafka Connect should create them with the correct configuration. 2. If the topics do exist, Kafka Connect should check their deletion policy and refuse to start if it isn't "compact" I'd love to do it (or have someone else do it) the moment the AdminClient is merged and to have it in 0.11.0.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5075) The position in the consumers may be advanced incorrectly when some exception is thrown from consumer.poll()
[ https://issues.apache.org/jira/browse/KAFKA-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969764#comment-15969764 ] Gwen Shapira commented on KAFKA-5075: - Cool. I'll check back on Monday. > The position in the consumers may be advanced incorrectly when some exception > is thrown from consumer.poll() > > > Key: KAFKA-5075 > URL: https://issues.apache.org/jira/browse/KAFKA-5075 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.10.2.0 >Reporter: Jiangjie Qin >Assignee: Dong Lin > Fix For: 0.11.0.0 > > > In Fetcher.fetchRecords() we iterate over the partition data to collect the > ConsumerRecords, after we collect some consumer records from a partition, we > advance the position of that partition then move on to the next partition. If > the next partition throws exceptions (e.g. OffsetOutOfRangeException), the > messages that have already been read out of the buffer will not be delivered > to the users. Since the positions of the previous partitions have been be > updated, those messages will not be consumed again either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5075) The position in the consumers may be advanced incorrectly when some exception is thrown from consumer.poll()
[ https://issues.apache.org/jira/browse/KAFKA-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969719#comment-15969719 ] Gwen Shapira commented on KAFKA-5075: - I don't mind doing another RC, since I agree data loss is serious. However, I don't want to delay a bugfix release that already has good bug fixes in it. When do you think you'll have the patch merged? > The position in the consumers may be advanced incorrectly when some exception > is thrown from consumer.poll() > > > Key: KAFKA-5075 > URL: https://issues.apache.org/jira/browse/KAFKA-5075 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.10.2.0 >Reporter: Jiangjie Qin >Assignee: Dong Lin > Fix For: 0.11.0.0 > > > In Fetcher.fetchRecords() we iterate over the partition data to collect the > ConsumerRecords, after we collect some consumer records from a partition, we > advance the position of that partition then move on to the next partition. If > the next partition throws exceptions (e.g. OffsetOutOfRangeException), the > messages that have already been read out of the buffer will not be delivered > to the users. Since the positions of the previous partitions have been be > updated, those messages will not be consumed again either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5075) The position in the consumers may be advanced incorrectly when some exception is thrown from consumer.poll()
[ https://issues.apache.org/jira/browse/KAFKA-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969681#comment-15969681 ] Gwen Shapira commented on KAFKA-5075: - I changed the fixVersion since this isn't marked as a blocker and we already have an RC for 0.10.2.1. If you think this is a blocker, please mark it as blocker and respond to the VOTE thread for 0.10.2.1 so we'll know we need to stop the vote. Is this issue new to 0.10.2 or did it exist in earlier releases too? > The position in the consumers may be advanced incorrectly when some exception > is thrown from consumer.poll() > > > Key: KAFKA-5075 > URL: https://issues.apache.org/jira/browse/KAFKA-5075 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.10.2.0 >Reporter: Jiangjie Qin > Fix For: 0.11.0.0 > > > In Fetcher.fetchRecords() we iterate over the partition data to collect the > ConsumerRecords, after we collect some consumer records from a partition, we > advance the position of that partition then move on to the next partition. If > the next partition throws exceptions (e.g. OffsetOutOfRangeException), the > messages that have already been read out of the buffer will not be delivered > to the users. Since the positions of the previous partitions have been be > updated, those messages will not be consumed again either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-5075) The position in the consumers may be advanced incorrectly when some exception is thrown from consumer.poll()
[ https://issues.apache.org/jira/browse/KAFKA-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5075: Fix Version/s: (was: 0.10.2.1) 0.11.0.0 > The position in the consumers may be advanced incorrectly when some exception > is thrown from consumer.poll() > > > Key: KAFKA-5075 > URL: https://issues.apache.org/jira/browse/KAFKA-5075 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 0.10.2.0 >Reporter: Jiangjie Qin > Fix For: 0.11.0.0 > > > In Fetcher.fetchRecords() we iterate over the partition data to collect the > ConsumerRecords, after we collect some consumer records from a partition, we > advance the position of that partition then move on to the next partition. If > the next partition throws exceptions (e.g. OffsetOutOfRangeException), the > messages that have already been read out of the buffer will not be delivered > to the users. Since the positions of the previous partitions have been be > updated, those messages will not be consumed again either. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5057) "Big Message Log"
[ https://issues.apache.org/jira/browse/KAFKA-5057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967819#comment-15967819 ] Gwen Shapira commented on KAFKA-5057: - I thought of a new configuration (which made me realize this will require a KIP). Basically, sometimes users bump up max.request.size, but only expect very few large messages (since large messages have impact on garbage collection, throughput, etc). Such log will let them track the number of large messages, their size and their source so they can see if their expectation is correct and adjust course if it isn't. So I will set max.request.size to 10MB, but the logging threshold to 1MB, because I expect very few messages between 1MB and 10MB. Does that make sense? > "Big Message Log" > - > > Key: KAFKA-5057 > URL: https://issues.apache.org/jira/browse/KAFKA-5057 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Really large requests can cause significant GC pauses which can cause quite a > few other symptoms on a broker. Will be nice to be able to catch them. > Lets add the option to log details (client id, topic, partition) for every > produce request that is larger than a configurable threshold. > /cc [~apurva] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-5057) "Big Message Log"
Gwen Shapira created KAFKA-5057: --- Summary: "Big Message Log" Key: KAFKA-5057 URL: https://issues.apache.org/jira/browse/KAFKA-5057 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Really large requests can cause significant GC pauses which can cause quite a few other symptoms on a broker. Will be nice to be able to catch them. Lets add the option to log details (client id, topic, partition) for every produce request that is larger than a configurable threshold. /cc [~apurva] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4876) WindowStore.fetch(key, 0, Long.MaxValue) is very slow
[
https://issues.apache.org/jira/browse/KAFKA-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4876:
Fix Version/s: (was: 0.10.2.1)
> WindowStore.fetch(key, 0, Long.MaxValue) is very slow
> -
>
> Key: KAFKA-4876
> URL: https://issues.apache.org/jira/browse/KAFKA-4876
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.1.0, 0.10.2.0
>Reporter: Xavier Léauté
>
> Relates to KAFKA-4851
> Querying a window store over the entire time range (i.e. {{\[0,
> Long.MAX_VALUE\]}} is extremely slow.
> Here's a simple benchmark that shows it takes about 400ms to query just a
> single key with a timestamp as of today's date.
> https://github.com/xvrl/kafka/commit/045b94d4c70d730d10ed492efeb0fd85a70757ae#diff-46a87bcb14bccc04c8ad716a043fc78eR150
> {code}
> BenchmarkMode CntScoreError
> Units
> RocksDBWindowStoreTest.benchmarkStoreFetch avgt 20 401.635 ± 46.130
> ms/op
> {code}
> Using timestamps far out in the future (e.g. 0x7a00L) the fetch
> operation essentially never completes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-4943) SCRAM secret's should be better protected with Zookeeper ACLs
[
https://issues.apache.org/jira/browse/KAFKA-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4943:
Resolution: Fixed
Fix Version/s: 0.11.0.0
Status: Resolved (was: Patch Available)
Issue resolved by pull request 2733
[https://github.com/apache/kafka/pull/2733]
> SCRAM secret's should be better protected with Zookeeper ACLs
> -
>
> Key: KAFKA-4943
> URL: https://issues.apache.org/jira/browse/KAFKA-4943
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.10.2.0
>Reporter: Johan Ström
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0, 0.10.2.1
>
>
> With the new SCRAM authenticator the secrets are stored in Zookeeper:
> {code}
> get /kafka/config/users/alice
> {"version":1,"config":{"SCRAM-SHA-512":"salt=ODhnZjNkdWZibTV1cG1zdnV6bmh6djF3Mg==,stored_key=BAbHWHuGEb4m5+U+p0M9oFQmOPhU6M7q5jtZY8deDDoZCvxaqVNLz41yPzdgcp1WpiEBmfwYOuFlo9hMFKM7mA==,server_key=JW3KhpMeyUgh0OAC0kejuFUvUSlXBv/Z68tlfOWcMw5f5jrBwyBnjNQ9VZsSYz1AcI9IYaQ5S6H3yN39SieNiA==,iterations=4096"}}
> {code}
> These are stored without any ACL, and zookeeper-security-migration.sh does
> not seem to change that either:
> {code}
> getAcl /kafka/config/users/alice
> 'world,'anyone
> : cdrwa
> getAcl /kafka/config/users
> 'world,'anyone
> : cdrwa
> getAcl /kafka
> 'world,'anyone
> : r
> 'sasl,'bob
> : cdrwa
> getAcl /kafka/config/changes
> 'world,'anyone
> : r
> 'sasl,'bob
> : cdrwa
> {code}
> The above output is after running security migrator, for some reason
> /kafka/config/users is ignored, but others are fixed..
> Even if these where to be stored with secure ZkUtils#DefaultAcls, they would
> be world readable.
> From my (limited) point of view, they should be readable by Kafka only.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-4814) ZookeeperLeaderElector not respecting zookeeper.set.acl
[
https://issues.apache.org/jira/browse/KAFKA-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4814:
Fix Version/s: (was: 0.10.2.1)
> ZookeeperLeaderElector not respecting zookeeper.set.acl
> ---
>
> Key: KAFKA-4814
> URL: https://issues.apache.org/jira/browse/KAFKA-4814
> Project: Kafka
> Issue Type: Bug
> Components: security
>Affects Versions: 0.10.1.1
>Reporter: Stevo Slavic
>Assignee: Rajini Sivaram
> Labels: newbie
> Fix For: 0.11.0.0
>
>
> By [migration
> guide|https://kafka.apache.org/documentation/#zk_authz_migration] for
> enabling ZooKeeper security on an existing Apache Kafka cluster, and [broker
> configuration
> documentation|https://kafka.apache.org/documentation/#brokerconfigs] for
> {{zookeeper.set.acl}} configuration property, when this property is set to
> false Kafka brokers should not be setting any ACLs on ZooKeeper nodes, even
> when JAAS config file is provisioned to broker.
> Problem is that there is broker side logic, like one in
> {{ZookeeperLeaderElector}} making use of {{JaasUtils#isZkSecurityEnabled}},
> which does not respect this configuration property, resulting in ACLs being
> set even when there's just JAAS config file provisioned to Kafka broker while
> {{zookeeper.set.acl}} is set to {{false}}.
> Notice that {{JaasUtils}} is in {{org.apache.kafka.common.security}} package
> of {{kafka-clients}} module, while {{zookeeper.set.acl}} is broker side only
> configuration property.
> To make it possible without downtime to enable ZooKeeper authentication on
> existing cluster, it should be possible to have all Kafka brokers in cluster
> first authenticate to ZooKeeper cluster, without ACLs being set. Only once
> all ZooKeeper clients (Kafka brokers and others) are authenticating to
> ZooKeeper cluster then ACLs can be started being set.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4814) ZookeeperLeaderElector not respecting zookeeper.set.acl
[
https://issues.apache.org/jira/browse/KAFKA-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961176#comment-15961176
]
Gwen Shapira commented on KAFKA-4814:
-
Hey guys,
I think this is going to be late for 0.10.2.1, so I'm moving the fixVersion.
Sorry about that, but there are so many important bug fixes that I don't want
to delay the release.
I'll revisit if we do another RC, so keep on the good work :)
> ZookeeperLeaderElector not respecting zookeeper.set.acl
> ---
>
> Key: KAFKA-4814
> URL: https://issues.apache.org/jira/browse/KAFKA-4814
> Project: Kafka
> Issue Type: Bug
> Components: security
>Affects Versions: 0.10.1.1
>Reporter: Stevo Slavic
>Assignee: Rajini Sivaram
> Labels: newbie
> Fix For: 0.11.0.0
>
>
> By [migration
> guide|https://kafka.apache.org/documentation/#zk_authz_migration] for
> enabling ZooKeeper security on an existing Apache Kafka cluster, and [broker
> configuration
> documentation|https://kafka.apache.org/documentation/#brokerconfigs] for
> {{zookeeper.set.acl}} configuration property, when this property is set to
> false Kafka brokers should not be setting any ACLs on ZooKeeper nodes, even
> when JAAS config file is provisioned to broker.
> Problem is that there is broker side logic, like one in
> {{ZookeeperLeaderElector}} making use of {{JaasUtils#isZkSecurityEnabled}},
> which does not respect this configuration property, resulting in ACLs being
> set even when there's just JAAS config file provisioned to Kafka broker while
> {{zookeeper.set.acl}} is set to {{false}}.
> Notice that {{JaasUtils}} is in {{org.apache.kafka.common.security}} package
> of {{kafka-clients}} module, while {{zookeeper.set.acl}} is broker side only
> configuration property.
> To make it possible without downtime to enable ZooKeeper authentication on
> existing cluster, it should be possible to have all Kafka brokers in cluster
> first authenticate to ZooKeeper cluster, without ACLs being set. Only once
> all ZooKeeper clients (Kafka brokers and others) are authenticating to
> ZooKeeper cluster then ACLs can be started being set.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-4901) Make ProduceRequest thread-safe
[ https://issues.apache.org/jira/browse/KAFKA-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-4901: Resolution: Fixed Status: Resolved (was: Patch Available) > Make ProduceRequest thread-safe > --- > > Key: KAFKA-4901 > URL: https://issues.apache.org/jira/browse/KAFKA-4901 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0, 0.10.2.1 > > > If request logging is enabled, ProduceRequest can be accessed > and mutated concurrently from a network thread (which calls > toString) and the request handler thread (which calls > clearPartitionRecords()). > That can lead to a ConcurrentModificationException when iterating > the partitionRecords map. > The underlying thread-safety issue has existed since the server > started using the Java implementation of ProduceRequest in 0.10.0. > However, we were incorrectly not clearing the underlying struct until > 0.10.2, so toString itself was thread-safe until that change. In 0.10.2, > toString is no longer thread-safe and we could potentially see a > NullPointerException given the right set of interleavings between > toString and clearPartitionRecords although we haven't seen that > happen yet. > In trunk, we changed the requests to have a toStruct method > instead of creating a struct in the constructor and toString was > no longer printing the contents of the Struct. This accidentally > fixed the race condition, but it meant that request logging was less > useful. > A couple of days ago, AbstractRequest.toString was changed to > print the contents of the request by calling toStruct().toString() > and reintroduced the race condition. The impact is more visible > because we iterate over a HashMap, which proactively > checks for concurrent modification (unlike arrays). > We will need a separate PR for 0.10.2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KAFKA-5042) InFlightRequests#isEmpty() always returns false
[
https://issues.apache.org/jira/browse/KAFKA-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira resolved KAFKA-5042.
-
Resolution: Fixed
Issue resolved by pull request 2823
[https://github.com/apache/kafka/pull/2823]
> InFlightRequests#isEmpty() always returns false
> ---
>
> Key: KAFKA-5042
> URL: https://issues.apache.org/jira/browse/KAFKA-5042
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Tommy Becker
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> While perusing the NetworkClient code I came across the following:
> InFlightRequests#isEmpty() always returns false.
> {code}
> public boolean isEmpty() {
> for (Deque deque :
> this.requests.values()) {
> if (!deque.isEmpty())
> return false;
> }
> return false;
> }
> {code}
> This method looks like a recent addtion.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-5034) Connect workers don't discover new Connector Plugins without Restart
[ https://issues.apache.org/jira/browse/KAFKA-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-5034: Component/s: KafkaConnect > Connect workers don't discover new Connector Plugins without Restart > > > Key: KAFKA-5034 > URL: https://issues.apache.org/jira/browse/KAFKA-5034 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira > > If I want to add a new Connector Plugin to a running distributed Connect > cluster, I need to copy the JAR to the classpath and then restart all the > workers so they will pick up the new plugin before I can create a connector. > This is both un-intuitive (most modern up can pick up changes dynamically) > and can get difficult when a connect cluster is shared between teams. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-5034) Connect workers don't discover new Connector Plugins without Restart
Gwen Shapira created KAFKA-5034: --- Summary: Connect workers don't discover new Connector Plugins without Restart Key: KAFKA-5034 URL: https://issues.apache.org/jira/browse/KAFKA-5034 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira If I want to add a new Connector Plugin to a running distributed Connect cluster, I need to copy the JAR to the classpath and then restart all the workers so they will pick up the new plugin before I can create a connector. This is both un-intuitive (most modern up can pick up changes dynamically) and can get difficult when a connect cluster is shared between teams. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4929) Transformation Key/Value type references should be to class name(), not canonicalName()
[
https://issues.apache.org/jira/browse/KAFKA-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959490#comment-15959490
]
Gwen Shapira commented on KAFKA-4929:
-
I'm cherrypicking this to 0.10.2 branch to be included in 0.10.2.1 bugfix
release (the docs and errors are misleading enough to constitute a bug in my
opinion).
Objections [~ewencp]?
> Transformation Key/Value type references should be to class name(), not
> canonicalName()
> ---
>
> Key: KAFKA-4929
> URL: https://issues.apache.org/jira/browse/KAFKA-4929
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: bruce szalwinski
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> The docs suggest that referencing the Key/Value transformations is done as
> follows:
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":
> "org.apache.kafka.connect.transforms.ReplaceField.Value"
> {code}
> But that results in a validation failure saying that the class cannot be
> found.
> {code}
> "value": {
> "errors": [
> "Invalid value
> org.apache.kafka.connect.transforms.ReplaceField.Value for configuration
> transforms.replaceFieldValue.type: Class
> org.apache.kafka.connect.transforms.ReplaceField.Value could not be found.",
> "Invalid value null for configuration
> transforms.replaceFieldValue.type: Not a Transformation"
> ],
> "name": "transforms.replaceFieldValue.type",
> "recommended_values": [
> "org.apache.kafka.connect.transforms.ExtractField.Key",
> "org.apache.kafka.connect.transforms.ExtractField.Value",
> "org.apache.kafka.connect.transforms.HoistField.Key",
> "org.apache.kafka.connect.transforms.HoistField.Value",
> "org.apache.kafka.connect.transforms.InsertField.Key",
> "org.apache.kafka.connect.transforms.InsertField.Value",
> "org.apache.kafka.connect.transforms.MaskField.Key",
> "org.apache.kafka.connect.transforms.MaskField.Value",
> "org.apache.kafka.connect.transforms.RegexRouter",
> "org.apache.kafka.connect.transforms.ReplaceField.Key",
> "org.apache.kafka.connect.transforms.ReplaceField.Value",
>
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
>
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
> "org.apache.kafka.connect.transforms.TimestampRouter",
> "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> {code}
> Since the Key / Value transformations are defined as static nested classes,
> the proper notation is
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":
> "org.apache.kafka.connect.transforms.ReplaceField$Value"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KAFKA-4878) Kafka Connect does not log connector configuration errors
[
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira resolved KAFKA-4878.
-
Resolution: Fixed
Fix Version/s: 0.11.0.0
Issue resolved by pull request 2722
[https://github.com/apache/kafka/pull/2722]
> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Armin Braun
>Priority: Blocker
> Fix For: 0.11.0.0, 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
> configuration is invalid (use the endpoint `/{connectorType}/config/validate`
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4916) Add streams tests with brokers failing
[ https://issues.apache.org/jira/browse/KAFKA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953722#comment-15953722 ] Gwen Shapira commented on KAFKA-4916: - This doesn't block the release, right? It doesn't fix any known issue and tests can be added to the branch at any point in time? > Add streams tests with brokers failing > -- > > Key: KAFKA-4916 > URL: https://issues.apache.org/jira/browse/KAFKA-4916 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska >Assignee: Eno Thereska >Priority: Blocker > Fix For: 0.10.2.1 > > > We need to add either integration or system tests with streams and have Kafka > brokers fail and come back up. A combination of transient and permanent > broker failures. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-5000) Framework should log some progress information regularly to assist in troubleshooting
Gwen Shapira created KAFKA-5000:
---
Summary: Framework should log some progress information regularly
to assist in troubleshooting
Key: KAFKA-5000
URL: https://issues.apache.org/jira/browse/KAFKA-5000
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Gwen Shapira
We get many questions of the type:
"I started a connector, it doesn't seem to make any progress, I don't know what
to do"
I think that periodic "progress reports" on the worker logs may help.
We have the offset commit message: "INFO
WorkerSinkTask{id=cassandra-sink-connector-0} Committing offsets
(org.apache.kafka.connect.runtime.WorkerSinkTask:244)"
But I think we'd also want to know: topic, partition, offsets, how many rows
were read from source/kafka and how many were successfully written.
This will help determine if there is any progress being made and whether some
partitions are "stuck" for some reason.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4930) Connect Rest API allows creating connectors with an empty name
[
https://issues.apache.org/jira/browse/KAFKA-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943621#comment-15943621
]
Gwen Shapira commented on KAFKA-4930:
-
[~sliebau] I like your validName approach. Can you create a PR? It is much
easier for me to review that way :)
> Connect Rest API allows creating connectors with an empty name
> --
>
> Key: KAFKA-4930
> URL: https://issues.apache.org/jira/browse/KAFKA-4930
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Sönke Liebau
>Priority: Minor
>
> The Connect Rest API allows to deploy connectors with an empty name field,
> which then cannot be removed through the api.
> Sending the following request:
> {code}
> {
> "name": "",
> "config": {
> "connector.class":
> "org.apache.kafka.connect.tools.MockSourceConnector",
> "tasks.max": "1",
> "topics": "test-topic"
>
> }
> }
> {code}
> Results in a connector being deployed which can be seen in the list of
> connectors:
> {code}
> [
> "",
> "testconnector"
> ]{code}
> But cannot be removed via a DELETE call, as the api thinks we are trying to
> delete the /connectors endpoint and declines the request.
> I don't think there is a valid case for the connector name to be empty so
> perhaps we should add a check for this. I am happy to work on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-4929) Transformation Key/Value type references should be to class name(), not canonicalName()
[
https://issues.apache.org/jira/browse/KAFKA-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4929:
Resolution: Fixed
Fix Version/s: 0.11.0.0
Status: Resolved (was: Patch Available)
Issue resolved by pull request 2720
[https://github.com/apache/kafka/pull/2720]
> Transformation Key/Value type references should be to class name(), not
> canonicalName()
> ---
>
> Key: KAFKA-4929
> URL: https://issues.apache.org/jira/browse/KAFKA-4929
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: bruce szalwinski
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> The docs suggest that referencing the Key/Value transformations is done as
> follows:
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":
> "org.apache.kafka.connect.transforms.ReplaceField.Value"
> {code}
> But that results in a validation failure saying that the class cannot be
> found.
> {code}
> "value": {
> "errors": [
> "Invalid value
> org.apache.kafka.connect.transforms.ReplaceField.Value for configuration
> transforms.replaceFieldValue.type: Class
> org.apache.kafka.connect.transforms.ReplaceField.Value could not be found.",
> "Invalid value null for configuration
> transforms.replaceFieldValue.type: Not a Transformation"
> ],
> "name": "transforms.replaceFieldValue.type",
> "recommended_values": [
> "org.apache.kafka.connect.transforms.ExtractField.Key",
> "org.apache.kafka.connect.transforms.ExtractField.Value",
> "org.apache.kafka.connect.transforms.HoistField.Key",
> "org.apache.kafka.connect.transforms.HoistField.Value",
> "org.apache.kafka.connect.transforms.InsertField.Key",
> "org.apache.kafka.connect.transforms.InsertField.Value",
> "org.apache.kafka.connect.transforms.MaskField.Key",
> "org.apache.kafka.connect.transforms.MaskField.Value",
> "org.apache.kafka.connect.transforms.RegexRouter",
> "org.apache.kafka.connect.transforms.ReplaceField.Key",
> "org.apache.kafka.connect.transforms.ReplaceField.Value",
>
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
>
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
> "org.apache.kafka.connect.transforms.TimestampRouter",
> "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> {code}
> Since the Key / Value transformations are defined as static nested classes,
> the proper notation is
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":
> "org.apache.kafka.connect.transforms.ReplaceField$Value"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KAFKA-4918) Continuous fetch requests for offset storage topic in kafka connect
[ https://issues.apache.org/jira/browse/KAFKA-4918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-4918. - Resolution: Not A Problem > Continuous fetch requests for offset storage topic in kafka connect > --- > > Key: KAFKA-4918 > URL: https://issues.apache.org/jira/browse/KAFKA-4918 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.1.0, 0.10.1.1, 0.10.2.0 > Environment: unix, osx >Reporter: Liju > Labels: enhancement, performance > > The kafka consumer in the KafkaOffsetBackingStore polls continuously with > timeout hardcoded as 0 ms , this leads to high fetch request load to kafka > server , and specifically for the sink connectors ( eg. kafka-connect-hdfs) > which doesn't uses the offset storage topic for offset tracking , this > becomes redundant and it continuously sends fetch request as there is no data > in this topic -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (KAFKA-4918) Continuous fetch requests for offset storage topic in kafka connect
[ https://issues.apache.org/jira/browse/KAFKA-4918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira reassigned KAFKA-4918: --- Assignee: Gwen Shapira > Continuous fetch requests for offset storage topic in kafka connect > --- > > Key: KAFKA-4918 > URL: https://issues.apache.org/jira/browse/KAFKA-4918 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 0.10.1.0, 0.10.1.1, 0.10.2.0 > Environment: unix, osx >Reporter: Liju >Assignee: Gwen Shapira > Labels: enhancement, performance > > The kafka consumer in the KafkaOffsetBackingStore polls continuously with > timeout hardcoded as 0 ms , this leads to high fetch request load to kafka > server , and specifically for the sink connectors ( eg. kafka-connect-hdfs) > which doesn't uses the offset storage topic for offset tracking , this > becomes redundant and it continuously sends fetch request as there is no data > in this topic -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4930) Connect Rest API allows creating connectors with an empty name
[
https://issues.apache.org/jira/browse/KAFKA-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936374#comment-15936374
]
Gwen Shapira commented on KAFKA-4930:
-
Good catch! If you fix this, we'll gladly take the PR. Let me know if you need
help or pointers.
> Connect Rest API allows creating connectors with an empty name
> --
>
> Key: KAFKA-4930
> URL: https://issues.apache.org/jira/browse/KAFKA-4930
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Sönke Liebau
>Priority: Minor
>
> The Connect Rest API allows to deploy connectors with an empty name field,
> which then cannot be removed through the api.
> Sending the following request:
> {code}
> {
> "name": "",
> "config": {
> "connector.class":
> "org.apache.kafka.connect.tools.MockSourceConnector",
> "tasks.max": "1",
> "topics": "test-topic"
>
> }
> }
> {code}
> Results in a connector being deployed which can be seen in the list of
> connectors:
> {code}
> [
> "",
> "testconnector"
> ]{code}
> But cannot be removed via a DELETE call, as the api thinks we are trying to
> delete the /connectors endpoint and declines the request.
> I don't think there is a valid case for the connector name to be empty so
> perhaps we should add a check for this. I am happy to work on this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KAFKA-4917) Our built-in file connector can't work with our built-in SMT
[ https://issues.apache.org/jira/browse/KAFKA-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-4917. - Resolution: Won't Fix Hoist is fine :) > Our built-in file connector can't work with our built-in SMT > > > Key: KAFKA-4917 > URL: https://issues.apache.org/jira/browse/KAFKA-4917 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira >Assignee: Manasvi Gupta > Labels: newbie > > Our built-in file connector always returns STRING schema. > All our transformations expect either STRUCT (if connectors return schema) or > a MAP (schemaless). > I understand why (how do you add a field to a STRING?), but it also means > that you can't have an example for SMT that works with Apache Kafka out of > the box. Which makes documentation kind of painful. > Either we modify the file connector or we modify the SMTs to deal with STRING > better, but something gotta change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4878) Kafka Connect does not log connector configuration errors
[
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934854#comment-15934854
]
Gwen Shapira commented on KAFKA-4878:
-
[~original-brownbear] - Thank you. Let me know if you need help / pointers.
> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Armin Braun
>Priority: Blocker
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
> configuration is invalid (use the endpoint `/{connectorType}/config/validate`
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4917) Our built-in file connector can't work with our built-in SMT
[ https://issues.apache.org/jira/browse/KAFKA-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930957#comment-15930957 ] Gwen Shapira commented on KAFKA-4917: - I guess I could use HOIST transformation to create a struct :) Maybe I'll just document this pattern and call it a day. > Our built-in file connector can't work with our built-in SMT > > > Key: KAFKA-4917 > URL: https://issues.apache.org/jira/browse/KAFKA-4917 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira > Labels: newbie > > Our built-in file connector always returns STRING schema. > All our transformations expect either STRUCT (if connectors return schema) or > a MAP (schemaless). > I understand why (how do you add a field to a STRING?), but it also means > that you can't have an example for SMT that works with Apache Kafka out of > the box. Which makes documentation kind of painful. > Either we modify the file connector or we modify the SMTs to deal with STRING > better, but something gotta change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4917) Our built-in file connector can't work with our built-in SMT
[ https://issues.apache.org/jira/browse/KAFKA-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-4917: Labels: newbie (was: ) > Our built-in file connector can't work with our built-in SMT > > > Key: KAFKA-4917 > URL: https://issues.apache.org/jira/browse/KAFKA-4917 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira > Labels: newbie > > Our built-in file connector always returns STRING schema. > All our transformations expect either STRUCT (if connectors return schema) or > a MAP (schemaless). > I understand why (how do you add a field to a STRING?), but it also means > that you can't have an example for SMT that works with Apache Kafka out of > the box. Which makes documentation kind of painful. > Either we modify the file connector or we modify the SMTs to deal with STRING > better, but something gotta change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-4917) Our built-in file connector can't work with our built-in SMT
Gwen Shapira created KAFKA-4917: --- Summary: Our built-in file connector can't work with our built-in SMT Key: KAFKA-4917 URL: https://issues.apache.org/jira/browse/KAFKA-4917 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Gwen Shapira Our built-in file connector always returns STRING schema. All our transformations expect either STRUCT (if connectors return schema) or a MAP (schemaless). I understand why (how do you add a field to a STRING?), but it also means that you can't have an example for SMT that works with Apache Kafka out of the box. Which makes documentation kind of painful. Either we modify the file connector or we modify the SMTs to deal with STRING better, but something gotta change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4878) Kafka Connect does not log connector configuration errors
[
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4878:
Fix Version/s: 0.10.2.1
> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
> configuration is invalid (use the endpoint `/{connectorType}/config/validate`
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-4878) Kafka Connect does not log connector configuration errors
[
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4878:
Priority: Blocker (was: Major)
> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Priority: Blocker
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
> configuration is invalid (use the endpoint `/{connectorType}/config/validate`
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (KAFKA-4878) Kafka Connect does not log connector configuration errors
[
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923586#comment-15923586
]
Gwen Shapira edited comment on KAFKA-4878 at 3/14/17 5:07 AM:
--
This is even worse in stand-alone mode because if you start it with bad config,
it won't start and you don't even have a REST API...
So far I ran into 4 users complaining about it, and 0.10.2.0 has only been out
for few weeks...
was (Author: gwenshap):
This is even worse in stand-alone mode because if you start it with bad config,
it won't start and you don't even have a REST API...
So far I ran into 4 users complaining about it, and 3.2 has only been out for a
week...
> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Priority: Blocker
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
> configuration is invalid (use the endpoint `/{connectorType}/config/validate`
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4878) Kafka Connect does not log connector configuration errors
[
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923586#comment-15923586
]
Gwen Shapira commented on KAFKA-4878:
-
This is even worse in stand-alone mode because if you start it with bad config,
it won't start and you don't even have a REST API...
So far I ran into 4 users complaining about it, and 3.2 has only been out for a
week...
> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
> configuration is invalid (use the endpoint `/{connectorType}/config/validate`
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KAFKA-4882) Remove internal converter configuration from example property files
Gwen Shapira created KAFKA-4882: --- Summary: Remove internal converter configuration from example property files Key: KAFKA-4882 URL: https://issues.apache.org/jira/browse/KAFKA-4882 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Our internal converter configuration is shown in connect-distributed.properties and connect-standalone.properties. This tempts users to change it. In particular, they seem to believe it needs to be identical to key/value converters. In reality, Connect doesn't deal well with anything other than schemaless JSON as the internal converter. Users get errors and find it hard to figure out what went wrong (since this is internal, they are not expected to). Let's stop tempting users into shooting their own feet? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-4878) Kafka Connect does not log connector configuration errors
Gwen Shapira created KAFKA-4878:
---
Summary: Kafka Connect does not log connector configuration errors
Key: KAFKA-4878
URL: https://issues.apache.org/jira/browse/KAFKA-4878
Project: Kafka
Issue Type: Bug
Reporter: Gwen Shapira
Currently, on connector configuration error, Kafka Connect (both distributed
and stand alone) logs:
org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector
configuration is invalid (use the endpoint `/{connectorType}/config/validate`
to get a full list of errors)
This is annoying because:
1. If I'm using stand-alone mode, I may have configured my connector via
configuration file and I don't want to know about the REST API at all.
2. The output of validate is rather annoying
What I'd like to see in the output is:
1. number of errors in my configuration
2. at least one error, preferably all of them
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-3491) Issue with consumer close() in finally block with 'enable.auto.commit=true'
[
https://issues.apache.org/jira/browse/KAFKA-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896561#comment-15896561
]
Gwen Shapira commented on KAFKA-3491:
-
Very late to the party, but it looks like the issue is still there, so we are
still losing events when using the common pattern shown in this JIRA? And the
work-around is still to call unsubscribe(), but we are not telling anyone about
it?
> Issue with consumer close() in finally block with 'enable.auto.commit=true'
> ---
>
> Key: KAFKA-3491
> URL: https://issues.apache.org/jira/browse/KAFKA-3491
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 0.9.0.0, 0.9.0.1
>Reporter: dan norwood
>Assignee: Jason Gustafson
>Priority: Minor
>
> imagine you have a run loop that looks like the following:
> {code:java}
> public void run() {
> try {
> consumer.subscribe(topics);
> while (true) {
> ConsumerRecords records = consumer.poll(Long.MAX_VALUE);
> records.forEach(record -> process(record));
> }
> } catch (WakeupException e) {
> // ignore, we're closing
> } catch (Exception e) {
> log.error("Unexpected error", e);
> } finally {
> consumer.close();
> }
> }
> {code}
> if you run this with 'enable.auto.commit=true' and throw an exception in the
> 'process()' method you will still try to commit all the read, but
> unprocessed, offsets in the most recent batch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KAFKA-4567) Connect Producer and Consumer ignore ssl parameters configured for worker
[ https://issues.apache.org/jira/browse/KAFKA-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-4567. - Resolution: Fixed Fix Version/s: 0.11.0.0 Issue resolved by pull request 2511 [https://github.com/apache/kafka/pull/2511] > Connect Producer and Consumer ignore ssl parameters configured for worker > - > > Key: KAFKA-4567 > URL: https://issues.apache.org/jira/browse/KAFKA-4567 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.10.1.1 >Reporter: Sönke Liebau >Priority: Minor > Fix For: 0.11.0.0 > > > When using Connect with a SSL enabled Kafka cluster, the configuration > options are either documented a bit misleading, or handled in an incorrect > way. > The documentation states the usual available SSL options > (ssl.keystore.location, ssl.truststore.location, ...) and these are picked up > and used for the producers and consumers that are used to communicate with > the status, offset and configs topics. > For the producers and consumers that are used for the actual data, these > parameters are ignored as can be seen > [here|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L98], > which results in plaintext communication on an SSL port, leading to an OOM > exception ([KAFKA-4493|https://issues.apache.org/jira/browse/KAFKA-4493]). > So in order to get Connect to communicate with a secured cluster you need to > override all SSL configs with the prefixes _consumer._ and _producer._ and > duplicate the values already set at a global level. > The documentation states: > bq. The most critical site-specific options, such as the Kafka bootstrap > servers, are already exposed via the standard worker configuration. > Since the address for the cluster is exposed here, I would propose that there > is no reason not to also pass the SSL parameters through to the consumers and > producers, as it is clearly intended that communication happens with the same > cluster. > In fringe cases, these can still be overridden manually to achieve different > behavior. > I am happy to create a pull request to address this or clarify the docs, > after we decide which one is the appropriate course of action. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KAFKA-4792) Kafka Connect: Add ByteArray Converter
[ https://issues.apache.org/jira/browse/KAFKA-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-4792. - Resolution: Duplicate > Kafka Connect: Add ByteArray Converter > -- > > Key: KAFKA-4792 > URL: https://issues.apache.org/jira/browse/KAFKA-4792 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira > > We currently have JSON and String converters. > Some data sources have non-string data (like database BLOBs) and some Kafka > have binary data they want to land in binary format in the target system. > We need a non-converting converter that will allow just dumping bytes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4280) Add REST resource for showing available connector plugin configs
[ https://issues.apache.org/jira/browse/KAFKA-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-4280: Labels: newbie (was: ) > Add REST resource for showing available connector plugin configs > > > Key: KAFKA-4280 > URL: https://issues.apache.org/jira/browse/KAFKA-4280 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Gwen Shapira > Labels: newbie > > Connector-plugins allow listing the plugs and validating configs, but we have > nothing (I think?) for listing available configuration properties. > If this doesn't exist, would be good for usability to add it. If it does > exist, perhaps document it? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4276) REST configuration not visible in connector properties config files
[ https://issues.apache.org/jira/browse/KAFKA-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-4276: Resolution: Fixed Fix Version/s: 0.11.0.0 Status: Resolved (was: Patch Available) Merged above PR. > REST configuration not visible in connector properties config files > --- > > Key: KAFKA-4276 > URL: https://issues.apache.org/jira/browse/KAFKA-4276 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Assignee: Akhilesh Naidu > Labels: newbie > Fix For: 0.11.0.0 > > > REST host and port configs are not visible in connect-distributed.properties. > I think this leads to some confusion as users don't know there's even a REST > port and need to read the docs to find about it and the default (and these > are marked as LOW configs). > We can easily improve that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KAFKA-4115) Grow default heap settings for distributed Connect from 256M to 1G
[
https://issues.apache.org/jira/browse/KAFKA-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4115:
Labels: newbie (was: )
> Grow default heap settings for distributed Connect from 256M to 1G
> --
>
> Key: KAFKA-4115
> URL: https://issues.apache.org/jira/browse/KAFKA-4115
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Assignee: Shikhar Bhushan
> Labels: newbie
>
> Currently, both {{connect-standalone.sh}} and {{connect-distributed.sh}}
> start the Connect JVM with the default heap settings from
> {{kafka-run-class.sh}} of {{-Xmx256M}}.
> At least for distributed connect, we should default to a much higher limit
> like 1G. While the 'correct' sizing is workload dependent, with a system
> where you can run arbitrary connector plugins which may perform buffering of
> data, we should provide for more headroom.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KAFKA-3832) Kafka Connect's JSON Converter never outputs a null value
[
https://issues.apache.org/jira/browse/KAFKA-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-3832:
Labels: newbie (was: )
> Kafka Connect's JSON Converter never outputs a null value
> -
>
> Key: KAFKA-3832
> URL: https://issues.apache.org/jira/browse/KAFKA-3832
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
> Labels: newbie
>
> Kafka Connect's JSON Converter will never output a null value when
> {{enableSchemas=true}}. This means that when a connector outputs a
> {{SourceRecord}} with a null value, the JSON Converter will always produce a
> message value with:
> {code:javascript}
> {
> "schema": null,
> "payload": null
> }
> {code}
> And, this means that while Kafka log compaction will always be able to remove
> earlier messages with the same key, log compaction will never remove _all_ of
> the messages with the same key.
> The JSON Connector's {{fromConnectData(...)}} should always return null when
> it is supplied a null value.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KAFKA-4793) Kafka Connect: POST /connectors/(string: name)/restart doesn't start failed tasks
Gwen Shapira created KAFKA-4793: --- Summary: Kafka Connect: POST /connectors/(string: name)/restart doesn't start failed tasks Key: KAFKA-4793 URL: https://issues.apache.org/jira/browse/KAFKA-4793 Project: Kafka Issue Type: Improvement Reporter: Gwen Shapira Sometimes tasks stop due to repeated failures. Users will want to restart the connector and have it retry after fixing an issue. We expected "POST /connectors/(string: name)/restart" to cause retry of failed tasks, but this doesn't appear to be the case. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-4792) Kafka Connect: Add ByteArray Converter
Gwen Shapira created KAFKA-4792: --- Summary: Kafka Connect: Add ByteArray Converter Key: KAFKA-4792 URL: https://issues.apache.org/jira/browse/KAFKA-4792 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Gwen Shapira We currently have JSON and String converters. Some data sources have non-string data (like database BLOBs) and some Kafka have binary data they want to land in binary format in the target system. We need a non-converting converter that will allow just dumping bytes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4564) When the destination brokers are down or misconfigured in config, Streams should fail fast
[ https://issues.apache.org/jira/browse/KAFKA-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856964#comment-15856964 ] Gwen Shapira commented on KAFKA-4564: - OMG, this is great. I got this: [2017-02-07 14:48:10,250] INFO Deleting obsolete state directory 0_0 for task 0_0 as cleanup delay of 0 ms has passed (org.apache.kafka.streams.processor.internals.StateDirectory) Exception in thread "main" org.apache.kafka.streams.errors.StreamsException: Could not find any available broker. at org.apache.kafka.streams.processor.internals.StreamsKafkaClient.getBrokerId(StreamsKafkaClient.java:205) at org.apache.kafka.streams.processor.internals.StreamsKafkaClient.checkBrokerCompatibility(StreamsKafkaClient.java:266) at org.apache.kafka.streams.KafkaStreams.checkBrokerVersionCompatibility(KafkaStreams.java:392) at org.apache.kafka.streams.KafkaStreams.start(KafkaStreams.java:416) at com.shapira.examples.streams.stockstats.StockStatsExample.main(StockStatsExample.java:65) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) It took about a minute to get it though. Is this timeout tunable? > When the destination brokers are down or misconfigured in config, Streams > should fail fast > -- > > Key: KAFKA-4564 > URL: https://issues.apache.org/jira/browse/KAFKA-4564 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Guozhang Wang >Assignee: Umesh Chaudhary > Labels: newbie > > Today if Kafka is down or users misconfigure the bootstrap list, Streams may > just hangs for a while without any error messages even with the log4j > enabled, which is quite confusing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (KAFKA-4564) When the destination brokers are down or misconfigured in config, Streams should fail fast
[
https://issues.apache.org/jira/browse/KAFKA-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856964#comment-15856964
]
Gwen Shapira edited comment on KAFKA-4564 at 2/7/17 10:51 PM:
--
OMG, this is great. I got this:
{code}
[2017-02-07 14:48:10,250] INFO Deleting obsolete state directory 0_0 for task
0_0 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
Exception in thread "main" org.apache.kafka.streams.errors.StreamsException:
Could not find any available broker.
at
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.getBrokerId(StreamsKafkaClient.java:205)
at
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.checkBrokerCompatibility(StreamsKafkaClient.java:266)
at
org.apache.kafka.streams.KafkaStreams.checkBrokerVersionCompatibility(KafkaStreams.java:392)
at org.apache.kafka.streams.KafkaStreams.start(KafkaStreams.java:416)
at
com.shapira.examples.streams.stockstats.StockStatsExample.main(StockStatsExample.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
{code}
It took about a minute to get it though. Is this timeout tunable?
was (Author: gwenshap):
OMG, this is great. I got this:
[2017-02-07 14:48:10,250] INFO Deleting obsolete state directory 0_0 for task
0_0 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
Exception in thread "main" org.apache.kafka.streams.errors.StreamsException:
Could not find any available broker.
at
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.getBrokerId(StreamsKafkaClient.java:205)
at
org.apache.kafka.streams.processor.internals.StreamsKafkaClient.checkBrokerCompatibility(StreamsKafkaClient.java:266)
at
org.apache.kafka.streams.KafkaStreams.checkBrokerVersionCompatibility(KafkaStreams.java:392)
at org.apache.kafka.streams.KafkaStreams.start(KafkaStreams.java:416)
at
com.shapira.examples.streams.stockstats.StockStatsExample.main(StockStatsExample.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
It took about a minute to get it though. Is this timeout tunable?
> When the destination brokers are down or misconfigured in config, Streams
> should fail fast
> --
>
> Key: KAFKA-4564
> URL: https://issues.apache.org/jira/browse/KAFKA-4564
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
> Labels: newbie
>
> Today if Kafka is down or users misconfigure the bootstrap list, Streams may
> just hangs for a while without any error messages even with the log4j
> enabled, which is quite confusing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KAFKA-4737) Streams apps hang if started when brokers are not available
[ https://issues.apache.org/jira/browse/KAFKA-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-4737. - Resolution: Duplicate > Streams apps hang if started when brokers are not available > --- > > Key: KAFKA-4737 > URL: https://issues.apache.org/jira/browse/KAFKA-4737 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Start a streams example while broker is down, and it will just hang there. It > will also hang on shutdown. > I'd expect it to exit with an error message if broker is not available. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4737) Streams apps hang if started when brokers are not available
[ https://issues.apache.org/jira/browse/KAFKA-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853423#comment-15853423 ] Gwen Shapira commented on KAFKA-4737: - It is! Sorry for the duplicate. I'll close this. > Streams apps hang if started when brokers are not available > --- > > Key: KAFKA-4737 > URL: https://issues.apache.org/jira/browse/KAFKA-4737 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Start a streams example while broker is down, and it will just hang there. It > will also hang on shutdown. > I'd expect it to exit with an error message if broker is not available. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-4737) Streams apps hang if started when brokers are not available
Gwen Shapira created KAFKA-4737: --- Summary: Streams apps hang if started when brokers are not available Key: KAFKA-4737 URL: https://issues.apache.org/jira/browse/KAFKA-4737 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Start a streams example while broker is down, and it will just hang there. It will also hang on shutdown. I'd expect it to exit with an error message if broker is not available. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (KAFKA-4733) Improve Streams Reset Tool console output
[
https://issues.apache.org/jira/browse/KAFKA-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira reassigned KAFKA-4733:
---
Assignee: Gwen Shapira
> Improve Streams Reset Tool console output
> -
>
> Key: KAFKA-4733
> URL: https://issues.apache.org/jira/browse/KAFKA-4733
> Project: Kafka
> Issue Type: Improvement
> Components: streams, tools
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Minor
> Labels: beginner, easyfix, newbie
>
> Currently, the console output of {{bin/kafka-streams-application-reset.sh}}
> is not helpful enough to users:
> - we should add a hint to clean up local state using
> {{KafkaStreams#cleanup()}}
> - we should clarify what to specify for each parameter (i,e, what is an input
> topic, what is an intermediate topics)
> - we should clarify, that it is not required to specify internal topics (and
> what those are)
> - we should clarify what the tool does for the different topics, ie.,
> seek+commit, delete etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KAFKA-1776) Re-factor out existing tools that have been implemented behind the CLI
[ https://issues.apache.org/jira/browse/KAFKA-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-1776. - Resolution: Duplicate > Re-factor out existing tools that have been implemented behind the CLI > -- > > Key: KAFKA-1776 > URL: https://issues.apache.org/jira/browse/KAFKA-1776 > Project: Kafka > Issue Type: Sub-task >Reporter: Joe Stein >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-4451) Recovering empty replica yields negative offsets in index of compact partitions
[
https://issues.apache.org/jira/browse/KAFKA-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-4451:
Labels: reliability (was: )
> Recovering empty replica yields negative offsets in index of compact
> partitions
> ---
>
> Key: KAFKA-4451
> URL: https://issues.apache.org/jira/browse/KAFKA-4451
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Michael Schiff
> Labels: reliability
>
> Bringing up an empty broker.
> the partition for a compact topic is not split into multiple log files. All
> data is written into a single log file, causing offsets to overflow.
> A dump of the affected broker shortly after it started replicating:
> {code}
> michael.schiff@stats-kafka09:~$ sudo /opt/kafka/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --files
> /kafka/attainment_event-0/.index | head -n 10
> Dumping /kafka/attainment_event-0/.index
> offset: 1022071124 position: 1037612
> offset: -1713432120 position: 1348740
> offset: -886291423 position: 2397130
> offset: -644750126 position: 3445630
> offset: -57889876 position: 4493972
> offset: 433950099 position: 5388461
> offset: 1071769472 position: 6436837
> offset: 1746859069 position: 7485367
> offset: 2090359736 position: 8533822
> ...
> {code}
> and the dump of the same log file from the leader of this partition
> {code}
> michael.schiff@stats-kafka12:~$ sudo /opt/kafka/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --files
> /kafka/attainment_event-0/.index
> [sudo] password for michael.schiff:
> Dumping /kafka/attainment_event-0/.index
> offset: 353690666 position: 262054
> offset: 633140428 position: 523785
> offset: 756537951 position: 785815
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
