[jira] [Commented] (KAFKA-1019) kafka-preferred-replica-election.sh will fail without clear error message if /brokers/topics/[topic]/partitions does not exist
[
https://issues.apache.org/jira/browse/KAFKA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084641#comment-14084641
]
Mickael Hemri commented on KAFKA-1019:
--
I tried with zookeeper 3.3.6 and we have the same issue.
To reproduce:
Create a topic named testid
{code}bin/kafka-topics.sh --topic testid --replication-factor 3 --partition 3
--zookeeper 127.0.0.1:2181/kafka --create
Created topic testid.{code}
{code}./bin/kafka-topics.sh --topic testid --zookeeper 127.0.0.1:2181/kafka
--describe
Topic:testidPartitionCount:3ReplicationFactor:3 Configs:
Topic: testid Partition: 0Leader: 31985 Replicas:
31985,9920,4580 Isr: 31985,9920,4580
Topic: testid Partition: 1Leader: 4580Replicas:
4580,31985,9920 Isr: 4580,31985,9920
Topic: testid Partition: 2Leader: 9920Replicas:
9920,4580,31985 Isr: 9920,4580,31985
{code}
Ok great, we have leaders and /brokers/topics/testid/partitions in zookeeper
Delete testid topic
{code}bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic testid
--zookeeper 127.0.0.1:2181/kafka
deletion succeeded!
{code}
Create again a topic named testid
{code}bin/kafka-topics.sh --topic testid --replication-factor 3 --partition 3
--zookeeper 127.0.0.1:2181/kafka --create
Created topic testid.{code}
Now check:
{code}./bin/kafka-topics.sh --topic testid --zookeeper 127.0.0.1:2181/kafka
--describe
Topic:testidPartitionCount:3ReplicationFactor:3 Configs:
Topic: testid Partition: 0Leader: noneReplicas:
31985,4580,9920 Isr:
Topic: testid Partition: 1Leader: noneReplicas:
4580,9920,31985 Isr:
Topic: testid Partition: 2Leader: noneReplicas:
9920,31985,4580 Isr:{code}
As you can see we have no leader when we create the topic after a deletion. And
there is no /brokers/topics/testid/partitions in zookeeper
It works again with a different topic name, so it seems that something is not
properly deleted with DeleteTopicCommand command.
We reproduced it on 3 differents zookeeper chroot: 127.0.0.1:2181/kafka,
127.0.0.1:2181/kafka2 and 127.0.0.1:2181/kafka3
Thanks
kafka-preferred-replica-election.sh will fail without clear error message if
/brokers/topics/[topic]/partitions does not exist
--
Key: KAFKA-1019
URL: https://issues.apache.org/jira/browse/KAFKA-1019
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Guozhang Wang
Labels: newbie
Fix For: 0.8.2
From Libo Yu:
I tried to run kafka-preferred-replica-election.sh on our kafka cluster.
But I got this expection:
Failed to start preferred replica election
org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /brokers/topics/uattoqaaa.default/partitions
I checked zookeeper and there is no
/brokers/topics/uattoqaaa.default/partitions. All I found is
/brokers/topics/uattoqaaa.default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1562) kafka-topics.sh alter add partitions resets cleanup.policy
[
https://issues.apache.org/jira/browse/KAFKA-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Rao updated KAFKA-1562:
---
Resolution: Fixed
Fix Version/s: 0.8.2
Assignee: Jonathan Natkins
Status: Resolved (was: Patch Available)
Thanks for the patch. +1 and committed to trunk.
kafka-topics.sh alter add partitions resets cleanup.policy
--
Key: KAFKA-1562
URL: https://issues.apache.org/jira/browse/KAFKA-1562
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.1.1
Reporter: Kenny
Assignee: Jonathan Natkins
Fix For: 0.8.2
Attachments: KAFKA-1562.patch, KAFKA-1562_2014-07-30_13:18:21.patch,
KAFKA-1562_2014-07-30_13:51:25.patch, KAFKA-1562_2014-08-02_11:10:34.patch
When partitions are added to an already existing topic the
cleanup.policy=compact is not retained.
{code}
./kafka-topics.sh --zookeeper localhost --create --partitions 1
--replication-factor 1 --topic KTEST --config cleanup.policy=compact
./kafka-topics.sh --zookeeper localhost --describe --topic KTEST
Topic:KTEST PartitionCount:1ReplicationFactor:1
Configs:cleanup.policy=compact
Topic: KTESTPartition: 0Leader: 0 Replicas: 0 Isr: 0
./kafka-topics.sh --zookeeper localhost --alter --partitions 3 --topic KTEST
--config cleanup.policy=compact
./kafka-topics.sh --zookeeper localhost --describe --topic KTEST
Topic:KTEST PartitionCount:3ReplicationFactor:1 Configs:
Topic: KTESTPartition: 0Leader: 0 Replicas: 0 Isr: 0
Topic: KTESTPartition: 1Leader: 0 Replicas: 0 Isr: 0
Topic: KTESTPartition: 2Leader: 0 Replicas: 0 Isr: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 23895: Patch for KAFKA-1419
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23895/#review49454 --- build.gradle https://reviews.apache.org/r/23895/#comment86528 Can we remove Annotations_2.8.scala and rename Annotations_2.9+.scala to just Annotations.scala? We can then potentially remove sourceSets setting completely. - Jun Rao On July 29, 2014, 3:13 p.m., Ivan Lyutov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23895/ --- (Updated July 29, 2014, 3:13 p.m.) Review request for kafka. Bugs: KAFKA-1419 https://issues.apache.org/jira/browse/KAFKA-1419 Repository: kafka Description --- KAFKA-1419 - cross build for scala 2.11 - dropped scala 2.8 support - minor bug fixes KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency version - updated scala version to 2.11.2 - added getBuffer to ByteBufferMessageSet classes Diffs - build.gradle a72905df824ba68bed5d5170d18873c23e1782c9 core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala fecee8d5f7b32f483bb1bfc6a5080d589906f9c4 core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 73401c5ff34d08abce22267aa9c4d86632c6fb74 gradle.properties 4827769a3f8e34f0fe7e783eb58e44d4db04859b gradle/buildscript.gradle 225e0a82708bc5f390e5e2c1d4d9a0d06f491b95 gradle/wrapper/gradle-wrapper.properties 610282a699afc89a82203ef0e4e71ecc53761039 scala.gradle ebd21b870c0746aade63248344ab65d9b5baf820 Diff: https://reviews.apache.org/r/23895/diff/ Testing --- Thanks, Ivan Lyutov
Re: Review Request 23895: Patch for KAFKA-1419
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23895/ --- (Updated Aug. 4, 2014, 2:43 p.m.) Review request for kafka. Bugs: KAFKA-1419 https://issues.apache.org/jira/browse/KAFKA-1419 Repository: kafka Description (updated) --- KAFKA-1419 - cross build for scala 2.11 - dropped scala 2.8 support - minor bug fixes KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency version - updated scala version to 2.11.2 - added getBuffer to ByteBufferMessageSet classes KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency version - updated scala version to 2.11.2 - added getBuffer to ByteBufferMessageSet classes - removed annotations 2.8 file Diffs (updated) - build.gradle a72905df824ba68bed5d5170d18873c23e1782c9 core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala fecee8d5f7b32f483bb1bfc6a5080d589906f9c4 core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 73401c5ff34d08abce22267aa9c4d86632c6fb74 core/src/main/scala/kafka/utils/Annotations_2.8.scala 28269eb037109f7680b9da732e4baa51c9a594b6 core/src/main/scala/kafka/utils/Annotations_2.9+.scala gradle.properties 4827769a3f8e34f0fe7e783eb58e44d4db04859b gradle/buildscript.gradle 225e0a82708bc5f390e5e2c1d4d9a0d06f491b95 gradle/wrapper/gradle-wrapper.properties 610282a699afc89a82203ef0e4e71ecc53761039 scala.gradle ebd21b870c0746aade63248344ab65d9b5baf820 Diff: https://reviews.apache.org/r/23895/diff/ Testing --- Thanks, Ivan Lyutov
[jira] [Updated] (KAFKA-1419) cross build for scala 2.11
[ https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Lyutov updated KAFKA-1419: --- Attachment: KAFKA-1419_2014-08-04_14:43:26.patch cross build for scala 2.11 -- Key: KAFKA-1419 URL: https://issues.apache.org/jira/browse/KAFKA-1419 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 0.8.1 Reporter: Scott Clasen Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1419.patch, KAFKA-1419.patch, KAFKA-1419_2014-07-28_15:05:16.patch, KAFKA-1419_2014-07-29_15:13:43.patch, KAFKA-1419_2014-08-04_14:43:26.patch Please publish builds for scala 2.11, hopefully just needs a small tweak to the gradle conf? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1419) cross build for scala 2.11
[ https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084715#comment-14084715 ] Ivan Lyutov commented on KAFKA-1419: Updated reviewboard https://reviews.apache.org/r/23895/diff/ against branch apache/trunk cross build for scala 2.11 -- Key: KAFKA-1419 URL: https://issues.apache.org/jira/browse/KAFKA-1419 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 0.8.1 Reporter: Scott Clasen Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1419.patch, KAFKA-1419.patch, KAFKA-1419_2014-07-28_15:05:16.patch, KAFKA-1419_2014-07-29_15:13:43.patch, KAFKA-1419_2014-08-04_14:43:26.patch Please publish builds for scala 2.11, hopefully just needs a small tweak to the gradle conf? -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Newer Zookeeper?
I have heard issues from installations running 3.4.X that I have not heard from installations running 3.3.X (i.e. zk breaking quorum and cluster going down). In none of these cases did I have an opportunity to isolate and reproduce and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was agreed to being a lower risk/cost solution to the problem. Once on 3.3.X the issues didn't happen again. So I can't say for sure if there are issues with running 3.4.X but I would suggest some due diligence in testing and production operation to validate that every case that Kafka requires operates correctly (and over some time). There is a cost to this so some company(s) will have to take that investment and do some cost vs the benefit of moving to 3.4.x. I currently recommend running a separate ZK cluster for Kafka production and not chroot into an existing one except for test/qa/dev. I don't know what others experience is with 3.4.X as I said the issues I have seen could have been coincidence. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
[jira] [Commented] (KAFKA-1499) Broker-side compression configuration
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084777#comment-14084777 ] Manikumar Reddy commented on KAFKA-1499: I would like to propose new server property *log.compression.type* and topic override property *compression.type* If set, this property is used for compression type at server-side. All non-compressed messages on the broker will be compressed to this compression type. log.compression.type=none|gzip|snappy (default = none) compression.type=compression.type Broker-side compression configuration - Key: KAFKA-1499 URL: https://issues.apache.org/jira/browse/KAFKA-1499 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Manikumar Reddy Labels: newbie++ Fix For: 0.8.2 Original Estimate: 72h Remaining Estimate: 72h A given topic can have messages in mixed compression codecs. i.e., it can also have a mix of uncompressed/compressed messages. It will be useful to support a broker-side configuration to recompress messages to a specific compression codec. i.e., all messages (for all topics) on the broker will be compressed to this codec. We could have per-topic overrides as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)
[
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084778#comment-14084778
]
Cosmin Lehene commented on KAFKA-1566:
--
The convention is to have it in conf/
We can have an env variable for it's location.
We generate these files at deploy time (they are Puppet .erb templates in our
case) based on some versioned configuration.
Kafka environment configuration (kafka-env.sh)
--
Key: KAFKA-1566
URL: https://issues.apache.org/jira/browse/KAFKA-1566
Project: Kafka
Issue Type: Improvement
Components: tools
Reporter: Cosmin Lehene
Assignee: Cosmin Lehene
Labels: newbie
Fix For: 0.8.2, 0.9.0
It would be useful (especially for automated deployments) to have an
environment configuration file that could be sourced from the launcher files
(e.g. kafka-run-server.sh).
This is how this could look like kafka-env.sh
{code}
export KAFKA_JVM_PERFORMANCE_OPTS=-XX:+UseCompressedOops
-XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC
-XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35' %
export KAFKA_HEAP_OPTS='-Xmx1G -Xms1G' %
export KAFKA_LOG4J_OPTS=-Dkafka.logs.dir=/var/log/kafka
{code}
kafka-server-start.sh
{code}
...
source $base_dir/config/kafka-env.sh
...
{code}
This approach is consistent with Hadoop and HBase. However the idea here is
to be able to set these values in a single place without having to edit
startup scripts.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (KAFKA-1499) Broker-side compression configuration
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy reassigned KAFKA-1499: -- Assignee: Manikumar Reddy Broker-side compression configuration - Key: KAFKA-1499 URL: https://issues.apache.org/jira/browse/KAFKA-1499 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Manikumar Reddy Labels: newbie++ Fix For: 0.8.2 Original Estimate: 72h Remaining Estimate: 72h A given topic can have messages in mixed compression codecs. i.e., it can also have a mix of uncompressed/compressed messages. It will be useful to support a broker-side configuration to recompress messages to a specific compression codec. i.e., all messages (for all topics) on the broker will be compressed to this codec. We could have per-topic overrides as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1499) Broker-side compression configuration
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084777#comment-14084777 ] Manikumar Reddy edited comment on KAFKA-1499 at 8/4/14 3:43 PM: I would like to propose new server property *log.compression.type* and topic override property *compression.type* If set, this property is used for compression type at server-side. All non-compressed messages on the broker will be compressed to this compression type. log.compression.type=none|gzip|snappy (default = none) compression.type=none|gzip|snappy was (Author: omkreddy): I would like to propose new server property *log.compression.type* and topic override property *compression.type* If set, this property is used for compression type at server-side. All non-compressed messages on the broker will be compressed to this compression type. log.compression.type=none|gzip|snappy (default = none) compression.type=compression.type Broker-side compression configuration - Key: KAFKA-1499 URL: https://issues.apache.org/jira/browse/KAFKA-1499 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Manikumar Reddy Labels: newbie++ Fix For: 0.8.2 Original Estimate: 72h Remaining Estimate: 72h A given topic can have messages in mixed compression codecs. i.e., it can also have a mix of uncompressed/compressed messages. It will be useful to support a broker-side configuration to recompress messages to a specific compression codec. i.e., all messages (for all topics) on the broker will be compressed to this codec. We could have per-topic overrides as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)
[
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084788#comment-14084788
]
Cosmin Lehene commented on KAFKA-1566:
--
Somehow related: KAFKA_LOG_DIR would be better than setting KAFKA_LOG4J_OPTS.
kafka-run-class.sh would check for this
{code}
LOG_DIR=${KAFKA_LOG_DIR:-$base_dir/logs}
{code}
Perhaps a new issue may be useful (otherwise I could just add them to this
patch).
Kafka environment configuration (kafka-env.sh)
--
Key: KAFKA-1566
URL: https://issues.apache.org/jira/browse/KAFKA-1566
Project: Kafka
Issue Type: Improvement
Components: tools
Reporter: Cosmin Lehene
Assignee: Cosmin Lehene
Labels: newbie
Fix For: 0.8.2, 0.9.0
It would be useful (especially for automated deployments) to have an
environment configuration file that could be sourced from the launcher files
(e.g. kafka-run-server.sh).
This is how this could look like kafka-env.sh
{code}
export KAFKA_JVM_PERFORMANCE_OPTS=-XX:+UseCompressedOops
-XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC
-XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35' %
export KAFKA_HEAP_OPTS='-Xmx1G -Xms1G' %
export KAFKA_LOG4J_OPTS=-Dkafka.logs.dir=/var/log/kafka
{code}
kafka-server-start.sh
{code}
...
source $base_dir/config/kafka-env.sh
...
{code}
This approach is consistent with Hadoop and HBase. However the idea here is
to be able to set these values in a single place without having to edit
startup scripts.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Build failed in Jenkins: Kafka-trunk #238
See https://builds.apache.org/job/Kafka-trunk/238/changes Changes: [junrao] kafka-1562; kafka-topics.sh alter add partitions resets cleanup.policy; patched by Jonathan Natkins; reviewed by Jun Rao -- [...truncated 671 lines...] there were 12 feature warning(s); re-run with -feature for details 7 warnings found :core:processResources UP-TO-DATE :core:classes :core:compileTestJava UP-TO-DATE :core:compileTestScala :core:processTestResources UP-TO-DATE :core:testClasses :core:test unit.kafka.utils.CommandLineUtilsTest testParseEmptyArg PASSED unit.kafka.utils.CommandLineUtilsTest testParseSingleArg PASSED unit.kafka.utils.CommandLineUtilsTest testParseArgs PASSED unit.kafka.common.TopicTest testInvalidTopicNames PASSED unit.kafka.common.ConfigTest testInvalidClientIds PASSED unit.kafka.common.ConfigTest testInvalidGroupIds PASSED kafka.server.LeaderElectionTest testLeaderElectionAndEpoch PASSED kafka.server.LeaderElectionTest testLeaderElectionWithStaleControllerEpoch PASSED kafka.server.LogRecoveryTest testHWCheckpointNoFailuresSingleLogSegment PASSED kafka.server.LogRecoveryTest testHWCheckpointWithFailuresSingleLogSegment PASSED kafka.server.LogRecoveryTest testHWCheckpointNoFailuresMultipleLogSegments PASSED kafka.server.LogRecoveryTest testHWCheckpointWithFailuresMultipleLogSegments PASSED kafka.server.IsrExpirationTest testIsrExpirationForStuckFollowers PASSED kafka.server.IsrExpirationTest testIsrExpirationForSlowFollowers PASSED kafka.server.HighwatermarkPersistenceTest testHighWatermarkPersistenceSinglePartition PASSED kafka.server.HighwatermarkPersistenceTest testHighWatermarkPersistenceMultiplePartitions PASSED kafka.server.DynamicConfigChangeTest testConfigChange PASSED kafka.server.DynamicConfigChangeTest testConfigChangeOnNonExistingTopic PASSED kafka.server.ServerShutdownTest testCleanShutdown PASSED kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled PASSED kafka.server.AdvertiseBrokerTest testBrokerAdvertiseToZK PASSED kafka.server.SimpleFetchTest testNonReplicaSeesHwWhenFetching PASSED kafka.server.SimpleFetchTest testReplicaSeesLeoWhenFetching PASSED kafka.server.OffsetCommitTest testUpdateOffsets PASSED kafka.server.OffsetCommitTest testCommitAndFetchOffsets PASSED kafka.server.OffsetCommitTest testLargeMetadataPayload PASSED kafka.server.ReplicaManagerTest testHighWaterMarkDirectoryMapping PASSED kafka.server.ReplicaManagerTest testHighwaterMarkRelativeDirectoryMapping PASSED kafka.server.LogOffsetTest testGetOffsetsForUnknownTopic PASSED kafka.server.LogOffsetTest testGetOffsetsBeforeLatestTime PASSED kafka.server.LogOffsetTest testEmptyLogsGetOffsets PASSED kafka.server.LogOffsetTest testGetOffsetsBeforeNow PASSED kafka.server.LogOffsetTest testGetOffsetsBeforeEarliestTime PASSED kafka.server.KafkaConfigTest testLogRetentionTimeHoursProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeMinutesProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeMsProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeNoConfigProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeBothMinutesAndHoursProvided PASSED kafka.server.KafkaConfigTest testLogRetentionTimeBothMinutesAndMsProvided PASSED kafka.server.KafkaConfigTest testAdvertiseDefaults PASSED kafka.server.KafkaConfigTest testAdvertiseConfigured PASSED kafka.server.KafkaConfigTest testUncleanLeaderElectionDefault PASSED kafka.server.KafkaConfigTest testUncleanElectionDisabled PASSED kafka.server.KafkaConfigTest testUncleanElectionEnabled PASSED kafka.server.KafkaConfigTest testUncleanElectionInvalid PASSED kafka.server.KafkaConfigTest testLogRollTimeMsProvided PASSED kafka.server.KafkaConfigTest testLogRollTimeBothMsAndHoursProvided PASSED kafka.server.KafkaConfigTest testLogRollTimeNoConfigProvided PASSED kafka.server.ReplicaFetchTest testReplicaFetcherThread PASSED kafka.server.RequestPurgatoryTest testRequestSatisfaction PASSED kafka.server.RequestPurgatoryTest testRequestExpiry PASSED kafka.utils.ReplicationUtilsTest testUpdateLeaderAndIsr PASSED kafka.utils.IteratorTemplateTest testIterator PASSED kafka.utils.SchedulerTest testMockSchedulerNonPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerPeriodicTask PASSED kafka.utils.SchedulerTest testReentrantTaskInMockScheduler PASSED kafka.utils.SchedulerTest testNonPeriodicTask PASSED kafka.utils.SchedulerTest testPeriodicTask PASSED kafka.utils.UtilsTest testSwallow PASSED kafka.utils.UtilsTest testCircularIterator PASSED kafka.utils.UtilsTest testReadBytes PASSED kafka.utils.UtilsTest testAbs PASSED kafka.utils.UtilsTest testReplaceSuffix PASSED kafka.utils.UtilsTest testReadInt PASSED kafka.utils.UtilsTest testCsvList PASSED kafka.utils.UtilsTest testInLock PASSED kafka.utils.JsonTest testJsonEncoding PASSED
Re: Newer Zookeeper?
Thanks for the heads-up, Joe. We've been shipping Zookeeper 3.4.X for over two years now (since CDH4.0) and have many production customers. I'll check if there are any known issues with breaking quorum. In any case I will take your comments into account and see if I can arrange for extra testing. Can you share more information about the 3.4.X issues you were seeing? Was there especially large clusters involved? large number of consumers? Also, I'm curious to hear more about the reasons for separate ZK cluster. I can see why you'll want it if you have thousands of consumers, but are there other reasons? Multiple zookeeper installs can be a pain to manage. Gwen On Mon, Aug 4, 2014 at 7:52 AM, Joe Stein joe.st...@stealth.ly wrote: I have heard issues from installations running 3.4.X that I have not heard from installations running 3.3.X (i.e. zk breaking quorum and cluster going down). In none of these cases did I have an opportunity to isolate and reproduce and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was agreed to being a lower risk/cost solution to the problem. Once on 3.3.X the issues didn't happen again. So I can't say for sure if there are issues with running 3.4.X but I would suggest some due diligence in testing and production operation to validate that every case that Kafka requires operates correctly (and over some time). There is a cost to this so some company(s) will have to take that investment and do some cost vs the benefit of moving to 3.4.x. I currently recommend running a separate ZK cluster for Kafka production and not chroot into an existing one except for test/qa/dev. I don't know what others experience is with 3.4.X as I said the issues I have seen could have been coincidence. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
Re: Newer Zookeeper?
Also, specific Zookeeper 3.4.X version where loss of quorum occurred will help. 3.4.5 fixed some pretty serious issues around hanging. Gwen On Mon, Aug 4, 2014 at 9:29 AM, Gwen Shapira gshap...@cloudera.com wrote: Thanks for the heads-up, Joe. We've been shipping Zookeeper 3.4.X for over two years now (since CDH4.0) and have many production customers. I'll check if there are any known issues with breaking quorum. In any case I will take your comments into account and see if I can arrange for extra testing. Can you share more information about the 3.4.X issues you were seeing? Was there especially large clusters involved? large number of consumers? Also, I'm curious to hear more about the reasons for separate ZK cluster. I can see why you'll want it if you have thousands of consumers, but are there other reasons? Multiple zookeeper installs can be a pain to manage. Gwen On Mon, Aug 4, 2014 at 7:52 AM, Joe Stein joe.st...@stealth.ly wrote: I have heard issues from installations running 3.4.X that I have not heard from installations running 3.3.X (i.e. zk breaking quorum and cluster going down). In none of these cases did I have an opportunity to isolate and reproduce and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was agreed to being a lower risk/cost solution to the problem. Once on 3.3.X the issues didn't happen again. So I can't say for sure if there are issues with running 3.4.X but I would suggest some due diligence in testing and production operation to validate that every case that Kafka requires operates correctly (and over some time). There is a cost to this so some company(s) will have to take that investment and do some cost vs the benefit of moving to 3.4.x. I currently recommend running a separate ZK cluster for Kafka production and not chroot into an existing one except for test/qa/dev. I don't know what others experience is with 3.4.X as I said the issues I have seen could have been coincidence. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
[jira] [Created] (KAFKA-1569) Creat tool to test end-to-end correctness in transactional mode
Raul Castro Fernandez created KAFKA-1569: Summary: Creat tool to test end-to-end correctness in transactional mode Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 24245: Patch for KAFKA-1524
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24245/ --- Review request for kafka. Bugs: KAFKA-1524 https://issues.apache.org/jira/browse/KAFKA-1524 Repository: kafka Description --- KAFKA-1524; implement transactional producer Diffs - clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 522881c972ca42ff4dfb6237a2db15b625334d7e clients/src/main/java/org/apache/kafka/clients/producer/AbortTransactionException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/InvalidTransactionStatusException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java c0f1d57e0feb894d9f246058cd0396461afe3225 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 36e8398416036cab84faad1f07159e5adefd8086 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f9de4af426449cceca12a8de9a9f54a6241d28d8 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 1ed3c28b436d28381d9402896e32d16f2586c65e clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java dd0af8aee98abed5d4a0dc50989e37888bb353fe clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionContext.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionControl.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/errors/TransactionCoordinatorNotAvailableException.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/errors/TransactionFailedException.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/record/Compressor.java 0323f5f7032dceb49d820c17a41b78c56591ffc4 clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 clients/src/main/java/org/apache/kafka/common/record/Record.java 10df9fd8d3f4ec8c277650fa7eab269f3ea30d85 clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 93b58d02eac0f8ca28440e3e0ebea28ed3a7673c clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 5489acac6806b3ae5e6d568d401d5a20c86cac05 clients/src/test/java/org/apache/kafka/clients/producer/TransactionContextTest.java PRE-CREATION clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 94a11121e207d5cf94dbc94443a8aa7edf387782 Diff: https://reviews.apache.org/r/24245/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Commented] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085063#comment-14085063 ] Raul Castro Fernandez commented on KAFKA-1524: -- Created reviewboard https://reviews.apache.org/r/24245/diff/ against branch origin/transactional_messaging Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch, KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1524: - Attachment: KAFKA-1524.patch Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch, KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] Subscription: outstanding kafka patches
Issue Subscription Filter: outstanding kafka patches (117 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1567 Metric memory leaking after closing the clients https://issues.apache.org/jira/browse/KAFKA-1567 KAFKA-1561 Data Loss for Incremented Replica Factor and Leader Election https://issues.apache.org/jira/browse/KAFKA-1561 KAFKA-1560 Make arguments to jira-python API more explicit in kafka-patch-review's get_jira() https://issues.apache.org/jira/browse/KAFKA-1560 KAFKA-1559 Upgrade Gradle wrapper to Gradle 2.0 https://issues.apache.org/jira/browse/KAFKA-1559 KAFKA-1550 Patch review tool should use git format-patch to generate patch https://issues.apache.org/jira/browse/KAFKA-1550 KAFKA-1543 Changing replication factor https://issues.apache.org/jira/browse/KAFKA-1543 KAFKA-1541 Add transactional request definitions to clients package https://issues.apache.org/jira/browse/KAFKA-1541 KAFKA-1536 Change the status of the JIRA to Patch Available in the kafka-review-tool https://issues.apache.org/jira/browse/KAFKA-1536 KAFKA-1528 Normalize all the line endings https://issues.apache.org/jira/browse/KAFKA-1528 KAFKA-1527 SimpleConsumer should be transaction-aware https://issues.apache.org/jira/browse/KAFKA-1527 KAFKA-1526 Producer performance tool should have an option to enable transactions https://issues.apache.org/jira/browse/KAFKA-1526 KAFKA-1525 DumpLogSegments should print transaction IDs https://issues.apache.org/jira/browse/KAFKA-1525 KAFKA-1524 Implement transactional producer https://issues.apache.org/jira/browse/KAFKA-1524 KAFKA-1523 Implement transaction manager module https://issues.apache.org/jira/browse/KAFKA-1523 KAFKA-1522 Transactional messaging request/response definitions https://issues.apache.org/jira/browse/KAFKA-1522 KAFKA-1517 Messages is a required argument to Producer Performance Test https://issues.apache.org/jira/browse/KAFKA-1517 KAFKA-1510 Force offset commits when migrating consumer offsets from zookeeper to kafka https://issues.apache.org/jira/browse/KAFKA-1510 KAFKA-1509 Restart of destination broker after unreplicated partition move leaves partitions without leader https://issues.apache.org/jira/browse/KAFKA-1509 KAFKA-1507 Using GetOffsetShell against non-existent topic creates the topic unintentionally https://issues.apache.org/jira/browse/KAFKA-1507 KAFKA-1500 adding new consumer requests using the new protocol https://issues.apache.org/jira/browse/KAFKA-1500 KAFKA-1498 new producer performance and bug improvements https://issues.apache.org/jira/browse/KAFKA-1498 KAFKA-1496 Using batch message in sync producer only sends the first message if we use a Scala Stream as the argument https://issues.apache.org/jira/browse/KAFKA-1496 KAFKA-1481 Stop using dashes AND underscores as separators in MBean names https://issues.apache.org/jira/browse/KAFKA-1481 KAFKA-1477 add authentication layer and initial JKS x509 implementation for brokers, producers and consumer for network communication https://issues.apache.org/jira/browse/KAFKA-1477 KAFKA-1476 Get a list of consumer groups https://issues.apache.org/jira/browse/KAFKA-1476 KAFKA-1475 Kafka consumer stops LeaderFinder/FetcherThreads, but application does not know https://issues.apache.org/jira/browse/KAFKA-1475 KAFKA-1471 Add Producer Unit Tests for LZ4 and LZ4HC compression https://issues.apache.org/jira/browse/KAFKA-1471 KAFKA-1468 Improve perf tests https://issues.apache.org/jira/browse/KAFKA-1468 KAFKA-1460 NoReplicaOnlineException: No replica for partition https://issues.apache.org/jira/browse/KAFKA-1460 KAFKA-1450 check invalid leader in a more robust way https://issues.apache.org/jira/browse/KAFKA-1450 KAFKA-1430 Purgatory redesign https://issues.apache.org/jira/browse/KAFKA-1430 KAFKA-1420 Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in unit tests https://issues.apache.org/jira/browse/KAFKA-1420 KAFKA-1419 cross build for scala 2.11 https://issues.apache.org/jira/browse/KAFKA-1419 KAFKA-1394 Ensure last segment isn't deleted on expiration when there are unflushed messages https://issues.apache.org/jira/browse/KAFKA-1394 KAFKA-1374 LogCleaner (compaction) does not support compressed topics https://issues.apache.org/jira/browse/KAFKA-1374 KAFKA-1372 Upgrade to Gradle 1.10 https://issues.apache.org/jira/browse/KAFKA-1372 KAFKA-1367 Broker topic metadata not kept in sync with ZooKeeper https://issues.apache.org/jira/browse/KAFKA-1367
Issues getting IntelliJ set up
Hi, I've been having some issues getting IntelliJ set up...I followed all the instructions on the wiki, and I've successfully imported the project, and run the jar Gradle target successfully. However, when I try to run a test in the IDE, I get a number of errors: /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java Error:(21, 30) java: package kafka.javaapi.producer does not exist Error:(22, 22) java: package kafka.producer does not exist Error:(23, 22) java: package kafka.producer does not exist Error:(24, 19) java: cannot find symbol symbol: class Utils location: package kafka.utils Error:(303, 39) java: cannot find symbol symbol: class KeyedMessage location: class kafka.tools.KafkaMigrationTool.MigrationThread And so on. The two classes that seem to be causing trouble are KafkaMigrationTool and ConsumerConnector. Has anybody run into this? Anyone know how to get around this issue? Thanks a lot, Natty
Re: Issues getting IntelliJ set up
Hi Johnathan, Did you update your scala version before you run gradle idea? Also try cleaning up all the artifacts and try it again, as perhaps your intellij is not picking up the right version and from the right build folder. Tim On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins na...@streamsets.com wrote: Hi, I've been having some issues getting IntelliJ set up...I followed all the instructions on the wiki, and I've successfully imported the project, and run the jar Gradle target successfully. However, when I try to run a test in the IDE, I get a number of errors: /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java Error:(21, 30) java: package kafka.javaapi.producer does not exist Error:(22, 22) java: package kafka.producer does not exist Error:(23, 22) java: package kafka.producer does not exist Error:(24, 19) java: cannot find symbol symbol: class Utils location: package kafka.utils Error:(303, 39) java: cannot find symbol symbol: class KeyedMessage location: class kafka.tools.KafkaMigrationTool.MigrationThread And so on. The two classes that seem to be causing trouble are KafkaMigrationTool and ConsumerConnector. Has anybody run into this? Anyone know how to get around this issue? Thanks a lot, Natty
Review Request 24253: Patch for KAFKA-1541
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24253/ --- Review request for kafka. Bugs: KAFKA-1541 https://issues.apache.org/jira/browse/KAFKA-1541 Repository: kafka Description --- KAFKA-1541; add transactional req definitions to clients package Diffs - clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 6fe7573973832615976defa37fe0dfbb8f911939 clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 3374bd98be8e565608c4e764ed10afdae383fb6f clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 044b03061802ee5e8ea4f1995fb0988e1a70e9a7 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionResponse.java PRE-CREATION Diff: https://reviews.apache.org/r/24253/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Commented] (KAFKA-1541) Add transactional request definitions to clients package
[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085171#comment-14085171 ] Raul Castro Fernandez commented on KAFKA-1541: -- Created reviewboard https://reviews.apache.org/r/24253/diff/ against branch origin/transactional_messaging Add transactional request definitions to clients package - Key: KAFKA-1541 URL: https://issues.apache.org/jira/browse/KAFKA-1541 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1541.patch, KAFKA-1541.patch Separate jira for this since KAFKA-1522 only adds definitions to the core package. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1541) Add transactional request definitions to clients package
[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1541: - Attachment: KAFKA-1541.patch Add transactional request definitions to clients package - Key: KAFKA-1541 URL: https://issues.apache.org/jira/browse/KAFKA-1541 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1541.patch, KAFKA-1541.patch Separate jira for this since KAFKA-1522 only adds definitions to the core package. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1569) Create tool to test end-to-end correctness when using transactions
[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1569: - Summary: Create tool to test end-to-end correctness when using transactions (was: Creat tool to test end-to-end correctness in transactional mode) Create tool to test end-to-end correctness when using transactions -- Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez Labels: transactions A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1569) Create tool to test correctness of transactions end-to-end
[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1569: - Summary: Create tool to test correctness of transactions end-to-end (was: Create tool to test end-to-end correctness when using transactions) Create tool to test correctness of transactions end-to-end -- Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez Labels: transactions A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Issues getting IntelliJ set up
I did. I actually tried this from a completely clean repo (cloned a new repo from github, changed gradle.properties, ran `gradlew idea`, then imported into IntelliJ) On Mon, Aug 4, 2014 at 12:18 PM, Timothy Chen tnac...@gmail.com wrote: Hi Johnathan, Did you update your scala version before you run gradle idea? Also try cleaning up all the artifacts and try it again, as perhaps your intellij is not picking up the right version and from the right build folder. Tim On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins na...@streamsets.com wrote: Hi, I've been having some issues getting IntelliJ set up...I followed all the instructions on the wiki, and I've successfully imported the project, and run the jar Gradle target successfully. However, when I try to run a test in the IDE, I get a number of errors: /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java Error:(21, 30) java: package kafka.javaapi.producer does not exist Error:(22, 22) java: package kafka.producer does not exist Error:(23, 22) java: package kafka.producer does not exist Error:(24, 19) java: cannot find symbol symbol: class Utils location: package kafka.utils Error:(303, 39) java: cannot find symbol symbol: class KeyedMessage location: class kafka.tools.KafkaMigrationTool.MigrationThread And so on. The two classes that seem to be causing trouble are KafkaMigrationTool and ConsumerConnector. Has anybody run into this? Anyone know how to get around this issue? Thanks a lot, Natty
Review Request 24255: Patch for KAFKA-1569
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24255/ --- Review request for kafka. Bugs: KAFKA-1569 https://issues.apache.org/jira/browse/KAFKA-1569 Repository: kafka Description --- KAFKA-1569; Create tool to test correctness of transactions end to end Diffs - bin/kafka-tx-consumer-test.sh PRE-CREATION bin/kafka-tx-producer-test.sh PRE-CREATION core/src/main/scala/kafka/tools/TransactionalConsumerTest.scala PRE-CREATION core/src/main/scala/kafka/tools/TransactionalProducerTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/24255/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Updated] (KAFKA-1569) Create tool to test correctness of transactions end-to-end
[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1569: - Attachment: KAFKA-1569.patch Create tool to test correctness of transactions end-to-end -- Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1569.patch A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1569) Create tool to test correctness of transactions end-to-end
[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085193#comment-14085193 ] Raul Castro Fernandez commented on KAFKA-1569: -- Created reviewboard https://reviews.apache.org/r/24255/diff/ against branch origin/transactional_messaging Create tool to test correctness of transactions end-to-end -- Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1569.patch A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Newer Zookeeper?
If Kafka installations are missing something(s) by not having or using the latest Zookeeper from a feature or stability perspective that would be something to understand maybe you could help with that Gwen? I know one of the implementations used this Hadoop version http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_releasenotes_hdp_2.1/content/ch_relnotes-hdp-2.1.3-product.html which appears to be using Zk 3.4.5. I will have to check on the other two (someone reminded me we saw this more than twice after I sent the email). I think maybe one of them was CDH but don't recall off the top of my head it was a while ago. A reason why another zookeeper cluster for Kafka vs other software systems (Hadoop, Mesos, etc) is to separate risk of dependent services. One zookeeper cluster can now take down more systems when it goes down (for whatever reason, rogue server/code, upgrade, whatever) and becomes one big single point of failure for everything. If you aren't using zookeeper for anything else that is mission critical it might not matter, it is relative (and have seen this too of course). We have also found deploying zookeeper to Mesos very (very (very))) fruitful for dealing with and managing multiple zookeeper ensembles without any headaches of course you can't do that with the Zookeeper ensemble for Mesos but that goes back to my separation. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 12:36 PM, Gwen Shapira gshap...@cloudera.com wrote: Also, specific Zookeeper 3.4.X version where loss of quorum occurred will help. 3.4.5 fixed some pretty serious issues around hanging. Gwen On Mon, Aug 4, 2014 at 9:29 AM, Gwen Shapira gshap...@cloudera.com wrote: Thanks for the heads-up, Joe. We've been shipping Zookeeper 3.4.X for over two years now (since CDH4.0) and have many production customers. I'll check if there are any known issues with breaking quorum. In any case I will take your comments into account and see if I can arrange for extra testing. Can you share more information about the 3.4.X issues you were seeing? Was there especially large clusters involved? large number of consumers? Also, I'm curious to hear more about the reasons for separate ZK cluster. I can see why you'll want it if you have thousands of consumers, but are there other reasons? Multiple zookeeper installs can be a pain to manage. Gwen On Mon, Aug 4, 2014 at 7:52 AM, Joe Stein joe.st...@stealth.ly wrote: I have heard issues from installations running 3.4.X that I have not heard from installations running 3.3.X (i.e. zk breaking quorum and cluster going down). In none of these cases did I have an opportunity to isolate and reproduce and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was agreed to being a lower risk/cost solution to the problem. Once on 3.3.X the issues didn't happen again. So I can't say for sure if there are issues with running 3.4.X but I would suggest some due diligence in testing and production operation to validate that every case that Kafka requires operates correctly (and over some time). There is a cost to this so some company(s) will have to take that investment and do some cost vs the benefit of moving to 3.4.x. I currently recommend running a separate ZK cluster for Kafka production and not chroot into an existing one except for test/qa/dev. I don't know what others experience is with 3.4.X as I said the issues I have seen could have been coincidence. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
[jira] [Updated] (KAFKA-1570) sbt assembly-package-dependency fails with errors
[
https://issues.apache.org/jira/browse/KAFKA-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuri Nagarin updated KAFKA-1570:
Component/s: (was: website)
build
Description:
xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt assembly-package-dependency
[info] Set current project to kafka-0-8-1-1-src (in build
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
[error] Not a valid command: assembly-package-dependency
[error] Not a valid project ID: assembly-package-dependency
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly-package-dependency (similar: sbt-dependency)
[error] assembly-package-dependency
[error]^
xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt about
[info] Set current project to kafka-0-8-1-1-src (in build
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
[info] This is sbt 0.13.5
[info] The current project is
{file:/home/xnag/Downloads/kafka-0.8.1.1-src/}kafka-0-8-1-1-src 0.1-SNAPSHOT
[info] The current project is built against Scala 2.10.4
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin,
sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4
xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 12.04.4 LTS
Release:12.04
Codename: precise
xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt tasks
[info] Set current project to kafka-0-8-1-1-src (in build
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
This is a list of tasks defined for the current project.
It does not list the scopes the tasks are defined in; use the 'inspect' command
for that.
Tasks produce values. Use the 'show' command to run the task and print the
resulting value.
cleanDeletes files produced by the build, such as generated
sources, compiled classes, and task caches.
compile Compiles sources.
console Starts the Scala interpreter with the project classes on the
classpath.
consoleProject Starts the Scala interpreter with the sbt and the build
definition on the classpath and useful imports.
consoleQuick Starts the Scala interpreter with the project dependencies
on the classpath.
copyResourcesCopies resources to the output directory.
doc Generates API documentation.
package Produces the main artifact, such as a binary jar. This is
typically an alias for the task that actually does the packaging.
packageBin Produces a main artifact, such as a binary jar.
packageDoc Produces a documentation artifact, such as a jar containing
API documentation.
packageSrc Produces a source artifact, such as a jar containing sources
and resources.
publish Publishes artifacts to a repository.
publishLocal Publishes artifacts to the local Ivy repository.
publishM2Publishes artifacts to the local Maven repository.
run Runs a main class, passing along arguments provided on the
command line.
runMain Runs the main class selected by the first argument, passing
the remaining arguments to the main method.
test Executes all tests.
testOnly Executes the tests provided as arguments or all tests if no
arguments are provided.
testQuickExecutes the tests that either failed before, were not run
or whose transitive dependencies changed, among those provided as arguments.
update Resolves and optionally retrieves dependencies, producing a
report.
More tasks may be viewed by increasing verbosity. See 'help tasks'.
was:
https://kafka.apache.org/08/quickstart.html says:
./sbt update
./sbt package
./sbt assembly-package-dependency
but assembly-package-dependency fails and is actually not needed to run the
rest of the code.
Environment: (was: 0.8 Git Revision 731ba90)
Affects Version/s: (was: 0.8.0)
0.8.1.1
sbt assembly-package-dependency fails with errors
-
Key: KAFKA-1570
URL: https://issues.apache.org/jira/browse/KAFKA-1570
Project: Kafka
Issue Type: Bug
Components: build
Affects Versions: 0.8.1.1
Reporter: Xuri Nagarin
xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt assembly-package-dependency
[info] Set current project to kafka-0-8-1-1-src (in build
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
[error] Not a valid command: assembly-package-dependency
[error] Not a valid project ID: assembly-package-dependency
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly-package-dependency (similar: sbt-dependency)
[error] assembly-package-dependency
Review Request 24265: Patch for KAFKA-1524
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24265/ --- Review request for kafka. Bugs: KAFKA-1524 https://issues.apache.org/jira/browse/KAFKA-1524 Repository: kafka Description --- KAFKA-1524; transactional producer Diffs - clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 522881c972ca42ff4dfb6237a2db15b625334d7e clients/src/main/java/org/apache/kafka/clients/producer/AbortTransactionException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/InvalidTransactionStatusException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java c0f1d57e0feb894d9f246058cd0396461afe3225 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 36e8398416036cab84faad1f07159e5adefd8086 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f9de4af426449cceca12a8de9a9f54a6241d28d8 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 1ed3c28b436d28381d9402896e32d16f2586c65e clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java dd0af8aee98abed5d4a0dc50989e37888bb353fe clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionContext.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionControl.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/errors/TransactionCoordinatorNotAvailableException.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/errors/TransactionFailedException.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/record/Compressor.java 0323f5f7032dceb49d820c17a41b78c56591ffc4 clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 clients/src/main/java/org/apache/kafka/common/record/Record.java 10df9fd8d3f4ec8c277650fa7eab269f3ea30d85 clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 93b58d02eac0f8ca28440e3e0ebea28ed3a7673c clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 5489acac6806b3ae5e6d568d401d5a20c86cac05 clients/src/test/java/org/apache/kafka/clients/producer/TransactionContextTest.java PRE-CREATION clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 94a11121e207d5cf94dbc94443a8aa7edf387782 Diff: https://reviews.apache.org/r/24265/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Updated] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1524: - Attachment: KAFKA-1524.patch Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch, KAFKA-1524.patch, KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085352#comment-14085352 ] Raul Castro Fernandez commented on KAFKA-1524: -- Created reviewboard https://reviews.apache.org/r/24265/diff/ against branch origin/transactional_messaging Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch, KAFKA-1524.patch, KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 24267: Patch for KAFKA-1541
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24267/ --- Review request for kafka. Bugs: KAFKA-1541 https://issues.apache.org/jira/browse/KAFKA-1541 Repository: kafka Description --- KAFKA-1541; tx request definitions in client package Diffs - clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 6fe7573973832615976defa37fe0dfbb8f911939 clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 3374bd98be8e565608c4e764ed10afdae383fb6f clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 044b03061802ee5e8ea4f1995fb0988e1a70e9a7 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/TransactionResponse.java PRE-CREATION Diff: https://reviews.apache.org/r/24267/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Commented] (KAFKA-1541) Add transactional request definitions to clients package
[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085369#comment-14085369 ] Raul Castro Fernandez commented on KAFKA-1541: -- Created reviewboard https://reviews.apache.org/r/24267/diff/ against branch origin/transactional_messaging Add transactional request definitions to clients package - Key: KAFKA-1541 URL: https://issues.apache.org/jira/browse/KAFKA-1541 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1541.patch, KAFKA-1541.patch, KAFKA-1541.patch Separate jira for this since KAFKA-1522 only adds definitions to the core package. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1541) Add transactional request definitions to clients package
[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1541: - Attachment: KAFKA-1541.patch Add transactional request definitions to clients package - Key: KAFKA-1541 URL: https://issues.apache.org/jira/browse/KAFKA-1541 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1541.patch, KAFKA-1541.patch, KAFKA-1541.patch Separate jira for this since KAFKA-1522 only adds definitions to the core package. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 24268: Patch for KAFKA-1569
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24268/ --- Review request for kafka. Bugs: KAFKA-1569 https://issues.apache.org/jira/browse/KAFKA-1569 Repository: kafka Description --- KAFKA-1569; end to end correctness test for transactions Diffs - bin/kafka-tx-consumer-test.sh PRE-CREATION bin/kafka-tx-producer-test.sh PRE-CREATION core/src/main/scala/kafka/tools/TransactionalConsumerTest.scala PRE-CREATION core/src/main/scala/kafka/tools/TransactionalProducerTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/24268/diff/ Testing --- Thanks, Raul Castro Fernandez
Re: Issues getting IntelliJ set up
Yep, it looks like you win the game. Managed to get a test running by doing this. Thanks a bunch, Gwen! On Mon, Aug 4, 2014 at 1:57 PM, Gwen Shapira gshap...@cloudera.com wrote: I think I found it :) Go to project settings Pick your module (I picked core) Paths tab And change the paths to: /Users/Natty/apache/kafka-new/core/build/classes/main and /Users/Natty/apache/kafka-new/core/build/classes/test Don't use the project inherited paths, because we have a build dir per module. Gwen On Mon, Aug 4, 2014 at 12:54 PM, Jonathan Natkins na...@streamsets.com wrote: I did. I actually tried this from a completely clean repo (cloned a new repo from github, changed gradle.properties, ran `gradlew idea`, then imported into IntelliJ) On Mon, Aug 4, 2014 at 12:18 PM, Timothy Chen tnac...@gmail.com wrote: Hi Johnathan, Did you update your scala version before you run gradle idea? Also try cleaning up all the artifacts and try it again, as perhaps your intellij is not picking up the right version and from the right build folder. Tim On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins na...@streamsets.com wrote: Hi, I've been having some issues getting IntelliJ set up...I followed all the instructions on the wiki, and I've successfully imported the project, and run the jar Gradle target successfully. However, when I try to run a test in the IDE, I get a number of errors: /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java Error:(21, 30) java: package kafka.javaapi.producer does not exist Error:(22, 22) java: package kafka.producer does not exist Error:(23, 22) java: package kafka.producer does not exist Error:(24, 19) java: cannot find symbol symbol: class Utils location: package kafka.utils Error:(303, 39) java: cannot find symbol symbol: class KeyedMessage location: class kafka.tools.KafkaMigrationTool.MigrationThread And so on. The two classes that seem to be causing trouble are KafkaMigrationTool and ConsumerConnector. Has anybody run into this? Anyone know how to get around this issue? Thanks a lot, Natty
[jira] [Commented] (KAFKA-1569) Create tool to test correctness of transactions end-to-end
[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085385#comment-14085385 ] Raul Castro Fernandez commented on KAFKA-1569: -- Created reviewboard https://reviews.apache.org/r/24268/diff/ against branch origin/transactional_messaging Create tool to test correctness of transactions end-to-end -- Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1569.patch, KAFKA-1569.patch A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1569) Create tool to test correctness of transactions end-to-end
[ https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1569: - Attachment: KAFKA-1569.patch Create tool to test correctness of transactions end-to-end -- Key: KAFKA-1569 URL: https://issues.apache.org/jira/browse/KAFKA-1569 Project: Kafka Issue Type: New Feature Reporter: Raul Castro Fernandez Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1569.patch, KAFKA-1569.patch A producer tool that creates an input file, reads it and sends it to the brokers according to some transaction configuration. And a consumer tool that read data from brokers with transaction boundaries and writes it to a file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1571) MetadataeTest hangs
Jun Rao created KAFKA-1571: -- Summary: MetadataeTest hangs Key: KAFKA-1571 URL: https://issues.apache.org/jira/browse/KAFKA-1571 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Saw the following stacktrace. Thread-47 prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() [0x7fb5af9f8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Thread-46 prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() [0x7fb5afbfa000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Test worker prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() [0x7fb5d4a5f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1186) - locked 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1239) at org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46) -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 24287: Patch for KAFKA-1571
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24287/ --- Review request for kafka. Bugs: KAFKA-1571 https://issues.apache.org/jira/browse/KAFKA-1571 Repository: kafka Description --- Fix the race condition btw the main thread and the asyncFetch threads. Diffs - clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 543304c8bb71d90b4af71b519d830a52595c4885 Diff: https://reviews.apache.org/r/24287/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1571) MetadataeTest hangs
[ https://issues.apache.org/jira/browse/KAFKA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085576#comment-14085576 ] Jun Rao commented on KAFKA-1571: Created reviewboard https://reviews.apache.org/r/24287/ against branch origin/trunk MetadataeTest hangs --- Key: KAFKA-1571 URL: https://issues.apache.org/jira/browse/KAFKA-1571 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Attachments: KAFKA-1571.patch Saw the following stacktrace. Thread-47 prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() [0x7fb5af9f8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Thread-46 prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() [0x7fb5afbfa000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Test worker prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() [0x7fb5d4a5f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1186) - locked 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1239) at org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1571) MetadataeTest hangs
[ https://issues.apache.org/jira/browse/KAFKA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1571: --- Attachment: KAFKA-1571.patch MetadataeTest hangs --- Key: KAFKA-1571 URL: https://issues.apache.org/jira/browse/KAFKA-1571 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Attachments: KAFKA-1571.patch Saw the following stacktrace. Thread-47 prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() [0x7fb5af9f8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Thread-46 prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() [0x7fb5afbfa000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Test worker prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() [0x7fb5d4a5f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1186) - locked 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1239) at org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24287: Patch for KAFKA-1571
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24287/#review49547 --- Ship it! Ship It! - Guozhang Wang On Aug. 5, 2014, 12:39 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24287/ --- (Updated Aug. 5, 2014, 12:39 a.m.) Review request for kafka. Bugs: KAFKA-1571 https://issues.apache.org/jira/browse/KAFKA-1571 Repository: kafka Description --- Fix the race condition btw the main thread and the asyncFetch threads. Diffs - clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 543304c8bb71d90b4af71b519d830a52595c4885 Diff: https://reviews.apache.org/r/24287/diff/ Testing --- Thanks, Jun Rao
[New Feature Request] Ability to Inject Queue Implementation Async Mode
Kafka Version: 0.8.x 1) Ability to define which messages get drooped (least recently instead of most recent in queue) 2) Try Unbounded Queue to find out the Upper Limit without drooping any messages for application (use case Stress test) 3) Priority Blocking Queue ( meaning a single Producer can send messages to multiple topic and I would like to give Delivery Priority to message for particular message for topic) We have use case to support #3 and #1 since we would like to deliver the Application Heartbeat first then any other event within the queue for any topics. To lower TCP connections, we only use one producer for 4 topics but one of the topics has priority for delivery. Please let me know if this is useful feature to have or not. Thanks in advance for great support !! Thanks, Bhavesh P.S. Sorry for asking this question again, but last time there was no conclusion.
Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance
How to achieve uniform distribution of non-keyed messages per topic across all partitions? We have tried to do this uniform distribution across partition using custom partitioning from each producer instance using round robing ( count(messages) % number of partition for topic). This strategy results in very poor performance. So we have switched back to random stickiness that Kafka provide out of box per some interval ( 10 minutes not sure exactly ) per topic. The above strategy results in consumer side lags sometime for some partitions because we have some applications/producers producing more messages for same topic than other servers. Can Kafka provide out of box uniform distribution by using coordination among all producers and rely on measure rate such as # messages per minute or # of bytes produce per minute to achieve uniform distribution and coordinate stickiness of partition among hundreds of producers for same topic ? Thanks, Bhavesh
Re: Newer Zookeeper?
We¹ve started running our test cluster against a Zookeeper 3.4.6 ensemble. So far, we¹ve had no problems with it that were specific to ZK (since we¹re using it for testing trunk version of Kafka, as well as mirror maker, we have plenty of problems with it. Just none that are ZK). We¹re probably going to start rolling that out to our Kafka clusters in the staging environments in the next month or so. That¹s a bigger step, since we treat those clusters more like production (they¹re staging for everyone else, but we¹re infrastructure). -Todd On 8/3/14, 9:56 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
Re: Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance
Bhavesh, take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? Maybe the root cause issue is something else? Even if producers produce more or less than what they are producing you should be able to make it random enough with a partitioner and a key. I don't think you should need more than what is in the FAQ but incase so maybe look into http://en.wikipedia.org/wiki/MurmurHash as another hash option. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 9:12 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: How to achieve uniform distribution of non-keyed messages per topic across all partitions? We have tried to do this uniform distribution across partition using custom partitioning from each producer instance using round robing ( count(messages) % number of partition for topic). This strategy results in very poor performance. So we have switched back to random stickiness that Kafka provide out of box per some interval ( 10 minutes not sure exactly ) per topic. The above strategy results in consumer side lags sometime for some partitions because we have some applications/producers producing more messages for same topic than other servers. Can Kafka provide out of box uniform distribution by using coordination among all producers and rely on measure rate such as # messages per minute or # of bytes produce per minute to achieve uniform distribution and coordinate stickiness of partition among hundreds of producers for same topic ? Thanks, Bhavesh
Re: Newer Zookeeper?
Thanks Todd, very good to know and learn! - Joestein On Mon, Aug 4, 2014 at 9:44 PM, Todd Palino tpal...@linkedin.com.invalid wrote: We¹ve started running our test cluster against a Zookeeper 3.4.6 ensemble. So far, we¹ve had no problems with it that were specific to ZK (since we¹re using it for testing trunk version of Kafka, as well as mirror maker, we have plenty of problems with it. Just none that are ZK). We¹re probably going to start rolling that out to our Kafka clusters in the staging environments in the next month or so. That¹s a bigger step, since we treat those clusters more like production (they¹re staging for everyone else, but we¹re infrastructure). -Todd On 8/3/14, 9:56 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode
Is it possible there is another solution to the problem? I think if you could better describe the problem(s) you are facing and how you are architected some then you may get responses from others that perhaps have faced the same problem with similar architectures ... or maybe folks can chime in with solution(s) to the problem(s). When only being presented with solutions it is hard to say much about if it is problem that folks will have and if this solution will work for them. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 8:52 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Kafka Version: 0.8.x 1) Ability to define which messages get drooped (least recently instead of most recent in queue) 2) Try Unbounded Queue to find out the Upper Limit without drooping any messages for application (use case Stress test) 3) Priority Blocking Queue ( meaning a single Producer can send messages to multiple topic and I would like to give Delivery Priority to message for particular message for topic) We have use case to support #3 and #1 since we would like to deliver the Application Heartbeat first then any other event within the queue for any topics. To lower TCP connections, we only use one producer for 4 topics but one of the topics has priority for delivery. Please let me know if this is useful feature to have or not. Thanks in advance for great support !! Thanks, Bhavesh P.S. Sorry for asking this question again, but last time there was no conclusion.
[jira] [Updated] (KAFKA-1485) Upgrade to Zookeeper 3.4.x
[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1485: - Fix Version/s: 0.8.2 Upgrade to Zookeeper 3.4.x -- Key: KAFKA-1485 URL: https://issues.apache.org/jira/browse/KAFKA-1485 Project: Kafka Issue Type: Wish Affects Versions: 0.8.1.1 Reporter: Machiel Groeneveld Fix For: 0.8.2 I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has been out for 2.5 years and seems to be ready for adoption. In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 0.9.2 release. I can't run both versions in my tests at the same time. The only compile problem I saw was in EmbeddedZookeeper.scala -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1485) Upgrade to Zookeeper 3.4.x
[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1485: - Labels: newbie (was: ) Upgrade to Zookeeper 3.4.x -- Key: KAFKA-1485 URL: https://issues.apache.org/jira/browse/KAFKA-1485 Project: Kafka Issue Type: Wish Affects Versions: 0.8.1.1 Reporter: Machiel Groeneveld Labels: newbie Fix For: 0.8.2 I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has been out for 2.5 years and seems to be ready for adoption. In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 0.9.2 release. I can't run both versions in my tests at the same time. The only compile problem I saw was in EmbeddedZookeeper.scala -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Newer Zookeeper?
I found an already open ticket in regards to this https://issues.apache.org/jira/browse/KAFKA-1485 It also references a conflict with storm upgrading and testing and some other conflicts too. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Aug 4, 2014 at 9:57 PM, Joe Stein joe.st...@stealth.ly wrote: Thanks Todd, very good to know and learn! - Joestein On Mon, Aug 4, 2014 at 9:44 PM, Todd Palino tpal...@linkedin.com.invalid wrote: We¹ve started running our test cluster against a Zookeeper 3.4.6 ensemble. So far, we¹ve had no problems with it that were specific to ZK (since we¹re using it for testing trunk version of Kafka, as well as mirror maker, we have plenty of problems with it. Just none that are ZK). We¹re probably going to start rolling that out to our Kafka clusters in the staging environments in the next month or so. That¹s a bigger step, since we treat those clusters more like production (they¹re staging for everyone else, but we¹re infrastructure). -Todd On 8/3/14, 9:56 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Kafka currently builds against Zookeeper 3.3.4, which is quite old. Perhaps we should move to the more recent 3.4.x branch? I tested the change on my system and the only impact is to EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which was refactored into its own class in 3.4). Here's what the change looks like: https://gist.github.com/gwenshap/d95b36e0bced53cab5bb Gwen
Re: Review Request 23895: Patch for KAFKA-1419
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23895/#review49559 --- Look good overall. We need to change the default scala version in bin/kafka-run-class.sh. Otherwise, the quickstart script will fail. build.gradle https://reviews.apache.org/r/23895/#comment86703 Since you are touching this part, could you also add clients:uploadArchives to uploadArchivesAll? - Jun Rao On Aug. 4, 2014, 2:43 p.m., Ivan Lyutov wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23895/ --- (Updated Aug. 4, 2014, 2:43 p.m.) Review request for kafka. Bugs: KAFKA-1419 https://issues.apache.org/jira/browse/KAFKA-1419 Repository: kafka Description --- KAFKA-1419 - cross build for scala 2.11 - dropped scala 2.8 support - minor bug fixes KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency version - updated scala version to 2.11.2 - added getBuffer to ByteBufferMessageSet classes KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency version - updated scala version to 2.11.2 - added getBuffer to ByteBufferMessageSet classes - removed annotations 2.8 file Diffs - build.gradle a72905df824ba68bed5d5170d18873c23e1782c9 core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala fecee8d5f7b32f483bb1bfc6a5080d589906f9c4 core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 73401c5ff34d08abce22267aa9c4d86632c6fb74 core/src/main/scala/kafka/utils/Annotations_2.8.scala 28269eb037109f7680b9da732e4baa51c9a594b6 core/src/main/scala/kafka/utils/Annotations_2.9+.scala gradle.properties 4827769a3f8e34f0fe7e783eb58e44d4db04859b gradle/buildscript.gradle 225e0a82708bc5f390e5e2c1d4d9a0d06f491b95 gradle/wrapper/gradle-wrapper.properties 610282a699afc89a82203ef0e4e71ecc53761039 scala.gradle ebd21b870c0746aade63248344ab65d9b5baf820 Diff: https://reviews.apache.org/r/23895/diff/ Testing --- Thanks, Ivan Lyutov
[jira] [Resolved] (KAFKA-1571) MetadataeTest hangs
[ https://issues.apache.org/jira/browse/KAFKA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1571. Resolution: Fixed Fix Version/s: 0.8.2 Thanks for the review. Committed to trunk. MetadataeTest hangs --- Key: KAFKA-1571 URL: https://issues.apache.org/jira/browse/KAFKA-1571 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Fix For: 0.8.2 Attachments: KAFKA-1571.patch Saw the following stacktrace. Thread-47 prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() [0x7fb5af9f8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Thread-46 prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() [0x7fb5afbfa000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107) - locked 0x0006b0925e40 (a org.apache.kafka.clients.producer.internals.Metadata) at org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57) Test worker prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() [0x7fb5d4a5f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1186) - locked 0x0006b0926700 (a org.apache.kafka.clients.producer.MetadataTest$1) at java.lang.Thread.join(Thread.java:1239) at org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time
[
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085792#comment-14085792
]
Joe Stein commented on KAFKA-1387:
--
Here is another way to reproduce this issue. I have seen it a few times now
with folks getting going with their clusters.
steps to reproduce. install a 3 node zk ensemble with 3 brokers cluster
e.g.
git clone https://github.com/stealthly/scala-kafka
git checkout -b zkbk3 origin/zkbk3
vagrant up provider=virtualbox
now setup each node in the cluster as you would broker 1,2,3 and the ensemble
e.g.
vagrant ssh zkbkOne
sudo su
cd /vagrant/vagrant/ ./up.sh
vagrant ssh zkbkTwo
sudo su
cd /vagrant/vagrant/ ./up.sh
vagrant ssh zkbkThree
sudo su
cd /vagrant/vagrant/ ./up.sh
start up zookeeper on all 3 nodes
cd /opt/apache/kafka bin/zookeeper-server-start.sh
config/zookeeper.properties 1/tmp/zk.log 2/tmp/zk.log
now, start up broker on node 2 only
cd /opt/apache/kafka bin/kafka-server-start.sh config/server.properties
1/tmp/bk.log 2/tmp/bk.log
ok, now here is where it gets wonky
- change the broker.id int server 3 to = 2
now you need to start up server 1 and 3 (even though it is 2) at the same time
cd /opt/apache/kafka bin/kafka-server-start.sh config/server.properties
1/tmp/bk.log 2/tmp/bk.log
cd /opt/apache/kafka bin/kafka-server-start.sh config/server.properties
1/tmp/bk.log 2/tmp/bk.log
( you can have two tabs, hit enter in one switch to other tab and hit enter is
close enough to same time)
and you get this looping forever
2014-08-05 04:34:38,591] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:2,timestamp:1407212148186}] at /controller a while
back in a different session, hence I will backoff for this node to be deleted
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,598] INFO conflict in /controller data:
{version:1,brokerid:2,timestamp:1407212148186} stored data:
{version:1,brokerid:2,timestamp:1407211911014} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,601] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:2,timestamp:1407212148186}] at /controller a while
back in a different session, hence I will backoff for this node to be deleted
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,610] INFO conflict in /controller data:
{version:1,brokerid:2,timestamp:1407212148186} stored data:
{version:1,brokerid:2,timestamp:1407211911014} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,614] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:2,timestamp:1407212148186}] at /controller a while
back in a different session, hence I will backoff for this node to be deleted
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:56,621] INFO conflict in /controller data:
{version:1,brokerid:2,timestamp:1407212148186} stored data:
{version:1,brokerid:2,timestamp:1407211911014} (kafka.utils.ZkUtils$)
the expected result that you get should be
[2014-08-05 04:07:20,917] INFO conflict in /brokers/ids/2 data:
{jmx_port:-1,timestamp:1407211640900,host:192.168.30.3,version:1,port:9092}
stored data: {jmx_port:-1,timestamp:140721119
9464,host:192.168.30.2,version:1,port:9092} (kafka.utils.ZkUtils$)
[2014-08-05 04:07:20,949] FATAL Fatal error during KafkaServerStable startup.
Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path
/brokers/ids/2. This probably indicates that you either have configured a
brokerid that is already in use, or else you have shutdown
this broker and restarted it faster than the zookeeper timeout so it appears to
be re-registering.
at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-08-05 04:07:20,952] INFO [Kafka Server 2], shutting down
(kafka.server.KafkaServer)
[2014-08-05 04:07:20,954] INFO [Socket Server on Broker 2], Shutting down
(kafka.network.SocketServer)
[2014-08-05 04:07:20,959] INFO [Socket Server on Broker 2], Shutdown completed
(kafka.network.SocketServer)
[2014-08-05 04:07:20,960] INFO [Kafka Request Handler on Broker 2], shutting
down (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:20,992] INFO [Kafka Request Handler on Broker 2], shut down
completely (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:21,263] INFO [Replica Manager on Broker 2]: Shut down
(kafka.server.ReplicaManager)
[2014-08-05 04:07:21,263] INFO [ReplicaFetcherManager on broker 2] shutting
down (kafka.server.ReplicaFetcherManager)
which is what
[jira] [Comment Edited] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time
[
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085792#comment-14085792
]
Joe Stein edited comment on KAFKA-1387 at 8/5/14 4:41 AM:
--
Here is another way to reproduce this issue. I have seen it a few times now
with folks getting going with their clusters.
steps to reproduce. install a 3 node zk ensemble with 3 brokers cluster
e.g.
git clone https://github.com/stealthly/scala-kafka
git checkout -b zkbk3 origin/zkbk3
vagrant up provider=virtualbox
now setup each node in the cluster as you would broker 1,2,3 and the ensemble
e.g.
vagrant ssh zkbkOne
sudo su
cd /vagrant/vagrant/ ./up.sh
vagrant ssh zkbkTwo
sudo su
cd /vagrant/vagrant/ ./up.sh
vagrant ssh zkbkThree
sudo su
cd /vagrant/vagrant/ ./up.sh
start up zookeeper on all 3 nodes
cd /opt/apache/kafka bin/zookeeper-server-start.sh
config/zookeeper.properties 1/tmp/zk.log 2/tmp/zk.log
now, start up broker on node 2 only
cd /opt/apache/kafka bin/kafka-server-start.sh config/server.properties
1/tmp/bk.log 2/tmp/bk.log
ok, now here is where it gets wonky
on server 3 change from broker.id=3 to broker.id=2
now you need to start up server 1 and 3 (even though it is broker.id=2) at the
same time
cd /opt/apache/kafka bin/kafka-server-start.sh config/server.properties
1/tmp/bk.log 2/tmp/bk.log
cd /opt/apache/kafka bin/kafka-server-start.sh config/server.properties
1/tmp/bk.log 2/tmp/bk.log
( you can have two tabs, hit enter in one switch to other tab and hit enter is
close enough to same time)
and you get this looping forever
2014-08-05 04:34:38,591] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:2,timestamp:1407212148186}] at /controller a while
back in a different session, hence I will backoff for this node to be deleted
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,598] INFO conflict in /controller data:
{version:1,brokerid:2,timestamp:1407212148186} stored data:
{version:1,brokerid:2,timestamp:1407211911014} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,601] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:2,timestamp:1407212148186}] at /controller a while
back in a different session, hence I will backoff for this node to be deleted
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,610] INFO conflict in /controller data:
{version:1,brokerid:2,timestamp:1407212148186} stored data:
{version:1,brokerid:2,timestamp:1407211911014} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,614] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:2,timestamp:1407212148186}] at /controller a while
back in a different session, hence I will backoff for this node to be deleted
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:56,621] INFO conflict in /controller data:
{version:1,brokerid:2,timestamp:1407212148186} stored data:
{version:1,brokerid:2,timestamp:1407211911014} (kafka.utils.ZkUtils$)
the expected result that you get should be
[2014-08-05 04:07:20,917] INFO conflict in /brokers/ids/2 data:
{jmx_port:-1,timestamp:1407211640900,host:192.168.30.3,version:1,port:9092}
stored data: {jmx_port:-1,timestamp:140721119
9464,host:192.168.30.2,version:1,port:9092} (kafka.utils.ZkUtils$)
[2014-08-05 04:07:20,949] FATAL Fatal error during KafkaServerStable startup.
Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path
/brokers/ids/2. This probably indicates that you either have configured a
brokerid that is already in use, or else you have shutdown
this broker and restarted it faster than the zookeeper timeout so it appears to
be re-registering.
at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-08-05 04:07:20,952] INFO [Kafka Server 2], shutting down
(kafka.server.KafkaServer)
[2014-08-05 04:07:20,954] INFO [Socket Server on Broker 2], Shutting down
(kafka.network.SocketServer)
[2014-08-05 04:07:20,959] INFO [Socket Server on Broker 2], Shutdown completed
(kafka.network.SocketServer)
[2014-08-05 04:07:20,960] INFO [Kafka Request Handler on Broker 2], shutting
down (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:20,992] INFO [Kafka Request Handler on Broker 2], shut down
completely (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:21,263] INFO [Replica Manager on Broker 2]: Shut down
(kafka.server.ReplicaManager)
[2014-08-05 04:07:21,263] INFO [ReplicaFetcherManager on broker 2]
[jira] [Comment Edited] (KAFKA-1070) Auto-assign node id
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085808#comment-14085808 ] Joe Stein edited comment on KAFKA-1070 at 8/5/14 5:06 AM: -- This sounds like a very cool and VERY useful feature. Excited to use it myself often. I know of a few (10) different clusters that not only use varying sized numbers for their broker.id but do so in what is a seemingly random (but not really when you think about it) way. so in a cluster there may be broker.id that is 1721632 and another 172164875 and another 172162240 . Making your brokers by replacing . in chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy just doing 2288, 2388, 17, 177 (where just the last two octets get used and . is replaced). I am not saying I agree with this approach and I actively advocate away from doing this but in some/many scenarios it is the best/only way to automate their deploys for how things are setup. It is also what seems to make sense when folks are building their automation scripts since they have no other option without doing more than they should be expected to-do (and the IP replace . is so obvious to them, and it is). So, for folks in these cases they would just pick the upper bound to be, lets say 17216255256 and then it would auto assign from there? Is there some better way to go about this where you might have a start increment and and some exclusion list? or a way to see broker.id already had that and not use it? I think a lot of folks would like to get having broker id be more continious and be easier to communicate but the desire to automate everything will outweigh that. We could give them some sanity back with brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster. not crucial and you may have already accounted for the scenario I brought up, but wanted to bring it up as a real use case for how people automate things. it might be better for folks to manually migrate their scripts but not sure they would do it and if they did would have to decommission brokers which in a prod environment could take a few weeks/months. If we let them start at 1 and exclude what they have then they can do it one at a time. After taking down the first broker and bring it back up (empty) it is broker.id=1, and so on (and if they have a 5 they don't have to take it down), etc. was (Author: joestein): This sounds like a very cool and VERY useful feature. Excited to use it myself often. I know of a few (10) different clusters that not only use varying sized numbers for their broker.id but do so in what is a seemingly random (but not really when you think about it) way. so in a cluster there may be broker.id that is 1721632 and another 172164875 and another 172162240 . Making your brokers by replacing . in chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy just doing 2288, 2388, 17, 177 (where just the last two octets get used and . is replaced). I am not saying I agree with this approach and I actively advocate away from doing this but in some/many scenarios it is the best/only way to automate their deploys for how things are setup. It is also what seems to make sense when folks are building their automation scripts since they have no other option without doing more than they should be expected to-do (and the IP replace . is so obvious to them, and it is). So, for folks in these cases they would just pick the upper bound to be, lets say 17216255256 and then it would auto assign from there? Is there some better way to go about this where you might have a start increment and and some exclusion list? or a way to see broker.id already had that and not use it? I think a lot of folks would like to get having broker id be more continious and be easier to communicate but the desire to automate everything will outweigh that. We could give them some sanity back with brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster. not crucial and you may have already accounted for the scenario I brought up, but wanted to bring it up as a real use case for how people automate things. it might be better for folks to manually migrate off what they have and then moving forward in their automation deal with the lower number or something. It is hard to say how folks find creative solutions to common problems without every speaking to each other. I don't know how this will work for them though. Auto-assign node id --- Key: KAFKA-1070 URL: https://issues.apache.org/jira/browse/KAFKA-1070 Project: Kafka Issue Type: Bug Reporter: Jay Kreps Assignee: Sriharsha Chintalapani Labels: usability Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch,
[jira] [Comment Edited] (KAFKA-1070) Auto-assign node id
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085808#comment-14085808 ] Joe Stein edited comment on KAFKA-1070 at 8/5/14 5:08 AM: -- This sounds like a very cool and VERY useful feature. Excited to use it myself often. I know of a few (10) different clusters that not only use varying sized numbers for their broker.id but do so in what is a seemingly random (but not really when you think about it) way. so in a cluster there may be broker.id that is 1721632 and another 172164875 and another 172162240 . Making your brokers by replacing . in chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy just doing 2288, 2388, 17, 177 (where just the last two octets get used and . is replaced). I am not saying I agree with this approach and I actively advocate away from doing this but in some/many scenarios it is the best/only way to automate their deploys for how things are setup. It is also what seems to make sense when folks are building their automation scripts since they have no other option without doing more than they should be expected to-do (and the IP replace . is so obvious to them, and it is). So, for folks in these cases they would just pick the upper bound to be, lets say 17216255256 and then it would auto assign from there? Is there some better way to go about this where you might have a start increment and and some exclusion list? or a way to see broker.id already had that and not use it? I think a lot of folks would like to get having broker id be more continious and be easier to communicate but the desire to automate everything will outweigh that. We could give them some sanity back with brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster. not crucial and you may have already accounted for the scenario I brought up, but wanted to bring it up as a real use case for how people automate things. it might be better for folks to manually migrate their scripts but not sure they would do it and if they did would have to decommission brokers which in a prod environment could take a few weeks/months. If we let them start at 1 and exclude what they have then they can do it one at a time. After taking down the first broker and bring it back up (empty) it is broker.id=1, and so on (and if they have a 5 they don't have to take it down), etc. For new clusters this is a slam dunk and wouldn't want to hold up the feature for existing users that have already decided a work around as I don't know what the intent of this was or not. Some folks might not change knowing broker.id=17216520 sometimes is nice you just login to that box but talking about broker 17216520 over and over again is a pita. was (Author: joestein): This sounds like a very cool and VERY useful feature. Excited to use it myself often. I know of a few (10) different clusters that not only use varying sized numbers for their broker.id but do so in what is a seemingly random (but not really when you think about it) way. so in a cluster there may be broker.id that is 1721632 and another 172164875 and another 172162240 . Making your brokers by replacing . in chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy just doing 2288, 2388, 17, 177 (where just the last two octets get used and . is replaced). I am not saying I agree with this approach and I actively advocate away from doing this but in some/many scenarios it is the best/only way to automate their deploys for how things are setup. It is also what seems to make sense when folks are building their automation scripts since they have no other option without doing more than they should be expected to-do (and the IP replace . is so obvious to them, and it is). So, for folks in these cases they would just pick the upper bound to be, lets say 17216255256 and then it would auto assign from there? Is there some better way to go about this where you might have a start increment and and some exclusion list? or a way to see broker.id already had that and not use it? I think a lot of folks would like to get having broker id be more continious and be easier to communicate but the desire to automate everything will outweigh that. We could give them some sanity back with brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster. not crucial and you may have already accounted for the scenario I brought up, but wanted to bring it up as a real use case for how people automate things. it might be better for folks to manually migrate their scripts but not sure they would do it and if they did would have to decommission brokers which in a prod environment could take a few weeks/months. If we let them start at 1 and exclude what they have then they can do it one at a time. After taking down the first broker and bring it back up (empty)
