[GitHub] kafka pull request: add change.log and check git branch

[darionyaphet]

GitHub user darionyaphet opened a pull request:

https://github.com/apache/kafka/pull/25

add change.log and check git branch



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/darionyaphet/kafka ring-buffer-dev

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/25.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #25


commit 098d08eb287fdd6ba1eca0329ec2c4d2bc796565
Author: darionyaphet 
Date:   2014-04-25T08:10:23Z

add change.log and check git branch




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: add change.log and check git branch

[darionyaphet]

Github user darionyaphet closed the pull request at:

https://github.com/apache/kafka/pull/25


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset

[Alexey Ozeritskiy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980971#comment-13980971
 ] 

Alexey Ozeritskiy commented on KAFKA-1414:
--

What to do with exceptions? Maybe we should use ParArray here?

{code}
  private def loadLogs(dirs: Seq[File]) {
dirs.toParArray.foreach(dir => loadDir(dir))
  }

  private def loadDir(dir: File) {
val recoveryPoints = this.recoveryPointCheckpoints(dir).read
/* load the logs */
val subDirs = dir.listFiles()
if(subDirs != null) {
  val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
  if(cleanShutDownFile.exists())
info("Found clean shutdown file. Skipping recovery for all logs in data 
directory '%s'".format(dir.getAbsolutePath))
  for(dir <- subDirs) {
if(dir.isDirectory) {
  info("Loading log '" + dir.getName + "'")
  val topicPartition = Log.parseTopicPartitionName(dir.getName)
  val config = topicConfigs.getOrElse(topicPartition.topic, 
defaultConfig)
  val log = new Log(dir,
config,
recoveryPoints.getOrElse(topicPartition, 0L),
scheduler,
time)
  val previous = addLogWithLock(topicPartition, log)
  if(previous != null)
throw new IllegalArgumentException("Duplicate log directories 
found: %s, %s!".format(log.dir.getAbsolutePath, previous.dir.getAbsolutePath))
}
  }
  cleanShutDownFile.delete()
}
  }

  private def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log 
= {
logCreationOrDeletionLock synchronized {
  this.logs.put(topicPartition, log)
}
  }
{code}

> Speedup broker startup after hard reset
> ---
>
> Key: KAFKA-1414
> URL: https://issues.apache.org/jira/browse/KAFKA-1414
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1
>Reporter: Dmitry Bugaychenko
>Assignee: Jay Kreps
>
> After hard reset due to power failure broker takes way too much time 
> recovering unflushed segments in a single thread. This could be easiliy 
> improved launching multiple threads (one per data dirrectory, assuming that 
> typically each data directory is on a dedicated drive). Localy we trie this 
> simple patch to LogManager.loadLogs and it seems to work, however I'm too new 
> to scala, so do not take it literally:
> {code}
>   /**
>* Recover and load all logs in the given data directories
>*/
>   private def loadLogs(dirs: Seq[File]) {
> val threads : Array[Thread] = new Array[Thread](dirs.size)
> var i: Int = 0
> val me = this
> for(dir <- dirs) {
>   val thread = new Thread( new Runnable {
> def run()
> {
>   val recoveryPoints = me.recoveryPointCheckpoints(dir).read
>   /* load the logs */
>   val subDirs = dir.listFiles()
>   if(subDirs != null) {
> val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
> if(cleanShutDownFile.exists())
>   info("Found clean shutdown file. Skipping recovery for all logs 
> in data directory '%s'".format(dir.getAbsolutePath))
> for(dir <- subDirs) {
>   if(dir.isDirectory) {
> info("Loading log '" + dir.getName + "'")
> val topicPartition = Log.parseTopicPartitionName(dir.getName)
> val config = topicConfigs.getOrElse(topicPartition.topic, 
> defaultConfig)
> val log = new Log(dir,
>   config,
>   recoveryPoints.getOrElse(topicPartition, 0L),
>   scheduler,
>   time)
> val previous = addLogWithLock(topicPartition, log)
> if(previous != null)
>   throw new IllegalArgumentException("Duplicate log 
> directories found: %s, %s!".format(log.dir.getAbsolutePath, 
> previous.dir.getAbsolutePath))
>   }
> }
> cleanShutDownFile.delete()
>   }
> }
>   })
>   thread.start()
>   threads(i) = thread
>   i = i + 1
> }
> for(thread <- threads) {
>   thread.join()
> }
>   }
>   def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log = {
> logCreationOrDeletionLock synchronized {
>   this.logs.put(topicPartition, log)
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset

[Dmitry Bugaychenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981025#comment-13981025
 ] 

Dmitry Bugaychenko commented on KAFKA-1414:
---

Actually similar improvement is needed for shutdown to - on a 20-disk with no 
forced flush during processing it could take more than 10 minutes to shutdown 
and flush logs in a single thread. We end up with the following workaround:

{code}
  def shutdown() {
info("Shutting down.")
try {
  // stop the cleaner first
  if (cleaner != null) {
Utils.swallow(cleaner.shutdown())
  }

  // Span single shutdown thread for each data dir.
  val threadCounter : AtomicInteger = new AtomicInteger()

  // TODO: This must be configurable
  val semaphore : Semaphore = new Semaphore(8)

  val threads : ArrayBuffer[Thread] = new ArrayBuffer[Thread]()
  threads ++= logDirs.map(
x => {
  val thread : Thread = new Thread(new Runnable {
override def run(): Unit = {
  threadCounter.incrementAndGet()

  semaphore.acquire()
  try {
val parent : String = x.toString
val thisDirLogs : Seq[(TopicAndPartition,Log)] = 
logs.filter(_._2.dir.toString.startsWith(parent)).toSeq

// flush the logs to ensure latest possible recovery point
info("Flushing logs at " + parent)
thisDirLogs.foreach(_._2.flush())
info("Closing logs at " + parent)
// close the logs
thisDirLogs.foreach(_._2.close())

// update the last flush point
info("Updating recovery points " + parent)
recoveryPointCheckpoints(x).write(
  thisDirLogs.groupBy(_._1).mapValues(_.head._2.recoveryPoint))

// mark that the shutdown was clean by creating the clean 
shutdown marker file
info("Writing clean shutdown marker " + parent)
Utils.swallow(new File(parent, 
Log.CleanShutdownFile).createNewFile())
  } finally {
semaphore.release()
threadCounter.decrementAndGet()
  }
}
  })

  thread.start()
  thread
})

  // Wait them all
  threads.foreach(_.join())

  // Check that all threads ended
  if (threadCounter.get() > 0) {
error("Not all shut down threads ended.")
  }
} finally {
  // regardless of whether the close succeeded, we need to unlock the data 
directories
  dirLocks.foreach(_.destroy())
}
info("Shutdown complete.")
  }
{code}

Howevere, there are evidences that running too many threads flushing at 
shutdown can cause JVM to terminate due to native out of memory:
{code}
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 4088 bytes for 
AllocateHeap
JVM exited unexpectedly while stopping the application
{code}

> Speedup broker startup after hard reset
> ---
>
> Key: KAFKA-1414
> URL: https://issues.apache.org/jira/browse/KAFKA-1414
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1
>Reporter: Dmitry Bugaychenko
>Assignee: Jay Kreps
>
> After hard reset due to power failure broker takes way too much time 
> recovering unflushed segments in a single thread. This could be easiliy 
> improved launching multiple threads (one per data dirrectory, assuming that 
> typically each data directory is on a dedicated drive). Localy we trie this 
> simple patch to LogManager.loadLogs and it seems to work, however I'm too new 
> to scala, so do not take it literally:
> {code}
>   /**
>* Recover and load all logs in the given data directories
>*/
>   private def loadLogs(dirs: Seq[File]) {
> val threads : Array[Thread] = new Array[Thread](dirs.size)
> var i: Int = 0
> val me = this
> for(dir <- dirs) {
>   val thread = new Thread( new Runnable {
> def run()
> {
>   val recoveryPoints = me.recoveryPointCheckpoints(dir).read
>   /* load the logs */
>   val subDirs = dir.listFiles()
>   if(subDirs != null) {
> val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
> if(cleanShutDownFile.exists())
>   info("Found clean shutdown file. Skipping recovery for all logs 
> in data directory '%s'".format(dir.getAbsolutePath))
> for(dir <- subDirs) {
>   if(dir.isDirectory) {
> info("Loading log '" + dir.getName + "'")
> val topicPartition = Log.parseTopicPartitionName(dir.getName)
> val config = topicConfigs.getOrElse(topic

[jira] [Created] (KAFKA-1424) transient unit test failure in testSendWithDeadBroker

[Jun Rao (JIRA)]

Jun Rao created KAFKA-1424:
--

 Summary: transient unit test failure in testSendWithDeadBroker
 Key: KAFKA-1424
 URL: https://issues.apache.org/jira/browse/KAFKA-1424
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Jun Rao


Saw the following transient unit test failure.

kafka.producer.ProducerTest > testSendWithDeadBroker FAILED
java.lang.AssertionError: Message set should have 1 message
at org.junit.Assert.fail(Assert.java:69)
at org.junit.Assert.assertTrue(Assert.java:32)
at 
kafka.producer.ProducerTest.testSendWithDeadBroker(ProducerTest.scala:245)




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20713: Patch for KAFKA-1424

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20713/
---

Review request for kafka.


Bugs: KAFKA-1424
https://issues.apache.org/jira/browse/KAFKA-1424


Repository: kafka


Description
---

Wait metadata is propagated before fetching.


Diffs
-

  core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
dc6a5ea4abcdff370f3b61c774bbde4e4157f412 

Diff: https://reviews.apache.org/r/20713/diff/


Testing
---


Thanks,

Jun Rao

[jira] [Updated] (KAFKA-1424) transient unit test failure in testSendWithDeadBroker

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1424:
---

Attachment: KAFKA-1424.patch

> transient unit test failure in testSendWithDeadBroker
> -
>
> Key: KAFKA-1424
> URL: https://issues.apache.org/jira/browse/KAFKA-1424
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Attachments: KAFKA-1424.patch
>
>
> Saw the following transient unit test failure.
> kafka.producer.ProducerTest > testSendWithDeadBroker FAILED
> java.lang.AssertionError: Message set should have 1 message
> at org.junit.Assert.fail(Assert.java:69)
> at org.junit.Assert.assertTrue(Assert.java:32)
> at 
> kafka.producer.ProducerTest.testSendWithDeadBroker(ProducerTest.scala:245)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1424) transient unit test failure in testSendWithDeadBroker

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981080#comment-13981080
 ] 

Jun Rao commented on KAFKA-1424:


Created reviewboard https://reviews.apache.org/r/20713/
 against branch origin/trunk

> transient unit test failure in testSendWithDeadBroker
> -
>
> Key: KAFKA-1424
> URL: https://issues.apache.org/jira/browse/KAFKA-1424
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Attachments: KAFKA-1424.patch
>
>
> Saw the following transient unit test failure.
> kafka.producer.ProducerTest > testSendWithDeadBroker FAILED
> java.lang.AssertionError: Message set should have 1 message
> at org.junit.Assert.fail(Assert.java:69)
> at org.junit.Assert.assertTrue(Assert.java:32)
> at 
> kafka.producer.ProducerTest.testSendWithDeadBroker(ProducerTest.scala:245)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20713: Patch for KAFKA-1424

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20713/#review41473
---

Ship it!


Ship It!

- Timothy Chen


On April 25, 2014, 2:54 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20713/
> ---
> 
> (Updated April 25, 2014, 2:54 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1424
> https://issues.apache.org/jira/browse/KAFKA-1424
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Wait metadata is propagated before fetching.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
> dc6a5ea4abcdff370f3b61c774bbde4e4157f412 
> 
> Diff: https://reviews.apache.org/r/20713/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 20380: Fix KAFKA-1395: Address Jun's comments

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20380/#review41472
---


 Great cleanup patch! Some minor comments.

1. TestUtils.createProducer: Could we keep the props parameter and treat it as 
the list of properties that we want to override?

2. Unused imports in ServerShutdownTest and LogRecoveryTest.

- Jun Rao


On April 24, 2014, 11:22 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20380/
> ---
> 
> (Updated April 24, 2014, 11:22 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1395
> https://issues.apache.org/jira/browse/KAFKA-1395
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> minor fix 2
> 
> 
> minor fix
> 
> 
> Move create server/producer and create topic to setUp in LogRecoveryTest
> 
> 
> Address Jun's comments
> 
> 
> Incoporate Neha's comments
> 
> 
> remove waitUntilLeaderIsElectedOrChanged
> 
> 
> KAFKA-1395.v1
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/integration/AutoOffsetResetTest.scala 
> 7125ec9c03e634d7625a315a944eaf5088074eed 
>   core/src/test/scala/unit/kafka/integration/FetcherTest.scala 
> 4075068a80d6e71a3446a579112541d5b4892854 
>   
> core/src/test/scala/unit/kafka/integration/ProducerConsumerTestHarness.scala 
> 731ee59e67e302f4cb1da12b1d4ae7ff59010aea 
>   core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
> d1d969e6c7a5183fa9877fcd12cf49de0cead62a 
>   
> core/src/test/scala/unit/kafka/javaapi/consumer/ZookeeperConsumerConnectorTest.scala
>  16e716440099d62198af3bbadaf5b7d7bc496735 
>   core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
> 7a0ef6faeaa419553f912c8a6d44099818e19833 
>   core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
> 481a400067ec97ed97a2bbd8506b34960a5ec000 
>   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
> addd11ab8be2a1c5de0e43f006de1087032fa6d8 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 130b6be8414268b9a992514ecbc7561872ccaca7 
> 
> Diff: https://reviews.apache.org/r/20380/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Resolved] (KAFKA-1424) transient unit test failure in testSendWithDeadBroker

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1424.


   Resolution: Fixed
Fix Version/s: 0.8.2

Thanks for the review. Committed to trunk.

> transient unit test failure in testSendWithDeadBroker
> -
>
> Key: KAFKA-1424
> URL: https://issues.apache.org/jira/browse/KAFKA-1424
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.2
>
> Attachments: KAFKA-1424.patch
>
>
> Saw the following transient unit test failure.
> kafka.producer.ProducerTest > testSendWithDeadBroker FAILED
> java.lang.AssertionError: Message set should have 1 message
> at org.junit.Assert.fail(Assert.java:69)
> at org.junit.Assert.assertTrue(Assert.java:32)
> at 
> kafka.producer.ProducerTest.testSendWithDeadBroker(ProducerTest.scala:245)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Kafka 0.8.1.1 Release Candidate 1

[Jun Rao]

+1. Verified quick start.

Thanks,

Jun


On Tue, Apr 22, 2014 at 4:18 PM, Joe Stein  wrote:

> This is the first candidate for release of Apache Kafka 0.8.1.1
>
> Release Notes for the 0.8.1.1 release
>
> https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, April 25th, 4pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1.1 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=34237371b662f617dd80c75cd6282f97e047ef56
>
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

Re: Review Request 20695: Patch for KAFKA-1392

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20695/#review41476
---

Ship it!


Ship It!

- Guozhang Wang


On April 25, 2014, 12:59 a.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20695/
> ---
> 
> (Updated April 25, 2014, 12:59 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1392
> https://issues.apache.org/jira/browse/KAFKA-1392
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> 4. Add failure messages in all calls to waitUntilTrue.
> 
> 
> 3. Address Joel's comments.
> 
> 
> 1. Throw exception in waitUntilTrue() if timeout is reached. 2. Fixed unit 
> test failure because of 1 in testResumePartitionReassignmentThatWasCompleted 
> (a. should wait until reassignment path is deleted; b. under replicated check 
> should be wrapped in waitUntilTrue).
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/AdminTest.scala 
> 3a6c5ffa7e27d871a5cfb250c85b0609644b7cf9 
>   core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
> 6d489adcb307d4bffc5d36ff65ff1b16f474d4e4 
>   core/src/test/scala/unit/kafka/network/SocketServerTest.scala 
> c3b1ac445cf496dede87267a9601a466e453fecc 
>   core/src/test/scala/unit/kafka/producer/ProducerTest.scala 
> dc6a5ea4abcdff370f3b61c774bbde4e4157f412 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> b278bb6eaa49d8dbe3b70463fdd4cc854f4796fe 
>   core/src/test/scala/unit/kafka/server/LogOffsetTest.scala 
> 3fb08e69446aba2363255d9853a02c455ac46e11 
>   core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
> 7a0ef6faeaa419553f912c8a6d44099818e19833 
>   core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
> 481a400067ec97ed97a2bbd8506b34960a5ec000 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 130b6be8414268b9a992514ecbc7561872ccaca7 
> 
> Diff: https://reviews.apache.org/r/20695/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

[jira] [Updated] (KAFKA-1308) Publish jar of test utilities to Maven

[Chris Campbell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Campbell updated KAFKA-1308:
--

Attachment: KAFKA-1308.patch

> Publish jar of test utilities to Maven
> --
>
> Key: KAFKA-1308
> URL: https://issues.apache.org/jira/browse/KAFKA-1308
> Project: Kafka
>  Issue Type: Wish
>Reporter: Martin Kleppmann
> Attachments: KAFKA-1308.patch
>
>
> For projects that use Kafka, and want to write tests that exercise Kafka (in 
> our case, Samza), it's useful to have access to Kafka's test utility classes 
> such as kafka.zk.EmbeddedZookeeper and kafka.utils.TestUtils. We can use 
> {{./gradlew testJar}} to build jar files that contain those classes, but as 
> far as I know, these are currently not made available in a binary release.
> At the moment, we have to check those kafka*-test.jar files into the Samza 
> repository. To avoid that, would it be possible to publish those jars of 
> tests to Maven, so that they fit into the normal dependency management?
> Or perhaps, if publishing the tests themselves is not appropriate, we could 
> move the test utilities into a separate module that is published, and make 
> the tests depend on that module?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1308) Publish jar of test utilities to Maven

[Chris Campbell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Campbell updated KAFKA-1308:
--

Affects Version/s: 0.8.1
   Status: Patch Available  (was: Open)

I'm attaching a small patch (see KAFKA-1308.patch) that adds a trivial 
testutils project containing only the kafka.utils.TestUtils class.  I'm not too 
experienced with Gradle so I don't know if this is the best approach (see the 
TODO), but this seems to get the job done.  Verified with our in-house Maven 
repo.


> Publish jar of test utilities to Maven
> --
>
> Key: KAFKA-1308
> URL: https://issues.apache.org/jira/browse/KAFKA-1308
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1
>Reporter: Martin Kleppmann
> Attachments: KAFKA-1308.patch
>
>
> For projects that use Kafka, and want to write tests that exercise Kafka (in 
> our case, Samza), it's useful to have access to Kafka's test utility classes 
> such as kafka.zk.EmbeddedZookeeper and kafka.utils.TestUtils. We can use 
> {{./gradlew testJar}} to build jar files that contain those classes, but as 
> far as I know, these are currently not made available in a binary release.
> At the moment, we have to check those kafka*-test.jar files into the Samza 
> repository. To avoid that, would it be possible to publish those jars of 
> tests to Maven, so that they fit into the normal dependency management?
> Or perhaps, if publishing the tests themselves is not appropriate, we could 
> move the test utilities into a separate module that is published, and make 
> the tests depend on that module?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20616: Fix KAFKA-1409: Address Jun and Neha's comments

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20616/
---

(Updated April 25, 2014, 5:22 p.m.)


Review request for kafka.


Summary (updated)
-

Fix KAFKA-1409: Address Jun and Neha's comments


Bugs: KAFKA-1409
https://issues.apache.org/jira/browse/KAFKA-1409


Repository: kafka


Description (updated)
---

Address Neha's comments


Update comments


Incorporate Jun's comments round three


Fix KAFKA-1409: Address Jun's comments round two


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 
c08eab0bc35aa2b0caa0dfc1987b4face4315d3b 
  core/src/main/scala/kafka/log/Log.scala 
46df8d99d977a3b010a9b9f4698187fa9bfb2498 
  core/src/main/scala/kafka/message/MessageSet.scala 
a1b5c63b4d9deaa2556427d995c41841838cf581 
  core/src/main/scala/kafka/server/KafkaApis.scala 
bb0359d933356b8fcc2c8df22ce6ea9cb838d6a2 
  core/src/main/scala/kafka/server/KafkaRequestHandler.scala 
f11f6e2c82b016f4dd3e92466ae1d2ede19feb30 

Diff: https://reviews.apache.org/r/20616/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1409) oversized messages can slow down the brokers

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981269#comment-13981269
 ] 

Guozhang Wang commented on KAFKA-1409:
--

Updated reviewboard https://reviews.apache.org/r/20616/
 against branch origin/trunk

> oversized messages can slow down the brokers
> 
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1409.patch, KAFKA-1409_2014-04-23_14:10:03.patch, 
> KAFKA-1409_2014-04-23_14:25:48.patch, KAFKA-1409_2014-04-23_15:20:43.patch, 
> KAFKA-1409_2014-04-24_16:26:06.patch, KAFKA-1409_2014-04-25_10:22:54.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to 
> assign new offsets) before validating the message size (we have to do the 
> validation after recompression since the message size could change). So, it 
> can spend many secs to decompress/recompress an oversized message, only to be 
> dropped later. While this was happening, a request thread was tied up, which 
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per 
> partition). So, if the producer is slow in appending the log, it will block 
> other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/
---

Review request for kafka.


Bugs: KAFKA-1384
https://issues.apache.org/jira/browse/KAFKA-1384


Repository: kafka


Description
---

KAFKA-1384: Logging kafka state metric


Diffs
-

  core/src/main/scala/kafka/controller/KafkaController.scala 
933de9dd324c7086efe6aa610335ef370d9e9c12 
  core/src/main/scala/kafka/log/Log.scala 
46df8d99d977a3b010a9b9f4698187fa9bfb2498 
  core/src/main/scala/kafka/log/LogManager.scala 
ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
  core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaServer.scala 
c208f83bed7fb91f07fae42f2b66892e6d46fecc 
  core/src/main/scala/kafka/server/KafkaServerStartable.scala 
acda52b801714bcc182edc0ced925f0e4b493fc1 

Diff: https://reviews.apache.org/r/20718/diff/


Testing
---


Thanks,

Timothy Chen

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981272#comment-13981272
 ] 

Timothy Chen commented on KAFKA-1384:
-

I'm planning just to expose it via JMX, as we don't have a need to write it to 
zookeeper as other brokers doesn't have a need to query it.

Less contention to ZK as well, you have anything in mind why you thought 
putting it in ZK would be good?


> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Timothy Chen
>Assignee: Timothy Chen
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981275#comment-13981275
 ] 

Timothy Chen commented on KAFKA-1384:
-

Created reviewboard https://reviews.apache.org/r/20718/
 against branch origin/trunk

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: Kafka-trunk #175

[Apache Jenkins Server]

See 

Changes:

[junrao] kafka-1424; transient unit test failure in testSendWithDeadBroker; 
patched by Jun Rao; reviewed by Timothy Chen

--
[...truncated 296 lines...]

kafka.log.LogManagerTest > testGetNonExistentLog PASSED

kafka.log.LogManagerTest > testCleanupExpiredSegments PASSED

kafka.log.LogManagerTest > testCleanupSegmentsToMaintainSize PASSED

kafka.log.LogManagerTest > testTimeBasedFlush PASSED

kafka.log.LogManagerTest > testLeastLoadedAssignment PASSED

kafka.log.LogManagerTest > testTwoLogManagersUsingSameDirFails PASSED

kafka.log.LogManagerTest > testCheckpointRecoveryPoints PASSED

kafka.log.LogManagerTest > testRecoveryDirectoryMappingWithTrailingSlash PASSED

kafka.log.LogManagerTest > testRecoveryDirectoryMappingWithRelativeDirectory 
PASSED

kafka.admin.AddPartitionsTest > testTopicDoesNotExist PASSED

kafka.admin.AddPartitionsTest > testWrongReplicaCount PASSED

kafka.admin.AddPartitionsTest > testIncrementPartitions PASSED

kafka.admin.AddPartitionsTest > testManualAssignmentOfReplicas PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.admin.DeleteTopicTest > testFake PASSED

kafka.admin.AdminTest > testReplicaAssignment PASSED

kafka.admin.AdminTest > testManualReplicaAssignment PASSED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderInNewReplicas PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderNotInNewReplicas 
PASSED

kafka.admin.AdminTest > testPartitionReassignmentNonOverlappingReplicas PASSED

kafka.admin.AdminTest > testReassigningNonExistingPartition PASSED

kafka.admin.AdminTest > testResumePartitionReassignmentThatWasCompleted PASSED

kafka.admin.AdminTest > testPreferredReplicaJsonData PASSED

kafka.admin.AdminTest > testBasicPreferredReplicaElection PASSED

kafka.admin.AdminTest > testShutdownBroker PASSED

kafka.admin.AdminTest > testTopicConfigChange PASSED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testInLock PASSED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization PASSED

kafka.api.ApiUtilsTest > testShortStringNonASCII PASSED

kafka.api.ApiUtilsTest > testShortStringASCII PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.api.ProducerFailureHandlingTest > testNonExistTopic PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.api.ProducerFailureHandlingTest > testNoResponse PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1] PASSED

kafka.api.test.ProducerSendTest > testSendOffset PASSED

kafka.api.test.ProducerSendTest > testClose PASSED

kafka.api.test.ProducerSendTest > testSendToPartition PASSED

kafka.api.test.ProducerSendTest > testAutoCreateTopic PASSED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.ConsumerIteratorTest > 
testConsumerIteratorDeduplicationDeepIterator PASSED

kafka.consumer.ConsumerIteratorTest > testConsumerIteratorDecodingFailure PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompression PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerDecoder PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.

Re: Review Request 20616: Fix KAFKA-1409: Address Jun and Neha's comments

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20616/#review41490
---



core/src/main/scala/kafka/log/Log.scala


entrySNize?



core/src/main/scala/kafka/log/Log.scala


entrySNnnnNize?


- Jun Rao


On April 25, 2014, 5:22 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20616/
> ---
> 
> (Updated April 25, 2014, 5:22 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1409
> https://issues.apache.org/jira/browse/KAFKA-1409
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Address Neha's comments
> 
> 
> Update comments
> 
> 
> Incorporate Jun's comments round three
> 
> 
> Fix KAFKA-1409: Address Jun's comments round two
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> c08eab0bc35aa2b0caa0dfc1987b4face4315d3b 
>   core/src/main/scala/kafka/log/Log.scala 
> 46df8d99d977a3b010a9b9f4698187fa9bfb2498 
>   core/src/main/scala/kafka/message/MessageSet.scala 
> a1b5c63b4d9deaa2556427d995c41841838cf581 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> bb0359d933356b8fcc2c8df22ce6ea9cb838d6a2 
>   core/src/main/scala/kafka/server/KafkaRequestHandler.scala 
> f11f6e2c82b016f4dd3e92466ae1d2ede19feb30 
> 
> Diff: https://reviews.apache.org/r/20616/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Updated] (KAFKA-1409) oversized messages can slow down the brokers

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1409:
-

Attachment: KAFKA-1409_2014-04-25_10:22:54.patch

> oversized messages can slow down the brokers
> 
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1409.patch, KAFKA-1409_2014-04-23_14:10:03.patch, 
> KAFKA-1409_2014-04-23_14:25:48.patch, KAFKA-1409_2014-04-23_15:20:43.patch, 
> KAFKA-1409_2014-04-24_16:26:06.patch, KAFKA-1409_2014-04-25_10:22:54.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to 
> assign new offsets) before validating the message size (we have to do the 
> validation after recompression since the message size could change). So, it 
> can spend many secs to decompress/recompress an oversized message, only to be 
> dropped later. While this was happening, a request thread was tied up, which 
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per 
> partition). So, if the producer is slow in appending the log, it will block 
> other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1384) Log Broker state

[Timothy Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Chen updated KAFKA-1384:


Attachment: KAFKA-1384.patch

> Log Broker state 
> -
>
> Key: KAFKA-1384
> URL: https://issues.apache.org/jira/browse/KAFKA-1384
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Timothy Chen
>Assignee: Timothy Chen
> Attachments: KAFKA-1384.patch
>
>
> Currently we don't have visibility into what state the broker is currently 
> in, ie: Startup -> Running -> Waiting Controlled shutdown -> Shutting down
> So without knowing what state the broker it is it's hard to figure out what 
> the current broker is performing.
> This ticket is to add a new metric to expose the current broker state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20616: Fix KAFKA-1409: Address Jun and Neha's comments

[Guozhang Wang]

> On April 25, 2014, 5:32 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/log/Log.scala, lines 274-275
> > 
> >
> > entrySNnnnNize?

Never search for classes while doing "git commit"  sorry about that.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20616/#review41490
---


On April 25, 2014, 5:22 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20616/
> ---
> 
> (Updated April 25, 2014, 5:22 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1409
> https://issues.apache.org/jira/browse/KAFKA-1409
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Address Neha's comments
> 
> 
> Update comments
> 
> 
> Incorporate Jun's comments round three
> 
> 
> Fix KAFKA-1409: Address Jun's comments round two
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> c08eab0bc35aa2b0caa0dfc1987b4face4315d3b 
>   core/src/main/scala/kafka/log/Log.scala 
> 46df8d99d977a3b010a9b9f4698187fa9bfb2498 
>   core/src/main/scala/kafka/message/MessageSet.scala 
> a1b5c63b4d9deaa2556427d995c41841838cf581 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> bb0359d933356b8fcc2c8df22ce6ea9cb838d6a2 
>   core/src/main/scala/kafka/server/KafkaRequestHandler.scala 
> f11f6e2c82b016f4dd3e92466ae1d2ede19feb30 
> 
> Diff: https://reviews.apache.org/r/20616/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Review Request 20616: Fix KAFKA-1409: Address Jun and Neha's comments

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20616/
---

(Updated April 25, 2014, 5:44 p.m.)


Review request for kafka.


Bugs: KAFKA-1409
https://issues.apache.org/jira/browse/KAFKA-1409


Repository: kafka


Description (updated)
---

typo fix


Address Neha's comments


Update comments


Incorporate Jun's comments round three


Fix KAFKA-1409: Address Jun's comments round two


Diffs (updated)
-

  core/src/main/scala/kafka/cluster/Partition.scala 
c08eab0bc35aa2b0caa0dfc1987b4face4315d3b 
  core/src/main/scala/kafka/log/Log.scala 
46df8d99d977a3b010a9b9f4698187fa9bfb2498 
  core/src/main/scala/kafka/message/MessageSet.scala 
a1b5c63b4d9deaa2556427d995c41841838cf581 
  core/src/main/scala/kafka/server/KafkaApis.scala 
bb0359d933356b8fcc2c8df22ce6ea9cb838d6a2 
  core/src/main/scala/kafka/server/KafkaRequestHandler.scala 
f11f6e2c82b016f4dd3e92466ae1d2ede19feb30 

Diff: https://reviews.apache.org/r/20616/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1409) oversized messages can slow down the brokers

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981305#comment-13981305
 ] 

Guozhang Wang commented on KAFKA-1409:
--

Updated reviewboard https://reviews.apache.org/r/20616/
 against branch origin/trunk

> oversized messages can slow down the brokers
> 
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1409.patch, KAFKA-1409_2014-04-23_14:10:03.patch, 
> KAFKA-1409_2014-04-23_14:25:48.patch, KAFKA-1409_2014-04-23_15:20:43.patch, 
> KAFKA-1409_2014-04-24_16:26:06.patch, KAFKA-1409_2014-04-25_10:22:54.patch, 
> KAFKA-1409_2014-04-25_10:44:27.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to 
> assign new offsets) before validating the message size (we have to do the 
> validation after recompression since the message size could change). So, it 
> can spend many secs to decompress/recompress an oversized message, only to be 
> dropped later. While this was happening, a request thread was tied up, which 
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per 
> partition). So, if the producer is slow in appending the log, it will block 
> other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1409) oversized messages can slow down the brokers

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1409:
-

Attachment: KAFKA-1409_2014-04-25_10:44:27.patch

> oversized messages can slow down the brokers
> 
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1409.patch, KAFKA-1409_2014-04-23_14:10:03.patch, 
> KAFKA-1409_2014-04-23_14:25:48.patch, KAFKA-1409_2014-04-23_15:20:43.patch, 
> KAFKA-1409_2014-04-24_16:26:06.patch, KAFKA-1409_2014-04-25_10:22:54.patch, 
> KAFKA-1409_2014-04-25_10:44:27.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to 
> assign new offsets) before validating the message size (we have to do the 
> validation after recompression since the message size could change). So, it 
> can spend many secs to decompress/recompress an oversized message, only to be 
> dropped later. While this was happening, a request thread was tied up, which 
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per 
> partition). So, if the producer is slow in appending the log, it will block 
> other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Kafka 0.8.1.1 Release Candidate 1

[Neha Narkhede]

+1. Verified quick start.

Thanks,
Neha


On Fri, Apr 25, 2014 at 8:39 AM, Jun Rao  wrote:

> +1. Verified quick start.
>
> Thanks,
>
> Jun
>
>
> On Tue, Apr 22, 2014 at 4:18 PM, Joe Stein  wrote:
>
> > This is the first candidate for release of Apache Kafka 0.8.1.1
> >
> > Release Notes for the 0.8.1.1 release
> >
> >
> https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Friday, April 25th, 4pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5, sha1
> > and sha2 (SHA256) checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1.1 tag
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=34237371b662f617dd80c75cd6282f97e047ef56
> >
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
>

Re: Review Request 20380: Fix KAFKA-1395: Address Jun's comments

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20380/
---

(Updated April 25, 2014, 6:03 p.m.)


Review request for kafka.


Bugs: KAFKA-1395
https://issues.apache.org/jira/browse/KAFKA-1395


Repository: kafka


Description (updated)
---

Address Jun's comments round two


minor fix 2


minor fix


Move create server/producer and create topic to setUp in LogRecoveryTest


Address Jun's comments


Incoporate Neha's comments


remove waitUntilLeaderIsElectedOrChanged


KAFKA-1395.v1


Diffs (updated)
-

  core/src/test/scala/unit/kafka/integration/AutoOffsetResetTest.scala 
7125ec9c03e634d7625a315a944eaf5088074eed 
  core/src/test/scala/unit/kafka/integration/FetcherTest.scala 
4075068a80d6e71a3446a579112541d5b4892854 
  core/src/test/scala/unit/kafka/integration/ProducerConsumerTestHarness.scala 
731ee59e67e302f4cb1da12b1d4ae7ff59010aea 
  core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
d1d969e6c7a5183fa9877fcd12cf49de0cead62a 
  
core/src/test/scala/unit/kafka/javaapi/consumer/ZookeeperConsumerConnectorTest.scala
 16e716440099d62198af3bbadaf5b7d7bc496735 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
7a0ef6faeaa419553f912c8a6d44099818e19833 
  core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
481a400067ec97ed97a2bbd8506b34960a5ec000 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
addd11ab8be2a1c5de0e43f006de1087032fa6d8 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
130b6be8414268b9a992514ecbc7561872ccaca7 

Diff: https://reviews.apache.org/r/20380/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1395:
-

Attachment: KAFKA-1395_2014-04-25_11:03:13.patch

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981333#comment-13981333
 ] 

Guozhang Wang commented on KAFKA-1395:
--

Updated reviewboard https://reviews.apache.org/r/20380/
 against branch origin/trunk

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Apache Kafka 0.8.1.1 Release Candidate 1

[Sriram Subramanian]

+1. Verified quick start + unit tests.

On 4/25/14 10:54 AM, "Neha Narkhede"  wrote:

>+1. Verified quick start.
>
>Thanks,
>Neha
>
>
>On Fri, Apr 25, 2014 at 8:39 AM, Jun Rao  wrote:
>
>> +1. Verified quick start.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Tue, Apr 22, 2014 at 4:18 PM, Joe Stein  wrote:
>>
>> > This is the first candidate for release of Apache Kafka 0.8.1.1
>> >
>> > Release Notes for the 0.8.1.1 release
>> >
>> >
>> 
>>https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/RELEASE_NOTE
>>S.html
>> >
>> > *** Please download, test and vote by Friday, April 25th, 4pm PT
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > https://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5,
>>sha1
>> > and sha2 (SHA256) checksum.
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > https://people.apache.org/~joestein/kafka-0.8.1.1-candidate1/
>> >
>> > * Maven artifacts to be voted upon prior to release:
>> > https://repository.apache.org/content/groups/staging/
>> >
>> > * The tag to be voted upon (off the 0.8.1 branch) is the 0.8.1.1 tag
>> >
>> >
>> 
>>https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=34237371b662f
>>617dd80c75cd6282f97e047ef56
>> >
>> >
>> > /***
>> >  Joe Stein
>> >  Founder, Principal Consultant
>> >  Big Data Open Source Security LLC
>> >  http://www.stealth.ly
>> >  Twitter: @allthingshadoop 
>> > /
>> >
>>

Re: Review Request 20718: Patch for KAFKA-1384

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review41501
---



core/src/main/scala/kafka/server/BrokerStates.scala


Just a thought: I'm wondering if it is better to have a bit-vector approach 
for states; although that will limit the number of possible states - but I 
think that is fine.

The main reason for this is that it will enable composing simultaneous 
states. E.g., in this approach you cannot distinguish state 5 from state 3 (if 
a shutting down broker is the controller). Although we can probably infer that 
from the fact that the other brokers are likely in state (2) or by looking at 
the active controller count separately. Still, we currently allow more than one 
broker to shut down.

It also _might_ help catch erroneous dual states (due to bugs).

What you have is probably fine for lifecycle states (except for the above 
caveat). However, if we ever want to allow more-than-lifecycle states (e.g., 
under-replicated is a state we _might_ want to include on this - even though we 
have a separate URP mbean and it's not a lifecycle state; another example is 
"loading consumer offsets").



core/src/main/scala/kafka/server/KafkaServer.scala


I think we can drop "CurrentBroker". Just "-State" (because it is shorter, 
as meaningful, and also because brokerstate does not fully fit with 
controllerstate)



core/src/main/scala/kafka/server/KafkaServerStartable.scala


Unused


- Joel Koshy


On April 25, 2014, 5:25 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20718/
> ---
> 
> (Updated April 25, 2014, 5:25 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1384
> https://issues.apache.org/jira/browse/KAFKA-1384
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1384: Logging kafka state metric
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/log/Log.scala 
> 46df8d99d977a3b010a9b9f4698187fa9bfb2498 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
> acda52b801714bcc182edc0ced925f0e4b493fc1 
> 
> Diff: https://reviews.apache.org/r/20718/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>

[jira] [Resolved] (KAFKA-1409) oversized messages can slow down the brokers

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1409.


Resolution: Fixed

Thanks for the patch. +1 and committed to trunk.

> oversized messages can slow down the brokers
> 
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1409.patch, KAFKA-1409_2014-04-23_14:10:03.patch, 
> KAFKA-1409_2014-04-23_14:25:48.patch, KAFKA-1409_2014-04-23_15:20:43.patch, 
> KAFKA-1409_2014-04-24_16:26:06.patch, KAFKA-1409_2014-04-25_10:22:54.patch, 
> KAFKA-1409_2014-04-25_10:44:27.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to 
> assign new offsets) before validating the message size (we have to do the 
> validation after recompression since the message size could change). So, it 
> can spend many secs to decompress/recompress an oversized message, only to be 
> dropped later. While this was happening, a request thread was tied up, which 
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per 
> partition). So, if the producer is slow in appending the log, it will block 
> other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981446#comment-13981446
 ] 

Guozhang Wang commented on KAFKA-1395:
--

Updated reviewboard  against branch origin/trunk

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1395:
-

Attachment: KAFKA-1395_2014-04-25_11:54:08.patch

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1395:
-

Attachment: KAFKA-1395_2014-04-25_11:56:47.patch

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch, 
> KAFKA-1395_2014-04-25_11:56:47.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981449#comment-13981449
 ] 

Guozhang Wang commented on KAFKA-1395:
--

Updated reviewboard  against branch origin/trunk

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch, 
> KAFKA-1395_2014-04-25_11:56:47.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20380: Fix KAFKA-1395: Address Jun's comments

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20380/#review41521
---


Looks good. Just the following minor comment.


core/src/test/scala/unit/kafka/utils/TestUtils.scala


Could we add a comment on producerProps?


- Jun Rao


On April 25, 2014, 6:03 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20380/
> ---
> 
> (Updated April 25, 2014, 6:03 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1395
> https://issues.apache.org/jira/browse/KAFKA-1395
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Address Jun's comments round two
> 
> 
> minor fix 2
> 
> 
> minor fix
> 
> 
> Move create server/producer and create topic to setUp in LogRecoveryTest
> 
> 
> Address Jun's comments
> 
> 
> Incoporate Neha's comments
> 
> 
> remove waitUntilLeaderIsElectedOrChanged
> 
> 
> KAFKA-1395.v1
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/integration/AutoOffsetResetTest.scala 
> 7125ec9c03e634d7625a315a944eaf5088074eed 
>   core/src/test/scala/unit/kafka/integration/FetcherTest.scala 
> 4075068a80d6e71a3446a579112541d5b4892854 
>   
> core/src/test/scala/unit/kafka/integration/ProducerConsumerTestHarness.scala 
> 731ee59e67e302f4cb1da12b1d4ae7ff59010aea 
>   core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
> d1d969e6c7a5183fa9877fcd12cf49de0cead62a 
>   
> core/src/test/scala/unit/kafka/javaapi/consumer/ZookeeperConsumerConnectorTest.scala
>  16e716440099d62198af3bbadaf5b7d7bc496735 
>   core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
> 7a0ef6faeaa419553f912c8a6d44099818e19833 
>   core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
> 481a400067ec97ed97a2bbd8506b34960a5ec000 
>   core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
> addd11ab8be2a1c5de0e43f006de1087032fa6d8 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 130b6be8414268b9a992514ecbc7561872ccaca7 
> 
> Diff: https://reviews.apache.org/r/20380/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Resolved] (KAFKA-1392) all TestUtiles.waitUntilTrue() should throw an exception if the return value is false

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1392.


   Resolution: Fixed
Fix Version/s: 0.8.2
 Assignee: Jun Rao

Thanks for the review. Committed to trunk.

> all TestUtiles.waitUntilTrue() should throw an exception if the return value 
> is false
> -
>
> Key: KAFKA-1392
> URL: https://issues.apache.org/jira/browse/KAFKA-1392
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.2
>
> Attachments: KAFKA-1392.patch, KAFKA-1392.patch, KAFKA-1392.patch, 
> KAFKA-1392_2014-04-24_15:53:56.patch, KAFKA-1392_2014-04-24_16:07:07.patch, 
> KAFKA-1392_2014-04-24_17:59:25.patch
>
>
> In all unit tests, we expect TestUtiles.waitUntilTrue() to be true. However, 
> this is not enforced in all unit tests.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20380: Fix KAFKA-1395: Address Jun's comments

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20380/
---

(Updated April 25, 2014, 9:09 p.m.)


Review request for kafka.


Bugs: KAFKA-1395
https://issues.apache.org/jira/browse/KAFKA-1395


Repository: kafka


Description (updated)
---

Added comments 2


Added comments


minor fix


Address Jun's comments round two


minor fix 2


minor fix


Move create server/producer and create topic to setUp in LogRecoveryTest


Address Jun's comments


Incoporate Neha's comments


remove waitUntilLeaderIsElectedOrChanged


KAFKA-1395.v1


Diffs (updated)
-

  core/src/test/scala/unit/kafka/integration/AutoOffsetResetTest.scala 
7125ec9c03e634d7625a315a944eaf5088074eed 
  core/src/test/scala/unit/kafka/integration/FetcherTest.scala 
4075068a80d6e71a3446a579112541d5b4892854 
  core/src/test/scala/unit/kafka/integration/ProducerConsumerTestHarness.scala 
731ee59e67e302f4cb1da12b1d4ae7ff59010aea 
  core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
d1d969e6c7a5183fa9877fcd12cf49de0cead62a 
  
core/src/test/scala/unit/kafka/javaapi/consumer/ZookeeperConsumerConnectorTest.scala
 16e716440099d62198af3bbadaf5b7d7bc496735 
  core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala 
1b87acf11f48f81d1186f5eb2e00017079eaef60 
  core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
faf466b4ed3d408e548db6e454f03020c2ff4808 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
addd11ab8be2a1c5de0e43f006de1087032fa6d8 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
498941d8d894fbb2120bfa10c5be1e046691f73b 

Diff: https://reviews.apache.org/r/20380/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1395:
-

Attachment: KAFKA-1395_2014-04-25_14:09:21.patch

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch, 
> KAFKA-1395_2014-04-25_11:56:47.patch, KAFKA-1395_2014-04-25_14:09:21.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981610#comment-13981610
 ] 

Guozhang Wang commented on KAFKA-1395:
--

Updated reviewboard https://reviews.apache.org/r/20380/
 against branch origin/trunk

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch, 
> KAFKA-1395_2014-04-25_11:56:47.patch, KAFKA-1395_2014-04-25_14:09:21.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1395) fix unit tests in AutoOffsetResetTest

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1395.


   Resolution: Fixed
Fix Version/s: 0.8.2

Thanks for the patch. +1 and committed to trunk.

> fix unit tests in AutoOffsetResetTest
> -
>
> Key: KAFKA-1395
> URL: https://issues.apache.org/jira/browse/KAFKA-1395
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Fix For: 0.8.2
>
> Attachments: KAFKA-1395.patch, KAFKA-1395_2014-04-16_10:11:30.patch, 
> KAFKA-1395_2014-04-22_10:43:54.patch, KAFKA-1395_2014-04-24_16:22:18.patch, 
> KAFKA-1395_2014-04-25_11:03:13.patch, KAFKA-1395_2014-04-25_11:54:08.patch, 
> KAFKA-1395_2014-04-25_11:56:47.patch, KAFKA-1395_2014-04-25_14:09:21.patch
>
>
> It's currently disabled after kafka-1390.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1425) provide logStartOffset in addition to LogEndOffset metric

[alexcb (JIRA)]

alexcb created KAFKA-1425:
-

 Summary: provide logStartOffset in addition to LogEndOffset metric
 Key: KAFKA-1425
 URL: https://issues.apache.org/jira/browse/KAFKA-1425
 Project: Kafka
  Issue Type: Improvement
Reporter: alexcb
Priority: Minor


Hello,

Would it be possible to include a metric to report the minimum message offset 
for each topic/partition? This would mirror the LogEndOffset which is already 
provided.

There's also a Size metric that's provided; however, this gives us the size of 
a topic/partition in bytes so I'm unable to calculate the Starting/beginning 
message offset based on the end offset.

Thanks,
Alex

Here's an initial patch which I have used against 0.8.1:

--- kafka-0.8.1-src/core/src/main/scala/kafka/log/Log.scala 2014-04-25 
09:53:00.252614353 -0700
+++ kafka-0.8.1-src-patch/core/src/main/scala/kafka/log/Log.scala   
2014-04-25 11:38:32.610614193 -0700
@@ -76,6 +76,15 @@
   newGauge(name + "-" + "LogEndOffset",
new Gauge[Long] { def value = logEndOffset })
 
+  newGauge(name + "-" + "logStartOffset",
+   new Gauge[Long] { def value = logStartOffset })
+
   /** The name of this log */
   def name  = dir.getName()
 
@@ -421,6 +430,11 @@
   def size: Long = logSegments.map(_.size).sum
 
   /**
+   * The earliest message offset in the log
+   */
+  def logStartOffset: Long = logSegments.map(_.baseOffset).min
+
+  /**
*  The offset of the next message that will be appended to the log
*/
   def logEndOffset: Long = nextOffset.get



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1425) provide logStartOffset in addition to LogEndOffset metric

[alexcb (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

alexcb updated KAFKA-1425:
--

Attachment: add-start-offset-to-kafka.patch

> provide logStartOffset in addition to LogEndOffset metric
> -
>
> Key: KAFKA-1425
> URL: https://issues.apache.org/jira/browse/KAFKA-1425
> Project: Kafka
>  Issue Type: Improvement
>Reporter: alexcb
>Priority: Minor
> Attachments: add-start-offset-to-kafka.patch
>
>
> Hello,
> Would it be possible to include a metric to report the minimum message offset 
> for each topic/partition? This would mirror the LogEndOffset which is already 
> provided.
> There's also a Size metric that's provided; however, this gives us the size 
> of a topic/partition in bytes so I'm unable to calculate the 
> Starting/beginning message offset based on the end offset.
> Thanks,
> Alex
> Here's an initial patch which I have used against 0.8.1:
> --- kafka-0.8.1-src/core/src/main/scala/kafka/log/Log.scala   2014-04-25 
> 09:53:00.252614353 -0700
> +++ kafka-0.8.1-src-patch/core/src/main/scala/kafka/log/Log.scala 
> 2014-04-25 11:38:32.610614193 -0700
> @@ -76,6 +76,15 @@
>newGauge(name + "-" + "LogEndOffset",
> new Gauge[Long] { def value = logEndOffset })
>  
> +  newGauge(name + "-" + "logStartOffset",
> +   new Gauge[Long] { def value = logStartOffset })
> +
>/** The name of this log */
>def name  = dir.getName()
>  
> @@ -421,6 +430,11 @@
>def size: Long = logSegments.map(_.size).sum
>  
>/**
> +   * The earliest message offset in the log
> +   */
> +  def logStartOffset: Long = logSegments.map(_.baseOffset).min
> +
> +  /**
> *  The offset of the next message that will be appended to the log
> */
>def logEndOffset: Long = nextOffset.get



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1425) provide logStartOffset in addition to LogEndOffset metric

[alexcb (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

alexcb updated KAFKA-1425:
--

Description: 
Hello,

Would it be possible to include a metric to report the minimum message offset 
for each topic/partition? This would mirror the LogEndOffset which is already 
provided.

There's also a Size metric that's provided; however, this gives us the size of 
a topic/partition in bytes so I'm unable to calculate the Starting/beginning 
message offset based on the end offset.

I've attached a patch I used with kafka 0.8.1 to expose this metric.

Thanks,
Alex

  was:
Hello,

Would it be possible to include a metric to report the minimum message offset 
for each topic/partition? This would mirror the LogEndOffset which is already 
provided.

There's also a Size metric that's provided; however, this gives us the size of 
a topic/partition in bytes so I'm unable to calculate the Starting/beginning 
message offset based on the end offset.

Thanks,
Alex

Here's an initial patch which I have used against 0.8.1:

--- kafka-0.8.1-src/core/src/main/scala/kafka/log/Log.scala 2014-04-25 
09:53:00.252614353 -0700
+++ kafka-0.8.1-src-patch/core/src/main/scala/kafka/log/Log.scala   
2014-04-25 11:38:32.610614193 -0700
@@ -76,6 +76,15 @@
   newGauge(name + "-" + "LogEndOffset",
new Gauge[Long] { def value = logEndOffset })
 
+  newGauge(name + "-" + "logStartOffset",
+   new Gauge[Long] { def value = logStartOffset })
+
   /** The name of this log */
   def name  = dir.getName()
 
@@ -421,6 +430,11 @@
   def size: Long = logSegments.map(_.size).sum
 
   /**
+   * The earliest message offset in the log
+   */
+  def logStartOffset: Long = logSegments.map(_.baseOffset).min
+
+  /**
*  The offset of the next message that will be appended to the log
*/
   def logEndOffset: Long = nextOffset.get


> provide logStartOffset in addition to LogEndOffset metric
> -
>
> Key: KAFKA-1425
> URL: https://issues.apache.org/jira/browse/KAFKA-1425
> Project: Kafka
>  Issue Type: Improvement
>Reporter: alexcb
>Priority: Minor
> Attachments: add-start-offset-to-kafka.patch
>
>
> Hello,
> Would it be possible to include a metric to report the minimum message offset 
> for each topic/partition? This would mirror the LogEndOffset which is already 
> provided.
> There's also a Size metric that's provided; however, this gives us the size 
> of a topic/partition in bytes so I'm unable to calculate the 
> Starting/beginning message offset based on the end offset.
> I've attached a patch I used with kafka 0.8.1 to expose this metric.
> Thanks,
> Alex



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1425) provide logStartOffset in addition to LogEndOffset metric

[alexcb (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

alexcb updated KAFKA-1425:
--

Attachment: (was: add-start-offset-to-kafka.patch)

> provide logStartOffset in addition to LogEndOffset metric
> -
>
> Key: KAFKA-1425
> URL: https://issues.apache.org/jira/browse/KAFKA-1425
> Project: Kafka
>  Issue Type: Improvement
>Reporter: alexcb
>Priority: Minor
>
> Hello,
> Would it be possible to include a metric to report the minimum message offset 
> for each topic/partition? This would mirror the LogEndOffset which is already 
> provided.
> There's also a Size metric that's provided; however, this gives us the size 
> of a topic/partition in bytes so I'm unable to calculate the 
> Starting/beginning message offset based on the end offset.
> I've attached a patch I used with kafka 0.8.1 to expose this metric.
> Thanks,
> Alex



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1425) provide logStartOffset in addition to LogEndOffset metric

[alexcb (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

alexcb updated KAFKA-1425:
--

Attachment: add-start-offset-to-kafka.patch

> provide logStartOffset in addition to LogEndOffset metric
> -
>
> Key: KAFKA-1425
> URL: https://issues.apache.org/jira/browse/KAFKA-1425
> Project: Kafka
>  Issue Type: Improvement
>Reporter: alexcb
>Priority: Minor
> Attachments: add-start-offset-to-kafka.patch
>
>
> Hello,
> Would it be possible to include a metric to report the minimum message offset 
> for each topic/partition? This would mirror the LogEndOffset which is already 
> provided.
> There's also a Size metric that's provided; however, this gives us the size 
> of a topic/partition in bytes so I'm unable to calculate the 
> Starting/beginning message offset based on the end offset.
> I've attached a patch I used with kafka 0.8.1 to expose this metric.
> Thanks,
> Alex



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Jenkins build is back to normal : Kafka-trunk #176

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-1417) Very slow initial high-level consumer startup in low traffic/blocking fetch scenario

[Sam Meder (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981825#comment-13981825
 ] 

Sam Meder commented on KAFKA-1417:
--

Looks fine in 0.8.1 

> Very slow initial high-level consumer startup in low traffic/blocking fetch 
> scenario
> 
>
> Key: KAFKA-1417
> URL: https://issues.apache.org/jira/browse/KAFKA-1417
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
>Reporter: Sam Meder
>Assignee: Neha Narkhede
> Fix For: 0.8.1
>
>
> We're seeing very slow startup times when starting a high level consumer in a 
> low traffic/blocking fetch type setup. The example we've come across has a 
> consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout. 
> What happens is that the leader finder thread adds partitions one by one and 
> since the offset is not know this causes a call to figure out the offset. 
> This call uses the fetcher threads simple consumer instance and locks around 
> the call. Initially this is not a problem, but as soon as the fetcher thread 
> has some partitions it will start fetching and since this is a low traffic 
> situation the fetch will at least sometimes take up to 20s (again locking 
> around the simple consumer). This leads to behavior like:
> # Finder thread adds a partition
> # Data thread notices it has partitions to fetch data for, locks the consumer 
> for 20s
> # Finder thread tries to add a partition, tries to lock consumer and blocks 
> for 20s
> # Rinse, repeat for each partition



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1417) Very slow initial high-level consumer startup in low traffic/blocking fetch scenario

[Sam Meder (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Meder resolved KAFKA-1417.
--

   Resolution: Fixed
Fix Version/s: 0.8.1

> Very slow initial high-level consumer startup in low traffic/blocking fetch 
> scenario
> 
>
> Key: KAFKA-1417
> URL: https://issues.apache.org/jira/browse/KAFKA-1417
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.0
>Reporter: Sam Meder
>Assignee: Neha Narkhede
> Fix For: 0.8.1
>
>
> We're seeing very slow startup times when starting a high level consumer in a 
> low traffic/blocking fetch type setup. The example we've come across has a 
> consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout. 
> What happens is that the leader finder thread adds partitions one by one and 
> since the offset is not know this causes a call to figure out the offset. 
> This call uses the fetcher threads simple consumer instance and locks around 
> the call. Initially this is not a problem, but as soon as the fetcher thread 
> has some partitions it will start fetching and since this is a low traffic 
> situation the fetch will at least sometimes take up to 20s (again locking 
> around the simple consumer). This leads to behavior like:
> # Finder thread adds a partition
> # Data thread notices it has partitions to fetch data for, locks the consumer 
> for 20s
> # Finder thread tries to add a partition, tries to lock consumer and blocks 
> for 20s
> # Rinse, repeat for each partition



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20718: Patch for KAFKA-1384

[Timothy Chen]

> On April 25, 2014, 6:35 p.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/KafkaServerStartable.scala, line 55
> > 
> >
> > Unused

This is actually exposed setting custom state that is not one of the defined 
enums. One example is for our internal custom kafka server startable override 
can allow more states that is not defined in BrokerStates. The alternative is 
to just come up with all possible enums and let that be the only option 
available, but I was thinking that might be a bit too limiting. I'm not sure if 
this approach is too permissive either, you have more thoughts?


> On April 25, 2014, 6:35 p.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/BrokerStates.scala, line 28
> > 
> >
> > Just a thought: I'm wondering if it is better to have a bit-vector 
> > approach for states; although that will limit the number of possible states 
> > - but I think that is fine.
> > 
> > The main reason for this is that it will enable composing simultaneous 
> > states. E.g., in this approach you cannot distinguish state 5 from state 3 
> > (if a shutting down broker is the controller). Although we can probably 
> > infer that from the fact that the other brokers are likely in state (2) or 
> > by looking at the active controller count separately. Still, we currently 
> > allow more than one broker to shut down.
> > 
> > It also _might_ help catch erroneous dual states (due to bugs).
> > 
> > What you have is probably fine for lifecycle states (except for the 
> > above caveat). However, if we ever want to allow more-than-lifecycle states 
> > (e.g., under-replicated is a state we _might_ want to include on this - 
> > even though we have a separate URP mbean and it's not a lifecycle state; 
> > another example is "loading consumer offsets").

Erroneous dual states is something I thought of, especially when state changes 
rapidly it will be harder to detect.
The down side of having a bit field is also that you need to then also have to 
unset that state when you're going into another lifecycle state.

I think coming back to the high level of what we want for broker states, since 
really a broker should really just be in a state at one point of time, setting 
a particular state I think makes more sense.

Composite states probably makes more sense for different component states. 


- Timothy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20718/#review41501
---


On April 25, 2014, 5:25 p.m., Timothy Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20718/
> ---
> 
> (Updated April 25, 2014, 5:25 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1384
> https://issues.apache.org/jira/browse/KAFKA-1384
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1384: Logging kafka state metric
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 933de9dd324c7086efe6aa610335ef370d9e9c12 
>   core/src/main/scala/kafka/log/Log.scala 
> 46df8d99d977a3b010a9b9f4698187fa9bfb2498 
>   core/src/main/scala/kafka/log/LogManager.scala 
> ac67f081e6219fd2181479e7a2bb88ea6044e6cc 
>   core/src/main/scala/kafka/server/BrokerStates.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> c208f83bed7fb91f07fae42f2b66892e6d46fecc 
>   core/src/main/scala/kafka/server/KafkaServerStartable.scala 
> acda52b801714bcc182edc0ced925f0e4b493fc1 
> 
> Diff: https://reviews.apache.org/r/20718/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Timothy Chen
> 
>