[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113232#comment-14113232
]
Mayuresh Gharat commented on KAFKA-1610:
Created reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-08-29_09:51:51.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115459#comment-14115459
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115478#comment-14115478
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-08-29_10:03:55.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-09-03_11:27:50.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120214#comment-14120214
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-09-16_13:08:17.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136111#comment-14136111
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-09-16_15:23:27.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136372#comment-14136372
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KAFKA-1592) Some INFO level logging needs to be DEBUG
[
https://issues.apache.org/jira/browse/KAFKA-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat resolved KAFKA-1592.
Resolution: Duplicate
Reviewer: Guozhang Wang
Closing a duplicate as the changes were taken care of in 1591
> Some INFO level logging needs to be DEBUG
> -
>
> Key: KAFKA-1592
> URL: https://issues.apache.org/jira/browse/KAFKA-1592
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
>
> Some of the INFO level log4j entries are not really useful, for example in
> SocketServer.Processor, due to metadata requests that reply on a separate and
> short-lived socket, the following log can be constantly printed:
> info("Closing socket connection to
> %s.".format(channelFor(key).socket.getInetAddress))
> We'd better move them to DEBUG if they are expected in normal state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145012#comment-14145012 ] Mayuresh Gharat commented on KAFKA-1013: Created reviewboard https://reviews.apache.org/r/25942/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Status: Patch Available (was: Open) > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145060#comment-14145060 ] Mayuresh Gharat commented on KAFKA-1013: Created reviewboard https://reviews.apache.org/r/25944/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013_2014-09-23_10:45:59.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145110#comment-14145110 ] Mayuresh Gharat commented on KAFKA-1013: Updated reviewboard https://reviews.apache.org/r/25944/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013_2014-09-23_10:48:07.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145116#comment-14145116 ] Mayuresh Gharat commented on KAFKA-1013: Updated reviewboard https://reviews.apache.org/r/25944/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-1648: -- Assignee: Mayuresh Gharat (was: Neha Narkhede) > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249) > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250) > at > org.eclipse
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14150332#comment-14150332 ] Mayuresh Gharat commented on KAFKA-1013: Updated reviewboard https://reviews.apache.org/r/25944/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch, > KAFKA-1013_2014-09-26_18:52:09.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013_2014-09-26_18:52:09.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch, > KAFKA-1013_2014-09-26_18:52:09.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-09-30_23:21:46.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch,
> KAFKA-1610_2014-09-30_23:21:46.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154426#comment-14154426
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch,
> KAFKA-1610_2014-09-30_23:21:46.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156056#comment-14156056 ] Mayuresh Gharat commented on KAFKA-1013: Updated reviewboard https://reviews.apache.org/r/25944/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch, > KAFKA-1013_2014-09-26_18:52:09.patch, KAFKA-1013_2014-10-01_21:05:00.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013_2014-10-01_21:05:00.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch, > KAFKA-1013_2014-09-26_18:52:09.patch, KAFKA-1013_2014-10-01_21:05:00.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-10-02_12:07:01.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch,
> KAFKA-1610_2014-09-30_23:21:46.patch, KAFKA-1610_2014-10-02_12:07:01.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156991#comment-14156991
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch,
> KAFKA-1610_2014-09-30_23:21:46.patch, KAFKA-1610_2014-10-02_12:07:01.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157000#comment-14157000
]
Mayuresh Gharat commented on KAFKA-1610:
Updated reviewboard https://reviews.apache.org/r/25136/diff/
against branch origin/trunk
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch,
> KAFKA-1610_2014-09-30_23:21:46.patch, KAFKA-1610_2014-10-02_12:07:01.patch,
> KAFKA-1610_2014-10-02_12:09:46.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost
[
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-10-02_12:09:46.patch
> Local modifications to collections generated from mapValues will be lost
>
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch,
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch,
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch,
> KAFKA-1610_2014-09-30_23:21:46.patch, KAFKA-1610_2014-10-02_12:07:01.patch,
> KAFKA-1610_2014-10-02_12:09:46.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it
> has an important semantic difference with map, which is that "map" creates a
> new map collection instance, while "mapValues" just create a map view of the
> original map, and hence any further value changes to the view will be
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to
> map
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157333#comment-14157333 ] Mayuresh Gharat commented on KAFKA-1648: Created reviewboard https://reviews.apache.org/r/26291/diff/ against branch origin/trunk > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContext
[jira] [Updated] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1648: --- Status: Patch Available (was: Open) > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249) > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
[jira] [Updated] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1648: --- Attachment: KAFKA-1648.patch > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249) > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013_2014-10-02_22:42:58.patch > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch, > KAFKA-1013_2014-09-26_18:52:09.patch, KAFKA-1013_2014-10-01_21:05:00.patch, > KAFKA-1013_2014-10-02_22:42:58.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157699#comment-14157699 ] Mayuresh Gharat commented on KAFKA-1013: Updated reviewboard https://reviews.apache.org/r/25944/diff/ against branch origin/trunk > Modify existing tools as per the changes in KAFKA-1000 > -- > > Key: KAFKA-1013 > URL: https://issues.apache.org/jira/browse/KAFKA-1013 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Tejas Patil > Assignee: Mayuresh Gharat >Priority: Minor > Attachments: KAFKA-1013.patch, KAFKA-1013.patch, > KAFKA-1013_2014-09-23_10:45:59.patch, KAFKA-1013_2014-09-23_10:48:07.patch, > KAFKA-1013_2014-09-26_18:52:09.patch, KAFKA-1013_2014-10-01_21:05:00.patch, > KAFKA-1013_2014-10-02_22:42:58.patch > > > Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools > below would be affected: > - ConsumerOffsetChecker > - ExportZkOffsets > - ImportZkOffsets > - UpdateOffsetsInZK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-1669) Default rebalance retries and backoff should be higher
[ https://issues.apache.org/jira/browse/KAFKA-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-1669: -- Assignee: Mayuresh Gharat > Default rebalance retries and backoff should be higher > -- > > Key: KAFKA-1669 > URL: https://issues.apache.org/jira/browse/KAFKA-1669 > Project: Kafka > Issue Type: Improvement >Reporter: Clark Haskins > Assignee: Mayuresh Gharat > Labels: newbie > > The default rebalance logic does not work for consumers with large numbers of > partitions and/or topics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1669) Default rebalance retries and backoff should be higher
[ https://issues.apache.org/jira/browse/KAFKA-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1669: --- Attachment: KAFKA-1669.patch > Default rebalance retries and backoff should be higher > -- > > Key: KAFKA-1669 > URL: https://issues.apache.org/jira/browse/KAFKA-1669 > Project: Kafka > Issue Type: Improvement > Components: consumer >Reporter: Clark Haskins > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1669.patch > > > The default rebalance logic does not work for consumers with large numbers of > partitions and/or topics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1669) Default rebalance retries and backoff should be higher
[ https://issues.apache.org/jira/browse/KAFKA-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158639#comment-14158639 ] Mayuresh Gharat commented on KAFKA-1669: Created reviewboard https://reviews.apache.org/r/26331/diff/ against branch origin/trunk > Default rebalance retries and backoff should be higher > -- > > Key: KAFKA-1669 > URL: https://issues.apache.org/jira/browse/KAFKA-1669 > Project: Kafka > Issue Type: Improvement > Components: consumer >Reporter: Clark Haskins > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1669.patch > > > The default rebalance logic does not work for consumers with large numbers of > partitions and/or topics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1669) Default rebalance retries and backoff should be higher
[ https://issues.apache.org/jira/browse/KAFKA-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1669: --- Status: Patch Available (was: Open) > Default rebalance retries and backoff should be higher > -- > > Key: KAFKA-1669 > URL: https://issues.apache.org/jira/browse/KAFKA-1669 > Project: Kafka > Issue Type: Improvement > Components: consumer >Reporter: Clark Haskins > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1669.patch > > > The default rebalance logic does not work for consumers with large numbers of > partitions and/or topics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1648: --- Attachment: KAFKA-1648_2014-10-04_17:40:47.patch > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249) > at > org.eclipse.jett
[jira] [Commented] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159366#comment-14159366 ] Mayuresh Gharat commented on KAFKA-1648: Updated reviewboard https://reviews.apache.org/r/26291/diff/ against branch origin/trunk > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContext
[jira] [Commented] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164465#comment-14164465 ] Mayuresh Gharat commented on KAFKA-1648: Updated reviewboard https://reviews.apache.org/r/26291/diff/ against branch origin/trunk > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch, > KAFKA-1648_2014-10-08_17:29:14.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > o
[jira] [Updated] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1648: --- Attachment: KAFKA-1648_2014-10-08_17:29:14.patch > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch, > KAFKA-1648_2014-10-08_17:29:14.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:249
[jira] [Updated] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1648: --- Attachment: KAFKA-1648_2014-10-08_17:46:45.patch > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch, > KAFKA-1648_2014-10-08_17:29:14.patch, KAFKA-1648_2014-10-08_17:46:45.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > org.eclipse.jetty.servlet.ServletContextH
[jira] [Commented] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164487#comment-14164487 ] Mayuresh Gharat commented on KAFKA-1648: Updated reviewboard https://reviews.apache.org/r/26291/diff/ against branch origin/trunk > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch, > KAFKA-1648_2014-10-08_17:29:14.patch, KAFKA-1648_2014-10-08_17:46:45.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.
[jira] [Updated] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1648: --- Attachment: KAFKA-1648_2014-10-09_11:56:44.patch > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch, > KAFKA-1648_2014-10-08_17:29:14.patch, KAFKA-1648_2014-10-08_17:46:45.patch, > KAFKA-1648_2014-10-09_11:56:44.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:763) > at > or
[jira] [Commented] (KAFKA-1648) Round robin consumer balance throws an NPE when there are no topics
[ https://issues.apache.org/jira/browse/KAFKA-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165549#comment-14165549 ] Mayuresh Gharat commented on KAFKA-1648: Updated reviewboard https://reviews.apache.org/r/26291/diff/ against branch origin/trunk > Round robin consumer balance throws an NPE when there are no topics > --- > > Key: KAFKA-1648 > URL: https://issues.apache.org/jira/browse/KAFKA-1648 > Project: Kafka > Issue Type: Bug > Components: consumer >Reporter: Todd Palino > Assignee: Mayuresh Gharat > Labels: newbie > Attachments: KAFKA-1648.patch, KAFKA-1648_2014-10-04_17:40:47.patch, > KAFKA-1648_2014-10-08_17:29:14.patch, KAFKA-1648_2014-10-08_17:46:45.patch, > KAFKA-1648_2014-10-09_11:56:44.patch > > > If you use the roundrobin rebalance method with a wildcard consumer, and > there are no topics in the cluster, rebalance throws a NullPointerException > in the consumer and fails. It retries the rebalance, but will continue to > throw the NPE. > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared all relevant > queues for this fetcher > 2014/09/23 17:51:16.147 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Cleared the data > chunks in all the consumer message iterators > 2014/09/23 17:51:16.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], Committing all > offsets after clearing the fetcher queues > 2014/09/23 17:51:46.148 [ZookeeperConsumerConnector] > [kafka-audit_lva1-app0007.corp-1411494404908-4e620544], begin rebalancing > consumer kafka-audit_lva1-app0007.corp-1411494404908-4e620544 try #0 > 2014/09/23 17:51:46.148 ERROR [OffspringServletRuntime] [main] > [kafka-console-audit] [] Boot listener > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener failed > kafka.common.ConsumerRebalanceFailedException: > kafka-audit_lva1-app0007.corp-1411494404908-4e620544 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:630) > at > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:897) > at > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:931) > at > kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:159) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.initWildcardIterators(TrackingConsumerImpl.java:88) > at > com.linkedin.tracker.consumer.TrackingConsumerImpl.getWildcardIterators(TrackingConsumerImpl.java:116) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.createAuditThreads(KafkaConsoleAudit.java:59) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAudit.initializeAudit(KafkaConsoleAudit.java:50) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:125) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditFactory.createInstance(KafkaConsoleAuditFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:20) > at > com.linkedin.util.factory.SimpleSingletonFactory.createInstance(SimpleSingletonFactory.java:14) > at com.linkedin.util.factory.Generator.doGetBean(Generator.java:337) > at com.linkedin.util.factory.Generator.getBean(Generator.java:270) > at > com.linkedin.kafkaconsoleaudit.KafkaConsoleAuditBootListener.onBoot(KafkaConsoleAuditBootListener.java:16) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.startGenerator(OffspringServletRuntime.java:147) > at > com.linkedin.offspring.servlet.OffspringServletRuntime.start(OffspringServletRuntime.java:73) > at > com.linkedin.offspring.servlet.OffspringServletContextListener.contextInitialized(OffspringServletContextListener.java:28) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:771) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:424) > at > org.eclip
[jira] [Assigned] (KAFKA-1742) ControllerContext removeTopic does not correctly update state
[ https://issues.apache.org/jira/browse/KAFKA-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-1742: -- Assignee: Mayuresh Gharat > ControllerContext removeTopic does not correctly update state > - > > Key: KAFKA-1742 > URL: https://issues.apache.org/jira/browse/KAFKA-1742 > Project: Kafka > Issue Type: Bug >Reporter: Onur Karaman > Assignee: Mayuresh Gharat > > removeTopic does not correctly update the state of ControllerContext. > This is because it removes the topic from some underlying maps through > dropWhile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1763) validate_index_log in system tests runs remotely but uses local paths
[
https://issues.apache.org/jira/browse/KAFKA-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207592#comment-14207592
]
Mayuresh Gharat commented on KAFKA-1763:
Yeah we will probably test out today.
> validate_index_log in system tests runs remotely but uses local paths
> -
>
> Key: KAFKA-1763
> URL: https://issues.apache.org/jira/browse/KAFKA-1763
> Project: Kafka
> Issue Type: Bug
> Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1763.patch
>
>
> validate_index_log is the only validation step in the system tests that needs
> to execute a Kafka binary and it's currently doing so remotely, like the rest
> of the test binaries. However, this is probably incorrect since it looks like
> logs are synced back to the driver host and in other cases are operated on
> locally. It looks like validate_index_log mixes up local/remote paths,
> causing an exception in DumpLogSegments:
> {quote}
> 2014-11-10 12:09:57,665 - DEBUG - executing command [ssh vagrant@worker1 -o
> 'HostName 127.0.0.1' -o 'Port ' -o 'UserKnownHostsFile /dev/null' -o
> 'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'IdentityFile
> /Users/ewencp/.vagrant.d/insecure_private_key' -o 'IdentitiesOnly yes' -o
> 'LogLevel FATAL' '/opt/kafka/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --file
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> --verify-index-only 2>&1'] (system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Dumping
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Exception in thread "main"
> java.io.FileNotFoundException:
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.log
> (No such file or directory) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at java.io.FileInputStream.open(Native
> Method) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> java.io.FileInputStream.(FileInputStream.java:146)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.utils.Utils$.openChannel(Utils.scala:162) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.log.FileMessageSet.(FileMessageSet.scala:74)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpIndex(DumpLogSegments.scala:108)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:80)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:73)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments$.main(DumpLogSegments.scala:73)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments.main(DumpLogSegments.scala)
> (kafka_system_test_utils)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1763) validate_index_log in system tests runs remotely but uses local paths
[
https://issues.apache.org/jira/browse/KAFKA-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14210190#comment-14210190
]
Mayuresh Gharat commented on KAFKA-1763:
Thanks for the patch.
We tested it out. This resolved the failures that were happening earlier, but
has 3 failures.
_test_case_name : testcase_0131
_test_class_name : ReplicaBasicTest
arg : auto_create_topic : true
arg : bounce_broker : true
arg : broker_type : leader
arg : message_producing_free_time_sec : 15
arg : num_iteration : 3
arg : num_partition : 3
arg : replica_factor : 2
arg : sleep_seconds_between_producer_calls : 1
validation_status :
No. of messages from consumer on [test_1] at
simple_consumer_test_1-0_r1.log : 0
No. of messages from consumer on [test_1] at
simple_consumer_test_1-0_r2.log : 2000
Test completed : FAILED
Validate leader election successful : PASSED
_test_case_name : testcase_0132
_test_class_name : ReplicaBasicTest
arg : auto_create_topic : true
arg : bounce_broker : true
arg : broker_type : leader
arg : message_producing_free_time_sec : 15
arg : num_iteration : 3
arg : num_partition : 3
arg : replica_factor : 2
arg : sleep_seconds_between_producer_calls : 1
validation_status :
No. of messages from consumer on [test_1] at
simple_consumer_test_1-0_r1.log : 2000
No. of messages from consumer on [test_1] at
simple_consumer_test_1-0_r2.log : 0
Test completed : FAILED
Validate leader election successful : PASSED
_test_case_name : testcase_0133
_test_class_name : ReplicaBasicTest
arg : auto_create_topic : true
arg : bounce_broker : true
arg : broker_type : leader
arg : message_producing_free_time_sec : 15
arg : num_iteration : 3
arg : num_partition : 3
arg : replica_factor : 2
arg : sleep_seconds_between_producer_calls : 1
validation_status :
No. of messages from consumer on [test_1] at
simple_consumer_test_1-0_r1.log : 2000
No. of messages from consumer on [test_1] at
simple_consumer_test_1-0_r2.log : 0
Test completed : FAILED
Validate leader election successful : PASSED
> validate_index_log in system tests runs remotely but uses local paths
> -
>
> Key: KAFKA-1763
> URL: https://issues.apache.org/jira/browse/KAFKA-1763
> Project: Kafka
> Issue Type: Bug
> Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1763.patch
>
>
> validate_index_log is the only validation step in the system tests that needs
> to execute a Kafka binary and it's currently doing so remotely, like the rest
> of the test binaries. However, this is probably incorrect since it looks like
> logs are synced back to the driver host and in other cases are operated on
> locally. It looks like validate_index_log mixes up local/remote paths,
> causing an exception in DumpLogSegments:
> {quote}
> 2014-11-10 12:09:57,665 - DEBUG - executing command [ssh vagrant@worker1 -o
> 'HostName 127.0.0.1' -o 'Port ' -o 'UserKnownHostsFile /dev/null' -o
> 'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'IdentityFile
> /Users/ewencp/.vagrant.d/insecure_private_key' -o 'IdentitiesOnly yes' -o
> 'LogLevel FATAL' '/opt/kafka/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --file
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> --verify-index-only 2>&1'] (system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Dumping
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Exception in thread "main"
> java.io.FileNotFoundException:
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.log
> (No such file or directory) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at java.io.FileInputStream.open(Native
> Method) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> java.io.FileInputStream.(FileInputStream.java:146)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG -
[jira] [Commented] (KAFKA-1763) validate_index_log in system tests runs remotely but uses local paths
[
https://issues.apache.org/jira/browse/KAFKA-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211289#comment-14211289
]
Mayuresh Gharat commented on KAFKA-1763:
Traceback (most recent call last):
File
"/mnt/u001/kafka_replication_system_test/system_test/replication_testsuite/
replica_basic_test.py", line 434, in runTest
kafka_system_test_utils.validate_simple_consumer_data_matched_across_replic
as(self.systemTestEnv, self.testcaseEnv)
File
"/mnt/u001/kafka_replication_system_test/system_test/utils/kafka_system_tes
t_utils.py", line 2223, in
validate_simple_consumer_data_matched_across_replicas
replicaIdxMsgIdList[replicaIdx - 1][topicPartition] = consumerMsgIdList
IndexError: list index out of range
> validate_index_log in system tests runs remotely but uses local paths
> -
>
> Key: KAFKA-1763
> URL: https://issues.apache.org/jira/browse/KAFKA-1763
> Project: Kafka
> Issue Type: Bug
> Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
> Attachments: KAFKA-1763.patch
>
>
> validate_index_log is the only validation step in the system tests that needs
> to execute a Kafka binary and it's currently doing so remotely, like the rest
> of the test binaries. However, this is probably incorrect since it looks like
> logs are synced back to the driver host and in other cases are operated on
> locally. It looks like validate_index_log mixes up local/remote paths,
> causing an exception in DumpLogSegments:
> {quote}
> 2014-11-10 12:09:57,665 - DEBUG - executing command [ssh vagrant@worker1 -o
> 'HostName 127.0.0.1' -o 'Port ' -o 'UserKnownHostsFile /dev/null' -o
> 'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'IdentityFile
> /Users/ewencp/.vagrant.d/insecure_private_key' -o 'IdentitiesOnly yes' -o
> 'LogLevel FATAL' '/opt/kafka/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --file
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> --verify-index-only 2>&1'] (system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Dumping
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Exception in thread "main"
> java.io.FileNotFoundException:
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.log
> (No such file or directory) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at java.io.FileInputStream.open(Native
> Method) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> java.io.FileInputStream.(FileInputStream.java:146)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.utils.Utils$.openChannel(Utils.scala:162) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.log.FileMessageSet.(FileMessageSet.scala:74)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpIndex(DumpLogSegments.scala:108)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:80)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:73)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments$.main(DumpLogSegments.scala:73)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments.main(DumpLogSegments.scala)
> (kafka_system_test_utils)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KAFKA-3995) Add a new configuration "enable.comrpession.ratio.estimation" to the producer config
[ https://issues.apache.org/jira/browse/KAFKA-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-3995: -- Assignee: Mayuresh Gharat > Add a new configuration "enable.comrpession.ratio.estimation" to the producer > config > > > Key: KAFKA-3995 > URL: https://issues.apache.org/jira/browse/KAFKA-3995 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Jiangjie Qin >Assignee: Mayuresh Gharat > Fix For: 0.10.1.0 > > > We recently see a few cases where RecordTooLargeException is thrown because > the compressed message sent by KafkaProducer exceeded the max message size. > The root cause of this issue is because the compressor is estimating the > batch size using an estimated compression ratio based on heuristic > compression ratio statistics. This does not quite work for the traffic with > highly variable compression ratios. > For example, if the batch size is set to 100KB and the max message size is > 1MB. Initially a the producer is sending messages (each message is 100KB) to > topic_1 whose data can be compressed to 1/10 of the original size. After a > while the estimated compression ratio in the compressor will be trained to > 1/10 and the producer would put 10 messages into one batch. Now the producer > starts to send messages (each message is also 100KB) to topic_2 whose message > can only be compress to 1/5 of the original size. The producer would still > use 1/10 as the estimated compression ratio and put 10 messages into a batch. > That batch would be 2 MB after compression which exceeds the maximum message > size. In this case the user do not have many options other than resend > everything or close the producer if they care about ordering. > This is especially an issue for services like MirrorMaker whose producer is > shared by many different topics. > To solve this issue, we can probably add a configuration > "enable.compression.ratio.estimation" to the producer. So when this > configuration is set to false, we stop estimating the compressed size but > will close the batch once the uncompressed bytes in the batch reaches the > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL
[ https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424982#comment-15424982 ] Mayuresh Gharat commented on KAFKA-4050: Just a heads up, this has also been seen as an issue in other systems, for example : https://issues.jenkins-ci.org/browse/JENKINS-20108 > Allow configuration of the PRNG used for SSL > > > Key: KAFKA-4050 > URL: https://issues.apache.org/jira/browse/KAFKA-4050 > Project: Kafka > Issue Type: Improvement > Components: security >Affects Versions: 0.10.0.1 >Reporter: Todd Palino >Assignee: Todd Palino > Labels: security, ssl > > This change will make the pseudo-random number generator (PRNG) > implementation used by the SSLContext configurable. The configuration is not > required, and the default is to use whatever the default PRNG for the JDK/JRE > is. Providing a string, such as "SHA1PRNG", will cause that specific > SecureRandom implementation to get passed to the SSLContext. > When enabling inter-broker SSL in our certification cluster, we observed > severe performance issues. For reference, this cluster can take up to 600 > MB/sec of inbound produce traffic over SSL, with RF=2, before it gets close > to saturation, and the mirror maker normally produces about 400 MB/sec > (unless it is lagging). When we enabled inter-broker SSL, we saw persistent > replication problems in the cluster at any inbound rate of more than about 6 > or 7 MB/sec per-broker. This was narrowed down to all the network threads > blocking on a single lock in the SecureRandom code. > It turns out that the default PRNG implementation on Linux is NativePRNG. > This uses randomness from /dev/urandom (which, by itself, is a non-blocking > read) and mixes it with randomness from SHA1. The problem is that the entire > application shares a single SecureRandom instance, and NativePRNG has a > global lock within the implNextBytes method. Switching to another > implementation (SHA1PRNG, which has better performance characteristics and is > still considered secure) completely eliminated the bottleneck and allowed the > cluster to work properly at saturation. > The SSLContext initialization has an optional argument to provide a > SecureRandom instance, which the code currently sets to null. This change > creates a new config to specify an implementation, and instantiates that and > passes it to SSLContext if provided. This will also let someone select a > stronger source of randomness (obviously at a performance cost) if desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KAFKA-3722) PlaintextChannelBuilder should not use ChannelBuilders.createPrincipalBuilder(configs) for creating instance of PrincipalBuilder
[ https://issues.apache.org/jira/browse/KAFKA-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3722 started by Mayuresh Gharat. -- > PlaintextChannelBuilder should not use > ChannelBuilders.createPrincipalBuilder(configs) for creating instance of > PrincipalBuilder > > > Key: KAFKA-3722 > URL: https://issues.apache.org/jira/browse/KAFKA-3722 > Project: Kafka > Issue Type: Bug > Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Consider this scenario : > 1) We have a Kafka Broker running on PlainText and SSL port simultaneously. > 2) We try to plugin a custom principal builder using the config > "principal.builder.class" for the request coming over the SSL port. > 3) The ChannelBuilders.createPrincipalBuilder(configs) first checks if a > config "principal.builder.class" is specified in the passed in configs and > tries to use that even when it is building the instance of PrincipalBuilder > for the PlainText port, when that custom principal class is only menat for > SSL port. > IMO, having a DefaultPrincipalBuilder for PalinText port should be fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3722) PlaintextChannelBuilder should not use ChannelBuilders.createPrincipalBuilder(configs) for creating instance of PrincipalBuilder
[ https://issues.apache.org/jira/browse/KAFKA-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-3722: --- Status: Patch Available (was: In Progress) > PlaintextChannelBuilder should not use > ChannelBuilders.createPrincipalBuilder(configs) for creating instance of > PrincipalBuilder > > > Key: KAFKA-3722 > URL: https://issues.apache.org/jira/browse/KAFKA-3722 > Project: Kafka > Issue Type: Bug > Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Consider this scenario : > 1) We have a Kafka Broker running on PlainText and SSL port simultaneously. > 2) We try to plugin a custom principal builder using the config > "principal.builder.class" for the request coming over the SSL port. > 3) The ChannelBuilders.createPrincipalBuilder(configs) first checks if a > config "principal.builder.class" is specified in the passed in configs and > tries to use that even when it is building the instance of PrincipalBuilder > for the PlainText port, when that custom principal class is only menat for > SSL port. > IMO, having a DefaultPrincipalBuilder for PalinText port should be fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2976) Mirror maker dies if we delete a topic from destination cluster
[ https://issues.apache.org/jira/browse/KAFKA-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat resolved KAFKA-2976. Resolution: Won't Fix > Mirror maker dies if we delete a topic from destination cluster > --- > > Key: KAFKA-2976 > URL: https://issues.apache.org/jira/browse/KAFKA-2976 > Project: Kafka > Issue Type: Bug > Components: clients > Reporter: Mayuresh Gharat > Assignee: Mayuresh Gharat > > In datapipeline, > 1) Suppose the Mirror Maker is producing to a cluster with Topic T and has > 128 partitions (Partition 0 to Partition 127) . The default setting on > creation of a new topic on that cluster is 8 partitions. > 2) After we delete the topic, the topic gets recreated with 8 partitions > (Partition 0 to Partition 7). > 3) The RecordAccumulator has batches for partitions from 9 to 127. Those > batches get expired and the mirror makers will die to avoid data loss. > We need a way to reassign those batches (batches for Partition 9 top > Partition 127) in the RecordAccumulator to the newly created Topic T with 8 > partitions (Partition 0 to Partition 7). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2014) Chaos Monkey / Failure Inducer for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat resolved KAFKA-2014. Resolution: Fixed > Chaos Monkey / Failure Inducer for Kafka > > > Key: KAFKA-2014 > URL: https://issues.apache.org/jira/browse/KAFKA-2014 > Project: Kafka > Issue Type: Task > Components: system tests > Reporter: Mayuresh Gharat > Assignee: Mayuresh Gharat > > Implement a Chaos Monkey for kafka, that will help us catch any shortcomings > in the test environment before going to production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2014) Chaos Monkey / Failure Inducer for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433359#comment-15433359 ] Mayuresh Gharat commented on KAFKA-2014: https://github.com/linkedin/simoorg > Chaos Monkey / Failure Inducer for Kafka > > > Key: KAFKA-2014 > URL: https://issues.apache.org/jira/browse/KAFKA-2014 > Project: Kafka > Issue Type: Task > Components: system tests > Reporter: Mayuresh Gharat > Assignee: Mayuresh Gharat > > Implement a Chaos Monkey for kafka, that will help us catch any shortcomings > in the test environment before going to production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2014) Chaos Monkey / Failure Inducer for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433359#comment-15433359 ] Mayuresh Gharat edited comment on KAFKA-2014 at 8/23/16 6:37 PM: - This is the failure inducer that was developed for kafka : https://github.com/linkedin/simoorg was (Author: mgharat): https://github.com/linkedin/simoorg > Chaos Monkey / Failure Inducer for Kafka > > > Key: KAFKA-2014 > URL: https://issues.apache.org/jira/browse/KAFKA-2014 > Project: Kafka > Issue Type: Task > Components: system tests > Reporter: Mayuresh Gharat > Assignee: Mayuresh Gharat > > Implement a Chaos Monkey for kafka, that will help us catch any shortcomings > in the test environment before going to production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3083) a soft failure in controller may leave a topic partition in an inconsistent state
[ https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437288#comment-15437288 ] Mayuresh Gharat commented on KAFKA-3083: [~fpj] do we have an umbrella jira where this issue is been tracked with the changes required to be made that are mentioned in this patch? > a soft failure in controller may leave a topic partition in an inconsistent > state > - > > Key: KAFKA-3083 > URL: https://issues.apache.org/jira/browse/KAFKA-3083 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: Mayuresh Gharat > > The following sequence can happen. > 1. Broker A is the controller and is in the middle of processing a broker > change event. As part of this process, let's say it's about to shrink the isr > of a partition. > 2. Then broker A's session expires and broker B takes over as the new > controller. Broker B sends the initial leaderAndIsr request to all brokers. > 3. Broker A continues by shrinking the isr of the partition in ZK and sends > the new leaderAndIsr request to the broker (say C) that leads the partition. > Broker C will reject this leaderAndIsr since the request comes from a > controller with an older epoch. Now we could be in a situation that Broker C > thinks the isr has all replicas, but the isr stored in ZK is different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-4074) Deleting a topic can make it unavailable even if delete.topic.enable is false
[
https://issues.apache.org/jira/browse/KAFKA-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491390#comment-15491390
]
Mayuresh Gharat commented on KAFKA-4074:
[~omkreddy] merging your PR with that in
https://issues.apache.org/jira/browse/KAFKA-3175 would be great.
> Deleting a topic can make it unavailable even if delete.topic.enable is false
> -
>
> Key: KAFKA-4074
> URL: https://issues.apache.org/jira/browse/KAFKA-4074
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
> Fix For: 0.10.1.0
>
>
> The {{delete.topic.enable}} configuration does not completely block the
> effects of delete topic since the controller may (indirectly) query the list
> of topics under the delete-topic znode.
> To reproduce:
> * Delete topic X
> * Force a controller move (either by bouncing or removing the controller
> znode)
> * The new controller will send out UpdateMetadataRequests with leader=-2 for
> the partitions of X
> * Producers eventually stop producing to that topic
> The reason for this is that when ControllerChannelManager adds
> UpdateMetadataRequests for brokers, we directly use the partitionsToBeDeleted
> field of the DeleteTopicManager (which is set to the partitions of the topics
> under the delete-topic znode on controller startup).
> In order to get out of the situation you have to remove X from the znode and
> then force another controller move.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KAFKA-3995) Add a new configuration "enable.comrpession.ratio.estimation" to the producer config
[ https://issues.apache.org/jira/browse/KAFKA-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat resolved KAFKA-3995. Resolution: Workaround > Add a new configuration "enable.comrpession.ratio.estimation" to the producer > config > > > Key: KAFKA-3995 > URL: https://issues.apache.org/jira/browse/KAFKA-3995 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Jiangjie Qin >Assignee: Mayuresh Gharat > Fix For: 0.10.2.0 > > > We recently see a few cases where RecordTooLargeException is thrown because > the compressed message sent by KafkaProducer exceeded the max message size. > The root cause of this issue is because the compressor is estimating the > batch size using an estimated compression ratio based on heuristic > compression ratio statistics. This does not quite work for the traffic with > highly variable compression ratios. > For example, if the batch size is set to 1MB and the max message size is 1MB. > Initially a the producer is sending messages (each message is 1MB) to topic_1 > whose data can be compressed to 1/10 of the original size. After a while the > estimated compression ratio in the compressor will be trained to 1/10 and the > producer would put 10 messages into one batch. Now the producer starts to > send messages (each message is also 1MB) to topic_2 whose message can only be > compress to 1/5 of the original size. The producer would still use 1/10 as > the estimated compression ratio and put 10 messages into a batch. That batch > would be 2 MB after compression which exceeds the maximum message size. In > this case the user do not have many options other than resend everything or > close the producer if they care about ordering. > This is especially an issue for services like MirrorMaker whose producer is > shared by many different topics. > To solve this issue, we can probably add a configuration > "enable.compression.ratio.estimation" to the producer. So when this > configuration is set to false, we stop estimating the compressed size but > will close the batch once the uncompressed bytes in the batch reaches the > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (KAFKA-3995) Add a new configuration "enable.comrpession.ratio.estimation" to the producer config
[ https://issues.apache.org/jira/browse/KAFKA-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reopened KAFKA-3995: > Add a new configuration "enable.comrpession.ratio.estimation" to the producer > config > > > Key: KAFKA-3995 > URL: https://issues.apache.org/jira/browse/KAFKA-3995 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.10.0.0 >Reporter: Jiangjie Qin >Assignee: Mayuresh Gharat > Fix For: 0.10.2.0 > > > We recently see a few cases where RecordTooLargeException is thrown because > the compressed message sent by KafkaProducer exceeded the max message size. > The root cause of this issue is because the compressor is estimating the > batch size using an estimated compression ratio based on heuristic > compression ratio statistics. This does not quite work for the traffic with > highly variable compression ratios. > For example, if the batch size is set to 1MB and the max message size is 1MB. > Initially a the producer is sending messages (each message is 1MB) to topic_1 > whose data can be compressed to 1/10 of the original size. After a while the > estimated compression ratio in the compressor will be trained to 1/10 and the > producer would put 10 messages into one batch. Now the producer starts to > send messages (each message is also 1MB) to topic_2 whose message can only be > compress to 1/5 of the original size. The producer would still use 1/10 as > the estimated compression ratio and put 10 messages into a batch. That batch > would be 2 MB after compression which exceeds the maximum message size. In > this case the user do not have many options other than resend everything or > close the producer if they care about ordering. > This is especially an issue for services like MirrorMaker whose producer is > shared by many different topics. > To solve this issue, we can probably add a configuration > "enable.compression.ratio.estimation" to the producer. So when this > configuration is set to false, we stop estimating the compressed size but > will close the batch once the uncompressed bytes in the batch reaches the > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3995) Add a new configuration "enable.comrpession.ratio.estimation" to the producer config
[ https://issues.apache.org/jira/browse/KAFKA-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589938#comment-15589938 ] Mayuresh Gharat commented on KAFKA-3995: If we disable compression, we would not have this issue right? Of course that's not recommended. The other way would be to reduce the linger.ms to be very very low. On thinking more about this I plan to reopen and work on this. Since this is a new config, we would probably require a KIP for this right? If Yes, I can write up a KIP and submit for review. > Add a new configuration "enable.comrpession.ratio.estimation" to the producer > config > > > Key: KAFKA-3995 > URL: https://issues.apache.org/jira/browse/KAFKA-3995 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.10.0.0 > Reporter: Jiangjie Qin >Assignee: Mayuresh Gharat > Fix For: 0.10.2.0 > > > We recently see a few cases where RecordTooLargeException is thrown because > the compressed message sent by KafkaProducer exceeded the max message size. > The root cause of this issue is because the compressor is estimating the > batch size using an estimated compression ratio based on heuristic > compression ratio statistics. This does not quite work for the traffic with > highly variable compression ratios. > For example, if the batch size is set to 1MB and the max message size is 1MB. > Initially a the producer is sending messages (each message is 1MB) to topic_1 > whose data can be compressed to 1/10 of the original size. After a while the > estimated compression ratio in the compressor will be trained to 1/10 and the > producer would put 10 messages into one batch. Now the producer starts to > send messages (each message is also 1MB) to topic_2 whose message can only be > compress to 1/5 of the original size. The producer would still use 1/10 as > the estimated compression ratio and put 10 messages into a batch. That batch > would be 2 MB after compression which exceeds the maximum message size. In > this case the user do not have many options other than resend everything or > close the producer if they care about ordering. > This is especially an issue for services like MirrorMaker whose producer is > shared by many different topics. > To solve this issue, we can probably add a configuration > "enable.compression.ratio.estimation" to the producer. So when this > configuration is set to false, we stop estimating the compressed size but > will close the batch once the uncompressed bytes in the batch reaches the > batch size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2120: --- Attachment: KAFKA-2120_2015-09-28_16:13:02.patch > Add a request timeout to NetworkClient > -- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature >Reporter: Jiangjie Qin > Assignee: Mayuresh Gharat >Priority: Blocker > Fix For: 0.9.0.0 > > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, > KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, > KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, > KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch, > KAFKA-2120_2015-09-18_19:27:48.patch, KAFKA-2120_2015-09-28_16:13:02.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934308#comment-14934308 ] Mayuresh Gharat commented on KAFKA-2120: Updated reviewboard https://reviews.apache.org/r/36858/diff/ against branch origin/trunk > Add a request timeout to NetworkClient > -- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature >Reporter: Jiangjie Qin > Assignee: Mayuresh Gharat >Priority: Blocker > Fix For: 0.9.0.0 > > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, > KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, > KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, > KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch, > KAFKA-2120_2015-09-18_19:27:48.patch, KAFKA-2120_2015-09-28_16:13:02.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2526) Console Producer / Consumer's serde config is not working
[ https://issues.apache.org/jira/browse/KAFKA-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935428#comment-14935428 ] Mayuresh Gharat commented on KAFKA-2526: Hi [~guozhang], can you list the steps to reproduce this issue. I think we had faced this issue sometime back at Linkedin. > Console Producer / Consumer's serde config is not working > - > > Key: KAFKA-2526 > URL: https://issues.apache.org/jira/browse/KAFKA-2526 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang > Assignee: Mayuresh Gharat > Labels: newbie > Fix For: 0.10.0.0 > > > Although in the console producer one can specify the key value serializer, > they are actually not used since 1) it always serialize the input string as > String.getBytes (hence always pre-assume the string serializer) and 2) it is > actually only passed into the old producer. The same issues exist in console > consumer. > In addition the configs in the console producer is messy: we have 1) some > config values exposed as cmd parameters, and 2) some config values in > --producer-property and 3) some in --property. > It will be great to clean the configs up in both console producer and > consumer, and put them into a single --property parameter which could > possibly take a file to reading in property values as well, and only leave > --new-producer as the other command line parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2526) Console Producer / Consumer's serde config is not working
[ https://issues.apache.org/jira/browse/KAFKA-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935428#comment-14935428 ] Mayuresh Gharat edited comment on KAFKA-2526 at 9/29/15 4:53 PM: - Hi [~guozhang], what steps did you followed to reproduce this? I think we had faced this issue sometime back at Linkedin. was (Author: mgharat): Hi [~guozhang], can you list the steps to reproduce this issue. I think we had faced this issue sometime back at Linkedin. > Console Producer / Consumer's serde config is not working > - > > Key: KAFKA-2526 > URL: https://issues.apache.org/jira/browse/KAFKA-2526 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang > Assignee: Mayuresh Gharat > Labels: newbie > Fix For: 0.10.0.0 > > > Although in the console producer one can specify the key value serializer, > they are actually not used since 1) it always serialize the input string as > String.getBytes (hence always pre-assume the string serializer) and 2) it is > actually only passed into the old producer. The same issues exist in console > consumer. > In addition the configs in the console producer is messy: we have 1) some > config values exposed as cmd parameters, and 2) some config values in > --producer-property and 3) some in --property. > It will be great to clean the configs up in both console producer and > consumer, and put them into a single --property parameter which could > possibly take a file to reading in property values as well, and only leave > --new-producer as the other command line parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2428) Add sanity test in kafkaConsumer for the timeouts. This is a followup ticket for Kafka-2120
[ https://issues.apache.org/jira/browse/KAFKA-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946106#comment-14946106 ] Mayuresh Gharat commented on KAFKA-2428: [~ijuma] Yes. I will submit a PR for this. > Add sanity test in kafkaConsumer for the timeouts. This is a followup ticket > for Kafka-2120 > --- > > Key: KAFKA-2428 > URL: https://issues.apache.org/jira/browse/KAFKA-2428 > Project: Kafka > Issue Type: Bug > Reporter: Mayuresh Gharat > Assignee: Mayuresh Gharat > > The request timeout should be the highest timeout across all the timeout. The > rules should be: > Request timeout > session timeout. > Request timeout > fetch.max.wait.timeout > request timeout won't kick in before the other timeout reached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2428) Add sanity test in kafkaConsumer for the timeouts. This is a followup ticket for Kafka-2120
[ https://issues.apache.org/jira/browse/KAFKA-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2428: --- Fix Version/s: 0.9.0.0 > Add sanity test in kafkaConsumer for the timeouts. This is a followup ticket > for Kafka-2120 > --- > > Key: KAFKA-2428 > URL: https://issues.apache.org/jira/browse/KAFKA-2428 > Project: Kafka > Issue Type: Bug > Reporter: Mayuresh Gharat > Assignee: Mayuresh Gharat > Fix For: 0.9.0.0 > > > The request timeout should be the highest timeout across all the timeout. The > rules should be: > Request timeout > session timeout. > Request timeout > fetch.max.wait.timeout > request timeout won't kick in before the other timeout reached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2635) After deleting the topic, you can still send and receive messages but kafka-topic --list does not show the topic.
Mayuresh Gharat created KAFKA-2635: -- Summary: After deleting the topic, you can still send and receive messages but kafka-topic --list does not show the topic. Key: KAFKA-2635 URL: https://issues.apache.org/jira/browse/KAFKA-2635 Project: Kafka Issue Type: Bug Reporter: Mayuresh Gharat Assignee: Mayuresh Gharat As per [~mingfang]'s comment on https://issues.apache.org/jira/browse/KAFKA-2094, it is possible to send and receive data for a deleted topic but the kafka-topic --list does not show that topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2094) Kafka does not create topic automatically after deleting the topic.
[
https://issues.apache.org/jira/browse/KAFKA-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953697#comment-14953697
]
Mayuresh Gharat commented on KAFKA-2094:
[~mingfang] We should create a separate Jira for this. I have created a ticket
:https://issues.apache.org/jira/browse/KAFKA-2635.
Can you can put some context on that about what you tried and what you saw. I
can than start working on it.
PS :
When one of the topic on our Linkedin cluster was deleted recently we saw
LeaderNotAvailableException while producing to that topic.
> Kafka does not create topic automatically after deleting the topic.
> ---
>
> Key: KAFKA-2094
> URL: https://issues.apache.org/jira/browse/KAFKA-2094
> Project: Kafka
> Issue Type: Bug
> Components: admin
>Affects Versions: 0.8.2.0
> Environment: Ubuntu 14.04 LTS
>Reporter: Hyukjin Kwon
>Priority: Critical
> Fix For: 0.8.2.0
>
>
> After I create a topic and then remove it (and wait for enough time to
> eventually delete it), it does not create a topic emitting errors even though
> auto-create topic option is true. It works okay when I manually create a
> topic after deleting it.
> Here is the command I run.
> ./bin/kafka-topics.sh --list --zookeeper 192.168.0.190:2181
> test
> ./bin/kafka-topics.sh --delete --zookeeper 192.168.0.190:2181 --topic test
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2015-04-06 20:51:44,542] WARN Property topic is not valid
> (kafka.utils.VerifiableProperties)
> test
> [2015-04-06 20:51:49,896] WARN Error while fetching metadata [{TopicMetadata
> for topic test ->
> No partition metadata for topic test due to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
> [2015-04-06 20:51:49,904] WARN Error while fetching metadata [{TopicMetadata
> for topic test->
> No partition metadata for topic testdue to
> kafka.common.LeaderNotAvailableException}] for topic [test]: class
> kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
> [2015-04-06 20:51:49,905] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic:
> test(kafka.producer.async.DefaultEventHandler)
> ^Csnowdrop@master:~/workspace/kafka/kafka_2.10-0.8.2.0$
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2015-04-06 20:52:08,733] WARN Property topic is not valid
> (kafka.utils.VerifiableProperties)
> Server side, I got this error
> [2015-04-06 21:41:08,491] ERROR [Replica Manager on Broker 0]: Error when
> processing fetch request for partition [test,0] offset 1687 from consumer
> with correlation id 5. Possible cause: Request for offset 1687 but we only
> have log segments in the range 0 to 107. (kafka.server.ReplicaManager)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1554) Corrupt index found on clean startup
[
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958225#comment-14958225
]
Mayuresh Gharat commented on KAFKA-1554:
[~wangbo23] thanks for the review.
We did not iterate on this patch. This was resolved in
https://issues.apache.org/jira/browse/KAFKA-2012.
> Corrupt index found on clean startup
>
>
> Key: KAFKA-1554
> URL: https://issues.apache.org/jira/browse/KAFKA-1554
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8.1
> Environment: ubuntu 12.04, oracle jdk 1.7
>Reporter: Alexis Midon
>Assignee: Mayuresh Gharat
>Priority: Critical
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-1554.patch
>
>
> On a clean start up, corrupted index files are found.
> After investigations, it appears that some pre-allocated index files are not
> "compacted" correctly and the end of the file is full of zeroes.
> As a result, on start up, the last relative offset is zero which yields an
> offset equal to the base offset.
> The workaround is to delete all index files of size 10MB (the size of the
> pre-allocated files), and restart. Index files will be re-created.
> {code}
> find $your_data_directory -size 10485760c -name *.index #-delete
> {code}
> This is issue might be related/similar to
> https://issues.apache.org/jira/browse/KAFKA-1112
> {code}
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514],
> Connecting to zookeeper on
> zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708
> INFO
> ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
> org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:host.name=i-6b948138.inst.aws.airbnb.com
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.version=1.7.0_55
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.vendor=Oracle Corporation
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.io.tmpdir=/tmp
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.compiler=
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:os.name=Linux
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:os.arch=amd64
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Clien
[jira] [Resolved] (KAFKA-1554) Corrupt index found on clean startup
[
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat resolved KAFKA-1554.
Resolution: Duplicate
> Corrupt index found on clean startup
>
>
> Key: KAFKA-1554
> URL: https://issues.apache.org/jira/browse/KAFKA-1554
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8.1
> Environment: ubuntu 12.04, oracle jdk 1.7
>Reporter: Alexis Midon
> Assignee: Mayuresh Gharat
>Priority: Critical
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-1554.patch
>
>
> On a clean start up, corrupted index files are found.
> After investigations, it appears that some pre-allocated index files are not
> "compacted" correctly and the end of the file is full of zeroes.
> As a result, on start up, the last relative offset is zero which yields an
> offset equal to the base offset.
> The workaround is to delete all index files of size 10MB (the size of the
> pre-allocated files), and restart. Index files will be re-created.
> {code}
> find $your_data_directory -size 10485760c -name *.index #-delete
> {code}
> This is issue might be related/similar to
> https://issues.apache.org/jira/browse/KAFKA-1112
> {code}
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514],
> Connecting to zookeeper on
> zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708
> INFO
> ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
> org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:host.name=i-6b948138.inst.aws.airbnb.com
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.version=1.7.0_55
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.vendor=Oracle Corporation
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.io.tmpdir=/tmp
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:java.compiler=
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:os.name=Linux
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:os.arch=amd64
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client
> environment:os.version=3.2.0-61-virtual
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717
> INFO main org.apache.zookeeper.ZooKeeper.lo
[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959152#comment-14959152 ] Mayuresh Gharat commented on KAFKA-2120: [~guozhang] The check is required to be done at each stage to see if we have crossed the maxBlockTime : 1) after fetching metadata. 2) after serialization of key. 3) after serialization of value. 4) after partitioning. > Add a request timeout to NetworkClient > -- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature >Reporter: Jiangjie Qin > Assignee: Mayuresh Gharat >Priority: Blocker > Fix For: 0.9.0.0 > > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, > KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, > KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, > KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch, > KAFKA-2120_2015-09-18_19:27:48.patch, KAFKA-2120_2015-09-28_16:13:02.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959196#comment-14959196 ] Mayuresh Gharat commented on KAFKA-2120: [~guozhang] Yes sure. Will submit a PR for it. > Add a request timeout to NetworkClient > -- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature >Reporter: Jiangjie Qin > Assignee: Mayuresh Gharat >Priority: Blocker > Fix For: 0.9.0.0 > > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, > KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, > KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, > KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch, > KAFKA-2120_2015-09-18_19:27:48.patch, KAFKA-2120_2015-09-28_16:13:02.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-2805) RecordAccumulator request timeout only enforced if partition leader is unknown
[ https://issues.apache.org/jira/browse/KAFKA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-2805: -- Assignee: Mayuresh Gharat > RecordAccumulator request timeout only enforced if partition leader is unknown > -- > > Key: KAFKA-2805 > URL: https://issues.apache.org/jira/browse/KAFKA-2805 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson > Assignee: Mayuresh Gharat > > From the user mailing list, the null check in batch expiration in > RecordAccumulator seems questionable: > https://github.com/apache/kafka/blob/ae5a5d7c08bb634576a414f6f2864c5b8a7e58a3/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L220. > > If this is correct behavior, it is probably worthwhile clarifying the purpose > of the check in a comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2805) RecordAccumulator request timeout only enforced if partition leader is unknown
[ https://issues.apache.org/jira/browse/KAFKA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999844#comment-14999844 ] Mayuresh Gharat commented on KAFKA-2805: Yes. That was the thinking. Also from the KIP-19 discussion it was decided that Request timeout will also be used when the batches in the accumulator that are ready but not drained due to metadata missing - we are reusing request timeout to timeout the batches in accumulator. When we say metadata is missing, it means that we don't have information about the leader. > RecordAccumulator request timeout only enforced if partition leader is unknown > -- > > Key: KAFKA-2805 > URL: https://issues.apache.org/jira/browse/KAFKA-2805 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson > Assignee: Mayuresh Gharat > > From the user mailing list, the null check in batch expiration in > RecordAccumulator seems questionable: > https://github.com/apache/kafka/blob/ae5a5d7c08bb634576a414f6f2864c5b8a7e58a3/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L220. > > If this is correct behavior, it is probably worthwhile clarifying the purpose > of the check in a comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2805) RecordAccumulator request timeout only enforced if partition leader is unknown
[
https://issues.apache.org/jira/browse/KAFKA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999884#comment-14999884
]
Mayuresh Gharat commented on KAFKA-2805:
I did this experiment :
1) I created a topic called "kip-19-followup" on one of our test cluster with 1
partition and replication factor 1.
2) started the producer class :
./kafka-verifiable-producer.sh --topic kip-19-followup --broker-list (BROKER
LIST) --max-messages 92233736 --acks 1
3) The producer started producing and I could see :
{"partition":0,"offset":3164368,"time_ms":1447212192651,"name":"producer_send_success","topic":"kip-19-followup","class":"class
org.apache.kafka.tools.VerifiableProducer","value":"9547","key":null}
4) Then I killed the broker that was hosting this topic and I could see :
{"exception":"class
org.apache.kafka.common.errors.TimeoutException","time_ms":1447212028704,"name":"producer_send_error","topic":"kip-19-followup","message":"Batch
Expired","class":"class
org.apache.kafka.tools.VerifiableProducer","value":"107100","key":null}
The producer did not hang. My broker is not on trunk though.
Thanks,
Mayuresh
> RecordAccumulator request timeout only enforced if partition leader is unknown
> ------
>
> Key: KAFKA-2805
> URL: https://issues.apache.org/jira/browse/KAFKA-2805
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Mayuresh Gharat
>
> When no brokers are left in the cluster, the producer seems not to enforce
> the request timeout as expected.
> From the user mailing list, the null check in batch expiration in
> RecordAccumulator seems questionable:
> https://github.com/apache/kafka/blob/ae5a5d7c08bb634576a414f6f2864c5b8a7e58a3/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L220.
>
> If this is correct behavior, it is probably worthwhile clarifying the purpose
> of the check in a comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2805) RecordAccumulator request timeout not enforced when all brokers are gone
[ https://issues.apache.org/jira/browse/KAFKA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1424#comment-1424 ] Mayuresh Gharat commented on KAFKA-2805: [~hachikuji] My producer was on trunk when I ran this test. But my request timeout was set to 30 seconds instead of 1 sec. I will test this tomorrow again and keep a patch ready if required. Thanks, Mayuresh > RecordAccumulator request timeout not enforced when all brokers are gone > > > Key: KAFKA-2805 > URL: https://issues.apache.org/jira/browse/KAFKA-2805 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson > Assignee: Mayuresh Gharat > > When no brokers are left in the cluster, the producer seems not to enforce > the request timeout as expected. > From the user mailing list, the null check in batch expiration in > RecordAccumulator seems questionable: > https://github.com/apache/kafka/blob/ae5a5d7c08bb634576a414f6f2864c5b8a7e58a3/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L220. > > If this is correct behavior, it is probably worthwhile clarifying the purpose > of the check in a comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2805) RecordAccumulator request timeout not enforced when all brokers are gone
[ https://issues.apache.org/jira/browse/KAFKA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001024#comment-15001024 ] Mayuresh Gharat commented on KAFKA-2805: Hi Jason, I was debugging to find exactly what is happening. Is it ok if I can get back with a patch ( if necessary ) by EOD. If I am not able to finish this today, you can go ahead with this jira. Is it ok? Thanks, Mayuresh > RecordAccumulator request timeout not enforced when all brokers are gone > > > Key: KAFKA-2805 > URL: https://issues.apache.org/jira/browse/KAFKA-2805 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 0.9.0.0 > > > When no brokers are left in the cluster, the producer seems not to enforce > the request timeout as expected. > From the user mailing list, the null check in batch expiration in > RecordAccumulator seems questionable: > https://github.com/apache/kafka/blob/ae5a5d7c08bb634576a414f6f2864c5b8a7e58a3/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L220. > > If this is correct behavior, it is probably worthwhile clarifying the purpose > of the check in a comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2805) RecordAccumulator request timeout not enforced when all brokers are gone
[ https://issues.apache.org/jira/browse/KAFKA-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001559#comment-15001559 ] Mayuresh Gharat commented on KAFKA-2805: Yes. That s right. The check was put in place because the KIP-19 explicitly mentioned that it should timeout the batches only if the metadata is unavailable. I debugged this today and found the same reasoning. I was going to write the same explanation before I read this comment. My bad, I should have considered that accumulator will not drain the batches for NOT ready nodes and the batches will not be expired since the metadata is stale and has the leader present. I have uploaded a patch for removing the check. > RecordAccumulator request timeout not enforced when all brokers are gone > > > Key: KAFKA-2805 > URL: https://issues.apache.org/jira/browse/KAFKA-2805 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson > Assignee: Mayuresh Gharat >Priority: Blocker > Fix For: 0.9.0.0 > > > When no brokers are left in the cluster, the producer seems not to enforce > the request timeout as expected. > From the user mailing list, the null check in batch expiration in > RecordAccumulator seems questionable: > https://github.com/apache/kafka/blob/ae5a5d7c08bb634576a414f6f2864c5b8a7e58a3/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L220. > > If this is correct behavior, it is probably worthwhile clarifying the purpose > of the check in a comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1911) Log deletion on stopping replicas should be async
[
https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003232#comment-15003232
]
Mayuresh Gharat commented on KAFKA-1911:
[~geoffra] are you actively working on this. If not I would like to take it up
as I have recently started testing DeleteTopic usecases.
Thanks,
Mayuresh
> Log deletion on stopping replicas should be async
> -
>
> Key: KAFKA-1911
> URL: https://issues.apache.org/jira/browse/KAFKA-1911
> Project: Kafka
> Issue Type: Bug
> Components: log, replication
>Reporter: Joel Koshy
>Assignee: Geoff Anderson
> Labels: newbie++, newbiee
>
> If a StopReplicaRequest sets delete=true then we do a file.delete on the file
> message sets. I was under the impression that this is fast but it does not
> seem to be the case.
> On a partition reassignment in our cluster the local time for stop replica
> took nearly 30 seconds.
> {noformat}
> Completed request:Name: StopReplicaRequest; Version: 0; CorrelationId: 467;
> ClientId: ;DeletePartitions: true; ControllerId: 1212; ControllerEpoch:
> 53 from
> client/...:45964;totalTime:29191,requestQueueTime:1,localTime:29190,remoteTime:0,responseQueueTime:0,sendTime:0
> {noformat}
> This ties up one API thread for the duration of the request.
> Specifically in our case, the queue times for other requests also went up and
> producers to the partition that was just deleted on the old leader took a
> while to refresh their metadata (see KAFKA-1303) and eventually ran out of
> retries on some messages leading to data loss.
> I think the log deletion in this case should be fully asynchronous although
> we need to handle the case when a broker may respond immediately to the
> stop-replica-request but then go down after deleting only some of the log
> segments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KAFKA-1911) Log deletion on stopping replicas should be async
[
https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat reassigned KAFKA-1911:
--
Assignee: Mayuresh Gharat (was: Geoff Anderson)
> Log deletion on stopping replicas should be async
> -
>
> Key: KAFKA-1911
> URL: https://issues.apache.org/jira/browse/KAFKA-1911
> Project: Kafka
> Issue Type: Bug
> Components: log, replication
>Reporter: Joel Koshy
> Assignee: Mayuresh Gharat
> Labels: newbie++, newbiee
>
> If a StopReplicaRequest sets delete=true then we do a file.delete on the file
> message sets. I was under the impression that this is fast but it does not
> seem to be the case.
> On a partition reassignment in our cluster the local time for stop replica
> took nearly 30 seconds.
> {noformat}
> Completed request:Name: StopReplicaRequest; Version: 0; CorrelationId: 467;
> ClientId: ;DeletePartitions: true; ControllerId: 1212; ControllerEpoch:
> 53 from
> client/...:45964;totalTime:29191,requestQueueTime:1,localTime:29190,remoteTime:0,responseQueueTime:0,sendTime:0
> {noformat}
> This ties up one API thread for the duration of the request.
> Specifically in our case, the queue times for other requests also went up and
> producers to the partition that was just deleted on the old leader took a
> while to refresh their metadata (see KAFKA-1303) and eventually ran out of
> retries on some messages leading to data loss.
> I think the log deletion in this case should be fully asynchronous although
> we need to handle the case when a broker may respond immediately to the
> stop-replica-request but then go down after deleting only some of the log
> segments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KAFKA-2937) Topics marked for delete in Zookeeper may become undeletable
[
https://issues.apache.org/jira/browse/KAFKA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat reassigned KAFKA-2937:
--
Assignee: Mayuresh Gharat
> Topics marked for delete in Zookeeper may become undeletable
>
>
> Key: KAFKA-2937
> URL: https://issues.apache.org/jira/browse/KAFKA-2937
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Mayuresh Gharat
>
> In our clusters, we occasionally see topics marked for delete, but never
> actually deleted. It may be due to brokers being restarted while tests were
> running, but further restarts of Kafka dont fix the problem. The topics
> remain marked for delete in Zookeeper.
> Topic describe shows:
> {quote}
> Topic:testtopic PartitionCount:1ReplicationFactor:3 Configs:
> Topic: testtopicPartition: 0Leader: noneReplicas: 3,4,0
> Isr:
> {quote}
> Kafka logs show:
> {quote}
> 2015-12-02 15:53:30,152] ERROR Controller 2 epoch 213 initiated state change
> of replica 3 for partition [testtopic,0] from OnlineReplica to OfflineReplica
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: Failed to change state of replica 3
> for partition [testtopic,0] since the leader and isr path in zookeeper is
> empty
> at
> kafka.controller.ReplicaStateMachine.handleStateChange(ReplicaStateMachine.scala:269)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:322)
> at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:978)
> at
> kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:114)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:342)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:334)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at
> kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:334)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:367)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:313)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:312)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:312)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:431)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:403)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply$mcV$sp(TopicDeletionManager.scala:403)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:397)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {quote}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2948) Kafka producer does not cope well with topic deletions
[ https://issues.apache.org/jira/browse/KAFKA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047220#comment-15047220 ] Mayuresh Gharat commented on KAFKA-2948: Adding TTL would mean another user exposed config. Can we not use the number of times we got "UNKNOWN_TOPIC_OR_PARTITION" and then get rid of the topic. > Kafka producer does not cope well with topic deletions > -- > > Key: KAFKA-2948 > URL: https://issues.apache.org/jira/browse/KAFKA-2948 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.9.0.0 >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram > > Kafka producer gets metadata for topics when send is invoked and thereafter > it attempts to keep the metadata up-to-date without any explicit requests > from the client. This works well in static environments, but when topics are > added or deleted, list of topics in Metadata grows but never shrinks. Apart > from being a memory leak, this results in constant requests for metadata for > deleted topics. > We are running into this issue with the Confluent REST server where topic > deletion from tests are filling up logs with warnings about unknown topics. > Auto-create is turned off in our Kafka cluster. > I am happy to provide a fix, but am not sure what the right fix is. Does it > make sense to remove topics from the metadata list when > UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding > sends? It doesn't look very straightforward to do this, so any alternative > suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()
Mayuresh Gharat created KAFKA-2975: -- Summary: The newtorkClient should request a metadata update after it gets an error in the handleResponse() Key: KAFKA-2975 URL: https://issues.apache.org/jira/browse/KAFKA-2975 Project: Kafka Issue Type: Bug Components: clients Reporter: Mayuresh Gharat Assignee: Mayuresh Gharat Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean tie the batches sitting in the accumulator will expire and the mirror makers die. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()
[ https://issues.apache.org/jira/browse/KAFKA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2975: --- Description: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss. To overcome this we need to refresh the metadata after 3). Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. was: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. > The newtorkClient should request a metadata update after it gets an error in > the handleResponse() > - > > Key: KAFKA-2975 > URL: https://issues.apache.org/jira/browse/KAFKA-2975 > Project: Kafka > Issue Type: Bug > Components: clients > Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Currently in data pipeline, > 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is > set to 5 min > 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries > torefresh its Metadata. > 3) It gets LeaderNotAvailableException, may be because the topic is not > created yet. > 4) Now its metadata does not have any information about that topic. > 5) It will wait for 5 min to do the next refresh. > 6) In the mean time the batches sitting in the accumulator will expire and > the mirror makers die to avoid data loss. > To overcome this we need to refresh the metadata after 3). > Well there is an alternative solution to have the metadataExpiry set to be > less then requestTimeout, but this will mean we make more metadataRequest > over the wire in normal scenario as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()
[ https://issues.apache.org/jira/browse/KAFKA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2975: --- Description: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean time the batches sitting in the accumulator will expire and the mirror makers die to avoid data loss. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. was: Currently in data pipeline, 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is set to 5 min 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries torefresh its Metadata. 3) It gets LeaderNotAvailableException, may be because the topic is not created yet. 4) Now its metadata does not have any information about that topic. 5) It will wait for 5 min to do the next refresh. 6) In the mean tie the batches sitting in the accumulator will expire and the mirror makers die. To overcome this we need to refresh the metadata after 3) before the timeout kicks in. Well there is an alternative solution to have the metadataExpiry set to be less then requestTimeout, but this will mean we make more metadataRequest over the wire in normal scenario as well. > The newtorkClient should request a metadata update after it gets an error in > the handleResponse() > - > > Key: KAFKA-2975 > URL: https://issues.apache.org/jira/browse/KAFKA-2975 > Project: Kafka > Issue Type: Bug > Components: clients > Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Currently in data pipeline, > 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is > set to 5 min > 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries > torefresh its Metadata. > 3) It gets LeaderNotAvailableException, may be because the topic is not > created yet. > 4) Now its metadata does not have any information about that topic. > 5) It will wait for 5 min to do the next refresh. > 6) In the mean time the batches sitting in the accumulator will expire and > the mirror makers die to avoid data loss. > To overcome this we need to refresh the metadata after 3) before the timeout > kicks in. > Well there is an alternative solution to have the metadataExpiry set to be > less then requestTimeout, but this will mean we make more metadataRequest > over the wire in normal scenario as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2976) Mirror maker dies if we delete a topic from destination cluster
Mayuresh Gharat created KAFKA-2976: -- Summary: Mirror maker dies if we delete a topic from destination cluster Key: KAFKA-2976 URL: https://issues.apache.org/jira/browse/KAFKA-2976 Project: Kafka Issue Type: Bug Components: clients Reporter: Mayuresh Gharat Assignee: Mayuresh Gharat In datapipeline, 1) Suppose the Mirror Maker is producing to a cluster with Topic T and has 128 partitions (Partition 0 to Partition 127) . The default setting on creation of a new topic on that cluster is 8 partitions. 2) After we delete the topic, the topic gets recreated with 8 partitions (Partition 0 to Partition 7). 3) The RecordAccumulator has batches for partitions from 9 to 127. Those batches get expired and the mirror makers will die to avoid data loss. We need a way to reassign those batches (batches for Partition 9 top Partition 127) in the RecordAccumulator to the newly created Topic T with 8 partitions (Partition 0 to Partition 7). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2975) The newtorkClient should request a metadata update after it gets an error in the handleResponse()
[ https://issues.apache.org/jira/browse/KAFKA-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat resolved KAFKA-2975. Resolution: Not A Problem > The newtorkClient should request a metadata update after it gets an error in > the handleResponse() > - > > Key: KAFKA-2975 > URL: https://issues.apache.org/jira/browse/KAFKA-2975 > Project: Kafka > Issue Type: Bug > Components: clients > Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat > > Currently in data pipeline, > 1) Lets say Mirror Maker requestTimeout is set to 2 min and metadataExpiry is > set to 5 min > 2) We delete a topic, the Mirror Maker get UNKNOWN_TOPIC_PARTITION and tries > torefresh its Metadata. > 3) It gets LeaderNotAvailableException, may be because the topic is not > created yet. > 4) Now its metadata does not have any information about that topic. > 5) It will wait for 5 min to do the next refresh. > 6) In the mean time the batches sitting in the accumulator will expire and > the mirror makers die to avoid data loss. > To overcome this we need to refresh the metadata after 3). > Well there is an alternative solution to have the metadataExpiry set to be > less then requestTimeout, but this will mean we make more metadataRequest > over the wire in normal scenario as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1860) File system errors are not detected unless Kafka tries to write
[
https://issues.apache.org/jira/browse/KAFKA-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062409#comment-15062409
]
Mayuresh Gharat commented on KAFKA-1860:
Cool.
> File system errors are not detected unless Kafka tries to write
> ---
>
> Key: KAFKA-1860
> URL: https://issues.apache.org/jira/browse/KAFKA-1860
> Project: Kafka
> Issue Type: Bug
>Reporter: Guozhang Wang
> Assignee: Mayuresh Gharat
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-1860.patch
>
>
> When the disk (raid with caches dir) dies on a Kafka broker, typically the
> filesystem gets mounted into read-only mode, and hence when Kafka tries to
> read the disk, they'll get a FileNotFoundException with the read-only errno
> set (EROFS).
> However, as long as there is no produce request received, hence no writes
> attempted on the disks, Kafka will not exit on such FATAL error (when the
> disk starts working again, Kafka might think some files are gone while they
> will reappear later as raid comes back online). Instead it keeps spilling
> exceptions like:
> {code}
> 2015/01/07 09:47:41.543 ERROR [KafkaScheduler] [kafka-scheduler-1]
> [kafka-server] [] Uncaught exception in scheduled task
> 'kafka-recovery-point-checkpoint'
> java.io.FileNotFoundException:
> /export/content/kafka/i001_caches/recovery-point-offset-checkpoint.tmp
> (Read-only file system)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:206)
> at java.io.FileOutputStream.(FileOutputStream.java:156)
> at kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:37)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2887) TopicMetadataRequest creates topic if it does not exist
[ https://issues.apache.org/jira/browse/KAFKA-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063131#comment-15063131 ] Mayuresh Gharat commented on KAFKA-2887: [~hachikuji] just wanted to be sure what you meant by leaving the current TopicMetadataRequest unchanged, is that it will not be creating the topic in future. Am I right? This is very important when we are dealing with deleting topics in a pipeline. > TopicMetadataRequest creates topic if it does not exist > --- > > Key: KAFKA-2887 > URL: https://issues.apache.org/jira/browse/KAFKA-2887 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.2.0 > Environment: Centos6, Java 1.7.0_75 >Reporter: Andrew Winterman >Priority: Minor > > We wired up a probe http endpoint to make TopicMetadataRequests with a > possible topic name. If no topic was found, we expected an empty response. > However if we asked for the same topic twice, it would exist the second time! > I think this is a bug because the purpose of the TopicMetadaRequest is to > provide information about the cluster, not mutate it. I can provide example > code if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2887) TopicMetadataRequest creates topic if it does not exist
[ https://issues.apache.org/jira/browse/KAFKA-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063131#comment-15063131 ] Mayuresh Gharat edited comment on KAFKA-2887 at 12/18/15 12:02 AM: --- [~hachikuji] just wanted to be sure what you meant by leaving the current TopicMetadataRequest unchanged, is that it will not be creating the topic in future if the topic is not present and this would require code change on server. Am I right? This is very important when we are dealing with deleting topics in a pipeline. was (Author: mgharat): [~hachikuji] just wanted to be sure what you meant by leaving the current TopicMetadataRequest unchanged, is that it will not be creating the topic in future. Am I right? This is very important when we are dealing with deleting topics in a pipeline. > TopicMetadataRequest creates topic if it does not exist > --- > > Key: KAFKA-2887 > URL: https://issues.apache.org/jira/browse/KAFKA-2887 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.8.2.0 > Environment: Centos6, Java 1.7.0_75 >Reporter: Andrew Winterman >Priority: Minor > > We wired up a probe http endpoint to make TopicMetadataRequests with a > possible topic name. If no topic was found, we expected an empty response. > However if we asked for the same topic twice, it would exist the second time! > I think this is a bug because the purpose of the TopicMetadaRequest is to > provide information about the cluster, not mutate it. I can provide example > code if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2937) Topics marked for delete in Zookeeper may become undeletable
[
https://issues.apache.org/jira/browse/KAFKA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064365#comment-15064365
]
Mayuresh Gharat commented on KAFKA-2937:
I ran a test with few commits behind trunk HEAD :
1) Started the kafka cluster.
2) Produced data to topic T for quite some time so that enough data is
accumulated on kafka brokers.
3) Delete a topic and immediately shutdown the cluster.
4) I could see the topic T under /admin/delete_topics.
5) After I restarted the cluster, the topic T was deleted.
> Topics marked for delete in Zookeeper may become undeletable
>
>
> Key: KAFKA-2937
> URL: https://issues.apache.org/jira/browse/KAFKA-2937
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Mayuresh Gharat
>
> In our clusters, we occasionally see topics marked for delete, but never
> actually deleted. It may be due to brokers being restarted while tests were
> running, but further restarts of Kafka dont fix the problem. The topics
> remain marked for delete in Zookeeper.
> Topic describe shows:
> {quote}
> Topic:testtopic PartitionCount:1ReplicationFactor:3 Configs:
> Topic: testtopicPartition: 0Leader: noneReplicas: 3,4,0
> Isr:
> {quote}
> Kafka logs show:
> {quote}
> 2015-12-02 15:53:30,152] ERROR Controller 2 epoch 213 initiated state change
> of replica 3 for partition [testtopic,0] from OnlineReplica to OfflineReplica
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: Failed to change state of replica 3
> for partition [testtopic,0] since the leader and isr path in zookeeper is
> empty
> at
> kafka.controller.ReplicaStateMachine.handleStateChange(ReplicaStateMachine.scala:269)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:322)
> at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:978)
> at
> kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:114)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:342)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:334)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at
> kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:334)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:367)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:313)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:312)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:312)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:431)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:403)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply$mcV$sp(TopicDeletionManager.scala:403)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:397)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:397)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {quote}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2937) Topics marked for delete in Zookeeper may become undeletable
[
https://issues.apache.org/jira/browse/KAFKA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064365#comment-15064365
]
Mayuresh Gharat edited comment on KAFKA-2937 at 12/18/15 6:21 PM:
--
I ran a test with few commits behind trunk HEAD :
1) Started the kafka cluster.
2) Produced data to topic T for quite some time so that enough data is
accumulated on kafka brokers.
3) Delete a topic and immediately shutdown the cluster.
4) I could see the topic T under /admin/delete_topics.
5) After I restarted the cluster, the topic T was deleted.
There might be a race condition here, were controller might have gone down
after it wrote to zookeeper but under removed the topic under
/admin/deleted_topics. I will have to investigate more on this.
[~rsivaram]What is the version of kafka that you are running?
was (Author: mgharat):
I ran a test with few commits behind trunk HEAD :
1) Started the kafka cluster.
2) Produced data to topic T for quite some time so that enough data is
accumulated on kafka brokers.
3) Delete a topic and immediately shutdown the cluster.
4) I could see the topic T under /admin/delete_topics.
5) After I restarted the cluster, the topic T was deleted.
> Topics marked for delete in Zookeeper may become undeletable
>
>
> Key: KAFKA-2937
> URL: https://issues.apache.org/jira/browse/KAFKA-2937
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Mayuresh Gharat
>
> In our clusters, we occasionally see topics marked for delete, but never
> actually deleted. It may be due to brokers being restarted while tests were
> running, but further restarts of Kafka dont fix the problem. The topics
> remain marked for delete in Zookeeper.
> Topic describe shows:
> {quote}
> Topic:testtopic PartitionCount:1ReplicationFactor:3 Configs:
> Topic: testtopicPartition: 0Leader: noneReplicas: 3,4,0
> Isr:
> {quote}
> Kafka logs show:
> {quote}
> 2015-12-02 15:53:30,152] ERROR Controller 2 epoch 213 initiated state change
> of replica 3 for partition [testtopic,0] from OnlineReplica to OfflineReplica
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: Failed to change state of replica 3
> for partition [testtopic,0] since the leader and isr path in zookeeper is
> empty
> at
> kafka.controller.ReplicaStateMachine.handleStateChange(ReplicaStateMachine.scala:269)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:322)
> at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:978)
> at
> kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:114)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:342)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:334)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at
> kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:334)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:367)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:313)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:312)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:312)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:431)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:403)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anon
[jira] [Created] (KAFKA-3013) Display the topic-partition in the exception message for expired batches in recordAccumulator
Mayuresh Gharat created KAFKA-3013: -- Summary: Display the topic-partition in the exception message for expired batches in recordAccumulator Key: KAFKA-3013 URL: https://issues.apache.org/jira/browse/KAFKA-3013 Project: Kafka Issue Type: Improvement Components: clients Reporter: Mayuresh Gharat Assignee: Mayuresh Gharat Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2937) Topics marked for delete in Zookeeper may become undeletable
[
https://issues.apache.org/jira/browse/KAFKA-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070294#comment-15070294
]
Mayuresh Gharat commented on KAFKA-2937:
[~rsivaram] This can occur if during delete topic, the controller writes the
updated LeaderISR to zookeeper and dies and a new controller is elected and
since the /admin/delete_topics/ path contains the topic, the deleteTopic is
retried and when the controller tries to send the stopReplicaRequest with
deletePartition = false when it tries to remove ReplicafromISR and since there
is no LeaderISR, the above exception is thrown.
I am thinking if while deleting a topic, is it necessary for the check :
if (leaderAndIsrIsEmpty)
throw new StateChangeFailedException(
"Failed to change state of replica %d for partition %s since the
leader and isr path in zookeeper is empty"
.format(replicaId, topicAndPartition))
On a side note did you see a log line "Retrying delete topic for topic
since replicaswere not successfully deleted".
> Topics marked for delete in Zookeeper may become undeletable
>
>
> Key: KAFKA-2937
> URL: https://issues.apache.org/jira/browse/KAFKA-2937
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.9.0.0
> Reporter: Rajini Sivaram
>Assignee: Mayuresh Gharat
>
> In our clusters, we occasionally see topics marked for delete, but never
> actually deleted. It may be due to brokers being restarted while tests were
> running, but further restarts of Kafka dont fix the problem. The topics
> remain marked for delete in Zookeeper.
> Topic describe shows:
> {quote}
> Topic:testtopic PartitionCount:1ReplicationFactor:3 Configs:
> Topic: testtopicPartition: 0Leader: noneReplicas: 3,4,0
> Isr:
> {quote}
> Kafka logs show:
> {quote}
> 2015-12-02 15:53:30,152] ERROR Controller 2 epoch 213 initiated state change
> of replica 3 for partition [testtopic,0] from OnlineReplica to OfflineReplica
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: Failed to change state of replica 3
> for partition [testtopic,0] since the leader and isr path in zookeeper is
> empty
> at
> kafka.controller.ReplicaStateMachine.handleStateChange(ReplicaStateMachine.scala:269)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> kafka.controller.ReplicaStateMachine$$anonfun$handleStateChanges$2.apply(ReplicaStateMachine.scala:114)
> at
> scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:322)
> at
> scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:978)
> at
> kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:114)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:342)
> at
> kafka.controller.TopicDeletionManager$$anonfun$startReplicaDeletion$2.apply(TopicDeletionManager.scala:334)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at
> kafka.controller.TopicDeletionManager.startReplicaDeletion(TopicDeletionManager.scala:334)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onPartitionDeletion(TopicDeletionManager.scala:367)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:313)
> at
> kafka.controller.TopicDeletionManager$$anonfun$kafka$controller$TopicDeletionManager$$onTopicDeletion$2.apply(TopicDeletionManager.scala:312)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> at
> kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$onTopicDeletion(TopicDeletionManager.scala:312)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:431)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1$$anonfun$apply$mcV$sp$4.apply(TopicDeletionManager.scala:403)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply$mcV$sp(TopicDeletionManager.scala:403)
> at
> kafka.controller.TopicDeletionManager$DeleteTopicsThread$$a
