Re: Contributor List

[Bill Bejeck]

My Jira user name is bbejeck.

Thanks Gwen!

-Bill

On Fri, Sep 4, 2015 at 7:46 PM, Gwen Shapira  wrote:

> Thank you!
>
> Please create a Jira user if you didn't do so already, and let me know your
> user name. I'll add you to the list.
>
> On Fri, Sep 4, 2015 at 4:33 PM, Bill Bejeck  wrote:
>
> > Hi can i get added to the contributor list? I'd like to take crack at
> > KAFKA-2058 
> >
> > Thanks!
> >
> > Bill Bejeck
> >
>

[jira] [Commented] (KAFKA-1070) Auto-assign node id

[Adrian Muraru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731941#comment-14731941
 ] 

Adrian Muraru commented on KAFKA-1070:
--

Saw it, that's why I raised the flag here, the patch in trunk seems broken with 
the .orig file added. A new jira issue might be needed to fix it

> Auto-assign node id
> ---
>
> Key: KAFKA-1070
> URL: https://issues.apache.org/jira/browse/KAFKA-1070
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
>  Labels: usability
> Fix For: 0.8.3
>
> Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch, 
> KAFKA-1070_2014-07-22_11:34:18.patch, KAFKA-1070_2014-07-24_20:58:17.patch, 
> KAFKA-1070_2014-07-24_21:05:33.patch, KAFKA-1070_2014-08-21_10:26:20.patch, 
> KAFKA-1070_2014-11-20_10:50:04.patch, KAFKA-1070_2014-11-25_20:29:37.patch, 
> KAFKA-1070_2015-01-01_17:39:30.patch, KAFKA-1070_2015-01-12_10:46:54.patch, 
> KAFKA-1070_2015-01-12_18:30:17.patch
>
>
> It would be nice to have Kafka brokers auto-assign node ids rather than 
> having that be a configuration. Having a configuration is irritating because 
> (1) you have to generate a custom config for each broker and (2) even though 
> it is in configuration, changing the node id can cause all kinds of bad 
> things to happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2072: Replace StopReplica Request/Respon...

[dajac]

GitHub user dajac opened a pull request:

https://github.com/apache/kafka/pull/196

KAFKA-2072: Replace StopReplica Request/Response with their 
org.apache.kafka.common.requests equivalents



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dajac/kafka KAFKA-2072-part-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #196


commit b064d2f9a0331789290b60dc3fa7fe83084f0167
Author: David Jacot 
Date:   2015-08-13T17:15:18Z

Replace k.a.StopReplicaRequest and k.a.StopReplicaResponse in KafkaApis by 
their org.apache.kafka.common.requests equivalents.

commit 41eecb1b7de296d3bf5a2fd20380280f8cf441f0
Author: David Jacot 
Date:   2015-08-14T18:53:32Z

Remove k.a.StopReplicaRequest and k.a.StopReplicaResponse.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732081#comment-14732081
 ] 

ASF GitHub Bot commented on KAFKA-2072:
---

GitHub user dajac opened a pull request:

https://github.com/apache/kafka/pull/196

KAFKA-2072: Replace StopReplica Request/Response with their 
org.apache.kafka.common.requests equivalents



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dajac/kafka KAFKA-2072-part-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #196


commit b064d2f9a0331789290b60dc3fa7fe83084f0167
Author: David Jacot 
Date:   2015-08-13T17:15:18Z

Replace k.a.StopReplicaRequest and k.a.StopReplicaResponse in KafkaApis by 
their org.apache.kafka.common.requests equivalents.

commit 41eecb1b7de296d3bf5a2fd20380280f8cf441f0
Author: David Jacot 
Date:   2015-08-14T18:53:32Z

Remove k.a.StopReplicaRequest and k.a.StopReplicaResponse.




> Add StopReplica request/response to o.a.k.common.requests and replace the 
> usage in core module
> --
>
> Key: KAFKA-2072
> URL: https://issues.apache.org/jira/browse/KAFKA-2072
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Gwen Shapira
>Assignee: David Jacot
>
> Add StopReplica request/response to o.a.k.common.requests and replace the 
> usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

[David Jacot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot updated KAFKA-2072:
---
Status: Patch Available  (was: In Progress)

> Add StopReplica request/response to o.a.k.common.requests and replace the 
> usage in core module
> --
>
> Key: KAFKA-2072
> URL: https://issues.apache.org/jira/browse/KAFKA-2072
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Gwen Shapira
>Assignee: David Jacot
>
> Add StopReplica request/response to o.a.k.common.requests and replace the 
> usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732096#comment-14732096
 ] 

Jay Kreps commented on KAFKA-2510:
--

@gwen I think maybe I'm confused. The discussion in the ticket is all about 
losing data or losing the cluster. But isn't this prevented by controlled 
shutdown?

> Prevent broker from re-replicating / losing data due to disk misconfiguration
> -
>
> Key: KAFKA-2510
> URL: https://issues.apache.org/jira/browse/KAFKA-2510
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> Currently Kafka assumes that whatever it sees in the data directory is the 
> correct state of the data.
> This means that if an admin mistakenly configures Chef to use wrong data 
> directory, one of the following can happen:
> 1. The broker will replicate a bunch of partitions and take over the network
> 2. If you did this to enough brokers, you can lose entire topics and 
> partitions.
> We have information about existing topics, partitions and their ISR in 
> zookeeper.
> We need a mode in which if a broker starts, is in ISR for a partition and 
> doesn't have any data or directory for the partition, the broker will issue a 
> huge ERROR in the log and refuse to do anything for the partition.
> [~fpj] worked on the problem for ZK and had some ideas on what is required 
> here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732096#comment-14732096
 ] 

Jay Kreps edited comment on KAFKA-2510 at 9/5/15 8:23 PM:
--

[~gwenshap] I think maybe I'm confused. The discussion in the ticket is all 
about losing data or losing the cluster. But isn't this prevented by controlled 
shutdown?


was (Author: jkreps):
@gwen I think maybe I'm confused. The discussion in the ticket is all about 
losing data or losing the cluster. But isn't this prevented by controlled 
shutdown?

> Prevent broker from re-replicating / losing data due to disk misconfiguration
> -
>
> Key: KAFKA-2510
> URL: https://issues.apache.org/jira/browse/KAFKA-2510
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> Currently Kafka assumes that whatever it sees in the data directory is the 
> correct state of the data.
> This means that if an admin mistakenly configures Chef to use wrong data 
> directory, one of the following can happen:
> 1. The broker will replicate a bunch of partitions and take over the network
> 2. If you did this to enough brokers, you can lose entire topics and 
> partitions.
> We have information about existing topics, partitions and their ISR in 
> zookeeper.
> We need a mode in which if a broker starts, is in ISR for a partition and 
> doesn't have any data or directory for the partition, the broker will issue a 
> huge ERROR in the log and refuse to do anything for the partition.
> [~fpj] worked on the problem for ZK and had some ideas on what is required 
> here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2520) log.cleaner.min.cleanable.ratio 1 The number of background threads to use for log cleaning.

[Chris Hiestand (JIRA)]

Chris Hiestand created KAFKA-2520:
-

 Summary: log.cleaner.min.cleanable.ratio   1  The number of 
background threads to use for log cleaning.
 Key: KAFKA-2520
 URL: https://issues.apache.org/jira/browse/KAFKA-2520
 Project: Kafka
  Issue Type: Bug
Reporter: Chris Hiestand


Documentation has copy/paste error. log.cleaner.min.cleanable.ratio is 
mentioned twice, but I believe the second instance should read 
log.cleaner.threads



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2521) Documentation: Unclear cleanup policy on wiki

[Chris Hiestand (JIRA)]

Chris Hiestand created KAFKA-2521:
-

 Summary: Documentation: Unclear cleanup policy on wiki
 Key: KAFKA-2521
 URL: https://issues.apache.org/jira/browse/KAFKA-2521
 Project: Kafka
  Issue Type: Bug
Reporter: Chris Hiestand


"The default policy for handling log tails. Can be either delete or dedupe."

The other documentation says this should either be delete or compact. 
 so I'm guessing this wiki 
needs to be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[jin xing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing reassigned KAFKA-2394:
---

Assignee: jin xing

> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: jin xing
>Priority: Minor
>  Labels: newbie
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Alexander Jipa (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732142#comment-14732142
 ] 

Alexander Jipa commented on KAFKA-1194:
---

Hello, could you please consider the proposed patch from [~mpoindexter]? This 
issue is really annoying for every Windows user...

The build seems fine:
asfbot commented 2 days ago
kafka-trunk-git-pr #346 SUCCESS
This pull request looks good

> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.9.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 1516723
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>  at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
>  at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
>  at scala.collection.immutable.List.foreach(List.scala:76)
>  at kafka.log.Log.deleteOldSegments(Log.scala:418)
>  at 
> kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>  at 
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
>  at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>  at 
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
>  at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>  at kafka.log.LogManager.cleanupLogs(LogManager.scala:314)
>  at 
> kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143)
>  at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
>  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:724)
> I think this error happens because kafka tries to rename the log file when it 
> is still opened.  So we should close the file first before rename.
> The index file uses a special data structure, the MappedByteBuffer. Javadoc 
> describes it as:
> A mapped byte buffer and the file mapping that it represents remain valid 
> until the buffer itself is garbage-collected.
> Fortunately, I find a forceUnmap function in kafka code, and perhaps it can 
> be used to free the MappedByteBuffer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)