Developer setup

2013-05-10 Thread Andrea Gazzarini 

Hi all,
I changed the Developer Setup page 
(https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup) a 
little bit. It is formally complete so I published it.

I say formally because there are these little issues:

*1) Unit tests*
Within an Eclipse workspace created using the first approach (i.e. 
Eclipse without sbt plugin) I have a failure:


kafka.log.LogTest.testTimeBasedLogRoll(kafka.log.LogTest)
junit.framework.AssertionFailedError: There should still be exactly one 
segment. expected:1 but was:2

at junit.framework.Assert.fail(Assert.java:47)
...
at kafka.log.LogTest.testTimeBasedLogRoll(LogTest.scala:80)
...

and an error

kafka.producer.AsyncProducerTest.testFailedSendRetryLogic(kafka.producer.AsyncProducerTest)
kafka.common.FailedToSendMessageException: Failed to send messages after 
3 tries.
at 
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:88)
at 
kafka.producer.AsyncProducerTest.testFailedSendRetryLogic(AsyncProducerTest.scala:439)

...

The other 158 tests are OK.

*2) Source folders*
Using the second approach (Eclipse with sbt plugin), the sbt plugin 
creates Eclipse projects with default source folders.
For core module this is not ok because tests are in test/unit and 
test/other. I suppose the core/build.sbt needs to be changed.


*3) Intellij**IDEA *
At the moment I don't have Intellij installed on my laptop so I left the 
content of this section ontouched.


Any kind of feedback is warmly welcome

Best,
Andrea

On 05/06/2013 03:08 AM, Jun Rao wrote:

Andrea,

Yes, you can update the wiki directly. Thanks for helping out.

Jun


On Sun, May 5, 2013 at 1:17 AM, Andrea Gazzarini andrea.gazzar...@gmail.com

wrote:
Thanks Jun,
I'm trying to setup a fresh dev environment and I met some problems.

Now, while I was changing the wiki page (Developer Setup) I inadvertently
saved the page, sending the notification about the changes, sorry about
that.

I believe the change I made is right (I changed just the line about the
Eclipse / Scala plugin setup by specifying the scala plugin version and a
recommendation about the corrispondence between scala plugin and eclipse
version) but I didn't finish to setup my ide, and therefore to change the
page;

I would like to complete the setup and update the wiki page accordingly
with eventual issues I get but, in case is not possible to proceed in this
way (by directly updating the wiki page) I apologize, please let me know
the right way to contribute.

Cheers,
Andrea


On 04/24/2013 04:56 PM, Jun Rao wrote:


We have moved to git. Updated the wiki.

Thanks,

Jun


On Wed, Apr 24, 2013 at 6:41 AM, Andrea Gazzarini 
andrea.gazzar...@gmail.com wrote:

  Hi, looking at these pages

http://kafka.apache.org/code.htmlhttp://kafka.apache.org/code.**html
http://kafka.apache.org/code.**html http://kafka.apache.org/code.html
https://cwiki.apache.org/confluence/display/KAFKA/
Developer+Setuphttps://cwiki.apache.org/**confluence/display/KAFKA/**Developer+Setup
https://cwiki.**apache.org/confluence/display/**KAFKA/Developer+Setuphttps://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup

seems that both Git and SVN are used...is that true? I'm more confident
with SVN but the front page (coding) in the web site reports a Git URL
while the SVN info are a little bit hidden in a wiki subpage...

Best,
Andrea

[jira] [Commented] (KAFKA-901) Kafka server can become unavailable if clients send several metadata requests

2013-05-10 Thread Joel Koshy (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13654667#comment-13654667
 ] 

Joel Koshy commented on KAFKA-901:
--

Haven't looked at the patch yet, but went through the overview. An alternate 
approach that we may want to consider is to maintain a metadata cache at every 
broker. The cache can be kept consistent by having the controller send a (new) 
update-metadata request to all brokers whenever it sends out a leaderAndIsr 
request. A new request type would avoid needing to overload the leader and 
isr request.

This would help avoid the herd effect of multiple clients flooding the 
controller with metadata requests (although these requests should return 
quickly with your patch).


 Kafka server can become unavailable if clients send several metadata requests
 -

 Key: KAFKA-901
 URL: https://issues.apache.org/jira/browse/KAFKA-901
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 0.8
Reporter: Neha Narkhede
Assignee: Neha Narkhede
Priority: Blocker
 Attachments: metadata-request-improvement.patch


 Currently, if a broker is bounced without controlled shutdown and there are 
 several clients talking to the Kafka cluster, each of the clients realize the 
 unavailability of leaders for some partitions. This leads to several metadata 
 requests sent to the Kafka brokers. Since metadata requests are pretty slow, 
 all the I/O threads quickly become busy serving the metadata requests. This 
 leads to a full request queue, that stalls handling of finished responses 
 since the same network thread handles requests as well as responses. In this 
 situation, clients timeout on metadata requests and send more metadata 
 requests. This quickly makes the Kafka cluster unavailable. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-876) Produce request: Leader not local for partition [test,0] on broker 0

2013-05-10 Thread Rob Withers (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655080#comment-13655080
 ] 

Rob Withers commented on KAFKA-876:
---

Jun, I changed the properties for the brokers to use backward slashes, as you 
suggested.  As in \tmp\kafka0-logs.  This worked and I was able to produce data 
for awhile.  It seems the bug this issue is about is fixed and could be closed. 
 

However, zookeeper accepts forward slashes (/tmp/zookeeper) and is successful 
with this.  Kafka should be agnostic as well, I think - to make it all it will 
be.

After producing for awhile, it logs this FATAL error, in one of the brokers, 
and both brokers are shutdown:

{2013-05-10 18:23:57,689} FATAL [highwatermark-checkpoint-thread1] 
(Logging.scala:109) - Attempt to swap the new high watermark file with the old 
one failed

I will log a new issue.

 Produce request: Leader not local for partition [test,0] on broker 0 
 -

 Key: KAFKA-876
 URL: https://issues.apache.org/jira/browse/KAFKA-876
 Project: Kafka
  Issue Type: Bug
  Components: clients, replication
Affects Versions: 0.8
 Environment: Windows
Reporter: Yin Yin
Assignee: Neha Narkhede
Priority: Blocker

 Follow the quick start to open zookeeper, one broker, one producer and one 
 consumer. In the producer console, there is an LeaderNotAvailableException 
 for the first message, and the broker complains Produce request: Leader not 
 local for partition [test,0] on broker 0 for all following messages. 
 Kafka-List-Topic shows [2013-04-25 10:21:24,689] INFO zookeeper state 
 changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) topic: test 
 partition: 0leader: 0   replicas: 0 isr: 0. With 
 --unavailable-partitions option, it doesn't list any topic.
 =Broker Log=
 Set JMX_PORT to default value : 
 C:\Projects\Kafka\kafka\bin\..
 log4j:ERROR Failed to rename [server.log] to [server.log.2013-04-25-09].
 [2013-04-25 10:08:49,531] INFO Verifying properties 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property socket.send.buffer.bytes is 
 overridden to 1048576 (kafka.utils.VerifiablePropert
 ies)
 [2013-04-25 10:08:49,578] INFO Property socket.request.max.bytes is 
 overridden to 104857600 (kafka.utils.VerifiablePrope
 rties)
 [2013-04-25 10:08:49,578] INFO Property log.dir is overridden to 
 /tmp/kafka-logs (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property log.cleanup.interval.mins is 
 overridden to 1 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property log.retention.hours is overridden to 
 168 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property num.io.threads is overridden to 2 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property broker.id is overridden to 0 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] WARN Property kafka.csv.metrics.reporter.enabled is 
 not valid (kafka.utils.VerifiablePropertie
 s)
 [2013-04-25 10:08:49,578] INFO Property port is overridden to 9092 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property log.flush.interval.messages is 
 overridden to 1 (kafka.utils.VerifiableProper
 ties)
 [2013-04-25 10:08:49,578] INFO Property zk.connection.timeout.ms is 
 overridden to 100 (kafka.utils.VerifiablePropert
 ies)
 [2013-04-25 10:08:49,578] WARN Property kafka.metrics.reporters is not valid 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] WARN Property kafka.csv.metrics.dir is not valid 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property log.flush.interval.ms is overridden 
 to 1000 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] WARN Property kafka.metrics.polling.interval.secs 
 is not valid (kafka.utils.VerifiableProperti
 es)
 [2013-04-25 10:08:49,578] INFO Property num.network.threads is overridden to 
 2 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property socket.receive.buffer.bytes is 
 overridden to 1048576 (kafka.utils.VerifiableProp
 erties)
 [2013-04-25 10:08:49,578] INFO Property log.segment.bytes is overridden to 
 536870912 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,578] INFO Property zk.connect is overridden to 
 localhost:2181 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,594] INFO Property num.partitions is overridden to 1 
 (kafka.utils.VerifiableProperties)
 [2013-04-25 10:08:49,609] INFO [Kafka Server 0], starting 
 (kafka.server.KafkaServer)
 [2013-04-25 10:08:49,625] INFO [Log Manager on Broker 0] Log directory 
 'C:\tmp\kafka-logs' not found, creating it. (kafk
 a.log.LogManager)
 [2013-04-25

[jira] [Created] (KAFKA-903) [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed

2013-05-10 Thread Rob Withers (JIRA) 
Rob Withers created KAFKA-903:
-

 Summary: [0.8.0 - windows]  FATAL - 
[highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the 
new high watermark file with the old one failed
 Key: KAFKA-903
 URL: https://issues.apache.org/jira/browse/KAFKA-903
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8
 Environment: Windows 7 with SP 1; jdk 7_0_17; scala-library-2.8.2, 
probably copied on 4/30. kafka-0.8, built current on 4/30.

-rwx--+ 1 reefedjib None   41123 Mar 19  2009 commons-cli-1.2.jar
-rwx--+ 1 reefedjib None   58160 Jan 11 13:45 commons-codec-1.4.jar
-rwx--+ 1 reefedjib None  575389 Apr 18 13:41 commons-collections-3.2.1.jar
-rwx--+ 1 reefedjib None  143847 May 21  2009 commons-compress-1.0.jar
-rwx--+ 1 reefedjib None   52543 Jan 11 13:45 commons-exec-1.1.jar
-rwx--+ 1 reefedjib None   57779 Jan 11 13:45 commons-fileupload-1.2.1.jar
-rwx--+ 1 reefedjib None  109043 Jan 20  2008 commons-io-1.4.jar
-rwx--+ 1 reefedjib None  279193 Jan 11 13:45 commons-lang-2.5.jar
-rwx--+ 1 reefedjib None   60686 Jan 11 13:45 commons-logging-1.1.1.jar
-rwx--+ 1 reefedjib None 1891110 Apr 18 13:41 guava-13.0.1.jar
-rwx--+ 1 reefedjib None  206866 Apr  7 21:24 jackson-core-2.1.4.jar
-rwx--+ 1 reefedjib None  232245 Apr  7 21:24 jackson-core-asl-1.9.12.jar
-rwx--+ 1 reefedjib None   69314 Apr  7 21:24 
jackson-dataformat-smile-2.1.4.jar
-rwx--+ 1 reefedjib None  780385 Apr  7 21:24 jackson-mapper-asl-1.9.12.jar
-rwx--+ 1 reefedjib None   47913 May  9 23:39 jopt-simple-3.0-rc2.jar
-rwx--+ 1 reefedjib None 2365575 Apr 30 13:06 kafka_2.8.0-0.8.0-SNAPSHOT.jar
-rwx--+ 1 reefedjib None  481535 Jan 11 13:46 log4j-1.2.16.jar
-rwx--+ 1 reefedjib None   20647 Apr 18 13:41 log4j-over-slf4j-1.6.6.jar
-rwx--+ 1 reefedjib None  251784 Apr 18 13:41 logback-classic-1.0.6.jar
-rwx--+ 1 reefedjib None  349706 Apr 18 13:41 logback-core-1.0.6.jar
-rwx--+ 1 reefedjib None   82123 Nov 26 13:11 metrics-core-2.2.0.jar
-rwx--+ 1 reefedjib None 1540457 Jul 12  2012 ojdbc14.jar
-rwx--+ 1 reefedjib None 6418368 Apr 30 08:23 scala-library-2.8.2.jar
-rwx--+ 1 reefedjib None 3114958 Apr  2 07:47 scalatest_2.10-1.9.1.jar
-rwx--+ 1 reefedjib None   25962 Apr 18 13:41 slf4j-api-1.6.5.jar
-rwx--+ 1 reefedjib None   62269 Nov 29 03:26 zkclient-0.2.jar
-rwx--+ 1 reefedjib None  601677 Apr 18 13:41 zookeeper-3.3.3.jar

Reporter: Rob Withers
Priority: Blocker


This FATAL shuts down both brokers on windows, 
{2013-05-10 18:23:57,636} DEBUG [local-vat] (Logging.scala:51) - Sending 1 
messages with no compression to [robert_v_2x0,0]
{2013-05-10 18:23:57,637} DEBUG [local-vat] (Logging.scala:51) - Producer 
sending messages with correlation id 178 for topics [robert_v_2x0,0] to 
broker 1 on 192.168.1.100:9093
{2013-05-10 18:23:57,689} FATAL [highwatermark-checkpoint-thread1] 
(Logging.scala:109) - Attempt to swap the new high watermark file with the 
old one failed
{2013-05-10 18:23:57,739}  INFO [Thread-4] (Logging.scala:67) - [Kafka 
Server 0], shutting down

Furthermore, attempts to restart them fail, with the following log:
{2013-05-10 19:14:52,156}  INFO [Thread-1] (Logging.scala:67) - [Kafka Server 
0], started
{2013-05-10 19:14:52,157}  INFO [ZkClient-EventThread-32-localhost:2181] 
(Logging.scala:67) - New leader is 0
{2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
(ZkEventThread.java:79) - Delivering event #1 done
{2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
(ZkEventThread.java:69) - Delivering event #4 ZkEvent[Data of /controller_epoch 
changed sent to kafka.controller.ControllerEpochListener@5cb88f42]
{2013-05-10 19:14:52,210} DEBUG [SyncThread:0] (FinalRequestProcessor.java:78) 
- Processing request:: sessionid:0x13e9127882e0001 type:exists cxid:0x1d 
zxid:0xfffe txntype:unknown reqpath:/controller_epoch
{2013-05-10 19:14:52,210} DEBUG [SyncThread:0] (FinalRequestProcessor.java:160) 
- sessionid:0x13e9127882e0001 type:exists cxid:0x1d zxid:0xfffe 
txntype:unknown reqpath:/controller_epoch
{2013-05-10 19:14:52,213} DEBUG [Thread-1-SendThread(localhost:2181)] 
(ClientCnxn.java:838) - Reading reply sessionid:0x13e9127882e0001, packet:: 
clientPath:null serverPath:null finished:false header:: 29,3  replyHeader:: 
29,37,0  request:: '/controller_epoch,T  response:: 
s{16,36,1368231712816,1368234889961,1,0,0,0,1,0,16} 
{2013-05-10 19:14:52,219}  INFO [Thread-5] (Logging.scala:67) - [Kafka Server 
0], shutting down


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-903) [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed

2013-05-10 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-903:
--

Status: Patch Available  (was: Open)

 [0.8.0 - windows]  FATAL - [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 ---

 Key: KAFKA-903
 URL: https://issues.apache.org/jira/browse/KAFKA-903
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8
 Environment: Windows 7 with SP 1; jdk 7_0_17; scala-library-2.8.2, 
 probably copied on 4/30. kafka-0.8, built current on 4/30.
 -rwx--+ 1 reefedjib None   41123 Mar 19  2009 commons-cli-1.2.jar
 -rwx--+ 1 reefedjib None   58160 Jan 11 13:45 commons-codec-1.4.jar
 -rwx--+ 1 reefedjib None  575389 Apr 18 13:41 
 commons-collections-3.2.1.jar
 -rwx--+ 1 reefedjib None  143847 May 21  2009 commons-compress-1.0.jar
 -rwx--+ 1 reefedjib None   52543 Jan 11 13:45 commons-exec-1.1.jar
 -rwx--+ 1 reefedjib None   57779 Jan 11 13:45 commons-fileupload-1.2.1.jar
 -rwx--+ 1 reefedjib None  109043 Jan 20  2008 commons-io-1.4.jar
 -rwx--+ 1 reefedjib None  279193 Jan 11 13:45 commons-lang-2.5.jar
 -rwx--+ 1 reefedjib None   60686 Jan 11 13:45 commons-logging-1.1.1.jar
 -rwx--+ 1 reefedjib None 1891110 Apr 18 13:41 guava-13.0.1.jar
 -rwx--+ 1 reefedjib None  206866 Apr  7 21:24 jackson-core-2.1.4.jar
 -rwx--+ 1 reefedjib None  232245 Apr  7 21:24 jackson-core-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   69314 Apr  7 21:24 
 jackson-dataformat-smile-2.1.4.jar
 -rwx--+ 1 reefedjib None  780385 Apr  7 21:24 
 jackson-mapper-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   47913 May  9 23:39 jopt-simple-3.0-rc2.jar
 -rwx--+ 1 reefedjib None 2365575 Apr 30 13:06 
 kafka_2.8.0-0.8.0-SNAPSHOT.jar
 -rwx--+ 1 reefedjib None  481535 Jan 11 13:46 log4j-1.2.16.jar
 -rwx--+ 1 reefedjib None   20647 Apr 18 13:41 log4j-over-slf4j-1.6.6.jar
 -rwx--+ 1 reefedjib None  251784 Apr 18 13:41 logback-classic-1.0.6.jar
 -rwx--+ 1 reefedjib None  349706 Apr 18 13:41 logback-core-1.0.6.jar
 -rwx--+ 1 reefedjib None   82123 Nov 26 13:11 metrics-core-2.2.0.jar
 -rwx--+ 1 reefedjib None 1540457 Jul 12  2012 ojdbc14.jar
 -rwx--+ 1 reefedjib None 6418368 Apr 30 08:23 scala-library-2.8.2.jar
 -rwx--+ 1 reefedjib None 3114958 Apr  2 07:47 scalatest_2.10-1.9.1.jar
 -rwx--+ 1 reefedjib None   25962 Apr 18 13:41 slf4j-api-1.6.5.jar
 -rwx--+ 1 reefedjib None   62269 Nov 29 03:26 zkclient-0.2.jar
 -rwx--+ 1 reefedjib None  601677 Apr 18 13:41 zookeeper-3.3.3.jar
Reporter: Rob Withers
Priority: Blocker
 Attachments: kafka-903.patch


 This FATAL shuts down both brokers on windows, 
 {2013-05-10 18:23:57,636} DEBUG [local-vat] (Logging.scala:51) - Sending 1 
 messages with no compression to [robert_v_2x0,0]
 {2013-05-10 18:23:57,637} DEBUG [local-vat] (Logging.scala:51) - Producer 
 sending messages with correlation id 178 for topics [robert_v_2x0,0] to 
 broker 1 on 192.168.1.100:9093
 {2013-05-10 18:23:57,689} FATAL [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 {2013-05-10 18:23:57,739}  INFO [Thread-4] (Logging.scala:67) - [Kafka 
 Server 0], shutting down
 Furthermore, attempts to restart them fail, with the following log:
 {2013-05-10 19:14:52,156}  INFO [Thread-1] (Logging.scala:67) - [Kafka Server 
 0], started
 {2013-05-10 19:14:52,157}  INFO [ZkClient-EventThread-32-localhost:2181] 
 (Logging.scala:67) - New leader is 0
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:79) - Delivering event #1 done
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:69) - Delivering event #4 ZkEvent[Data of 
 /controller_epoch changed sent to 
 kafka.controller.ControllerEpochListener@5cb88f42]
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:78) - Processing request:: 
 sessionid:0x13e9127882e0001 type:exists cxid:0x1d zxid:0xfffe 
 txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:160) - sessionid:0x13e9127882e0001 type:exists 
 cxid:0x1d zxid:0xfffe txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,213} DEBUG [Thread-1-SendThread(localhost:2181)] 
 (ClientCnxn.java:838) - Reading reply sessionid:0x13e9127882e0001, packet:: 
 clientPath:null serverPath:null finished:false header:: 29,3  replyHeader:: 
 29,37,0  request:: '/controller_epoch,T  response::

[jira] [Updated] (KAFKA-903) [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed

2013-05-10 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-903:
--

Attachment: kafka-903.patch

Attach a patch. Rob, could you give it a try?

 [0.8.0 - windows]  FATAL - [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 ---

 Key: KAFKA-903
 URL: https://issues.apache.org/jira/browse/KAFKA-903
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8
 Environment: Windows 7 with SP 1; jdk 7_0_17; scala-library-2.8.2, 
 probably copied on 4/30. kafka-0.8, built current on 4/30.
 -rwx--+ 1 reefedjib None   41123 Mar 19  2009 commons-cli-1.2.jar
 -rwx--+ 1 reefedjib None   58160 Jan 11 13:45 commons-codec-1.4.jar
 -rwx--+ 1 reefedjib None  575389 Apr 18 13:41 
 commons-collections-3.2.1.jar
 -rwx--+ 1 reefedjib None  143847 May 21  2009 commons-compress-1.0.jar
 -rwx--+ 1 reefedjib None   52543 Jan 11 13:45 commons-exec-1.1.jar
 -rwx--+ 1 reefedjib None   57779 Jan 11 13:45 commons-fileupload-1.2.1.jar
 -rwx--+ 1 reefedjib None  109043 Jan 20  2008 commons-io-1.4.jar
 -rwx--+ 1 reefedjib None  279193 Jan 11 13:45 commons-lang-2.5.jar
 -rwx--+ 1 reefedjib None   60686 Jan 11 13:45 commons-logging-1.1.1.jar
 -rwx--+ 1 reefedjib None 1891110 Apr 18 13:41 guava-13.0.1.jar
 -rwx--+ 1 reefedjib None  206866 Apr  7 21:24 jackson-core-2.1.4.jar
 -rwx--+ 1 reefedjib None  232245 Apr  7 21:24 jackson-core-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   69314 Apr  7 21:24 
 jackson-dataformat-smile-2.1.4.jar
 -rwx--+ 1 reefedjib None  780385 Apr  7 21:24 
 jackson-mapper-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   47913 May  9 23:39 jopt-simple-3.0-rc2.jar
 -rwx--+ 1 reefedjib None 2365575 Apr 30 13:06 
 kafka_2.8.0-0.8.0-SNAPSHOT.jar
 -rwx--+ 1 reefedjib None  481535 Jan 11 13:46 log4j-1.2.16.jar
 -rwx--+ 1 reefedjib None   20647 Apr 18 13:41 log4j-over-slf4j-1.6.6.jar
 -rwx--+ 1 reefedjib None  251784 Apr 18 13:41 logback-classic-1.0.6.jar
 -rwx--+ 1 reefedjib None  349706 Apr 18 13:41 logback-core-1.0.6.jar
 -rwx--+ 1 reefedjib None   82123 Nov 26 13:11 metrics-core-2.2.0.jar
 -rwx--+ 1 reefedjib None 1540457 Jul 12  2012 ojdbc14.jar
 -rwx--+ 1 reefedjib None 6418368 Apr 30 08:23 scala-library-2.8.2.jar
 -rwx--+ 1 reefedjib None 3114958 Apr  2 07:47 scalatest_2.10-1.9.1.jar
 -rwx--+ 1 reefedjib None   25962 Apr 18 13:41 slf4j-api-1.6.5.jar
 -rwx--+ 1 reefedjib None   62269 Nov 29 03:26 zkclient-0.2.jar
 -rwx--+ 1 reefedjib None  601677 Apr 18 13:41 zookeeper-3.3.3.jar
Reporter: Rob Withers
Priority: Blocker
 Attachments: kafka-903.patch


 This FATAL shuts down both brokers on windows, 
 {2013-05-10 18:23:57,636} DEBUG [local-vat] (Logging.scala:51) - Sending 1 
 messages with no compression to [robert_v_2x0,0]
 {2013-05-10 18:23:57,637} DEBUG [local-vat] (Logging.scala:51) - Producer 
 sending messages with correlation id 178 for topics [robert_v_2x0,0] to 
 broker 1 on 192.168.1.100:9093
 {2013-05-10 18:23:57,689} FATAL [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 {2013-05-10 18:23:57,739}  INFO [Thread-4] (Logging.scala:67) - [Kafka 
 Server 0], shutting down
 Furthermore, attempts to restart them fail, with the following log:
 {2013-05-10 19:14:52,156}  INFO [Thread-1] (Logging.scala:67) - [Kafka Server 
 0], started
 {2013-05-10 19:14:52,157}  INFO [ZkClient-EventThread-32-localhost:2181] 
 (Logging.scala:67) - New leader is 0
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:79) - Delivering event #1 done
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:69) - Delivering event #4 ZkEvent[Data of 
 /controller_epoch changed sent to 
 kafka.controller.ControllerEpochListener@5cb88f42]
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:78) - Processing request:: 
 sessionid:0x13e9127882e0001 type:exists cxid:0x1d zxid:0xfffe 
 txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:160) - sessionid:0x13e9127882e0001 type:exists 
 cxid:0x1d zxid:0xfffe txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,213} DEBUG [Thread-1-SendThread(localhost:2181)] 
 (ClientCnxn.java:838) - Reading reply sessionid:0x13e9127882e0001, packet:: 
 clientPath:null serverPath:null finished:false header:: 29,3  replyHeader:: 
 29,37,0  request::

[jira] [Commented] (KAFKA-903) [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed

2013-05-10 Thread Rob Withers (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655169#comment-13655169
 ] 

Rob Withers commented on KAFKA-903:
---

I'd love too.  Could you build me a jar and email it too me, please?  :)


 [0.8.0 - windows]  FATAL - [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 ---

 Key: KAFKA-903
 URL: https://issues.apache.org/jira/browse/KAFKA-903
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8
 Environment: Windows 7 with SP 1; jdk 7_0_17; scala-library-2.8.2, 
 probably copied on 4/30. kafka-0.8, built current on 4/30.
 -rwx--+ 1 reefedjib None   41123 Mar 19  2009 commons-cli-1.2.jar
 -rwx--+ 1 reefedjib None   58160 Jan 11 13:45 commons-codec-1.4.jar
 -rwx--+ 1 reefedjib None  575389 Apr 18 13:41 
 commons-collections-3.2.1.jar
 -rwx--+ 1 reefedjib None  143847 May 21  2009 commons-compress-1.0.jar
 -rwx--+ 1 reefedjib None   52543 Jan 11 13:45 commons-exec-1.1.jar
 -rwx--+ 1 reefedjib None   57779 Jan 11 13:45 commons-fileupload-1.2.1.jar
 -rwx--+ 1 reefedjib None  109043 Jan 20  2008 commons-io-1.4.jar
 -rwx--+ 1 reefedjib None  279193 Jan 11 13:45 commons-lang-2.5.jar
 -rwx--+ 1 reefedjib None   60686 Jan 11 13:45 commons-logging-1.1.1.jar
 -rwx--+ 1 reefedjib None 1891110 Apr 18 13:41 guava-13.0.1.jar
 -rwx--+ 1 reefedjib None  206866 Apr  7 21:24 jackson-core-2.1.4.jar
 -rwx--+ 1 reefedjib None  232245 Apr  7 21:24 jackson-core-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   69314 Apr  7 21:24 
 jackson-dataformat-smile-2.1.4.jar
 -rwx--+ 1 reefedjib None  780385 Apr  7 21:24 
 jackson-mapper-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   47913 May  9 23:39 jopt-simple-3.0-rc2.jar
 -rwx--+ 1 reefedjib None 2365575 Apr 30 13:06 
 kafka_2.8.0-0.8.0-SNAPSHOT.jar
 -rwx--+ 1 reefedjib None  481535 Jan 11 13:46 log4j-1.2.16.jar
 -rwx--+ 1 reefedjib None   20647 Apr 18 13:41 log4j-over-slf4j-1.6.6.jar
 -rwx--+ 1 reefedjib None  251784 Apr 18 13:41 logback-classic-1.0.6.jar
 -rwx--+ 1 reefedjib None  349706 Apr 18 13:41 logback-core-1.0.6.jar
 -rwx--+ 1 reefedjib None   82123 Nov 26 13:11 metrics-core-2.2.0.jar
 -rwx--+ 1 reefedjib None 1540457 Jul 12  2012 ojdbc14.jar
 -rwx--+ 1 reefedjib None 6418368 Apr 30 08:23 scala-library-2.8.2.jar
 -rwx--+ 1 reefedjib None 3114958 Apr  2 07:47 scalatest_2.10-1.9.1.jar
 -rwx--+ 1 reefedjib None   25962 Apr 18 13:41 slf4j-api-1.6.5.jar
 -rwx--+ 1 reefedjib None   62269 Nov 29 03:26 zkclient-0.2.jar
 -rwx--+ 1 reefedjib None  601677 Apr 18 13:41 zookeeper-3.3.3.jar
Reporter: Rob Withers
Priority: Blocker
 Attachments: kafka-903.patch


 This FATAL shuts down both brokers on windows, 
 {2013-05-10 18:23:57,636} DEBUG [local-vat] (Logging.scala:51) - Sending 1 
 messages with no compression to [robert_v_2x0,0]
 {2013-05-10 18:23:57,637} DEBUG [local-vat] (Logging.scala:51) - Producer 
 sending messages with correlation id 178 for topics [robert_v_2x0,0] to 
 broker 1 on 192.168.1.100:9093
 {2013-05-10 18:23:57,689} FATAL [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 {2013-05-10 18:23:57,739}  INFO [Thread-4] (Logging.scala:67) - [Kafka 
 Server 0], shutting down
 Furthermore, attempts to restart them fail, with the following log:
 {2013-05-10 19:14:52,156}  INFO [Thread-1] (Logging.scala:67) - [Kafka Server 
 0], started
 {2013-05-10 19:14:52,157}  INFO [ZkClient-EventThread-32-localhost:2181] 
 (Logging.scala:67) - New leader is 0
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:79) - Delivering event #1 done
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:69) - Delivering event #4 ZkEvent[Data of 
 /controller_epoch changed sent to 
 kafka.controller.ControllerEpochListener@5cb88f42]
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:78) - Processing request:: 
 sessionid:0x13e9127882e0001 type:exists cxid:0x1d zxid:0xfffe 
 txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:160) - sessionid:0x13e9127882e0001 type:exists 
 cxid:0x1d zxid:0xfffe txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,213} DEBUG [Thread-1-SendThread(localhost:2181)] 
 (ClientCnxn.java:838) - Reading reply sessionid:0x13e9127882e0001, packet:: 
 clientPath:null serverPath:null finished:false

[jira] [Updated] (KAFKA-903) [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed

2013-05-10 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-903:
--

Attachment: kafka_2.8.0-0.8.0-SNAPSHOT.jar

Attach the kafka jar with the patch.

 [0.8.0 - windows]  FATAL - [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 ---

 Key: KAFKA-903
 URL: https://issues.apache.org/jira/browse/KAFKA-903
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8
 Environment: Windows 7 with SP 1; jdk 7_0_17; scala-library-2.8.2, 
 probably copied on 4/30. kafka-0.8, built current on 4/30.
 -rwx--+ 1 reefedjib None   41123 Mar 19  2009 commons-cli-1.2.jar
 -rwx--+ 1 reefedjib None   58160 Jan 11 13:45 commons-codec-1.4.jar
 -rwx--+ 1 reefedjib None  575389 Apr 18 13:41 
 commons-collections-3.2.1.jar
 -rwx--+ 1 reefedjib None  143847 May 21  2009 commons-compress-1.0.jar
 -rwx--+ 1 reefedjib None   52543 Jan 11 13:45 commons-exec-1.1.jar
 -rwx--+ 1 reefedjib None   57779 Jan 11 13:45 commons-fileupload-1.2.1.jar
 -rwx--+ 1 reefedjib None  109043 Jan 20  2008 commons-io-1.4.jar
 -rwx--+ 1 reefedjib None  279193 Jan 11 13:45 commons-lang-2.5.jar
 -rwx--+ 1 reefedjib None   60686 Jan 11 13:45 commons-logging-1.1.1.jar
 -rwx--+ 1 reefedjib None 1891110 Apr 18 13:41 guava-13.0.1.jar
 -rwx--+ 1 reefedjib None  206866 Apr  7 21:24 jackson-core-2.1.4.jar
 -rwx--+ 1 reefedjib None  232245 Apr  7 21:24 jackson-core-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   69314 Apr  7 21:24 
 jackson-dataformat-smile-2.1.4.jar
 -rwx--+ 1 reefedjib None  780385 Apr  7 21:24 
 jackson-mapper-asl-1.9.12.jar
 -rwx--+ 1 reefedjib None   47913 May  9 23:39 jopt-simple-3.0-rc2.jar
 -rwx--+ 1 reefedjib None 2365575 Apr 30 13:06 
 kafka_2.8.0-0.8.0-SNAPSHOT.jar
 -rwx--+ 1 reefedjib None  481535 Jan 11 13:46 log4j-1.2.16.jar
 -rwx--+ 1 reefedjib None   20647 Apr 18 13:41 log4j-over-slf4j-1.6.6.jar
 -rwx--+ 1 reefedjib None  251784 Apr 18 13:41 logback-classic-1.0.6.jar
 -rwx--+ 1 reefedjib None  349706 Apr 18 13:41 logback-core-1.0.6.jar
 -rwx--+ 1 reefedjib None   82123 Nov 26 13:11 metrics-core-2.2.0.jar
 -rwx--+ 1 reefedjib None 1540457 Jul 12  2012 ojdbc14.jar
 -rwx--+ 1 reefedjib None 6418368 Apr 30 08:23 scala-library-2.8.2.jar
 -rwx--+ 1 reefedjib None 3114958 Apr  2 07:47 scalatest_2.10-1.9.1.jar
 -rwx--+ 1 reefedjib None   25962 Apr 18 13:41 slf4j-api-1.6.5.jar
 -rwx--+ 1 reefedjib None   62269 Nov 29 03:26 zkclient-0.2.jar
 -rwx--+ 1 reefedjib None  601677 Apr 18 13:41 zookeeper-3.3.3.jar
Reporter: Rob Withers
Priority: Blocker
 Attachments: kafka_2.8.0-0.8.0-SNAPSHOT.jar, kafka-903.patch


 This FATAL shuts down both brokers on windows, 
 {2013-05-10 18:23:57,636} DEBUG [local-vat] (Logging.scala:51) - Sending 1 
 messages with no compression to [robert_v_2x0,0]
 {2013-05-10 18:23:57,637} DEBUG [local-vat] (Logging.scala:51) - Producer 
 sending messages with correlation id 178 for topics [robert_v_2x0,0] to 
 broker 1 on 192.168.1.100:9093
 {2013-05-10 18:23:57,689} FATAL [highwatermark-checkpoint-thread1] 
 (Logging.scala:109) - Attempt to swap the new high watermark file with the 
 old one failed
 {2013-05-10 18:23:57,739}  INFO [Thread-4] (Logging.scala:67) - [Kafka 
 Server 0], shutting down
 Furthermore, attempts to restart them fail, with the following log:
 {2013-05-10 19:14:52,156}  INFO [Thread-1] (Logging.scala:67) - [Kafka Server 
 0], started
 {2013-05-10 19:14:52,157}  INFO [ZkClient-EventThread-32-localhost:2181] 
 (Logging.scala:67) - New leader is 0
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:79) - Delivering event #1 done
 {2013-05-10 19:14:52,193} DEBUG [ZkClient-EventThread-32-localhost:2181] 
 (ZkEventThread.java:69) - Delivering event #4 ZkEvent[Data of 
 /controller_epoch changed sent to 
 kafka.controller.ControllerEpochListener@5cb88f42]
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:78) - Processing request:: 
 sessionid:0x13e9127882e0001 type:exists cxid:0x1d zxid:0xfffe 
 txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,210} DEBUG [SyncThread:0] 
 (FinalRequestProcessor.java:160) - sessionid:0x13e9127882e0001 type:exists 
 cxid:0x1d zxid:0xfffe txntype:unknown reqpath:/controller_epoch
 {2013-05-10 19:14:52,213} DEBUG [Thread-1-SendThread(localhost:2181)] 
 (ClientCnxn.java:838) - Reading reply sessionid:0x13e9127882e0001, packet:: 
 clientPath:null serverPath:null finished:false header:: 29,3