Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-28 Thread khandelwalanuj
Hi, Yes !! The broker process was still running. I verified it with "ps" command. I have updated the JIRA with details as you mentioned in last update. Thanks, Anuj -- View this message in context: http://activemq.2283324.n4.nabble.com/kahadb-corruption-Checkpoint-failed-java-io-IOException

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-28 Thread Tim Bain
You said the broker still held the file lock, but I assumed that the broker process exited without releasing the lock (since it couldn't write to disk). Can you confirm that the master broker process really was still running (as seen by ps, not just the state of the file lock)? If the broker proc

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-27 Thread khandelwalanuj
Hi, Master was not completely killed. Master has stopped it's transport connectors and plugins but it didn't release it's lock from the kahadb. Thanks, Anuj -- View this message in context: http://activemq.2283324.n4.nabble.com/kahadb-corruption-Checkpoint-failed-java-io-IOException-Input-ou

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-27 Thread Tim Bain
I thought the master was killed completely, and the problem was solely that the slave didn't take over. Can you please describe how the master wasn't killed completely, since you've never before mentioned that in either your emails here or the JIRA you submitted? On Tue, Jan 27, 2015 at 7:46 AM,

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-27 Thread khandelwalanuj
Hi, There was some failures on filer because of which applications (ActiveMQ) was not able to read/write on kahadb. As you mentioned that kahadb should handle this if master broker is not writing than failover should take over; I have logged a request https://issues.apache.org/jira/browse/AMQ-554

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-27 Thread Tim Bain
Submit this as a JIRA, stating that KahaDB can't fail over to the slave if the master is unable to write to disk when it shuts down (because it couldn't write to disk). I'm not sure how feasible it'll be for the slave to detect this (maybe there's a file modification timestamp that can be used, or

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-26 Thread khandelwalanuj
Did anyone get a chance to look at this ? -- View this message in context: http://activemq.2283324.n4.nabble.com/kahadb-corruption-Checkpoint-failed-java-io-IOException-Input-output-error-tp4690378p4690442.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-25 Thread khandelwalanuj
Attaching the complete stack trace for more details. ActiveMQ_prod_25_Jan.txt -- View this message in context: http://activemq.2283324.n4.nabble.com/kahadb-corruption-Checkpoint-failed-java-io-IOException-Input-

kahadb corruption: "Checkpoint failed java.io.IOException: Input/output error"

2015-01-25 Thread khandelwalanuj
Hi, Broker verison : 5.10.0 using Master-slave topology with shared kahadb. Today we facing very critical production issue due to Kahadb. We got below mentioned error in broker logs, and after that broker stopped it's transport connectors and stopped it's services but still it didn't release the