Re: Encountered unhandled exception during HA process

2015-02-27 Thread Daan Hoogland 
don't hesistate to report in the meantime, though. have fun

On Fri, Feb 27, 2015 at 12:21 PM, Martin Emrich
martin.emr...@empolis.com wrote:
 Yes... I more and more learn the first rule with Cloudstack: If something
 does not work: Wait a day. If something is strange: Wait a week. ;)

 Cheers

 Martin

 Am 26.02.2015 um 21:19 schrieb Somesh Naidu:

 Wonderful! Guess the HA task eventually hit the retry attempt and ended in
 Error state.

 Regards,
 Somesh


 -Original Message-
 From: Martin Emrich [mailto:martin.emr...@empolis.com]
 Sent: Thursday, February 26, 2015 5:44 AM
 To: users@cloudstack.apache.org
 Subject: AW: Encountered unhandled exception during HA process

 Hmm, without doing anything, the messages stopped by themselves ;)

 Thanks

 Martin

 -Ursprüngliche Nachricht-
 Von: Somesh Naidu [mailto:somesh.na...@citrix.com]
 Gesendet: Dienstag, 17. Februar 2015 17:16
 An: users@cloudstack.apache.org
 Betreff: RE: Encountered unhandled exception during HA process

 You'd probably need to delete the corresponding record from op_ha_work
 table. I guess there is a HA task being scheduled for a VM that may no
 longer exists or something similar.

 If you believe you haven't performed any manual DB updates prior to this
 then this NPE should be treated as a defect and you should file a bug report
 for the same.

 Regards,
 Somesh


 -Original Message-
 From: Martin Emrich [mailto:martin.emr...@empolis.com]
 Sent: Tuesday, February 17, 2015 7:48 AM
 To: users@cloudstack.apache.org
 Subject: Encountered unhandled exception during HA process

 Hello!

 I just discovered that I periodically (every few minutes) a lot of these
 messages in the server log:

 
 2015-02-17 11:50:03,649 INFO  [c.c.h.HighAvailabilityManagerImpl]
 (HA-Worker-3:ctx-ee9d5d55 work-793) Processing
 HAWork[793-Migration-2-Stopped-Migrating]
 2015-02-17 11:50:03,651 WARN  [c.c.h.HighAvailabilityManagerImpl]
 (HA-Worker-3:ctx-ee9d5d55 work-793) Encountered unhandled exception during
 HA process, reschedule retry java.lang.NullPointerException
   at

 com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
   at

 org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
   at

 org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
   at

 org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
 2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl]
 (HA-Worker-4:ctx-029c212c work-794) Processing
 HAWork[794-Migration-2-Stopped-Migrating]
 2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl]
 (HA-Worker-3:ctx-ee9d5d55 work-793) Rescheduling
 HAWork[793-Migration-2-Stopped-Migrating] to try again at Tue Feb 17
 12:00:17 CET 2015
 2015-02-17 11:50:03,651 ERROR [c.c.h.HighAvailabilityManagerImpl]
 (HA-Worker-3:ctx-ee9d5d55 work-793) Caught this throwable,
 java.lang.NullPointerException
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:925)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
   at

 org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
   at

 org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
   at

 org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
 2015-02-17 11:50:03,652 WARN  [c.c.h.HighAvailabilityManagerImpl]
 (HA-Worker-4:ctx-029c212c work-794) Encountered unhandled exception during
 HA process, reschedule retry java.lang.NullPointerException
   at

 com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
   at

 com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848

Re: Encountered unhandled exception during HA process

2015-02-27 Thread Martin Emrich 
Yes... I more and more learn the first rule with Cloudstack: If 
something does not work: Wait a day. If something is strange: Wait a 
week. ;)


Cheers

Martin

Am 26.02.2015 um 21:19 schrieb Somesh Naidu:

Wonderful! Guess the HA task eventually hit the retry attempt and ended in 
Error state.

Regards,
Somesh


-Original Message-
From: Martin Emrich [mailto:martin.emr...@empolis.com]
Sent: Thursday, February 26, 2015 5:44 AM
To: users@cloudstack.apache.org
Subject: AW: Encountered unhandled exception during HA process

Hmm, without doing anything, the messages stopped by themselves ;)

Thanks

Martin

-Ursprüngliche Nachricht-
Von: Somesh Naidu [mailto:somesh.na...@citrix.com]
Gesendet: Dienstag, 17. Februar 2015 17:16
An: users@cloudstack.apache.org
Betreff: RE: Encountered unhandled exception during HA process

You'd probably need to delete the corresponding record from op_ha_work table. I 
guess there is a HA task being scheduled for a VM that may no longer exists or 
something similar.

If you believe you haven't performed any manual DB updates prior to this then 
this NPE should be treated as a defect and you should file a bug report for the 
same.

Regards,
Somesh


-Original Message-
From: Martin Emrich [mailto:martin.emr...@empolis.com]
Sent: Tuesday, February 17, 2015 7:48 AM
To: users@cloudstack.apache.org
Subject: Encountered unhandled exception during HA process

Hello!

I just discovered that I periodically (every few minutes) a lot of these 
messages in the server log:


2015-02-17 11:50:03,649 INFO  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Processing 
HAWork[793-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,651 WARN  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Encountered unhandled exception during HA 
process, reschedule retry java.lang.NullPointerException
  at
com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-029c212c work-794) Processing 
HAWork[794-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Rescheduling 
HAWork[793-Migration-2-Stopped-Migrating] to try again at Tue Feb 17
12:00:17 CET 2015
2015-02-17 11:50:03,651 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Caught this throwable, 
java.lang.NullPointerException
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:925)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,652 WARN  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-029c212c work-794) Encountered unhandled exception during HA 
process, reschedule retry java.lang.NullPointerException
  at
com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
  at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
  at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56

RE: Encountered unhandled exception during HA process

2015-02-26 Thread Somesh Naidu 
Wonderful! Guess the HA task eventually hit the retry attempt and ended in 
Error state.

Regards,
Somesh


-Original Message-
From: Martin Emrich [mailto:martin.emr...@empolis.com] 
Sent: Thursday, February 26, 2015 5:44 AM
To: users@cloudstack.apache.org
Subject: AW: Encountered unhandled exception during HA process

Hmm, without doing anything, the messages stopped by themselves ;)

Thanks

Martin

-Ursprüngliche Nachricht-
Von: Somesh Naidu [mailto:somesh.na...@citrix.com] 
Gesendet: Dienstag, 17. Februar 2015 17:16
An: users@cloudstack.apache.org
Betreff: RE: Encountered unhandled exception during HA process

You'd probably need to delete the corresponding record from op_ha_work table. I 
guess there is a HA task being scheduled for a VM that may no longer exists or 
something similar.

If you believe you haven't performed any manual DB updates prior to this then 
this NPE should be treated as a defect and you should file a bug report for the 
same.

Regards,
Somesh


-Original Message-
From: Martin Emrich [mailto:martin.emr...@empolis.com]
Sent: Tuesday, February 17, 2015 7:48 AM
To: users@cloudstack.apache.org
Subject: Encountered unhandled exception during HA process

Hello!

I just discovered that I periodically (every few minutes) a lot of these 
messages in the server log:


2015-02-17 11:50:03,649 INFO  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Processing 
HAWork[793-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,651 WARN  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Encountered unhandled exception during HA 
process, reschedule retry java.lang.NullPointerException
 at
com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-029c212c work-794) Processing 
HAWork[794-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Rescheduling 
HAWork[793-Migration-2-Stopped-Migrating] to try again at Tue Feb 17
12:00:17 CET 2015
2015-02-17 11:50:03,651 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-ee9d5d55 work-793) Caught this throwable, 
java.lang.NullPointerException
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:925)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,652 WARN  [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-029c212c work-794) Encountered unhandled exception during HA 
process, reschedule retry java.lang.NullPointerException
 at
com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
 at
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53

RE: Encountered unhandled exception during HA process

2015-02-17 Thread Somesh Naidu 
You'd probably need to delete the corresponding record from op_ha_work table. I 
guess there is a HA task being scheduled for a VM that may no longer exists or 
something similar.

If you believe you haven't performed any manual DB updates prior to this then 
this NPE should be treated as a defect and you should file a bug report for the 
same.

Regards,
Somesh


-Original Message-
From: Martin Emrich [mailto:martin.emr...@empolis.com] 
Sent: Tuesday, February 17, 2015 7:48 AM
To: users@cloudstack.apache.org
Subject: Encountered unhandled exception during HA process

Hello!

I just discovered that I periodically (every few minutes) a lot of these 
messages in the server log:


2015-02-17 11:50:03,649 INFO  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-3:ctx-ee9d5d55 work-793) Processing 
HAWork[793-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,651 WARN  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-3:ctx-ee9d5d55 work-793) Encountered unhandled exception 
during HA process, reschedule retry
java.lang.NullPointerException
 at 
com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-4:ctx-029c212c work-794) Processing 
HAWork[794-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,651 INFO  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-3:ctx-ee9d5d55 work-793) Rescheduling 
HAWork[793-Migration-2-Stopped-Migrating] to try again at Tue Feb 17 
12:00:17 CET 2015
2015-02-17 11:50:03,651 ERROR [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-3:ctx-ee9d5d55 work-793) Caught this throwable,
java.lang.NullPointerException
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:925)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,652 WARN  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-4:ctx-029c212c work-794) Encountered unhandled exception 
during HA process, reschedule retry
java.lang.NullPointerException
 at 
com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:631)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.runWithContext(HighAvailabilityManagerImpl.java:891)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.access$000(HighAvailabilityManagerImpl.java:848)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread$1.run(HighAvailabilityManagerImpl.java:860)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
 at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
 at 
com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:857)
2015-02-17 11:50:03,652 INFO  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-4:ctx-029c212c work-794) Rescheduling 
HAWork[794-Migration-2-Stopped-Migrating] to try again at Tue Feb 17 
12:00:17 CET 2015
2015-02-17 11:50:03,653 INFO  [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-1:ctx-30ba9813 work-795) Processing 
HAWork[795-Migration-2-Stopped-Migrating]
2015-02-17 11:50:03,653 ERROR [c.c.h.HighAvailabilityManagerImpl] 
(HA-Worker-4:ctx-029c212c work-794) Caught this throwable,
java.lang.NullPointerException
 at