Ok here are the results of the suggestions.
Stuart Roe:
Try: If you stop SMX can you delete the data folder in the SMX installation?
Result: Success
Selected "data" directory in ${SMX_HOME} and successfully deleted
directory via Windows Explorer > Delete
Gert Vanthienen:
virus scanners - Not tried.
Gert Vanthienen:
Try: Use hotdeploy instead of the jbi maven plugin.
Result: Success.
Fresh SMX start, copy the <installer>.zip created
INFO - AutoDeploymentService - Directory: hotdeploy:
Finished installation of archive: mcp-sa-1.0-SNAPSHOT.zip
Even tried deleting the hotdeploy archive and that uninstalls correctly as well.
Also rebuilt sa and re-hotdeployed and that works with no errors.
Barrie:
Stop SMX
Delete data directory
Start SMX
Wait for re-installation of data/ directory.
Restart SMX to avoid any re-installation potential issues.
Searched for "mcp" in ${SMX_HOME} - found nothing.
Hotdeploy
Searched for "mcp" in ${SMX_HOME}
- found in data\smx\service-assemblies\mcp-sa\...
- found in data\amq\kr-store\data (hash index files)
- found in hotdeploy\ (obviously)
Hotundeploy
Searched for "mcp" in ${SMX_HOME}
- found in data\smx\service-assemblies\mcp-sa\...
- found in data\amq\kr-store\data (hash index files)
Delete directory
- Fails because java has the locks on the files - specifically the
libraries in one of my service-units.
My code is pretty dumb (its only a proof of concept) and I can't see
anything that would be left in memory after an un-install.
I thought perhaps it was Log4J and the static Loggers holding
references to classes, but I've removed that code and it still
happens.
I was just about to remove the guts of my code thinking something like
XStream might be holding onto classes, when I realized that I am not
actually running the service-unit.
I'm just installing/uninstalling which therefore should not create any locks.
I suspect that SMX is not unloading classes correctly then.
So doing a full thread dump I can see that there are threads blocked
on waiting condition - which is probably why the classes are not being
released.
So the problem appears to be uninstall does not stop running threads
created during the install.
Scanning JIRA on "thread waiting" shows
http://issues.apache.org/activemq/browse/SM-1807 "Classes, Objects and
Threads still alive after Undeploying a servicemix component"
http://issues.apache.org/activemq/browse/SM-1578 "When shutting down
endpoints, the component should wait for all exchanges to be
completely processed before unregistering the endpoint"
Does this sound familiar to anyone? Have I identified the correct defects?
Cheers
"QueueThread:queue://org.apache.servicemix.jca.{urn:mcp:testing}file:sender"
daemon prio=6 tid=0x00a39090 nid=0x172c waiting on condition
[0x3088f000..0x3088fa98]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jca.{urn:mcp}incoming:endpoint"
daemon prio=6 tid=0x2af22d58 nid=0x11d8 waiting on condition
[0x307cf000..0x307cfb18]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jms.{urn:mcp}incoming:endpoint"
daemon prio=6 tid=0x28fbfc18 nid=0xce4 waiting on condition
[0x3078f000..0x3078fd98]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jms.{urn:mcp:testing}file:sender"
daemon prio=6 tid=0x2b898db0 nid=0x1420 waiting on condition
[0x3068f000..0x3068fb18]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jms.{http://activemq.apache.org/camel/schema/jbi}provider:camel:mcp-eip-su-controlBus";
daemon prio=6 tid=0x28b90638 nid=0xb50 waiting on condition
[0x304cf000..0x304cfb18]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jca.{http://activemq.apache.org/camel/schema/jbi}provider:camel:mcp-eip-su-controlBus";
daemon prio=6 tid=0x28f6d408 nid=0x7e0 waiting on condition
[0x3040f000..0x3040fc18]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jca.{urn:mcp}outgoing:endpoint"
daemon prio=6 tid=0x2878b390 nid=0x828 waiting on condition
[0x3038f000..0x3038fb18]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
"QueueThread:queue://org.apache.servicemix.jms.{urn:mcp}outgoing:endpoint"
daemon prio=6 tid=0x2bc35408 nid=0x14e4 waiting on condition
[0x300df000..0x300dfa18]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
at java.lang.Thread.run(Thread.java:595)
