[
https://issues.apache.org/jira/browse/OOZIE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334999#comment-16334999
]
Attila Sasvari commented on OOZIE-2775:
---------------------------------------
{color:#333333}[~andras.piros] Thanks for your comments!{color}
* {color:#333333}When a {color}ProvisionException is thrown during
{{guiceInjector.getInstance()}} , {{embeddedOozieServer}} will be null. In the
current form, without additional changes: you can't get a reference to the
Jetty server as it does not exist at this point of the execution (it is
actually created after the Oozie services started). The (failed) Oozie services
created by the provider cannot be stopped either. I wonder if it was really
necessary. If so, we can probably handle it in the ServicesProvider. I will try
to address this in a new patch uploaded to *Reviewboard* *first*.
* Regarding a stop() method: can you expand how it should work? Right now,
stopping the Oozie server happens via a SIGTERM signal sent to the process (see
{{bin/oozie-jetty-server.sh}}. This behaviour is pretty similar to the earlier
version (with Tomcat) where this case was not handled by the means of the
[shutdown hook
mechanism|https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/signals006.html]
as far as I remember. I suppose it is OK that we add such a handler, but of
course there might be races (for example what if signal arrives before we
install it). What do you think?
* I believe
[ExceptionInInitializerError|https://docs.oracle.com/javase/7/docs/api/java/lang/ExceptionInInitializerError.html]
is not suitable for handling this sort of service initialization problems,
[ExceptionInInitializerError|https://docs.oracle.com/javase/7/docs/api/java/lang/ExceptionInInitializerError.html]
is a kind of LinkageError. Documentation says "{{ExceptionInInitializerError}}
is thrown to indicate that an exception occurred during evaluation of a static
initializer or the initializer for a static variable.". A more pragmatic answer
is that BoundProviderFactory's {{get()}} by guice only catches Runtime
exceptions.
> Oozie server does not stop if there is an exception during service
> initalization at startup
> -------------------------------------------------------------------------------------------
>
> Key: OOZIE-2775
> URL: https://issues.apache.org/jira/browse/OOZIE-2775
> Project: Oozie
> Issue Type: Bug
> Reporter: Attila Sasvari
> Assignee: Attila Sasvari
> Priority: Major
> Attachments: OOZIE-2775-01.patch
>
>
> OOZIE-2666 brought in embedded jetty in Oozie. {{com.google.inject.guice}}
> was selected for dependency management. Unfortunately if there is an
> exception at starting Oozie services (e.g. JPAService initialization, called
> via
> https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L194),
> Oozie won't stop if it cannot destroy Oozie services.
> In OOZIE-2666 a shutdown hook
> (https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L184)
> was added to print out startup errors and exceptions, destroy pending oozie
> services, and exit eventually, but it does not work.
> Steps to reproduce a startup failure (for example simulate that Oozie cannot
> find required database driver):
> {code}
> $ cd oozie
> $ mvn clean install assembly:single -DskipTests -DjavaVersion=1.8
> -DtargetVersion=1.7 -Phadoop-2 -Dhadoop.version=2.6.0 -Puber;
> $ cd distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/
> $ export JETTY_OPTS="-Djavax.net.debug=all
> -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
> -Dlog4j.logger.org.eclipse.jetty=TRACE
> -Dlog4j.configuration=./conf/oozie-log4j.properties"
> $ bin/oozied.sh run
> $ mv embedded-oozie-server/webapp/WEB-INF/lib/derby-10.10.1.1.jar ..
> <attach to the process with a debugger and continue execution>
> {code}
> Exception is printed to stdout
> {code}
> <openjpa-2.4.1-r422266:1730418 fatal general error>
> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver
> class 'org.apache.derby.jdbc.EmbeddedDriver'
> at
> org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:106)
> at
> org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
> at
> org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1520)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:533)
> at
> org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:458)
> at
> org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:121)
> at
> org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
> at
> org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
> at
> org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:642)
> at
> org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:202)
> at
> org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:154)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:226)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:153)
> at
> org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:59)
> at
> org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:514)
> at org.apache.oozie.service.JPAService.init(JPAService.java:215)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:387)
> at org.apache.oozie.service.Services.setService(Services.java:373)
> at org.apache.oozie.service.Services.loadServices(Services.java:305)
> at org.apache.oozie.service.Services.init(Services.java:213)
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:31)
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:25)
> at
> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at
> com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at
> com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at
> com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at
> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at
> com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at
> com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at
> com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at
> com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
> at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
> at
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
> at
> org.apache.oozie.server.EmbeddedOozieServer.main(EmbeddedOozieServer.java:194)
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC
> driver class 'org.apache.derby.jdbc.EmbeddedDriver'
> at
> org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1429)
> at
> org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> at
> org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
> at
> org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:86)
> at
> org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:90)
> ... 48 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.derby.jdbc.EmbeddedDriver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1420)
> ... 53 more
> Exception in thread "main" java.lang.ExceptionInInitializerError: Could not
> instantiate Oozie services. [e.message=E0103: Could not load service classes,
> Cannot load JDBC driver class 'org.apache.derby.jdbc.EmbeddedDriver']
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:36)
> at
> org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:25)
> at
> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at
> com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at
> com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at
> com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at
> com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at
> com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at
> com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at
> com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at
> com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at
> com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at
> com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at
> com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
> at
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
> at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
> at
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
> at
> org.apache.oozie.server.EmbeddedOozieServer.main(EmbeddedOozieServer.java:194)
> {code}
> I checked the JVM process with {{jstack}} and saw:
> {code}
> "JDWP Transport Listener: dt_socket" #45 daemon prio=10 os_prio=31
> tid=0x00007fb70b001000 nid=0x3d0b runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "DestroyJavaVM" #44 prio=5 os_prio=31 tid=0x00007fb70ad9b000 nid=0x1003
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Attach Listener" #39 daemon prio=9 os_prio=31 tid=0x00007fb70b383000
> nid=0x670f waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Scheduler-0" #29 prio=5 os_prio=31 tid=0x00007fb70da9b000 nid=0x5e0f waiting
> on condition [0x0000700001ae4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006c00187f0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "FileWatchdog" #15 daemon prio=5 os_prio=31 tid=0x00007fb70b2a3000 nid=0x5a03
> waiting on condition [0x00007000019e1000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
> "com.google.inject.internal.util.$Finalizer" #14 daemon prio=5 os_prio=31
> tid=0x00007fb70abaf800 nid=0x5803 in Object.wait() [0x00007000018de000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c00188d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000006c00188d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
> "Service Thread" #12 daemon prio=9 os_prio=31 tid=0x00007fb70e834000
> nid=0x5403 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C1 CompilerThread3" #11 daemon prio=9 os_prio=31 tid=0x00007fb70e00a800
> nid=0x5203 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007fb70a80f000
> nid=0x5003 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007fb70c002000
> nid=0x4e03 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007fb70d81a000
> nid=0x1707 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31
> tid=0x00007fb70a80e000 nid=0x4c03 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fb70b041000
> nid=0x3e0f runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fb70e002000 nid=0x3803 in
> Object.wait() [0x0000700000d3a000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c00265d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000006c00265d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fb70d802000
> nid=0x3603 in Object.wait() [0x0000700000c37000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c0026670> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000006c0026670> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> "VM Thread" os_prio=31 tid=0x00007fb70b82b000 nid=0x3403 runnable
> "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fb70d003800 nid=0x2403
> runnable
> "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fb70d004000 nid=0x2603
> runnable
> "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fb70d004800 nid=0x2803
> runnable
> "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fb70a80a800 nid=0x2a03
> runnable
> "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fb70a80b000 nid=0x2c03
> runnable
> "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fb70a80b800 nid=0x2e03
> runnable
> "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fb70d005000 nid=0x3003
> runnable
> "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fb70d006000 nid=0x3203
> runnable
> "VM Periodic Task Thread" os_prio=31 tid=0x00007fb70b042000 nid=0x5603
> waiting on condition
> JNI global references: 5741
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
