Hi, Do you also have pmml-model-moxy as a dependency in your job? Using mvn dependency:tree, I do not see that pmml-evaluator has a compile time dependency on jaxb-api. The jaxb-api dependency actually comes from pmml- model-moxy. The exclusion should be therefore defined on pmml-model-moxy.
You can also try "parent-first" ClassLoader resolution order [1]. Best, Gary [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order On Tue, Sep 4, 2018 at 3:24 AM, Sameer W <sam...@axiomine.com> wrote: > Hi, > > I am using PMML dependency as below to execute ML models at prediction > time within a Flink Map operator > > <dependency> > > <groupId>org.jpmml</groupId> > > <artifactId>pmml-evaluator</artifactId> > > <version>1.4.3</version> > > > <exclusions> > > <exclusion> > > <groupId>javax.xml.bind</groupId> > > <artifactId>jaxb-api</artifactId> > > </exclusion> > > <exclusion> > > <groupId>org.glassfish.jaxb</groupId> > > <artifactId>jaxb-runtime</artifactId> > > </exclusion> > > <exclusion> > > <artifactId>guava</artifactId> > > <groupId>com.google.guava</groupId> > > </exclusion> > > </exclusions> > > </dependency> > Environment is EMR, OpenJDK 1.8 and Flink 1.4.2. My programs run fine in > my Eclipse Development environment. However when we deploy on the cluster > we get Classloading exceptions which are primarily due to the PMML classes > loaded via the Flink Classloader while the JAXB classes are loaded by the > boot classloader. Also the problem seems like the version of the jaxb > classes referenced within the PMML library is different from the ones > loaded by the open JDK. > > For example I keep getting this type of error. I have also listed another > error after this which is linked to not being able to use reflection and > unsafe library to set private instances within the PMML class instance > using JAXB Unmarshaller. - > java.lang.LinkageError: loader constraint violation: when resolving > interface method "javax.xml.bind.Unmarshaller.unmarshal(Ljavax/xml/ > transform/Source;)Ljava/lang/Object;" the class loader (instance of > org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) > of the current class, > com/comcast/mlarche/featurecreationflows/xreerrors/MyJAXBUtil, > and the class loader (instance of <bootloader>) for the method's defining > class, javax/xml/bind/Unmarshaller, have different Class objects for the > type javax/xml/transform/Source used in the signature > at com.comcast.mlarche.featurecreationflows.xreerrors.MyJAXBUtil. > unmarshal(MyJAXBUtil.java:52) > at com.comcast.mlarche.featurecreationflows.xreerrors.MyJAXBUtil. > unmarshalPMML(MyJAXBUtil.java:38) > at com.comcast.mlarche.featurecreationflows. > xreerrors.PMMLModelExecution.getMiningModelEvaluator( > PMMLModelExecution.java:67) > at com.comcast.mlarche.featurecreationflows. > xreerrors.PMMLModelExecution.predict(PMMLModelExecution.java:126) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorModelsPredictionServiceService.predict( > XreErrorModelsPredictionServiceService.java:61) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorModelsPredictionServiceService.predictSystemRefresh( > XreErrorModelsPredictionServiceService.java:44) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XREErrorsModelExecutionMap.flatMap(XREErrorsModelExecutionMap.java:46) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XREErrorsModelExecutionMap.flatMap(XREErrorsModelExecutionMap.java:17) > at org.apache.flink.streaming.api.operators.StreamFlatMap. > processElement(StreamFlatMap.java:50) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > CopyingChainingOutput.pushToOperator(OperatorChain.java:549) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > CopyingChainingOutput.collect(OperatorChain.java:524) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > CopyingChainingOutput.collect(OperatorChain.java:504) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$ > CountingOutput.collect(AbstractStreamOperator.java:830) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$ > CountingOutput.collect(AbstractStreamOperator.java:808) > at org.apache.flink.streaming.api.operators. > TimestampedCollector.collect(TimestampedCollector.java:51) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorsModelsApplyFunction.apply(XreErrorsModelsApplyFunction.java:65) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorsModelsApplyFunction.apply(XreErrorsModelsApplyFunction.java:20) > at org.apache.flink.streaming.runtime.operators.windowing.functions. > InternalIterableWindowFunction.process(InternalIterableWindowFunction > .java:44) > at org.apache.flink.streaming.runtime.operators.windowing.functions. > InternalIterableWindowFunction.process(InternalIterableWindowFunction > .java:32) > at org.apache.flink.streaming.runtime.operators.windowing. > EvictingWindowOperator.emitWindowContents(EvictingWindowOperator.java:357) > at org.apache.flink.streaming.runtime.operators.windowing. > EvictingWindowOperator.processElement(EvictingWindowOperator.java:218) > at org.apache.flink.streaming.runtime.io.StreamInputProcessor. > processInput(StreamInputProcessor.java:207) > at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run( > OneInputStreamTask.java:69) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > invoke(StreamTask.java:264) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) > at java.lang.Thread.run(Thread.java:748) > > Finally when I started using InputStream based constructor (so as not to > use SAXSource classe) below is the last error I got when I finally got as > deep within the library as possible without using any util classes. > > java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not > set org.xml.sax.Locator field org.dmg.pmml.PMMLObject.locator to > org.xml.sax.helpers.LocatorImpl > at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution. > getMiningModelEvaluator(PMMLModelExecution.java:81) > at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution. > predict(PMMLModelExecution.java:129) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorModelsPredictionServiceService.predict( > XreErrorModelsPredictionServiceService.java:61) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorModelsPredictionServiceService.predictSystemRefresh( > XreErrorModelsPredictionServiceService.java:44) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XREErrorsModelExecutionMap.flatMap(XREErrorsModelExecutionMap.java:46) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XREErrorsModelExecutionMap.flatMap(XREErrorsModelExecutionMap.java:17) > at org.apache.flink.streaming.api.operators.StreamFlatMap. > processElement(StreamFlatMap.java:50) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > CopyingChainingOutput.pushToOperator(OperatorChain.java:549) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > CopyingChainingOutput.collect(OperatorChain.java:524) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > CopyingChainingOutput.collect(OperatorChain.java:504) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$ > CountingOutput.collect(AbstractStreamOperator.java:830) > at org.apache.flink.streaming.api.operators.AbstractStreamOperator$ > CountingOutput.collect(AbstractStreamOperator.java:808) > at org.apache.flink.streaming.api.operators.TimestampedCollector.collect( > TimestampedCollector.java:51) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorsModelsApplyFunction.apply(XreErrorsModelsApplyFunction.java:65) > at com.comcast.mlarche.featurecreationflows.xreerrors. > XreErrorsModelsApplyFunction.apply(XreErrorsModelsApplyFunction.java:20) > at org.apache.flink.streaming.runtime.operators.windowing.functions. > InternalIterableWindowFunction.process(InternalIterableWindowFunction > .java:44) > at org.apache.flink.streaming.runtime.operators.windowing.functions. > InternalIterableWindowFunction.process(InternalIterableWindowFunction > .java:32) > at org.apache.flink.streaming.runtime.operators.windowing. > EvictingWindowOperator.emitWindowContents(EvictingWindowOperator.java:357) > at org.apache.flink.streaming.runtime.operators.windowing. > EvictingWindowOperator.processElement(EvictingWindowOperator.java:218) > at org.apache.flink.streaming.runtime.io.StreamInputProcessor. > processInput(StreamInputProcessor.java:207) > at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run( > OneInputStreamTask.java:69) > at org.apache.flink.streaming.runtime.tasks.StreamTask. > invoke(StreamTask.java:264) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.IllegalArgumentException: Can not set > org.xml.sax.Locator field org.dmg.pmml.PMMLObject.locator to > org.xml.sax.helpers.LocatorImpl > at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentExcepti > on(UnsafeFieldAccessorImpl.java:167) > at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentExcepti > on(UnsafeFieldAccessorImpl.java:171) > at sun.reflect.UnsafeObjectFieldAccessorImpl.set( > UnsafeObjectFieldAccessorImpl.java:81) > at java.lang.reflect.Field.set(Field.java:764) > at com.sun.xml.bind.v2.runtime.reflect.Accessor$ > FieldReflection.set(Accessor.java:271) > at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.createInstance( > ClassBeanInfoImpl.java:282) > at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext. > createInstance(UnmarshallingContext.java:608) > at com.sun.xml.bind.v2.runtime.unmarshaller.StructureLoader. > startElement(StructureLoader.java:185) > at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext._ > startElement(UnmarshallingContext.java:486) > at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext. > startElement(UnmarshallingContext.java:464) > at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector. > startElement(SAXConnector.java:152) > at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown > Source) > at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown > Source) > at > org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown > Source) > at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl > $FragmentContentDispatcher.dispatch(Unknown Source) > at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown > Source) > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) > at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) > at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) > at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown > Source) > at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl. > unmarshal0(UnmarshallerImpl.java:216) > at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl. > unmarshal(UnmarshallerImpl.java:189) > at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal( > AbstractUnmarshallerImpl.java:157) > at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal( > AbstractUnmarshallerImpl.java:204) > at com.comcast.mlarche.featurecreationflows.xreerrors.MyJAXBUtil. > unmarshal(MyJAXBUtil.java:64) > at com.comcast.mlarche.featurecreationflows.xreerrors.MyJAXBUtil. > unmarshalPMML(MyJAXBUtil.java:43) > at com.comcast.mlarche.featurecreationflows.xreerrors.PMMLModelExecution. > getMiningModelEvaluator(PMMLModelExecution.java:66) >
