Aaron, The question might be best served on the d...@apex.apache.org mailing list as you are starting to make changes to the sources. Nothing obvious jumps out to be, you don't need hadoop on your system to build the sources successfully with all the tests passing. The test failures you are seeing may or may not be related to the kryo change. What happens when you try to build the original sources with changing the Kryo version? What is your environment, mac, linux?
On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com> wrote: > I know it has been a little while since I brought this up...I pulled down > the 3.7.1-RELEASE branch from GitHub in order to test out the change to > Kryo v. 4.0.2 and encountered no problems while building, however, when I > try to do mvn install, there are several failed tests. I am hoping, that > other than the obvious (to me, anyway) errors such as not finding > yarn-site.xml on my laptop which does not have hadoop installed, someone > may be able to see what is going on? > > > ------------------------------------------------------------------------------- > Test set: com.datatorrent.stram.StramRecoveryTest > > ------------------------------------------------------------------------------- > Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.372 sec > <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest > testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: > 0.115 sec <<< FAILURE! > java.lang.AssertionError: flush count expected:<1> but was:<2> > at > com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) > > > > ------------------------------------------------------------------------------- > Test set: com.datatorrent.stram.engine.StatsTest > > ------------------------------------------------------------------------------- > Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.894 sec > <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest > testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) > Time elapsed: 3.266 sec <<< FAILURE! > java.lang.AssertionError: Validate input port queue size -1 > at > com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) > at > com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) > > > ------------------------------------------------------------------------------- > Test set: com.datatorrent.stram.StramMiniClusterTest > > ------------------------------------------------------------------------------- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 sec > <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest > com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec <<< > ERROR! > java.io.FileNotFoundException: > /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml > (No such file or directory) > at > com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) > > > ------------------------------------------------------------------------------- > Test set: com.datatorrent.stram.StramLocalClusterTest > > ------------------------------------------------------------------------------- > Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 15.081 sec > <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest > testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time elapsed: > 0.138 sec <<< FAILURE! > java.lang.AssertionError: checkpoint path exists > at > com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) > > testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) > Time elapsed: 5.191 sec <<< FAILURE! > java.lang.AssertionError: > ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out > exists > at > com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) > > > ------------------------------------------------------------------------------- > Test set: com.datatorrent.stram.client.AppPackageTest > > ------------------------------------------------------------------------------- > Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec > <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest > testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time elapsed: > 0 sec <<< FAILURE! > java.lang.AssertionError: Number of applications expected:<2> but was:<0> > at > com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) > > testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) > Time elapsed: 0.001 sec <<< FAILURE! > java.lang.AssertionError: Should consist of an app called > MyFirstApplication > at > com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) > > > > ------------------------------------------------------------------------------- > Test set: com.datatorrent.stram.cli.ApexCliTest > > ------------------------------------------------------------------------------- > Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: 0.938 sec > <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest > testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) Time > elapsed: 0.104 sec <<< ERROR! > com.datatorrent.stram.cli.ApexCli$CliException: No applications in > Application Package > at > com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) > > testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) > Time elapsed: 0.045 sec <<< FAILURE! > org.junit.ComparisonFailure: expected:<[app]-default> but > was:<[package]-default> > at > com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) > > testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) > Time elapsed: 0.045 sec <<< ERROR! > com.datatorrent.stram.cli.ApexCli$CliException: No applications in > Application Package matching "MyFirstApplication" > at > com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) > > testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) > Time elapsed: 0.058 sec <<< FAILURE! > org.junit.ComparisonFailure: expected:<[app]-default> but > was:<[package]-default> > at > com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) > > > On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert <aa...@punchcyber.com> > wrote: > >> Thanks, Thomas! I have implemented the fix you suggested >> (@FieldSerializer.BInd) and also created a JIRA ticket: >> https://issues.apache.org/jira/browse/APEXCORE-816 >> >> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> wrote: >> >>> Hi, >>> >>> It is not necessary to use a different stream codec when you have >>> control over the class that is causing the error. You can annotate a >>> different field serializer instead, using the @FieldSerializer.Bind >>> annotation. Here is an example: >>> >>> >>> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50 >>> >>> It would be good to upgrade Kryo to 4.x in apex-core master. Would you >>> like to create a JIRA for it? (http://apex.apache.org/contributing.html) >>> >>> Thanks, >>> Thomas >>> >>> >>> >>> >>> >>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <aa...@punchcyber.com> >>> wrote: >>> >>>> I am writing an application that includes a custom object. One of the >>>> fields is of type Instant and seems to not agree with the version of Kryo >>>> that is used in Apex (see exception below). I now that kryo 4.X supports >>>> serializing Instant fields, but for the life of me, cannot seem to find a >>>> way to either get Apex to use a newer version of Kryo. Can anyone help? >>>> Perhaps setting a different streamcodec that uses the newer version of kryo >>>> would work, but I cannot seem to find an example of how to set a different >>>> streamcodec despite references to the two (Kryo and Java) that ship with >>>> Apex. >>>> >>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >>>> Catastrophic Error: Execution halted due to Kryo exception! >>>> com.esotericsoftware.kryo.KryoException: Class cannot be created >>>> (missing no-arg constructor): java.time.Instant >>>> Serialization trace: >>>> systemTime >>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent) >>>> at >>>> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) >>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >>>> at >>>> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) >>>> at >>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) >>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>>> at >>>> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) >>>> at >>>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>> at com.datatorrent.stram.codec.De >>>> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98) >>>> at >>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391) >>>> at >>>> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339) >>>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >>>> at >>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429) >>>> >>>> >>>> -- >>>> >>>> M. Aaron Bossert >>>> (571) 242-4021 >>>> Punch Cyber Analytics Group >>>> >>>> >>>> >>> >> >> >> -- >> >> M. Aaron Bossert >> (571) 242-4021 >> Punch Cyber Analytics Group >> >> >> > > > -- > > M. Aaron Bossert > (571) 242-4021 > Punch Cyber Analytics Group > > >
