cool! I'm on it. On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni <pramod.imman...@gmail.com> wrote:
> Hi Aaron, > > That would be great if you can pitch in wherever you can. Let us know how > we can help. Also, I was going to try the new kryo version later today and > see if there are any internal errors that are bubbling up to those test > failures but not getting logged. > > Thanks > > On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> P.S. I am happy to roll up my sleeves, but from my initial attempt, >> errors >> I was seeing were not obviously related to the Kryo version update (though >> I am certain they are...at least indirectly because they only show up when >> the version is changed), in terms of tracking down where the changes would >> be required...I am not really familiar with the inner workings of Apex >> yet, >> though this would definitely be a good excuse to learn... >> >> I also saw a bunch of annoying warnings when building related to use of >> deprecated methods/classes within the com.datatorrent.* namespace...I'd be >> happy to start tracking some of those down too if it would be helpful... >> >> On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >> > Pramod, >> > >> > In a nutshell, yes. I can set a @FieldSerializer to the fields in >> > question...the only one I have run into thus far is Instant...but it >> would >> > be great to not need to annotate each field this way...Also, I could see >> > this becoming problematic should I encounter more than a couple of >> fields >> > that are not serializable using the older version of Kryo (but >> presumably >> > would work with a more current one)...though admitedly, it would not be >> > that big a deal in the interim. It would be much cleaner to have an >> update >> > to the most current version of Kryo, IMHO...though I understand that is >> > something of a lift. By the way, Kryo 5 just came out yesterday, so if >> > there is to be effort expended, it might be good to go with the latest >> > version...though, heads up, I tried it out and there seem to be quite a >> few >> > changes that will be needed...new methods, and definitely some replaced >> > ones... >> > >> > >> > >> > On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni < >> > pramod.imman...@gmail.com> wrote: >> > >> >> Hi Aaron, >> >> >> >> While we are debugging the test failures on dev, I didn't fully >> >> understand the last question in your email. Looks like you mentioned >> that >> >> the workaround for adding the @FieldSerializer to the field is >> working. Are >> >> you looking for an alternative to this workaround and trying to set a >> >> default serializer for all fields of type Instant (the field type you >> >> mentioned in the earlier email) so that you don't have to set the >> >> annotation each time? >> >> >> >> Thanks >> >> >> >> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert <aa...@punchcyber.com> >> >> wrote: >> >> >> >>> I sent an e-mail to the dev list...but have not heard any responses. >> >>> How active is that list? Do you know? Basically, I have a >> workaround that >> >>> will deal with those "things" that Kryo v 2.24.0 will not handle (but >> 4.0.2 >> >>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this is not >> ideal. I >> >>> have two options left: go it alone and try to update Kryo in the >> apex-core, >> >>> or simply change the default serializer. The hiccup I am running >> into is >> >>> that I don't see how (from the documentation) to set the default >> >>> serializer...perhaps I am missing it. Can someone point me to where >> that >> >>> is in the docs? >> >>> >> >>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert <aa...@punchcyber.com> >> >>> wrote: >> >>> >> >>>> Ah, you make a good point about the dev list...I am using IntelliJ >> on a >> >>>> Mac. When I tried building without any changes, I also encountered >> >>>> failures. I can re-run the non-modified sources to do a diff and >> see if >> >>>> the errors were different... >> >>>> >> >>>> Sent from my iPhone >> >>>> >> >>>> On Jun 11, 2018, at 15:11, Pramod Immaneni < >> pramod.imman...@gmail.com> >> >>>> wrote: >> >>>> >> >>>> Aaron, >> >>>> >> >>>> The question might be best served on the dev@apex.apache.org mailing >> >>>> list as you are starting to make changes to the sources. Nothing >> obvious >> >>>> jumps out to be, you don't need hadoop on your system to build the >> sources >> >>>> successfully with all the tests passing. The test failures you are >> seeing >> >>>> may or may not be related to the kryo change. What happens when you >> try to >> >>>> build the original sources with changing the Kryo version? What is >> your >> >>>> environment, mac, linux? >> >>>> >> >>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert <aa...@punchcyber.com >> > >> >>>> wrote: >> >>>> >> >>>>> I know it has been a little while since I brought this up...I pulled >> >>>>> down the 3.7.1-RELEASE branch from GitHub in order to test out the >> change >> >>>>> to Kryo v. 4.0.2 and encountered no problems while building, >> however, when >> >>>>> I try to do mvn install, there are several failed tests. I am >> hoping, that >> >>>>> other than the obvious (to me, anyway) errors such as not finding >> >>>>> yarn-site.xml on my laptop which does not have hadoop installed, >> someone >> >>>>> may be able to see what is going on? >> >>>>> >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Test set: com.datatorrent.stram.StramRecoveryTest >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: >> 6.372 >> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest >> >>>>> testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time >> >>>>> elapsed: 0.115 sec <<< FAILURE! >> >>>>> java.lang.AssertionError: flush count expected:<1> but was:<2> >> >>>>> at >> >>>>> >> com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Test set: com.datatorrent.stram.engine.StatsTest >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: >> 21.894 >> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest >> >>>>> >> testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) >> >>>>> Time elapsed: 3.266 sec <<< FAILURE! >> >>>>> java.lang.AssertionError: Validate input port queue size -1 >> >>>>> at >> >>>>> >> com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) >> >>>>> at >> >>>>> >> com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) >> >>>>> >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Test set: com.datatorrent.stram.StramMiniClusterTest >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: >> 0.002 >> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramMiniClusterTest >> >>>>> com.datatorrent.stram.StramMiniClusterTest Time elapsed: 0.002 sec >> >>>>> <<< ERROR! >> >>>>> java.io.FileNotFoundException: >> >>>>> >> /Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml >> >>>>> (No such file or directory) >> >>>>> at >> >>>>> >> com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151) >> >>>>> >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Test set: com.datatorrent.stram.StramLocalClusterTest >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: >> 15.081 >> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.StramLocalClusterTest >> >>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest) Time >> >>>>> elapsed: 0.138 sec <<< FAILURE! >> >>>>> java.lang.AssertionError: checkpoint path exists >> >>>>> at >> >>>>> >> com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413) >> >>>>> >> >>>>> >> testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest) >> >>>>> Time elapsed: 5.191 sec <<< FAILURE! >> >>>>> java.lang.AssertionError: >> >>>>> >> ./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out >> >>>>> exists >> >>>>> at >> >>>>> >> com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117) >> >>>>> >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Test set: com.datatorrent.stram.client.AppPackageTest >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: >> 0.105 >> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.client.AppPackageTest >> >>>>> testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time >> >>>>> elapsed: 0 sec <<< FAILURE! >> >>>>> java.lang.AssertionError: Number of applications expected:<2> but >> >>>>> was:<0> >> >>>>> at >> >>>>> >> com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91) >> >>>>> >> >>>>> >> testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest) >> >>>>> Time elapsed: 0.001 sec <<< FAILURE! >> >>>>> java.lang.AssertionError: Should consist of an app called >> >>>>> MyFirstApplication >> >>>>> at >> >>>>> >> com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171) >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Test set: com.datatorrent.stram.cli.ApexCliTest >> >>>>> >> >>>>> >> ------------------------------------------------------------------------------- >> >>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time elapsed: >> 0.938 >> >>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest >> >>>>> testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> >>>>> Time elapsed: 0.104 sec <<< ERROR! >> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >> >>>>> Application Package >> >>>>> at >> >>>>> >> com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235) >> >>>>> >> >>>>> >> testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest) >> >>>>> Time elapsed: 0.045 sec <<< FAILURE! >> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but >> >>>>> was:<[package]-default> >> >>>>> at >> >>>>> >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143) >> >>>>> >> >>>>> >> testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> >>>>> Time elapsed: 0.045 sec <<< ERROR! >> >>>>> com.datatorrent.stram.cli.ApexCli$CliException: No applications in >> >>>>> Application Package matching "MyFirstApplication" >> >>>>> at >> >>>>> >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155) >> >>>>> >> >>>>> >> testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest) >> >>>>> Time elapsed: 0.058 sec <<< FAILURE! >> >>>>> org.junit.ComparisonFailure: expected:<[app]-default> but >> >>>>> was:<[package]-default> >> >>>>> at >> >>>>> >> com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204) >> >>>>> >> >>>>> >> >>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert < >> aa...@punchcyber.com> >> >>>>> wrote: >> >>>>> >> >>>>>> Thanks, Thomas! I have implemented the fix you suggested >> >>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket: >> >>>>>> https://issues.apache.org/jira/browse/APEXCORE-816 >> >>>>>> >> >>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise <t...@apache.org> >> wrote: >> >>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> It is not necessary to use a different stream codec when you have >> >>>>>>> control over the class that is causing the error. You can >> annotate a >> >>>>>>> different field serializer instead, using the >> @FieldSerializer.Bind >> >>>>>>> annotation. Here is an example: >> >>>>>>> >> >>>>>>> >> >>>>>>> >> https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50 >> >>>>>>> >> >>>>>>> It would be good to upgrade Kryo to 4.x in apex-core master. Would >> >>>>>>> you like to create a JIRA for it? ( >> >>>>>>> http://apex.apache.org/contributing.html) >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Thomas >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert < >> >>>>>>> aa...@punchcyber.com> wrote: >> >>>>>>> >> >>>>>>>> I am writing an application that includes a custom object. One >> of >> >>>>>>>> the fields is of type Instant and seems to not agree with the >> version of >> >>>>>>>> Kryo that is used in Apex (see exception below). I now that >> kryo 4.X >> >>>>>>>> supports serializing Instant fields, but for the life of me, >> cannot seem to >> >>>>>>>> find a way to either get Apex to use a newer version of Kryo. >> Can anyone >> >>>>>>>> help? Perhaps setting a different streamcodec that uses the >> newer version >> >>>>>>>> of kryo would work, but I cannot seem to find an example of how >> to set a >> >>>>>>>> different streamcodec despite references to the two (Kryo and >> Java) that >> >>>>>>>> ship with Apex. >> >>>>>>>> >> >>>>>>>> ERROR com.datatorrent.stram.codec.DefaultStatefulStreamCodec - >> >>>>>>>> Catastrophic Error: Execution halted due to Kryo exception! >> >>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot be created >> >>>>>>>> (missing no-arg constructor): java.time.Instant >> >>>>>>>> Serialization trace: >> >>>>>>>> systemTime >> >>>>>>>> (com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent) >> >>>>>>>> at >> >>>>>>>> >> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) >> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) >> >>>>>>>> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) >> >>>>>>>> at >> >>>>>>>> >> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) >> >>>>>>>> at >> >>>>>>>> >> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) >> >>>>>>>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >> >>>>>>>> at >> >>>>>>>> >> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) >> >>>>>>>> at >> >>>>>>>> >> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528) >> >>>>>>>> at >> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >> >>>>>>>> at com.datatorrent.stram.codec.De >> >>>>>>>> >> faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98) >> >>>>>>>> at >> >>>>>>>> >> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391) >> >>>>>>>> at >> >>>>>>>> >> com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339) >> >>>>>>>> at >> >>>>>>>> >> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269) >> >>>>>>>> at >> >>>>>>>> >> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429) >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> >> >>>>>>>> M. Aaron Bossert >> >>>>>>>> (571) 242-4021 >> >>>>>>>> Punch Cyber Analytics Group >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> M. Aaron Bossert >> >>>>>> (571) 242-4021 >> >>>>>> Punch Cyber Analytics Group >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> >> >>>>> M. Aaron Bossert >> >>>>> (571) 242-4021 >> >>>>> Punch Cyber Analytics Group >> >>>>> >> >>>>> >> >>>>> >> >>> >> >>> -- >> >>> >> >>> M. Aaron Bossert >> >>> (571) 242-4021 >> >>> Punch Cyber Analytics Group >> >>> >> >>> >> >>> >> > >> > -- >> > >> > M. Aaron Bossert >> > (571) 242-4021 >> > Punch Cyber Analytics Group >> > >> > >> > >> >> -- >> >> M. Aaron Bossert >> (571) 242-4021 >> Punch Cyber Analytics Group >> > -- M. Aaron Bossert (571) 242-4021 Punch Cyber Analytics Group
