cool! I'm on it.
On Tue, Jun 19, 2018 at 11:00 PM Pramod Immaneni
<pramod.imman...@gmail.com <mailto:pramod.imman...@gmail.com>> wrote:
Hi Aaron,
That would be great if you can pitch in wherever you can. Let us
know how we can help. Also, I was going to try the
new kryo version later today and see if there are any internal
errors that are bubbling up to those test failures but not getting
logged.
Thanks
On Tue, Jun 19, 2018 at 7:54 PM Aaron Bossert
<aa...@punchcyber.com <mailto:aa...@punchcyber.com>> wrote:
P.S. I am happy to roll up my sleeves, but from my initial
attempt, errors
I was seeing were not obviously related to the Kryo version
update (though
I am certain they are...at least indirectly because they only
show up when
the version is changed), in terms of tracking down where the
changes would
be required...I am not really familiar with the inner workings
of Apex yet,
though this would definitely be a good excuse to learn...
I also saw a bunch of annoying warnings when building related
to use of
deprecated methods/classes within the com.datatorrent.*
namespace...I'd be
happy to start tracking some of those down too if it would be
helpful...
On Tue, Jun 19, 2018 at 10:48 PM Aaron Bossert
<aa...@punchcyber.com <mailto:aa...@punchcyber.com>> wrote:
> Pramod,
>
> In a nutshell, yes. I can set a @FieldSerializer to the
fields in
> question...the only one I have run into thus far is
Instant...but it would
> be great to not need to annotate each field this way...Also,
I could see
> this becoming problematic should I encounter more than a
couple of fields
> that are not serializable using the older version of Kryo
(but presumably
> would work with a more current one)...though admitedly, it
would not be
> that big a deal in the interim. It would be much cleaner to
have an update
> to the most current version of Kryo, IMHO...though I
understand that is
> something of a lift. By the way, Kryo 5 just came out
yesterday, so if
> there is to be effort expended, it might be good to go with
the latest
> version...though, heads up, I tried it out and there seem to
be quite a few
> changes that will be needed...new methods, and definitely
some replaced
> ones...
>
>
>
> On Tue, Jun 19, 2018 at 10:37 PM Pramod Immaneni <
> pramod.imman...@gmail.com
<mailto:pramod.imman...@gmail.com>> wrote:
>
>> Hi Aaron,
>>
>> While we are debugging the test failures on dev, I didn't fully
>> understand the last question in your email. Looks like you
mentioned that
>> the workaround for adding the @FieldSerializer to the field
is working. Are
>> you looking for an alternative to this workaround and
trying to set a
>> default serializer for all fields of type Instant (the
field type you
>> mentioned in the earlier email) so that you don't have to
set the
>> annotation each time?
>>
>> Thanks
>>
>> On Tue, Jun 19, 2018 at 12:59 PM Aaron Bossert
<aa...@punchcyber.com <mailto:aa...@punchcyber.com>>
>> wrote:
>>
>>> I sent an e-mail to the dev list...but have not heard any
responses.
>>> How active is that list? Do you know? Basically, I have a
workaround that
>>> will deal with those "things" that Kryo v 2.24.0 will not
handle (but 4.0.2
>>> or 5.0.0-RC1 would) using the @FieldSerializer ...but this
is not ideal. I
>>> have two options left: go it alone and try to update Kryo
in the apex-core,
>>> or simply change the default serializer. The hiccup I am
running into is
>>> that I don't see how (from the documentation) to set the
default
>>> serializer...perhaps I am missing it. Can someone point
me to where that
>>> is in the docs?
>>>
>>> On Mon, Jun 11, 2018 at 4:20 PM Aaron Bossert
<aa...@punchcyber.com <mailto:aa...@punchcyber.com>>
>>> wrote:
>>>
>>>> Ah, you make a good point about the dev list...I am using
IntelliJ on a
>>>> Mac. When I tried building without any changes, I also
encountered
>>>> failures. I can re-run the non-modified sources to do a
diff and see if
>>>> the errors were different...
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 11, 2018, at 15:11, Pramod Immaneni
<pramod.imman...@gmail.com <mailto:pramod.imman...@gmail.com>>
>>>> wrote:
>>>>
>>>> Aaron,
>>>>
>>>> The question might be best served on the
dev@apex.apache.org <mailto:dev@apex.apache.org> mailing
>>>> list as you are starting to make changes to the sources.
Nothing obvious
>>>> jumps out to be, you don't need hadoop on your system to
build the sources
>>>> successfully with all the tests passing. The test
failures you are seeing
>>>> may or may not be related to the kryo change. What
happens when you try to
>>>> build the original sources with changing the Kryo
version? What is your
>>>> environment, mac, linux?
>>>>
>>>> On Mon, Jun 11, 2018 at 12:03 PM Aaron Bossert
<aa...@punchcyber.com <mailto:aa...@punchcyber.com>>
>>>> wrote:
>>>>
>>>>> I know it has been a little while since I brought this
up...I pulled
>>>>> down the 3.7.1-RELEASE branch from GitHub in order to
test out the change
>>>>> to Kryo v. 4.0.2 and encountered no problems while
building, however, when
>>>>> I try to do mvn install, there are several failed
tests. I am hoping, that
>>>>> other than the obvious (to me, anyway) errors such as
not finding
>>>>> yarn-site.xml on my laptop which does not have hadoop
installed, someone
>>>>> may be able to see what is going on?
>>>>>
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramRecoveryTest
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 6.372
>>>>> sec <<< FAILURE! - in
com.datatorrent.stram.StramRecoveryTest
>>>>>
testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time
>>>>> elapsed: 0.115 sec <<< FAILURE!
>>>>> java.lang.AssertionError: flush count expected:<1> but
was:<2>
>>>>> at
>>>>>
com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326)
>>>>>
>>>>>
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.engine.StatsTest
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 21.894
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest
>>>>>
testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest)
>>>>> Time elapsed: 3.266 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Validate input port queue size -1
>>>>> at
>>>>>
com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270)
>>>>> at
>>>>>
com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285)
>>>>>
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramMiniClusterTest
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 0.002
>>>>> sec <<< FAILURE! - in
com.datatorrent.stram.StramMiniClusterTest
>>>>> com.datatorrent.stram.StramMiniClusterTest Time
elapsed: 0.002 sec
>>>>> <<< ERROR!
>>>>> java.io.FileNotFoundException:
>>>>>
/Users/mbossert/Library/Mobile%20Documents/com~apple~CloudDocs/Punch/IdeaProjects/apex-core/engine/target/test-classes/yarn-site.xml
>>>>> (No such file or directory)
>>>>> at
>>>>>
com.datatorrent.stram.StramMiniClusterTest.setup(StramMiniClusterTest.java:151)
>>>>>
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.StramLocalClusterTest
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
elapsed: 15.081
>>>>> sec <<< FAILURE! - in
com.datatorrent.stram.StramLocalClusterTest
>>>>> testAppPath(com.datatorrent.stram.StramLocalClusterTest)
Time
>>>>> elapsed: 0.138 sec <<< FAILURE!
>>>>> java.lang.AssertionError: checkpoint path exists
>>>>> at
>>>>>
com.datatorrent.stram.StramLocalClusterTest.testAppPath(StramLocalClusterTest.java:413)
>>>>>
>>>>>
testLocalClusterInitShutdown(com.datatorrent.stram.StramLocalClusterTest)
>>>>> Time elapsed: 5.191 sec <<< FAILURE!
>>>>> java.lang.AssertionError:
>>>>>
./target/com.datatorrent.stram.StramLocalClusterTest-testLocalClusterInitShutdown.out
>>>>> exists
>>>>> at
>>>>>
com.datatorrent.stram.StramLocalClusterTest.testLocalClusterInitShutdown(StramLocalClusterTest.java:117)
>>>>>
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.client.AppPackageTest
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Tests run: 4, Failures: 2, Errors: 0, Skipped: 0, Time
elapsed: 0.105
>>>>> sec <<< FAILURE! - in
com.datatorrent.stram.client.AppPackageTest
>>>>>
testAppPackage(com.datatorrent.stram.client.AppPackageTest) Time
>>>>> elapsed: 0 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Number of applications
expected:<2> but
>>>>> was:<0>
>>>>> at
>>>>>
com.datatorrent.stram.client.AppPackageTest.testAppPackage(AppPackageTest.java:91)
>>>>>
>>>>>
testAppLevelRequiredAndDefaultProperties(com.datatorrent.stram.client.AppPackageTest)
>>>>> Time elapsed: 0.001 sec <<< FAILURE!
>>>>> java.lang.AssertionError: Should consist of an app called
>>>>> MyFirstApplication
>>>>> at
>>>>>
com.datatorrent.stram.client.AppPackageTest.testAppLevelRequiredAndDefaultProperties(AppPackageTest.java:171)
>>>>>
>>>>>
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Test set: com.datatorrent.stram.cli.ApexCliTest
>>>>>
>>>>>
-------------------------------------------------------------------------------
>>>>> Tests run: 7, Failures: 2, Errors: 2, Skipped: 0, Time
elapsed: 0.938
>>>>> sec <<< FAILURE! - in com.datatorrent.stram.cli.ApexCliTest
>>>>>
testAppFromOnlyConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.104 sec <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
applications in
>>>>> Application Package
>>>>> at
>>>>>
com.datatorrent.stram.cli.ApexCliTest.testAppFromOnlyConfigPackage(ApexCliTest.java:235)
>>>>>
>>>>>
testLaunchAppPackagePropertyPrecedence(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.045 sec <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>>
com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePropertyPrecedence(ApexCliTest.java:143)
>>>>>
>>>>>
testLaunchAppPackageParametersWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.045 sec <<< ERROR!
>>>>> com.datatorrent.stram.cli.ApexCli$CliException: No
applications in
>>>>> Application Package matching "MyFirstApplication"
>>>>> at
>>>>>
com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackageParametersWithConfigPackage(ApexCliTest.java:155)
>>>>>
>>>>>
testLaunchAppPackagePrecedenceWithConfigPackage(com.datatorrent.stram.cli.ApexCliTest)
>>>>> Time elapsed: 0.058 sec <<< FAILURE!
>>>>> org.junit.ComparisonFailure: expected:<[app]-default> but
>>>>> was:<[package]-default>
>>>>> at
>>>>>
com.datatorrent.stram.cli.ApexCliTest.testLaunchAppPackagePrecedenceWithConfigPackage(ApexCliTest.java:204)
>>>>>
>>>>>
>>>>> On Thu, May 17, 2018 at 1:31 PM, Aaron Bossert
<aa...@punchcyber.com <mailto:aa...@punchcyber.com>>
>>>>> wrote:
>>>>>
>>>>>> Thanks, Thomas! I have implemented the fix you suggested
>>>>>> (@FieldSerializer.BInd) and also created a JIRA ticket:
>>>>>> https://issues.apache.org/jira/browse/APEXCORE-816
>>>>>>
>>>>>> On Thu, May 17, 2018 at 1:31 AM, Thomas Weise
<t...@apache.org <mailto:t...@apache.org>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It is not necessary to use a different stream codec
when you have
>>>>>>> control over the class that is causing the error. You
can annotate a
>>>>>>> different field serializer instead, using the
@FieldSerializer.Bind
>>>>>>> annotation. Here is an example:
>>>>>>>
>>>>>>>
>>>>>>>
https://github.com/apache/apex-malhar/blob/2fe2903bfe65055c7fd361a97549b6e7e12d4289/library/src/main/java/org/apache/apex/malhar/lib/window/WindowOption.java#L50
>>>>>>>
>>>>>>> It would be good to upgrade Kryo to 4.x in apex-core
master. Would
>>>>>>> you like to create a JIRA for it? (
>>>>>>> http://apex.apache.org/contributing.html)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 16, 2018 at 10:31 AM, Aaron Bossert <
>>>>>>> aa...@punchcyber.com <mailto:aa...@punchcyber.com>> wrote:
>>>>>>>
>>>>>>>> I am writing an application that includes a custom
object. One of
>>>>>>>> the fields is of type Instant and seems to not agree
with the version of
>>>>>>>> Kryo that is used in Apex (see exception below). I
now that kryo 4.X
>>>>>>>> supports serializing Instant fields, but for the life
of me, cannot seem to
>>>>>>>> find a way to either get Apex to use a newer version
of Kryo. Can anyone
>>>>>>>> help? Perhaps setting a different streamcodec that
uses the newer version
>>>>>>>> of kryo would work, but I cannot seem to find an
example of how to set a
>>>>>>>> different streamcodec despite references to the two
(Kryo and Java) that
>>>>>>>> ship with Apex.
>>>>>>>>
>>>>>>>> ERROR
com.datatorrent.stram.codec.DefaultStatefulStreamCodec -
>>>>>>>> Catastrophic Error: Execution halted due to Kryo
exception!
>>>>>>>> com.esotericsoftware.kryo.KryoException: Class cannot
be created
>>>>>>>> (missing no-arg constructor): java.time.Instant
>>>>>>>> Serialization trace:
>>>>>>>> systemTime
>>>>>>>>
(com.punchcyber.streaming.pipeline.apex.datatypes.evtx.EvtxEvent)
>>>>>>>> at
>>>>>>>>
com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228)
>>>>>>>> at
com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049)
>>>>>>>> at
com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058)
>>>>>>>> at
>>>>>>>>
com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547)
>>>>>>>> at
>>>>>>>>
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523)
>>>>>>>> at
com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
>>>>>>>> at
>>>>>>>>
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
>>>>>>>> at
>>>>>>>>
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
>>>>>>>> at
com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
>>>>>>>> at com.datatorrent.stram.codec.De
<http://com.datatorrent.stram.codec.De>
>>>>>>>>
faultStatefulStreamCodec.fromDataStatePair(DefaultStatefulStreamCodec.java:98)
>>>>>>>> at
>>>>>>>>
com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.processPayload(BufferServerSubscriber.java:391)
>>>>>>>> at
>>>>>>>>
com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:339)
>>>>>>>> at
>>>>>>>>
com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
>>>>>>>> at
>>>>>>>>
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1429)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> M. Aaron Bossert
>>>>>>>> (571) 242-4021
>>>>>>>> Punch Cyber Analytics Group
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> M. Aaron Bossert
>>>>>> (571) 242-4021
>>>>>> Punch Cyber Analytics Group
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (571) 242-4021
>>>>> Punch Cyber Analytics Group
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>>
>>> M. Aaron Bossert
>>> (571) 242-4021
>>> Punch Cyber Analytics Group
>>>
>>>
>>>
>
> --
>
> M. Aaron Bossert
> (571) 242-4021
> Punch Cyber Analytics Group
>
>
>
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
--
M. Aaron Bossert
(571) 242-4021
Punch Cyber Analytics Group
