GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/321
METRON-443 Exception seen while running stellar or fixed query on pcap data
I believe this resolves this ticket, but was unable to prove it directly
with the provided information. However, this does solve a specific, input
dependent, bug that would result in the described error.
The original implementation of the comparator was generating microseconds
since epoch as Longs from the provided packets (seconds * 1000000 +
microseconds). While Long is sufficient for holding these values, a narrowing
conversion occurs during the intValue() call on the difference. The problem is
that narrowing conversions do not necessarily maintain sign. This can then
result in incorrect comparisons and the error seen. See
http://docs.oracle.com/javase/specs/jls/se8/html/jls-5.html#jls-5.1.3
> A narrowing conversion of a signed integer to an integral type T simply
discards all but the n lowest order bits, where n is the number of bits used to
represent type T. In addition to a possible loss of information about the
magnitude of the numeric value, this may cause the sign of the resulting value
to differ from the sign of the input value.
The code has been updated and cleaned up a bit to avoid this narrowing
conversion and avoid some unnecessary boxing while I was already in the code.
Given that it has plain longs, rather than calculate a difference, it just
calls `Long.compare()` on the generated microseconds since epoch. Debug
message is modified accordingly to not use the difference anymore, because it's
no longer calculated or used.
Unit tests are added, and the original function fails
`testLargeDifference()`, while the new one succeeds. This function adds twenty
years to the original date, and does a comparison. `compare(earlier, later)`
should return negative, but due to the narrowing conversion actually return
positive as demonstrated here:
> p1time:1469521273000000 p2time: 2100673273999999 delta: -631152000999999
-631152000999999 - After longValue()
2033081793 - After intValue()
I haven't run this up in a dev environment, given the relatively small size
of the change and the ability to directly unit test and prove the error. If
anyone wants me to run it up and do more involved testing, I'm happy to do that
too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron METRON-443
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/321.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #321
----
commit 1feb815869f3e0a3ffffcfdbc7b79fc26d645caf
Author: justinjleet <justinjl...@gmail.com>
Date: 2016-10-25T13:44:55Z
Fixing intValue call to use long comparator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
