down votefavorite
<http://stackoverflow.com/questions/35009560/anyone-had-issues-with-subtraction-of-a-long-within-an-rdd?noredirect=1#>
I am having an issue with the subtraction of a long within an RDD to filter
out items in the RDD that are within a certain time range.
So my code filters an RDD of case class auctions, with an object of
successfulAuctions(Long, Int, String):
auctions.filter(it => relevantAuctions(it, successfulAuctions))
The successfulAuctions object is made up of a timestamp: Long, an itemID:
Int, and a direction: String (BUY/SELL).
The relevantAuctions function basically uses tail recursion to find the
auctions in a time range for the exact item and direction.
@tailrec
def relevantAuctions(auction: Auction, successfulAuctions:
List[(Long, String, String)]): Boolean = successfulAuctions match
{
case sample :: xs => if (isRelevantAuction(auction, sample) )
true else relevantAuctions(auction, xs)
case Nil => false
}
This then feeds into another method in the if statement that checks the
timestamp in the sample is within a 10ms range, and the item ID is the
same, as is the direction.
def isRelevantAuction(auction: Auction, successfulAuction: (Long,
String, String)): Boolean = {(successfulAuction.timestampNanos -
auction.timestampNanos) >= 0 &&
(successfulAuction.timestampNanos - auction.timestampNanos) < 10000000L &&
auction.itemID == successfulAuction.itemID &&
auction.direction== successfulAuction.direction
}
I am having issues where the range option is not entirely working. The
timestamps I am receiving back are not within the required range. Although
the Item ID and direction seems to be working successfully.
The results I am getting are as follows, when I have a timestamp of
1431651108749267459 for the successful auction, I am receiving other
auctions of a time GREATER than this, where it should be less.
The auctions I am receiving have the timestamps of:
143165110874932660314316511087493307321431651108749537901
Has anyone experienced this phenomenon?
Thanks!
