It is wonderful to see some idea. Now the questions: 1) What is a track segment? Ans) It is the line that contains two adjacent points when all points are arranged by time. Say a vehicle moves (t1, p1) -> (t2, p2) -> (t3, p3). Then the segments are (p1, p2), (p2, p3) when the time ordering is (t1 < t2 < t3) 2) What is Lag function. Ans) Sean's link explains it.
Little bit more to my requirement: What I need to calculate is a density Map of vehicles in a certain area. Because of a user specific requirement I can't use just points but I will have to use segments. I already have a gridRDD containing 1km polygons for the whole world. My approach is 1) create a tracksegmentRDD of Vehicle, segment 2) do a cartesian of tracksegmentRDD and gridRDD and for each row check if the segment intersects the polygon. If it does then count it as 1. 3) Group the result above by vehicle(probably reduceByKey(_ + _) ) to get the density Map I am checking an issue http://apache-spark-user-list.1001560.n3.nabble.com/Finding-previous-and-next-element-in-a-sorted-RDD-td12621.html which seems to have some potential. I will give it a try. ..Manas On Wed, Oct 15, 2014 at 2:55 AM, sowen [via Apache Spark User List] < ml-node+s1001560n16471...@n3.nabble.com> wrote: > You say you reduceByKey but are you really collecting all the tuples > for a vehicle in a collection, like what groupByKey does already? Yes, > if one vehicle has a huge amount of data that could fail. > > Otherwise perhaps you are simply not increasing memory from the default. > > Maybe you can consider using something like vehicle and *day* as a > key. This would make you process each day of data separately, but if > that's fine for you, might drastically cut down the data associated to > a single key. > > Spark Streaming has a windowing function, and there is a window > function for an entire RDD, but I am not sure if there is support for > a 'window by key' anywhere. You can perhaps get your direct approach > of collecting events working with some of the changes above. > > Otherwise I think you have to roll your own to some extent, creating > the overlapping buckets of data, which will mean mapping the data to > several copies of itself. This might still be quite feasible depending > on how big a lag you are thinking of. > > PS for the interested, this is what LAG is: > > http://www.oracle-base.com/articles/misc/lag-lead-analytic-functions.php#lag > > On Wed, Oct 15, 2014 at 1:37 AM, Manas Kar <[hidden email] > <http://user/SendEmail.jtp?type=node&node=16471&i=0>> wrote: > > > Hi, > > I have an RDD containing Vehicle Number , timestamp, Position. > > I want to get the "lag" function equivalent to my RDD to be able to > create > > track segment of each Vehicle. > > > > Any help? > > > > PS: I have tried reduceByKey and then splitting the List of position in > > tuples. For me it runs out of memory every time because of the volume of > > data. > > > > ...Manas > > > > For some reason I have never got any reply to my emails to the user > group. I > > am hoping to break that trend this time. :) > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [hidden email] > <http://user/SendEmail.jtp?type=node&node=16471&i=1> > For additional commands, e-mail: [hidden email] > <http://user/SendEmail.jtp?type=node&node=16471&i=2> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/Lag-function-equivalent-in-an-RDD-tp16448p16471.html > To start a new topic under Apache Spark User List, email > ml-node+s1001560n1...@n3.nabble.com > To unsubscribe from Lag function equivalent in an RDD, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16448&code=bWFuYXNkZWJhc2hpc2thckBnbWFpbC5jb218MTY0NDh8LTM0Nzc4MjUwMg==> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > ----- Manas Kar -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Lag-function-equivalent-in-an-RDD-tp16448p16498.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
