Optimal way to avoid processing null returns in Spark Scala

[swetha]

Hi,

I have the following functions that I am using for my job in Scala. If you
see the getSessionId function I am returning null sometimes. If I return
null the only way that I can avoid processing those records is by filtering
out null records. I wanted to avoid having another pass for filtering so I
tried returning "None" . But, it seems to be having issues as it demands the
return type as optional. What is the optimal way to avoid processing null
records and at the same way avoid having Option as the return type using
None? The use of Option[] and Some(()) seems to be having type issues in
subsequent function calls.


    val sessions = filteredStream.transform(rdd=>getBeaconMap(rdd))

  def getBeaconMap(rdd: RDD[(String, String)]): RDD[(String, (Long,
String))] = {
    rdd.map[(String, (Long, String))]{ case (x, y) =>
      ((getSessionId(y), (getTimeStamp(y).toLong,y)))
    }
  }

  def getSessionId(eventRecord:String): String = {
    val beaconTestImpl: BeaconTestLoader = new BeaconTestImpl//This needs to
be changed.
    val beaconEvent: BeaconEventData =
beaconTestImpl.getBeaconEventData(eventRecord)

    if(beaconEvent!=null){
       beaconEvent.getSessionID //This might be in Set Cookie header
    }else{
     null
}


    val groupedAndSortedSessions =
sessions.transform(rdd=>ExpoJobCommonNew.getGroupedAndSortedSessions(rdd))




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Optimal-way-to-avoid-processing-null-returns-in-Spark-Scala-tp24972.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org