Re: Unit test with sqlContext

2016-03-19 Thread Vikas Kawadia
bootstrapping for you. >>>> >>>> https://github.com/holdenk/spark-testing-base >>>> >>>> DataFrame examples are here: >>>> https://github.com/holdenk/spark-testing-base/blob/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleD

Re: Unit test with sqlContext

2016-02-05 Thread Steve Annessa
-testing-base/blob/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala >>> >>> Thanks, >>> Silvio >>> >>> From: Steve Annessa <steve.anne...@gmail.com> >>> Date: Thursday, February 4, 2016 at 8:36 PM >>>

Unit test with sqlContext

2016-02-04 Thread Steve Annessa
I'm trying to unit test a function that reads in a JSON file, manipulates the DF and then returns a Scala Map. The function has signature: def ingest(dataLocation: String, sc: SparkContext, sqlContext: SQLContext) I've created a bootstrap spec for spark jobs that instantiates the Spark Context

Re: Unit test with sqlContext

2016-02-04 Thread Silvio Fiorito
uot; <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Unit test with sqlContext I'm trying to unit test a function that reads in a JSON file, manipulates the DF and then returns a Scala Map. The function has signature: def ingest(dataLocation: String, sc: SparkContex

Re: Unit test with sqlContext

2016-02-04 Thread Rishi Mishra
ting-base/blob/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala >> >> Thanks, >> Silvio >> >> From: Steve Annessa <steve.anne...@gmail.com> >> Date: Thursday, February 4, 2016 at 8:36 PM >> To: "user@spark.apac

Re: Unit test with sqlContext

2016-02-04 Thread Holden Karau
e/blob/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala > > Thanks, > Silvio > > From: Steve Annessa <steve.anne...@gmail.com> > Date: Thursday, February 4, 2016 at 8:36 PM > To: "user@spark.apache.org" <user@spark.apache.org&