nswer. I should sit down
>> and play with that some more.
>> >
>> >
>> >>
>> >> This poll and sleep strategy both makes tests quick in successful
>> >> cases, but still robust to occasional delays. The strategy does not
>> >>
in successful
>> >> cases, but still robust to occasional delays. The strategy does not
>> >> work if you want to test for absence, e.g. ensure that a particular
>> >> message if filtered. You can work around it by adding another message
>> >> af
gt;> message if filtered. You can work around it by adding another message
> >> afterwards and polling for its effect before testing for absence of
> >> the first. Be aware that messages can be processed out of order in
> >> Spark Streaming depending on partitioning, howeve
messages can be processed out of order in
>> Spark Streaming depending on partitioning, however.
>>
>>
>> I have tested Spark applications with both strategies described above,
>> and it is straightforward to set up. Let me know if you want
>> clarifications or as
2, 2016 at 6:54 PM, SRK <swethakasire...@gmail.com> wrote:
>> Hi,
>>
>> What is a good unit testing framework for Spark batch/streaming jobs? I have
>> core spark, spark sql with dataframes and streaming
egards,
>
>
>
> Lars Albertsson
> Data engineering consultant
> www.mapflat.com
> +46 70 7687109
>
>
> On Wed, Mar 2, 2016 at 6:54 PM, SRK <swethakasire...@gmail.com> wrote:
> > Hi,
> >
> > What is a good unit testing framework for Spark batch/s
-
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp26380.html
>> To start a new topic under Apache Spark User List, email [hidden ema
e spark, spark sql with dataframes and streaming api getting used. Any
> good framework to cover unit tests for these APIs?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp2638
/apache-spark-user-list.1001560.n3.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp26380.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1...@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-us
, spark sql with dataframes and streaming api getting used.
> Any good framework to cover unit tests for these APIs?
>
> Thanks!
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spa
at is a good unit testing framework for Spark batch/streaming jobs? I have
>core spark, spark sql with dataframes and streaming api getting used. Any
>good framework to cover unit tests for these APIs?
>
>Thanks!
>
>
>
>--
>View this message in context:
>http://apache-spark
k sql with dataframes and streaming api getting used. Any
> good framework to cover unit tests for these APIs?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp26380.html
>
.nabble.com/Unit-testing-framework-for-Spark-Jobs-tp26380.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h
13 matches
Mail list logo
