Agreed with the statement in quotes below whether one wants to do unit
tests or not It is a good practice to write code that way. But I think the
more painful and tedious task is to mock/emulate all the nodes such as
spark workers/master/hdfs/input source stream and all that. I wish there is
>
> Basically you abstract your transformations to take in a dataframe and
> return one, then you assert on the returned df
>
+1 to this suggestion. This is why we wanted streaming and batch
dataframes to share the same API.
ali <kanth...@gmail.com> wrote:
>
> Hi All,
>
> How to unit test spark streaming or spark in general? How do I test the
> results of my transformations? Also, more importantly don't we need to spawn
> master and worker JVM's either in one or multiple
in a dataframe and
return one, then you assert on the returned df
Regards
Sam
On Tue, 7 Mar 2017 at 12:05, kant kodali <kanth...@gmail.com> wrote:
> Hi All,
>
> How to unit test spark streaming or spark in general? How do I test the
> results of my transformations? Also, more importa
Hi All,
How to unit test spark streaming or spark in general? How do I test the
results of my transformations? Also, more importantly don't we need to
spawn master and worker JVM's either in one or multiple nodes?
Thanks!
kant
