Why do you need to mock the read/write at all? Why not have your test CSV
file, and invoke it (which will perform the real Spark DF read of CSV),
write it, and assert on the output?
On Tue, Jul 14, 2020 at 12:19 PM Dark Crusader
wrote:
> Sorry I wasn't very clear in my last email.
>
> I have a
Hi,
For testing things like this you have a couple of options, you could isolate
all your business logic separately from your read/write/spark code which, in my
experience, makes the code harder to write and manage.
The other option is to accept that tests will be slower than you would expect
Sorry I wasn't very clear in my last email.
I have a function like this:
def main( read_file):
df = spark.read.csv(read_file)
** Some other code **
df.write.csv(path)
Which I need to write a unit test for.
Would pythons unittest mock help me here?
When I googled this, I
