Hi
I can run this in my pc.
I check the email chian. bitfox install the spark measure with python2 and he
launch the pyspark with python3. I think it's the reason.
Regards.
Hollis
Replied mail
| From | Mich Talebzadeh |
| Date | 12/25/2021 00:25 |
| To | Sean Owen |
| Cc |
is the reason you got the IOM and analysis exception.
my suggestion is you need checkpoint the dataframe when joined 200 dataframes.
so you can trancate the lineage. so the optimizer only analysis the 200
dataframe.
this will reduce the pressure of spark engine.
| |
Hollis
|
Replied
(1000) cross join range(1000) cross join range(100)").show()')
+-+
| count(1)|
+-+
|1|
+-+
|
Regards,
Hollis
At 2021-12-24 09:18:19, bit...@bitfox.top wrote:
>Hello list
