Yes, the same code, the same result.
In fact, the code has been running for a more one month. Before 1.5.0, the
performance is quite the same, So I doubt that it is causd by tungsten.
Gen
On Wed, Nov 4, 2015 at 4:05 PM, Rick Moritz wrote:
> Something to check (just in case):
Something to check (just in case):
Are you getting identical results each time?
On Wed, Nov 4, 2015 at 8:54 AM, gen tang wrote:
> Hi sparkers,
>
> I am using dataframe to do some large ETL jobs.
> More precisely, I create dataframe from HIVE table and do some operations.
>
Hi sparkers,
I am using dataframe to do some large ETL jobs.
More precisely, I create dataframe from HIVE table and do some operations.
And then I save it as json.
When I used spark-1.4.1, the whole process is quite fast, about 1 mins.
However, when I use the same code with spark-1.5.1(with