Please share the results of df.explain()[1] for both. That should give us
some clues of what the differences are
[1]https://github.com/apache/spark/blob/e1c90d66bbea5b4cb97226610701b0389b734651/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L499
--
Sent from:
Hi,
Without more information it’s very difficult to work out what’s going on. If
possible can you do the following and make available to us.
1) for each query call explain() and post the output.
2) Run each query and then go to the sql tab in the spark ui. For each query
show us the plan.
3)
Hi All,
Can anyone help me here with my query?
Regards,
Neeraj
On Mon, Apr 1, 2019 at 9:44 AM neeraj bhadani
wrote:
> In Both the cases, I am trying to create a HIVE table based on Union on 2
> same queries.
>
> Not sure how internally it differs on the process of creation of HIVE
> table?
In Both the cases, I am trying to create a HIVE table based on Union on 2
same queries.
Not sure how internally it differs on the process of creation of HIVE table?
Regards,
Neeraj
On Sun, Mar 31, 2019 at 1:29 PM Jörn Franke wrote:
> Is the select taking longer or the saving to a file. You
Is the select taking longer or the saving to a file. You seem to only save in
the second case to a file
> Am 29.03.2019 um 15:10 schrieb neeraj bhadani :
>
> Hi Team,
>I am executing same spark code using the Spark SQL API and DataFrame API,
> however, Spark SQL is taking longer than
qry_1 and qry_2 are simple select query with groupBy clause.
Are there any specific queries which works in a different way for Spark SQL
and DataFrame API?
Regards,
Neeraj
On Sat, Mar 30, 2019 at 7:27 PM Jason Nerothin
wrote:
> Can you please quantify the difference and provide the query
Can you please quantify the difference and provide the query code?
On Fri, Mar 29, 2019 at 9:11 AM neeraj bhadani
wrote:
> Hi Team,
>I am executing same spark code using the Spark SQL API and DataFrame
> API, however, Spark SQL is taking longer than expected.
>
> PFB Sudo code.
>
>
Hi Team,
I am executing same spark code using the Spark SQL API and DataFrame
API, however, Spark SQL is taking longer than expected.
PFB Sudo code.
---
Case 1 : Spark SQL
