That's a good catch. I just checked the jstack, ps -ef of executor process.
they are progressing and completing much after the result generation.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To un
Is it possible there is a bug for the UI? If you can run jstack on the
executor process to see whether anything is actually running, that can help
narrow down the issue.
On Tue, Dec 26, 2017 at 10:28 PM ckhari4u wrote:
> Hi Reynold,
>
> I am running a Spark SQL query.
>
> val df = spark.sql("sel
Hi Reynold,
I am running a Spark SQL query.
val df = spark.sql("select * from table1 t1 join table2 t2 on
t1.col1=t2.col1")
df.count()
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To unsubscrib
What did you run?
On Tue, Dec 26, 2017 at 10:21 PM, ckhari4u wrote:
> Hi Sean,
>
> Thanks for the reply. I believe I am not facing the scenarios you
> mentioned.
>
> Timestamp conflict: I see the Spark driver logs on the console (tried with
> INFO and DEBUG). In all the scenarios, I see the res
Hi Sean,
Thanks for the reply. I believe I am not facing the scenarios you mentioned.
Timestamp conflict: I see the Spark driver logs on the console (tried with
INFO and DEBUG). In all the scenarios, I see the result getting printed and
the application execution continues for 4 more minutes.
ie
My guess is that either they haven't actually finished before the result
and something about timestamps you're comparing is misleading, or else,
you're looking at stages executing that are part of a later part of the
program.
On Tue, Dec 26, 2017 at 3:49 PM ckhari4u wrote:
> I found this interes
I found this interesting behavior while running some adhoc analysis query. I
have a Spark SQL query where I am joining 2 tables and then performing a
count operation. In the Spark Web UI, I see there are 4 Stages getting
launched.
The interesting behavior I see here is that I see the result befor
