[jira] [Commented] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

[discipleforteen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000419#comment-16000419
 ] 

discipleforteen commented on SPARK-20591:
-

i think that's better if it can be consistent

> Succeeded tasks num not equal in job page and job detail page on spark web ui 
> when speculative task(s) exist
> 
>
> Key: SPARK-20591
> URL: https://issues.apache.org/jira/browse/SPARK-20591
> Project: Spark
>  Issue Type: Bug
>  Components: Web UI
>Affects Versions: 2.0.2
>Reporter: Jinhua Fu
>Priority: Minor
> Attachments: job detail page(stages).png, job page.png
>
>
> when spark.speculation is enabled,and there are some speculative tasks, then 
> we can see succeeded tasks num include speculative tasks on the job page, 
> which however not being included on the job detail page(job stages page).
> When I consider some tasks may run a little slow by the job page's  succeeded 
> tasks more than total tasks,which make me want to known which tasks and why,I 
> have to check every stage to find the speculative tasks which is beacause 
> speculative tasks not being included in the stage succeeded task num.
> Can it be improved?
> update two screenshots, succeeded task num is 557 on job page,but 550(by sum) 
> on job detail page(stages),the extra 7 tasks are speculative tasks. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-19225) Spark SQL round constant double return null

[discipleforteen (JIRA)]

discipleforteen created SPARK-19225:
---

 Summary: Spark SQL round constant double return null 
 Key: SPARK-19225
 URL: https://issues.apache.org/jira/browse/SPARK-19225
 Project: Spark
  Issue Type: Bug
Affects Versions: 2.1.0, 2.0.2, 2.0.1, 2.0.0
Reporter: discipleforteen
 Fix For: 2.1.1


Spark SQL round constant double value may return null . like 'select round(4.4, 
2)' return null in spark 2.x. it's not compatible with spark 1.x version. it 
seems 4.4 is casted to decimal with new SqlBase.g4 , which is casted to double 
in spark 1.x version.  round decimal 4.4, 2 get null in changeprecision...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-19225) Spark SQL round constant double return null

[discipleforteen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-19225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823348#comment-15823348
 ] 

discipleforteen commented on SPARK-19225:
-

spark 1.4.1

> select round(4.4, 2);
2017-01-16 09:47:21,573 INFO ParseDriver: Parsing command: select round(4.4, 2)
2017-01-16 09:47:21,783 INFO ParseDriver: Parse Completed
2017-01-16 09:47:22,318 INFO SparkContext: Starting job: processCmd at 
CliDriver.java:423
2017-01-16 09:47:22,335 INFO DAGScheduler: Got job 0 (processCmd at 
CliDriver.java:423) with 1 output partitions (allowLocal=false)
2017-01-16 09:47:22,335 INFO DAGScheduler: Final stage: ResultStage 
0(processCmd at CliDriver.java:423)
2017-01-16 09:47:22,335 INFO DAGScheduler: Parents of final stage: List()
2017-01-16 09:47:22,339 INFO DAGScheduler: Missing parents: List()
2017-01-16 09:47:22,344 INFO DAGScheduler: Submitting ResultStage 0 
(MapPartitionsRDD[2] at processCmd at CliDriver.java:423), which has no missing 
parents
2017-01-16 09:47:22,382 INFO MemoryStore: ensureFreeSpace(3232) called with 
curMem=0, maxMem=278302556
2017-01-16 09:47:22,384 INFO MemoryStore: Block broadcast_0 stored as values in 
memory (estimated size 3.2 KB, free 265.4 MB)
2017-01-16 09:47:22,516 INFO MemoryStore: ensureFreeSpace(1852) called with 
curMem=3232, maxMem=278302556
2017-01-16 09:47:22,516 INFO MemoryStore: Block broadcast_0_piece0 stored as 
bytes in memory (estimated size 1852.0 B, free 265.4 MB)
2017-01-16 09:47:22,519 INFO BlockManagerInfo: Added broadcast_0_piece0 in 
memory on localhost:45436 (size: 1852.0 B, free: 265.4 MB)
2017-01-16 09:47:22,520 INFO SparkContext: Created broadcast 0 from broadcast 
at DAGScheduler.scala:874
2017-01-16 09:47:22,525 INFO DAGScheduler: Submitting 1 missing tasks from 
ResultStage 0 (MapPartitionsRDD[2] at processCmd at CliDriver.java:423)
2017-01-16 09:47:22,526 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
2017-01-16 09:47:22,562 INFO TaskSetManager: Starting task 0.0 in stage 0.0 
(TID 0, localhost, PROCESS_LOCAL, 1392 bytes)
2017-01-16 09:47:22,571 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
2017-01-16 09:47:22,643 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 
955 bytes result sent to driver
2017-01-16 09:47:22,658 INFO TaskSetManager: Finished task 0.0 in stage 0.0 
(TID 0) in 113 ms on localhost (1/1)
2017-01-16 09:47:22,658 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose 
tasks have all completed, from pool
2017-01-16 09:47:22,661 INFO DAGScheduler: ResultStage 0 (processCmd at 
CliDriver.java:423) finished in 0.126 s
2017-01-16 09:47:22,671 INFO DAGScheduler: Job 0 finished: processCmd at 
CliDriver.java:423, took 0.352237 s
4.4

 
 
spark 2.1.0

> select round(4.4, 2);
17/01/16 09:48:13 INFO SparkSqlParser: Parsing command: select round(4.4, 2)
17/01/16 09:48:15 INFO CodeGenerator: Code generated in 215.145435 ms
17/01/16 09:48:15 INFO SparkContext: Starting job: processCmd at 
CliDriver.java:376
17/01/16 09:48:15 INFO DAGScheduler: Got job 0 (processCmd at 
CliDriver.java:376) with 1 output partitions
17/01/16 09:48:15 INFO DAGScheduler: Final stage: ResultStage 0 (processCmd at 
CliDriver.java:376)
17/01/16 09:48:15 INFO DAGScheduler: Parents of final stage: List()
17/01/16 09:48:15 INFO DAGScheduler: Missing parents: List()
17/01/16 09:48:15 INFO DAGScheduler: Submitting ResultStage 0 
(MapPartitionsRDD[3] at processCmd at CliDriver.java:376), which has no missing 
parents
17/01/16 09:48:15 INFO MemoryStore: Block broadcast_0 stored as values in 
memory (estimated size 5.9 KB, free 408.9 MB)
17/01/16 09:48:15 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in 
memory (estimated size 3.1 KB, free 408.9 MB)
17/01/16 09:48:15 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 
10.9.233.62:34512 (size: 3.1 KB, free: 408.9 MB)
17/01/16 09:48:15 INFO SparkContext: Created broadcast 0 from broadcast at 
DAGScheduler.scala:996
17/01/16 09:48:15 INFO DAGScheduler: Submitting 1 missing tasks from 
ResultStage 0 (MapPartitionsRDD[3] at processCmd at CliDriver.java:376)
17/01/16 09:48:15 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/01/16 09:48:16 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 
localhost, executor driver, partition 0, PROCESS_LOCAL, 6244 bytes)
17/01/16 09:48:16 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/01/16 09:48:16 INFO CodeGenerator: Code generated in 8.289946 ms
17/01/16 09:48:16 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1295 
bytes result sent to driver
17/01/16 09:48:16 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) 
in 77 ms on localhost (executor driver) (1/1)
17/01/16 09:48:16 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have 
all completed, from pool
17/01/16 09:48:16 INFO DAGScheduler: ResultStage 0 (processCmd at 
CliDriver.java:376) finished in 0.097 s
17/01/16 09:48:16 INFO DAGScheduler: Job