Awesome! And Patrick just gave his LGTM ;-)
On Wed Nov 19 2014 at 5:13:17 PM Aniket Bhatnagar
aniket.bhatna...@gmail.com wrote:
Thanks for pointing this out Mark. Had totally missed the existing JIRA
items
On Wed Nov 19 2014 at 21:42:19 Mark Hamstra m...@clearstorydata.com
wrote:
This is
I have for now submitted a JIRA ticket @
https://issues.apache.org/jira/browse/SPARK-4473. I will collate all my
experiences ( hacks) and submit them as a feature request for public API.
On Tue Nov 18 2014 at 20:35:00 andy petrella andy.petre...@gmail.com
wrote:
yep, we should also propose to
Thanks for pointing this out Mark. Had totally missed the existing JIRA
items
On Wed Nov 19 2014 at 21:42:19 Mark Hamstra m...@clearstorydata.com wrote:
This is already being covered by SPARK-2321 and SPARK-4145. There are
pull requests that are already merged or already very far along --
This is already being covered by SPARK-2321 and SPARK-4145. There are pull
requests that are already merged or already very far along -- e.g.,
https://github.com/apache/spark/pull/3009
If there is anything that needs to be added, please add it to those issues
or PRs.
On Wed, Nov 19, 2014 at
I am writing yet another Spark job server and have been able to submit jobs
and return/save results. I let multiple jobs use the same spark context but
I set job group while firing each job so that I can in future cancel jobs.
Further, what I deserve to do is provide some kind of status
I started some quick hack for that in the notebook, you can head to:
https://github.com/andypetrella/spark-notebook/blob/master/common/src/main/scala/notebook/front/widgets/SparkInfo.scala
On Tue Nov 18 2014 at 2:44:48 PM Aniket Bhatnagar
aniket.bhatna...@gmail.com wrote:
I am writing yet
Thanks Andy. This is very useful. This gives me all active stages their
percentage completion but I am unable to tie stages to job group (or
specific job). I looked at Spark's code and to me, it
seems org.apache.spark.scheduler.ActiveJob's group ID should get propagated
to StageInfo (possibly in
yep, we should also propose to add this stuffs in the public API.
Any other ideas?
On Tue Nov 18 2014 at 4:03:35 PM Aniket Bhatnagar
aniket.bhatna...@gmail.com wrote:
Thanks Andy. This is very useful. This gives me all active stages their
percentage completion but I am unable to tie stages
