Hey Jim, There are some experimental (unstable) API's that support running jobs which might short-circuit:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1126 This can be used for doing online aggregations like you are describing. And in one or two cases we've exposed functions that rely on this: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala#L334 I would expect more robust support for online aggregation to show up in a future version of Spark. - Patrick On Tue, Oct 28, 2014 at 7:27 AM, Jim Carroll <jimfcarr...@gmail.com> wrote: > > We have some very large datasets where the calculation converge on a result. > Our current implementation allows us to track how quickly the calculations > are converging and end the processing early. This can significantly speed up > some of our processing. > > Is there a way to do the same thing is spark? > > A trivial example might be a column average on a dataset. As we're > 'aggregating' rows into columnar averages I can track how fast these > averages are moving and decide to stop after a low percentage of the rows > has been processed, producing an estimate rather than an exact value. > > Within a partition, or better yet, within a worker across 'reduce' steps, is > there a way to stop all of the aggregations and just continue on with > reduces of already processed data? > > Thanks > JIm > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Ending-a-job-early-tp17505.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org