Hi, I made some changes to SPARK-24020 (https://github.com/apache/spark/pull/21109) and implemented spill-over to disk. I believe there are no objections to the implementation left and that this can now be merged.
Please take a look. Thanks, Petar Zečević Wenchen Fan @ 1970-01-01 01:00 CET: > Some updates for the JIRA tickets that we want to resolve before Spark 2.4. > > green: merged > orange: in progress > red: likely to miss > > SPARK-24374: Support Barrier Execution Mode in Apache Spark > The core functionality is finished, but we still need to add Python API. > Tracked by SPARK-24822 > > SPARK-23899: Built-in SQL Function Improvement > I think it's ready to go. Although there are still some functions working in > progress, the common ones are all merged. > > SPARK-14220: Build and test Spark against Scala 2.12 > It's close, just one last piece. Tracked by SPARK-25029 > > SPARK-4502: Spark SQL reads unnecessary nested fields from Parquet > Being reviewed. > > SPARK-24882: data source v2 API improvement > PR is out, being reviewed. > > SPARK-24252: Add catalog support in Data Source V2 > Being reviewed. > > SPARK-24768: Have a built-in AVRO data source implementation > It's close, just one last piece: the decimal type support > > SPARK-23243: Shuffle+Repartition on an RDD could lead to incorrect answers > It turns out to be a very complicated issue, there is no consensus about what > is the right fix yet. Likely to miss it in Spark 2.4 because it's a > long-standing issue, not a regression. > > SPARK-24598: Datatype overflow conditions gives incorrect result > We decided to keep the current behavior in Spark 2.4 and add some > document(already done). We will re-consider this change in Spark 3.0. > > SPARK-24020: Sort-merge join inner range optimization > There are some discussions about the design, I don't think we can get to a > consensus within Spark 2.4. > > SPARK-24296: replicating large blocks over 2GB > Being reviewed. > > SPARK-23874: upgrade to Apache Arrow 0.10.0 > Apache Arrow 0.10.0 has some critical bug fixes and is being voted, we should > wait a few days. > > According to the status, I think we should wait a few more days. Any > objections? > > Thanks, > Wenchen > > On Tue, Aug 7, 2018 at 3:39 AM Sean Owen <sro...@gmail.com> wrote: > > ... and we still have a few snags with Scala 2.12 support at > https://issues.apache.org/jira/browse/SPARK-25029 > > There is some hope of resolving it on the order of a week, so for the > moment, seems worth holding 2.4 for. > > On Mon, Aug 6, 2018 at 2:37 PM Bryan Cutler <cutl...@gmail.com> wrote: > > Hi All, > > I'd like to request a few days extension to the code freeze to complete the > upgrade to Apache Arrow 0.10.0, SPARK-23874. This upgrade includes several > key improvements and bug fixes. The RC vote just passed this morning and code > changes are complete in https://github.com/apache/spark/pull/21939. We just > need some time for the release artifacts to be available. Thoughts? > > Thanks, > Bryan --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
