Re: [VOTE] Release Spark 2.4.7 (RC1)

2020-08-08 Thread Takeshi Yamamuro
Thanks for letting us know about the two issues above, Dongjoon. I've checked the release materials (signatures, tag, ...) and it looks fine, too. Also, I run the tests on my local Mac (java 1.8.0) with the options `-Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Psparkr`

Re: [VOTE] Release Spark 2.4.7 (RC1)

2020-08-08 Thread Dongjoon Hyun
Another instance is SPARK-31703 which filed on May 13th and the PR arrived two days ago. [SPARK-31703][SQL] Parquet RLE float/double are read incorrectly on big endian platforms https://github.com/apache/spark/pull/29383 It seems that the patch is already ready in this case. I raised the

Re: Why is V2SessionCatalog not a CatalogExtension?

2020-08-08 Thread Russell Spitzer
A v2catalog can stand alone, it doesn't have to extend the built in catalog. On Sat, Aug 8, 2020, 7:36 AM Jacek Laskowski wrote: > Hi, > > Just started exploring Catalog Plugin API and noticed these two classes: > CatalogExtension and V2SessionCatalog. > > Why is V2SessionCatalog not a CatalogEx

Re: [VOTE] Release Spark 2.4.7 (RC1)

2020-08-08 Thread Holden Karau
I'm going to go ahead and vote -0 then based on that then. On Fri, Aug 7, 2020 at 11:36 PM Dongjoon Hyun wrote: > Hi, All. > > Unfortunately, there is an on-going discussion about the new decimal > correctness. > > Although we fixed one correctness issue at master and backported it > partially t

Why is V2SessionCatalog not a CatalogExtension?

2020-08-08 Thread Jacek Laskowski
Hi, Just started exploring Catalog Plugin API and noticed these two classes: CatalogExtension and V2SessionCatalog. Why is V2SessionCatalog not a CatalogExtension? - V2SessionCatalog extends TableCatalog with SupportsNamespaces [1] - CatalogExtension extends TableCatalog, SupportsNamespaces [2]

Re: Async RDD saves

2020-08-08 Thread Antonin Delpeuch (lists)
Hi both, Thanks for your replies! Sean, your proposal to use a driver-side future wrapping the blocking call sounds a lot easier indeed. But I want to ensure that canceling the future in the driver code kills the corresponding tasks on all executors. If I wrap the driver-side call in a standard

Re: Async RDD saves

2020-08-08 Thread kalyan
This looks interesting.. anyways, it will be good if you can elaborate more on the expectations and the various other ways you had tried before deciding to do it this way... Regards, Kalyan. On Fri, Aug 7, 2020, 11:24 PM Edward Mitchell wrote: > I will agree that the side effects of using Futur