Re: Spark 2.4.1 on Kubernetes - DNS resolution of driver fails

2019-05-03 Thread Olivier Girardot
Hi, I did not try on another vendor, so I can't say if it's only related to gke, and no, I did not notice anything on the kubelet or kube-dns processes... Regards Le ven. 3 mai 2019 à 03:05, Li Gao a écrit : > hi Olivier, > > This seems a GKE specific issue? have you tried on other vendors ? Al

Re: [VOTE] Release Apache Spark 2.4.3

2019-05-03 Thread antonkulaga
Can you prove release version for Hadoop 3 and Scala 2.12 this time? -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 2.4.3

2019-05-03 Thread Sean Owen
Hadoop 3 has not been supported in 2.4.x. 2.12 has been since 2.4.0, and 2.12 artifacts have always been released where available. What are you referring to? On Fri, May 3, 2019 at 9:28 AM antonkulaga wrote: > > Can you prove release version for Hadoop 3 and Scala 2.12 this time? > -

Howto force spark to honor parquet partitioning

2019-05-03 Thread Tomas Bartalos
Hello, I have partitioned parquet files based on "event_hour" column. After reading parquet files to spark: spark.read.format("parquet").load("...") Files from the same parquet partition are scattered in many spark partitions. Example of mapping spark partition -> parquet partition: Spark partit

Re: Preserving cache name and storage level upon table refresh

2019-05-03 Thread William Wong
Hi @gatorsmile, @cloud-fan and Sean, Thanks for previous review and suggestions. I updated the test case to cover storage level, fixed the typo on the migration note and applied some other enhancements. Sorry that I accidentally reformat the QueryTest.scala and introduced many unnecessary changes