Re: Release Apache Spark 2.4.4

Thank you, DB, Takeshi, Hyukjin, Sean, Kazuaki, Holden, Wenchen! I'll create tag for 2.4.4-rc1 next Monday. For SPARK-27234, it looks like that to me, too. Thanks, Dongjoon. On Wed, Aug 14, 2019 at 9:13 AM Holden Karau wrote: > That looks like more of a feature than a bug fix unless I’m

Spark Structured Streaming XML content

I'm trying to analyze data using Kinesis source in PySpark Structured Streaming on Databricks. Ceeated a Dataframe as shown below. kinDF = spark.readStream.format("kinesis").("streamName", "test-stream-1").load() Converted the data from base64 encoding as below. df =

Re: Release Apache Spark 2.4.4

That looks like more of a feature than a bug fix unless I’m missing something? On Tue, Aug 13, 2019 at 11:58 PM Hyukjin Kwon wrote: > Adding Shixiong > > WDYT? > > 2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성: > >> Can the following be included? >> >> [SPARK-27234][SS][PYTHON] Use

Re: Spark SQL reads all leaf directories on a partitioned Hive table

Thank you, Subash. It works! On Tue, Aug 13, 2019 at 5:58 AM Subash Prabakar wrote: > I had the similar issue reading the external parquet table . In my case I > had permission issue in one partition so I added filter to exclude that > partition but still the spark didn’t prune it. Then I read

Re: Release Apache Spark 2.4.4

Adding Shixiong WDYT? 2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성: > Can the following be included? > > [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in > EpochTracker (to support Python UDFs) > > > Thanks, > Terry > > On Tue,