Re: spark 3.2.1: Unexpected reuse of dynamic PVC

2022-03-16 Thread Andreas Weise
minor correction: >> (hence our *ReadWriteOnce* Storage should be sufficient right?... On Wed, Mar 16, 2022 at 11:33 AM Andreas Weise wrote: > Hi, > > when using dynamic allocation on k8s with dynamic pvc reuse, I face that > only few executors are running.

Re: RebaseDateTime with dynamicAllocation

2022-03-09 Thread Andreas Weise
teTime$.(RebaseDateTime.scala:300) at org.apache.spark.sql.catalyst.util.RebaseDateTime$.(RebaseDateTime.scala) ... 38 more On Wed, Mar 9, 2022 at 5:30 PM Andreas Weise wrote: > Full trace doesn't provide any further details. It looks like this: > > Py4JJavaError: An error occurred while calling

Re: RebaseDateTime with dynamicAllocation

2022-03-09 Thread Andreas Weise
> On Wed, Mar 9, 2022 at 10:08 AM Andreas Weise > wrote: > >> Hi, >> >> When playing around with spark.dynamicAllocation.enabled I face the >> following error after the first round of executors have been killed. >> >> Py4JJava

RebaseDateTime with dynamicAllocation

2022-03-09 Thread Andreas Weise
Hi, When playing around with spark.dynamicAllocation.enabled I face the following error after the first round of executors have been killed. Py4JJavaError: An error occurred while calling o337.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 18.0

Spark 2.4 Regression with posexplode and structs

2019-02-08 Thread Andreas Weise
Hi, after upgrading from 2.3.2 to 2.4.0 we recognized a regression when using posexplode() in conjunction with select of another struct fields. Given a structure like this: = >>> df = (spark.range(1) ... .withColumn("my_arr", array(lit("1"), lit("2"))) ...

Re: Structured streaming: Tried to fetch $offset but the returned record offset was ${record.offset}"

2018-08-16 Thread andreas . weise
On 2018/04/17 22:34:25, Cody Koeninger wrote: > Is this possibly related to the recent post on > https://issues.apache.org/jira/browse/SPARK-18057 ? > > On Mon, Apr 16, 2018 at 11:57 AM, ARAVIND SETHURATHNAM < > asethurath...@homeaway.com.invalid> wrote: > > > Hi, > > > > We have several