Raise exception whilst casting instead of defaulting to null

Hi all, The default behaviour of Spark is to add a null value for casts that fail, unless ANSI SQL is enabled, SPARK-30292 . Whilst I understand that this is a subset of ANSI compliant behaviour, I don't understand why this feature is so

Re: Potability of dockers built on different cloud platforms

The whole idea of creating a docker container is to have a reployable self contained utility. A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings. The

Re: Troubleshooting ArrayIndexOutOfBoundsException in long running Spark application

OK Spark Structured Streaming. How are you getting messages into Spark? Is it Kafka? This to me index that the message is incomplete or having another value in Json HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies London United Kingdom view my Linkedin

Troubleshooting ArrayIndexOutOfBoundsException in long running Spark application

Dear Apache Spark users, I have a long running Spark application that is encountering an ArrayIndexOutOfBoundsException once every two weeks. The exception does not disrupt the operation of my app, but I'm still concerned about it and would like to find a solution. Here's some

Troubleshooting ArrayIndexOutOfBoundsException in long running Spark application

Dear Apache Spark users, I have a long running Spark application that is encountering an ArrayIndexOutOfBoundsException once every two weeks. The exception does not disrupt the operation of my app, but I'm still concerned about it and would like to find a solution. Here's some

Re: Potability of dockers built on different cloud platforms

ashok34...@yahoo.com.INVALID wrote: Is it possible to use Spark docker built on GCP on AWS without rebuilding from new on AWS? I am using the spark image from bitnami for running on k8s. And yes, it's deployed by helm. -- https://kenpeng.pages.dev/

Potability of dockers built on different cloud platforms

Hello team Is it possible to use Spark docker built on GCP on AWS without rebuilding from new on AWS? Will that work please. AK