Re: [Structured Streaming] NullPointerException in long running query

It should be a problem of my data quality. It's curious why the driver-side exception stack has no specific exception information. Edgardo Szrajber 于2020年4月28日周二 下午3:32写道： > The exception occured while aborting the stage. It might be interesting to > try to understand the reason for the

Re: Read Hive ACID Managed table in Spark

Hi Amogh, Thanks for your reply. I'll send you seperate email for the communication. On Thu, Apr 9, 2020 at 1:11 PM amogh margoor wrote: > Sorry for the late reply. > I can help you with getting started with > https://github.com/qubole/spark-acid to read Hive ACID tables. Feel free > to drop

Re: Unablee to get to_timestamp with Timezone Information

Thanks Enrico, Magnus On Thu, Apr 2, 2020 at 11:49 AM Enrico Minack wrote: > Once parsed into a Timestamp the timestamp is store internally as UTC and > printed as your local timezone (e.g. as defined by > spark.sql.session.timeZone). Spark is good at hiding timezone information > from you. > >

How can I add extra mounted disk to HDFS

Hi Spark Users, My spark job gave me an error No Space left on the device

[no subject]

Unsubscribe Get Outlook for Android

Re: Converting a date to milliseconds with time zone in Scala with fixed date str

import java.time._ import java.util.Date import java.text.SimpleDateFormat import java.util.Locale import java.util.SimpleTimeZone import org.joda.time import org.joda.time.DateTime object CalendarDemo extends App { println("Calendar Demo") val pattern = "E dd M

Re: Spark 2.3 and Kafka client library version

You should be able to override the Kafka client version. The Kafka APIs used by Structured Streaming exist in new Kafka versions. There is a known correctness issue in Kafka 0.10.1.*. Other versions should be fine. Best Regards, Ryan On Tue,

Re: Converting a date to milliseconds with time zone in Scala

Hi, Thank you all, I am just thinking of passing that date 06/04/2020 12:03:43 and getting the correct format from the module. In effect This date format -MM-dd'T'HH:mm:ss.SZ as pattern in other words rather than new Date() pass "06/04/2020 12:03:43" as string REgards, Dr

Spark 2.3 and Kafka client library version

I have a keberized HDFS cluster. When I use structured streaming with Kafka (with SASL_SSL/PLAINTEXT), I believe I’m blocked by Kafka-5294. It seems like fix version in 0.11.0.0 Kafka client library. I have a Spark 2.3 cluster, and it’s using 0.10.0.1 kafka client library. Do you know if I can

Re: Converting a date to milliseconds with time zone in Scala

import java.time._ import java.util.Date import java.text.SimpleDateFormat import java.util.Locale import java.util.SimpleTimeZone object CalendarDemo extends App { println("Calendar Demo") val pattern = "E dd M HH:mm:ss.SSSZ"; val simpleDateFormat = (new

Re: Converting a date to milliseconds with time zone in Scala

import java.time._ import java.util.Date import java.text.SimpleDateFormat import java.util.Locale import java.util.SimpleTimeZone object CalendarDemo extends App { println("Calendar Demo") val pattern =3D "E dd M HH:mm:ss.SSSZ"; val simpleDateFormat =3D (new

Re: Converting a date to milliseconds with time zone in Scala

Hiplease check combining unix_timestamp and from_unixtime, Something like:  from_unixtime(unix_timestamp( "06-04-2020 12:03:43"),"-MM-dd'T'HH:mm:ss Z") please note that I just wrote without any validation. In any case, you might want to check the documentation of both functions to check all

Re: Converting a date to milliseconds with time zone in Scala

Unfortunately that did not work. any other suggestions? thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Converting a date to milliseconds with time zone in Scala

Thanks Neeraj, I'll check it out. ! Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it

Re: Converting a date to milliseconds with time zone in Scala

Hi Mich, You can try Spark DateTime function here and see if that helps. https://medium.com/expedia-group-tech/deep-dive-into-apache-spark-datetime-functions-b66de737950a Regards, Neeraj On Tue, Apr 28, 2020 at 5:15 PM Mich Talebzadeh wrote: > Hi, > > I have a date in format like

Converting a date to milliseconds with time zone in Scala

Hi, I have a date in format like 06/04/2020 12:03:43 and we want it to be displayed as follows: -MM-dd'T'HH:mm:ss.SZ format So the input date is GMT date time just we do not receive the information with it The output should have timezone information Appreciate any ideas.

Unsubscribe

unsubscribe From: Yasir Elgohary [mailto:ygoh...@gmail.com] Sent: Tuesday, April 28, 2020 8:18 AM To: user@spark.apache.org Subject: Unsubscribe... unsubscribe This communication and any attached files may contain information that is confidential or privileged. If this communication has been

Unsubscribe...

unsubscribe

Structured Streaming using Kafka Avro Record in 2.3.0

Hi How can we deserialise avro record read from kafka in spark 2.3.0 in optimised manner. I could see that native support for avro was added in 2.4.x. Currently i am using following library which is very slow. com.twitter bijection-avro_2.11 Kind Regards Harsh Takkar

Spark stable release for Hadoop 3

Hello, We are considering whether to use Hadoop or Kubernetes as the cluster manager for Spark. We would prefer to have Hadoop 3 because of its native support for scheduling GPUs. Although there is a Spark 3.0.0 pre-view2 version available that is pre-built for Hadoop 3, I would like to know

Re: [Structured Streaming] NullPointerException in long running query

The exception occured while aborting the stage. It might be interesting to try to understand the reason for the abortion.Maybe timeout? How long the query run?Bentzi Sent from Yahoo Mail on Android On Tue, Apr 28, 2020 at 9:25, Jungtaek Lim wrote: The root cause of exception is occurred

Re: SparkLauncher reliability and scalability

How about Apache Livy? the purpose is similar as SparkLauncher, but through Restful API to launch Spark Jobs From: mhd wrk Sent: Monday, April 27, 2020 11:38 PM To: user@spark.apache.org Subject: SparkLauncher reliability and scalability [Warning]: This email

Re: [Structured Streaming] NullPointerException in long running query

The root cause of exception is occurred in executor side "Lost task 10.3 in stage 1.0 (TID 81, spark6, executor 1)" so you may need to check there. On Tue, Apr 28, 2020 at 2:52 PM lec ssmi wrote: > Hi： > One of my long-running queries occasionally encountered the following > exception: > > >