????

Re: Spark Version 3.0.1 Gui Display Query

Hi Ranju, The screenshots and logs you shared are from spark driver and executor. I meant for you to check the web page logs in chrome console. There might be some error logs indicating why UI is unable to fetch the information. I have faced a similar problem when I was accessing spark UI via a

RE: Spark Version 3.0.1 Gui Display Query

Hi Attila, Ok , I understood. I will switch on event logs . Regards Ranju -Original Message- From: Attila Zsolt Piros Sent: Thursday, March 4, 2021 11:38 PM To: user@spark.apache.org Subject: RE: Spark Version 3.0.1 Gui Display Query Hi Ranju! I meant the event log would be very

Re: Spark Version 3.0.1 Gui Display Query

Well I cannot recall in Spark 3.0.1. However, this looks fine in Spark 3.1.1 (recent release) See attached the image from the executor tabs LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: [Spark SQL, intermediate+] possible bug or weird behavior of insertInto

That certainly is a solution if you know about the issue and we've used it in the end. I'm trying to find out if there is a solution that would prevent users who don't know about it from accidentally corrupting data. Something like "enable strict schema matching".

RE: Spark Version 3.0.1 Gui Display Query

Hi Ranju! I meant the event log would be very helpful for analyzing the problem at your side. The three logs together (driver, executors, event) is the best from the same run of course. I know you want check the executors tab during the job is running. And for this you do not need to

RE: Spark Version 3.0.1 Gui Display Query

Hi Attila, I checked the Section < https://spark.apache.org/docs/latest/monitoring.html#web-interfaces> and Web UI Page What document is saying that if I want to view information only for the duration of the application, then I do not need to generate the event logs and do not need to set

Re: Possible upgrade path from Spark 3.1.1-RC2 to Spark 3.1.1 GA

Ok, thanks. LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction

Re: Possible upgrade path from Spark 3.1.1-RC2 to Spark 3.1.1 GA

I think you're still asking about GCP and Dataproc, and that's really nothing to do with Spark itself. Whatever issues you are having concern Dataproc and how it's run and possibly customizations in Dataproc. 3.1.1-RC2 is not a release, but, also nothing meaningfully changed between it and the

Possible upgrade path from Spark 3.1.1-RC2 to Spark 3.1.1 GA

Hi, Is there any direct upgrade path from 3.1.1 RC-2 to 3.1.1 GA. If there is, will that involve replacing the Spark binaries? thanks, Mich LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: [Spark SQL, intermediate+] possible bug or weird behavior of insertInto

Why not perform a df.select(...) before the final write to ensure a consistent ordering. On Thu, Mar 4, 2021, 7:39 AM Oldrich Vlasic wrote: > Thanks for reply! Is there something to be done, setting a config property > for example? I'd like to prevent users (mainly data scientists) from >

Re: [Spark SQL, intermediate+] possible bug or weird behavior of insertInto

Thanks for reply! Is there something to be done, setting a config property for example? I'd like to prevent users (mainly data scientists) from falling victim to this. From: Russell Spitzer Sent: Wednesday, March 3, 2021 3:31 PM To: Sean Owen Cc: Oldrich Vlasic