Re: [pyspark 2.3.0] Task was denied committing errors

Any suggestions? On Wed, Nov 6, 2019 at 7:30 AM Rishi Shah wrote: > Hi All, > > I have two relatively big tables and join on them keeps throwing > TaskCommitErrors, eventually job succeeds but I was wondering what these > errors are and if there's any solution? > > -- > Regards, > > Rishi Shah

Re: Build customized resource manager

Any suggestions? - Klaus On Mon, Nov 4, 2019 at 5:04 PM Klaus Ma wrote: > Hi team, > > AFAIK, we built k8s/yarn/mesos as resource manager; but I'd like to did > some enhancement to them, e.g. integrate with Volcano > in k8s. Is that possible to do > that

What's the deal with --proxy-user?

Hi all, I'm trying to understand if the --proxy-user parameter to spark-submit is deprecated, or something similar? The reason I ask is because it's hard to find documentation really talking about it. The Spark Security doc doesn't mention it

Re: [DISCUSS] Remove sorting of fields in PySpark SQL Row construction

Sounds reasonable to me. We should make the behavior consistent within Spark. On Tue, Nov 5, 2019 at 6:29 AM Bryan Cutler wrote: > Currently, when a PySpark Row is created with keyword arguments, the > fields are sorted alphabetically. This has created a lot of confusion with > users because it

[pyspark 2.3.0] Task was denied committing errors

Hi All, I have two relatively big tables and join on them keeps throwing TaskCommitErrors, eventually job succeeds but I was wondering what these errors are and if there's any solution? -- Regards, Rishi Shah