Re: Dataproc serverless for Spark

2022-11-28 Thread Mich Talebzadeh
Thanks Can you please confirm when that work was being carried out if you recall? I opened the same question in Google Cloud Dataproc Discussions < cloud-dataproc-disc...@googlegroups.com>, see someone will have a better answer Also there is another feature called Dataproc on GKE which currently

Re: Dataproc serverless for Spark

2022-11-28 Thread Holden Karau
This sounds like a great question for the Google DataProc folks (I know there was some interesting work being done around it but I left before it was finished so I don't want to provide a possibly incorrect answer). If your a GCP customer try reaching out to their support for details. On Mon,

Re: [PySpark] Join using condition where each record may be joined multiple times

2022-11-28 Thread Oliver Ruebenacker
Hello, Thanks, I can do that. What I was hoping to hear is whether what I'm trying to do is even considered possible, and what would be the correct 'how' parameter? Best, Oliver On Sun, Nov 27, 2022 at 2:50 PM Artemis User wrote: > What if you just do a join with the first

Create Jira account

2022-11-28 Thread Gerben van der Huizen
Hello, I would like to contribute to the Apache Spark project through Jira, but according to this blog post I need to request an account via email ( https://infra.apache.org/jira-guidelines.html#who). Please let me know if you need any more details to create an account. Kind regards, Gerben van

Implement custom datasource (writer) for Spark3

2022-11-28 Thread guenterh.lists
Dear list, I'm trying to implement my own custom datasource writer for spark 3 to serialize DataSets to an external storage (in my case RDF Triple ). After reading various resources (books, articles, internet) I learned that the implementation changed from Spark1 via datasources v2 in

Re: Create Jira account

2022-11-28 Thread Sean Owen
-user@ Send me your preferred email and username for the ASF JIRA and I'll create it. On Mon, Nov 28, 2022 at 10:55 AM Gerben van der Huizen < gerbenvanderhui...@gmail.com> wrote: > Hello, > > I would like to contribute to the Apache Spark project through Jira, but > according to this blog post