Re: Spark Doubts

These are the basic concepts in spark :) You may take a bit time to read this small book: https://cloudcache.net/resume/PDDWS2-V2.pdf regards On Wed, Jun 22, 2022 at 3:17 AM Sid wrote: > Hi Team, > > I have a few doubts about the below questions: > > 1) data frame will reside where? memory?

Re: Spark Doubts

Dear Sid. You are asking questions for which answers exist in the Apache Spark website or in books or in MOOCS or in other URLs. For example, take a look at this one: https://sparkbyexamples.com/spark/spark-dataframe-cache-and-persist-explained/

Spark Doubts

Hi Team, I have a few doubts about the below questions: 1) data frame will reside where? memory? disk? memory allocation about data frame? 2) How do you configure each partition? 3) Is there any way to calculate the exact partitions needed to load a specific file? Thanks, Sid

Re: Spark Summit Europe

It's still held, just called the Data and AI Summit. https://databricks.com/dataaisummit/ Next one is next week; last one in Europe was in November 2020, and think it might be virtual in Europe if held separately this year. On Tue, Jun 21, 2022 at 7:38 AM Gowran, Declan wrote: > Announcing

spark-submit on kubernetes

Hi I am developing python applications. I use kubernetes to containerise my applications. I want to set up a spark cluster in kubernetes. I only want specific spark jobs to be processed by my spark cluster (for example large data ETL processes that would take long using python alone). I don’t

Spark Summit Europe

Announcing Spark Summit Europe | Apache Spark Hello, I see this link is linked to 2015 and does not appear to have update. Assume its not held anymore ? Declan Declan Gowran | Optum Global Advantage