: Ranju Jain
Cc: user@spark.apache.org
Subject: Re: Dynamic Allocation Backlog Property in Spark on Kubernetes
Hi!
For dynamic allocation you do not need to run the Spark jobs in parallel.
Dynamic allocation simply means Spark scales up by requesting more executors
when there are pending tasks
Hi All,
I have set dynamic allocation enabled while running spark on Kubernetes . But
new executors are requested if pending tasks are backlogged for more than
configured duration in property
"spark.dynamicAllocation.schedulerBacklogTimeout".
My Use Case is:
There are number of parallel jobs
.
Regards
Ranju
From: Attila Zsolt Piros
Sent: Monday, March 22, 2021 11:07 AM
To: Ranju Jain
Cc: Mich Talebzadeh ; user@spark.apache.org
Subject: Re: Can JVisual VM monitoring tool be used to Monitor Spark Executor
Memory and CPU
Hi Ranju!
I am quite sure for your requirement "monitor
,
damage or destruction of data or any other property which may arise from
relying on this email's technical content is explicitly disclaimed. The author
will in no case be liable for any monetary damages arising from such loss,
damage or destruction.
On Sat, 20 Mar 2021 at 16:06, Ranju Ja
process this data
further within Spark then please consider something way better: a columnar
storage format namely ORC or Parquet.
Best Regards,
Attila
From: Ranju Jain
Sent: Sunday, March 21, 2021 8:10 AM
To: user@spark.apache.org
Subject: Spark saveAsTextFile Disk Recommendation
Hi All,
I have a
Hi All,
I have a large RDD dataset of around 60-70 GB which I cannot send to driver
using collect so first writing that to disk using saveAsTextFile and then this
data gets saved in the form of multiple part files on each node of the cluster
and after that driver reads the data from that stora
Hi All,
Virtual Machine running an application, this application is having various
other 3PPs components running such as spark, database etc .
My requirement is to monitor every component and isolate the resources
consuming individually by every component.
I am thinking of using a common tool
Ok!
Thanks for all guidance :-)
Regards
Ranju
From: Mich Talebzadeh
Sent: Thursday, March 11, 2021 11:07 PM
To: Ranju Jain
Cc: user@spark.apache.org
Subject: Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS
I don't have any specific reference. However, you can do a Google search.
r any monetary damages arising from such loss,
damage or destruction.
On Thu, 11 Mar 2021 at 12:01, Ranju Jain
mailto:ranju.j...@ericsson.com>> wrote:
Hi Mich,
No, it is not Google cloud. It is simply Kubernetes deployed over Bare Metal
Platform.
I am not clear for pros and cons of Shared Vo
the other sides [drawback].
Regards
Ranju
From: Mich Talebzadeh
Sent: Thursday, March 11, 2021 5:22 PM
To: Ranju Jain
Cc: user@spark.apache.org
Subject: Re: Spark on Kubernetes | 3.0.1 | Shared Volume or NFS
Ok this is on Google Cloud correct?
LinkedIn
https://www.linkedin.com/profile
Hi,
I need to write all Executors pods data on some common location which can be
accessed and retrieved by driver pod.
I was first planning to go with NFS, but I think Shared Volume is equally good.
Please suggest Is there any major drawback in using Shared Volume instead of
NFS when many pods
terminates.
Or it works as ephemeral storage , which will
be there till executor is live?
1. GCP bucket shareable across driver Pod and Executor Pods.
Regards
Ranju
From: Mich Talebzadeh
Sent: Monday, March 8, 2021 8:32 PM
To: Ranju Jain
Cc: Ranju Jain
for Shared Storage Options available to persist the part files.
What is the best shared storage can be used to collate all executors part files
at one place.
Regards
Ranju
From: Mich Talebzadeh
Sent: Monday, March 8, 2021 8:06 PM
To: Ranju Jain
Cc: Attila Zsolt Piros ; user@spark.apache.org
Storage I should go for?
Regards
Ranju
From: Jacek Laskowski
Sent: Monday, March 8, 2021 4:14 PM
To: Ranju Jain
Cc: Attila Zsolt Piros ; user@spark.apache.org
Subject: Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part
Files Storage
Hi,
> as Executors terminates after their w
Hi,
I need to save the Executors processed data in the form of part files , but I
think persistent Volume is not an option for this as Executors terminates after
their work completes.
So I am thinking to use shared volume across executor pods.
Should I go with NFS or is there any other Volume o
Hi Attila,
Ok , I understood. I will switch on event logs .
Regards
Ranju
-Original Message-
From: Attila Zsolt Piros
Sent: Thursday, March 4, 2021 11:38 PM
To: user@spark.apache.org
Subject: RE: Spark Version 3.0.1 Gui Display Query
Hi Ranju!
I meant the event log would be very help
Hi Attila,
I checked the Section <
https://spark.apache.org/docs/latest/monitoring.html#web-interfaces> and Web
UI Page
What document is saying that if I want to view information only for the
duration of the application, then I do not need to
generate the event logs and do not need to set sp
. But currently it is blank shown.
[cid:image001.jpg@01D70F5B.0A47FED0]
[cid:image002.png@01D70F5B.0A47FED0]
Regards
Ranju
From: Kapil Garg
mailto:kapi...@flipkart.com.INVALID>>
Sent: Tuesday, March 2, 2021 11:39 AM
To: Ranju Jain
mailto:ranju.j...@ericsson.com.invalid>&
. But currently it is blank shown.
[cid:image004.jpg@01D70F5A.8D200C40]
[cid:image003.png@01D70F5A.0E04D530]
Regards
Ranju
From: Kapil Garg
Sent: Tuesday, March 2, 2021 11:39 AM
To: Ranju Jain
Cc: user@spark.apache.org
Subject: Re: Spark Version 3.0.1 Gui Display Query
Hi Ranju,
Is it
Hi ,
I started using Spark 3.0.1 version recently and noticed the Executors Tab on
Spark GUI appears as blank.
Please suggest what could be the reason of this type of display?
Regards
Ranju
Hi,
I submitted the spark job and pods goes in Pending state because of
insufficient resources.
But they are not getting deleted after this timeout of 60 sec. Please help me
in understanding.
Regards
Ranju
21 matches
Mail list logo
