Re: Spark stand-alone mode

2023-10-17 Thread Ilango
g.html#dynamic-resource-allocation > > On Mon, Sep 18, 2023 at 3:53 PM Ilango wrote: > >> >> Thanks all for your suggestions. Noted with thanks. >> Just wanted share few more details about the environment >> 1. We use NFS for data storage and data is in parquet for

Re: Spark stand-alone mode

2023-09-18 Thread Ilango
>> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >&

Spark stand-alone mode

2023-09-14 Thread Ilango
Hi all, We have 4 HPC nodes and installed spark individually in all nodes. Spark is used as local mode(each driver/executor will have 8 cores and 65 GB) in Sparklyr/pyspark using Rstudio/Posit workbench. Slurm is used as scheduler. As this is local mode, we are facing performance issue(as only

Datastore for GrpahX

2015-11-21 Thread Ilango Ravi
Hi I am trying to figure which Datastore I can use for storing data to be used with GraphX. Is there a good Graph database out there which I can use for storing Graph data for efficient data storage/retireval. thanks, ravi