You can make a PVC on K8S call it 300GB make a folder in yours dockerfile WORKDIR /opt/spark/work-dir RUN chmod g+w /opt/spark/work-dir
start spark with adding this .config("spark.kubernetes.driver.volumes.persistentVolumeClaim.300gb.options.claimName", "300gb") \ .config("spark.kubernetes.driver.volumes.persistentVolumeClaim.300gb.mount.path", "/opt/spark/work-dir") \ .config("spark.kubernetes.driver.volumes.persistentVolumeClaim.300gb.mount.readOnly", "False") \ .config("spark.kubernetes.executor.volumes.persistentVolumeClaim.300gb.options.claimName", "300gb") \ .config("spark.kubernetes.executor.volumes.persistentVolumeClaim.300gb.mount.path", "/opt/spark/work-dir") \ .config("spark.kubernetes.executor.volumes.persistentVolumeClaim.300gb.mount.readOnly", "False") \ .config("spark.local.dir", "/opt/spark/work-dir") lør. 6. apr. 2024 kl. 15:45 skrev Mich Talebzadeh <mich.talebza...@gmail.com >: > I have seen some older references for shuffle service for k8s, > although it is not clear they are talking about a generic shuffle > service for k8s. > > Anyhow with the advent of genai and the need to allow for a larger > volume of data, I was wondering if there has been any more work on > this matter. Specifically larger and scalable file systems like HDFS, > GCS , S3 etc, offer significantly larger storage capacity than local > disks on individual worker nodes in a k8s cluster, thus allowing > handling much larger datasets more efficiently. Also the degree of > parallelism and fault tolerance with these files systems come into > it. I will be interested in hearing more about any progress on this. > > Thanks > . > > Mich Talebzadeh, > > Technologist | Solutions Architect | Data Engineer | Generative AI > > London > United Kingdom > > > view my Linkedin profile > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > Disclaimer: The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner Von Braun)". > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297