Hi, Where do you see this? In spark UI.
So data is skewed most probably as one node gets all the data and others nothing as I understand? HTH Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 18 Aug 2023 at 17:17, marc nicole <mk1853...@gmail.com> wrote: > Hi, > > Spark 3.2, Hadoop 3.2, using YARN cluster mode, if one wants to read a > dataset that is found in one node of the cluster and not in the others, how > to tell Spark that? > > I expect through DataframeReader and using path like > *IP:port/pathOnLocalNode* > > PS: loading the dataset in HDFS is not an option. > > Thanks >
