Everything is working fine now
Thanks again
Loïc
De : German Schiavon
Envoyé : mercredi 16 décembre 2020 19:23
À : Loic DESCOTTE
Cc : user@spark.apache.org
Objet : Re: Spark on Kubernetes : unable to write files to HDFS
We all been there! no reason to be
We all been there! no reason to be ashamed :)
On Wed, 16 Dec 2020 at 18:14, Loic DESCOTTE <
loic.desco...@kaizen-solutions.net> wrote:
> Oh thank you you're right!! I feel shameful
>
> --
> *De :* German Schiavon
> *Envoyé :* mercredi 16 décembre 2020 18:01
> *À :*
Oh thank you you're right!! I feel shameful ??
De : German Schiavon
Envoyé : mercredi 16 décembre 2020 18:01
À : Loic DESCOTTE
Cc : user@spark.apache.org
Objet : Re: Spark on Kubernetes : unable to write files to HDFS
Hi,
seems that you have a typo no?
Hi,
seems that you have a typo no?
Exception in thread "main" java.io.IOException: No FileSystem for scheme:
hfds
data.write.mode("overwrite").format("text").save("hfds://
hdfs-namenode/user/loic/result.txt")
On Wed, 16 Dec 2020 at 17:02, Loic DESCOTTE <
loic.desco...@kaizen-solutions.net>
So I've tried several other things, including building a fat jar with hdfs
dependency inside my app jar, and added this to the Spark configuration in the
code :
val spark = SparkSession
.builder()
.appName("Hello Spark 7")
.config("fs.hdfs.impl",
Hello,
I am using Spark On Kubernetes and I have the following error when I try to
write data on HDFS : "no filesystem for scheme hdfs"
More details :
I am submitting my application with Spark submit like this :
spark-submit --master k8s://https://myK8SMaster:6443 \
--deploy-mode cluster \
