RE: Spark on Kubernetes : unable to write files to HDFS

Everything is working fine now  Thanks again Loïc De : German Schiavon Envoyé : mercredi 16 décembre 2020 19:23 À : Loic DESCOTTE Cc : user@spark.apache.org Objet : Re: Spark on Kubernetes : unable to write files to HDFS We all been there! no reason to be

Find difference between two dataframes in spark structured streaming

I am creating a spark structured streaming job, where I need to find the difference between two dataframes. Dataframe 1 : [1, item1, value1] [2, item2, value2] [3, item3, value3] [4, item4, value4] [5, item5, value5] Dataframe 2: [4, item4, value4] [5, item5, value5] New Dataframe with

Re: Spark on Kubernetes : unable to write files to HDFS

We all been there! no reason to be ashamed :) On Wed, 16 Dec 2020 at 18:14, Loic DESCOTTE < loic.desco...@kaizen-solutions.net> wrote: > Oh thank you you're right!! I feel shameful  > > -- > *De :* German Schiavon > *Envoyé :* mercredi 16 décembre 2020 18:01 > *À :*

RE: Spark on Kubernetes : unable to write files to HDFS

Oh thank you you're right!! I feel shameful ?? De : German Schiavon Envoyé : mercredi 16 décembre 2020 18:01 À : Loic DESCOTTE Cc : user@spark.apache.org Objet : Re: Spark on Kubernetes : unable to write files to HDFS Hi, seems that you have a typo no?

Re: Spark on Kubernetes : unable to write files to HDFS

Hi, seems that you have a typo no? Exception in thread "main" java.io.IOException: No FileSystem for scheme: hfds data.write.mode("overwrite").format("text").save("hfds:// hdfs-namenode/user/loic/result.txt") On Wed, 16 Dec 2020 at 17:02, Loic DESCOTTE < loic.desco...@kaizen-solutions.net>

RE: Spark on Kubernetes : unable to write files to HDFS

So I've tried several other things, including building a fat jar with hdfs dependency inside my app jar, and added this to the Spark configuration in the code : val spark = SparkSession .builder() .appName("Hello Spark 7") .config("fs.hdfs.impl",

[no subject]

Unsubscribe 发自网易邮箱大师

11

发自网易邮箱大师

Re: unsubscribe

https://gist.github.com/jeff303/ba1906bb7bcb2f2501528a8bb1521b8e On Wed, Dec 16, 2020, 6:45 AM 张洪斌 wrote: > how to unsubscribe this ? > > 发自网易邮箱大师 > 在2020年12月16日 20:43，张洪斌 > 写道： > > > unsubscribe > 学生张洪斌 > 邮箱：hongbinzh...@163.com > >

回复：unsubscribe

how to unsubscribe this ? 发自网易邮箱大师 在2020年12月16日 20:43，张洪斌 写道： unsubscribe | | 学生张洪斌 | | 邮箱：hongbinzh...@163.com | 签名由 网易邮箱大师 定制

unsubscribe

unsubscribe | | 学生张洪斌 | | 邮箱：hongbinzh...@163.com | 签名由 网易邮箱大师 定制

Spark on Kubernetes : unable to write files to HDFS

Hello, I am using Spark On Kubernetes and I have the following error when I try to write data on HDFS : "no filesystem for scheme hdfs" More details : I am submitting my application with Spark submit like this : spark-submit --master k8s://https://myK8SMaster:6443 \ --deploy-mode cluster \

copy_to() of dplyr lib returning previous date

Hi I am trying to convert date from string to date format using R with as.date() as shown below: dfTest <- data.frame(StringDate=c("2020-12-01","2020-12-02"), DateDate=as.Date(c("2020-12-01","2020-12-02"))) dfTest StringDate DateDate 1 2020-12-01 2020-12-01 2 2020-12-02 2020-12-02 The above

copy_to() of sparklyr lib returning previous date

Hi I am trying to convert date from string to date format using R with as.date() as shown below: dfTest <- data.frame(StringDate=c("2020-12-01","2020-12-02"), DateDate=as.Date(c("2020-12-01","2020-12-02"))) dfTest StringDate DateDate 1 2020-12-01 2020-12-01 2 2020-12-02 2020-12-02 The above

Running spark code using wheel file

Hi All, I have created a wheel file and I am using the following command to run the spark job: spark-submit --py-files application.whl main_flow.py My application is unable to reference the modules. Do I need to do the pip install of the wheel first? Kind Regards, Sachit Murarka