Unsubscribe

Unsubscribe

How can I write data to ftp

DF.write.format("csv") .option("timestampFormat", "/MM/dd HH:mm:ss ZZ") .mode(SaveMode.Overwrite) .save("ftp://ftp:ftp@127.0.0.1:21/sparkftp/write/;) I get a error 21/08/10 08:32:04 WARN FileOutputCommitter: Could not delete

Performance of PySpark jobs on the Kubernetes cluster

Hi, I have a basic question to ask. I am running a Google k8s cluster (AKA GKE) with three nodes each having configuration below e2-standard-2 (2 vCPUs, 8 GB memory) spark-submit is launched from another node (actually a data proc single node that I have just upgraded to e2-custom (4 vCPUs, 8

unsubscribe

unsubscribe Regards, Vijay Gharge

Is the pandas version in doc of using pyarrow in spark wrong

The doc says that the minimum supported pandas version is 0.23.2 which is only supported in python2. IIRC, python2 is not supported in pyspark a long time ago. Can any one confirm whether the doc is wrong and what is the right version of pandas and pyarrow ?

Re: How can I read ftp

We have solved it using an orchestrator, which copied data from FTP to HDFS. But of course, you can just use a Java FTP client to just read files, put them somewhere, and then read with Spark. пн, 9 авг. 2021 г. в 06:39, Sean Owen : > FTP is definitely not supported. Read the files to

How can I write to ftp

DF.write.format(sourceType) .option("timestampFormat", "/MM/dd HH:mm:ss ZZ") .mode(SaveMode.Overwrite) // .partitionBy("created") .save("ftp://ftp:ftpu@127.0.0.1:21/sparkftp/write/;) I get a error java.io.IOException: Cannot rename source: