Spark 3.0 plugins

2021-12-19 Thread Anil Dasari
Hello everyone, I was going through Apache Spark Performance Monitoring in Spark 3.0 talk and wanted to collect IO metrics for my spark application. Couldn’t find Spark 3.0 built-in plugins for IO metrics like https://github.com/cerndb/SparkPlugins

Re: [R] SparkR on conda-forge

2021-12-19 Thread Hyukjin Kwon
Awesome! On Mon, 20 Dec 2021 at 09:43, yonghua wrote: > Nice release. thanks for sharing. > > On 2021/12/20 3:55, Maciej wrote: > > FYI ‒ thanks to good folks from conda-forge we have now these: > > - > To unsubscribe e-mail:

Re: [R] SparkR on conda-forge

2021-12-19 Thread yonghua
Nice release. thanks for sharing. On 2021/12/20 3:55, Maciej wrote: FYI ‒ thanks to good folks from conda-forge we have now these: - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

[R] SparkR on conda-forge

2021-12-19 Thread Maciej
Hi everyone, FYI ‒ thanks to good folks from conda-forge we have now these: * https://github.com/conda-forge/r-sparkr-feedstock * https://anaconda.org/conda-forge/r-sparkr -- Best regards, Maciej Szymkiewicz Web: https://zero323.net PGP: A30CEF0C31A501EC OpenPGP_signature Description:

Re: question about data skew and memory issues

2021-12-19 Thread Gourav Sengupta
Hi, also if you are using SPARK 3.2.x please try to see the documentation on handling skew using SPARK settings. Regards, Gourav Sengupta On Tue, Dec 14, 2021 at 6:01 PM David Diebold wrote: > Hello all, > > I was wondering if it possible to encounter out of memory exceptions on > spark

Re: spark 3.2.0 the different dataframe createOrReplaceTempView the same name TempView

2021-12-19 Thread Gourav Sengupta
Hi, I am pretty sure that Sean already answered the question. Also i do not think that creating iterative table definitions or data frame definitions are best practice. Regards, Gourav On Mon, Dec 13, 2021 at 4:00 PM Sean Owen wrote: > ... but the error is not "because that already exists".

Re: Unable to use WriteStream to write to delta file.

2021-12-19 Thread Alex Ott
Make sure that you're using compatible version of Delta Lake library. For Spark 3.2 it's 1.1.0: https://github.com/delta-io/delta/releases/tag/v1.1.0 For other Spark versions, check this table: https://docs.delta.io/latest/releases.html On Fri, Dec 17, 2021 at 2:36 PM Stelios Philippou wrote: >