WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped

2020-06-03 Thread YuqingWan
I installed Spark and run it, then I get an error "WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped" Something like:

Re: [PySpark 2.3+] Reading parquet entire path vs a set of file paths

2020-06-03 Thread Rishi Shah
Hi All, Just following up on below to see if anyone has any suggestions. Appreciate your help in advance. Thanks, Rishi On Mon, Jun 1, 2020 at 9:33 AM Rishi Shah wrote: > Hi All, > > I use the following to read a set of parquet file paths when files are > scattered across many many

Re: NoClassDefFoundError: scala/Product$class

2020-06-03 Thread Pol Santamaria
Hi Charles, I believe Spark 3.0 removed the support for Scala 2.11, and that error is a version compatibility issue. You should try Spark 2.4.5 with your current setup (works with Scala 2.11 by default). Pol Santamaria On Wed, Jun 3, 2020 at 7:44 AM charles_cai <1620075...@qq.com> wrote: > Hi,