Re: Ever increasing physical memory for a Spark Application in YARN

Hi Daniel, I could indeed discover the problem in my case and it turned out to be a bug at parquet side and I had raised and contributed to the following issue :- https://issues.apache.org/jira/browse/PARQUET-353 Hope this helps! Thanks -Nitin On Mon, May 2, 2016 at 9:15 PM, Daniel Darabos <

Unable To Find Proto Buffer Class Error With RDD

Hi, I ran into an issue when using proto buffer in spark RDD. I googled this and found it seems to be a known compatible issue. Have anyone ran into the same issue before and found any solutions? The detailed description could be found in this link.

Re: SQLContext and "stable identifier required"

Probably not. Want to submit a pull request? On Tuesday, May 3, 2016, Koert Kuipers wrote: > yes it works fine if i switch to using the implicits on the SparkSession > (which is a val) > > but do we want to break the old way of doing the import? > > On Tue, May 3, 2016 at

Re: SQLContext and "stable identifier required"

sure i can do that On Tue, May 3, 2016 at 1:21 PM, Reynold Xin wrote: > Probably not. Want to submit a pull request? > > > On Tuesday, May 3, 2016, Koert Kuipers wrote: > >> yes it works fine if i switch to using the implicits on the SparkSession >>

Re: SQLContext and "stable identifier required"

yes it works fine if i switch to using the implicits on the SparkSession (which is a val) but do we want to break the old way of doing the import? On Tue, May 3, 2016 at 12:56 PM, Ted Yu wrote: > Have you tried the following ? > > scala> import spark.implicits._ > import

Re: SQLContext and "stable identifier required"

Have you tried the following ? scala> import spark.implicits._ import spark.implicits._ scala> spark res0: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@323d1fa2 Cheers On Tue, May 3, 2016 at 9:16 AM, Koert Kuipers wrote: > with the introduction of

SQLContext and "stable identifier required"

with the introduction of SparkSession SQLContext changed from being a lazy val to a def. however this is troublesome if you want to do: import someDataset.sqlContext.implicits._ because it is no longer a stable identifier, i think? i get: stable identifier required, but