Re: Add Caller Context in Spark

2016-06-09 Thread Weiqing Yang
Yes, it is a string. Jira SPARK-15857 is created. Thanks, WQ On Thu, Jun 9, 2016 at 4:40 PM, Reynold Xin wrote: > Is this just to set some string? That makes sense. One thing you would > need to make sure is that Spark

Add Caller Context in Spark

2016-06-09 Thread Weiqing Yang
Hi, Hadoop has implemented a feature of log tracing – caller context (Jira: HDFS-9184 and YARN-4349 ). The motivation is to better diagnose and understand how specific applications impacting parts

Re: JIRA SPARK-2984

2016-06-09 Thread Holden Karau
I'd do some searching and see if there is a JIRA related to this problem on s3 and if you don't find one go ahead and make one. Even if it is an intrinsic problem with s3 (and I'm not super sure since I'm just reading this on mobile) - it would maybe be a good thing for us to document. On

Re: JIRA SPARK-2984

2016-06-09 Thread Holden Karau
I think your error could possibly be different - looking at the original JIRA the issue was happening on HDFS and you seem to be experiencing the issue on s3n, and while I don't have full view of the problem I could see this being s3 specific (read-after-write on s3 is trickier than

Strange exception while reading Parquet files

2016-06-09 Thread Rostyslav Sotnychenko
Hello! I have faced a very strange exception (stack-trace in the end of this email) while trying to read Parquet file using Hive Context from Spark 1.3.1, Hive 0.13. This issue appears only on YARN (standalone and local are working fine) and only when HiveContext is used (from SqlContext

Re: Spark 2.0.0 preview docs uploaded

2016-06-09 Thread Sean Owen
Available but mostly as JIRA output: https://spark.apache.org/news/spark-2.0.0-preview.html On Thu, Jun 9, 2016 at 7:33 AM, Pete Robbins wrote: > It would be nice to have a "what's new in 2.0.0" equivalent to > https://spark.apache.org/releases/spark-release-1-6-0.html

Re: Spark 2.0.0 preview docs uploaded

2016-06-09 Thread Pete Robbins
It would be nice to have a "what's new in 2.0.0" equivalent to https://spark.apache.org/releases/spark-release-1-6-0.html available or am I just missing it? On Wed, 8 Jun 2016 at 13:15 Sean Owen wrote: > OK, this is done: > > http://spark.apache.org/documentation.html >