Re: Reparitioning Hive tables - Container killed by YARN for exceeding memory limits

either increase overall executor memory if you have scope. or try to give more % to overhead memory from default of .7. Read this for more details. On Wed, Aug 2, 2017 at 11:03 PM Chetan Khatri

Re: Spark 2.0.2 : Hang at "org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:623)"

olExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) So clearly there is a race condition. May be only option is to avoid it... but how ?? On Fri, Mar 24, 2017 at 5:40 PM Ra

Spark 2.0.2 : Hang at "org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:623)"

Hi All, My Spark job hangs here... Looking into the thread dump I noticed that it hangs here (stack trace given below) on the count action on dataframe (given below). Data is very small. Its actually not more than even 10 rows. I noticed some JIRAs about this issue but all are resolved-closed in

Re: Spark 2.0.2 - hiveContext.emptyDataFrame.except(hiveContext.emptyDataFrame).count()

Thanks a lot young for explanation. But its sounds like an API behaviour change. For now I do the counts != o on both dataframes before these operations. Not good from performance point of view hence have created a JIRA (SPARK-20008) to track it. Thanks, Ravindra. On Fri, Mar 17, 2017 at 8:51 PM

Spark 2.0.2 - hiveContext.emptyDataFrame.except(hiveContext.emptyDataFrame).count()

nother way to check full equality of 2 dataframes. Thanks, Ravindra.

Re: Spark Task is not created

f you can paste the piece of code that you are executing. > > On Sun, Jun 26, 2016 at 11:21 AM, Ravindra <ravindra.baj...@gmail.com> > wrote: > >> Hi All, >> >> May be I need to just set some property or its a known issue. My spark >> application hangs i

Spark Task is not created

Hi All, May be I need to just set some property or its a known issue. My spark application hangs in test environment whenever I see following message - 16/06/26 11:13:34 INFO DAGScheduler: *Submitting 2 missing tasks from ShuffleMapStage* 145 (MapPartitionsRDD[590] at rdd at

Re: Forcing data from disk to memory

yup, cache is a transformation and hence lazy. you need to run action to get the data into it. http://apache-spark-user-list.1001560.n3.nabble.com/How-to-enforce-RDD-to-be-cached-td20230.html On Fri, Mar 25, 2016 at 2:32 PM Jörn Franke wrote: > I am not 100% sure of the

Spark 1.6.1 : SPARK-12089 : java.lang.NegativeArraySizeException

java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Thanks. -- Regards Ravindra

Re: About Huawei-Spark/Spark-SQL-on-HBase

Hi censj, Please try the new repo at https://github.com/HuaweiBigData/astro , not maintaining the old repo. Please let me know if you still get the error. You can contact on my personal mail also at ravi.pes...@gmail.com Thanks, Ravindra. On Sat 19 Dec, 2015 2:45 pm censj <ce...@lotuseed.

Re: Best practices to handle corrupted records

+1 Erwan.. May be a trivial solution like this - class Result (msg: String, record: Record) class Success (msgSuccess: String, val msg: String, val record: Record) extends Result(msg, record) class Failure (msgFailure: String, val msg: String, val record: Record) extends Result (msg, record)

Re: Research ideas using spark

Look at this : http://www.forbes.com/sites/lisabrownlee/2015/07/10/the-11-trillion-internet-of-things-big-data-and-pattern-of-life-pol-analytics/ On Wed, Jul 15, 2015 at 10:19 PM shahid ashraf sha...@trialx.com wrote: Sorry Guys! I mistakenly added my question to this thread( Research ideas

Re: Spark can not access jar from HDFS !!

:40 AM Michael Armbrust mich...@databricks.com wrote: That code path is entirely delegated to hive. Does hive support this? You might try instead using sparkContext.addJar. On Sat, May 9, 2015 at 12:32 PM, Ravindra ravindra.baj...@gmail.com wrote: Hi All, I am trying to create custom udfs

Re: Spark can not access jar from HDFS !!

After upgrading to spark 1.3, these statements on hivecontext are working fine. Thanks On Mon, May 11, 2015, 12:15 Ravindra ravindra.baj...@gmail.com wrote: Hi All, Thanks for suggestions. What I tried is - hiveContext.sql (add jar ) and that helps to complete the create temporary

Spark can not access jar from HDFS !!

Hi All, I am trying to create custom udfs with hiveContext as given below - scala hiveContext.sql (CREATE TEMPORARY FUNCTION sample_to_upper AS 'com.abc.api.udf.MyUpper' USING JAR 'hdfs:///users/ravindra/customUDF2.jar') I have put the udf jar in the hdfs at the path given above. The same

Registring UDF from a different package fails

Hi All, I have all my UDFs defined in the classes residing in a different package than where I am instantiating my HiveContext. I have a register function in my UDF class. I pass HiveContext to this function. and in this function I call hiveContext.registerFunction(myudf, myudf _) All goes well

Re: Writing wide parquet file in Spark SQL

Even I am keen to learn an answer for this but as an alternate you can use hive to create a table stored as parquet and then use it in spark. On Wed, Mar 11, 2015 at 1:44 AM kpeng1 kpe...@gmail.com wrote: Hi All, I am currently trying to write a very wide file into parquet using spark sql.

Re: ANSI Standard Supported by the Spark-SQL

Armbrust mich...@databricks.com wrote: Spark SQL supports a subset of HiveQL: http://spark.apache.org/docs/latest/sql-programming-guide.html#compatibility-with-apache-hive On Mon, Mar 9, 2015 at 11:32 PM, Ravindra ravindra.baj...@gmail.com wrote: From the archives in this user list, It seems

Re: ANSI Standard Supported by the Spark-SQL

92 compliant? Please clarify. Regards, Ravi On Tue, Mar 10, 2015 at 11:46 AM Ravindra ravindra.baj...@gmail.com wrote: Hi All, I am new to spark and trying to understand what SQL Standard is supported by the Spark. I googled around a lot but didn't get clear answer. Some where I saw