What is the meaning to of 'STATE' in a worker/ an executor?

Hi, I have noticed in the Spark UI, workers and executors run on several states, ALIVE, LOADING, RUNNING, DEAD etc? What exactly are these states mean and what is the effect it has on working with those executor? ex: whether an executor can not be used in the loading state, etc cheers --

Re: what is the roadmap for Spark SQL dialect in the coming releases?

optimizer / execution engine instead of the catalyst optimizer that is shipped with Spark. On Thu, Jan 22, 2015 at 3:12 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, would like to know if there is an update on this? rgds On Mon, Jan 12, 2015 at 10:44 AM, Niranda Perera niranda.per

Re: what is the roadmap for Spark SQL dialect in the coming releases?

Hi, would like to know if there is an update on this? rgds On Mon, Jan 12, 2015 at 10:44 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, I found out that SparkSQL supports only a relatively small subset of SQL dialect currently. I would like to know the roadmap for the coming

what is the roadmap for Spark SQL dialect in the coming releases?

Hi, I found out that SparkSQL supports only a relatively small subset of SQL dialect currently. I would like to know the roadmap for the coming releases. And, are you focusing more on popularizing the 'Hive on Spark' SQL dialect or the Spark SQL dialect? Rgds -- Niranda

example insert statement in Spark SQL

Hi, Are insert statements supported in Spark? if so, can you please give me an example? Rgds -- Niranda

Re: Guava 11 dependency issue in Spark 1.2.0

of these problems are solved. As we've seen though, this one is tricky. What's your Spark version? and what are you executing? what mode -- standalone, YARN? What Hadoop version? On Tue, Jan 6, 2015 at 8:38 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, I have been running a simple Spark app

Guava 11 dependency issue in Spark 1.2.0

: com.google.common.hash.HashFunction.hashInt error occurs, which is understandable because hashInt is not available before Guava 12. So, I''m wondering why this occurs? Cheers -- Niranda Perera

Spark avro: Sample app fails to run in a spark standalone cluster

) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) what might be the reason behind this? I'm using intelliJ idea as my IDE I have attached the logs herewith Rgds -- Niranda Perera /usr

SparkSQL 1.2.0 sources API error

Hi all, I am evaluating the spark sources API released with Spark 1.2.0. But I'm getting a ava.lang.NoSuchMethodError: org.jboss.netty.channel.socket.nio.NioWorkerPool.init(Ljava/util/concurrent/Executor;I)V error running the program. Error log: 15/01/03 10:41:30 ERROR ActorSystemImpl: Uncaught

Re: Creating a SchemaRDD from an existing API

library for reading Avro data https://github.com/databricks/spark-avro. On Thu, Nov 27, 2014 at 10:31 PM, Niranda Perera nira...@wso2.com wrote: Hi, I am evaluating Spark for an analytic component where we do batch processing of data using SQL. So, I am particularly interested in Spark

Creating a SchemaRDD from an existing API

[1] https://github.com/wso2-dev/carbon-analytics/tree/master/components/xanalytics -- *Niranda Perera* Software Engineer, WSO2 Inc. Mobile: +94-71-554-8430 Twitter: @n1r44 https://twitter.com/N1R44