I noticed when querying struct data in spark sql, we are requesting the
whole column from parquet files. Is this intended or is there some kind of
config to control this behaviour? Wouldn't it be better to request just the
struct field?
ssues.apache.org/jira/browse/SPARK-4502
>
> On Thu, Oct 29, 2015 at 6:00 PM, Sadhan Sood <sadhan.s...@gmail.com>
> wrote:
>
>> I noticed when querying struct data in spark sql, we are requesting the
>> whole column from parquet files. Is this intended or is the
Hi All,
Does anyone have fair scheduling working for them in a hive server? I have
one hive thriftserver running and multiple users trying to run queries at
the same time on that server using a beeline client. I see that a big query
is stopping all other queries from making any progress. Is this
che.org/docs/latest/sql-programming-guide.html#scheduling
>
> You likely want to put each user in their own pool.
>
> On Tue, Oct 20, 2015 at 11:55 AM, Sadhan Sood <sadhan.s...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Does anyone have fair scheduling
I am trying to run a query on a month of data. The volume of data is not
much, but we have a partition per hour and per day. The table schema is
heavily nested with total of 300 leaf fields. I am trying to run a simple
select count(*) query on the table and running into this exception:
SELECT
Hi Spark users,
We are running Spark on Yarn and often query table partitions as big as
100~200 GB from hdfs. Hdfs is co-located on the same cluster on which Spark
and Yarn run. I've noticed a much higher I/O read rates when I increase the
number of executors cores from 2 to 8( Most tasks run in
Interestingly, if there is nothing running on dev spark-shell, it recovers
successfully and regains the lost executors. Attaching the log for that.
Notice, the Registering block manager .. statements in the very end after
all executors were lost.
On Wed, Aug 26, 2015 at 11:27 AM, Sadhan Sood
Attaching log for when the dev job gets stuck (once all its executors are
lost due to preemption). This is a spark-shell job running in yarn-client
mode.
On Wed, Aug 26, 2015 at 10:45 AM, Sadhan Sood sadhan.s...@gmail.com wrote:
Hi All,
We've set up our spark cluster on aws running on yarn
Hi All,
We've set up our spark cluster on aws running on yarn (running on hadoop
2.3) with fair scheduling and preemption turned on. The cluster is shared
for prod and dev work where prod runs with a higher fair share and can
preempt dev jobs if there are not enough resources available for it.
It
Hi Xu-dong,
Thats probably because your table's partition path don't look like
hdfs://somepath/key=value/*.parquet. Spark is trying to extract the
partition key's value from the path while caching and hence the exception
is being thrown since it can't find one.
On Mon, Jan 26, 2015 at 10:45 AM,
and struct are not supported.
On 12/20/14 6:17 AM, Sadhan Sood wrote:
Hey Michael,
Thank you for clarifying that. Is tachyon the right way to get compressed
data in memory or should we explore the option of adding compression to
cached data. This is because our uncompressed data set is too big
:
There is only column level encoding (run length encoding, delta encoding,
dictionary encoding) and no generic compression.
On Thu, Dec 18, 2014 at 12:07 PM, Sadhan Sood sadhan.s...@gmail.com
wrote:
Hi All,
Wondering if when caching a table backed by lzo compressed parquet data,
if spark
at 2:17 PM, Sadhan Sood sadhan.s...@gmail.com
wrote:
Hey Michael,
Thank you for clarifying that. Is tachyon the right way to get compressed
data in memory or should we explore the option of adding compression to
cached data. This is because our uncompressed data set is too big to fit
Hi All,
Wondering if when caching a table backed by lzo compressed parquet data, if
spark also compresses it (using lzo/gzip/snappy) along with column level
encoding or just does the column level encoding when
*spark.sql.inMemoryColumnarStorage.compressed
*is set to true. This is because when I
Also attaching the parquet file if anyone wants to take a further look.
On Thu, Nov 20, 2014 at 8:54 AM, Sadhan Sood sadhan.s...@gmail.com wrote:
So, I am seeing this issue with spark sql throwing an exception when
trying to read selective columns from a thrift parquet file and also when
We are loading parquet data as temp tables but wondering if there is a way
to add a partition to the data without going through hive (we still want to
use spark's parquet serde as compared to hive). The data looks like -
/date1/file1, /date1/file2 ... , /date2/file1,
/date2/file2,/daten/filem
I am running on master, pulled yesterday I believe but saw the same issue
with 1.2.0
On Thu, Nov 20, 2014 at 1:37 PM, Michael Armbrust mich...@databricks.com
wrote:
Which version are you running on again?
On Thu, Nov 20, 2014 at 8:17 AM, Sadhan Sood sadhan.s...@gmail.com
wrote:
Also
Ah awesome, thanks!!
On Thu, Nov 20, 2014 at 3:01 PM, Michael Armbrust mich...@databricks.com
wrote:
In 1.2 by default we use Spark parquet support instead of Hive when the
SerDe contains the word Parquet. This should work with hive partitioning.
On Thu, Nov 20, 2014 at 10:33 AM, Sadhan
Thanks Michael, opened this https://issues.apache.org/jira/browse/SPARK-4520
On Thu, Nov 20, 2014 at 2:59 PM, Michael Armbrust mich...@databricks.com
wrote:
Can you open a JIRA?
On Thu, Nov 20, 2014 at 10:39 AM, Sadhan Sood sadhan.s...@gmail.com
wrote:
I am running on master, pulled
ah makes sense - Thanks Michael!
On Mon, Nov 17, 2014 at 6:08 PM, Michael Armbrust mich...@databricks.com
wrote:
You are perhaps hitting an issue that was fixed by #3248
https://github.com/apache/spark/pull/3248?
On Mon, Nov 17, 2014 at 9:58 AM, Sadhan Sood sadhan.s...@gmail.com
wrote
While testing sparkSQL, we were running this group by with expression query
and got an exception. The same query worked fine on hive.
SELECT from_unixtime(floor(xyz.whenrequestreceived/1000.0 - 25200),
'/MM/dd') as pst_date,
count(*) as num_xyzs
FROM
all_matched_abc
GROUP BY
to narrow down the problem by trying to scan exactly the same Parquet file
with some other systems (e.g. Hive or Impala)? If other systems work, then
there must be something wrong with Spark SQL.
Cheng
On Sun, Nov 16, 2014 at 1:19 PM, Sadhan Sood sadhan.s...@gmail.com
wrote:
Hi Cheng,
Thanks
While testing SparkSQL on a bunch of parquet files (basically used to be a
partition for one of our hive tables), I encountered this error:
import org.apache.spark.sql.SchemaRDD
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
for this in master and branch-1.2 is 10,000
rows per batch.
On 11/14/14 1:27 AM, Sadhan Sood wrote:
Thanks Chneg, Just one more question - does that mean that we still
need enough memory in the cluster to uncompress the data before it can be
compressed again or does that just read the raw data
spark.sql.inMemoryColumnarStorage.compressed to true. This property is
already set to true by default in master branch and branch-1.2.
On 11/13/14 7:16 AM, Sadhan Sood wrote:
We noticed while caching data from our hive tables which contain data
in compressed sequence file format that it gets uncompressed in memory when
We are running spark on yarn with combined memory 1TB and when trying to
cache a table partition(which is 100G), seeing a lot of failed collect
stages in the UI and this never succeeds. Because of the failed collect, it
seems like the mapPartitions keep getting resubmitted. We have more than
(Logging.scala:logError(75)) - Asked to remove non-existent executor 372
2014-11-12 19:11:21,655 INFO scheduler.DAGScheduler
(Logging.scala:logInfo(59)) - Executor lost: 372 (epoch 3)
On Wed, Nov 12, 2014 at 12:31 PM, Sadhan Sood sadhan.s...@gmail.com wrote:
We are running spark on yarn with combined
Just making sure but are you looking for the tar in assembly/target dir ?
On Wed, Nov 12, 2014 at 3:14 PM, Ashwin Shankar ashwinshanka...@gmail.com
wrote:
Hi,
I just cloned spark from the github and I'm trying to build to generate a
tar ball.
I'm doing : mvn -Pyarn -Phadoop-2.4
I think you can provide -Pbigtop-dist to build the tar.
On Wed, Nov 12, 2014 at 3:21 PM, Sean Owen so...@cloudera.com wrote:
mvn package doesn't make tarballs. It creates artifacts that will
generally appear in target/ and subdirectories, and likewise within
modules. Look at
an
output location for shuffle 0
The data is lzo compressed sequence file with compressed size ~ 26G. Is
there a way to understand why shuffle keeps failing for one partition. I
believe we have enough memory to store the uncompressed data in memory.
On Wed, Nov 12, 2014 at 2:50 PM, Sadhan Sood sadhan.s
We noticed while caching data from our hive tables which contain data in
compressed sequence file format that it gets uncompressed in memory when
getting cached. Is there a way to turn this off and cache the compressed
data as is ?
didn't start
successfully because the HiveServer2 occupied the port, and your Beeline
session was probably linked against HiveServer2.
Cheng
On 11/11/14 8:29 AM, Sadhan Sood wrote:
I was testing out the spark thrift jdbc server by running a simple query
in the beeline client. The spark
I was testing out the spark thrift jdbc server by running a simple query in
the beeline client. The spark itself is running on a yarn cluster.
However, when I run a query in beeline - I see no running jobs in the
spark UI(completely empty) and the yarn UI seem to indicate that the
submitted query
Hi,
Trying to run a query on spark-sql but it keeps failing with this error on
the cli ( we are running spark-sql on a yarn cluster):
org.apache.spark.SparkException: Job cancelled because SparkContext was
shut down
at
Is there a way to cache certain (or most latest) partitions of certain
tables ?
On Fri, Oct 24, 2014 at 2:35 PM, Michael Armbrust mich...@databricks.com
wrote:
It does have support for caching using either CACHE TABLE tablename or
CACHE TABLE tablename AS SELECT
On Fri, Oct 24, 2014 at
at 12:06 PM, Sadhan Sood sadhan.s...@gmail.com
wrote:
Is there a way to cache certain (or most latest) partitions of certain
tables ?
On Fri, Oct 24, 2014 at 2:35 PM, Michael Armbrust mich...@databricks.com
wrote:
It does have support for caching using either CACHE TABLE tablename or
CACHE
We want to run multiple instances of spark sql cli on our yarn cluster.
Each instance of the cli is to be used by a different user. This looks
non-optimal if each user brings up a different cli given how spark works on
yarn by running executor processes (and hence consuming resources) on
worker
How can we read all parquet files in a directory in spark-sql. We are
following this example which shows a way to read one file:
// Read in the parquet file created above. Parquet files are
self-describing so the schema is preserved.// The result of loading a
Parquet file is also a SchemaRDD.val
on Parquet files to
request
this feature.
Nick
On Mon, Oct 13, 2014 at 12:21 PM, Sadhan Sood sadhan.s...@gmail.com
wrote:
How can we read all parquet files in a directory in spark-sql. We are
following this example which shows a way to read one file:
// Read in the parquet file created
-- Forwarded message --
From: Sadhan Sood sadhan.s...@gmail.com
Date: Sat, Oct 11, 2014 at 10:26 AM
Subject: Re: how to find the sources for spark-project
To: Stephen Boesch java...@gmail.com
Thanks, I still didn't find it - is it under some particular branch ? More
specifically
40 matches
Mail list logo
