Hi, all
The builtin hive version for spark 2.x is hive-1.2.1.spark2, I'd like know
whether it works for hive meta store version 2.1 or not.
If not, I'd like to build a spark package with -Dhive.version=2.x.spark2 but
find no such a maven artifact there, is there any process to deploy one?
Or I
I would like to truncate date to his day or hour. currently it is only
possible to truncate MONTH or YEAR.
1.How can achieve that?
2.Is there any pull request about this issue?
3.If there is not any open pull request about this issue, what are the
implications that I should be aware of when coding
This is my experience too when running under yarn at least
On Thu, 9 Nov 2017, 07:11 Nicolas Paris, wrote:
> Le 06 nov. 2017 à 19:56, Nicolas Paris écrivait :
> > Can anyone clarify the driver memory aspects of pySpark?
> > According to [1], spark.driver.memory limits JVM +
Confidentiality: This communication and any attachments are intended for the
above-named persons only and may be confidential and/or legally privileged. Any
opinions expressed in this communication are not necessarily those of NICE
Actimize. If this communication has come to you in error you
I'd recommend against using the built-in jars for a different version
of Hive. You don't need to build your own Spark; just set
spark.sql.hive.metastore.jars / spark.sql.hive.metastore.version (see
documentation).
On Thu, Nov 9, 2017 at 2:10 AM, yaooqinn wrote:
> Hi, all
>
There are functions for day (called dayOfMonth and dayOfYear) and hour
(called hour). You can view them here:
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions
Example:
import org.apache.spark.sql.functions._
val df = df.select(hour($"myDateColumn"),
Can we pass the Calcite streaming sql queries to spark sql?
https://calcite.apache.org/docs/stream.html#references
I dont think so. Calcite's SQL is an extension of standard SQL (keywords
like STREAM, etc.) which we dont support; we just support regular SQL, so
queries like "SELECT STREAM " will not work.
On Thu, Nov 9, 2017 at 11:50 AM, kant kodali wrote:
> Can we pass the Calcite
Hi,
I am not successful when using using spark 2.1 with Kafka 0.9, can anyone
please share the code snippet to use it.
val sparkSession: SparkSession = runMode match {
case "local" => SparkSession.builder.config(sparkConfig).getOrCreate
case "yarn" =>
Thank you for your reply.
But,sometimes successed, when i rerun the job.
And the job process the same data using the same code.
From: Margusja
Date: 2017-11-09 14:25
To: bing...@iflytek.com
CC: user
Subject: Re: spark job paused(active stages finished)
You have to deal with
Hi,
I have compressed data of size 500GB .I am repartitioning this data since
the underlying data is very skewed and is causing a lot of issues for the
downstream jobs.
During repartioning the *shuffles writes* are not getting compressed due to
this I am running into disk space issues.Below is the
Hi everyone.
I try to build a middleware that executes injection-safe dynamic SQL
queries using Spark, with Cassandra as storage engine. Results of
executed queries need to be stored into a newly created Cassandra table.
Spark SQL does not provide any feature for safe parameter binding, so I
I forgot to precise that I'm using Spark 2.1.0.
On 2017/11/10 16:08, Arnaud Wolf wrote:
Hi everyone.
I try to build a middleware that executes injection-safe dynamic SQL
queries using Spark, with Cassandra as storage engine. Results of
executed queries need to be stored into a newly created
13 matches
Mail list logo