Re: Enable hql on the JDBC thrift server

I forgot to precise that I'm using Spark 2.1.0. On 2017/11/10 16:08, Arnaud Wolf wrote: Hi everyone. I try to build a middleware that executes injection-safe dynamic SQL queries using Spark, with Cassandra as storage engine. Results of executed queries need to be stored into a newly created

Enable hql on the JDBC thrift server

Hi everyone. I try to build a middleware that executes injection-safe dynamic SQL queries using Spark, with Cassandra as storage engine. Results of executed queries need to be stored into a newly created Cassandra table. Spark SQL does not provide any feature for safe parameter binding, so I

Re: Re: spark job paused(active stages finished)

Thank you for your reply. But，sometimes successed， when i rerun the job. And the job process the same data using the same code. From: Margusja Date: 2017-11-09 14:25 To: bing...@iflytek.com CC: user Subject: Re: spark job paused(active stages finished) You have to deal with

Compression during shuffle writes

Hi, I have compressed data of size 500GB .I am repartitioning this data since the underlying data is very skewed and is causing a lot of issues for the downstream jobs. During repartioning the *shuffles writes* are not getting compressed due to this I am running into disk space issues.Below is the

Spark Streaming in Spark 2.1 with Kafka 0.9

Hi, I am not successful when using using spark 2.1 with Kafka 0.9, can anyone please share the code snippet to use it. val sparkSession: SparkSession = runMode match { case "local" => SparkSession.builder.config(sparkConfig).getOrCreate case "yarn" =>

Re: Can we pass the Calcite streaming sql queries to spark sql?

I dont think so. Calcite's SQL is an extension of standard SQL (keywords like STREAM, etc.) which we dont support; we just support regular SQL, so queries like "SELECT STREAM " will not work. On Thu, Nov 9, 2017 at 11:50 AM, kant kodali wrote: > Can we pass the Calcite

Can we pass the Calcite streaming sql queries to spark sql?

Can we pass the Calcite streaming sql queries to spark sql? https://calcite.apache.org/docs/stream.html#references

Re: Spark SQL - Truncate Day / Hour

There are functions for day (called dayOfMonth and dayOfYear) and hour (called hour). You can view them here: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions Example: import org.apache.spark.sql.functions._ val df = df.select(hour($"myDateColumn"),

Re: Does the builtin hive jars talk of spark to HiveMetaStore(2.1) without any issues?

I'd recommend against using the built-in jars for a different version of Hive. You don't need to build your own Spark; just set spark.sql.hive.metastore.jars / spark.sql.hive.metastore.version (see documentation). On Thu, Nov 9, 2017 at 2:10 AM, yaooqinn wrote: > Hi, all >

Spark SQL - Truncate Day / Hour

I would like to truncate date to his day or hour. currently it is only possible to truncate MONTH or YEAR. 1.How can achieve that? 2.Is there any pull request about this issue? 3.If there is not any open pull request about this issue, what are the implications that I should be aware of when coding

Does the builtin hive jars talk of spark to HiveMetaStore(2.1) without any issues?

Hi, all The builtin hive version for spark 2.x is hive-1.2.1.spark2, I'd like know whether it works for hive meta store version 2.1 or not. If not, I'd like to build a spark package with -Dhive.version=2.x.spark2 but find no such a maven artifact there, is there any process to deploy one? Or I

Testing spark e-mail list

Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you

Re: pySpark driver memory limit

This is my experience too when running under yarn at least On Thu, 9 Nov 2017, 07:11 Nicolas Paris, wrote: > Le 06 nov. 2017 à 19:56, Nicolas Paris écrivait : > > Can anyone clarify the driver memory aspects of pySpark? > > According to [1], spark.driver.memory limits JVM +