Re: Query parsing error for the join query between different database

[JaeSung Jun]

Thanks Herman,
I didn't recognise the "user" is reserved word. it works now.

On 19 May 2016 at 08:02, Herman van Hövell tot Westerflier <
hvanhov...@questtec.nl> wrote:

> 'User' is a SQL2003 keyword. This is normally not a problem, except when
> you use it as a table alias (which you are doing). Change the alias or
> place it between backticks and you should be fine.
>
>
> 2016-05-18 23:51 GMT+02:00 JaeSung Jun :
>
>> It's spark 1.6.1 and hive 1.2.1 (spark-sql saying "SET
>> spark.sql.hive.version=1.2.1").
>>
>> Thanks
>>
>> On 18 May 2016 at 23:31, Ted Yu  wrote:
>>
>>> Which release of Spark / Hive are you using ?
>>>
>>> Cheers
>>>
>>> On May 18, 2016, at 6:12 AM, JaeSung Jun  wrote:
>>>
>>> Hi,
>>>
>>> I'm working on custom data source provider, and i'm using fully
>>> qualified table name in FROM clause like following :
>>>
>>> SELECT user. uid, dept.name
>>> FROM userdb.user user, deptdb.dept
>>> WHERE user.dept_id = dept.id
>>>
>>> and i've got the following error :
>>>
>>> MismatchedTokenException(279!=26)
>>> at
>>> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
>>> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
>>> at
>>> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
>>> at
>>> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
>>> at
>>> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
>>> at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
>>> at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
>>> at
>>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
>>> at
>>> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
>>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
>>> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
>>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
>>> at
>>> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>>> at
>>> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
>>> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
>>> at
>>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>>> at
>>> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
>>> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>>> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
>>> at
>>> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
>>> at
>>> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
>>> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)
>>>
>>> Any idea?
>>>
>>> Thanks
>>> Jason
>>>
>>>
>>
>

Re: Query parsing error for the join query between different database

[JaeSung Jun]

It's spark 1.6.1 and hive 1.2.1 (spark-sql saying "SET
spark.sql.hive.version=1.2.1").

Thanks

On 18 May 2016 at 23:31, Ted Yu  wrote:

> Which release of Spark / Hive are you using ?
>
> Cheers
>
> On May 18, 2016, at 6:12 AM, JaeSung Jun  wrote:
>
> Hi,
>
> I'm working on custom data source provider, and i'm using fully qualified
> table name in FROM clause like following :
>
> SELECT user. uid, dept.name
> FROM userdb.user user, deptdb.dept
> WHERE user.dept_id = dept.id
>
> and i've got the following error :
>
> MismatchedTokenException(279!=26)
> at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
> at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
> at
> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
> at
> org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
> at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)
>
> Any idea?
>
> Thanks
> Jason
>
>

Query parsing error for the join query between different database

[JaeSung Jun]

Hi,

I'm working on custom data source provider, and i'm using fully qualified
table name in FROM clause like following :

SELECT user. uid, dept.name
FROM userdb.user user, deptdb.dept
WHERE user.dept_id = dept.id

and i've got the following error :

MismatchedTokenException(279!=26)
at
org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableSource(HiveParser_FromClauseParser.java:4608)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3729)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1873)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
at
org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45861)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.spark.sql.hive.HiveQl$.getAst(HiveQl.scala:276)
at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:303)
at
org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
at
org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
at scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
at
scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
at
scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
at
scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
at
scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
at
org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:34)
at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:295)

Any idea?

Thanks
Jason

Unit test error

[JaeSung Jun]

Hi All,

I'm developing custom data source & relation provider based on spark 1.6.1.
Every unit test has its own Spark Context, and it runs successfully when
running one by one.
But when running in sbt(sbt:test), error pops up when initializing spark
contest like followings :

org.apache.spark.rpc.RpcEndpointNotFoundException: Cannot find endpoint:
spark://HeartbeatReceiver@192.168.123.101:54079

at
org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$asyncSetupEndpointRefByURI$1.apply(NettyRpcEnv.scala:148)

at
org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$asyncSetupEndpointRefByURI$1.apply(NettyRpcEnv.scala:144)

at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)

at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:249)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)

at
org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)

at
scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)

at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)

at
scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallback(Promise.scala:280)

at
scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)

at scala.concurrent.Future$class.flatMap(Future.scala:249)

at scala.concurrent.impl.Promise$DefaultPromise.flatMap(Promise.scala:153)

at
org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:150)

at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:97)

at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:106)

at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:36)

at org.apache.spark.executor.Executor.(Executor.scala:115)

at
org.apache.spark.scheduler.local.LocalEndpoint.(LocalBackend.scala:58)

at
org.apache.spark.scheduler.local.LocalBackend.start(LocalBackend.scala:125)

at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)

at org.apache.spark.SparkContext.(SparkContext.scala:530)


Anyone any idea?


Thanks Jason

Does RDD[Type1, Iterable[Type2]] split into multiple partitions?

[JaeSung Jun]

Hi,

I'm currently working on Iterable type of RDD, which is like :

val keyValueIterableRDD[CaseClass1, Iterable[CaseClass2]] = buildRDD(...)

If there is only one unique key and Iterable is big enough, would this
Iterable be partitioned across all executors like followings ?

(executor1)
(xxx, iterator from 0 to 10,000)

(executor2)
(xxx, iterator from 10,001 to 20,000)

(executor2)
(xxx, iterator from 20,001 to 30,000)

...

Thanks
Jason

unit test failure for hive query

[JaeSung Jun]

Hi,
I'm working on custom sql processing on top of Spark-SQL, and i'm upgrading
it along with spark 1.4.1.
I've got an error regarding multiple test suites access hive meta store at
the same time like :

 Cause: org.apache.derby.impl.jdbc.EmbedSQLException: Another instance of
Derby may have already booted the database /Users/~~~/metastore_db.

  at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)

  at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)

  at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)

  at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)

  at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)

  at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)

  at org.apache.derby.impl.jdbc.EmbedConnection40.(Unknown Source)

  at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)

  at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)

  at org.apache.derby.jdbc.Driver20.connect(Unknown Source)


It was okay with spark 1.3.0.

Any idea to fist this?


thanks in advance.

Jason

Re: databases currently supported by Spark SQL JDBC

[JaeSung Jun]

As long as JDBC driver is provided, any database can be used in JDBC
datasource provider.
you can provide driver class in options field like followings :

CREATE TEMPORARY TABLE jdbcTable
USING org.apache.spark.sql.jdbc
OPTIOS(
url "jdbc:oracle:thin:@myhost:1521:orcl"
driver "oracle.jdbc.driver.OracleDriver"
dbtable "users"
)

thx
jason

On 9 July 2015 at 22:09, Niranda Perera  wrote:

> Hi,
>
> I'm planning to use Spark SQL JDBC datasource provider in various RDBMS
> databases.
>
> what are the databases currently supported by Spark JDBC relation provider?
>
> rgds
>
> --
> Niranda
> @n1r44 
> https://pythagoreanscript.wordpress.com/
>

Re: Can't find postgresql jdbc driver when using external datasource

[JaeSung Jun]

Thanks Felix,
It worked with spark class path variable as follows :
SPARK_CLASSPATH=postgresql-9.3-1102-jdbc41.jar

I think it should be working with driver class path.

Thanks
Jason

On 21 April 2015 at 22:27, Felix C  wrote:

> It works with --driver-class-path?
>
> Please see
> https://eradiating.wordpress.com/2015/04/17/using-spark-data-sources-to-load-data-from-postgresql/
>
> --- Original Message ---
>
> From: "JaeSung Jun" 
> Sent: April 21, 2015 1:05 AM
> To: dev@spark.apache.org
> Subject: Can't find postgresql jdbc driver when using external datasource
>
> Hi,
>
> I tried to get external data base table running sitting on postgresql.
> i've got java.lang.ClassNotFoundException even if i added driver jar using
> --jars option like followings :
>
> is it class loader hierarchy problem or any idea?
>
> thanks
>
> -
>
> spark-sql --jars ../lib/postgresql-9.4-1200.jdbc41.jar
>
> spark-sql> CREATE TEMPORARY TABLE uusr
>
>  > USING org.apache.spark.sql.jdbc
>
>  > OPTIONS (
>
>  > url
> "jdbc:postgresql://localhost/spark_db?user=postgres&password=password",
>
>  > driver "org.postgresql.Driver",
>
>  > dbtable "uusr"
>
>  > );
>
> ..
>
> java.lang.ClassNotFoundException: org.postgresql.Driver
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:191)
>
>  at
>
> org.apache.spark.sql.jdbc.DefaultSource.createRelation(JDBCRelation.scala:97)
>
> -
>

Can't find postgresql jdbc driver when using external datasource

[JaeSung Jun]

Hi,

I tried to get external data base table running sitting on postgresql.
i've got java.lang.ClassNotFoundException even if i added driver jar using
--jars option like followings :

is it class loader hierarchy problem or any idea?

thanks

-

spark-sql --jars ../lib/postgresql-9.4-1200.jdbc41.jar

spark-sql> CREATE TEMPORARY TABLE uusr

 > USING org.apache.spark.sql.jdbc

 > OPTIONS (

 > url
"jdbc:postgresql://localhost/spark_db?user=postgres&password=password",

 > driver "org.postgresql.Driver",

 > dbtable "uusr"

 > );

..

java.lang.ClassNotFoundException: org.postgresql.Driver

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:191)

 at
org.apache.spark.sql.jdbc.DefaultSource.createRelation(JDBCRelation.scala:97)

-

Re: DDL parser class parsing DDL in spark-sql cli

[JaeSung Jun]

Thanks Michael,

I was wondering how HiveContext.sql() is hooked up HiveQL..I'll have a look
at it.
much appreciated.

Thanks
Jason

On 15 April 2015 at 04:15, Michael Armbrust  wrote:

> HiveQL
> <https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala>
>
> On Tue, Apr 14, 2015 at 7:13 AM, JaeSung Jun  wrote:
>
>> Hi,
>>
>> Wile I've been walking through spark-sql source code, I typed the
>> following
>> HiveQL:
>>
>> CREATE EXTERNAL TABLE user (uid STRING, age INT, gender STRING, job
>> STRING,
>> ts STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION
>> '/hive/user';
>>
>> , and I finally came across ddl.scala after analysing a couple of scala
>> files.
>>
>> What I found here is createTable method, which doesn't parse the above
>> statement.
>> Please let me know which class is responsible for parsing the above
>> statement.
>>
>> Thanks
>> Jason
>>
>
>

DDL parser class parsing DDL in spark-sql cli

[JaeSung Jun]

Hi,

Wile I've been walking through spark-sql source code, I typed the following
HiveQL:

CREATE EXTERNAL TABLE user (uid STRING, age INT, gender STRING, job STRING,
ts STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION
'/hive/user';

, and I finally came across ddl.scala after analysing a couple of scala
files.

What I found here is createTable method, which doesn't parse the above
statement.
Please let me know which class is responsible for parsing the above
statement.

Thanks
Jason