[jira] [Updated] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.

2018-05-31 Thread Volodymyr Glushak (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Glushak updated SPARK-24430:
--
Description: 
When I executes following SQL statement:
{code:java}
spark.sql('CREATE VIEW view_12 AS

SELECT * FROM (

  SELECT * FROM table1

  UNION ALL

  SELECT * FROM table2

) UT')

{code}
 

It successfully creates view in HIVE, which I can query via Apache Spark.

However if I'm trying to query the same view directly via HIVE, I've got an 
error:
{code:java}
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'.

Failed rule: 'identifier' in subquery source

{code}
 

*Investigation*

Under hood, spark generate following SQL statement for this view:
{code:java}
CREATE VIEW `view_12` AS

SELECT *

FROM (SELECT * FROM

    (

   (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
gen_subquery_0)

     UNION ALL

    (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
gen_subquery_1)

    ) AS UT

  ) AS UT

{code}
If I try to execute this statement in HIVE it fails with the same reason.

The easiest way to fix it, is to remove parentheses from outer queries on lines 
5 and 7.

  was:
When I executes following SQL statement:
{code:java}
spark.sql('CREATE VIEW view_12 AS

SELECT * FROM (

  SELECT * FROM table1

  UNION ALL

  SELECT * FROM table2

) UT')

{code}
 

It successfully creates view in HIVE, which I can query via Apache Spark.

However if I'm trying to query the same view directly via HIVE, I've got an 
error:
{code:java}
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'.

Failed rule: 'identifier' in subquery source

{code}
 

*Investigation*

Under hood, spark generate following SQL statement for this view:
{code:java}
CREATE VIEW `view_12` AS

SELECT *

FROM (SELECT * FROM

    (

   (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
gen_subquery_0)

     UNION ALL

    (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
gen_subquery_1)

    ) AS UT

  ) AS UT

{code}
If I try to execute this statement in HIVE it fails with the same reason.

The easiest way to fix it, is to unwrap outer queries on lines 5 and 7.


> CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
> 
>
> Key: SPARK-24430
> URL: https://issues.apache.org/jira/browse/SPARK-24430
> Project: Spark
>  Issue Type: Request
>  Components: Spark Core, SQL
>Affects Versions: 2.2.1
>Reporter: Volodymyr Glushak
>Priority: Major
>
> When I executes following SQL statement:
> {code:java}
> spark.sql('CREATE VIEW view_12 AS
> SELECT * FROM (
>   SELECT * FROM table1
>   UNION ALL
>   SELECT * FROM table2
> ) UT')
> {code}
>  
> It successfully creates view in HIVE, which I can query via Apache Spark.
> However if I'm trying to query the same view directly via HIVE, I've got an 
> error:
> {code:java}
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: ParseException line 6:6 Failed to recognize predicate 
> 'UNION'.
> Failed rule: 'identifier' in subquery source
> {code}
>  
> *Investigation*
> Under hood, spark generate following SQL statement for this view:
> {code:java}
> CREATE VIEW `view_12` AS
> SELECT *
> FROM (SELECT * FROM
>     (
>    (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
> gen_subquery_0)
>      UNION ALL
>     (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
> gen_subquery_1)
>     ) AS UT
>   ) AS UT
> {code}
> If I try to execute this statement in HIVE it fails with the same reason.
> The easiest way to fix it, is to remove parentheses from outer queries on 
> lines 5 and 7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.

2018-05-31 Thread Volodymyr Glushak (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Glushak updated SPARK-24430:
--
Description: 
When I executes following SQL statement:
{code:java}
spark.sql('CREATE VIEW view_12 AS

SELECT * FROM (

  SELECT * FROM table1

  UNION ALL

  SELECT * FROM table2

) UT')

{code}
 

It successfully creates view in HIVE, which I can query via Apache Spark.

However if I'm trying to query the same view directly via HIVE, I've got an 
error:
{code:java}
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'.

Failed rule: 'identifier' in subquery source

{code}
 

*Investigation*

Under hood, spark generate following SQL statement for this view:
{code:java}
CREATE VIEW `view_12` AS

SELECT *

FROM (SELECT * FROM

    (

   (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
gen_subquery_0)

     UNION ALL

    (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
gen_subquery_1)

    ) AS UT

  ) AS UT

{code}
If I try to execute this statement in HIVE it fails with the same reason.

The easiest way to fix it, is to unwrap outer queries on lines 5 and 7.

  was:
When I executes following SQL statement:

{code}

spark.sql('CREATE VIEW view_12 AS

SELECT * FROM (

  SELECT * FROM table1

  UNION ALL

  SELECT * FROM table2

) UT')

{code}

 

It successfully creates view in HIVE, which I can query via Apache Spark.

However if I'm trying to query the same view directly via HIVE, I've got an 
error:

{code}

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'.

Failed rule: 'identifier' in subquery source

{code}

 

*Investigation*

Under hood, spark generate following SQL statement for this view:

{code}

CREATE VIEW `view_12` AS

SELECT *

FROM (SELECT * FROM

    (

   (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
gen_subquery_0)

     UNION ALL

    (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
gen_subquery_1)

    ) AS UT

  ) AS UT

{code}

If I try to executes this statement in HIVE it fails with the same reason.

The easiest way to fix it, is to unwrap outer queries on lines 5 and 7.


> CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
> 
>
> Key: SPARK-24430
> URL: https://issues.apache.org/jira/browse/SPARK-24430
> Project: Spark
>  Issue Type: Request
>  Components: Spark Core, SQL
>Affects Versions: 2.2.1
>Reporter: Volodymyr Glushak
>Priority: Major
>
> When I executes following SQL statement:
> {code:java}
> spark.sql('CREATE VIEW view_12 AS
> SELECT * FROM (
>   SELECT * FROM table1
>   UNION ALL
>   SELECT * FROM table2
> ) UT')
> {code}
>  
> It successfully creates view in HIVE, which I can query via Apache Spark.
> However if I'm trying to query the same view directly via HIVE, I've got an 
> error:
> {code:java}
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: ParseException line 6:6 Failed to recognize predicate 
> 'UNION'.
> Failed rule: 'identifier' in subquery source
> {code}
>  
> *Investigation*
> Under hood, spark generate following SQL statement for this view:
> {code:java}
> CREATE VIEW `view_12` AS
> SELECT *
> FROM (SELECT * FROM
>     (
>    (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
> gen_subquery_0)
>      UNION ALL
>     (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
> gen_subquery_1)
>     ) AS UT
>   ) AS UT
> {code}
> If I try to execute this statement in HIVE it fails with the same reason.
> The easiest way to fix it, is to unwrap outer queries on lines 5 and 7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.

2018-05-30 Thread Volodymyr Glushak (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495374#comment-16495374
 ] 

Volodymyr Glushak commented on SPARK-24430:
---

I wonder, if that is legal code, and this request should be moved to HIVE JIRA.

> CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.
> 
>
> Key: SPARK-24430
> URL: https://issues.apache.org/jira/browse/SPARK-24430
> Project: Spark
>  Issue Type: Request
>  Components: Spark Core, SQL
>Affects Versions: 2.2.1
>Reporter: Volodymyr Glushak
>Priority: Major
>
> When I executes following SQL statement:
> {code}
> spark.sql('CREATE VIEW view_12 AS
> SELECT * FROM (
>   SELECT * FROM table1
>   UNION ALL
>   SELECT * FROM table2
> ) UT')
> {code}
>  
> It successfully creates view in HIVE, which I can query via Apache Spark.
> However if I'm trying to query the same view directly via HIVE, I've got an 
> error:
> {code}
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: ParseException line 6:6 Failed to recognize predicate 
> 'UNION'.
> Failed rule: 'identifier' in subquery source
> {code}
>  
> *Investigation*
> Under hood, spark generate following SQL statement for this view:
> {code}
> CREATE VIEW `view_12` AS
> SELECT *
> FROM (SELECT * FROM
>     (
>    (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
> gen_subquery_0)
>      UNION ALL
>     (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
> gen_subquery_1)
>     ) AS UT
>   ) AS UT
> {code}
> If I try to executes this statement in HIVE it fails with the same reason.
> The easiest way to fix it, is to unwrap outer queries on lines 5 and 7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-24430) CREATE VIEW with UNION statement: Failed to recognize predicate 'UNION'.

2018-05-30 Thread Volodymyr Glushak (JIRA) 
Volodymyr Glushak created SPARK-24430:
-

 Summary: CREATE VIEW with UNION statement: Failed to recognize 
predicate 'UNION'.
 Key: SPARK-24430
 URL: https://issues.apache.org/jira/browse/SPARK-24430
 Project: Spark
  Issue Type: Request
  Components: Spark Core, SQL
Affects Versions: 2.2.1
Reporter: Volodymyr Glushak


When I executes following SQL statement:

{code}

spark.sql('CREATE VIEW view_12 AS

SELECT * FROM (

  SELECT * FROM table1

  UNION ALL

  SELECT * FROM table2

) UT')

{code}

 

It successfully creates view in HIVE, which I can query via Apache Spark.

However if I'm trying to query the same view directly via HIVE, I've got an 
error:

{code}

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: ParseException line 6:6 Failed to recognize predicate 'UNION'.

Failed rule: 'identifier' in subquery source

{code}

 

*Investigation*

Under hood, spark generate following SQL statement for this view:

{code}

CREATE VIEW `view_12` AS

SELECT *

FROM (SELECT * FROM

    (

   (SELECT * FROM (SELECT * FROM `db1`.`table1`) AS 
gen_subquery_0)

     UNION ALL

    (SELECT * FROM (SELECT  * FROM `db1`.`tabl2`) AS 
gen_subquery_1)

    ) AS UT

  ) AS UT

{code}

If I try to executes this statement in HIVE it fails with the same reason.

The easiest way to fix it, is to unwrap outer queries on lines 5 and 7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-23993) Support DESC FORMATTED table_name column_name

2018-05-01 Thread Volodymyr Glushak (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Glushak resolved SPARK-23993.
---
   Resolution: Duplicate
Fix Version/s: 2.3.0

Duplicate of SPARK-17642.

Thanks [~ksunitha] for investigation.

> Support DESC FORMATTED table_name column_name
> -
>
> Key: SPARK-23993
> URL: https://issues.apache.org/jira/browse/SPARK-23993
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.1.2
>Reporter: Volodymyr Glushak
>Priority: Major
> Fix For: 2.3.0
>
>
> Hive and Spark both supports:
> {code}
> DESC FORMATTED table_name{code}
> which gives table metadata.
> If you want to get metadata for particular column in hive you can execute:
> {code}
> DESC FORMATTED table_name column_name{code}
> Thos is not supported in Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-23993) Support DESC FORMATTED table_name column_name

2018-04-16 Thread Volodymyr Glushak (JIRA) 
Volodymyr Glushak created SPARK-23993:
-

 Summary: Support DESC FORMATTED table_name column_name
 Key: SPARK-23993
 URL: https://issues.apache.org/jira/browse/SPARK-23993
 Project: Spark
  Issue Type: Improvement
  Components: SQL
Affects Versions: 2.1.2
Reporter: Volodymyr Glushak


Hive and Spark both supports:
{code}
DESC FORMATTED table_name{code}
which gives table metadata.
If you want to get metadata for particular column in hive you can execute:
{code}
DESC FORMATTED table_name column_name{code}
Thos is not supported in Spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-15276) CREATE TABLE with LOCATION should imply EXTERNAL

2018-03-27 Thread Volodymyr Glushak (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415825#comment-16415825
 ] 

Volodymyr Glushak edited comment on SPARK-15276 at 3/27/18 3:55 PM:


[~andrewor14],

HIVE does not "externalise" table  if LOCATION specified. (I reckon Impala 
neither)

Why does Apache Spark introduce different behaviour? 

 

 


was (Author: rumoku):
[~andrewor14],

HIVE does not "externalise" table  if LOCATION specified.

Why does Apache Spark introduce different behaviour? 

 

 

> CREATE TABLE with LOCATION should imply EXTERNAL
> 
>
> Key: SPARK-15276
> URL: https://issues.apache.org/jira/browse/SPARK-15276
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>Assignee: Andrew Or
>Priority: Major
> Fix For: 2.0.0
>
>
> If the user runs `CREATE TABLE some_table ... LOCATION /some/path`, then this 
> will still be a managed table even though the table's data is stored at 
> /some/path. The problem is that when we drop the table we'll also delete the 
> data /some/path. This could cause problems if /some/path contains existing 
> data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-15276) CREATE TABLE with LOCATION should imply EXTERNAL

2018-03-27 Thread Volodymyr Glushak (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415825#comment-16415825
 ] 

Volodymyr Glushak edited comment on SPARK-15276 at 3/27/18 3:51 PM:


[~andrewor14],

HIVE does not "externalise" table  if LOCATION specified.

Why does Apache Spark introduce different behaviour? 

 

 


was (Author: rumoku):
[~andrewor14],

HIVE is not adding EXTERNAL keyword  if LOCATION specified.

Why does Apache Spark introduce different behaviour? 

 

 

> CREATE TABLE with LOCATION should imply EXTERNAL
> 
>
> Key: SPARK-15276
> URL: https://issues.apache.org/jira/browse/SPARK-15276
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>Assignee: Andrew Or
>Priority: Major
> Fix For: 2.0.0
>
>
> If the user runs `CREATE TABLE some_table ... LOCATION /some/path`, then this 
> will still be a managed table even though the table's data is stored at 
> /some/path. The problem is that when we drop the table we'll also delete the 
> data /some/path. This could cause problems if /some/path contains existing 
> data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-15276) CREATE TABLE with LOCATION should imply EXTERNAL

2018-03-27 Thread Volodymyr Glushak (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415825#comment-16415825
 ] 

Volodymyr Glushak commented on SPARK-15276:
---

[~andrewor14],

HIVE is not adding EXTERNAL keyword  if LOCATION specified.

Why does Apache Spark introduce different behaviour? 

 

 

> CREATE TABLE with LOCATION should imply EXTERNAL
> 
>
> Key: SPARK-15276
> URL: https://issues.apache.org/jira/browse/SPARK-15276
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Andrew Or
>Assignee: Andrew Or
>Priority: Major
> Fix For: 2.0.0
>
>
> If the user runs `CREATE TABLE some_table ... LOCATION /some/path`, then this 
> will still be a managed table even though the table's data is stored at 
> /some/path. The problem is that when we drop the table we'll also delete the 
> data /some/path. This could cause problems if /some/path contains existing 
> data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org