[jira] [Updated] (SPARK-45403) Spark SQL returns table column names as literal data values for Hive tables

2023-10-03 Thread Reece Robinson (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-45403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reece Robinson updated SPARK-45403:
---
Description: 
When using Spark SQL and Hive JDBC driver to access a Hive table the resulting 
row data is replaced with the literal column name in the resulting dataframe 
result.

When I run this:

jdbcDF = spark.read \
  .format("jdbc") \
  .options(driver="org.apache.hive.jdbc.HiveDriver",
           url="jdbc:hive2://10.20.174.171:10009",
           user="10009",
           password="123",
           query="select * from demo.hospitals limit 10"
           ) \
  .load()

 

I get:

+```+

++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+

+```+

I should see:

```

++++-+-+---++++--+---+--+-+-+
 | person_pk| race_value|sex_code|poverty_value|veteran_value|ppr_pro| 
patient_pk| di_dk| pov_pk|vet_pk|veteran|total_paid|num_drugs|immunized| 
++++-+-+---++++--+---+--+-+-+
 |001252a7-a1e7-428...|01 - American Ind...| F| 37.0| null| 
2|65007233-424e-4c2...|9d66f5b7-ab10-47f...|1f3d76c8-d039-483...| |unknown| 
null| null| true| |002673d4-579a-4d1...|01 - American Ind...| M| 64.0| null| 
2|a3c89a7f-d57d-4be...|2f6ffa09-e5b3-419...|7dbfc730-64bc-4a9...| |unknown| 
null| null| true| |00267822-8192-44f...|01 - American Ind...| F| 0.0| null| 
2|cd318b72-35d4-422...|44646492-60ef-44e...|d5f462ef-cd4c-497...| |unknown| 
null| null| true| |0028fece-59ec-4db...|01 - American Ind...| F| 0.0| null| 
2|ee9e09aa-67bc-47e...|3be068de-7fe3-44d...|63a04010-c381-4aa...| |unknown| 
null| null| true| |003470e7-b548-444...|06 - American Ind...| M| 171.0| null| 
2|7ed5b0f9-02b3-459...|1b778c9f-71ab-45a...|84ecc23a-6c39-44d...| |unknown| 
null| null| false| |0044a493-e226-409...|01 - American Ind...| F| 0.0| null| 
2|c821f5b2-d0af-428...|26144dac-81f0-44e...|f7355eeb-89a3-4f0...| |unknown| 
null| null| true| |004d44d0-fdf7-403...|01 - American Ind...| F| 37.0| null| 
2|cb6c8e5c-71ab-409...|88eaf3c4-5f00-4e9...|78679644-f4e7-450...| |unknown| 
null| null| true| |0059c1bf-5263-42a...|03 - Black or Afr...| M| 0.0| null| 
2|da9247d1-96fb-44d...|6831544a-faf9-426...|3534f3a8-a367-41e...| |unknown| 
null| null| true| |007b82b6-ae2e-49e...|01 - American Ind...| M| 43.0| null| 
2|3e6fcc8c-c484-465...|90e2a03f-f0a4-48f...|5c9c71e1-901b-481...| |unknown| 
null| null|

[jira] [Updated] (SPARK-45403) Spark SQL returns table column names as literal data values for Hive tables

2023-10-03 Thread Reece Robinson (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-45403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reece Robinson updated SPARK-45403:
---
Description: 
When using Spark SQL and Hive JDBC driver to access a Hive table the resulting 
row data is replaced with the literal column name in the resulting dataframe 
result.

When I run this:

jdbcDF = spark.read \
  .format("jdbc") \
  .options(driver="org.apache.hive.jdbc.HiveDriver",
           url="jdbc:hive2://10.20.174.171:10009",
           user="10009",
           password="123",
           query="select * from demo.hospitals limit 10"
           ) \
  .load()

 

I get:

{+}---{-}{-}{+}-{-}++{-}--{-}{-}-{-}++{-}--{-}{-}---{-}++{-}-{-}{-}-{-}++{-}-{-}{-}-{-}++{-}---{-}{-}{-}++{-}--{-}{-}---{-}++{-}--{-}{-}-{-}++{-}-{-}{-}---{-}++{-}-{-}{-}{-}+
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
++{-}--{-}{-}-{-}++{-}--{-}{-}-{-}++{-}--{-}{-}---{-}++{-}-{-}{-}-{-}++{-}-{-}{-}-{-}++{-}---{-}{-}{-}++{-}--{-}{-}---{-}++{-}--{-}{-}-{-}++{-}-{-}{-}---{-}++{-}-{-}{-}{-}+
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
++{-}--{-}{-}-{-}++{-}--{-}{-}-{-}++{-}--{-}{-}---{-}++{-}-{-}{-}-{-}++{-}-{-}{-}-{-}++{-}---{-}{-}{-}++{-}--{-}{-}---{-}++{-}--{-}{-}-{-}++{-}-{-}{-}---{-}++{-}-{-}{-}-+

 

I should see:

{+}---{-}{-}{+}--{-}++{-}--{-}{-}---{-}++{-}---{-}{-}-{-}++{-}--{-}{-}--{-}++{-}--{-}{-}{-}++{-}-{-}{-}{-}++{-}---{-}{-}---{-}+
 | person_pk| race_value|sex_code|poverty_value|veteran_value|ppr_pro| 
patient_pk| di_dk| pov_pk|vet_pk|veteran|total_paid|num_drugs|immunized| 
++{-}--{-}{-}--{-}++{-}--{-}{-}---{-}++{-}---{-}{-}-{-}++{-}--{-}{-}--{-}++{-}--{-}{-}{-}++{-}-{-}{-}{-}++{-}---{-}{-}---{-}+
 |001252a7-a1e7-428...|01 - American Ind...| F| 37.0| null| 
2|65007233-424e-4c2...|9d66f5b7-ab10-47f...|1f3d76c8-d039-483...| |unknown| 
null| null| true| |002673d4-579a-4d1...|01 - American Ind...| M| 64.0| null| 
2|a3c89a7f-d57d-4be...|2f6ffa09-e5b3-419...|7dbfc730-64bc-4a9...| |unknown| 
null| null| true| |00267822-8192-44f...|01 - American Ind...| F| 0.0| null| 
2|cd318b72-35d4-422...|44646492-60ef-44e...|d5f462ef-cd4c-497...| |unknown| 
null| null| true| |0028fece-59ec-4db...|01 - American Ind...| F| 0.0| null| 
2|ee9e09aa-67bc-47e...|3be068de-7fe3-44d...|63a04010-c381-4aa...| |unknown| 
null| null| true| |003470e7-b548-444...|06 - American Ind...| M| 171.0| null| 
2|7ed5b0f9-02b3-459...|1b778c9f-71ab-45a...|84ecc23a-6c39-44d...| |unknown| 
null| null| false| |0044a493-e226-409...|01 - American Ind...| F| 0.0| null| 
2|c821f5b2-d0af-428...|26144dac-81f0-44e...|f7355eeb-89a3-4f0...| |unknown| 
null| null| true| |004d44d0-fdf7-403...|01 - American Ind...| F| 37.0| null|

[jira] [Updated] (SPARK-45403) Spark SQL returns table column names as literal data values for Hive tables

2023-10-03 Thread Reece Robinson (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-45403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reece Robinson updated SPARK-45403:
---
Attachment: Screenshot 2023-10-04 at 11.11.28 AM.png

> Spark SQL returns table column names as literal data values for Hive tables
> ---
>
> Key: SPARK-45403
> URL: https://issues.apache.org/jira/browse/SPARK-45403
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.4.0
> Environment: I am using Spark 3.4.0 however this has been an issue 
> for years.
>Reporter: Reece Robinson
>Priority: Major
> Attachments: Screenshot 2023-10-04 at 11.11.28 AM.png
>
>
> When using Spark SQL and Hive JDBC driver to access a Hive table the 
> resulting row data is replaced with the literal column name in the resulting 
> dataframe result.
> When I run this:
> jdbcDF = spark.read \
>   .format("jdbc") \
>   .options(driver="org.apache.hive.jdbc.HiveDriver",
>            url="jdbc:hive2://10.20.174.171:10009",
>            user="10009",
>            password="123",
>            query="select * from demo.hospitals limit 10"
>            ) \
>   .load()
>  
> I get:
> ++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> ++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
>  
> ++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
>  
> I should see:
> ++++-+-+---++++--+---+--+-+-+
>  | person_pk| race_value|sex_code|poverty_value|veteran_value|ppr_pro| 
> patient_pk| di_dk| pov_pk|vet_pk|veteran|total_paid|num_drugs|immunized| 
> ++++-+-+---++++--+---+--+-+-+
>  |001252a7-a1e7-428...|01 - American Ind...| F| 37.0| null| 
> 2|65007233-424e-4c2...|9d66f5b7-ab10-47f...|1f3d76c8-d039-483...| |unknown| 
> null| null| true| |002673d4-579a-4d1...|01 - American Ind...| M| 64.0| null| 
> 2|a3c89a7f-d57d-4be...|2f6ffa09-e5b3-419...|7dbfc730-64bc-4a9...| |unknown| 
> null| null| true| |00267822-8192-44f...|01 - American Ind...| F| 0.0| null| 
> 2|cd318b72-35d4-422...|44646492-60ef-44e...|d5f462ef-cd4c-497...| |unknown| 
> null| null| true| |0028fece-59ec-4db...|01 - American Ind...| F| 0.0| null| 
> 2|ee9e09aa-67bc-47e...|3be068de-7fe3-44d...|63a04010-c381-4aa...| |unknown| 
> null| null| true| |003470e7-b548-444...|06 - American

[jira] [Created] (SPARK-45403) Spark SQL returns table column names as literal data values for Hive tables

2023-10-03 Thread Reece Robinson (Jira) 
Reece Robinson created SPARK-45403:
--

 Summary: Spark SQL returns table column names as literal data 
values for Hive tables
 Key: SPARK-45403
 URL: https://issues.apache.org/jira/browse/SPARK-45403
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.4.0
 Environment: I am using Spark 3.4.0 however this has been an issue for 
years.
Reporter: Reece Robinson


When using Spark SQL and Hive JDBC driver to access a Hive table the resulting 
row data is replaced with the literal column name in the resulting dataframe 
result.

When I run this:

jdbcDF = spark.read \
  .format("jdbc") \
  .options(driver="org.apache.hive.jdbc.HiveDriver",
           url="jdbc:hive2://10.20.174.171:10009",
           user="10009",
           password="123",
           query="select * from demo.hospitals limit 10"
           ) \
  .load()

 

I get:

++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
|provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK|
 
++---++---++-+---+---+---+---+-+--++-++---+---+-+---+--+

 

I should see:

++++-+-+---++++--+---+--+-+-+
 | person_pk| race_value|sex_code|poverty_value|veteran_value|ppr_pro| 
patient_pk| di_dk| pov_pk|vet_pk|veteran|total_paid|num_drugs|immunized| 
++++-+-+---++++--+---+--+-+-+
 |001252a7-a1e7-428...|01 - American Ind...| F| 37.0| null| 
2|65007233-424e-4c2...|9d66f5b7-ab10-47f...|1f3d76c8-d039-483...| |unknown| 
null| null| true| |002673d4-579a-4d1...|01 - American Ind...| M| 64.0| null| 
2|a3c89a7f-d57d-4be...|2f6ffa09-e5b3-419...|7dbfc730-64bc-4a9...| |unknown| 
null| null| true| |00267822-8192-44f...|01 - American Ind...| F| 0.0| null| 
2|cd318b72-35d4-422...|44646492-60ef-44e...|d5f462ef-cd4c-497...| |unknown| 
null| null| true| |0028fece-59ec-4db...|01 - American Ind...| F| 0.0| null| 
2|ee9e09aa-67bc-47e...|3be068de-7fe3-44d...|63a04010-c381-4aa...| |unknown| 
null| null| true| |003470e7-b548-444...|06 - American Ind...| M| 171.0| null| 
2|7ed5b0f9-02b3-459...|1b778c9f-71ab-45a...|84ecc23a-6c39-44d...| |unknown| 
null| null| false| |0044a493-e226-409...|01 - American Ind...| F| 0.0| null| 
2|c821f5b2-d0af-428...|26144dac-81f0-44e...|f7355eeb-89a3-4f0...| |unknown| 
null| null| true| |004d44d0-fdf7-403...|01 - American Ind...| F| 37.0| null| 
2|cb6c8e5c-71ab-409...|88eaf3c4-5f00-4e9...|78679644-f4e7-450...| |unknown| 
null| null| true|

[jira] [Updated] (SPARK-27943) Implement default constraint with Column for Hive table

2023-10-03 Thread Reece Robinson (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-27943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reece Robinson updated SPARK-27943:
---
Attachment: (was: Screenshot 2023-10-04 at 11.11.28 AM.png)

> Implement default constraint with Column for Hive table
> ---
>
> Key: SPARK-27943
> URL: https://issues.apache.org/jira/browse/SPARK-27943
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Jiaan Geng
>Priority: Major
>
>  
>  *Background*
> Default constraint with column is ANSI standard.
> Hive 3.0+ has supported default constraint 
> ref:https://issues.apache.org/jira/browse/HIVE-18726
> But Spark SQL implement this feature not yet.
> *Design*
> Hive is widely used in production environments and is the standard in the 
> field of big data in fact.
> But Hive exists many version used in production and the feature between each 
> version are different.
> Spark SQL need to implement default constraint, but there are three points to 
> pay attention to in design:
> _First_, Spark SQL should reduce coupling with Hive.
> _Second_, default constraint could compatible with different versions of Hive.
> _Thrid_, Which expression of default constraint should Spark SQL support? I 
> think should support `literal`, `current_date()`, `current_timestamp()`. 
> Maybe other expression should also supported, like `Cast(1 as float)`, `1 + 
> 2` and so on.
> We want to save the metadata of default constraint into properties of Hive 
> table, and then we restore metadata from the properties after client gets 
> newest metadata.The implement is the same as other metadata (e.g. 
> partition,bucket,statistics).
> Because default constraint is part of column, so I think could reuse the 
> metadata of StructField. The default constraint will cached by metadata of 
> StructField.
>  
> *Tasks*
> This is a big work, wo I want to split this work into some sub tasks, as 
> follows:
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-27943) Implement default constraint with Column for Hive table

2023-10-03 Thread Reece Robinson (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-27943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reece Robinson updated SPARK-27943:
---
Attachment: Screenshot 2023-10-04 at 11.11.28 AM.png

> Implement default constraint with Column for Hive table
> ---
>
> Key: SPARK-27943
> URL: https://issues.apache.org/jira/browse/SPARK-27943
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Jiaan Geng
>Priority: Major
> Attachments: Screenshot 2023-10-04 at 11.11.28 AM.png
>
>
>  
>  *Background*
> Default constraint with column is ANSI standard.
> Hive 3.0+ has supported default constraint 
> ref:https://issues.apache.org/jira/browse/HIVE-18726
> But Spark SQL implement this feature not yet.
> *Design*
> Hive is widely used in production environments and is the standard in the 
> field of big data in fact.
> But Hive exists many version used in production and the feature between each 
> version are different.
> Spark SQL need to implement default constraint, but there are three points to 
> pay attention to in design:
> _First_, Spark SQL should reduce coupling with Hive.
> _Second_, default constraint could compatible with different versions of Hive.
> _Thrid_, Which expression of default constraint should Spark SQL support? I 
> think should support `literal`, `current_date()`, `current_timestamp()`. 
> Maybe other expression should also supported, like `Cast(1 as float)`, `1 + 
> 2` and so on.
> We want to save the metadata of default constraint into properties of Hive 
> table, and then we restore metadata from the properties after client gets 
> newest metadata.The implement is the same as other metadata (e.g. 
> partition,bucket,statistics).
> Because default constraint is part of column, so I think could reuse the 
> metadata of StructField. The default constraint will cached by metadata of 
> StructField.
>  
> *Tasks*
> This is a big work, wo I want to split this work into some sub tasks, as 
> follows:
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip

2016-08-22 Thread Reece Robinson (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432109#comment-15432109
 ] 

Reece Robinson commented on SPARK-4563:
---

+1 I also disagree with the minor rating. This is essential for our production 
strategy that we containerise our workloads.

> Allow spark driver to bind to different ip then advertise ip
> 
>
> Key: SPARK-4563
> URL: https://issues.apache.org/jira/browse/SPARK-4563
> Project: Spark
>  Issue Type: Improvement
>  Components: Deploy
>Reporter: Long Nguyen
>Priority: Minor
>
> Spark driver bind ip and advertise is not configurable. spark.driver.host is 
> only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option 
> to set advertised ip/hostname



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org