[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661305#comment-14661305 ] Apache Spark commented on SPARK-7550: - User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/8021 > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao >Priority: Blocker > Fix For: 1.5.0 > > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660739#comment-14660739 ] Reynold Xin commented on SPARK-7550: I cherry-picked it into branch-1.5. > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao >Priority: Blocker > Fix For: 1.5.0 > > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660726#comment-14660726 ] Reynold Xin commented on SPARK-7550: [~lian cheng] shouldn't this also be merged into branch-1.5? > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao >Priority: Blocker > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658616#comment-14658616 ] Apache Spark commented on SPARK-7550: - User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/7967 > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao >Priority: Blocker > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585489#comment-14585489 ] Apache Spark commented on SPARK-7550: - User 'chenghao-intel' has created a pull request for this issue: https://github.com/apache/spark/pull/5733 > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581315#comment-14581315 ] Cheng Hao commented on SPARK-7550: -- I will start working on this today, sorry for the delay. > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581311#comment-14581311 ] Nathan McCarthy commented on SPARK-7550: This is a major blocker for us as we use spark to write to hive. Is there a work around? > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561541#comment-14561541 ] Yin Huai commented on SPARK-7550: - I think it will also address https://issues.apache.org/jira/browse/SPARK-6413. > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561416#comment-14561416 ] Yin Huai commented on SPARK-7550: - [~chenghao] Will you have time to take a look at it? It is a related issue of SPARK-6923. I think once we store serde info using Hive's data structures, our CLI will work correctly. But for SPARK-6923, it also needs to handle data source tables that only have schema info in the table properties. > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin >Assignee: Cheng Hao > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561222#comment-14561222 ] Yin Huai commented on SPARK-7550: - Yeah. That is the plan. > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560455#comment-14560455 ] Reynold Xin commented on SPARK-7550: cc [~yhuai]. I think the proposed design is to write the schema out according to Hive's format, for data types that Hive supports. For UDTs, just write them out like what we do right now. > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560450#comment-14560450 ] Cheng Hao commented on SPARK-7550: -- Similar issue with SPARK-6923 > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560444#comment-14560444 ] Reynold Xin commented on SPARK-7550: [~chenghao] can you work on this one? > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7550) Support setting the right schema & serde when writing to Hive metastore
[ https://issues.apache.org/jira/browse/SPARK-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14538936#comment-14538936 ] Apache Spark commented on SPARK-7550: - User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/6067 > Support setting the right schema & serde when writing to Hive metastore > --- > > Key: SPARK-7550 > URL: https://issues.apache.org/jira/browse/SPARK-7550 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.4.0 >Reporter: Reynold Xin > > As of 1.4, Spark SQL does not properly set the table schema and serde when > writing a table to Hive's metastore. Would be great to do that properly so > users can use non-Spark SQL systems to read those tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
