[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925141#comment-13925141 ] Lefty Leverenz commented on HIVE-3682: -- [~lars_francke] added this note to the wiki: "As of Hive 0.11.0 the separator used can be specified, in earlier versions it was always the ^A character (\001)" and [~prasadm] added the ROW FORMAT syntax. More details and some examples would be helpful. * [LanguageManual DML: Writing data into the filesystem from queries |https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Writingdataintothefilesystemfromqueries] > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898805#comment-13898805 ] Amareshwari Sriramadasu commented on HIVE-3682: --- Though above suggestion of creating a table and insert overwrite table works, it enforces the user to know schema of the output and create the table ahead. When queries are automated, it is difficult to always create the table ahead. I have created the issue HIVE-6410 for adding the functionality in this issue for INSERT OVERWRITE DIRECTORY as well. > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831773#comment-13831773 ] Sushanth Sowmyan commented on HIVE-3682: [~caofangkun] : Thanks for bringing that up, apologies for not noticing it till now, I'll add it in to the wiki. [~vratnagiri] : Well, for writing out to hdfs, there already exists a way to do this, and that is to write out to a new table at that location. What was lacking was the ability to be able to support a write to a local directory with the features that exist for a hdfs write, and therefore, this was added. Basically, you can do a CREATE TABLE with whatever format you want, at an appropriate hdfs location, and then do an INSERT OVERWRITE into that table with the results of whatever SELECT you desire. :) > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831625#comment-13831625 ] Vijay Ratnagiri commented on HIVE-3682: --- Hey Guys, I was really delighted to find that the export finally supported choosing the format, but unfortunately, my delight was short lived when I discovered thet this feature is supported only for 'insert overwrite LOCAL directory' and not when I'm exporting to an HDFS directory. I get a syntax/parse error when I try to export to an HDFS directory with a custom row format. How come this feature was implimented like this? If this wasn't intentional, then, does this warrant reopening this ticket? Thanks! > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13679310#comment-13679310 ] Navis commented on HIVE-3682: - [~caofangkun] I don't know anything about this issue. I think you've meant to refer [~sushanth]? > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13677784#comment-13677784 ] caofangkun commented on HIVE-3682: -- Hi [~navis] could you please put this into the wiki ? https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Writingdataintofilesystemfromqueries > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646304#comment-13646304 ] Hudson commented on HIVE-3682: -- Integrated in Hive-trunk-hadoop2 #183 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/183/]) HIVE-3682 : when output hive table to file,users should could have a separator of their own choice (Sushanth Sowmyan via Ashutosh Chauhan) (Revision 1477368) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477368 Files : * /hive/trunk/data/files/array_table.txt * /hive/trunk/data/files/map_table.txt * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java * /hive/trunk/ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q * /hive/trunk/ql/src/test/results/clientpositive/insert_overwrite_local_directory_1.q.out > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646149#comment-13646149 ] Hudson commented on HIVE-3682: -- Integrated in Hive-trunk-h0.21 #2086 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2086/]) HIVE-3682 : when output hive table to file,users should could have a separator of their own choice (Sushanth Sowmyan via Ashutosh Chauhan) (Revision 1477368) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477368 Files : * /hive/trunk/data/files/array_table.txt * /hive/trunk/data/files/map_table.txt * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java * /hive/trunk/ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q * /hive/trunk/ql/src/test/results/clientpositive/insert_overwrite_local_directory_1.q.out > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.D10275.4.patch, > HIVE-3682.D10275.4.patch.for.0.11, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643742#comment-13643742 ] Phabricator commented on HIVE-3682: --- ashutoshc has accepted the revision "HIVE-3682 [jira] when output hive table to file,users should could have a separator of their own choice". +1 will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D10275 BRANCH HIVE-3682 ARCANIST PROJECT hive To: JIRA, ashutoshc, omalley, khorgath > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Fix For: 0.11.0 > > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.D10275.2.patch, HIVE-3682.D10275.3.patch, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[
https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640811#comment-13640811
]
Phabricator commented on HIVE-3682:
---
ashutoshc has requested changes to the revision "HIVE-3682 [jira] when output
hive table to file,users should could have a separator of their own choice".
Some comments.
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/plan/LocalDirectoryDesc.java:29 Since
all of these fields are subset of fields defined in CreateTableDesc. I wonder
if you can reuse that class instead of creating a new one? In case you can not,
atleast consider extending that.
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1208 As I
have indicated below if you can reuse/extend CreateTableDesc instead of
LocalDirectoryDesc, you probably can refactor and reuse much of the parsing
logic from create table and that will be good to have.
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java:102 As I mentioned
earlier, if we reuse CreateTableDesc class, probably much of this code could be
avoided.
ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q:8 Can
you add a test which results in MR job. e.g, doing join, group-by etc?
ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q:72
After HIVE-4369 we need to use dfs ${system:test.dfs.mkdir} for mkdir command
because of hadoop incompatibility issues.
REVISION DETAIL
https://reviews.facebook.net/D10275
BRANCH
HIVE-3682
ARCANIST PROJECT
hive
To: JIRA, ashutoshc, omalley, khorgath
> when output hive table to file,users should could have a separator of their
> own choice
> --
>
> Key: HIVE-3682
> URL: https://issues.apache.org/jira/browse/HIVE-3682
> Project: Hive
> Issue Type: New Feature
> Components: CLI
>Affects Versions: 0.8.1
> Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21
> 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
> java version "1.6.0_25"
> hadoop-0.20.2-cdh3u0
> hive-0.8.1
>Reporter: caofangkun
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch,
> HIVE-3682.with.serde.patch
>
>
> By default,when output hive table to file ,columns of the Hive table are
> separated by ^A character (that is \001).
> But indeed users should have the right to set a seperator of their own choice.
> Usage Example:
> create table for_test (key string, value string);
> load data local inpath './in1.txt' into table for_test
> select * from for_test;
> UT-01：default separator is \001 line separator is \n
> insert overwrite local directory './test-01'
> select * from src ;
> create table array_table (a array, b array)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ',';
> load data local inpath "../hive/examples/files/arraytest.txt" overwrite into
> table table2;
> CREATE TABLE map_table (foo STRING , bar MAP)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ','
> MAP KEYS TERMINATED BY ':'
> STORED AS TEXTFILE;
> UT-02：defined field separator as ':'
> insert overwrite local directory './test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select * from src ;
> UT-03: line separator DO NOT ALLOWED to define as other separator
> insert overwrite local directory './test-03'
> row format delimited
> FIELDS TERMINATED BY ':'
> select * from src ;
> UT-04: define map separators
> insert overwrite local directory './test-04'
> row format delimited
> FIELDS TERMINATED BY '\t'
> COLLECTION ITEMS TERMINATED BY ','
> MAP KEYS TERMINATED BY ':'
> select * from src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640809#comment-13640809 ] Ashutosh Chauhan commented on HIVE-3682: Left some comments on phabricator. > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Sushanth Sowmyan > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640700#comment-13640700 ] caofangkun commented on HIVE-3682: -- Hi [~gangtimliu] , I'm not a committer yet, so I could not assign this issue to myself. Please feel free and assign this issue . Thanks > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Gang Tim Liu > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639575#comment-13639575 ] Gang Tim Liu commented on HIVE-3682: Hi ~caofangkun, thank you for working on it. Would you please assign this issue to yourself? thanks > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Gang Tim Liu > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632647#comment-13632647 ] Phabricator commented on HIVE-3682: --- khorgath has added reviewers to the revision "HIVE-3682 [jira] when output hive table to file,users should could have a separator of their own choice". Added Reviewers: ashutoshc, omalley Updated patch based on [~caofangkun]'s initial patch to support STORED-AS and SerDe specification. REVISION DETAIL https://reviews.facebook.net/D10275 To: JIRA, ashutoshc, omalley, khorgath > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Gang Tim Liu > Attachments: HIVE-3682-1.patch, HIVE-3682.D10275.1.patch, > HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13629787#comment-13629787 ] caofangkun commented on HIVE-3682: -- Thanks [~sushanth] and the "STORED AS " feature is very useful for me too. > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Gang Tim Liu >Priority: Minor > Attachments: HIVE-3682-1.patch, HIVE-3682.with.serde.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. > Usage Example: > create table for_test (key string, value string); > load data local inpath './in1.txt' into table for_test > select * from for_test; > UT-01：default separator is \001 line separator is \n > insert overwrite local directory './test-01' > select * from src ; > create table array_table (a array, b array) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ','; > load data local inpath "../hive/examples/files/arraytest.txt" overwrite into > table table2; > CREATE TABLE map_table (foo STRING , bar MAP) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > STORED AS TEXTFILE; > UT-02：defined field separator as ':' > insert overwrite local directory './test-02' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-03: line separator DO NOT ALLOWED to define as other separator > insert overwrite local directory './test-03' > row format delimited > FIELDS TERMINATED BY ':' > select * from src ; > UT-04: define map separators > insert overwrite local directory './test-04' > row format delimited > FIELDS TERMINATED BY '\t' > COLLECTION ITEMS TERMINATED BY ',' > MAP KEYS TERMINATED BY ':' > select * from src; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3682) when output hive table to file,users should could have a separator of their own choice
[ https://issues.apache.org/jira/browse/HIVE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612464#comment-13612464 ] caofangkun commented on HIVE-3682: -- https://reviews.apache.org/r/10115/ > when output hive table to file,users should could have a separator of their > own choice > -- > > Key: HIVE-3682 > URL: https://issues.apache.org/jira/browse/HIVE-3682 > Project: Hive > Issue Type: New Feature > Components: CLI >Affects Versions: 0.8.1 > Environment: Linux 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 > 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux > java version "1.6.0_25" > hadoop-0.20.2-cdh3u0 > hive-0.8.1 >Reporter: caofangkun >Assignee: Gang Tim Liu >Priority: Minor > Attachments: HIVE-3682-1.patch > > > By default,when output hive table to file ,columns of the Hive table are > separated by ^A character (that is \001). > But indeed users should have the right to set a seperator of their own choice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
