[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814317#comment-15814317 ] Rui Li commented on HIVE-15519: --- Ping [~thejas] again. [~xuefuz], could you also have a look? Thanks. > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, HIVE-15519.2.patch, > data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) scale is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, scale > is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796059#comment-15796059 ] Bharat Viswanadham commented on HIVE-15519: --- [~thejas] Could you please take a look in to this patch. > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, HIVE-15519.2.patch, > data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) scale is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, scale > is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786741#comment-15786741 ] Rui Li commented on HIVE-15519: --- The failures are not related because I can't reproduce them locally. [~thejas], please take a look. Thanks! > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, HIVE-15519.2.patch, > data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) scale is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, scale > is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784921#comment-15784921 ] Hive QA commented on HIVE-15519: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12845022/HIVE-15519.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10863 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=233) TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=144) [vectorized_rcfile_columnar.q,vector_elt.q,explainuser_1.q,multi_insert.q,tez_dml.q,vector_bround.q,schema_evol_orc_acid_table.q,vector_when_case_null.q,orc_ppd_schema_evol_1b.q,vector_join30.q,vectorization_11.q,cte_3.q,update_tmp_table.q,vector_decimal_cast.q,groupby_grouping_id2.q,vector_decimal_round.q,tez_smb_empty.q,orc_merge6.q,vector_char_mapjoin1.q,vector_decimal_trailing.q,cte_5.q,tez_union.q,vector_decimal_2.q,columnStatsUpdateForStatsOptimizer_1.q,vector_outer_join3.q,schema_evol_text_vec_part_all_complex.q,tez_dynpart_hashjoin_2.q,auto_sortmerge_join_12.q,offset_limit.q,tez_union_multiinsert.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=113) [bucketmapjoin3.q,union_date.q,cbo_gby.q,auto_join31.q,auto_sortmerge_join_1.q,join_cond_pushdown_unqual1.q,ppd_outer_join3.q,bucket_map_join_spark3.q,union28.q,statsfs.q,escape_sortby1.q,leftsemijoin.q,groupby_multi_single_reducer3.q,union_remove_6.q,join29.q] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=134) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=93) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=92) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] (batchId=93) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2745/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2745/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2745/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12845022 - PreCommit-HIVE-Build > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, HIVE-15519.2.patch, > data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) precison is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, > precision is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784767#comment-15784767 ] Rui Li commented on HIVE-15519: --- Update patch v2 to add a UT. And also put it in an RB entry. [~thejas], I don't think it's related to HIVE-12049, because we initially found this issue in Hive-1.2.1. > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, HIVE-15519.2.patch, > data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) precison is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, > precision is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783708#comment-15783708 ] Bharat Viswanadham commented on HIVE-15519: --- [~lirui] Thanks for providing the patch. I have tested the patch, and it is working properly. Precision is returning as expected. > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) precison is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, > precision is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782680#comment-15782680 ] Hive QA commented on HIVE-15519: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12844904/HIVE-15519.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10884 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=233) TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=109) [groupby_grouping_id2.q,input17.q,bucketmapjoin12.q,ppd_gby_join.q,auto_join10.q,ptf_rcfile.q,vectorized_rcfile_columnar.q,vector_elt.q,ppd_join5.q,ppd_join.q,join_filters_overlap.q,join_cond_pushdown_1.q,timestamp_3.q,load_dyn_part6.q,stats_noscan_2.q] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=134) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=93) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] (batchId=93) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct] (batchId=105) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2735/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2735/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2735/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12844904 - PreCommit-HIVE-Build > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: HIVE-15519.1.patch, data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) precison is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, > precision is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782474#comment-15782474 ] Rui Li commented on HIVE-15519: --- Thanks [~bharatviswa] for raising the issue. I think we have a bug in computing subsets of the ResultSet. Specifically, we don't compute the null BitSet for each subset properly. Therefore we may get incorrect results in subsequent batches. I'll upload a patch to demonstrate the idea. > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Assignee: Rui Li >Priority: Critical > Attachments: data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) precison is returning as zero for that column. > Tried with below program. > {code} >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > {code} > Default fetch size is 50. if any column no is under 50 with decimal type, > precision is returning properly, when the column no is greater than 50, > precision is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15781014#comment-15781014 ] Bharat Viswanadham commented on HIVE-15519: --- [~xufeng] Any comments on this issue. > Hive Decimal Type column scale is returning as zero > --- > > Key: HIVE-15519 > URL: https://issues.apache.org/jira/browse/HIVE-15519 > Project: Hive > Issue Type: Bug > Components: Hive, JDBC >Reporter: Bharat Viswanadham >Priority: Critical > Attachments: data_type_test(1).txt > > > Hive decimal type column precision is returning as zero, even though column > has precision set. > Example: col67 decimal(18,2) precison is returning as zero for that column. > Tried with below program. >System.out.println("Opening connection"); > Class.forName("org.apache.hive.jdbc.HiveDriver"); > System.out.println("hi"); >Connection con = > DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default"); > DatabaseMetaData dbMeta = con.getMetaData(); >ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null); > while (rs.next()) { > if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") || > rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){ > System.out.println(rs.getString("COLUMN_NAME") + "\t" + > rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS")); > } >} >rs.close(); >con.close(); > } catch (Exception e) { >e.printStackTrace(); >; > } > Default fetch size is 50. if any column no is under 50 with decimal type > precision is returning properly, when the column no is greater than 50, > precision is returning as zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332)