[jira] [Commented] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260340#comment-14260340 ] Yongzhi Chen commented on HIVE-9201: The four failed tests are not related to the change: The first 2 tests are old failures aged 3 or more. I ran tests org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_remote_script org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority on my own machine, both succeeded. Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Affects Versions: 0.14.0, 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-9201.1.patch Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258326#comment-14258326 ] Yongzhi Chen commented on HIVE-9201: we will escape new line, changing it to '\' 'n' in stream. We also escape \ in the customer's data. So if customer's data is newline, char \ char n, then in the stream it will be \n\\n, then later it will be converted back to newline char \ char n by the method copyAndEscapeStringDataToText Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Affects Versions: 0.14.0, 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-9201.1.patch Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-9201: --- Summary: Lazy functions do not handle newlines and carriage returns properly (was: Lazy functions does not handle newlines and carriage returns properly) Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Reporter: Yongzhi Chen Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9201) Lazy functions does not handle newlines and carriage returns properly
Yongzhi Chen created HIVE-9201: -- Summary: Lazy functions does not handle newlines and carriage returns properly Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Reporter: Yongzhi Chen Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-9201: -- Assignee: Yongzhi Chen Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Reporter: Yongzhi Chen Assignee: Yongzhi Chen Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257688#comment-14257688 ] Yongzhi Chen commented on HIVE-9201: Three rows are returned because hadoop method org.apache.hadoop.mapred.LineRecordReader.readDefaultLine use \r and \n as line terminator. So hive need to process the \r and \n chars before call the method. Map job uses LazyUtils.writeEscaped method to escape special chars (such as control characters). The method just blindly add escape chars before the chars needing escaped. There are two issues: first \r and \n not in the chars needed to be escaped. second, even they are added, they should be escaped differently: for just adding escape char (such as \ ) before them can not solve our problem, the char with value 13 and 10 still in the stream. So we should process the two chars differently. For example replace '\r' with two chars: escape char and char 'r' . These logic can be add in the LazyUtils.writeEscaped method. The processed stream can go through org.apache.hadoop.mapred.LineRecordReader.readDefaultLine method without logic error(such errors as one row becomes 3 rows). Then in LazyString.init method, when we remove the escape chars, we know convert '\' '\r' to char 13. Attach the fix patch. Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Reporter: Yongzhi Chen Assignee: Yongzhi Chen Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-9201: --- Attachment: HIVE-9201.1.patch Need code review. Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-9201.1.patch Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-9201: --- Affects Version/s: 0.14.0 0.13.1 Status: Patch Available (was: Open) Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.14.0 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-9201.1.patch Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214710#comment-14214710 ] Yongzhi Chen commented on HIVE-8872: The test temp_table_windowing_expressions has nothing to do with hbase. For this change is only related to the HBase, so the test failure is not caused by the change. Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8872.1.patch, HIVE-8872.2.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214842#comment-14214842 ] Yongzhi Chen commented on HIVE-8872: [~brocknoland], I created a hbase bug and link to it. HBASE-12492 Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8872.1.patch, HIVE-8872.2.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
Yongzhi Chen created HIVE-8872: -- Summary: Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212472#comment-14212472 ] Yongzhi Chen commented on HIVE-8872: Incorrect data returns only when hive.optimize.ppd=true Fixed by adding synchronized block for method getSplit in HiveHBaseTableInputFormat Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8872: --- Status: Patch Available (was: Open) Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8872.1.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8872: --- Attachment: HIVE-8872.1.patch Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8872.1.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212486#comment-14212486 ] Yongzhi Chen commented on HIVE-8872: created a review request: https://reviews.apache.org/r/28041/ Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8872.1.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8874) Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster
Yongzhi Chen created HIVE-8874: -- Summary: Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster Key: HIVE-8874 URL: https://issues.apache.org/jira/browse/HIVE-8874 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Yongzhi Chen Assignee: Yongzhi Chen A Hive action workflow on a secure cluster, that does an INSERT INTO regular table FROM hbase table as part of its script will reproduce the issue. And it can be reproduced in CDH5.2 cluster. 10309 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Error while configuring input job properties org.apache.hadoop.hive.ql.parse.SemanticException: Error while configuring input job properties at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:94) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9261) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:206) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:988) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1053) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:464) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:474) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:227) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.IllegalStateException: Error while configuring input job properties at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:343) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:279) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:804) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:774) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:241) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:207) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:112) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:83) ... 35 more Caused by: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:124) at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$1.getAuthenticationToken(AuthenticationProtos.java:4267) at
[jira] [Commented] (HIVE-8874) Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster
[ https://issues.apache.org/jira/browse/HIVE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212556#comment-14212556 ] Yongzhi Chen commented on HIVE-8874: The problem caused by addHBaseDelegationToken method in HBaseStorageHandler, attempting to grab a token even if one already exists in the environment. Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster - Key: HIVE-8874 URL: https://issues.apache.org/jira/browse/HIVE-8874 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Yongzhi Chen Assignee: Yongzhi Chen A Hive action workflow on a secure cluster, that does an INSERT INTO regular table FROM hbase table as part of its script will reproduce the issue. And it can be reproduced in CDH5.2 cluster. 10309 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Error while configuring input job properties org.apache.hadoop.hive.ql.parse.SemanticException: Error while configuring input job properties at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:94) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9261) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:206) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:988) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1053) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:464) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:474) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:227) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.IllegalStateException: Error while configuring input job properties at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:343) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:279) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:804) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:774) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:241) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:207) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:112) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:83) ... 35 more Caused by: org.apache.hadoop.hbase.security.AccessDeniedException:
[jira] [Updated] (HIVE-8874) Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster
[ https://issues.apache.org/jira/browse/HIVE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8874: --- Attachment: HIVE-8874.1.patch Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster - Key: HIVE-8874 URL: https://issues.apache.org/jira/browse/HIVE-8874 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8874.1.patch A Hive action workflow on a secure cluster, that does an INSERT INTO regular table FROM hbase table as part of its script will reproduce the issue. And it can be reproduced in CDH5.2 cluster. 10309 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Error while configuring input job properties org.apache.hadoop.hive.ql.parse.SemanticException: Error while configuring input job properties at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:94) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9261) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:206) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:988) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1053) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:464) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:474) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:227) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.IllegalStateException: Error while configuring input job properties at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:343) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:279) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:804) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:774) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:241) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:207) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:112) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:83) ... 35 more Caused by: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients at
[jira] [Updated] (HIVE-8874) Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster
[ https://issues.apache.org/jira/browse/HIVE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8874: --- Status: Patch Available (was: Open) Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster - Key: HIVE-8874 URL: https://issues.apache.org/jira/browse/HIVE-8874 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8874.1.patch A Hive action workflow on a secure cluster, that does an INSERT INTO regular table FROM hbase table as part of its script will reproduce the issue. And it can be reproduced in CDH5.2 cluster. 10309 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Error while configuring input job properties org.apache.hadoop.hive.ql.parse.SemanticException: Error while configuring input job properties at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:94) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9261) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:206) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:988) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1053) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:464) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:474) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:227) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.IllegalStateException: Error while configuring input job properties at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:343) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:279) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:804) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:774) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:241) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:207) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:112) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:83) ... 35 more Caused by: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients at
[jira] [Updated] (HIVE-8872) Hive view of HBase range scan intermittently returns incorrect data.
[ https://issues.apache.org/jira/browse/HIVE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8872: --- Attachment: HIVE-8872.2.patch Hive view of HBase range scan intermittently returns incorrect data. Key: HIVE-8872 URL: https://issues.apache.org/jira/browse/HIVE-8872 Project: Hive Issue Type: Bug Affects Versions: 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8872.1.patch, HIVE-8872.2.patch This need running in cluster: 1. Create a hive external table pointing to a hbase table. 2. Create views to the hive table(for example 30 views), each view looks like following with different range check: CREATE VIEW hview_nn AS SELECT * FROM hivehbasetable WHERE (pk ='pk_nn_0' AND pk = pk_nn_A') 3. Create same number of hive new tables as views. 4. then runs several queries in parallel (30 threads): INSERT OVERWRITE TABLE hivenewtable_nn SELECT * FROM hview_nn //nn is from 01 to 30 5 After insert, check the hivenewtables, some values are not right. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8874) Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster
[ https://issues.apache.org/jira/browse/HIVE-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212648#comment-14212648 ] Yongzhi Chen commented on HIVE-8874: submitted review: https://reviews.apache.org/r/28047/ Error Accessing HBase from Impala via Oozie on Kerberos 5.0.1 cluster - Key: HIVE-8874 URL: https://issues.apache.org/jira/browse/HIVE-8874 Project: Hive Issue Type: Bug Components: HBase Handler Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8874.1.patch A Hive action workflow on a secure cluster, that does an INSERT INTO regular table FROM hbase table as part of its script will reproduce the issue. And it can be reproduced in CDH5.2 cluster. 10309 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException Error while configuring input job properties org.apache.hadoop.hive.ql.parse.SemanticException: Error while configuring input job properties at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:94) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9261) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:206) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:434) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:332) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:988) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1053) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:464) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:474) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:227) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.IllegalStateException: Error while configuring input job properties at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:343) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:279) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:804) at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:774) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:241) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:207) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:112) at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:83) ... 35 more Caused by: org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only
[jira] [Commented] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173744#comment-14173744
]
Yongzhi Chen commented on HIVE-8448:
[~szehon] and [~jdere], I checked the failure tests. 2 of them are new ones,
but it related to login so it should not be caused by the change.
The rest of the failures are aged more than 2 days, and do not have union
operations in the queries. So I think the all the test failures
are not related to the change.
Thanks.
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.4.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: HIVE-8448.1.patch
Need code review.
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Affects Version/s: 0.13.1
Status: Patch Available (was: Open)
when SemanticAnalyzer check union plan, it use getCommonClassForUnionAll, but
when evaluate union operator it uses getCommonClass. The inconsistency cause
some queries with multiple union all on date type column can pass the analyzer
but fail with HiveException: Incompatible types at execute time.
Fixed by use new updateForUnionAll method which use getCommonClassForUnionAll
to update column for union operator.
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: HIVE-8448.2.patch
fixes after review
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch, HIVE-8448.2.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172726#comment-14172726
]
Yongzhi Chen commented on HIVE-8448:
Thanks [~szehon], I attached the new patch following your review advice, I also
submit a review request for the jira:
https://reviews.apache.org/r/26763/
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch, HIVE-8448.2.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Status: Open (was: Patch Available)
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch, HIVE-8448.2.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: HIVE-8448.3.patch
need review
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch, HIVE-8448.2.patch, HIVE-8448.3.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Status: Patch Available (was: Open)
Fixed code style after review
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.1.patch, HIVE-8448.2.patch, HIVE-8448.3.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: (was: HIVE-8448.3.patch)
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: (was: HIVE-8448.2.patch)
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: (was: HIVE-8448.1.patch)
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Attachment: HIVE-8448.4.patch
need code review
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.4.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Status: Open (was: Patch Available)
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-8448:
---
Status: Patch Available (was: Open)
Fixed code style after review
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Affects Versions: 0.13.1
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
Attachments: HIVE-8448.4.patch
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8448) Union All might not work due to the type conversion issue
[
https://issues.apache.org/jira/browse/HIVE-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen reassigned HIVE-8448:
--
Assignee: Yongzhi Chen (was: Chaoyu Tang)
Union All might not work due to the type conversion issue
-
Key: HIVE-8448
URL: https://issues.apache.org/jira/browse/HIVE-8448
Project: Hive
Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Yongzhi Chen
Priority: Minor
create table t1 (val date);
insert overwrite table t1 select '2014-10-10' from src limit 1;
create table t2 (val varchar(10));
insert overwrite table t2 select '2014-10-10' from src limit 1;
==
Query:
select t.val from
(select val from t1
union all
select val from t1
union all
select val from t2
union all
select val from t1) t;
==
Will throw exception:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible
types for union operator
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:86)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:65)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:420)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:380)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:133)
... 22 more
{code}
It was because at this query parse step, getCommonClassForUnionAll is used,
but at execution getCommonClass is used. They are not used consistently in
union. The later one does not support the implicit conversion from date to
string, which is the problem cause.
The change might be simple to fix this particular union issue but I noticed
that there are three versions of getCommonClass: getCommonClass,
getCommonClassForComparison, getCommonClassForUnionAll, and wonder if they
need to be cleaned and refactored.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8182) beeline fails when executing multiple-line queries with trailing spaces
[
https://issues.apache.org/jira/browse/HIVE-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151701#comment-14151701
]
Yongzhi Chen commented on HIVE-8182:
Is trimming very line a good idea? I think to be consistent with single line
case, maybe only trim last line is better choice.
My suggestion is add
line = line.trim();
before
if (line.endsWith(;)) { line = line.substring(0, line.length() - 1); }
beeline fails when executing multiple-line queries with trailing spaces
---
Key: HIVE-8182
URL: https://issues.apache.org/jira/browse/HIVE-8182
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0, 0.13.1
Reporter: Yongzhi Chen
Assignee: Sergio Peña
Fix For: 0.14.0
Attachments: HIVE-8181.1.patch, HIVE-8182.1.patch
As title indicates, when executing a multi-line query with trailing spaces,
beeline reports syntax error:
Error: Error while compiling statement: FAILED: ParseException line 1:76
extraneous input ';' expecting EOF near 'EOF' (state=42000,code=4)
If put this query in one single line, beeline succeeds to execute it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8189) A select statement with a subquery is failing with HBaseSerde
[ https://issues.apache.org/jira/browse/HIVE-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146409#comment-14146409 ] Yongzhi Chen commented on HIVE-8189: Add the patch to the review board: https://reviews.apache.org/r/25993/ A select statement with a subquery is failing with HBaseSerde - Key: HIVE-8189 URL: https://issues.apache.org/jira/browse/HIVE-8189 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.12.0, 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-8189.1.patch, hbase_ppd_join.q Hive tables in the query are hbase tables, and the subquery is a join statement. When set hive.optimize.ppd=true; and set hive.auto.convert.join=false; The query does not return data. While hive.optimize.ppd=true and hive.auto.convert.join=true return values back. See attached query file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8189) A select statement with a subquery is failing.
Yongzhi Chen created HIVE-8189: -- Summary: A select statement with a subquery is failing. Key: HIVE-8189 URL: https://issues.apache.org/jira/browse/HIVE-8189 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.12.0 Reporter: Yongzhi Chen Hive tables in the query are hbase tables, and the subquery is a join statement. When set hive.optimize.ppd=true; and set hive.auto.convert.join=false; The query does not return data. While hive.optimize.ppd=true and hive.auto.convert.join=true return values back. See attached query file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8189) A select statement with a subquery is failing.
[ https://issues.apache.org/jira/browse/HIVE-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8189: --- Attachment: hbase_ppd_join.q The query can reproduce the issue. A select statement with a subquery is failing. --- Key: HIVE-8189 URL: https://issues.apache.org/jira/browse/HIVE-8189 Project: Hive Issue Type: Bug Affects Versions: 0.12.0, 0.13.1 Reporter: Yongzhi Chen Attachments: hbase_ppd_join.q Hive tables in the query are hbase tables, and the subquery is a join statement. When set hive.optimize.ppd=true; and set hive.auto.convert.join=false; The query does not return data. While hive.optimize.ppd=true and hive.auto.convert.join=true return values back. See attached query file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8189) A select statement with a subquery is failing with HBaseSerde
[ https://issues.apache.org/jira/browse/HIVE-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8189: --- Attachment: HIVE-8189.1.patch Need code review. The patch is for trunk, Problem: 1) Predicates are used to filter out regions in hbase which do not need to 2) The predicates are sticking around in the jobConf from table with predic Solution: removing the predicates before we reset them we remove this bad stat A select statement with a subquery is failing with HBaseSerde - Key: HIVE-8189 URL: https://issues.apache.org/jira/browse/HIVE-8189 Project: Hive Issue Type: Bug Affects Versions: 0.12.0, 0.13.1 Reporter: Yongzhi Chen Attachments: HIVE-8189.1.patch, hbase_ppd_join.q Hive tables in the query are hbase tables, and the subquery is a join statement. When set hive.optimize.ppd=true; and set hive.auto.convert.join=false; The query does not return data. While hive.optimize.ppd=true and hive.auto.convert.join=true return values back. See attached query file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8189) A select statement with a subquery is failing with HBaseSerde
[ https://issues.apache.org/jira/browse/HIVE-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-8189: --- Status: Patch Available (was: Open) need code review A select statement with a subquery is failing with HBaseSerde - Key: HIVE-8189 URL: https://issues.apache.org/jira/browse/HIVE-8189 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.12.0 Reporter: Yongzhi Chen Attachments: HIVE-8189.1.patch, hbase_ppd_join.q Hive tables in the query are hbase tables, and the subquery is a join statement. When set hive.optimize.ppd=true; and set hive.auto.convert.join=false; The query does not return data. While hive.optimize.ppd=true and hive.auto.convert.join=true return values back. See attached query file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8182) beeline fails when executing multiple-line queries with trailing spaces
Yongzhi Chen created HIVE-8182: -- Summary: beeline fails when executing multiple-line queries with trailing spaces Key: HIVE-8182 URL: https://issues.apache.org/jira/browse/HIVE-8182 Project: Hive Issue Type: Bug Affects Versions: 0.13.1, 0.12.0 Reporter: Yongzhi Chen As title indicates, when executing a multi-line query with trailing spaces, beeline reports syntax error: Error: Error while compiling statement: FAILED: ParseException line 1:76 extraneous input ';' expecting EOF near 'EOF' (state=42000,code=4) If put this query in one single line, beeline succeeds to execute it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
