[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15267887#comment-15267887
]
WangMeng commented on HIVE-11880:
-
Thanks [~aihuaxu].
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch, HIVE-11880.04.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This following query (with filter "type incompatible column 'o_custkey' ")
> will fail with java.lang.IndexOutOfBoundsException :
> {code}
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> CREATE TABLE `orders`(
> `o_orderkey` int,
> `o_custkey` int,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string);
> SELECT o_orderkey
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse when create external database
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033397#comment-15033397 ] WangMeng commented on HIVE-12231: - [~thejas] Many thanks for your detailed explaination. > StorageBasedAuthorization requires write permission of default Warehouse when > create external database > -- > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization, I set > external location to create database. However, it will also check write > permission of default warehouse "/user/hive/warehouse" : > > CREATE DATABASE test LOCATION '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse PATH when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' "
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng reassigned HIVE-12231: --- Assignee: WangMeng > StorageBasedAuthorization requires write permission of default Warehouse PATH > when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' " > > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > Please look at the stacktrace, when enabled StorageBasedAuthorization , I > set external Location of creating database, it will also check write > permission of default Warehouse "/user/hive/warehouse" : > > create database test location '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' "
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-12231: Summary: StorageBasedAuthorization requires write permission of default Warehouse when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' " (was: StorageBasedAuthorization requires write permission of default Warehouse PATH when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' ") > StorageBasedAuthorization requires write permission of default Warehouse when > execute "CREATE DATABASE $Name LOCATION '$ExternalPath' " > --- > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization , I > set external Location of creating database, it will also check write > permission of default Warehouse "/user/hive/warehouse" : > > create database test location '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' "
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-12231: Description: Please look at the stacktrace, when enabled StorageBasedAuthorization, I set external location to create database. However, it will also check write permission of default warehouse "/user/hive/warehouse" : > CREATE DATABASE test LOCATION '/tmp/wangmeng/test' ; Error: Error while compiling statement: FAILED: HiveException java.security.AccessControlException: Permission denied: user=wangmeng, access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) was: Please look at the stacktrace, when enabled StorageBasedAuthorization , I set external Location of creating database, it will also check write permission of default Warehouse "/user/hive/warehouse" : > create database test location '/tmp/wangmeng/test' ; Error: Error while compiling statement: FAILED: HiveException java.security.AccessControlException: Permission denied: user=wangmeng, access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) > StorageBasedAuthorization requires write permission of default Warehouse when > execute "CREATE DATABASE $Name LOCATION '$ExternalPath' " > --- > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization, I set > external location to create database. However, it will also check write > permission of default warehouse "/user/hive/warehouse" : > > CREATE DATABASE test LOCATION '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse PATH when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' "
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-12231: Attachment: HIVE-12231.01.patch > StorageBasedAuthorization requires write permission of default Warehouse PATH > when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' " > > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization , I > set external Location of creating database, it will also check write > permission of default Warehouse "/user/hive/warehouse" : > > create database test location '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse when create external database
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-12231: Summary: StorageBasedAuthorization requires write permission of default Warehouse when create external database (was: StorageBasedAuthorization requires write permission of default Warehouse when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' ") > StorageBasedAuthorization requires write permission of default Warehouse when > create external database > -- > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization, I set > external location to create database. However, it will also check write > permission of default warehouse "/user/hive/warehouse" : > > CREATE DATABASE test LOCATION '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse when create external database
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970294#comment-14970294 ] WangMeng commented on HIVE-12231: - [~thejas] Thanks for your detailed clarification. However, if user has write permission on the external path which is used in creating external database, this check mechanism maybe a little inappropriate. User should own the right to create database in path which user owns write permission such as his own dir. How about checking the destination path of external database directly? In production environment, creating external database in permission dir is relative common. Thanks. > StorageBasedAuthorization requires write permission of default Warehouse when > create external database > -- > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization, I set > external location to create database. However, it will also check write > permission of default warehouse "/user/hive/warehouse" : > > CREATE DATABASE test LOCATION '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12232) Create external table failed when enabled StorageBasedAuthorization
[ https://issues.apache.org/jira/browse/HIVE-12232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-12232: Attachment: HIVE-12232.01.patch > Create external table failed when enabled StorageBasedAuthorization > --- > > Key: HIVE-12232 > URL: https://issues.apache.org/jira/browse/HIVE-12232 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12232.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization, > creating external table will failed with write permission about the default > warehouse path "/user/hive/warehouse": > > CREATE EXTERNAL TABLE test(id int) LOCATION '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12231) StorageBasedAuthorization requires write permission of default Warehouse when execute "CREATE DATABASE $Name LOCATION '$ExternalPath' "
[ https://issues.apache.org/jira/browse/HIVE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968670#comment-14968670 ] WangMeng commented on HIVE-12231: - [~xuefuz] [~thejas] : Please take a look about this small bug. Thanks. > StorageBasedAuthorization requires write permission of default Warehouse when > execute "CREATE DATABASE $Name LOCATION '$ExternalPath' " > --- > > Key: HIVE-12231 > URL: https://issues.apache.org/jira/browse/HIVE-12231 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-12231.01.patch > > > Please look at the stacktrace, when enabled StorageBasedAuthorization, I set > external location to create database. However, it will also check write > permission of default warehouse "/user/hive/warehouse" : > > CREATE DATABASE test LOCATION '/tmp/wangmeng/test' ; > Error: Error while compiling statement: FAILED: HiveException > java.security.AccessControlException: Permission denied: user=wangmeng, > access=WRITE, inode="/user/hive/warehouse":hive:hive:drwxr-x--t > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:255) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:236) > at > org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948023#comment-14948023
]
WangMeng commented on HIVE-11880:
-
[~jpullokkaran]
I have added Review Board in Issue Links.
The execution engine is MR( I don't use TEZ) .You can use
TPC-H(http://www.tpc.org/tpch/) to reproduce this Jira according to the
descriptition above. Thanks.
Different from HIVE-11919 , only when occurs "union type mismatch" and one
of the type mismatch column is constant and this type mismatch column is
filter column, then UNION ALL will throws HIVE-11880．
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch, HIVE-11880.04.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935024#comment-14935024
]
WangMeng commented on HIVE-11880:
-
[~ashutoshc] [~jpullokkaran] I have published this patch on Review Board:
https://reviews.apache.org/r/38805/
Please help review it . Thanks.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch, HIVE-11880.04.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Attachment: HIVE-11880.04.patch
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch, HIVE-11880.04.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933230#comment-14933230
]
WangMeng commented on HIVE-11880:
-
[~jpullokkaran] this exception is as follows:
Caused by: java.lang.RuntimeException: Map operator initialization failed
..
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.exec.UnionOperator.initializeOp(UnionOperator.java:83).
.
And I uploaded a new patch:HIVE-11880.04.patch, please check it again. Thanks.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Attachment: HIVE-11880.03.patch
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907667#comment-14907667
]
WangMeng commented on HIVE-11880:
-
[~xuefuz]I have rebased it and uploaded a new patch.
[~ashutoshc] I tried the patch of HIVE-11919 again after I rebased, it can not
fix this bug also.
Please check it again . Thanks.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch,
> HIVE-11880.03.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906211#comment-14906211
]
WangMeng commented on HIVE-11880:
-
[~ashutoshc] [~xuefuz]
I feel it will wait a very long time until [~hiveqa] runs tests in recent
period.
Does the QA queue have been blocked ? Thanks.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906213#comment-14906213 ] WangMeng commented on HIVE-11149: - [~sershe] I feel it will wait a very long time until [~hiveqa] runs tests in recent period. Does the QA queue have been blocked ? Thanks. > Fix issue with sometimes HashMap in PerfLogger.java hangs > -- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, sometimes the HashMap in PerfLogger.java > will casue massive Java Processes hang and cost large amounts of > unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Affects Version/s: (was: 1.2.0) > Fix issue with sometimes HashMap in PerfLogger.java hangs > -- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, sometimes the HashMap in PerfLogger.java > will casue massive Java Processes hang and cost large amounts of > unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Affects Version/s: 1.2.1 > Fix issue with sometimes HashMap in PerfLogger.java hangs > -- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, sometimes the HashMap in PerfLogger.java > will casue massive Java Processes hang and cost large amounts of > unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Description:
For UNION ALL , when an union operator is constant column (such as '0L',
BIGINT Type) and its corresponding column has incompatible type (such as INT
type).
Query with filter condition on type incompatible column on this UNION ALL
will cause IndexOutOfBoundsException.
Such as TPC-H table "orders",in the following query:
Type of 'orders'.'o_custkey' is INT normally, while the type of
corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
This query (with filter "type incompatible column 'o_custkey' ") will fail
with java.lang.IndexOutOfBoundsException :
{code}
SELECT Count(1)
FROM (
SELECT `o_orderkey` ,
`o_custkey`
FROM `orders`
UNION ALL
SELECT `o_orderkey`,
0L AS `o_custkey`
FROM `orders`) `oo`
WHERE o_custkey<10 limit 4 ;
{code}
When
{code}
set hive.ppd.remove.duplicatefilters=true
{code}
was:
For UNION ALL , when an union operator is constant(column 'a' such as '0L')
and it has incompatible type with the corresponding column A(INT type).
Query with filter condition on type incompatible column on this UNION ALL
will cause IndexOutOfBoundsException.
Such as TPC-H table "orders",in the fllowing query:
Type of 'o_custkey' is INT normally, while the type of corresponding column
constant "0" is BIGINT( `0L AS `o_custkey` ).
This query (with filter " incompatible column 'o_custkey' ") will fail with
java.lang.IndexOutOfBoundsException :
{code}
SELECT Count(1)
FROM (
SELECT `o_orderkey` ,
`o_custkey`
FROM `orders`
UNION ALL
SELECT `o_orderkey`,
0L AS `o_custkey`
FROM `rcfileorders`) `oo`
WHERE o_custkey<10 limit 4 ;
{code}
When
{code}
set hive.ppd.remove.duplicatefilters=true
{code}
the fllowing query (with filter " incompatible column 'o_custkey' ") will
fail with java.lang.IndexOutOfBoundsException:
{code}
select count(1) from view_orders where o_custkey<10
{code}
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905982#comment-14905982
]
WangMeng commented on HIVE-11880:
-
[~ashutoshc] I tried the patch of HIVE-11919, it can not fix this bug.
The two bugs are different.I think HIVE-11919 focuses on type mismatch and
HIVE-11880 focuses on filter bug when hive.ppd.remove.duplicatefilters=true
> IndexOutOfBoundsException when query with filter condition on type
> incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
> --
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905983#comment-14905983
]
WangMeng commented on HIVE-11880:
-
[~ashutoshc] I tried the patch of HIVE-11919, it can not fix this bug.
The two bugs are different.I think HIVE-11919 focuses on type mismatch and
HIVE-11880 focuses on filter bug when hive.ppd.remove.duplicatefilters=true
> IndexOutOfBoundsException when query with filter condition on type
> incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
> --
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Summary: filter bug of UNION ALL when
hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible
column (was: IndexOutOfBoundsException when query with filter condition on
type incompatible column of UNION ALL when
hive.ppd.remove.duplicatefilters=true)
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Attachment: HIVE-11880.02.patch
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and filter condition is type incompatible column
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906092#comment-14906092
]
WangMeng commented on HIVE-11880:
-
[~ashutoshc] Hi , I uploaded a new patch to add notes for this issue and
alterd jira title and description.
Please check it again . Thanks.
> filter bug of UNION ALL when hive.ppd.remove.duplicatefilters=true and
> filter condition is type incompatible column
> -
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch, HIVE-11880.02.patch
>
>
>For UNION ALL , when an union operator is constant column (such as '0L',
> BIGINT Type) and its corresponding column has incompatible type (such as INT
> type).
> Query with filter condition on type incompatible column on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders",in the following query:
> Type of 'orders'.'o_custkey' is INT normally, while the type of
> corresponding constant column "0" is BIGINT( `0L AS `o_custkey` ).
> This query (with filter "type incompatible column 'o_custkey' ") will fail
> with java.lang.IndexOutOfBoundsException :
> {code}
> SELECT Count(1)
> FROM (
> SELECT `o_orderkey` ,
> `o_custkey`
> FROM `orders`
> UNION ALL
> SELECT `o_orderkey`,
> 0L AS `o_custkey`
> FROM `orders`) `oo`
> WHERE o_custkey<10 limit 4 ;
> {code}
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Component/s: (was: Query Processor)
Logical Optimizer
> IndexOutOfBoundsException when query with filter condition on type
> incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
> --
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903977#comment-14903977
]
WangMeng commented on HIVE-11880:
-
[~hiveqa]
> IndexOutOfBoundsException when query with filter condition on type
> incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
> --
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when execute query with filter condition on type incompatible column on data(generated by UNION ALL with an union column is constant and it h
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Attachment: HIVE-11880.01.patch >IndexOutOfBoundsException when execute query with filter condition on type > incompatible column on data(generated by UNION ALL with an union column is > constant and it has incompatible type with corresponding column) > -- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11880.01.patch > > >For Hive UNION ALL , when a union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. The > query with filter condition on type incompatible column a on this UNION-ALL > results will cause IndexOutOfBoundsException > Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select > `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , > `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select > `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from > `tpch270g`.`textfileorders`) `oo`. >Type of 'o_custkey' is INT normally, while the type of corresponding > column constant "0" is BIGINT. >Then the fllowing query(with filter incompatible column 0_custkey) will > fail with java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Summary: IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL (was:IndexOutOfBoundsException when execute query with filter condition on type incompatible column on data(generated by UNION ALL with an union column is constant and it has incompatible type with corresponding column) ) > IndexOutOfBoundsException when query with filter condition on type > incompatible column of UNION ALL > --- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11880.01.patch > > >For Hive UNION ALL , when a union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. The > query with filter condition on type incompatible column a on this UNION-ALL > results will cause IndexOutOfBoundsException > Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select > `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , > `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select > `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from > `tpch270g`.`textfileorders`) `oo`. >Type of 'o_custkey' is INT normally, while the type of corresponding > column constant "0" is BIGINT. >Then the fllowing query(with filter incompatible column 0_custkey) will > fail with java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Description: For Hive UNION ALL , when an union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A(INT Type). Query with filter condition on type incompatible column a on this UNION ALL will cause IndexOutOfBoundsException. Such as TPC-H table "orders", we create View by : CREATE VIEW `view_orders` AS SELECT `oo`.`o_orderkey` , `oo`.`o_custkey` FROM ( SELECT `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` FROM `rcfileorders` UNION ALL SELECT `orcfileorders`.`o_orderkey` , 0L AS `o_custkey` FROM `textfileorders`) `oo`. In view_orders , type of 'o_custkey' is INT normally, while the type of corresponding column constant "0" is BIGINT. When hive.ppd.remove.duplicatefilters=true, the fllowing query (with filter " incompatible column 'o_custkey' ") will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. was: For Hive UNION ALL , when an union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. Query with filter condition on type incompatible column a on this UNION ALL will cause IndexOutOfBoundsException. Such as TPC-H table "orders", we create View by : CREATE VIEW `view_orders` AS SELECT `oo`.`o_orderkey` , `oo`.`o_custkey` FROM ( SELECT `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` FROM `rcfileorders` UNION ALL SELECT `orcfileorders`.`o_orderkey` , 0L AS `o_custkey` FROM `textfileorders`) `oo`. In view_orders , type of 'o_custkey' is INT normally, while the type of corresponding column constant "0" is BIGINT. Then the fllowing query(with filter " incompatible column 'o_custkey' ") will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. > IndexOutOfBoundsException when query with filter condition on type > incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true > -- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11880.01.patch > > >For Hive UNION ALL , when an union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A(INT > Type). > Query with filter condition on type incompatible column a on this UNION ALL > will cause IndexOutOfBoundsException. > Such as TPC-H table "orders", we create View by : > CREATE VIEW `view_orders` AS > SELECT `oo`.`o_orderkey` , > `oo`.`o_custkey` > FROM ( > SELECT `orders`.`o_orderkey` , >`rcfileorders`.`o_custkey` > FROM `rcfileorders` > UNION ALL > SELECT `orcfileorders`.`o_orderkey` , > 0L AS `o_custkey` > FROM `textfileorders`) `oo`. > In view_orders , type of 'o_custkey' is INT normally, while the type of > corresponding column constant "0" is BIGINT. > When hive.ppd.remove.duplicatefilters=true, the fllowing query (with filter > " incompatible column 'o_custkey' ") will fail with > java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Summary: IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true (was: IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL) > IndexOutOfBoundsException when query with filter condition on type > incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true > -- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11880.01.patch > > >For Hive UNION ALL , when an union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. > Query with filter condition on type incompatible column a on this UNION ALL > will cause IndexOutOfBoundsException. > Such as TPC-H table "orders", we create View by : > CREATE VIEW `view_orders` AS > SELECT `oo`.`o_orderkey` , > `oo`.`o_custkey` > FROM ( > SELECT `orders`.`o_orderkey` , >`rcfileorders`.`o_custkey` > FROM `rcfileorders` > UNION ALL > SELECT `orcfileorders`.`o_orderkey` , > 0L AS `o_custkey` > FROM `textfileorders`) `oo`. > In view_orders , type of 'o_custkey' is INT normally, while the type of > corresponding column constant "0" is BIGINT. > Then the fllowing query(with filter " incompatible column 'o_custkey' ") > will fail with java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Description: For Hive UNION ALL , when an union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. Query with filter condition on type incompatible column a on this UNION ALL will cause IndexOutOfBoundsException. Such as TPC-H table "orders", we create View by : CREATE VIEW `view_orders` AS SELECT `oo`.`o_orderkey` , `oo`.`o_custkey` FROM ( SELECT `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` FROM `rcfileorders` UNION ALL SELECT `orcfileorders`.`o_orderkey` , 0L AS `o_custkey` FROM `textfileorders`) `oo`. In view_orders , type of 'o_custkey' is INT normally, while the type of corresponding column constant "0" is BIGINT. Then the fllowing query(with filter " incompatible column 'o_custkey' ") will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. was: For Hive UNION ALL , when a union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of 'o_custkey' is INT normally, while the type of corresponding column constant "0" is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. > IndexOutOfBoundsException when query with filter condition on type > incompatible column of UNION ALL > --- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11880.01.patch > > >For Hive UNION ALL , when an union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. > Query with filter condition on type incompatible column a on this UNION ALL > will cause IndexOutOfBoundsException. > Such as TPC-H table "orders", we create View by : > CREATE VIEW `view_orders` AS > SELECT `oo`.`o_orderkey` , > `oo`.`o_custkey` > FROM ( > SELECT `orders`.`o_orderkey` , >`rcfileorders`.`o_custkey` > FROM `rcfileorders` > UNION ALL > SELECT `orcfileorders`.`o_orderkey` , > 0L AS `o_custkey` > FROM `textfileorders`) `oo`. > In view_orders , type of 'o_custkey' is INT normally, while the type of > corresponding column constant "0" is BIGINT. > Then the fllowing query(with filter " incompatible column 'o_custkey' ") > will fail with java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903975#comment-14903975
]
WangMeng commented on HIVE-11880:
-
[~xuefuz]
I uploaded a new patch for this issue.
Please check it. Thanks.
> IndexOutOfBoundsException when query with filter condition on type
> incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
> --
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when query with filter condition on type incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
[
https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangMeng updated HIVE-11880:
Description:
For Hive UNION ALL , when an union column is constant(column a such as '0L')
and it has incompatible type with the corresponding column A(INT Type).
Query with filter condition on type incompatible column a on this UNION ALL
will cause IndexOutOfBoundsException.
Such as TPC-H table "orders", we create View by :
{code}
CREATE VIEW `view_orders` AS
SELECT `oo`.`o_orderkey` ,
`oo`.`o_custkey`
FROM (
SELECT`o_orderkey` , `0L AS `o_custkey`
FROM `rcfileorders`
UNION ALL
SELECT `o_orderkey` ,`o_custkey`
FROM `textfileorders`) `oo`.
{code}
In VIEW view_orders , type of 'o_custkey' is INT normally, while the type of
corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
When
{code}
set hive.ppd.remove.duplicatefilters=true
{code}
the fllowing query (with filter " incompatible column 'o_custkey' ") will
fail with java.lang.IndexOutOfBoundsException:
{code}
select count(1) from view_orders where o_custkey<10
{code}
was:
For Hive UNION ALL , when an union column is constant(column a such as '0L')
and it has incompatible type with the corresponding column A(INT Type).
Query with filter condition on type incompatible column a on this UNION ALL
will cause IndexOutOfBoundsException.
Such as TPC-H table "orders", we create View by :
CREATE VIEW `view_orders` AS
SELECT `oo`.`o_orderkey` ,
`oo`.`o_custkey`
FROM (
SELECT `orders`.`o_orderkey` ,
`rcfileorders`.`o_custkey`
FROM `rcfileorders`
UNION ALL
SELECT `orcfileorders`.`o_orderkey` ,
0L AS `o_custkey`
FROM `textfileorders`) `oo`.
In view_orders , type of 'o_custkey' is INT normally, while the type of
corresponding column constant "0" is BIGINT.
When hive.ppd.remove.duplicatefilters=true, the fllowing query (with filter "
incompatible column 'o_custkey' ") will fail with
java.lang.IndexOutOfBoundsException:
'select count(1) from view_orders where o_custkey<10 '.
> IndexOutOfBoundsException when query with filter condition on type
> incompatible column of UNION ALL when hive.ppd.remove.duplicatefilters=true
> --
>
> Key: HIVE-11880
> URL: https://issues.apache.org/jira/browse/HIVE-11880
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11880.01.patch
>
>
>For Hive UNION ALL , when an union column is constant(column a such as
> '0L') and it has incompatible type with the corresponding column A(INT
> Type).
> Query with filter condition on type incompatible column a on this UNION ALL
> will cause IndexOutOfBoundsException.
> Such as TPC-H table "orders", we create View by :
> {code}
> CREATE VIEW `view_orders` AS
> SELECT `oo`.`o_orderkey` ,
> `oo`.`o_custkey`
> FROM (
> SELECT`o_orderkey` , `0L AS `o_custkey`
> FROM `rcfileorders`
> UNION ALL
> SELECT `o_orderkey` ,`o_custkey`
> FROM `textfileorders`) `oo`.
> {code}
> In VIEW view_orders , type of 'o_custkey' is INT normally, while the type
> of corresponding column constant "0" is BIGINT( `0L AS `o_custkey` ).
> When
> {code}
> set hive.ppd.remove.duplicatefilters=true
> {code}
> the fllowing query (with filter " incompatible column 'o_custkey' ") will
> fail with java.lang.IndexOutOfBoundsException:
> {code}
> select count(1) from view_orders where o_custkey<10
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900430#comment-14900430 ] WangMeng commented on HIVE-11149: - [~hiveqa] > Fix issue with sometimes HashMap in PerfLogger.java hangs > -- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, sometimes the HashMap in PerfLogger.java > will casue massive Java Processes hang and cost large amounts of > unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Attachment: HIVE-11149.03.patch > Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in > Multi-thread environment > --- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the Thread unsafe Class HashMap in > PerfLogger.java will casue massive Java Processes hang and cost large > amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Summary: Fix issue with sometimes HashMap in PerfLogger.java hangs (was: Fix issue with HashMap in PerfLogger.java hangs ) > Fix issue with sometimes HashMap in PerfLogger.java hangs > -- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the HashMap in PerfLogger.java will casue > massive Java Processes hang and cost large amounts of unnecessary CPU and > Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Description: In Multi-thread environment, sometimes the HashMap in PerfLogger.java will casue massive Java Processes hang and cost large amounts of unnecessary CPU and Memory. (was: In Multi-thread environment, the HashMap in PerfLogger.java will casue massive Java Processes hang and cost large amounts of unnecessary CPU and Memory.) > Fix issue with sometimes HashMap in PerfLogger.java hangs > -- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, sometimes the HashMap in PerfLogger.java > will casue massive Java Processes hang and cost large amounts of > unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Description: In Multi-thread environment, the HashMap in PerfLogger.java will casue massive Java Processes hang and cost large amounts of unnecessary CPU and Memory. (was: In Multi-thread environment, the Thread unsafe Class HashMap in PerfLogger.java will casue massive Java Processes hang and cost large amounts of unnecessary CPU and Memory.) > Fix issue with HashMap in PerfLogger.java hangs > > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the HashMap in PerfLogger.java will casue > massive Java Processes hang and cost large amounts of unnecessary CPU and > Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with HashMap in PerfLogger.java hangs
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Summary: Fix issue with HashMap in PerfLogger.java hangs (was: Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment) > Fix issue with HashMap in PerfLogger.java hangs > > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the Thread unsafe Class HashMap in > PerfLogger.java will casue massive Java Processes hang and cost large > amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805164#comment-14805164 ] WangMeng commented on HIVE-11149: - [~xuefuz] [~sershe] I uploaded a new patch for this issue. Please check it again. Thanks. > Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in > Multi-thread environment > --- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the Thread unsafe Class HashMap in > PerfLogger.java will casue massive Java Processes hang and cost large > amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11149) Fix issue with sometimes HashMap in PerfLogger.java hangs
[
https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876805#comment-14876805
]
WangMeng commented on HIVE-11149:
-
[~sershe] According to my understanding,it may not occur the problem you
mentioned above.
According to code logic in Hive,each thread accessing the session owns the
initial default value"resetPerfLogger == true" when it begin initialization and
then it registered as ThreadLocal thread by perfLogger.set() method.
Do I understand your problem? Thanks.
//Reset the perf logger in Driver.runInternal()
PerfLogger perfLogger = PerfLogger.getPerfLogger(true);
//PerfLogger.java
public static PerfLogger getPerfLogger(boolean resetPerfLogger) {
if (SessionState.get() == null) {
if (perfLogger.get() == null || resetPerfLogger) {
perfLogger.set(new PerfLogger());
}
return perfLogger.get();
} else {
perfLogger.set(SessionState.get().getPerfLogger(resetPerfLogger));
return perfLogger.get();
}
}
> Fix issue with sometimes HashMap in PerfLogger.java hangs
> --
>
> Key: HIVE-11149
> URL: https://issues.apache.org/jira/browse/HIVE-11149
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 1.2.0
>Reporter: WangMeng
>Assignee: WangMeng
> Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch,
> HIVE-11149.03.patch
>
>
> In Multi-thread environment, sometimes the HashMap in PerfLogger.java
> will casue massive Java Processes hang and cost large amounts of
> unnecessary CPU and Memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when execute query with filter condition on type incompatible column(A) on data(composed by UNION ALL when a union column is constant and it
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Description: For Hive UNION ALL , when a union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of "o_custkey" is INT, the type of corresponding constant "0" is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail with java.lang.IndexOutOfBoundsException: "select count(1) from view_orders where o_custkey<10 ". was: For Hive UNION ALL , when a union column is constant(column a) and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException such as TPC-H table orders: CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of "o_custkey" is INT, the type of corresponding constant column 0 is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail: select count(1) from view_orders where o_custkey<10 with java.lang.IndexOutOfBoundsException. >IndexOutOfBoundsException when execute query with filter condition on type > incompatible column(A) on data(composed by UNION ALL when a union column is > constant and it has incompatible type with corresponding column) > > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > >For Hive UNION ALL , when a union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. The > query with filter condition on type incompatible column a on this UNION-ALL > results will cause IndexOutOfBoundsException > Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select > `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , > `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select > `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from > `tpch270g`.`textfileorders`) `oo`. >Type of "o_custkey" is INT, the type of corresponding constant "0" is > BIGINT. >Then the fllowing query(with filter incompatible column 0_custkey) will > fail with java.lang.IndexOutOfBoundsException: > "select count(1) from view_orders where o_custkey<10 ". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when execute query with filter condition on type incompatible column on data(generated by UNION ALL with an union column is constant and it h
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Summary:IndexOutOfBoundsException when execute query with filter condition on type incompatible column on data(generated by UNION ALL with an union column is constant and it has incompatible type with corresponding column) (was:IndexOutOfBoundsException when execute query with filter condition on type incompatible column(A) on data(composed by UNION ALL when a union column is constant and it has incompatible type with corresponding column) ) >IndexOutOfBoundsException when execute query with filter condition on type > incompatible column on data(generated by UNION ALL with an union column is > constant and it has incompatible type with corresponding column) > -- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > >For Hive UNION ALL , when a union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. The > query with filter condition on type incompatible column a on this UNION-ALL > results will cause IndexOutOfBoundsException > Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select > `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , > `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select > `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from > `tpch270g`.`textfileorders`) `oo`. >Type of "o_custkey" is INT, the type of corresponding constant "0" is > BIGINT. >Then the fllowing query(with filter incompatible column 0_custkey) will > fail with java.lang.IndexOutOfBoundsException: > "select count(1) from view_orders where o_custkey<10 ". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11695) If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw any exception and fail silently.
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14852764#comment-14852764 ] WangMeng commented on HIVE-11695: - [~ashutoshc] and [~xuefuz] :The above failures are about hcatalog ,this patch may have little associatition with Hcatlog. Thanks. > If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw > any exception and fail silently. > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11695.01.patch > > > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the query does not throw any exception and > fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when execute query with filter condition on type incompatible column on data(generated by UNION ALL with an union column is constant and it h
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Description: For Hive UNION ALL , when a union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of 'o_custkey' is INT normally, while the type of corresponding column constant "0" is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. was: For Hive UNION ALL , when a union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of 'o_custkey' is INT, the type of corresponding constant "0" is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. >IndexOutOfBoundsException when execute query with filter condition on type > incompatible column on data(generated by UNION ALL with an union column is > constant and it has incompatible type with corresponding column) > -- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > >For Hive UNION ALL , when a union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. The > query with filter condition on type incompatible column a on this UNION-ALL > results will cause IndexOutOfBoundsException > Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select > `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , > `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select > `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from > `tpch270g`.`textfileorders`) `oo`. >Type of 'o_custkey' is INT normally, while the type of corresponding > column constant "0" is BIGINT. >Then the fllowing query(with filter incompatible column 0_custkey) will > fail with java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11880) IndexOutOfBoundsException when execute query with filter condition on type incompatible column on data(generated by UNION ALL with an union column is constant and it h
[ https://issues.apache.org/jira/browse/HIVE-11880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11880: Description: For Hive UNION ALL , when a union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of 'o_custkey' is INT, the type of corresponding constant "0" is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail with java.lang.IndexOutOfBoundsException: 'select count(1) from view_orders where o_custkey<10 '. was: For Hive UNION ALL , when a union column is constant(column a such as '0L') and it has incompatible type with the corresponding column A. The query with filter condition on type incompatible column a on this UNION-ALL results will cause IndexOutOfBoundsException Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from `tpch270g`.`textfileorders`) `oo`. Type of "o_custkey" is INT, the type of corresponding constant "0" is BIGINT. Then the fllowing query(with filter incompatible column 0_custkey) will fail with java.lang.IndexOutOfBoundsException: "select count(1) from view_orders where o_custkey<10 ". >IndexOutOfBoundsException when execute query with filter condition on type > incompatible column on data(generated by UNION ALL with an union column is > constant and it has incompatible type with corresponding column) > -- > > Key: HIVE-11880 > URL: https://issues.apache.org/jira/browse/HIVE-11880 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > >For Hive UNION ALL , when a union column is constant(column a such as > '0L') and it has incompatible type with the corresponding column A. The > query with filter condition on type incompatible column a on this UNION-ALL > results will cause IndexOutOfBoundsException > Such as TPC-H table "orders", we CREATE VIEW `view_orders` AS select > `oo`.`o_orderkey` , `oo`.`o_custkey` from ( select `orders`.`o_orderkey` , > `rcfileorders`.`o_custkey` from `tpch270g`.`rcfileorders` union all select > `orcfileorders`.`o_orderkey` , 0L as `o_custkey` from > `tpch270g`.`textfileorders`) `oo`. >Type of 'o_custkey' is INT, the type of corresponding constant "0" is > BIGINT. >Then the fllowing query(with filter incompatible column 0_custkey) will > fail with java.lang.IndexOutOfBoundsException: > 'select count(1) from view_orders where o_custkey<10 '. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Attachment: (was: HIVE-11149.03.patch) > Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in > Multi-thread environment > --- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the Thread unsafe Class HashMap in > PerfLogger.java will casue massive Java Processes hang and cost large > amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Attachment: HIVE-11149.03.patch > Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in > Multi-thread environment > --- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch, > HIVE-11149.03.patch > > > In Multi-thread environment, the Thread unsafe Class HashMap in > PerfLogger.java will casue massive Java Processes hang and cost large > amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11695) If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw any exception and fail silently.
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804863#comment-14804863 ] WangMeng commented on HIVE-11695: - [~hiveqa] > If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw > any exception and fail silently. > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11695.01.patch > > > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the query does not throw any exception and > fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw any exception and fail silently.
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Fix Version/s: (was: 1.2.1) (was: 1.3.0) > If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw > any exception and fail silently. > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11695.01.patch > > > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the query does not throw any exception and > fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804885#comment-14804885 ] WangMeng commented on HIVE-11149: - [~sershe] I know PerfLogger is threadlocal, but in current environment, some times Hive Client's Java Process will hang at " java.util.HashMap.put(HashMap.java:494) "and never exit. such as the following: "main" prio=10 tid=0x7f938c014800 nid=0x4e1a runnable [0x7f9392f3c000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:494) at org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:109) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1282) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:750) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in > Multi-thread environment > --- > > Key: HIVE-11149 > URL: https://issues.apache.org/jira/browse/HIVE-11149 > Project: Hive > Issue Type: Bug > Components: Logging >Affects Versions: 1.2.0 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11149.01.patch, HIVE-11149.02.patch > > > In Multi-thread environment, the Thread unsafe Class HashMap in > PerfLogger.java will casue massive Java Processes hang and cost large > amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw any exception and fail silently.
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw any exception and fail silently. (was: If user have no permission to create LOCAL DIRECTORY such as "/data/wangmeng/hiveserver2" ,the Hql does not throw any exception and fail silently.) > If user have no permission to create LOCAL DIRECTORY ，the Hql does not throw > any exception and fail silently. > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Fix For: 1.3.0, 1.2.1 > > Attachments: HIVE-11695.01.patch > > > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the query does not throw any exception and > fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) If user have no permission to create LOCAL DIRECTORY such as "/data/wangmeng/hiveserver2" ,the Hql does not throw any exception and fail silently.
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: If user have no permission to create LOCAL DIRECTORY such as "/data/wangmeng/hiveserver2" ,the Hql does not throw any exception and fail silently. (was: Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive user does not have write-permission of the DIRECTORY) > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the Hql does not throw any exception and fail > silently. > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Fix For: 1.3.0, 1.2.1 > > Attachments: HIVE-11695.01.patch > > > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the query does not throw any exception and > fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Attachment: HIVE-11695.01.patch > Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive > user does not have write-permission of the DIRECTORY > > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Attachments: HIVE-11695.01.patch > > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791586#comment-14791586 ] WangMeng commented on HIVE-11695: - if hive user can not create local target directory and return false, this should throw an exception rather than fail sliently. [~ashutoshc] Can you please take a look at this small patch? Thanks. > Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive > user does not have write-permission of the DIRECTORY > > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Fix For: 1.3.0, 1.2.1 > > Attachments: HIVE-11695.01.patch > > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Description: If user have no permission to create LOCAL DIRECTORY such as "/data/wangmeng/hiveserver2" ,the query does not throw any exception and fail silently. (was: If user have no write permission to LOCAL DIRECTORY such as "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY "/data/wangmeng/hiveserver2" does not throw any exception and fail silently.) > Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive > user does not have write-permission of the DIRECTORY > > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > Fix For: 1.3.0, 1.2.1 > > Attachments: HIVE-11695.01.patch > > > If user have no permission to create LOCAL DIRECTORY such as > "/data/wangmeng/hiveserver2" ,the query does not throw any exception and > fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: Hql "insert overwrite to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY (was: Hql "write to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY) > Hql "insert overwrite to LOCAL DIRECTORY " can not throw exception when > Hive user does not have write-permission of the DIRECTORY > --- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fails silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: Hql "insert overwrite LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY (was: Hql "insert overwrite to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY) > Hql "insert overwrite LOCAL DIRECTORY " can not throw exception when Hive > user does not have write-permission of the DIRECTORY > - > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fails silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "write to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Description: If user have no write permission to LOCAL DIRECTORY such as "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY "/data/wangmeng/hiveserver2" does not throw any exception and fails silently. (was: For Hive user who dose not have write permission of LOCAL DIRECTORY such as "/data/wangmeng/" , when the user executes Hql "insert overwrite LOCAL DIRECTORY "/data/wangmeng/hiveserver2" ,this query can not throw any exception and pretend to have finished successfully.) > Hql "write to LOCAL DIRECTORY " can not throw exception when Hive user does > not have write-permission of the DIRECTORY > --- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fails silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive user does not have write-permission of the DIRECTORY (was: Hql "insert overwrite LOCAL DIRECTORY " does not throw exception when Hive user does not have write-permission of the DIRECTORY) > Hql "insert overwrite LOCAL DIRECTORY " does not throw exception if Hive > user does not have write-permission of the DIRECTORY > > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "write to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-promotion of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: Hql "write to LOCAL DIRECTORY " can not throw exception when Hive user does not have write-promotion of the DIRECTORY (was: Hql "write to LOCAL DIRECTORY " can not throws exception when Hive user does not have write-promotion of the DIRECTORY) > Hql "write to LOCAL DIRECTORY " can not throw exception when Hive user does > not have write-promotion of the DIRECTORY > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > For Hive user who dose not have write promotion of LOCAL DIRECTORY such as > "/data/wangmeng/" , when the user executes Hql "insert overwrite LOCAL > DIRECTORY "/data/wangmeng/hiveserver2" ,this query can not throw any > exception and pretend to have finished successfully. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " does not throw exception when Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Description: If user have no write permission to LOCAL DIRECTORY such as "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY "/data/wangmeng/hiveserver2" does not throw any exception and fail silently. (was: If user have no write permission to LOCAL DIRECTORY such as "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY "/data/wangmeng/hiveserver2" does not throw any exception and fails silently.) > Hql "insert overwrite LOCAL DIRECTORY " does not throw exception when Hive > user does not have write-permission of the DIRECTORY > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fail silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11695) Hql "insert overwrite LOCAL DIRECTORY " does not throw exception when Hive user does not have write-permission of the DIRECTORY
[ https://issues.apache.org/jira/browse/HIVE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11695: Summary: Hql "insert overwrite LOCAL DIRECTORY " does not throw exception when Hive user does not have write-permission of the DIRECTORY (was: Hql "insert overwrite LOCAL DIRECTORY " can not throw exception when Hive user does not have write-permission of the DIRECTORY) > Hql "insert overwrite LOCAL DIRECTORY " does not throw exception when Hive > user does not have write-permission of the DIRECTORY > -- > > Key: HIVE-11695 > URL: https://issues.apache.org/jira/browse/HIVE-11695 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1 >Reporter: WangMeng >Assignee: WangMeng > > If user have no write permission to LOCAL DIRECTORY such as > "/data/wangmeng/" ,the query "insert overwrite LOCAL DIRECTORY > "/data/wangmeng/hiveserver2" does not throw any exception and fails silently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Description: In Multi-thread environment, the Thread unsafe Class HashMap in PerfLogger.java will casue massive Java Processes hang and cost large amounts of unnecessary CPU and Memory. (was: In Multi-thread environment, the Thread unsafe Class HashMap in PerfLogger.java will hang and cost large amounts of unnecessary CPU and Memory.) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment --- Key: HIVE-11149 URL: https://issues.apache.org/jira/browse/HIVE-11149 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 1.2.0 Reporter: WangMeng Assignee: WangMeng Fix For: 1.2.0 Attachments: HIVE-11149.01.patch In Multi-thread environment, the Thread unsafe Class HashMap in PerfLogger.java will casue massive Java Processes hang and cost large amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11149) Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment
[ https://issues.apache.org/jira/browse/HIVE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangMeng updated HIVE-11149: Attachment: HIVE-11149.01.patch Fix issue with Thread unsafe Class HashMap in PerfLogger.java hangs in Multi-thread environment --- Key: HIVE-11149 URL: https://issues.apache.org/jira/browse/HIVE-11149 Project: Hive Issue Type: Bug Components: Logging Affects Versions: 1.2.0 Reporter: WangMeng Assignee: WangMeng Fix For: 1.2.0 Attachments: HIVE-11149.01.patch In Multi-thread environment, the Thread unsafe Class HashMap in PerfLogger.java will hang and cost large amounts of unnecessary CPU and Memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangmeng updated HIVE-10971:
Attachment: HIVE-10971.01.patch
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
Attachments: HIVE-10971.01.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangmeng updated HIVE-10971:
Component/s: (was: Hive)
Logical Optimizer
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangmeng updated HIVE-10971:
Attachment: HIVE-10971.01.patch
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
Attachments: HIVE-10971.01.patch, HIVE-10971.01.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangmeng updated HIVE-10971:
Attachment: (was: HIVE-10971.01.patch)
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
Attachments: HIVE-10971.01.patch
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578602#comment-14578602
]
wangmeng commented on HIVE-10971:
-
{code}
hive set hive.groupby.skewindata=true;
hive explain select l_returnflag,count(*),count(distinct l_linestatus) from
lineitem group by l_returnflag limit 10;
OK
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 1008537518 Data size: 201707503616 Basic
stats: COMPLETE Column stats: NONE
Select Operator
expressions: l_returnflag (type: string), l_linestatus (type:
string)
outputColumnNames: l_returnflag, l_linestatus
Statistics: Num rows: 1008537518 Data size: 201707503616 Basic
stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(), count(DISTINCT l_linestatus)
keys: l_returnflag (type: string), l_linestatus (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 1008537518 Data size: 201707503616 Basic
stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1008537518 Data size: 201707503616
Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: bigint)
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(DISTINCT KEY._col1:0._col0)
keys: KEY._col0 (type: string)
mode: complete
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 504268759 Data size: 100853751808 Basic stats:
COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: bigint), _col2
(type: bigint)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 504268759 Data size: 100853751808 Basic
stats: COMPLETE Column stats: NONE
Limit
Number of rows: 10
Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE
Column stats: NONE
File Output Operator
compressed: true
Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE
Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: 10
{code}
When hive.groupby.skewindata=false, the Group By operator has mode
mergepartial, which gives the correct results.
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10971) count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true
[
https://issues.apache.org/jira/browse/HIVE-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangmeng updated HIVE-10971:
Description:
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
was:
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two,
which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
count(*) with count(distinct) gives wrong results when
hive.groupby.skewindata=true
---
Key: HIVE-10971
URL: https://issues.apache.org/jira/browse/HIVE-10971
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.2.0
Reporter: wangmeng
Assignee: wangmeng
When hive.groupby.skewindata=true, the following query based on TPC-H gives
wrong results:
{code}
set hive.groupby.skewindata=true;
select l_returnflag, count(*), count(distinct l_linestatus)
from lineitem
group by l_returnflag
limit 10;
{code}
The query plan shows that it generates only one MapReduce job instead of two
theoretically, which is dictated by hive.groupby.skewindata=true.
The problem arises only when {noformat}count(*){noformat} and
{noformat}count(distinct){noformat} exist together.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10478) resolved
[ https://issues.apache.org/jira/browse/HIVE-10478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579978#comment-14579978 ] wangmeng commented on HIVE-10478: - Hi, I also encountered the same problem ,how did you solve it ? SET hive.exec.parallel=false? Thanks. resolved Key: HIVE-10478 URL: https://issues.apache.org/jira/browse/HIVE-10478 Project: Hive Issue Type: Task Components: Hive Reporter: anna ken Labels: hadoop, hive, hue, kryo resolved -- This message was sent by Atlassian JIRA (v6.3.4#6332)
