[jira] [Commented] (HIVE-21543) Use FilterHooks for show compactions
[ https://issues.apache.org/jira/browse/HIVE-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904386#comment-16904386 ] Peter Vary commented on HIVE-21543: --- CC: [~szita]: Could you please review? Thanks, Peter > Use FilterHooks for show compactions > > > Key: HIVE-21543 > URL: https://issues.apache.org/jira/browse/HIVE-21543 > Project: Hive > Issue Type: Improvement >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21543.01.patch, HIVE-21543.02.patch > > > Use FilterHooks for checking dbs/tables/partitions for showCompactions -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui An reassigned HIVE-22088:
-
Assignee: Hui An
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> OS: Ubuntu 16.04.4 LTS
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Assignee: Hui An
>Priority: Major
>
> If external table location spec has a '=' sign (coincidentally partition
> specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized.
> Like in below example, there are individual customer specific tables and
> queries/jobs to insert data partitioned by type. And there is another table
> to give the consolidated data view of all the customers, thus have two level
> partition customer and type.
> The job to insert customer specific data into customer specific table fails
> if we use dynamic partitioning. Static partition insert on same table works
> fine though.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains
> non-partition columns
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are
> under the defined table root/base path. If the path itself has partition
> style format (customer=abc in above example) then that should not be
> considered as partition as it is outside the scope of the table.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui An updated HIVE-22088:
--
Affects Version/s: 2.3.4
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.4
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> OS: Ubuntu 16.04.4 LTS
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Assignee: Hui An
>Priority: Major
>
> If external table location spec has a '=' sign (coincidentally partition
> specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized.
> Like in below example, there are individual customer specific tables and
> queries/jobs to insert data partitioned by type. And there is another table
> to give the consolidated data view of all the customers, thus have two level
> partition customer and type.
> The job to insert customer specific data into customer specific table fails
> if we use dynamic partitioning. Static partition insert on same table works
> fine though.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains
> non-partition columns
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are
> under the defined table root/base path. If the path itself has partition
> style format (customer=abc in above example) then that should not be
> considered as partition as it is outside the scope of the table.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22088) Dynamic partition insert problem on external table with "=" in location path spec
[
https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904404#comment-16904404
]
Hui An commented on HIVE-22088:
---
MoveTask will read HDFS files to get partitions info, the problem might occur
there. Working on this issue.
> Dynamic partition insert problem on external table with "=" in location path
> spec
> -
>
> Key: HIVE-22088
> URL: https://issues.apache.org/jira/browse/HIVE-22088
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.4
> Environment: Hive 2.6.0.10-2 Executing on Tez.
> OS: Ubuntu 16.04.4 LTS
> Config settings used:
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
>Reporter: Puneet Khatod
>Assignee: Hui An
>Priority: Major
>
> If external table location spec has a '=' sign (coincidentally partition
> specifier) in it, then dynamic partition loading fails.
> *Use cases:*
> Quite often the same data is used in different contexts by creating different
> external tables on top of the data. Many times the tables have different
> partition depths depending on how data is organized.
> Like in below example, there are individual customer specific tables and
> queries/jobs to insert data partitioned by type. And there is another table
> to give the consolidated data view of all the customers, thus have two level
> partition customer and type.
> The job to insert customer specific data into customer specific table fails
> if we use dynamic partitioning. Static partition insert on same table works
> fine though.
> *Replication:*
> To replicate following simple setup could be done. Below execution is on
> 'Tez'.
> *Source table**-*
> CREATE EXTERNAL TABLE temp_dummy_table
> (id STRING, type STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/source/';
>
> *Destination Table-*
> CREATE EXTERNAL TABLE temp_dummy_dest_table
> (id STRING)
> PARTITIONED BY (type string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/home/destination/{color:#ff}customer=abc{color}/';
>
> *Insert into destination-*
> insert overwrite table temp_dummy_dest_table partition (type)
> select i.id as id, i.type as type
> from temp_dummy_table i
> where i.type in ('type1','type2');
>
> *Log and Error Msgs on CLI*-
> Loading data to table temp_dummy_dest_table partition (type=null)
> Failed with exception Partition spec \{type=type1, customer=abc} contains
> non-partition columns
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> *Possible resolution:*
> The dynamic partitioning should consider only those partition specs which are
> under the defined table root/base path. If the path itself has partition
> style format (customer=abc in above example) then that should not be
> considered as partition as it is outside the scope of the table.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[ https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22083: --- Attachment: HIVE-22083.4.patch > Values of tag order cannot be null, so it can be "byte" instead of "Byte" > - > > Key: HIVE-22083 > URL: https://issues.apache.org/jira/browse/HIVE-22083 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22083.1.patch, HIVE-22083.2.patch, > HIVE-22083.3.patch, HIVE-22083.4.patch > > > Values of tag order cannot be null, so it can be "byte" instead of "Byte". > Switching between Byte and byte is "cheap" - the Byte objects are cached by > the JVM - but it still costs a bit more memory and CPU usage. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[
https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904415#comment-16904415
]
Hive QA commented on HIVE-22083:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 599 unchanged - 53
fixed = 599 total (was 652) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
15s{color} | {color:red} ql generated 4 new + 2237 unchanged - 13 fixed = 2241
total (was 2250) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.plan.HashTableSinkDesc.getTagOrder() may expose
internal representation by returning HashTableSinkDesc.tagOrder At
HashTableSinkDesc.java:by returning HashTableSinkDesc.tagOrder At
HashTableSinkDesc.java:[line 263] |
| | org.apache.hadoop.hive.ql.plan.HashTableSinkDesc.setTagOrder(byte[]) may
expose internal representation by storing an externally mutable object into
HashTableSinkDesc.tagOrder At HashTableSinkDesc.java:by storing an externally
mutable object into HashTableSinkDesc.tagOrder At HashTableSinkDesc.java:[line
268] |
| | org.apache.hadoop.hive.ql.plan.JoinDesc.getTagOrder() may expose internal
representation by returning JoinDesc.tagOrder At JoinDesc.java:by returning
JoinDesc.tagOrder At JoinDesc.java:[line 419] |
| | org.apache.hadoop.hive.ql.plan.JoinDesc.setTagOrder(byte[]) may expose
internal representation by storing an externally mutable object into
JoinDesc.tagOrder At JoinDesc.java:by storing an externally mutable object
into JoinDesc.tagOrder At JoinDesc.java:[line 429] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18302/dev-support/hive-personality.sh
|
| git revision | master / 1729d77 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18302/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18302/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Values of tag order cannot be null, so it can be "byte" instead of "Byte"
> -
>
> Key: HIVE-22083
> URL: https://issues.apache.org/jira/browse/HIVE-22083
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Su
[jira] [Commented] (HIVE-22083) Values of tag order cannot be null, so it can be "byte" instead of "Byte"
[
https://issues.apache.org/jira/browse/HIVE-22083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904422#comment-16904422
]
Hive QA commented on HIVE-22083:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12977219/HIVE-22083.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 16732 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query23]
(batchId=298)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query88]
(batchId=298)
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testConcurrentDropPartitions
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestFunctions.addNoSuchCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.dropNoSuchCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.getFunctionsNoSuchCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.getNoSuchCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.otherCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionAlreadyExists[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionCaseInsensitive[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionEmptyNameInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionInvalidNameInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNoSuchDatabaseInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNoSuchDatabase[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNoSuchFunctionInThisDatabase[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNoSuchFunction[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullClassInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullDatabaseNameInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullDatabase[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullFunctionNameInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullFunctionName[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullFunctionTypeInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullFunction[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunctionNullOwnerTypeInNew[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testAlterFunction[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testCreateFunctionAlreadyExists[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testCreateFunctionCaseInsensitive[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testCreateFunctionDefaultValues[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testCreateFunctionEmptyName[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testCreateFunctionInvalidName[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestFunctions.testCreateFu
[jira] [Work started] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views
[
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21344 started by Jesus Camacho Rodriguez.
--
> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views
[
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21344:
---
Status: Patch Available (was: In Progress)
> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views
[
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904513#comment-16904513
]
Hive QA commented on HIVE-21344:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12960551/calcite-planner-after-fix.svg.zip
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18303/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18303/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18303/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-08-10 21:23:21.237
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18303/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-08-10 21:23:21.240
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1729d77 HIVE-21579: Introduce more complex SQL:2016 datetime
formats (Karen Coppage via Marta Kuczora)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1729d77 HIVE-21579: Introduce more complex SQL:2016 datetime
formats (Karen Coppage via Marta Kuczora)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-08-10 21:23:24.038
+ rm -rf ../yetus_PreCommit-HIVE-Build-18303
+ mkdir ../yetus_PreCommit-HIVE-Build-18303
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18303
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18303/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: unrecognized input
fatal: unrecognized input
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-18303
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12960551 - PreCommit-HIVE-Build
> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views
[
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904514#comment-16904514
]
Jesus Camacho Rodriguez commented on HIVE-21344:
I uploaded a patch that changes the current logic.
With new logic, we retrieve the materialized views only from the registry (no
access to metastore) and we check whether the materialized view is valid or not
once we have produced the rewriting. This improves compilation times for those
queries for which a materialized view rewritting is not produced (we do not
need to query the metastore) and also for queries for which the rewriting is
produced (we only check the validity for the materialization that is being used
by the query).
Registry is updated proactively for ops happening in the HS2 where it is
contained, and after detecting a materialization is outdated and periodically
for ops happening in a remote HS2.
> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: calcite-planner-after-fix.svg.zip, mv-get-from-remote.png
>
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22094) queries failing with ClassCastException: hive.ql.exec.vector.DecimalColumnVector cannot be cast to hive.ql.exec.vector.Decimal64ColumnVector
[
https://issues.apache.org/jira/browse/HIVE-22094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22094:
Summary: queries failing with ClassCastException:
hive.ql.exec.vector.DecimalColumnVector cannot be cast to
hive.ql.exec.vector.Decimal64ColumnVector (was: Mondrian queries failing with
ClassCastException: hive.ql.exec.vector.DecimalColumnVector cannot be cast to
hive.ql.exec.vector.Decimal64ColumnVector)
> queries failing with ClassCastException:
> hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> hive.ql.exec.vector.Decimal64ColumnVector
>
>
> Key: HIVE-22094
> URL: https://issues.apache.org/jira/browse/HIVE-22094
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22094.1.patch, HIVE-22094.2.patch
>
>
> When running a query like this
> select sum(salary.salary_paid) from salary, employee_closure where
> salary.employee_id = employee_closure.employee_id;
> with hive.auto.convert.join=true and hive.vectorized.execution.enabled=true
> the following exception occurs
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal64ToDecimal.aggregateInput(VectorUDAFSumDecimal64ToDecimal.java:320)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:217)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:182)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1124)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:706)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator.generateHashMultiSetResultMultiValue(VectorMapJoinInnerBigOnlyGenerateResultOperator.java:268)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator.finishInnerBigOnly(VectorMapJoinInnerBigOnlyGenerateResultOperator.java:180)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyLongOperator.processBatch(VectorMapJoinInnerBigOnlyLongOperator.java:379)
> ... 28 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22094) queries failing with ClassCastException: hive.ql.exec.vector.DecimalColumnVector cannot be cast to hive.ql.exec.vector.Decimal64ColumnVector
[
https://issues.apache.org/jira/browse/HIVE-22094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904524#comment-16904524
]
Ashutosh Chauhan commented on HIVE-22094:
-
+1
> queries failing with ClassCastException:
> hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> hive.ql.exec.vector.Decimal64ColumnVector
>
>
> Key: HIVE-22094
> URL: https://issues.apache.org/jira/browse/HIVE-22094
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22094.1.patch, HIVE-22094.2.patch
>
>
> When running a query like this
> select sum(salary.salary_paid) from salary, employee_closure where
> salary.employee_id = employee_closure.employee_id;
> with hive.auto.convert.join=true and hive.vectorized.execution.enabled=true
> the following exception occurs
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal64ToDecimal.aggregateInput(VectorUDAFSumDecimal64ToDecimal.java:320)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:217)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:182)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1124)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:706)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator.generateHashMultiSetResultMultiValue(VectorMapJoinInnerBigOnlyGenerateResultOperator.java:268)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator.finishInnerBigOnly(VectorMapJoinInnerBigOnlyGenerateResultOperator.java:180)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyLongOperator.processBatch(VectorMapJoinInnerBigOnlyLongOperator.java:379)
> ... 28 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22094) queries failing with ClassCastException: hive.ql.exec.vector.DecimalColumnVector cannot be cast to hive.ql.exec.vector.Decimal64ColumnVector
[
https://issues.apache.org/jira/browse/HIVE-22094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22094:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Attila!
> queries failing with ClassCastException:
> hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> hive.ql.exec.vector.Decimal64ColumnVector
>
>
> Key: HIVE-22094
> URL: https://issues.apache.org/jira/browse/HIVE-22094
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22094.1.patch, HIVE-22094.2.patch
>
>
> When running a query like this
> select sum(salary.salary_paid) from salary, employee_closure where
> salary.employee_id = employee_closure.employee_id;
> with hive.auto.convert.join=true and hive.vectorized.execution.enabled=true
> the following exception occurs
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal64ToDecimal.aggregateInput(VectorUDAFSumDecimal64ToDecimal.java:320)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:217)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:182)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1124)
> at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:706)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator.generateHashMultiSetResultMultiValue(VectorMapJoinInnerBigOnlyGenerateResultOperator.java:268)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator.finishInnerBigOnly(VectorMapJoinInnerBigOnlyGenerateResultOperator.java:180)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyLongOperator.processBatch(VectorMapJoinInnerBigOnlyLongOperator.java:379)
> ... 28 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21344) CBO: Reduce compilation time in presence of materialized views
[
https://issues.apache.org/jira/browse/HIVE-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21344:
---
Attachment: HIVE-21344.patch
> CBO: Reduce compilation time in presence of materialized views
> --
>
> Key: HIVE-21344
> URL: https://issues.apache.org/jira/browse/HIVE-21344
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21344.patch, calcite-planner-after-fix.svg.zip,
> mv-get-from-remote.png
>
>
> For every query, {{getAllValidMaterializedViews}} still requires a call to
> metastore to verify that the materializations exist, whether they are
> outdated or not, etc. Since this is only useful for active-active HS2
> deployments, we could take a less aggressive approach and check this
> information only after rewriting has been triggered. In addition, we could
> refresh the information in the HS2 registry periodically in a background
> thread.
> {code}
> // This is not a rebuild, we retrieve all the materializations. In turn, we
> do not need
> // to force the materialization contents to be up-to-date, as this is not a
> rebuild, and
> // we apply the user parameters
> (HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
> materializations = db.getAllValidMaterializedViews(getTablesUsed(basePlan),
> false, getTxnMgr());
> {code}
> !mv-get-from-remote.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22092) Fetch is failing with IllegalArgumentException: No ValidTxnList when refetch is done
[ https://issues.apache.org/jira/browse/HIVE-22092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22092: --- Status: Open (was: Patch Available) > Fetch is failing with IllegalArgumentException: No ValidTxnList when refetch > is done > > > Key: HIVE-22092 > URL: https://issues.apache.org/jira/browse/HIVE-22092 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-22092.01.patch, HIVE-22092.02.patch > > > The fetch task is created during query compilation with the config of the > driver. That config will have the valid txn list set. Thus the fetch task > will have valid txn list set while doing fetch for ACID tables. But when user > does a refetch with cusrsor set to first position it reinitializes the fetch > task with the driver config (cached in task config). But by that time, the > select query would have cleaned up the valid txn list from the config and the > fetch will happen with valid txn list as null. This will cause illegal > argument exception. > > Caused by: java.lang.IllegalArgumentException: No ValidTxnList fo > at org.apache.hadoop.hive.ql.io.AcidUtils.isDirUsable(AcidUtils.java:1418) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.io.AcidUtils.getChildState(AcidUtils.java:1393) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1205) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.processForWriteIds(HiveInputFormat.java:633) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.processPathsForMmRead(HiveInputFormat.java:597) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.processCurrPathForMmWriteIds(FetchOperator.java:469) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:396) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:322) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:563) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:532) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2362) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22092) Fetch is failing with IllegalArgumentException: No ValidTxnList when refetch is done
[ https://issues.apache.org/jira/browse/HIVE-22092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22092: --- Status: Patch Available (was: Open) > Fetch is failing with IllegalArgumentException: No ValidTxnList when refetch > is done > > > Key: HIVE-22092 > URL: https://issues.apache.org/jira/browse/HIVE-22092 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-22092.01.patch, HIVE-22092.02.patch > > > The fetch task is created during query compilation with the config of the > driver. That config will have the valid txn list set. Thus the fetch task > will have valid txn list set while doing fetch for ACID tables. But when user > does a refetch with cusrsor set to first position it reinitializes the fetch > task with the driver config (cached in task config). But by that time, the > select query would have cleaned up the valid txn list from the config and the > fetch will happen with valid txn list as null. This will cause illegal > argument exception. > > Caused by: java.lang.IllegalArgumentException: No ValidTxnList fo > at org.apache.hadoop.hive.ql.io.AcidUtils.isDirUsable(AcidUtils.java:1418) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.io.AcidUtils.getChildState(AcidUtils.java:1393) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1205) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.processForWriteIds(HiveInputFormat.java:633) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.processPathsForMmRead(HiveInputFormat.java:597) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.processCurrPathForMmWriteIds(FetchOperator.java:469) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:396) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:322) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:563) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:532) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2362) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22092) Fetch is failing with IllegalArgumentException: No ValidTxnList when refetch is done
[ https://issues.apache.org/jira/browse/HIVE-22092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22092: --- Attachment: HIVE-22092.02.patch > Fetch is failing with IllegalArgumentException: No ValidTxnList when refetch > is done > > > Key: HIVE-22092 > URL: https://issues.apache.org/jira/browse/HIVE-22092 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-22092.01.patch, HIVE-22092.02.patch > > > The fetch task is created during query compilation with the config of the > driver. That config will have the valid txn list set. Thus the fetch task > will have valid txn list set while doing fetch for ACID tables. But when user > does a refetch with cusrsor set to first position it reinitializes the fetch > task with the driver config (cached in task config). But by that time, the > select query would have cleaned up the valid txn list from the config and the > fetch will happen with valid txn list as null. This will cause illegal > argument exception. > > Caused by: java.lang.IllegalArgumentException: No ValidTxnList fo > at org.apache.hadoop.hive.ql.io.AcidUtils.isDirUsable(AcidUtils.java:1418) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.io.AcidUtils.getChildState(AcidUtils.java:1393) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1205) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.processForWriteIds(HiveInputFormat.java:633) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.processPathsForMmRead(HiveInputFormat.java:597) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.processCurrPathForMmWriteIds(FetchOperator.java:469) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:396) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:322) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:563) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:532) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2362) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] > at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) > ~[data_analytics_studio-webapp-1.3.1.1.3.1.0-12.jar:?] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
