[jira] [Commented] (HIVE-21214) MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
[
https://issues.apache.org/jira/browse/HIVE-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761153#comment-16761153
]
Jason Dere commented on HIVE-21214:
---
I'm not totally sure about the decision to change duplicate filename resolution
from file size to task attempt number. If you just fixed the file size logic to
take directories into account this would allow the existing logic to work in
the directory case. With task attempts we might have to worry about if this
breaks any existing cases. If we are convinced that we just need to worry about
Tez execution then I guess this could work, but this does not work on M/R with
speculative execution.
In terms of code comments, might be better with RB, but I'll add comments here:
* For the comments at the top of compareTempOrDuplicateFiles(), add a comment
this this breaks speculative execution.
* getDirSize() may not be the best name - this is really getting the file
size, and doing so recursively in the case that the file turns out to be a
directory. So maybe getFileSizeRecursivey() or something.
* Log at debug level in getDirSize()
I still need to make sense of the parsing changes
> MoveTask : Use attemptId instead of file size for deduplication of files
> compareTempOrDuplicateFiles()
> --
>
> Key: HIVE-21214
> URL: https://issues.apache.org/jira/browse/HIVE-21214
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-21214.1.patch
>
>
> For a given task, if there is more than one attempt then deduplication logic
> kicks in.
> {noformat}
> Utilities.compareTempOrDuplicateFiles(){noformat}
> The logic uses file size and picks the one with largest size. This logic is
> very fragile.
> ideally, it should pick the successful attempt's file.
> However, a simpler solution is to pick the newest attempt and also checking
> the file size for the newest attempt is the largest.
> If not, throw an exception.
>
> cc [~gopalv] [~thejas] [~jdere] [~ekoifman]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21132) Semi join edge is not being removed despite max bloomfilter entries set to 1
[
https://issues.apache.org/jira/browse/HIVE-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754239#comment-16754239
]
Jason Dere commented on HIVE-21132:
---
+1
> Semi join edge is not being removed despite max bloomfilter entries set to 1
>
>
> Key: HIVE-21132
> URL: https://issues.apache.org/jira/browse/HIVE-21132
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21132.1.patch, HIVE-21132.2.patch,
> HIVE-21132.3.patch
>
>
> * Reproducer
> {code:sql}
> --! qt:dataset:lineitem
> --! qt:dataset:part
> --! qt:dataset:src
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> --set hive.compute.query.using.stats=false;
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.optimize.ppd=true;
> set hive.ppd.remove.duplicatefilters=true;
> set hive.tez.dynamic.partition.pruning=true;
> set hive.tez.dynamic.semijoin.reduction=true;
> set hive.optimize.metadataonly=false;
> set hive.optimize.index.filter=true;
> set hive.stats.autogather=true;
> set hive.tez.bigtable.minsize.semijoin.reduction=1;
> set hive.tez.min.bloom.filter.entries=1;
> set hive.stats.fetch.column.stats=true;
> set hive.tez.bloom.filter.factor=1.0f;
> set hive.auto.convert.join=false;
> set hive.optimize.shared.work=false;
> create database tpch_test;
> use tpch_test;
> CREATE TABLE `customer`(
> `c_custkey` bigint,
> `c_name` string,
> `c_address` string,
> `c_nationkey` bigint,
> `c_phone` string,
> `c_acctbal` double,
> `c_mktsegment` string,
> `c_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026723');
> CREATE TABLE `lineitem`(
> `l_orderkey` bigint,
> `l_partkey` bigint,
> `l_suppkey` bigint,
> `l_linenumber` int,
> `l_quantity` double,
> `l_extendedprice` double,
> `l_discount` double,
> `l_tax` double,
> `l_returnflag` string,
> `l_linestatus` string,
> `l_shipdate` string,
> `l_commitdate` string,
> `l_receiptdate` string,
> `l_shipinstruct` string,
> `l_shipmode` string,
> `l_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543027179');
> CREATE TABLE `orders`(
> `o_orderkey` bigint,
> `o_custkey` bigint,
> `o_orderstatus` string,
> `o_totalprice` double,
> `o_orderdate` string,
> `o_orderpriority` string,
> `o_clerk` string,
> `o_shippriority` int,
> `o_comment` string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1543026824');
> alter table customer update statistics
> set('numRows'='15000','rawDataSize'='8633707142');
> alter table lineitem update statistics
> set('numRows'='589709','rawDataSize'='184245066955');
> alter table orders update statistics
> set('numRows'='15','rawDataSize'='46741318253');
> create view q18_tmp_cached as
> select l_orderkey, sum(l_quantity) as t_sum_quantity
> from lineitem
> where l_orderkey is not null
> group by l_orderkey;
> -- Set bloom filter size to huge number so we get any possible semijoin
> reductions
> set hive.tez.min.bloom.filter.entries=0;
> set hive.tez.max.bloom.filter.entries=1;
> create table q18_large_volume_customer_cached stored as orc tblproperties
> ('transactional'='true', 'transactional_properties'='default') as
> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity)
> from customer, orders, q18_tmp_cached t, lineitem l
> where
> c_custkey = o_custkey and o_orderkey = t.l_orderkey
> and o_orderkey is not null and t.t_sum_quantity > 300
> and o_orderkey = l.l_orderkey and l.l_orderkey is not null
> group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
> order by o_totalprice
[jira] [Updated] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20998: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20998.1.patch > > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20998: -- Attachment: HIVE-20998.1.patch > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20998.1.patch > > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20998: -- Attachment: (was: HIVE-20998.1.patch) > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20998: -- Attachment: HIVE-20998.1.patch > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20998.1.patch > > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20998: -- Status: Patch Available (was: Open) > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20998.1.patch > > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21020) log which table/partition is being processed by a txn in Worker
[
https://issues.apache.org/jira/browse/HIVE-21020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715562#comment-16715562
]
Jason Dere commented on HIVE-21020:
---
+1
> log which table/partition is being processed by a txn in Worker
> ---
>
> Key: HIVE-21020
> URL: https://issues.apache.org/jira/browse/HIVE-21020
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-21020.01.patch
>
>
> Make sure we have info in the log that ties cat.table.part with txnid of the
> compactor
> in Worker
> {\{LOG.info("Starting " + ci.type.toString() + " compaction for " +
> ci.getFullPartitionName());}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20998) HiveStrictManagedMigration utility should update DB/Table location as last migration steps
[ https://issues.apache.org/jira/browse/HIVE-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20998: - > HiveStrictManagedMigration utility should update DB/Table location as last > migration steps > -- > > Key: HIVE-20998 > URL: https://issues.apache.org/jira/browse/HIVE-20998 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > When processing a database or table, the HiveStrictManagedMigration utility > currently changes the database/table locations as the first step in > processing that database/table. Unfortunately if an error occurs while > processing this database or table, then there may still be migration work > that needs to continue for that db/table by running the migration again. > However the migration tool only processes dbs/tables that have the old > warehouse location, then the tool will skip over the db/table when the > migration is run again. > One fix here is to set the new location as the last step after all of the > migration work is done: > - The new table location will not be set until all of its partitions have > been successfully migrated. > - The new database location will not be set until all of its tables have > been successfully migrated. > For existing migrations that failed with an error, the following workaround > can be done so that the db/tables can be re-processed by the migration tool: > 1) Use the migration tool logs to find which databases/tables failed during > processing. > 2) For each db/table, change location of of the database and table back to > old location: > ALTER DATABASE tpcds_bin_partitioned_orc_10 SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db'; > ALTER TABLE tpcds_bin_partitioned_orc_10.store_sales SET LOCATION > 'hdfs://ns1/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/store_sales'; > 2) Rerun the migration tool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20985) If select operator inputs are temporary columns vectorization may reuse some of them as output
[ https://issues.apache.org/jira/browse/HIVE-20985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20985: -- Attachment: HIVE-20985.04.patch > If select operator inputs are temporary columns vectorization may reuse some > of them as output > -- > > Key: HIVE-20985 > URL: https://issues.apache.org/jira/browse/HIVE-20985 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20985.01.patch, HIVE-20985.02.patch, > HIVE-20985.03.patch, HIVE-20985.03.patch, HIVE-20985.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20985) If select operator inputs are temporary columns vectorization may reuse some of them as output
[ https://issues.apache.org/jira/browse/HIVE-20985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704379#comment-16704379 ] Jason Dere commented on HIVE-20985: --- Uploaded new patch to fix TestVectorizationContext failures > If select operator inputs are temporary columns vectorization may reuse some > of them as output > -- > > Key: HIVE-20985 > URL: https://issues.apache.org/jira/browse/HIVE-20985 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20985.01.patch, HIVE-20985.02.patch, > HIVE-20985.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20985) If select operator inputs are temporary columns vectorization may reuse some of them as output
[ https://issues.apache.org/jira/browse/HIVE-20985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20985: -- Attachment: HIVE-20985.03.patch > If select operator inputs are temporary columns vectorization may reuse some > of them as output > -- > > Key: HIVE-20985 > URL: https://issues.apache.org/jira/browse/HIVE-20985 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20985.01.patch, HIVE-20985.02.patch, > HIVE-20985.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20985) If select operator inputs are temporary columns vectorization may reuse some of them as output
[ https://issues.apache.org/jira/browse/HIVE-20985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703927#comment-16703927 ] Jason Dere commented on HIVE-20985: --- Looks like the patch ends up disabling most of the scratch column re-use. I don't have a lot of knowledge of the vectorization details, cc [~teddy.choi] [~gopalv] > If select operator inputs are temporary columns vectorization may reuse some > of them as output > -- > > Key: HIVE-20985 > URL: https://issues.apache.org/jira/browse/HIVE-20985 > Project: Hive > Issue Type: Bug > Components: Vectorization >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20985.01.patch, HIVE-20985.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20981) streaming/AbstractRecordWriter leaks HeapMemoryMonitor
[ https://issues.apache.org/jira/browse/HIVE-20981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20981: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master > streaming/AbstractRecordWriter leaks HeapMemoryMonitor > -- > > Key: HIVE-20981 > URL: https://issues.apache.org/jira/browse/HIVE-20981 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20981.1.patch > > > Each record writer registers a memory monitor with the MemoryMXBean but they > aren't removed. So the listener objects/lambdas accumulate over time in the > bean. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20981) streaming/AbstractRecordWriter leaks HeapMemoryMonitor
[ https://issues.apache.org/jira/browse/HIVE-20981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702550#comment-16702550 ] Jason Dere commented on HIVE-20981: --- +1 > streaming/AbstractRecordWriter leaks HeapMemoryMonitor > -- > > Key: HIVE-20981 > URL: https://issues.apache.org/jira/browse/HIVE-20981 > Project: Hive > Issue Type: Bug >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20981.1.patch > > > Each record writer registers a memory monitor with the MemoryMXBean but they > aren't removed. So the listener objects/lambdas accumulate over time in the > bean. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20935) Upload of llap package tarball fails in EC2 causing LLAP service start failure
[
https://issues.apache.org/jira/browse/HIVE-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20935:
--
Attachment: HIVE-20935.02.patch
> Upload of llap package tarball fails in EC2 causing LLAP service start failure
> --
>
> Key: HIVE-20935
> URL: https://issues.apache.org/jira/browse/HIVE-20935
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: HIVE-20935.01.patch, HIVE-20935.02.patch
>
>
> Even though package dir is defined as below (with a / at the end) -
> {code}
> LLAP_PACKAGE_DIR = ".yarn/package/LLAP/";
> {code}
> copyLocalFileToHdfs API fails to create the dir hierarchy of
> .yarn/package/LLAP/ first and then copy the file under it. It instead uploads
> the file under .yarn/package with name LLAP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20935) Upload of llap package tarball fails in EC2 causing LLAP service start failure
[
https://issues.apache.org/jira/browse/HIVE-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692051#comment-16692051
]
Jason Dere commented on HIVE-20935:
---
Yeah we need green CI .. I'll just re-upload the patch as a new version number
> Upload of llap package tarball fails in EC2 causing LLAP service start failure
> --
>
> Key: HIVE-20935
> URL: https://issues.apache.org/jira/browse/HIVE-20935
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: HIVE-20935.01.patch
>
>
> Even though package dir is defined as below (with a / at the end) -
> {code}
> LLAP_PACKAGE_DIR = ".yarn/package/LLAP/";
> {code}
> copyLocalFileToHdfs API fails to create the dir hierarchy of
> .yarn/package/LLAP/ first and then copy the file under it. It instead uploads
> the file under .yarn/package with name LLAP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[
https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685830#comment-16685830
]
Jason Dere commented on HIVE-20900:
---
Looks like TIMESTAMP WITH LOCAL TIME ZONE never implemented the
timestamp.formats support:
{noformat}
Caused by: org.apache.hadoop.hive.serde2.SerDeException: struct field __time:
Could not convert from string to map type timestamp with local time zone
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parseStruct(HiveJsonStructReader.java:203)
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parseDispatcher(HiveJsonStructReader.java:117)
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parseInternal(HiveJsonStructReader.java:100)
... 55 more
Caused by: java.io.IOException: Could not convert from string to map type
timestamp with local time zone
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parsePrimitiveValue(HiveJsonStructReader.java:370)
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.getObjectOfCorrespondingPrimitiveType(HiveJsonStructReader.java:322)
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parsePrimitive(HiveJsonStructReader.java:308)
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parseDispatcher(HiveJsonStructReader.java:113)
at
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.parseStruct(HiveJsonStructReader.java:201)
... 57 more
{noformat}
> serde2.JsonSerDe no longer supports timestamp.formats
> -
>
> Key: HIVE-20900
> URL: https://issues.apache.org/jira/browse/HIVE-20900
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20900.1.patch, HIVE-20900.2.patch
>
>
> Looks like HIVE-18545 broke this.
> Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the
> format in that test used the ISO timestamp format which apparently is now
> parsed by the default timestamp parsing, so the test was too simple.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20900: -- Attachment: HIVE-20900.2.patch > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20900.1.patch, HIVE-20900.2.patch > > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685577#comment-16685577 ] Jason Dere commented on HIVE-20900: --- [~kgyrtkirk] how about this fix, primitive value parsing will go through the same logic whether it is the hcat or serde2 version of JsonSerde. > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20900.1.patch, HIVE-20900.2.patch > > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20900: -- Status: Patch Available (was: Open) > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20900.1.patch > > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20900: -- Attachment: HIVE-20900.1.patch > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20900.1.patch > > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20900) serde2.JsonSerDe no longer supports timestamp.formats
[ https://issues.apache.org/jira/browse/HIVE-20900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20900: - Assignee: Jason Dere > serde2.JsonSerDe no longer supports timestamp.formats > - > > Key: HIVE-20900 > URL: https://issues.apache.org/jira/browse/HIVE-20900 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Looks like HIVE-18545 broke this. > Also json_serde_tsformat.q only tested the hcat version of JsonSerde, and the > format in that test used the ISO timestamp format which apparently is now > parsed by the default timestamp parsing, so the test was too simple. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch,
> HIVE-20839.3.patch, HIVE-20839.4.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674760#comment-16674760
]
Jason Dere commented on HIVE-20839:
---
Hey [~vgarg], ptests went through, is this patch good to commit?
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch,
> HIVE-20839.3.patch, HIVE-20839.4.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Attachment: HIVE-20839.4.patch
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch,
> HIVE-20839.3.patch, HIVE-20839.4.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) I've committed this to master > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20834.1.patch, HIVE-20834.2.patch, > HIVE-20834.3.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Attachment: HIVE-20834.3.patch > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, HIVE-20834.2.patch, > HIVE-20834.3.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Attachment: HIVE-20834.2.patch > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, HIVE-20834.2.patch, > dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Attachment: HIVE-20839.3.patch
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch,
> HIVE-20839.3.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Status: Patch Available (was: In Progress)
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670446#comment-16670446
]
Jason Dere commented on HIVE-20839:
---
Thanks [~vgarg]. Attaching new patch using .equals() rather than .isSame().
Also removed the test, will create a new Jira with the testcase once HIVE-20833
is done.
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Attachment: HIVE-20839.2.patch
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20787) MapJoinBytesTableContainer dummyRow case doesn't handle reuse
[ https://issues.apache.org/jira/browse/HIVE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669328#comment-16669328 ] Jason Dere commented on HIVE-20787: --- +1 pending test results > MapJoinBytesTableContainer dummyRow case doesn't handle reuse > - > > Key: HIVE-20787 > URL: https://issues.apache.org/jira/browse/HIVE-20787 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20787.01.patch, HIVE-20787.patch > > > Discovered while investigating some (probably) unrelated issue. > MapJoinBytesTableContainer was not intended to be reused, but it looks like > some code might reuse it. If that happens, the dummyRow case will not work > correctly (dummyRow is cleared on first(), so another call to first() will > behave differently). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669107#comment-16669107
]
Jason Dere commented on HIVE-20839:
---
Patch to fix the issue by checking for the folded-from column in the case of
constant expressions.
An alternative fix would be if ExprNodeConstantDesc.isSame() also took into
account the folded-from column. However is that something that is appropriate
[~vgarg] [~ashutoshc]?
The test case required tables with lots of columns unfortunately, so the tests
for this patch cannot run until HIVE-20833 is committed.
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
>
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Attachment: HIVE-20839.1.patch
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669100#comment-16669100
]
Jason Dere commented on HIVE-20839:
---
Issue seems to be that there are several column expressions that end up being
converted to constant value 'SSS' during constant folding. Some of these
constant columns end up being pruned from the query, and others might be join
keys which are required in the query. The problem was that
ExprNodeDescUtils.resolveJoinKeysAsRSColumns() (which is used to setup the join
key expressions in the case of dynamically partitioned hashjoin) was trying to
resolve the join key expression for the 'SSS' column and picked the wrong
column - it was also constant 'SSS', but it was actually a different column
that ended up getting pruned.
The ExprNodeConstDesc actually has a getFoldedFromCol() field that shows what
column this constant expression has been folded from, I think this can be used
to help resolve to the correct expression.
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
>
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Description:
Occurs in some cases in the non-CBO optimized queries, either if CBO is
disabled or has failed due to error.
{noformat}
2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
(1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
while closing operators - failing tree
2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
(1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
166:_col169, 167:_col170, 168:_col171, 169:_col318]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
at
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
at
org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
{noformat}
was:
{noformat}
2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
(1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
while closing operators - failing tree
2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
(1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
[jira] [Work started] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20839 started by Jason Dere.
-
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere reassigned HIVE-20839:
-
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669068#comment-16669068 ] Jason Dere commented on HIVE-20834: --- Yes, the Supplier inner class ended up holding a ref to the SemanticAnalyzer. Does the fix in the patch make sense, or did you have another fix in mind? > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Status: Patch Available (was: Open) > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668122#comment-16668122 ] Jason Dere commented on HIVE-20834: --- Patch to just fetch the ValidTxnWriteIdList upfront when creating the results cache entry, rather than trying to defer the call to fetch the ValidTxnWriteIdList. Looked at heap dumps before/after this patch and the results cache entry is no longer hanging onto a reference to the SemanticAnalyzer. [~jcamachorodriguez] [~gopalv] can you review? > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Attachment: HIVE-20834.1.patch > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20834.1.patch, dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668001#comment-16668001
]
Jason Dere commented on HIVE-20833:
---
+1
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code` string,
>
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667912#comment-16667912
]
Jason Dere commented on HIVE-20833:
---
Thanks for finding this. +1
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, first.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code` string,
>
[jira] [Updated] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20834: -- Attachment: dominator_tree.png > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: dominator_tree.png > > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20834) Hive QueryResultCache entries keeping reference to SemanticAnalyzer from cached query
[ https://issues.apache.org/jira/browse/HIVE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20834: - > Hive QueryResultCache entries keeping reference to SemanticAnalyzer from > cached query > - > > Key: HIVE-20834 > URL: https://issues.apache.org/jira/browse/HIVE-20834 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > QueryResultCache.LookupInfo ends up keeping a reference to the > SemanticAnalyzer from the cached query, for as long as the cached entry is in > the cache. We should not be keeping the SemanticAnalyzer around after the > query is done executing since they can hold on to quite a bit of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20707) Automatic partition management
[ https://issues.apache.org/jira/browse/HIVE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667796#comment-16667796 ] Jason Dere commented on HIVE-20707: --- +1 > Automatic partition management > -- > > Key: HIVE-20707 > URL: https://issues.apache.org/jira/browse/HIVE-20707 > Project: Hive > Issue Type: New Feature >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20702.3.patch, HIVE-20707.1.patch, > HIVE-20707.2.patch, HIVE-20707.4.patch, HIVE-20707.5.patch, > HIVE-20707.6.patch, HIVE-20707.6.patch, HIVE-20707.7.patch > > > In current scenario, to add partitions for external tables to metastore, MSCK > REPAIR command has to be executed manually. To avoid this manual step, > external tables can be specified a table property based on which a background > metastore thread can sync partitions periodically. Tables can also be > specified with partition retention period. Any partition whose age exceeds > the retention period will be dropped automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664135#comment-16664135 ] Jason Dere commented on HIVE-20259: --- RB at https://reviews.apache.org/r/69173/ > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664134#comment-16664134 ] Jason Dere commented on HIVE-20259: --- Re-attaching patch, though actually this had a green run last time. [~gopalv], can you review this one? > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20259: -- Attachment: HIVE-20259.2.patch > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20250) Option to allow external tables to use query results cache
[ https://issues.apache.org/jira/browse/HIVE-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664120#comment-16664120 ] Jason Dere edited comment on HIVE-20250 at 10/25/18 6:27 PM: - Hey [~thai.bui], sorry didn't get around to working on this one. How about going with the approach outlined [here|https://issues.apache.org/jira/browse/HIVE-20250?focusedCommentId=16565715=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16565715]. That would enable your case with the timeout approach. Feel free to try to take up, or I will eventually get around to it when I get a chance. Regarding the ability to receive notifications to invalidate the cache, there is also HIVE-19154, where HS2 is able to listen on metastore notification events to invalidate result cache entries. If you are able to generate a basic ALTER TABLE or ALTER PARTITION statement on the table which results in the metastore generating the notification event, this could suit your purposes. Something like ALTER TABLE TOUCH might be useful here. was (Author: jdere): Hey [~thai.bui], sorry didn't get around to working on this one. How about going with the approach outlined [here|https://issues.apache.org/jira/browse/HIVE-20250?focusedCommentId=16565715=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16565715]. That would enable your case with the timeout approach. Feel free to try to take up, or I will eventually get around to it when I get a chance. Regarding the ability to receive notifications to invalidate the cache, there is also HIVE-19154, where HS2 is able to listen on metastore notification events to invalidate result cache entries. If you are able to generate a basic ALTER TABLE or ALTER PARTITION statement on the table which results in the metastore generating the notification event, this could suit your purposes. > Option to allow external tables to use query results cache > -- > > Key: HIVE-20250 > URL: https://issues.apache.org/jira/browse/HIVE-20250 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Priority: Major > Attachments: HIVE-20250.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20250) Option to allow external tables to use query results cache
[ https://issues.apache.org/jira/browse/HIVE-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664120#comment-16664120 ] Jason Dere commented on HIVE-20250: --- Hey [~thai.bui], sorry didn't get around to working on this one. How about going with the approach outlined [here|https://issues.apache.org/jira/browse/HIVE-20250?focusedCommentId=16565715=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16565715]. That would enable your case with the timeout approach. Feel free to try to take up, or I will eventually get around to it when I get a chance. Regarding the ability to receive notifications to invalidate the cache, there is also HIVE-19154, where HS2 is able to listen on metastore notification events to invalidate result cache entries. If you are able to generate a basic ALTER TABLE or ALTER PARTITION statement on the table which results in the metastore generating the notification event, this could suit your purposes. > Option to allow external tables to use query results cache > -- > > Key: HIVE-20250 > URL: https://issues.apache.org/jira/browse/HIVE-20250 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Priority: Major > Attachments: HIVE-20250.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20705) Vectorization: Native Vector MapJoin doesn't support Complex Big Table values
[ https://issues.apache.org/jira/browse/HIVE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20705: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master > Vectorization: Native Vector MapJoin doesn't support Complex Big Table values > - > > Key: HIVE-20705 > URL: https://issues.apache.org/jira/browse/HIVE-20705 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-20705.01.patch, HIVE-20705.02.patch, > HIVE-20705.03.patch, HIVE-20705.04.patch, HIVE-20705.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20705) Vectorization: Native Vector MapJoin doesn't support Complex Big Table values
[ https://issues.apache.org/jira/browse/HIVE-20705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642277#comment-16642277 ] Jason Dere commented on HIVE-20705: --- +1 on the changes, can you confirm the test failures are not related to the patch? > Vectorization: Native Vector MapJoin doesn't support Complex Big Table values > - > > Key: HIVE-20705 > URL: https://issues.apache.org/jira/browse/HIVE-20705 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-20705.01.patch, HIVE-20705.02.patch, > HIVE-20705.03.patch, HIVE-20705.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20603:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20603.1.patch, HIVE-20603.2.patch,
> HIVE-20603.3.patch
>
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20603:
--
Attachment: HIVE-20603.3.patch
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20603.1.patch, HIVE-20603.2.patch,
> HIVE-20603.3.patch
>
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622950#comment-16622950
]
Jason Dere commented on HIVE-20603:
---
updated patch to fix archive_insert tests. Other failures don't appear related.
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20603.1.patch, HIVE-20603.2.patch
>
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20603:
--
Attachment: HIVE-20603.2.patch
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20603.1.patch, HIVE-20603.2.patch
>
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20603:
--
Status: Patch Available (was: Open)
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20603.1.patch
>
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20603:
--
Attachment: HIVE-20603.1.patch
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20603.1.patch
>
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621336#comment-16621336
]
Jason Dere commented on HIVE-20603:
---
Looks like HIVE-19891 changed some behavior relevant to this, but missed out on
the case where the new table location used a different FS.
cc [~sershe] [~ashutoshc]
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20603) "Wrong FS" error when inserting to partition after changing table location filesystem
[
https://issues.apache.org/jira/browse/HIVE-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere reassigned HIVE-20603:
-
> "Wrong FS" error when inserting to partition after changing table location
> filesystem
> -
>
> Key: HIVE-20603
> URL: https://issues.apache.org/jira/browse/HIVE-20603
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
>
> Inserting into an existing partition, after changing a table's location to
> point to a different HDFS filesystem:
> {noformat}
>query += "CREATE TABLE test_managed_tbl (id int, name string, dept string)
> PARTITIONED BY (year int);\n"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2016) VALUES
> (8,'Henry','CSE');\n"
> query += "ALTER TABLE test_managed_tbl ADD PARTITION (year=2017);\n"
> query += "ALTER TABLE test_managed_tbl SET LOCATION
>
> 'hdfs://ns2/warehouse/tablespace/managed/hive/test_managed_tbl'"
> query += "INSERT INTO test_managed_tbl PARTITION (year=2017) VALUES
> (9,'Harris','CSE');\n"
> {noformat}
> Results in the following error:
> {noformat}
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ns1/warehouse/tablespace/managed/hive/test_managed_tbl/year=2017,
> expected: hdfs://ns2
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4141)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1966)
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:477)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:397)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20524) Schema Evolution checking is broken in going from Hive version 2 to version 3 for ALTER TABLE VARCHAR to DECIMAL
[
https://issues.apache.org/jira/browse/HIVE-20524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612830#comment-16612830
]
Jason Dere commented on HIVE-20524:
---
+1 pending test results
> Schema Evolution checking is broken in going from Hive version 2 to version 3
> for ALTER TABLE VARCHAR to DECIMAL
>
>
> Key: HIVE-20524
> URL: https://issues.apache.org/jira/browse/HIVE-20524
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-20524.01.patch, HIVE-20524.02.patch
>
>
> Issue that started this JIRA:
> {code}
> create external table varchar_decimal (c1 varchar(25));
> alter table varchar_decimal change c1 c1 decimal(31,0);
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following
> columns have types incompatible with the existing columns in their respective
> positions :
> c1
> {code}
> There appear to be 2 issues here:
> 1) When hive.metastore.disallow.incompatible.col.type.changes is true (the
> default) we only allow StringFamily (STRING, CHAR, VARCHAR) conversion to a
> number that can hold the largest numbers. The theory being we don't want
> data loss you would get by converting the StringFamily field into integers,
> etc. In Hive version 2 the hierarchy of numbers had DECIMAL at the top. At
> some point during Hive version 2 we realized this was incorrect and put
> DOUBLE the top.
> However, the Hive2 Hive version 2 TypeInfoUtils.implicitConversion method
> allows StringFamily to either DOUBLE or DECIMAL conversion.
> The new org.apache.hadoop.hive.metastore.ColumnType class under Hive version
> 3 hive-standalone-metadata-server method checkColTypeChangeCompatible only
> allows DOUBLE.
> This JIRA fixes that problem.
> 2) Also, the checkColTypeChangeCompatible method lost a version 2 series bug
> fix that drops CHAR/VARCHAR (and DECIMAL I think) type decorations when
> checking for Schema Evolution compatibility. So, when that code is checking
> if a data type "varchar(25)" is StringFamily it fails because the "(25)"
> didn't get removed properly.
> This JIRA fixes issue #2 also.
> NOTE: Hive1 version 2 did undecoratedTypeName(oldType) and Hive2 version
> performed the logic in TypeInfoUtils.implicitConvertible on the
> PrimitiveCategory not the raw type string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20412) NPE in HiveMetaHook
[
https://issues.apache.org/jira/browse/HIVE-20412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20412:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master
> NPE in HiveMetaHook
> ---
>
> Key: HIVE-20412
> URL: https://issues.apache.org/jira/browse/HIVE-20412
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20412.1.patch, HIVE-20412.2.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.HiveMetaHook.preAlterTable(HiveMetaHook.java:113)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:427)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:415)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2933)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:708)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.util.HiveStrictManagedMigration$HiveUpdater.updateTableProperties(HiveStrictManagedMigration.java:954)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20412) NPE in HiveMetaHook
[
https://issues.apache.org/jira/browse/HIVE-20412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20412:
--
Attachment: HIVE-20412.2.patch
> NPE in HiveMetaHook
> ---
>
> Key: HIVE-20412
> URL: https://issues.apache.org/jira/browse/HIVE-20412
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20412.1.patch, HIVE-20412.2.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.HiveMetaHook.preAlterTable(HiveMetaHook.java:113)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:427)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:415)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2933)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:708)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.util.HiveStrictManagedMigration$HiveUpdater.updateTableProperties(HiveStrictManagedMigration.java:954)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20514) Query with outer join filter is failing with dynamic partition join
[
https://issues.apache.org/jira/browse/HIVE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607505#comment-16607505
]
Jason Dere commented on HIVE-20514:
---
+1. Interesting that this also makes a vertex vectorizable in
tez_dynpart_hashjoin_3
> Query with outer join filter is failing with dynamic partition join
> ---
>
> Key: HIVE-20514
> URL: https://issues.apache.org/jira/browse/HIVE-20514
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20514.1.patch, HIVE-20514.2.patch
>
>
> *Reproducer*
> Copy the following query in {{tez_dynpart_hashjoin_1.q}} and run the test
> {code:sql}
> select
> *
> from alltypesorc a left outer join alltypesorc b on a.cint = b.cint and
> a.csmallint != a.cint
> where
> a.cint between 100 and 300
> order by a.cint;
> {code}
> *Exception*
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1536275581088_0001_5_02,
> diagnostics=[Task failed, taskId=task_1536275581088_0001_5_02_09,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1536275581088_0001_5_02_09_0:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:key,
> 1:value]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:537)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:56)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:266)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:155)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:193)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1536275581088_0001_5_02_09_1:java.lang.RuntimeException:
> java.lang.RuntimeException: cannot find field _col1 from [0:key, 1:value]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
[jira] [Updated] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20515: -- Attachment: HIVE-20515.2.patch > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch, HIVE-20515.2.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606574#comment-16606574 ] Jason Dere commented on HIVE-20515: --- [~thejas], can you review? > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20515: -- Attachment: HIVE-20515.1.patch Status: Patch Available (was: Open) > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20515.1.patch > > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20515) Empty query results when using results cache and query temp dir, results cache dir in different filesystems
[ https://issues.apache.org/jira/browse/HIVE-20515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20515: - > Empty query results when using results cache and query temp dir, results > cache dir in different filesystems > --- > > Key: HIVE-20515 > URL: https://issues.apache.org/jira/browse/HIVE-20515 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > If the scratchdir for temporary query results and the results cache dir are > in different filesystems, moving the query from the temp directory to results > cache will fail. > Looking at the moveResultsToCacheDirectory() logic in QueryResultsCache.java, > I see the following issues: > - FileSystem.rename() is used, which only works if the files are on the same > filesystem. Need to use something like Hive.mvFile or something similar which > can work between different filesystems. > - The return code from rename() was not checked which might possibly have > caught the error here. This may not be applicable if a different method from > FS.rename() is used in the proper fix. > With some filesystems (noticed this with WASB), if FileSystem.rename() > returns false on failure rather than throwing an exception, then this results > in empty results showing up for the query because the return code was not > checked properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20407) add txn table alter support to HiveStrictManagedMigration
[ https://issues.apache.org/jira/browse/HIVE-20407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584409#comment-16584409 ] Jason Dere commented on HIVE-20407: --- +1 > add txn table alter support to HiveStrictManagedMigration > - > > Key: HIVE-20407 > URL: https://issues.apache.org/jira/browse/HIVE-20407 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20407.patch > > > On top of HIVE-20397 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20412) NPE in HiveMetaHook
[
https://issues.apache.org/jira/browse/HIVE-20412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20412:
--
Assignee: Jason Dere
Status: Patch Available (was: Open)
[~ashutoshc] [~sershe] can you take a look? One line fix.
> NPE in HiveMetaHook
> ---
>
> Key: HIVE-20412
> URL: https://issues.apache.org/jira/browse/HIVE-20412
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20412.1.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.HiveMetaHook.preAlterTable(HiveMetaHook.java:113)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:427)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:415)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2933)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:708)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.util.HiveStrictManagedMigration$HiveUpdater.updateTableProperties(HiveStrictManagedMigration.java:954)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20412) NPE in HiveMetaHook
[
https://issues.apache.org/jira/browse/HIVE-20412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20412:
--
Attachment: HIVE-20412.1.patch
> NPE in HiveMetaHook
> ---
>
> Key: HIVE-20412
> URL: https://issues.apache.org/jira/browse/HIVE-20412
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20412.1.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.HiveMetaHook.preAlterTable(HiveMetaHook.java:113)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:427)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.alter_table(SessionHiveMetaStoreClient.java:415)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_112]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2933)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at com.sun.proxy.$Proxy37.alter_table(Unknown Source) ~[?:?]
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:708)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> at
> org.apache.hadoop.hive.ql.util.HiveStrictManagedMigration$HiveUpdater.updateTableProperties(HiveStrictManagedMigration.java:954)
> ~[hive-exec-3.1.0.3.0.1.0-104.jar:3.1.0.3.0.1.0-104]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20397: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20397.1.patch > > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20378) don't update stats during alter for txn table conversion
[ https://issues.apache.org/jira/browse/HIVE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582882#comment-16582882 ] Jason Dere commented on HIVE-20378: --- +1 > don't update stats during alter for txn table conversion > > > Key: HIVE-20378 > URL: https://issues.apache.org/jira/browse/HIVE-20378 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20378.01.patch, HIVE-20378.02.patch, > HIVE-20378.03.patch, HIVE-20378.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20397: -- Status: Patch Available (was: Open) [~ekoifman] can you take a look? > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20397.1.patch > > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20397: -- Attachment: HIVE-20397.1.patch > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20397.1.patch > > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20397) HiveStrictManagedMigration updates
[ https://issues.apache.org/jira/browse/HIVE-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20397: - > HiveStrictManagedMigration updates > -- > > Key: HIVE-20397 > URL: https://issues.apache.org/jira/browse/HIVE-20397 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > - Switch from using Driver instance to using metastore calls via > Hive.alterDatabase/Hive.alterTable > - For tables converted from ORC to ACID tables, handle renaming of the files > - Fix error handling so utility does not terminate after the first error > encountered -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20298:
--
Attachment: (was: HIVE-20298.1.patch)
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579090#comment-16579090
]
Jason Dere commented on HIVE-20298:
---
+1
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20298.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Commented] (HIVE-19800) Create separate submodules for pre and post upgrade and add rename file logic
[ https://issues.apache.org/jira/browse/HIVE-19800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576950#comment-16576950 ] Jason Dere commented on HIVE-19800: --- +1 pending tests > Create separate submodules for pre and post upgrade and add rename file logic > - > > Key: HIVE-19800 > URL: https://issues.apache.org/jira/browse/HIVE-19800 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19800.01.patch, HIVE-19800.02.patch, > HIVE-19800.03.patch, HIVE-19800.04.patch, HIVE-19800.05.patch, > HIVE-19800.06.patch > > > this is a followup to HIVE-19751 which includes HIVE-19751 since it hasn't > landed yet > this includes file rename logic and HIVE-19750 since it hasn't landed yet > either > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20290) Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during GetSplits
[
https://issues.apache.org/jira/browse/HIVE-20290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20290:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master
> Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during
> GetSplits
> ---
>
> Key: HIVE-20290
> URL: https://issues.apache.org/jira/browse/HIVE-20290
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-20290.1.patch, HIVE-20290.2.patch
>
>
> When using {{GenericUDTFGetSplits}} to create {{LlapInputSplit}} for
> submission to {{LlapOutputFormatService}}, the physical plan generation
> initializes whatever SerDe is being used.
> {{ArrowColumnarBatchSerDe}} initializes buffers for Arrow and
> {{VectorizedRowBatch}} at this point inside HS2 which are never used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20300) VectorFileSinkArrowOperator
[ https://issues.apache.org/jira/browse/HIVE-20300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568956#comment-16568956 ] Jason Dere commented on HIVE-20300: --- Took a look a the use of the LlapOutputFormatService within VectorFileSinkArrowOperator and left one comment. [~mmccline]/[~teddy.choi] might be able to take a better look at the vectorization changes than I can. > VectorFileSinkArrowOperator > --- > > Key: HIVE-20300 > URL: https://issues.apache.org/jira/browse/HIVE-20300 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20300.1.patch, HIVE-20300.2.patch > > > Bypass the row-mode FileSinkOperator for pushing Arrow format to the > LlapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567500#comment-16567500 ] Jason Dere commented on HIVE-20259: --- Attached patch with utility DirectoryMarkerUpdate/Cleanup classes to create .cacheupdate files in the cache directory, to indicate that this directory should not be cleaned up by any other process performing DirectoryMarkerCleanup. This uses the last modify date of the .cacheupdate file to determine whether the file should be cleaned up, if the instance running cleanup determines this date is too old then the directory will be deleted. Another option, rather than relying on the lastModifyDate of the .cacheupdate file, would be for the .cacheupdate file contents to contain a long string indicating when this directory should be considered stale and safe to delete. This would have the benefit that the determination of when the directory should be cleaned up would depend on the settings of the application that wrote the .cacheupdate file, rather than on the settings of the application instance that is performing the cleanup. Though it means more file operations - it would involve having to read the file rather than just dealing with the file metadata. > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20259: -- Status: Patch Available (was: Open) > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20259: -- Attachment: HIVE-20259.1.patch > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567485#comment-16567485
]
Jason Dere commented on HIVE-20298:
---
Actually looking at the definition of the TBLS table in
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql,
this column is meant to have a default value
{noformat}
`WRITE_ID` bigint(20) DEFAULT 0,
{noformat}
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
>
[jira] [Commented] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567302#comment-16567302
]
Jason Dere commented on HIVE-20298:
---
Attaching a patch that got around the problem for me. [~sershe] [~ekoifman]
does this look ok, or does WRITE_ID require a default value when the column is
added to the TBLS metastore table?
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
>
[jira] [Updated] (HIVE-20298) Illegal null value in column `TBLS`.`WRITE_ID`
[
https://issues.apache.org/jira/browse/HIVE-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20298:
--
Attachment: HIVE-20298.1.patch
> Illegal null value in column `TBLS`.`WRITE_ID`
> --
>
> Key: HIVE-20298
> URL: https://issues.apache.org/jira/browse/HIVE-20298
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jason Dere
>Priority: Major
> Attachments: HIVE-20298.1.patch
>
>
> Manually upgraded my existing local metastore using
> upgrade-3.0.0-to-3.1.0.mysql.sql, upgrade-3.1.0-to-3.2.0.mysql.sql,
> upgrade-3.2.0-to-4.0.0.mysql.sql. When running DESCRIBE EXTENDED of an
> existing table, I was getting the following error in hive.log. It looks like
> the ObjectStore/MTable classes don't seem to be able to support null values
> in the new writeId column that was added to the TBLS table in the metastore.
> cc [~sershe] [~ekoifman]
> {noformat}
> Caused by: javax.jdo.JDODataStoreException: Illegal null value in column
> `TBLS`.`WRITE_ID`
> NestedThrowables:
> org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal null value
> in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:553)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1802)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:1838)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1424)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy39.getTable(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2950)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTableInternal(HiveMetaStore.java:2898)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_req(HiveMetaStore.java:2882)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ... 36 more
> Caused by: org.datanucleus.store.rdbms.exceptions.NullValueException: Illegal
> null value in column `TBLS`.`WRITE_ID`
> at
> org.datanucleus.store.rdbms.mapping.datastore.BigIntRDBMSMapping.getLong(BigIntRDBMSMapping.java:140)
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getLong(SingleFieldMapping.java:155)
> at
> org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchLongField(ResultSetGetter.java:124)
> at
> org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:1549)
> at
> org.datanucleus.state.StateManagerImpl.replacingLongField(StateManagerImpl.java:120)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceField(MTable.java)
> at
> org.apache.hadoop.hive.metastore.model.MTable.dnReplaceFields(MTable.java)
> at
> org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:3109)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:465)
> at
> org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:2238)
> at
> org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:263)
> at
> org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:112)
> at
> org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3097)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObjectForDatastoreId(PersistentClassROF.java:460)
> at
> org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:385)
> at
>
[jira] [Commented] (HIVE-20290) Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during GetSplits
[
https://issues.apache.org/jira/browse/HIVE-20290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566050#comment-16566050
]
Jason Dere commented on HIVE-20290:
---
+1
> Lazy initialize ArrowColumnarBatchSerDe so it doesn't allocate buffers during
> GetSplits
> ---
>
> Key: HIVE-20290
> URL: https://issues.apache.org/jira/browse/HIVE-20290
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: HIVE-20290.1.patch
>
>
> When using {{GenericUDTFGetSplits}} to create {{LlapInputSplit}} for
> submission to {{LlapOutputFormatService}}, the physical plan generation
> initializes whatever SerDe is being used.
> {{ArrowColumnarBatchSerDe}} initializes buffers for Arrow and
> {{VectorizedRowBatch}} at this point inside HS2 which are never used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20250) Option to allow external tables to use query results cache
[ https://issues.apache.org/jira/browse/HIVE-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565715#comment-16565715 ] Jason Dere commented on HIVE-20250: --- Had some offline discussion with [~hagleitn] on this and there is concern that users might too easily enable hive.query.results.cache.external.tables.enabled/hive.query.results.cache.nontransactional.tables.enabled without considering the ramifications - that Hive will continue to serve stale cached results if such tables are modified outside of Hive. One option we have talked about is replacing hive.query.results.cache.max.entry.lifetime with a new timeout setting that only applies to non-transactional/external tables, as both [~gopalv] and [~hagleitn] have argued that cached results from transactional tables do not need to be timed out (there is other logic to automatically invalidate results on transactional tables if a transactional table has been updated using a transaction). So hive.query.results.cache.external.tables.enabled/hive.query.results.cache.nontransactional.tables.enabled/hive.query.results.cache.max.entry.lifetime would be replaced by some new setting like hive.query.results.cache.nontransactional.result.lifetime (defaulted to 0), where setting this value with a positive value would enable results caching for non-transactional tables, where the cached result would be considered valid for as long as the value of that setting. > Option to allow external tables to use query results cache > -- > > Key: HIVE-20250 > URL: https://issues.apache.org/jira/browse/HIVE-20250 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Priority: Major > Attachments: HIVE-20250.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565684#comment-16565684 ] Jason Dere commented on HIVE-18513: --- Created a simple patch for this at HIVE-20250, but had some pushback from [~hagleitn] over concern that customers would misunderstand some of the settings. Will leave more comments at HIVE-20250. Will try to add some docs when I get a chance. There aren't any memory management settings (the results are kept in HDFS in the results cache directory), but are settings related to size of the results kept by a single Hive instance (hive.query.results.cache.max.size, hive.query.results.cache.max.entry.size). > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch, > HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch, HIVE-18513.6.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560444#comment-16560444 ] Jason Dere commented on HIVE-20259: --- Yeah, I had originally proposed using the ClearDanglingScratchDir functionality for cleanup, though [~hagleitn] had some reservations with this approach as this depends on very specific behavior of HDFS (file leases). Another possible option rather than using the file lease, is to simply periodically write a file with a known naming convention (.cleanup.timestamp_val?) to the cache directory. A cleanup thread would look for the .cleanup file in the base cache directory, and delete any cache directories where the age of the file is too old. As long as the Hive process is still alive and creating new versions of the cleanup file, its cache directory would not be cleaned up by the cleanup thread. > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-20259: - > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20250) Option to allow external tables to use query results cache
[ https://issues.apache.org/jira/browse/HIVE-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559062#comment-16559062 ] Jason Dere commented on HIVE-20250: --- Patch to enable caching of external tables by setting both hive.query.results.cache.nontransactional.tables.enabled and hive.query.results.cache.external.tables.enabled. > Option to allow external tables to use query results cache > -- > > Key: HIVE-20250 > URL: https://issues.apache.org/jira/browse/HIVE-20250 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Priority: Major > Attachments: HIVE-20250.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20250) Option to allow external tables to use query results cache
[ https://issues.apache.org/jira/browse/HIVE-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20250: -- Attachment: HIVE-20250.1.patch > Option to allow external tables to use query results cache > -- > > Key: HIVE-20250 > URL: https://issues.apache.org/jira/browse/HIVE-20250 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Priority: Major > Attachments: HIVE-20250.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20240) Semijoin Reduction : Use local variable to check for external table condition
[ https://issues.apache.org/jira/browse/HIVE-20240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559036#comment-16559036 ] Jason Dere commented on HIVE-20240: --- +1 > Semijoin Reduction : Use local variable to check for external table condition > - > > Key: HIVE-20240 > URL: https://issues.apache.org/jira/browse/HIVE-20240 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20240.1.patch > > > This condition, > > semiJoin = semiJoin && > !disableSemiJoinOptDueToExternalTable(parseContext.getConf(), ts, ctx); > > may set semiJoin to false if an external table is encountered and will remain > false for subsequent cases. It should only disable it for that particular > case. > > cc [~jdere] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20242) Query results cache: Improve ability of queries to use pending query results
[ https://issues.apache.org/jira/browse/HIVE-20242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20242: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master > Query results cache: Improve ability of queries to use pending query results > > > Key: HIVE-20242 > URL: https://issues.apache.org/jira/browse/HIVE-20242 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20242.1.patch > > > HIVE-19138 allowed a currently running query to wait on the pending results > of an already running query. [~gopalv], after testing with high concurrency, > suggested further improving this by having a way to use the switch to using > the results cache even at the end of query compilation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
