[
https://issues.apache.org/jira/browse/HIVE-25976?focusedWorklogId=784084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784084
]
ASF GitHub Bot logged work on HIVE-25976:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Jun/22 08:54
Start Date: 23/Jun/22 08:54
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3289:
URL: https://github.com/apache/hive/pull/3289#discussion_r904756053
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java:
##########
@@ -66,32 +70,13 @@ public void initialize(QueryState queryState, QueryPlan
queryPlan, TaskQueue tas
super.initialize(queryState, queryPlan, taskQueue, context);
work.initializeForFetch(context.getOpContext());
+ cachingEnabled = HiveConf.getBoolVar(conf,
HiveConf.ConfVars.HIVEFETCHTASKCACHING);
+ fetchedData = new ArrayList<>();
+
try {
// Create a file system handle
- if (job == null) {
- // The job config should be initilaized once per fetch task. In case
of refetch, we should use the
- // same config.
- job = new JobConf(conf);
- }
-
- Operator<?> source = work.getSource();
- if (source instanceof TableScanOperator) {
- TableScanOperator ts = (TableScanOperator) source;
- // push down projections
- ColumnProjectionUtils.appendReadColumns(job, ts.getNeededColumnIDs(),
ts.getNeededColumns(),
- ts.getNeededNestedColumnPaths(),
ts.getConf().hasVirtualCols());
- // push down filters and as of information
- HiveInputFormat.pushFiltersAndAsOf(job, ts, null);
-
- AcidUtils.setAcidOperationalProperties(job,
ts.getConf().isTranscationalTable(),
- ts.getConf().getAcidOperationalProperties());
- }
- sink = work.getSink();
- fetch = new FetchOperator(work, job, source, getVirtualColumns(source));
- source.initialize(conf, new
ObjectInspector[]{fetch.getOutputObjectInspector()});
- totalRows = 0;
- ExecMapper.setDone(false);
-
+ job = new JobConf(conf);
Review Comment:
Initialize now called only once, FetchTask reinitalization moved to the
private initFetch() method which is called from resetFetch(). So from now on
there is no need to check if the field has already initialized or not.
Issue Time Tracking
-------------------
Worklog Id: (was: 784084)
Time Spent: 1h (was: 50m)
> Cleaner may remove files being accessed from a fetch-task-converted reader
> --------------------------------------------------------------------------
>
> Key: HIVE-25976
> URL: https://issues.apache.org/jira/browse/HIVE-25976
> Project: Hive
> Issue Type: Bug
> Reporter: Zoltan Haindrich
> Assignee: László Végh
> Priority: Major
> Labels: pull-request-available
> Attachments: fetch_task_conv_compactor_test.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> in a nutshell the following happens:
> * query is compiled in fetch-task-converted mode
> * no real execution happens....but the locks are released
> * the HS2 is communicating with the client and uses the fetch-task to get the
> rows - which in this case will directly read files from the table's
> directory....
> * client sleeps between reads - so there is ample time for other events...
> * cleaner wakes up and removes some files....
> * in the next read the fetch-task encounters a read error...
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
