[
https://issues.apache.org/jira/browse/DRILL-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831371#comment-16831371
]
ASF GitHub Bot commented on DRILL-7223:
---------------------------------------
amansinha100 commented on pull request #1776: DRILL-7223: Create an option to
control timeout for REFRESH METADATA
URL: https://github.com/apache/drill/pull/1776#discussion_r280269517
##########
File path: exec/java-exec/src/main/resources/drill-module.conf
##########
@@ -629,6 +629,7 @@ drill.exec.options: {
store.parquet.reader.columnreader.async: false,
store.parquet.reader.int96_as_timestamp: false,
store.parquet.reader.strings_signed_min_max: "",
+ store.parquet.refresh_timeout_per_runnable_in_msec: 15000,
Review comment:
See prior comment about omitting 'refresh' . Also I think that we should
not put this in the 'parquet' namespace even though currently it is mainly used
for Parquet metadata read. The reason is the runnable timeout could in theory
be used for other formats also. How about `store.timeout_per_runnable_in_msec`
?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Make the timeout in TimedCallable a configurable boot time parameter
> --------------------------------------------------------------------
>
> Key: DRILL-7223
> URL: https://issues.apache.org/jira/browse/DRILL-7223
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.16.0
> Reporter: Aman Sinha
> Assignee: Boaz Ben-Zvi
> Priority: Minor
> Fix For: 1.17.0
>
>
> The
> [TimedCallable.TIMEOUT_PER_RUNNABLE_IN_MSECS|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java#L52]
> is currently an internal Drill constant defined as 15 secs. This has been
> there from day 1 of the introduction. Drill's TimedCallable implements the
> Java concurrency's Callable interface to create timed threads. It is used by
> the REFRESH METADATA command which creates multiple threads on the Foreman
> node to gather Parquet metadata to build the metadata cache.
> Depending on the load on the system or for very large scale number of parquet
> files (millions) it is possible to exceed this timeout. While the exact root
> cause of exceeding the timeout is being investigated, it makes sense to make
> this timeout a configurable parameter to aid with large scale testing. This
> JIRA is to make this a configurable bootstrapping option in the
> drill-override.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
