Mark Mc Keown created HDFS-16825: ------------------------------------ Summary: hadoop-azure flush timing out and triggering retry Key: HDFS-16825 URL: https://issues.apache.org/jira/browse/HDFS-16825 Project: Hadoop HDFS Issue Type: Bug Reporter: Mark Mc Keown
>From AbfsHttpOperation the code to create a HTTP connection to Azure is: {code} public AbfsHttpOperation(final URL url, final String method, final List<AbfsHttpHeader> requestHeaders) throws IOException { this.isTraceEnabled = LOG.isTraceEnabled(); this.url = url; this.method = method; this.clientRequestId = UUID.randomUUID().toString(); this.connection = openConnection(); if (this.connection instanceof HttpsURLConnection) { HttpsURLConnection secureConn = (HttpsURLConnection) this.connection; SSLSocketFactory sslSocketFactory = SSLSocketFactoryEx.getDefaultFactory(); if (sslSocketFactory != null) { secureConn.setSSLSocketFactory(sslSocketFactory); } } this.connection.setConnectTimeout(CONNECT_TIMEOUT); this.connection.setReadTimeout(READ_TIMEOUT); this.connection.setRequestMethod(method); for (AbfsHttpHeader header : requestHeaders) { this.connection.setRequestProperty(header.getName(), header.getValue()); } this.connection.setRequestProperty(HttpHeaderConfigurations.X_MS_CLIENT_REQUEST_ID, clientRequestId); } {code} The READ_TIMEOUT is hard coded to 30 seconds. When a file uploaded to Azure and closed it triggers a flush operation - Azure sometimes takes longer than 30 seconds to respond and this is triggering a retry within hadoop-azure library. (This can cause issues with DataBricks Autoloader which monitors EventGrid for tiggers to ingest data - multiple flush/close can confuse it, this is an Autoloader bug as retries can happen normally). Can the READ_TIMEOUT be increased or made configurable? -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org