Created https://issues.apache.org/jira/browse/MRESOLVER-361 to track the issue.
On Fri, May 19, 2023 at 11:27 PM Finkelman, Jacob <jfink...@amazon.com.invalid> wrote: > During a maven publish if the server closes the TCP connection after the > upload of an asset sometimes the publish will fail with a failed to respond > while attempting to upload the checksum. The correct behavior to wanting to > send more HTTP requests on a closed TCP connection, is to open a new TCP > connection. If the client and server have a low latency connection, say in > the same data center, when the connection is closed a new one appears to > reliably be opened. During higher latency scenarios, in our case when the > client and server are on different continents, maven attempts to reuse the > connection that the server has already closed receiving a failed to respond > error about half the time. On even slower connections we get a Connection > or outbound has closed error. > > Unfortunately, this connection does not get retried. It immediately fails > to build no matter how many retries are configured. I believe this is the > known https://issues.apache.org/jira/browse/MDEPLOY-162 If these requests > were retried on this error, I suspect the entire situation would not have > been a problem. The lack of retries go through different code paths > depending on the transport implementation. > > The wagon transport fails with > org.apache.maven.wagon.providers.http.httpclient.client.NonRepeatableRequestException: > Cannot retry request with a non-repeatable request entity. Fixing the wagon > transport requires changes in maven-wagon and in maven-resolver. In > particular the WagonHttpEntity needs to use a stream supplier in order to > be retryable. My colleague sketched out these changes a year ago. See > https://github.com/mmarston/maven-wagon/commit/4419140ff7794a548f3f9cb9e320fcd20420b5e0 > and > https://github.com/mmarston/maven-resolver/commit/cf00080ac546bda588bd86a44f93c037f7638145 > > The native transport does not have this limitation because when uploading > checksums it uses a PutTaskEntity that is retryable because the PutTask > that it wraps around is essentially a stream supplier. Here< > https://github.com/apache/maven-resolver/blob/89dadad860dc802fd328f2c7a29e7285b599ecde/maven-resolver-transport-http/src/main/java/org/eclipse/aether/transport/http/HttpTransporter.java#L635>. > For the native transport my colleagues think the issue is that it only > supports retry when the client detects the closed connection before or > while sending the request. But if the client sends the request and then > detects the closed connection while trying to read the response, then it > won't retry. In particular the native transport uses > DefaultHttpRequestRetryHandler with requestSentRetryEnabled set to false > (see here< > https://github.com/apache/maven-resolver/blob/d0f8564846863c8a88f2fb1da8391fc860517192/maven-resolver-transport-http/src/main/java/org/eclipse/aether/transport/http/HttpTransporter.java#L239>). > One way to fix the native transport is to change requestSentRetryEnabled to > true when constructing DefaultHttpRequestRetryHandler. Another option is to > use StandardHttpRequestRetryHandler instead because it treats PUT requests > as idempotent so will retry a PUT request even when requestSentRetryEnabled > is false. > > Oddly, if only one checksum algorithm is selected by using > -Daether.checksums.algorithms=MD5 then the entire build fails on this error > but if two or more algorithms are selected > -Daether.checksums.algorithms=MD5,SHA-1 each checksum throws a warning but > the overall build succeeds. It seems odd that the behavior is inconsistent > here. I would expect it either to always throw a warning if a requested > checksum failed upload, or to always fail the build if any checksum failed > upload. The behavior with multiple checksums requested makes it clear that > this failed to respond error can be caught by mvn. > > Using -Daether.connector.http.reuseConnections=false makes this error go > away, because maven never tries sending data over an existing TCP > connection. > > I am new to the maven dev community and was not sure what the best > practice was for reporting these kinds of issues. Starting a conversation > on the mailing list seems like a good place to start. What should happen > next? Which of these issues deserve tickets? What more information should I > provide and where? What PRs would be appreciated to help fix the issues? > > A stack trace from running "mvn deploy" with Apache Maven 3.9.2 > (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c) > > org.apache.http.NoHttpResponseException: ...:443 failed to respond > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead > (DefaultHttpResponseParser.java:141) > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead > (DefaultHttpResponseParser.java:56) > at org.apache.http.impl.io.AbstractMessageParser.parse > (AbstractMessageParser.java:259) > at > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader > (DefaultBHttpClientConnection.java:163) > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader > (CPoolProxy.java:157) > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse > (HttpRequestExecutor.java:273) > at org.apache.http.protocol.HttpRequestExecutor.execute > (HttpRequestExecutor.java:125) > at org.apache.http.impl.execchain.MainClientExec.execute > (MainClientExec.java:272) > at org.apache.http.impl.execchain.ProtocolExec.execute > (ProtocolExec.java:186) > at org.apache.http.impl.execchain.RetryExec.execute (RetryExec.java:89) > at org.apache.http.impl.execchain.RedirectExec.execute > (RedirectExec.java:110) > at org.apache.http.impl.client.InternalHttpClient.doExecute > (InternalHttpClient.java:185) > at org.apache.http.impl.client.CloseableHttpClient.execute > (CloseableHttpClient.java:72) > at org.eclipse.aether.transport.http.HttpTransporter.execute > (HttpTransporter.java:393) > at org.eclipse.aether.transport.http.HttpTransporter.implPut > (HttpTransporter.java:377) > at org.eclipse.aether.spi.connector.transport.AbstractTransporter.put > (AbstractTransporter.java:107) > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum > (BasicRepositoryConnector.java:608) > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums > (BasicRepositoryConnector.java:591) at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask > (BasicRepositoryConnector.java:565) > at > org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run > (BasicRepositoryConnector.java:414) > at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put > (BasicRepositoryConnector.java:319) > at org.eclipse.aether.internal.impl.DefaultDeployer.deploy > (DefaultDeployer.java:298) > at org.eclipse.aether.internal.impl.DefaultDeployer.deploy > (DefaultDeployer.java:201) > at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy > (DefaultRepositorySystem.java:392) > at org.apache.maven.plugins.deploy.AbstractDeployMojo.deploy > (AbstractDeployMojo.java:139) > at org.apache.maven.plugins.deploy.DeployMojo.execute > (DeployMojo.java:191) > at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo > (DefaultBuildPluginManager.java:126) > at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 > (MojoExecutor.java:342) > at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute > (MojoExecutor.java:330) > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > (MojoExecutor.java:213) > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > (MojoExecutor.java:175) > at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 > (MojoExecutor.java:76) > at org.apache.maven.lifecycle.internal.MojoExecutor$1.run > (MojoExecutor.java:163) > at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute > (DefaultMojosExecutionStrategy.java:39) > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > (MojoExecutor.java:160) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject > (LifecycleModuleBuilder.java:105) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject > (LifecycleModuleBuilder.java:73) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build > (SingleThreadedBuilder.java:53) > at org.apache.maven.lifecycle.internal.LifecycleStarter.execute > (LifecycleStarter.java:118) > at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261) > at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173) > at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101) > at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910) > at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283) > at org.apache.maven.cli.MavenCli.main (MavenCli.java:206) > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native > Method) > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke > (NativeMethodAccessorImpl.java:62) > at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke > (DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke (Method.java:566) > at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced > (Launcher.java:283) > at org.codehaus.plexus.classworlds.launcher.Launcher.launch > (Launcher.java:226) > at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode > (Launcher.java:407) > at org.codehaus.plexus.classworlds.launcher.Launcher.main > (Launcher.java:348) > >
