[ https://issues.apache.org/jira/browse/HDFS-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701179#comment-16701179 ]
Siyao Meng commented on HDFS-14092: ----------------------------------- I've taken a look at the client. {code:java} /** Expects HTTP response 307 "Temporary Redirect". */ public static class TemporaryRedirectOp implements Op { static final TemporaryRedirectOp CREATE = new TemporaryRedirectOp( PutOpParam.Op.CREATE); static final TemporaryRedirectOp APPEND = new TemporaryRedirectOp( PostOpParam.Op.APPEND); static final TemporaryRedirectOp OPEN = new TemporaryRedirectOp( GetOpParam.Op.OPEN); static final TemporaryRedirectOp GETFILECHECKSUM = new TemporaryRedirectOp( GetOpParam.Op.GETFILECHECKSUM); ... {code} Only for those 4 operations the WebHDFS client would: 1. Send a request to server and expect 307 Temporary Redirect; 2. Grab the new URL from HTTP header Location key and send a second request to it (which points to a DN). WebHdfsFileSystem#connect(URL) performs the logic. So my understanding for the problem is that, the Java 6 library was faulty so it couldn't redirect the request automatically (like user browsers normally do when encountered HTTP 3xx). Therefore, we had to do it manually. CMIIW > Remove two-step create/append in WebHdfsFileSystem > -------------------------------------------------- > > Key: HDFS-14092 > URL: https://issues.apache.org/jira/browse/HDFS-14092 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs > Affects Versions: 3.2.0 > Reporter: Daniel Templeton > Assignee: Siyao Meng > Priority: Major > > Per the javadoc on the {{WebHdfsFileSystem.connect()}} method: > {code} /** > * Two-step requests redirected to a DN > * > * Create/Append: > * Step 1) Submit a Http request with neither auto-redirect nor data. > * Step 2) Submit another Http request with the URL from the Location > header > * with data. > * > * The reason of having two-step create/append is for preventing clients > to > * send out the data before the redirect. This issue is addressed by the > * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3. > * Unfortunately, there are software library bugs (e.g. Jetty 6 http > server > * and Java 6 http client), which do not correctly implement "Expect: > * 100-continue". The two-step create/append is a temporary workaround for > * the software library bugs. > * > * Open/Checksum > * Also implements two-step connects for other operations redirected to > * a DN such as open and checksum > */{code} > We should validate that it's safe to remove the two-step process and do so. > FYI, [~smeng]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org