[jira] [Updated] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-11243: --- Status: Patch Available (was: Open) > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-11243-HDFS-10285-00.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-11243: --- Attachment: HDFS-11243-HDFS-10285-00.patch Attached initial patch to address. Please review. > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-11243-HDFS-10285-00.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11280) Allow WebHDFS to reuse HTTP connections to NN
[ https://issues.apache.org/jira/browse/HDFS-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15792232#comment-15792232 ] Brahma Reddy Battula commented on HDFS-11280: - *There are so many testcases are failing after this commit..Following is the stacktrace.* will you look into this..? {noformat} java.io.IOException: localhost:52693: Server returned HTTP response code: 403 for URL: http://localhost:52693/webhdfs/v1/srcdat/three/1925354346738733427?op=OPEN=bob=4096=0 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926) at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:664) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.connect(WebHdfsFileSystem.java:1997) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:740) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:584) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:615) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:611) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.read(WebHdfsFileSystem.java:1946) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.read(WebHdfsFileSystem.java:1804) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.read(WebHdfsFileSystem.java:1799) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.testAuditWebHdfsDenied(TestAuditLogs.java:249) {noformat} *FYR* https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/273/testReport/junit/ > Allow WebHDFS to reuse HTTP connections to NN > - > > Key: HDFS-11280 > URL: https://issues.apache.org/jira/browse/HDFS-11280 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Zheng Shao >Assignee: Zheng Shao > Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HDFS-11280.for.2.7.and.below.patch, > HDFS-11280.for.2.8.and.beyond.2.patch, HDFS-11280.for.2.8.and.beyond.3.patch, > HDFS-11280.for.2.8.and.beyond.4.patch, HDFS-11280.for.2.8.and.beyond.patch > > > WebHDFSClient calls "conn.disconnect()", which disconnects from the NameNode. > When we use webhdfs as the source in distcp, this used up all ephemeral > ports on the client side since all closed connections continue to occupy the > port with TIME_WAIT status for some time. > According to http://tinyurl.com/java7-http-keepalive, we should call > conn.getInputStream().close() instead to make sure the connection is kept > alive. This will get rid of the ephemeral port problem. > Manual steps used to verify the bug fix: > 1. Build original hadoop jar. > 2. Try out distcp from webhdfs as source, and "netstat -n | grep TIME_WAIT | > grep -c 50070" on the local machine shows a big number (100s). > 3. Build hadoop jar with this diff. > 4. Try out distcp from webhdfs as source, and "netstat -n | grep TIME_WAIT | > grep -c 50070" on the local machine shows 0. > 5. The explanation: distcp's client side does a lot of directory scanning, > which would create and close a lot of connections to the namenode HTTP port. > Reference: > 2.7 and below: > https://github.com/apache/hadoop/blob/branch-2.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L743 > 2.8 and above: >
[jira] [Updated] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[ https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-8741: - Status: Patch Available (was: Open) > Proper error msg to be printed when invalid operation type is given to > WebHDFS operations. > -- > > Key: HDFS-8741 > URL: https://issues.apache.org/jira/browse/HDFS-8741 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Minor > Attachments: HDFS-8741.001.patch > > > When wrong operation type is given to WebHDFS operations, following Error > message is printed -- > For ex: CREATE is called with GET instead of PUT-- > HTTP/1.1 400 Bad Request > .. > {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid > value for webhdfs parameter \"op\": {color:red}No enum constant > org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color} > Expected-- > Valid Error message to be printed -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8741) Proper error msg to be printed when invalid operation type is given to WebHDFS operations.
[ https://issues.apache.org/jira/browse/HDFS-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-8741: - Attachment: HDFS-8741.001.patch Thanks [~archanat] for reporting this issue.. Attached initial patch.. Please review.. > Proper error msg to be printed when invalid operation type is given to > WebHDFS operations. > -- > > Key: HDFS-8741 > URL: https://issues.apache.org/jira/browse/HDFS-8741 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Minor > Attachments: HDFS-8741.001.patch > > > When wrong operation type is given to WebHDFS operations, following Error > message is printed -- > For ex: CREATE is called with GET instead of PUT-- > HTTP/1.1 400 Bad Request > .. > {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid > value for webhdfs parameter \"op\": {color:red}No enum constant > org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.CREATE"}}{color} > Expected-- > Valid Error message to be printed -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org