[jira] [Resolved] (HDFS-5465) Update the package names for hsftp / hftp in the documentation

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-5465.
--
Resolution: Not A Problem

This is no longer an issue as hftp / hsftp have been deprecated.

> Update the package names for hsftp / hftp in the documentation
> --
>
> Key: HDFS-5465
> URL: https://issues.apache.org/jira/browse/HDFS-5465
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
>
> HDFS-5436 move HftpFileSystem and HsftpFileSystem to a different package. The 
> documentation should be updated as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-7239) Create a servlet for HDFS UI

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-7239.
--
Resolution: Won't Fix

It turns out that JMX is pretty good at providing such information. Closing 
this as won't fix.

> Create a servlet for HDFS UI
> 
>
> Key: HDFS-7239
> URL: https://issues.apache.org/jira/browse/HDFS-7239
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> Currently the HDFS UI gathers most of its information from JMX. There are a 
> couple disadvantages:
> * JMX is also used by management tools, thus Hadoop needs to maintain 
> compatibility across minor releases.
> * JMX organizes information as  pairs. The organization does not 
> fit well with emerging use cases like startup progress report and nntop.
> This jira proposes to introduce a new servlet in the NN for the purpose of 
> serving information to the UI.
> It should be viewed as a part of the UI. There is *no* compatibility 
> guarantees for the output of the servlet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-594) Add support for byte-ranges to hsftp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-594.
-
  Resolution: Won't Fix
Release Note: As hftp / hsftp have been deprecated a long time a ago. 
Closing this as won't fix.

> Add support for byte-ranges to hsftp
> 
>
> Key: HDFS-594
> URL: https://issues.apache.org/jira/browse/HDFS-594
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: hdfs-client
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Haohui Mai
>
> HsftpFileSystem should be modified to support byte-ranges so it has the same 
> semantics as HftpFileSystem after committing HDFS-235. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11588) Output Avro format in the offline editlog viewer

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-11588:
-

 Summary: Output Avro format in the offline editlog viewer
 Key: HDFS-11588
 URL: https://issues.apache.org/jira/browse/HDFS-11588
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


We found that it is handy to import the edit logs into query engines (e.g., 
Hive / Presto) to understand the usages of the cluster. Some examples include:

* The size of the data and the number of files that are written into a directory
* The distribution of the operations, for different directories.
* The number of files that are created by a user.

The answers to the above questions give insights on the usages of the clusters 
and have significant values on capacity planning.

Importing the edit log into query engines simplifies the tasks of answering 
these questions, and they can be answered efficiently.

While the Offline Editlog Viewer (OEV) supports outputting editlogs in XML 
formats, we found that it is time-consuming to transforming the XML format to 
formats that query engines recognize, because the generating the editlogs in 
XML formats and transforming them into formats that the query engine 
understands takes significant amount of time. In our environment it takes 
minutes to prepare a 100MB editlog file into a corresponding Parquet file.

This jira proposes to extend the OEV to output Avro files to make this process 
efficient. As an internal tool, the Avro output format has certain pre-defined 
schemas but it does not have the constraint of maintaining backward 
compatibility of the output, which is similar to the XML output format.







--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10184) Introduce unit tests framework for HDFS UI

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-10184:
-

 Summary: Introduce unit tests framework for HDFS UI
 Key: HDFS-10184
 URL: https://issues.apache.org/jira/browse/HDFS-10184
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Haohui Mai


The current HDFS UI is based on HTML5 and it does not have unit tests yet. 
Occasionally things break and we can't catch it. We should investigate and 
introduce unit test frameworks such as Mocha for the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9815) Move o.a.h.fs.Hdfs to hadoop-hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9815:


 Summary: Move o.a.h.fs.Hdfs to hadoop-hdfs-client
 Key: HDFS-9815
 URL: https://issues.apache.org/jira/browse/HDFS-9815
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai


the {{o.a.h.fs.Hdfs}} is used by the {{FileContext}} API. It should belong to 
the hadoop-hdfs-client module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9680) Doing a lsr against WebImageViewer is slow

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9680:


 Summary: Doing a lsr against WebImageViewer is slow
 Key: HDFS-9680
 URL: https://issues.apache.org/jira/browse/HDFS-9680
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai


We have experienced a performance issue that doing lsr against the 
WebImageViewer.

For a fsimage that has around 140m files, it takes around ~35 minutes to do the 
lsr across the live cluster, but ~12 hours to do the same operation against the 
WebImageViewer.

I believe that the root cause is that WebImageViewer decodes the protobuf 
messages on-demand which creates a lot of GC pressure. It might be better to 
decode it at the very beginning.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9380) HDFS-8707 builds are failing with protobuf directories as undef

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-9380.
--
Resolution: Duplicate

> HDFS-8707 builds are failing with protobuf directories as undef
> ---
>
> Key: HDFS-9380
> URL: https://issues.apache.org/jira/browse/HDFS-9380
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Bob Hansen
>Assignee: Haohui Mai
>
> See recent builds in HDFS-9320 and HDFS-9103.
> {code}
>  [exec] CMake Error: The following variables are used in this project, 
> but they are set to NOTFOUND.
>  [exec] Please set them or make sure they are set and tested correctly in 
> the CMake files:
>  [exec] PROTOBUF_LIBRARY (ADVANCED)
>  [exec] linked by target "protoc-gen-hrpc" in directory 
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto
>  [exec] linked by target "inputstream_test" in directory 
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
>  [exec] linked by target "remote_block_reader_test" in directory 
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
>  [exec] linked by target "rpc_engine_test" in directory 
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
>  [exec] PROTOBUF_PROTOC_LIBRARY (ADVANCED)
>  [exec] linked by target "protoc-gen-hrpc" in directory 
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9417) Clean up the RAT warnings in the HDFS-8707 branch.

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9417:


 Summary: Clean up the RAT warnings in the HDFS-8707 branch.
 Key: HDFS-9417
 URL: https://issues.apache.org/jira/browse/HDFS-9417
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Xiaobing Zhou


Recent jenkins builds reveals that the pom.xml in the HDFS-8707 branch does not 
currently exclude third-party files. The RAT plugin generates warnings as these 
files do not have Apache headers.

The warnings need to be suppressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9416:


 Summary: Respect OpenSSL and protobuf definitions in maven 
configuration when building libhdfspp
 Key: HDFS-9416
 URL: https://issues.apache.org/jira/browse/HDFS-9416
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Xiaobing Zhou
Priority: Blocker


As discovered in HDFS-9380 the current pom.xml / CMakeLists.txt in libhdfspp 
does not respect the configuration from the maven command line. Subsequently it 
breaks the Jenkins build.

Both pom.xml and CMakeLists.txt need to be fixed to get Jenkins working again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-8855) Webhdfs client leaks active NameNode connections

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai reopened HDFS-8855:
--

This patch breaks Jenkins around 2 weeks. I reverted this patch in trunk and 
branch-2 for now to keep Jenkins happy.

[~xiaobingo], can you please consolidate the effort of all the follow up jiras 
and post a new patch? Thanks.


> Webhdfs client leaks active NameNode connections
> 
>
> Key: HDFS-8855
> URL: https://issues.apache.org/jira/browse/HDFS-8855
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Reporter: Bob Hansen
>Assignee: Xiaobing Zhou
> Fix For: 2.8.0
>
> Attachments: HDFS-8855.005.patch, HDFS-8855.006.patch, 
> HDFS-8855.007.patch, HDFS-8855.1.patch, HDFS-8855.2.patch, HDFS-8855.3.patch, 
> HDFS-8855.4.patch, HDFS_8855.prototype.patch
>
>
> The attached script simulates a process opening ~50 files via webhdfs and 
> performing random reads.  Note that there are at most 50 concurrent reads, 
> and all webhdfs sessions are kept open.  Each read is ~64k at a random 
> position.  
> The script periodically (once per second) shells into the NameNode and 
> produces a summary of the socket states.  For my test cluster with 5 nodes, 
> it took ~30 seconds for the NameNode to have ~25000 active connections and 
> fails.
> It appears that each request to the webhdfs client is opening a new 
> connection to the NameNode and keeping it open after the request is complete. 
>  If the process continues to run, eventually (~30-60 seconds), all of the 
> open connections are closed and the NameNode recovers.  
> This smells like SoftReference reaping.  Are we using SoftReferences in the 
> webhdfs client to cache NameNode connections but never re-using them?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9367) Get libhdfs++ gmock tests running with CI

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-9367.
--
Resolution: Duplicate

This can be done in trunk. The issue should be fixed once HDFS-9369 lands.

> Get libhdfs++ gmock tests running with CI
> -
>
> Key: HDFS-9367
> URL: https://issues.apache.org/jira/browse/HDFS-9367
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Haohui Mai
>
> The gmock tests build with maven but there's no 'make test' target exposed to 
> maven/antrun.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9340) libhdfspp fails to compile after HDFS-9207

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9340:


 Summary: libhdfspp fails to compile after HDFS-9207
 Key: HDFS-9340
 URL: https://issues.apache.org/jira/browse/HDFS-9340
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


After the refactor of HDFS-9207 the {{hadoop-hdfs-client}} module fails to 
compile as it invokes {{cmake}} against a directory that does not exist. It 
should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6327) Clean up FSDirectory

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-6327.
--
Resolution: Fixed

Closing this jira as all the subtasks have been completed.

> Clean up FSDirectory
> 
>
> Key: HDFS-6327
> URL: https://issues.apache.org/jira/browse/HDFS-6327
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This is an umbrella jira that coves the clean up work on the FSDirectory 
> class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8128) hadoop-hdfs-client dependency convergence error

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8128.
--
Resolution: Cannot Reproduce

Resolving this issue. It looks like it is no longer reproducible in trunk. 

> hadoop-hdfs-client dependency convergence error
> ---
>
> Key: HDFS-8128
> URL: https://issues.apache.org/jira/browse/HDFS-8128
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: build
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Haohui Mai
>
> Found the following in 
> https://builds.apache.org/job/PreCommit-HDFS-Build/10258/consoleFull
> {noformat}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence 
> failed with message:
> Failed while enforcing releasability the error(s) are [
> Dependency convergence error for 
> org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT paths to dependency are:
> +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT
>   +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
> and
> +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT
>   +-org.apache.hadoop:hadoop-annotations:3.0.0-20150410.234534-6484
> ]
> [INFO] 
> 
> [INFO] BUILD FAILURE
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9253) Refactor tests of libhdfs into a directory

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9253:


 Summary: Refactor tests of libhdfs into a directory
 Key: HDFS-9253
 URL: https://issues.apache.org/jira/browse/HDFS-9253
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to refactor the current tests in libhdfs into a separate 
directory. The refactor opens up the opportunity to reuse tests in libhdfs, 
libwebhdfs and libhdfspp in HDFS-8707 and to also allow cross validation of 
different implementation of the libhdfs API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9207) Move the implementation to the hdfs-native-client module

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-9207.
--
Resolution: Fixed

Committed to the HDFS-8707 branch. Thanks James and Bob for the reviews!

> Move the implementation to the hdfs-native-client module
> 
>
> Key: HDFS-9207
> URL: https://issues.apache.org/jira/browse/HDFS-9207
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9207.000.patch
>
>
> The implementation of libhdfspp should be moved to the new hdfs-native-client 
> module as HDFS-9170 has landed in trunk and branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9222) Add hadoop-hdfs-client as a dependency of hadoop-hdfs-native-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9222:


 Summary: Add hadoop-hdfs-client as a dependency of 
hadoop-hdfs-native-client
 Key: HDFS-9222
 URL: https://issues.apache.org/jira/browse/HDFS-9222
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Mingliang Liu


libhdfs requires the jars in hadoop-hdfs-client in order to work. This jira 
proposes to add the missing dependency in hadoop-hdfs-native-client.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9215) Suppress the RAT warnings in hdfs-native-client module

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9215:


 Summary: Suppress the RAT warnings in hdfs-native-client module
 Key: HDFS-9215
 URL: https://issues.apache.org/jira/browse/HDFS-9215
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor


HDFS-9170 moves the native client implementation to the hdfs-native-client 
module. This is a follow-up jira to suppress the RAT warning that was 
suppressed in the original hadoop-hdfs module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9207) Move the implementation to the hdfs-native-client module

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9207:


 Summary: Move the implementation to the hdfs-native-client module
 Key: HDFS-9207
 URL: https://issues.apache.org/jira/browse/HDFS-9207
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai


The implementation of libhdfspp should be moved to the new hdfs-native-client 
module as HDFS-9170 has landed in trunk and branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9170) Move libhdfs / fuse-dfs / libwebhdfs to a separate module

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9170:


 Summary: Move libhdfs / fuse-dfs / libwebhdfs to a separate module
 Key: HDFS-9170
 URL: https://issues.apache.org/jira/browse/HDFS-9170
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


After HDFS-6200 the Java implementation of hdfs-client has be moved to a 
separate hadoop-hdfs-client module.

libhdfs, fuse-dfs and libwebhdfs still reside in the hadoop-hdfs module. 
Ideally these modules should reside in the hadoop-hdfs-client. However, to 
write unit tests for these components, it is often necessary to run 
MiniDFSCluster which resides in the hadoop-hdfs module.

This jira is to discuss how these native modules should layout after HDFS-6200.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9165) Move the rest of the entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9165:


 Summary: Move the rest of the entries in 
META-INF/services/o.a.h.fs.FileSystem to hdfs-client
 Key: HDFS-9165
 URL: https://issues.apache.org/jira/browse/HDFS-9165
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai


After HDFS-8740 the entries in META-INF/services/o.a.h.fs.FileSystem should be 
updated accordingly similar to HDFS-9041.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9166) Move hftp / hsftp filesystem to hfds-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9166:


 Summary: Move hftp / hsftp filesystem to hfds-client
 Key: HDFS-9166
 URL: https://issues.apache.org/jira/browse/HDFS-9166
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai


The hftp / hsftp filesystems in branch-2 need to be moved to the hdfs-client 
module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9168) Move client side uni test to hadoop-hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9168:


 Summary: Move client side uni test to hadoop-hdfs-client
 Key: HDFS-9168
 URL: https://issues.apache.org/jira/browse/HDFS-9168
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


We need to identify and move the unit tests on the client of hdfs to the 
hdfs-client module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9167) Update pom.xml in other modules to depend on hdfs-client instead of hdfs

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9167:


 Summary: Update pom.xml in other modules to depend on hdfs-client 
instead of hdfs
 Key: HDFS-9167
 URL: https://issues.apache.org/jira/browse/HDFS-9167
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai


Since now the implementation of the client has been moved to the 
hadoop-hdfs-client, we should update the poms of other modules in hadoop to use 
hdfs-client instead of hdfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7416) Revisit the abstraction between NameNodeRpcServer, FSNameSystem and FSDirectory

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-7416.
--
Resolution: Fixed

The desired code refactoring has completed. Closing this jira.

Thanks everyone for the work!

> Revisit the abstraction between NameNodeRpcServer, FSNameSystem and 
> FSDirectory
> ---
>
> Key: HDFS-7416
> URL: https://issues.apache.org/jira/browse/HDFS-7416
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> {{NameNodeRpcServer}}, {{FSNameSystem}} and {{FSDirectory}} implement the 
> namespace of the NN. In the current implementation the boundary of these 
> classes are not fully clear.
> This jira tracks the efforts of clarifying the boundaries between these three 
> classes so that they can be more easily maintained in the long term.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9129) Move the safemode block count into BlockManager

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9129:


 Summary: Move the safemode block count into BlockManager
 Key: HDFS-9129
 URL: https://issues.apache.org/jira/browse/HDFS-9129
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Mingliang Liu


The {{SafeMode}} needs to track whether there are enough blocks so that the NN 
can get out of the safemode. These fields can moved to the {{BlockManager}} 
class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9115) Create documentation to describe the overall architecture and rationales of the library

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9115:


 Summary: Create documentation to describe the overall architecture 
and rationales of the library
 Key: HDFS-9115
 URL: https://issues.apache.org/jira/browse/HDFS-9115
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707


It's beneficial to have documentations to describe the design decisions and 
rationales of the library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9116) Suppress false positives from Valgrind on uninitialized variables in tests

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9116:


 Summary: Suppress false positives from Valgrind on uninitialized 
variables in tests
 Key: HDFS-9116
 URL: https://issues.apache.org/jira/browse/HDFS-9116
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor


Valgrind complains about uninitialized variables in the unit tests. It should 
be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9095) RPC client should fail gracefully when the connection is timed out or reset

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9095:


 Summary: RPC client should fail gracefully when the connection is 
timed out or reset
 Key: HDFS-9095
 URL: https://issues.apache.org/jira/browse/HDFS-9095
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


The RPC client should fail gracefully when the connection is timed out or 
reset. instead of bailing out. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9093) Initialize protobuf fields in RemoteBlockReaderTest

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9093:


 Summary: Initialize protobuf fields in RemoteBlockReaderTest
 Key: HDFS-9093
 URL: https://issues.apache.org/jira/browse/HDFS-9093
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


Protobuf 2.6.1 complains that the {{ExtendedBlockProto}} objects in 
{{remote_block_reader_test.cc}} are not initialized.

The test should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9067) o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9067:


 Summary: o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter 
is failing in trunk
 Key: HDFS-9067
 URL: https://issues.apache.org/jira/browse/HDFS-9067
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai


The test TestLazyWriter is consistently failing in trunk. For example:

https://builds.apache.org/job/PreCommit-HDFS-Build/12407/testReport/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8952) InputStream.PositionRead() should be aware of available DNs

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8952.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks James and Bob for the reviews.

> InputStream.PositionRead() should be aware of available DNs
> ---
>
> Key: HDFS-8952
> URL: https://issues.apache.org/jira/browse/HDFS-8952
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: HDFS-8707
>
> Attachments: HDFS-8952.000.patch
>
>
> This jira provides basic functionality to allow libraries to recover from DN 
> failures.
> {{InputStream.PositionRead()}} should (1) report the DN that when it serves 
> reads, and (2) allows the users exclude dead nodes during reads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9048) DistCp documentation is out-of-dated

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9048:


 Summary: DistCp documentation is out-of-dated
 Key: HDFS-9048
 URL: https://issues.apache.org/jira/browse/HDFS-9048
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai


There are a couple issues with the current distcp document:

* It recommends hftp / hsftp filesystem to copy data between different hadoop 
version. hftp / hsftp have been deprecated in the flavor of webhdfs.
* If the users are copying between Hadoop 2.x they can use the hdfs protocol 
directly for better performance.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9045) DatanodeHttpServer is not setting Endpoint based on configured policy and not loading ssl configuration.

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-9045.
--
Resolution: Invalid

This is invalid as the Netty serves as a reverse proxy for the Jetty server. 
The jetty server is supposed to listen to localhost only.

> DatanodeHttpServer is not setting Endpoint based on configured policy and not 
> loading ssl configuration.
> 
>
> Key: HDFS-9045
> URL: https://issues.apache.org/jira/browse/HDFS-9045
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Surendra Singh Lilhore
>Priority: Critical
>
> Always DN is starting in http mode.
> {code}
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0;))
> .setFindPort(true);
> {code}
> Should be based on configured policy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9041) Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-9041:


 Summary: Move entries in META-INF/services/o.a.h.fs.FileSystem to 
hdfs-client
 Key: HDFS-9041
 URL: https://issues.apache.org/jira/browse/HDFS-9041
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Mingliang Liu


This is a follow up of HDFS-8052. It looks like HDFS-8052 breaks  
{{TestLocalJobSubmission}} in MR.

HDFS-8052 moves the implementation of {{WebHdfsFileSystem}} and 
{{SWebHdfsFileSystem}} to hdfs-client.

According the usage of ServiceLoader, the corresponding entries in 
{{hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
 should be moved as well.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8991) Provide information on BPOfferService in DN JMX and UI

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8991:


 Summary: Provide information on BPOfferService in DN JMX and UI
 Key: HDFS-8991
 URL: https://issues.apache.org/jira/browse/HDFS-8991
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Mingliang Liu


In cases like HDFS-7714 where the BPOfferService thread is missing, which 
require nontrivial effort to debug that which NN that the DN thinks it active / 
standby.

It would make sense to make the information more accessible through JMX or Web 
UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8970) Clean up checkstyle warnings in shortcircuit package

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8970.
--
Resolution: Duplicate

The granularity of cleaning things up might be too small -- let's combine the 
effort with HDFS-8979.

 Clean up checkstyle warnings in shortcircuit package
 

 Key: HDFS-8970
 URL: https://issues.apache.org/jira/browse/HDFS-8970
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu

 We moved the {{shortcircuit}} package from {{hadoop-hdfs}} to 
 {{hadoop-hdfs-client}} module in JIRA 
 [HDFS-8934|https://issues.apache.org/jira/browse/HDFS-8934] and 
 [HDFS-8951|https://issues.apache.org/jira/browse/HDFS-8951]. There are some 
 checkstyle issues which are not fixed in those commits since they only 
 tracked the effort of moving.
 This jira tracks the effort of fixing the checkstyle warnings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-8938) Extract BlockToMarkCorrupt and ReplicationWork as standalone classes from BlockManager

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai reopened HDFS-8938:
--

It looks like the Jenkins report is on the v6 patch instead of v7 patch.
Although I don't see significant differences between them, I'm going to revert 
this patch in branch-2 and trunk for now. I'll recommit it once the Jenkins 
report is back.

 Extract BlockToMarkCorrupt and ReplicationWork as standalone classes from 
 BlockManager
 --

 Key: HDFS-8938
 URL: https://issues.apache.org/jira/browse/HDFS-8938
 Project: Hadoop HDFS
  Issue Type: Task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
 Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch, 
 HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch, 
 HDFS-8938.005.patch, HDFS-8938.006.patch, HDFS-8938.007.patch


 This jira proposes to refactor two inner static classes, 
 {{BlockToMarkCorrupt}} and {{ReplicationWork}} from {{BlockManager}} to 
 standalone classes. The refactor also improves readability by abstracting the 
 complexity of scheduling and validating replications to corresponding helper 
 methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai reopened HDFS-8982:
--

 Consolidate getFileReplication and getPreferredBlockReplication in INodeFile
 

 Key: HDFS-8982
 URL: https://issues.apache.org/jira/browse/HDFS-8982
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.7.1
Reporter: Zhe Zhang

 Currently {{INodeFile}} provides both {{getFileReplication}} and 
 {{getPreferredBlockReplication}} interfaces. At the very least they should be 
 renamed (e.g. {{getCurrentFileReplication}} and 
 {{getMaxConfiguredFileReplication}}), with clearer Javadoc.
 I also suspect we are not using them correctly in all places right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8984) Move replication queues related methods in FSNamesystem to BlockManager

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8984:


 Summary: Move replication queues related methods in FSNamesystem 
to BlockManager
 Key: HDFS-8984
 URL: https://issues.apache.org/jira/browse/HDFS-8984
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


Currently {{FSNamesystem}} controls whether replication queue should be 
populated based on whether the NN is in safe mode or whether it is an active NN.

Replication is a concept on the block management layer. It is more natural to 
place the functionality in the {{BlockManager}} class.

This jira proposes to move the these methods to the {{BlockManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8982.
--
Resolution: Won't Fix

The behavior is expected.

{{getFileReplication()}} records the replication factor of the current 
(w.r.t. snapshot state) replicate factor of the file.

The so-called {{getPerferredBlockReplication()}} records the maximum 
replication factor of the file w.r.t. the current and all snapshot state of the 
file.

 Consolidate getFileReplication and getPreferredBlockReplication in INodeFile
 

 Key: HDFS-8982
 URL: https://issues.apache.org/jira/browse/HDFS-8982
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.7.1
Reporter: Zhe Zhang

 Currently {{INodeFile}} provides both {{getFileReplication}} and 
 {{getPreferredBlockReplication}} interfaces. At the very least they should be 
 renamed (e.g. {{getCurrentFileReplication}} and 
 {{getMaxConfiguredFileReplication}}), with clearer Javadoc.
 I also suspect we are not using them correctly in all places right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8982) Consolidate getFileReplication and getPreferredBlockReplication in INodeFile

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8982.
--
Resolution: Invalid

 Consolidate getFileReplication and getPreferredBlockReplication in INodeFile
 

 Key: HDFS-8982
 URL: https://issues.apache.org/jira/browse/HDFS-8982
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.7.1
Reporter: Zhe Zhang

 Currently {{INodeFile}} provides both {{getFileReplication}} and 
 {{getPreferredBlockReplication}} interfaces. At the very least they should be 
 renamed (e.g. {{getCurrentFileReplication}} and 
 {{getMaxConfiguredFileReplication}}), with clearer Javadoc.
 I also suspect we are not using them correctly in all places right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8961) Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8961:


 Summary: Investigate lock issue in 
o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager
 Key: HDFS-8961
 URL: https://issues.apache.org/jira/browse/HDFS-8961
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Mingliang Liu


There are two clauses in {{hadoop-hdfs}} to filter out the findbugs warnings in 
{{org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager}}:

{code}
Match
  Class 
name=org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager
 /
  Method name=allocSlot /
  Bug pattern=UL_UNRELEASED_LOCK_EXCEPTION_PATH /
/Match
Match
  Class 
name=org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager
 /
  Method name=allocSlot /
  Bug pattern=UL_UNRELEASED_LOCK /
/Match
{code}

These two warnings show up in the Jenkins run as these classes are moved into 
the {{hadoop-hdfs-client}} module. We either need to fix the code or move these 
clauses to the {{hadoop-hdfs-client}} module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8963) Fix findbugs warnings introduced in HDFS-8900

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8963:


 Summary: Fix findbugs warnings introduced in HDFS-8900
 Key: HDFS-8963
 URL: https://issues.apache.org/jira/browse/HDFS-8963
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai


HDFS-8900 introduced two new findbugs warnings:

https://builds.apache.org/job/PreCommit-HDFS-Build/12120/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8966) Separate the lock used in namespace and block management layer

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8966:


 Summary: Separate the lock used in namespace and block management 
layer
 Key: HDFS-8966
 URL: https://issues.apache.org/jira/browse/HDFS-8966
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


Currently the namespace and the block management layer share one giant lock. 
One consequence that we have seen more and more often is that the namespace 
hangs due to excessive activities from the block management layer. For example, 
the NN might take a couple hundred milliseconds to handle a large block report. 
Because the NN holds the write lock during processing the block report, all 
namespace requests are paused. In production we have seen these lock 
contentions cause long latencies and instabilities in the cluster.

This umbrella jira proposes to separate the lock used by namespace and the 
block management layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8967:


 Summary: Create a BlockManagerLock class to represent the lock 
used in the BlockManager
 Key: HDFS-8967
 URL: https://issues.apache.org/jira/browse/HDFS-8967
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to create a {{BlockManagerLock}} class to represent the lock 
used in {{BlockManager}}.

Currently it directly points to the {{FSNamesystem}} lock thus there are no 
functionality changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8934) Move ShortCircuitShm to hdfs-client

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8934.
--
   Resolution: Fixed
Fix Version/s: (was: 3.0.0)
   2.8.0

Fixed the merge issue and recommitted the patch in branch-2.

 Move ShortCircuitShm to hdfs-client
 ---

 Key: HDFS-8934
 URL: https://issues.apache.org/jira/browse/HDFS-8934
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
 Fix For: 2.8.0

 Attachments: HDFS-8934.000.patch, HDFS-8934.001.patch


 This jira tracks the effort of moving the {{ShortCircuitShm}} class into the 
 hdfs-client module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8943.
--
Resolution: Invalid

 Read apis in ByteRangeInputStream does not read all the bytes specified when 
 chunked transfer-encoding is used in the server
 

 Key: HDFS-8943
 URL: https://issues.apache.org/jira/browse/HDFS-8943
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
 Attachments: HDFS-8943.000.patch


 With the default Webhdfs server implementation the read apis in 
 ByteRangeInputStream work as expected reading the correct number of bytes for 
 these apis :
 {{public int read(byte b[], int off, int len)}}
 {{public int read(long position, byte[] buffer, int offset, int length)}}
 But when a custom Webhdfs server implementation is plugged in which uses 
 chunked Transfer-encoding, these apis read only the first chunk. Simple fix 
 would be to loop and read till bytes specified similar to {{readfully()}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8952) InputStream.PositionRead() should be aware of available DNs

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8952:


 Summary: InputStream.PositionRead() should be aware of available 
DNs
 Key: HDFS-8952
 URL: https://issues.apache.org/jira/browse/HDFS-8952
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira provides basic functionality to allow libraries to recover from DN 
failures.

{{InputStream.PositionRead()}} should (1) report the DN that when it serves 
reads, and (2) allows the users exclude dead nodes during reads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8774) Implement FileSystem and InputStream API for libhdfspp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8774.
--
  Resolution: Fixed
Hadoop Flags: Reviewed

Committed to the HDFS-8707 branch. Thanks James and Bob for the reviews!

 Implement FileSystem and InputStream API for libhdfspp
 --

 Key: HDFS-8774
 URL: https://issues.apache.org/jira/browse/HDFS-8774
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8774.000.patch


 This jira proposes to implement FileSystem and InputStream APIs for libhdfspp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8775) SASL support for data transfer protocol in libhdfspp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8775.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks James and Bob for the reviews!

 SASL support for data transfer protocol in libhdfspp
 

 Key: HDFS-8775
 URL: https://issues.apache.org/jira/browse/HDFS-8775
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8775.000.patch


 This jira proposes to implement basic SASL support for the data transfer 
 protocol which allows libhdfspp to talk to secure clusters.
 Support for encryption is deferred to subsequent jiras.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6290) File is not closed in OfflineImageViewerPB#run()

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-6290.
--
Resolution: Won't Fix

I don't think this is worth fixing as the life cycle of the file closely 
matches the life cycle of the process. The file will be automatically closed 
when the process exits.

 File is not closed in OfflineImageViewerPB#run()
 

 Key: HDFS-6290
 URL: https://issues.apache.org/jira/browse/HDFS-6290
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Reporter: Ted Yu
Priority: Minor

 {code}
   } else if (processor.equals(XML)) {
 new PBImageXmlWriter(conf, out).visit(new RandomAccessFile(inputFile,
 r));
 {code}
 The RandomAccessFile instance should be closed before the method returns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8052) Move WebHdfsFileSystem into hadoop-hdfs-client

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8052.
--
Resolution: Fixed

Closing this jira.

{{RetryUtils}} is not annotated as a public API thus it might change as the 
project evolve. This is not an incompatible change as it is an internal 
implementation detail.

It looks like that the fix in SLIDER-923 is correct to me. Do I miss anything?

 Move WebHdfsFileSystem into hadoop-hdfs-client
 --

 Key: HDFS-8052
 URL: https://issues.apache.org/jira/browse/HDFS-8052
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: build
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: 2.8.0

 Attachments: HDFS-8052.000.patch, HDFS-8052.001.patch, 
 HDFS-8052.002.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8888) Support the volume concepts in HDFS

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-:


 Summary: Support the volume concepts in HDFS
 Key: HDFS-
 URL: https://issues.apache.org/jira/browse/HDFS-
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai


There are multiple types of zones (e.g., snapshot, encryption zone) which are 
conceptually close to namespace volumes in traditional filesystems.

This jira proposes to introduce the concept of volume to simplify the 
implementation of snapshots and encryption zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8823) Move replication factor into individual blocks

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8823:


 Summary: Move replication factor into individual blocks
 Key: HDFS-8823
 URL: https://issues.apache.org/jira/browse/HDFS-8823
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to record the replication factor in the {{BlockInfo}} class. 
The changes have two advantages:

* Decoupling the namespace and the block management layer. It is a prerequisite 
step to move block management off the heap or to a separate process.
* Increased flexibility on replicating blocks. Currently the replication 
factors of all blocks have to be the same. The replication factors of these 
blocks are equal to the highest replication factor across all snapshots. The 
changes will allow blocks in a file to have different replication factor, 
potentially saving some space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8816) Improve visualization for the Datanode tab in the NN UI

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8816:


 Summary: Improve visualization for the Datanode tab in the NN UI
 Key: HDFS-8816
 URL: https://issues.apache.org/jira/browse/HDFS-8816
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


The information of the datanode tab in the NN UI is clogged. This jira proposes 
to improve the visualization of the datanode tab in the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8803) Move DfsClientConf to hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8803:


 Summary: Move DfsClientConf to hdfs-client
 Key: HDFS-8803
 URL: https://issues.apache.org/jira/browse/HDFS-8803
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira tracks the effort of moving the {{DfsClientConf}} class into the 
hdfs-client module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8764) Generate Hadoop RPC stubs from protobuf definitions

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8764.
--
   Resolution: Fixed
Fix Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks Jing and James for the reviews.

 Generate Hadoop RPC stubs from protobuf definitions
 ---

 Key: HDFS-8764
 URL: https://issues.apache.org/jira/browse/HDFS-8764
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8764.000.patch


 It would be nice to have the the RPC stubs generated from the protobuf 
 definitions which is similar to what the HADOOP-10388 has achieved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8788) Implement unit tests for remote block reader in libhdfspp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8788.
--
   Resolution: Fixed
Fix Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks James for the reviews.

 Implement unit tests for remote block reader in libhdfspp
 -

 Key: HDFS-8788
 URL: https://issues.apache.org/jira/browse/HDFS-8788
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8788.000.patch


 This jira proposes to implement unit tests for the remote block reader in 
 gmock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8788) Implement unit tests for remote block reader in libhdfspp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8788:


 Summary: Implement unit tests for remote block reader in libhdfspp
 Key: HDFS-8788
 URL: https://issues.apache.org/jira/browse/HDFS-8788
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to implement unit tests for the remote block reader in gmock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8759) Implement remote block reader in libhdfspp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8759.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks Jing and James for the reviews.

 Implement remote block reader in libhdfspp
 --

 Key: HDFS-8759
 URL: https://issues.apache.org/jira/browse/HDFS-8759
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8759.000.patch, HDFS-8759.001.patch, 
 HDFS-8759.002.patch


 This jira tracks the effort of implementing the remote block reader that 
 communicates with DN in libhdfspp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8782) Upgrade to block ID-based DN storage layout delays DN registration

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8782:


 Summary: Upgrade to block ID-based DN storage layout delays DN 
registration
 Key: HDFS-8782
 URL: https://issues.apache.org/jira/browse/HDFS-8782
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Priority: Critical


We have seen multiple incidents at production sites that there are long delays 
for DNs to register to the NN when upgrading to post 2.6 release.

Further investigation shows that the DN is blocked when upgrading the storage 
layout introduced in HDFS-6482. The new storage layout requires making up to 
64k directories in the underlying file system. Unfortunately the current 
implementation calls {{mkdirs()}} sequentially and upgrades each volume in 
sequential order.

As a result, upgrading a DN with a lot of disks or with blocks that have random 
block ID takes a long time (usually in hours), and the DN won't register to the 
NN unless it finishes upgrading all the storage directory. The excessive delays 
confuse operations and break the assumption of rolling upgrades.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8774) Implement FileSystem and InputStream API for libhdfspp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8774:


 Summary: Implement FileSystem and InputStream API for libhdfspp
 Key: HDFS-8774
 URL: https://issues.apache.org/jira/browse/HDFS-8774
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707


This jira proposes to implement FileSystem and InputStream APIs for libhdfspp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8775) SASL support for data transfer protocol in libhdfspp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8775:


 Summary: SASL support for data transfer protocol in libhdfspp
 Key: HDFS-8775
 URL: https://issues.apache.org/jira/browse/HDFS-8775
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to implement basic SASL support for the data transfer 
protocol which allows libhdfspp to talk to secure clusters.

Support for encryption is deferred to subsequent jiras.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8758) Implement the continuation library for libhdfspp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8758.
--
  Resolution: Fixed
Hadoop Flags: Reviewed
   Fix Version/s: HDFS-8707
Target Version/s: HDFS-8707

Committed to HDFS-8707. Thanks Jing for reviews.

 Implement the continuation library for libhdfspp
 

 Key: HDFS-8758
 URL: https://issues.apache.org/jira/browse/HDFS-8758
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8758.000.patch


 libhdfspp uses continuations as basic building blocks to implement 
 asynchronous operations. This jira imports the continuation library into the 
 repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8764) Generate Hadoop RPC stubs from protobuf definitions

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8764:


 Summary: Generate Hadoop RPC stubs from protobuf definitions
 Key: HDFS-8764
 URL: https://issues.apache.org/jira/browse/HDFS-8764
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


It would be nice to have the the RPC stubs generated from the protobuf 
definitions which is similar to what the HADOOP-10388 has achieved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8767) RawLocalFileSystem.listStatus() returns null for UNIX pipefile

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8767:


 Summary: RawLocalFileSystem.listStatus() returns null for UNIX 
pipefile
 Key: HDFS-8767
 URL: https://issues.apache.org/jira/browse/HDFS-8767
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Priority: Critical


Calling FileSystem.listStatus() on a UNIX pipe file returns null instead of the 
file. The bug breaks Hive when Hive loads data from UNIX pipe file.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8758) Implement the continuation library for libhdfspp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8758:


 Summary: Implement the continuation library for libhdfspp
 Key: HDFS-8758
 URL: https://issues.apache.org/jira/browse/HDFS-8758
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


libhdfspp uses continuations as basic building blocks to implement asynchronous 
operations. This jira imports the continuation library into the repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8745) Use Doxygen to generate documents for libhdfspp

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8745.
--
  Resolution: Fixed
Hadoop Flags: Reviewed
   Fix Version/s: HDFS-8707
Target Version/s: HDFS-8707

Committed to HDFS-8707 branch. Thank Jing for the reviews.

 Use Doxygen to generate documents for libhdfspp
 ---

 Key: HDFS-8745
 URL: https://issues.apache.org/jira/browse/HDFS-8745
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
 Fix For: HDFS-8707

 Attachments: HDFS-8745.000.patch, HDFS-8745.001.patch


 This jira proposes to add Doxygen hooks to generate documentation for the 
 library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8759) Implement remote block reader in libhdfspp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8759:


 Summary: Implement remote block reader in libhdfspp
 Key: HDFS-8759
 URL: https://issues.apache.org/jira/browse/HDFS-8759
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira tracks the effort of implementing the remote block reader that 
communicates with DN in libhdfspp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8745) Use Doxygen to generate documents

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8745:


 Summary: Use Doxygen to generate documents
 Key: HDFS-8745
 URL: https://issues.apache.org/jira/browse/HDFS-8745
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor


This jira proposes to add Doxygen hooks to generate documentation for the 
library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8737) Implement the Hadoop RPC v9 protocol

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8737:


 Summary: Implement the Hadoop RPC v9 protocol
 Key: HDFS-8737
 URL: https://issues.apache.org/jira/browse/HDFS-8737
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira tracks the effort of implementing the Hadoop RPC v9 protocol.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8723) Integrate the build infrastructure with hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8723:


 Summary: Integrate the build infrastructure with hdfs-client
 Key: HDFS-8723
 URL: https://issues.apache.org/jira/browse/HDFS-8723
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to integrate the build infrastructures of libhdfspp with the 
one in hdfs-client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8724) Import third_party libraries into the repository

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8724:


 Summary: Import third_party libraries into the repository
 Key: HDFS-8724
 URL: https://issues.apache.org/jira/browse/HDFS-8724
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


libhdfspp currently uses the gtest and asio library. This jira tracks the 
effort of importing them into the repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8725) Use std::chrono to implement the timer in the asio library

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8725:


 Summary: Use std::chrono to implement the timer in the asio library
 Key: HDFS-8725
 URL: https://issues.apache.org/jira/browse/HDFS-8725
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


Currently the asio library uses boost to implement its timer service. Since the 
date time API is available in C++11, this jira proposes to implement the timer 
service with C++11's timer when boost is unavailable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8724) Import third_party libraries into the repository

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8724.
--
  Resolution: Fixed
   Fix Version/s: HDFS-8707
Target Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks Jing for the reviews.

 Import third_party libraries into the repository
 

 Key: HDFS-8724
 URL: https://issues.apache.org/jira/browse/HDFS-8724
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8724.000.patch, HDFS-8724.001.patch


 libhdfspp currently uses the gtest and asio library. This jira tracks the 
 effort of importing them into the repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8725) Use std::chrono to implement the timer in the asio library

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8725.
--
  Resolution: Fixed
   Fix Version/s: HDFS-8707
Target Version/s: HDFS-8707

Committed to the HDFS-8707 branch. Thanks Jing for the reviews.

 Use std::chrono to implement the timer in the asio library
 --

 Key: HDFS-8725
 URL: https://issues.apache.org/jira/browse/HDFS-8725
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client
Reporter: Haohui Mai
Assignee: Haohui Mai
 Fix For: HDFS-8707

 Attachments: HDFS-8725.000.patch


 Currently the asio library uses boost to implement its timer service. Since 
 the date time API is available in C+\+11, this jira proposes to implement the 
 timer service with C\++11's timer when boost is unavailable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8726) Move protobuf files that define the client-sever protocols to hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8726:


 Summary: Move protobuf files that define the client-sever 
protocols to hdfs-client
 Key: HDFS-8726
 URL: https://issues.apache.org/jira/browse/HDFS-8726
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


The protobuf files that defines the RPC protocols between the HDFS clients and 
servers current sit in the hdfs package. They should be moved the the 
hdfs-client package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8592) SafeModeException never get unwrapped

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8592:


 Summary: SafeModeException never get unwrapped
 Key: HDFS-8592
 URL: https://issues.apache.org/jira/browse/HDFS-8592
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


{{RemoteException#unwrapRemoteException}} fails to instantiate 
{{SafeModeException}} because {{SafeModeException}} does not have the 
corresponding constructor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8588) DN should not support SPNEGO authenticator

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8588:


 Summary: DN should not support SPNEGO authenticator
 Key: HDFS-8588
 URL: https://issues.apache.org/jira/browse/HDFS-8588
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai


Currently {{HttpServer2}} initializes SPNEGO authentication filter for all 
HttpServer instances. However, DNs are not supposed to initialize any SPNEGO 
authentication handler.

The class needs to be refactor to support this use case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8590) Surpress bad_cert SSLException and provide more information in the DN log

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8590:


 Summary: Surpress bad_cert SSLException and provide more 
information in the DN log
 Key: HDFS-8590
 URL: https://issues.apache.org/jira/browse/HDFS-8590
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai


The Netty server in DN throws a long list of exception when the client does not 
trust the certificate of the server.

This jira proposes to surpress the exception and to print out the origin of the 
request to ease the debugging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8572) DN uses incorrect kerberos principals in spnego authentication

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8572:


 Summary: DN uses incorrect kerberos principals in spnego 
authentication
 Key: HDFS-8572
 URL: https://issues.apache.org/jira/browse/HDFS-8572
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Blocker


After HDFS-7279, the DN always uses {{HTTP/locahost@REALM}} to authenticate 
spnego requests, which breaks all the security deployments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8493) Consolidate truncate() related implementation in a single class

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8493:


 Summary: Consolidate truncate() related implementation in a single 
class
 Key: HDFS-8493
 URL: https://issues.apache.org/jira/browse/HDFS-8493
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai


This jira proposes to consolidate truncate() related methods into a single 
class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8135) Remove the deprecated FSConstants class

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8135.
--
  Resolution: Fixed
   Fix Version/s: (was: 2.8.0)
  3.0.0
Target Version/s:   (was: 2.8.0)

 Remove the deprecated FSConstants class
 ---

 Key: HDFS-8135
 URL: https://issues.apache.org/jira/browse/HDFS-8135
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
 Fix For: 3.0.0

 Attachments: HDFS-8135-041315.patch


 The {{FSConstants}} class has been marked as deprecated since 0.23. There is 
 no uses of this class in the current code base and it can be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-8452) In WebHDFS, duplicate directory creation is not throwing exception.

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-8452.
--
Resolution: Not A Problem

The behavior is expected. in HDFS {{mkdirs()}} is an idempotent operation, 
meaning executing multiple {{mkdirs()}} for the same directory will succeed.

 In WebHDFS, duplicate directory creation is not throwing exception.
 ---

 Key: HDFS-8452
 URL: https://issues.apache.org/jira/browse/HDFS-8452
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Jagadesh Kiran N
Priority: Minor
 Fix For: 3.0.0


 *Case 1 (CLI):*
 a. In HDFS Create a new Directory 
   {code}./hdfs dfs -mkdir /new  , A New directory will be 
 created{code}
b. Now Execute the same Command again 
 {code}   mkdir: `/new': File exists  , Error message will be shown  {code}
 *Case 2 (RestAPI) :*
 a. In HDFS Create a new Directory
  {code}curl -i -X PUT -L 
 http://host1:50070/webhdfs/v1/new1?op=MKDIRSoverwrite=false{code}
   A New Directory will be created 
  b. Now Execute the same webhdfs  command again 
 No exception will be thrown back to the client.
{code}
 HTTP/1.1 200 OK
 Cache-Control: no-cache
 Expires: Thu, 21 May 2015 15:11:57 GMT
 Date: Thu, 21 May 2015 15:11:57 GMT
 Pragma: no-cache
 Content-Type: application/json
 Transfer-Encoding: chunked
{code}
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8446) Separate safemode related operations in GetBlockLocations()

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8446:


 Summary: Separate safemode related operations in 
GetBlockLocations()
 Key: HDFS-8446
 URL: https://issues.apache.org/jira/browse/HDFS-8446
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor


Currently {{FSNamesystem#GetBlockLocations()}} has some special cases when the 
NN is in SafeMode. This jira proposes to refactor the code to improve 
readability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8447) Decouple information of files in GetLocatedBlocks

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8447:


 Summary: Decouple information of files in GetLocatedBlocks
 Key: HDFS-8447
 URL: https://issues.apache.org/jira/browse/HDFS-8447
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


The current implementation of {{BlockManager.getLocatedBlocks()}} requires the 
information of files to be passed as parameters. These information does not 
affect the results of getting the physical locations of blocks.

This jira proposes to refactor the call so that 
{{BlockManager.getLocatedBlocks()}} depends only on the block information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8434) Jenkins has race conditions when reading the maven repo

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8434:


 Summary: Jenkins has race conditions when reading the maven repo
 Key: HDFS-8434
 URL: https://issues.apache.org/jira/browse/HDFS-8434
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai


Jenkins builds have race conditions when accessing the directory of the maven 
repo:

{noformat}
Error Message

org.xml.sax.SAXParseException; systemId: 
jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT.jar!/core-default.xml;
 lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
Stacktrace

java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: 
jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-common/3.0.0-SNAPSHOT/hadoop-common-3.0.0-SNAPSHOT.jar!/core-default.xml;
 lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
{noformat}

More details can be found in 
https://builds.apache.org/job/PreCommit-HDFS-Build/11050/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeHotSwapVolumes/testAddOneNewVolume/

What it looks like here is that there is another build that triggers {{mvn 
clean}} when {{mvn test}} is still running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8421) Move startFile() and related operations into FSDirWriteFileOp

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8421:


 Summary: Move startFile() and related operations into 
FSDirWriteFileOp
 Key: HDFS-8421
 URL: https://issues.apache.org/jira/browse/HDFS-8421
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


This jira proposes to move startFile() and related functions into 
FSDirWriteFileOp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8394) Move getAdditionalBlock() and related functionalities into a separate class

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8394:


 Summary: Move getAdditionalBlock() and related functionalities 
into a separate class
 Key: HDFS-8394
 URL: https://issues.apache.org/jira/browse/HDFS-8394
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-8394.000.patch

This jira proposes to move the implementation of getAdditionalBlock() and 
related functionalities to a separate class to open up the possibilities of 
further refactoring and improvements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8357) Consolidate parameters of INode.CleanSubtree() into a parameter objects.

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8357:


 Summary: Consolidate parameters of INode.CleanSubtree() into a 
parameter objects.
 Key: HDFS-8357
 URL: https://issues.apache.org/jira/browse/HDFS-8357
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Li Lu


{{INode.CleanSubtree()}} takes multiple parameters including 
BlockStoragePolicySuite, removedBlocks and removedINodes. These parameters are 
pass multiple layers down the call chains.

This jira proposes to refactor them into a parameter object so that it is 
easier to make changes like HDFS-6757.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8321) CacheDirectives and CachePool operations should throw RetriableException in safemode

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8321:


 Summary: CacheDirectives and CachePool operations should throw 
RetriableException in safemode
 Key: HDFS-8321
 URL: https://issues.apache.org/jira/browse/HDFS-8321
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


Operations such as {{addCacheDirectives()}} throws {{SafeModeException}} when 
the NN is in safemode:

{code}
  if (isInSafeMode()) {
throw new SafeModeException(
Cannot add cache directive, safeMode);
  }
{code}

While other NN operations throws {{RetriableException}} when HA is enabled:

{code}
  void checkNameNodeSafeMode(String errorMsg)
  throws RetriableException, SafeModeException {
if (isInSafeMode()) {
  SafeModeException se = new SafeModeException(errorMsg, safeMode);
  if (haEnabled  haContext != null
   haContext.getState().getServiceState() == HAServiceState.ACTIVE
   shouldRetrySafeMode(this.safeMode)) {
throw new RetriableException(se);
  } else {
throw se;
  }
}
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8314) Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and SMALL_BUFFER_SIZE to the users

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8314:


 Summary: Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and 
SMALL_BUFFER_SIZE to the users
 Key: HDFS-8314
 URL: https://issues.apache.org/jira/browse/HDFS-8314
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


Currently HdfsServerConstants reads the configuration and to set the value of 
IO_FILE_BUFFER_SIZE and SMALL_BUFFER_SIZE, thus they are configurable instead 
of being constants.

This jira proposes to move these two variables to the users in the upper-level 
so that HdfsServerConstants only stores constant values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8286) Scaling out the namespace using KV store

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8286:


 Summary: Scaling out the namespace using KV store
 Key: HDFS-8286
 URL: https://issues.apache.org/jira/browse/HDFS-8286
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


Currently the NN keeps the namespace in the memory. To improve the scalability 
of the namespace, users can scale up by using more RAM or scale out using 
Federation (i.e., statically partitioning the namespace).

We would like to remove the limitation of scaling the global namespace. Our 
vision is that that HDFS should adopt a scalable underlying architecture that 
allows the global namespace scales linearly.

We propose to implement the HDFS namespace on top of a key-value (KV) store. 
Adopting the KV store interfaces allows HDFS to leverage the capability of 
modern KV store and to become much easier to scale. Going forward, the 
architecture allows distributing the namespace across multiple machines, or  
storing only the working set in the memory (HDFS-5389), both of which allows  
HDFS to manage billions of files using the commodity hardware available today.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8269) getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atime

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8269:


 Summary: getBlockLocations() does not resolve the .reserved path 
and generates incorrect edit logs when updating the atime
 Key: HDFS-8269
 URL: https://issues.apache.org/jira/browse/HDFS-8269
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Yesha Vora
Assignee: Haohui Mai
Priority: Blocker


When {{FSNamesystem#getBlockLocations}} updates the access time of the INode, 
it uses the path passed from the client, which generates incorrect edit logs 
entries:

{noformat}
  RECORD
OPCODEOP_TIMES/OPCODE
DATA
  TXID5085/TXID
  LENGTH0/LENGTH
  PATH/.reserved/.inodes/18230/PATH
  MTIME-1/MTIME
  ATIME1429908236392/ATIME
/DATA
  /RECORD
{noformat}

Note that the NN does not resolve the {{/.reserved}} path when processing the 
edit log, therefore it eventually leads to a NPE when loading the edit logs.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8249) Separate HdfsConstants into the client and the server side class

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8249:


 Summary: Separate HdfsConstants into the client and the server 
side class
 Key: HDFS-8249
 URL: https://issues.apache.org/jira/browse/HDFS-8249
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


The constants in {{HdfsConstants}} are used by both the client side and the 
server side. There are two types of constants in the class:

1. Constants that are used internally by the servers or not part of the APIs. 
These constants are free to evolve without breaking compatibilities. For 
example, {{MAX_PATH_LENGTH}} is used by the NN to enforce the length of the 
path does not go too long. Developers are free to change the name of the 
constants and to move it around if necessary.
1. Constants that are used by the clients, but not parts of the APIs. For 
example, {{QUOTA_DONT_SET}} represents an unlimited quota. The value is part of 
the wire protocol but the value is not. Developers are free to rename the 
constants but are not allowed to change the value of the constants.
1. Constants that are parts of the APIs. For example, {{SafeModeAction}} is 
used in {{DistributedFileSystem}}. Changing the name / value of the constant 
will break binary compatibility, but not source code compatibility.

This jira proposes to separate the above three types of constants into 
different classes:

* Creating a new class {{HdfsConstantsServer}} to hold the first type of 
constants.
* Move {{HdfsConstants}} into the {{hdfs-client}} package. The work of 
separating the second and the third types of constants will be postponed in a 
separate jira.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8248:


 Summary: Store INodeId instead of the INodeFile object in 
BlockInfoContiguous
 Key: HDFS-8248
 URL: https://issues.apache.org/jira/browse/HDFS-8248
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-8248.000.patch

Currently the namespace and the block manager are tightly coupled together. 
There are two couplings in terms of implementation:

1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that 
owns the block, so that the block manager can look up the corresponding file 
when replicating blocks, recovering from pipeline failures, etc.
1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file owns.

Decoupling the namespace and the block manager allows the BM to be separated 
out from the Java heap or even as a standalone process. This jira proposes to 
remove the first coupling by storing the id of the inode instead of the object 
reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8236) Merge HDFS-8227 into EC branch

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8236:


 Summary: Merge HDFS-8227 into EC branch
 Key: HDFS-8236
 URL: https://issues.apache.org/jira/browse/HDFS-8236
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-8236.000.patch

This jira proposes to merge the changes proposed in HDFS-8227 into the EC 
branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-8237) Move all protocol classes used by ClientProtocol to hdfs-client

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-8237:


 Summary: Move all protocol classes used by ClientProtocol to 
hdfs-client
 Key: HDFS-8237
 URL: https://issues.apache.org/jira/browse/HDFS-8237
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
 Attachments: HDFS-8237.000.patch

This jira proposes to move the classes in the hdfs project referred by 
ClientProtocol into the hdfs-client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)