Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

> On Jul 22, 2016, at 5:47 PM, Zheng, Kai wrote: > > For the leveldb thing, wouldn't we have an alternative option in Java for the > platforms where leveldb isn't supported yet due to whatever reasons. IMO, > native library would be best to be used for optimization and

[VOTE] Release Apache Hadoop 2.7.3 RC0

Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.3. As discussed before, this is the next maintenance release to follow up 2.7.2. The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ The

RE: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

For the leveldb thing, wouldn't we have an alternative option in Java for the platforms where leveldb isn't supported yet due to whatever reasons. IMO, native library would be best to be used for optimization and production for performance. For development and pure Java platform, by default

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

But if I don't use ApplicationClassLoader, my java app is basically screwed then, right? Also: right now, the non-Linux and/or non-x86 platforms have to supply their own leveldbjni jar (or at least the C level library?) in order to make YARN even functional. How is that going to work with

[jira] [Created] (HDFS-10682) Add metric to measure lock held time in FSDataSetImpl

Chen Liang created HDFS-10682: - Summary: Add metric to measure lock held time in FSDataSetImpl Key: HDFS-10682 URL: https://issues.apache.org/jira/browse/HDFS-10682 Project: Hadoop HDFS Issue

[jira] [Resolved] (HDFS-9836) RequestHedgingInvocationHandler can't be cast to org.apache.hadoop.ipc.RpcInvocationHandler

[ https://issues.apache.org/jira/browse/HDFS-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guocui Mi resolved HDFS-9836. - Resolution: Won't Fix > RequestHedgingInvocationHandler can't be cast to >

Re: Multiple namenodes

Thanks a lot for your suggestions. On Fri, Jul 22, 2016 at 3:58 PM, Daniel Templeton wrote: > On 7/22/16 12:23 PM, Kun Ren wrote: > >> Thanks a lot for your reply, Daniel, very helpful. >> >> About (1) : I will consider this way, thanks. Also beside multiple >> clusters,

Re: Multiple namenodes

On 7/22/16 12:23 PM, Kun Ren wrote: Thanks a lot for your reply, Daniel, very helpful. About (1) : I will consider this way, thanks. Also beside multiple clusters, are there any other options to do so? Thanks. HDFS does not support two active NNs in a single cluster. Each DN belongs to a

Re: Multiple namenodes

Thanks a lot for your reply, Daniel, very helpful. About (1) : I will consider this way, thanks. Also beside multiple clusters, are there any other options to do so? Thanks. About (2), if I understand correctly, HDFS used the quorum journal manager(QJM) for HA, and the client still only

[jira] [Created] (HDFS-10681) DiskBalancer: query command should report Plan file path apart from PlanID

Manoj Govindassamy created HDFS-10681: - Summary: DiskBalancer: query command should report Plan file path apart from PlanID Key: HDFS-10681 URL: https://issues.apache.org/jira/browse/HDFS-10681

Re: Multiple namenodes

On 7/22/16 8:45 AM, Kun Ren wrote: (1). How to create/start multiple namenodes? Just pretend like you have two separate HDFS clusters and set them up that way. (That is actually what you have.) (2). Once I have multiple Namenodes running, do you think what is the best/simple way to

[jira] [Created] (HDFS-10680) TestUTF8 fails in branch-2.6

Zhe Zhang created HDFS-10680: Summary: TestUTF8 fails in branch-2.6 Key: HDFS-10680 URL: https://issues.apache.org/jira/browse/HDFS-10680 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-10653) Optimize conversion from path string to components

[ https://issues.apache.org/jira/browse/HDFS-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang resolved HDFS-10653. -- Resolution: Fixed Local test-patch shows 2 failures but those tests fail even without the patch. I'll

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

The work on HADOOP-13070 and the ApplicationClassLoader are generic and go beyond YARN. It can be used in any JVM that uses hadoop. The current use cases are MR containers, hadoop's RunJar (as in "hadoop jar"), and the YARN node manager auxiliary services. I'm not sure if that's what you were

[jira] [Created] (HDFS-10679) libhdfs++: Implement parallel find with wildcards tool

Anatoli Shein created HDFS-10679: Summary: libhdfs++: Implement parallel find with wildcards tool Key: HDFS-10679 URL: https://issues.apache.org/jira/browse/HDFS-10679 Project: Hadoop HDFS

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

My work on HADOOP-11804 *only* helps processes that sit outside of YARN. :) On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer wrote: > > Does any of this work actually help processes that sit outside of YARN? > >> On Jul 21, 2016, at 12:29 PM, Sean Busbey

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

Does any of this work actually help processes that sit outside of YARN? > On Jul 21, 2016, at 12:29 PM, Sean Busbey wrote: > > thanks for bringing this up! big +1 on upgrading dependencies for 3.0. > > I have an updated patch for HADOOP-11804 ready to post this week. I've

Multiple namenodes

Hi Genius, I am currently involved in a project that will create/start multiple namenodes(It is different with Federation that: We want to partition the metadata not only by directory, and may support other partition schemes, and we want to support the distributed operations that cross multiple

[jira] [Resolved] (HDFS-10385) LocalFileSystem rename() function should return false when destination file exists

[ https://issues.apache.org/jira/browse/HDFS-10385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor resolved HDFS-10385. - Resolution: Later > LocalFileSystem rename() function should return false when destination file

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/110/ [Jul 21, 2016 11:41:02 PM] (xiao) HDFS-10225. DataNode hot swap drives should disallow storage type [Jul 22, 2016 6:21:47 AM] (cdouglas) HADOOP-13393. Omit unsupported fs.defaultFS setting in ADLS -1

Start client side daemon

Hi Genius, I understand that we use the command to start namenode and datanode. But I don't know how HDFS starts client side and creates the Client side object(Like DistributedFileSystem), and client side RPC server? Could you please point it out how HDFS start the client side dameon? If the

Re: Improving recovery performance for degraded reads

I'm adding one more point to the above. In my previous mail reply, I've explained the striped block reconstruction task which will be triggered by the Namenode on identifying a missing/bad block. Similarly, in case of hdfs client read failure, currently hdfs client internally submitting read

Re: Improving recovery performance for degraded reads

Hi Roy, Thanks for the interest in hdfs erasure coding feature and helping us in making the feature more attractive to the users by sharing performance improvement ideas. Presently, the reconstruction work has been implemented in a centralized manner in which the reconstruction task will be

Improving recovery performance for degraded reads

Greetings! We are evaluating erasure coding on HDFS to reduce storage cost. However, the degraded read latency seems like a crucial bottleneck for our system. After exploring some strategies for alleviating the pain of degraded read latency, I found a "tree-like recovery" technique might be

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/109/ No changes -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint