Re: [VOTE] Adopt HDSL as a new Hadoop subproject

+1 (binding) sanjay > On Mar 20, 2018, at 11:20 AM, Owen O'Malley wrote: > > All, > > Following our discussions on the previous thread (Merging branch HDFS-7240 > to trunk), I'd like to propose the following: > > * HDSL become a subproject of Hadoop. > * HDSL will

Re: [VOTE] Merging branch HDFS-7240 to trunk

Owen, Thanks for your proposal. While I would have prefered to have HDSL in HDFS and also to be part of Hadoop releases for the reasons stated earlier in this thread, I am willing to accept your proposal as a compromise to move this forward. Jitendra, Anu, Daryn, Andrew, Konstantine your

Re: [VOTE] Merging branch HDFS-7240 to trunk

> On Mar 5, 2018, at 4:08 PM, Andrew Wang wrote: > > - NN on top HDSL where the NN uses the new block layer (Both Daryn and Owen > acknowledge the benefit of the new block layer). We have two choices here > ** a) Evolve NN so that it can interact with both old and

Re: [VOTE] Merging branch HDFS-7240 to trunk

Joep, You raise a number of points: (1) Ozone vs and object stores. “Some users would choose Ozone as that layer, some might use S3, others GCS, or Azure, or something else”. (2) How HDSL/Ozone fits into Hadoop and whether it is necessary. (3) You raise the release issue which we will respond

Re: [VOTE] Merging branch HDFS-7240 to trunk

HDSL), which >> is >>> a distributed, replicated block layer. >>>The old HDFS namespace and NN can be connected to this new block >> layer >>> as we have described in HDFS-10419. >>>We also introduce a key-value namespace called Ozone built on HDSL. &

Re: [VOTE] Merging branch HDFS-7240 to trunk

e a key-value namespace called Ozone built on HDSL. >>> >>>The code is in a separate module and is turned off by default. In a >>> secure setup, HDSL and Ozone daemons cannot be started. >>> >>>The detailed documentation

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Konstantine Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns? sanjay > On Feb 13, 2018, at 6:28 PM, sanjay Radia <sanjayo...@gmail.com> wrote: > > Sorry the formatting got messed by my email client. H

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Sorry the formatting got messed by my email client. Here it is again Dear Hadoop Community Members, We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Dear Hadoop Community Members, We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following How the new block storage

[jira] [Created] (HDFS-12952) Change OzoneFS's semtics to allow readers to see file content while being written

Sanjay Radia created HDFS-12952: --- Summary: Change OzoneFS's semtics to allow readers to see file content while being written Key: HDFS-12952 URL: https://issues.apache.org/jira/browse/HDFS-12952

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

Konstantine, Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

Erik Great stuff. BTW did you build on top of the “simulated data nodes” in HDFS which has a way to storing only the length of data (but not real data)? That work allowed supplementing with a matching editsLog for the NN. Your approach of using a real image has the advantage of being able

[jira] [Created] (HDFS-8075) Revist layout version

Sanjay Radia created HDFS-8075: -- Summary: Revist layout version Key: HDFS-8075 URL: https://issues.apache.org/jira/browse/HDFS-8075 Project: Hadoop HDFS Issue Type: Bug Components

[jira] [Resolved] (HDFS-8075) Revist layout version

[ https://issues.apache.org/jira/browse/HDFS-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved HDFS-8075. Resolution: Duplicate Closed: duplicate of HDFS-5223 Revist layout version

Re: Looking to a Hadoop 3 release

On Mar 5, 2015, at 3:21 PM, Siddharth Seth ss...@apache.org wrote: 2) Simplification of configs - potentially separating client side configs and those used by daemons. This is another source of perpetual confusion for users. + 1 on this. sanjay

Re: Looking to a Hadoop 3 release

Andrew Thanks for bringing up the issue of moving to Java8. Java8 is important However, I am not seeing a strong motivation for changing the major number. We can go to Java8 in the 2.series. The classpath issue for Hadoop-11656 is too minor to force a major number change (no pun intended).

[jira] [Created] (HDFS-7745) HDFS should have its own daemon command and not rely on the one in common

Sanjay Radia created HDFS-7745: -- Summary: HDFS should have its own daemon command and not rely on the one in common Key: HDFS-7745 URL: https://issues.apache.org/jira/browse/HDFS-7745 Project: Hadoop

Re: [VOTE] Merge fs-encryption branch to trunk

transparent encryption in a 2.x release for pubic consumption, let us at least complete item 1 (ie distcp and cp) and the flag to turn this feature on/of. This is a great work; thanks team for contributing this important feature. sanjay On Aug 14, 2014, at 1:05 AM, sanjay Radia san

Re: [VOTE] Merge fs-encryption branch to trunk

While I was originally skeptical of transparent encryption, I like the value proposition of transparent encryption. HDFS has several layers, protocols and tools. While the HDFS core part seems to be well done in the Jira, inserting the matching transparency in the other tools or protocols need

Re: [VOTE] Migration from subversion to git for version control

+1 sanjay On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2.

Re: [VOTE] Release Apache Hadoop 2.4.0

+1 binding Verified binaries, ran from binary on single node cluster. Tested some HDFS clis and wordcount. sanjay On Apr 7, 2014, at 9:52 AM, Suresh Srinivas sur...@hortonworks.com wrote: +1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source,

Re: [VOTE] Release Apache Hadoop 2.4.0

On Thu, Apr 3, 2014 at 4:55 PM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.comwrote: Hi, Ran tests and confirmed that some tests(TestSymlinkLocalFSFileSystem) fail. The log of the test failure is as follows: https://gist.github.com/oza/9965197 Should we fix or disable the feature? Symlinks is

Re: [Proposal] Pluggable Namespace

On Oct 6, 2013, at 5:58 PM, Milind Bhandarkar wrote: Vinod, I have received a few emails about concerns that this effort somehow conflicts with federated namenodes. Most of these emails are from folks who are directly or remotely associated with Hortonworks. Three weeks ago, I sent

Re: [Proposal] Pluggable Namespace

On Oct 3, 2013, at 12:17 PM, Milind Bhandarkar wrote: Exec Summary: For the last couple of months, we, at Pivotal, along with a couple of folks in the community have been working on making Namespace implementation in the namenode pluggable. We have demonstrated that it can be done without

Re: symlink support in Hadoop 2 GA

There are a number of issues (some minor, some more than minor). GA is close and we are are still in discussion on the some of them; while I believe we will close on these very very shortly, code change like this so close to GA is dangerous. I suggest we do the following: 1) Disable Symlinks

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

+1 for naming the new branch 2.2.0 sanjay On Oct 1, 2013, at 4:55 PM, Suresh Srinivas wrote: (This time copying all the lists) I am +1 for naming the new branch 2.2.0. On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy a...@hortonworks.com wrote: Guys, I took a look at the content in

[jira] [Created] (HDFS-4691) Disallow .reserved path component, given that HDFS-4434 does not allow /.reserved

Sanjay Radia created HDFS-4691: -- Summary: Disallow .reserved path component, given that HDFS-4434 does not allow /.reserved Key: HDFS-4691 URL: https://issues.apache.org/jira/browse/HDFS-4691 Project

[jira] [Created] (HDFS-4606) HDFS API to move file replicas to caller's location

Sanjay Radia created HDFS-4606: -- Summary: HDFS API to move file replicas to caller's location Key: HDFS-4606 URL: https://issues.apache.org/jira/browse/HDFS-4606 Project: Hadoop HDFS Issue Type

Re: [Vote] Merge branch-trunk-win to trunk

On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote: Commitment is a good thing. I think the two builds that I proposed are a prerequisite for Win support. If we commit windows patch people will start breaking it the next day. Which we wont know without the nightly build and wont be able

Re: [Vote] Merge branch-trunk-win to trunk

+1 Java has done the bulk of the work in making Hadoop multi-platform. Windows specific code is a tiny percentage of the code. Jeninks support for windows is going help us keep the platform portable going forward. I expect that the vast majority of new commits have no problems. I propose that

Re: VOTE: HDFS-347 merge

On Feb 26, 2013, at 1:51 PM, Eli Collins wrote: it doesn't seem right to hold up 347 up for Windows support given that Windows support has not been merged to trunk yet, is not in any Apache release, etc. Personally I don't like establishing the precedent here that we can hold up a merge due

Re: VOTE: HDFS-347 merge

On Feb 20, 2013, at 5:12 PM, Aaron T. Myers wrote: On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas cdoug...@apache.org wrote: Given that HDFS-347 is a strictly better approach, once committed, there will be ample motivation to add support for other OSes and remove HDFS-2246 entirely. Nobody

[jira] [Resolved] (HDFS-4256) Backport concatenation of files into a single file to branch-1

[ https://issues.apache.org/jira/browse/HDFS-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved HDFS-4256. Resolution: Fixed Target Version/s: 1.2.0 Backport concatenation of files

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

I have posted a comment on 3077 listing two areas that i would like to fix in the QJM. If we agree on that we can merge now and fix those two later. Todd has volunteered to fix one and I or suresh can fix the other. sanjay

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

at 12:06 PM, sanjay Radia san...@hortonworks.com wrote: Suresh and I are still reviewing this design and patch. The 3077 code along with the code pulled from 3092 is fairly substrantial. The design is also fairly complex and involved. I would request that we postpone the merge for another week

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

I am in favor of 3077 being part of HDFS. - It is one of several way of configuring HDFS HA (shared storage is an option or if we extend the BackupNode a little, then it will be another option for HA.) - I believe I can use part of 3092/3077 to move journals to the Datanodes down

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

Suresh and I are still reviewing this design and patch. The 3077 code along with the code pulled from 3092 is fairly substrantial. The design is also fairly complex and involved. I would request that we postpone the merge for another week to give folks time to review this fully. sanjay

[jira] [Created] (HDFS-3630) Modify TestPersistBlocks to use both flush and hflush

Sanjay Radia created HDFS-3630: -- Summary: Modify TestPersistBlocks to use both flush and hflush Key: HDFS-3630 URL: https://issues.apache.org/jira/browse/HDFS-3630 Project: Hadoop HDFS Issue

[jira] [Resolved] (HDFS-3424) TestDatanodeBlockScanner and TestReplication fail intermittently on Windows

[ https://issues.apache.org/jira/browse/HDFS-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved HDFS-3424. Resolution: Fixed Target Version/s: HADOOP-1-Windows Committed to windows branch. Thanks

Hdfs Contributors meetup at the Hadoop summit

We will have a Hdfs Contributors meetup at the Hadoop summit. Details: http://www.meetup.com/Hadoop-Contributors/events/60841502/ sanjay

[jira] [Created] (HDFS-3278) Umbrella Jira for HDFS-HA Phase 2

Umbrella Jira for HDFS-HA Phase 2 - Key: HDFS-3278 URL: https://issues.apache.org/jira/browse/HDFS-3278 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee

[jira] [Created] (HDFS-3153) For HA, a logical name is visible in URIs - add an explicit logical name

Type: Improvement Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see

Re: [DISCUSS] Remove append?

On Tue, Mar 20, 2012 at 5:37 PM, Eli Collins e...@cloudera.com wrote: Append introduces non-trivial design and code complexity, which is not worth the cost if we don't have real users. The bulk of the complexity of HDFS-265 (the new Append) was around Hflush, concurrent readers, the pipeline

HDFS HA Next Steps - Developers Meeting

for approx 15 folks - please send me (san...@hortonworks.com) an email if you are interested in attending so that I can reserve a spot and confirm. Minutes will be posted on HDFS-1623. sanjay On Feb 21, 2012, at 9:16 PM, sanjay Radia wrote: Hello, We would like organize a meeting to discuss

HDFS HA Status and Next Steps

Hello, We would like organize a meeting to discuss the 2nd phase of HDFS HA development. Right now the HA branch supports HOT-Failover, except that it is manual failover. We are now moving into a phase to implement automatic failover. Please see HDFS-1623 for more details. Significant

[jira] [Created] (HDFS-2907) Make FSDataset in Datanode Pluggable

Make FSDataset in Datanode Pluggable Key: HDFS-2907 URL: https://issues.apache.org/jira/browse/HDFS-2907 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia

[jira] [Created] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol

Enable protobuf rpc for ClientNamenodeProtocol -- Key: HDFS-2669 URL: https://issues.apache.org/jira/browse/HDFS-2669 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia

[jira] [Created] (HDFS-2597) ClientNameNodeProtocol in Protocol Buffers

ClientNameNodeProtocol in Protocol Buffers --- Key: HDFS-2597 URL: https://issues.apache.org/jira/browse/HDFS-2597 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia

[jira] [Created] (HDFS-2598) ClientDatanodeProtocol in Protocol Buffers

ClientDatanodeProtocol in Protocol Buffers -- Key: HDFS-2598 URL: https://issues.apache.org/jira/browse/HDFS-2598 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia

[jira] [Created] (HDFS-2478) HDFS Protocols in Protocol Buffers

HDFS Protocols in Protocol Buffers -- Key: HDFS-2478 URL: https://issues.apache.org/jira/browse/HDFS-2478 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee

[jira] [Created] (HDFS-2479) HDFS Client Data Types in Protocol Buffers

HDFS Client Data Types in Protocol Buffers -- Key: HDFS-2479 URL: https://issues.apache.org/jira/browse/HDFS-2479 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia

[jira] [Created] (HDFS-2399) Remove all Writable from HDFS data type since the wire protocol types have been segregated in protocolR23Compatible.

://issues.apache.org/jira/browse/HDFS-2399 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA

[jira] [Created] (HDFS-2400) HDFS wire types are seperated, move the HDFS data types in o.a.h.hdfs.protocol to o.a.h.hdfs.server

/HDFS-2400 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure

[jira] [Created] (HDFS-2351) Change Namenode and Datanode to register each of their protocols seperately

Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia This take advantage of HADOOP-7524 and allow separation of datatypes (HDFS-2181) jira to be completed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http

[jira] [Created] (HDFS-2284) RW Http access to HDFS

RW Http access to HDFS -- Key: HDFS-2284 URL: https://issues.apache.org/jira/browse/HDFS-2284 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Tsz Wo (Nicholas), SZE

[jira] [Created] (HDFS-2181) Seperate HDFS wire protocol data types

Seperate HDFS wire protocol data types -- Key: HDFS-2181 URL: https://issues.apache.org/jira/browse/HDFS-2181 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia

Re: Developer meetup after the Hadoop Summit - Friday July 1st

I have updated meetup: http://www.meetup.com/Hadoop-Contributors/events/23890191/ sanjay On Jun 28, 2011, at 4:27 PM, Sanjay Radia wrote: On Jun 28, 2011, at 11:17 AM, Sanjay Radia wrote: We have a room confirmed from 10-3pm on Friday July 1st at Yahoo's sunnyvale campus. Will update

[jira] [Created] (HDFS-2005) Client should not timeout during a NN start - esp useful for HA NN

-task Reporter: Sanjay Radia -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-1911) HDFS tests for viewfs

HDFS tests for viewfs - Key: HDFS-1911 URL: https://issues.apache.org/jira/browse/HDFS-1911 Project: Hadoop HDFS Issue Type: Test Reporter: Sanjay Radia Assignee: Sanjay Radia Fix

Re: Merging Namenode Federation feature (HDFS-1052) to trunk

On Mar 14, 2011, at 10:57 AM, Sanjay Radia wrote: On Mar 12, 2011, at 8:43 AM, Allen Wittenauer wrote: To me, this series of changes looks like it is going to make running a grid much much harder for very little benefit. In particular, I don't see the difference between running

[jira] Created: (HDFS-1623) High Availability Framework for HDFS NN

High Availability Framework for HDFS NN --- Key: HDFS-1623 URL: https://issues.apache.org/jira/browse/HDFS-1623 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia

[jira] Created: (HDFS-1599) Umbrella Jira for Improving HBASE support in HDFS

: Sanjay Radia Umbrella Jira for improved HBase support in HDFS -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HDFS-1491) Update Hdfs to match the change of methods from protected to public in AbstractFileSystem (Hadoop-6903)

[ https://issues.apache.org/jira/browse/HDFS-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved HDFS-1491. Resolution: Fixed Committed on Nov 10. Update Hdfs to match the change of methods from protected

Re: svn commit: r1032846 - in /hadoop/hdfs/trunk: .eclipse.templates/.classpath .eclipse.templates/.project CHANGES.txt build.xml

Nigel At my previous company we simply checked-in the .classpath file and the .project file rather then generate it through an ant command. Folks who were not using eclipse ignored it. Should we consider such an approach for eclipse in the future. It is much easier to manage. sanjay

[jira] Created: (HDFS-1491) Update Hdfs to match the change of methods from protected to public in AbstractFileSystem (Hadoop-6903)

/browse/HDFS-1491 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 0.22.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment

Hadoop Contributors Meetings

be found at: - Meetup home page: http://www.meetup.com/Hadoop-Contributors/ - May 6th registration: http://www.meetup.com/Hadoop-Contributors/calendar/13249370/ thanks Sanjay Radia Hadoop Team, Yahoo!

[jira] Created: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs

: Sanjay Radia The naming and handling of NN's fsImage and edit logs can be significantly improved resulting simpler and more robust code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.

[jira] Created: (HDFS-1064) NN Availability - umbrella Jira

NN Availability - umbrella Jira --- Key: HDFS-1064 URL: https://issues.apache.org/jira/browse/HDFS-1064 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia This is an umbrella jira

[jira] Created: (HDFS-1051) Umbrella Jira for Scaling the HDFS Name Service

Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 0.22.0 The HDFS Name service currently uses a single Namenode which limits its scalability. This is a master jira to track sub-jiras to address this problem. -- This message is automatically generated

[jira] Created: (HDFS-1053) A client side mount table to give per-application/per-job file system view

Issue Type: New Feature Affects Versions: 0.22.0 Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 0.22.0 This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. -- This message is automatically

[jira] Created: (HDFS-759) Extend DataNodeCluster to allow block injection for real (non simulated) DNs

Issue Type: Improvement Reporter: Sanjay Radia Fix For: 0.22.0 The DataNodeCluster (built using the MiniDFSCluster) allows injection of blocks only if the data nodes are simulated. Extend the implementation to allow block injection for DataNodeClusters that have real

[jira] Created: (HDFS-725) Support the build error fix for HADOOP-6327

: Sanjay Radia Attachments: buildError-hdfs.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.

[jira] Created: (HDFS-610) Add support for FileContext

Reporter: Sanjay Radia Assignee: Sanjay Radia Add support for FileContext (linked jira) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.