On Wed, Oct 15, 2014 at 4:54 PM, Prabath Abeysekera <[email protected]> wrote:
> Hi Paul, > > We indeed tried using namenode federation as an alternative and IMO, it > doesn't really fit in. One of the issues with that approach is, one > namenode is only capable of handling just one namespace in it. Therefore, > if we are to map tenants with those namespaces, then the deployment would > really grow big particularly, when the number of tenants increase, as we > might need to deploy at least a namenode each for all the tenants > (excluding supportive nodes deployed to achieve HA for each namenode). > MT with multiple name nodes more suitable for system that can automatically instantiate name nodes. This model needs to get help form an infrastructure like Stratos to instantiate name nodes. Therefore MT with mutiple name nodes do not scale well in a public cloud environment. > However, the federation approach would be ideal if we are to deal with a > small number of tenants though. > > Deep and Shani (CC'ed) should be able to share more detailed notes on this > as they've been working on this over the last couple of months. > In the second approach we use one HDFS cluster with single name space but update the INode file path information based on the tenant. In this approach we do not have to update HDFS client tools. Also we can keep single cluster and share HDFS resources across tenants. > Cheers, > Prabath > > On Wed, Oct 15, 2014 at 4:20 PM, Paul Fremantle <[email protected]> wrote: > >> Prabath >> >> Does HDFS support MT via multiple namenodes? >> >> Paul >> >> On 15 October 2014 09:49, Prabath Abeysekera <[email protected]> wrote: >> >>> Hi Folks, >>> >>> I've got a 3rd party dependency (i.e Hadoop/HDFS) to be adapted into GIT >>> and wondering where exactly I should do the initial developments, etc with >>> the newly introduced GIT model? I do realize that I could have simply used >>> a private repository or something and do the initial forking and stuff, but >>> just checking if there's any recommended space available within any >>> WSO2-managed repository where I could simply get the stuff checked in and >>> make them available for collaborative developments within the team. >>> >>> What's the recommended approach? >>> >>> Let me explain why we need to get this in created as a separate >>> repository as well. We'd already done a lot of R&D around getting MT >>> implemented for HDFS using the extension points exposed within Hadoop >>> without any success. Tried getting ourselves into discussions with the >>> Hadoop community also just to see if there's an alternative approach >>> available to what we'd suggested for implementing the same but haven't >>> received any positive response yet. Therefore, I'm afraid the only approach >>> we're left with right now is to get the sources forked into our own code >>> base and moving forward with the developments. I certainly do realize that >>> this would incur maintenance issues and all while trying to upgrade the >>> versions, etc. However, I do not see any other feasible alternative option >>> that we can make use of, in order to continue developments on this front. >>> >>> Please do share your feedback as to how we could resolve the >>> aforementioned as well. >>> >>> >>> Cheers, >>> Prabath >>> >>> P.S. Pardon if this is something that's already been discussed within >>> the list, I'm just trying to get my head around this. :) >>> >>> -- >>> Prabath Abeysekara >>> Associate Technical Lead, Data TG. >>> WSO2 Inc. >>> Email: [email protected] >>> Mobile: +94774171471 >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Paul Fremantle >> CTO and Co-Founder, WSO2 >> OASIS WS-RX TC Co-chair, Apache Member >> >> UK: +44 207 096 0336 >> >> blog: http://pzf.fremantle.org >> twitter.com/pzfreo >> [email protected] >> >> wso2.com Lean Enterprise Middleware >> >> Disclaimer: This communication may contain privileged or other >> confidential information and is intended exclusively for the addressee/s. >> If you are not the intended recipient/s, or believe that you may have >> received this communication in error, please reply to the sender indicating >> that fact and delete the copy you received and in addition, you should not >> print, copy, retransmit, disseminate, or otherwise use the information >> contained in this communication. Internet communications cannot be >> guaranteed to be timely, secure, error or virus-free. The sender does not >> accept liability for any errors or omissions. >> > > > > -- > Prabath Abeysekara > Associate Technical Lead, Data TG. > WSO2 Inc. > Email: [email protected] > Mobile: +94774171471 > -- Deependra Ariyadewa WSO2, Inc. http://wso2.com/ http://wso2.org email [email protected]; cell +94 71 403 5996 ; Blog http://risenfall.wordpress.com/ PGP info: KeyID: 'DC627E6F' *WSO2 - Lean . Enterprise . Middleware*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
