All - I broke this list of questions out into a separate DISCUSS thread where we can iterate over how a security audit process at merge time might look and whether it is even something that we want to take on.
I will try and continue discussion on that thread and drive that to some conclusion before bringing it into any particular merge discussion. thanks, --larry On Fri, Oct 20, 2017 at 12:37 PM, larry mccay <lmc...@apache.org> wrote: > I previously sent this same email from my work email and it doesn't seem > to have gone through - resending from apache account (apologizing up from > for the length).... > > For such sizable merges in Hadoop, I would like to start doing security > audits in order to have an initial idea of the attack surface, the > protections available for known threats, what sort of configuration is > being used to launch processes, etc. > > I dug into the architecture documents while in the middle of this list - > nice docs! > I do intend to try and make a generic check list like this for such > security audits in the future so a lot of this is from that but I tried to > also direct specific questions from those docs as well. > > 1. UIs > I see there are at least two UIs - Storage Container Manager and Key Space > Manager. There are a number of typical vulnerabilities that we find in UIs > > 1.1. What sort of validation is being done on any accepted user input? > (pointers to code would be appreciated) > 1.2. What explicit protections have been built in for (pointers to code > would be appreciated): > 1.2.1. cross site scripting > 1.2.2. cross site request forgery > 1.2.3. click jacking (X-Frame-Options) > 1.3. What sort of authentication is required for access to the UIs? > 1.4. What authorization is available for determining who can access what > capabilities of the UIs for either viewing, modifying data or affecting > object stores and related processes? > 1.5. Are the UIs built with proxying in mind by leveraging X-Forwarded > headers? > 1.6. Is there any input that will ultimately be persisted in configuration > for executing shell commands or processes? > 1.7. Do the UIs support the trusted proxy pattern with doas impersonation? > 1.8. Is there TLS/SSL support? > > 2. REST APIs > > 2.1. Do the REST APIs support the trusted proxy pattern with doas > impersonation capabilities? > 2.2. What explicit protections have been built in for: > 2.2.1. cross site scripting (XSS) > 2.2.2. cross site request forgery (CSRF) > 2.2.3. XML External Entity (XXE) > 2.3. What is being used for authentication - Hadoop Auth Module? > 2.4. Are there separate processes for the HTTP resources (UIs and REST > endpoints) or are the part of existing HDFS processes? > 2.5. Is there TLS/SSL support? > 2.6. Are there new CLI commands and/or clients for access the REST APIs? > 2.7. Bucket Level API allows for setting of ACLs on a bucket - what > authorization is required here - is there a restrictive ACL set on creation? > 2.8. Bucket Level API allows for deleting a bucket - I assume this is > dependent on ACLs based access control? > 2.9. Bucket Level API to list bucket returns up to 1000 keys - is there > paging available? > 2.10. Storage Level APIs indicate “Signed with User Authorization” what > does this refer to exactly? > 2.11. Object Level APIs indicate that there is no ACL support and only > bucket owners can read and write - but there are ACL APIs on the Bucket > Level are they meaningless for now? > 2.12. How does a REST client know which Ozone Handler to connect to or am > I missing some well known NN type endpoint in the architecture doc > somewhere? > > 3. Encryption > > 3.1. Is there any support for encryption of persisted data? > 3.2. If so, is KMS and the hadoop key command used for key management? > > 4. Configuration > > 4.1. Are there any passwords or secrets being added to configuration? > 4.2. If so, are they accessed via Configuration.getPassword() to allow for > provisioning in credential providers? > 4.3. Are there any settings that are used to launch docker containers or > shell out any commands, etc? > > 5. HA > > 5.1. Are there provisions for HA? > 5.2. Are we leveraging the existing HA capabilities in HDFS? > 5.3. Is Storage Container Manager a SPOF? > 5.4. I see HA listed in future work in the architecture doc - is this > still an open issue? > > On Fri, Oct 20, 2017 at 11:19 AM, Anu Engineer <aengin...@hortonworks.com> > wrote: > >> Hi Steve, >> >> In addition to everything Weiwei mentioned (chapter 3 of user guide), if >> you really want to drill down to REST protocol you might want to apply this >> patch and build ozone. >> >> https://issues.apache.org/jira/browse/HDFS-12690 >> >> This will generate an Open API (https://www.openapis.org , >> http://swagger.io) based specification which can be accessed from KSM UI >> or just as a json file. >> Unfortunately, this patch is still at code review stage, so you will have >> to apply the patch and build it yourself. >> >> Thanks >> Anu >> >> >> On 10/20/17, 6:09 AM, "Yang Weiwei" <cheersy...@hotmail.com> wrote: >> >> Hi Steve >> >> >> The code is available in HDFS-7240 feature branch, public git repo >> here<https://github.com/apache/hadoop/tree/HDFS-7240>. >> >> I am not sure if there is a "public" API for object stores, but the >> design doc<https://issues.apache.org/jira/secure/attachment/1279954 >> 9/ozone_user_v0.pdf> uses most common syntax so I believe it should be >> compliance. You can find the rest API doc here<https://github.com/apache >> /hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/ >> site/markdown/OzoneRest.md> (with some example usages), and commandline >> API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop- >> hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>. >> >> >> Look forward for your feedback! >> >> >> --Weiwei >> >> >> ________________________________ >> 发件人: Steve Loughran <ste...@hortonworks.com> >> 发送时间: 2017年10月20日 11:49 >> 收件人: Yang Weiwei >> 抄送: hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; >> yarn-...@hadoop.apache.org; common-dev@hadoop.apache.org >> 主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk >> >> >> Wow, big piece of work >> >> 1. Where is a PR/branch on github with rendered docs for us to look >> at? >> 2. Have you made any public APi changes related to object stores? >> That's probably something I'll have opinions on more than implementation >> details. >> >> thanks >> >> > On 19 Oct 2017, at 02:54, Yang Weiwei <cheersy...@hotmail.com> >> wrote: >> > >> > Hello everyone, >> > >> > >> > I would like to start this thread to discuss merging Ozone >> (HDFS-7240) to trunk. This feature implements an object store which can >> co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with >> cluster sizes varying from 1 to 100 data nodes. >> > >> > >> > >> > The merge payload includes the following: >> > >> > 1. All services, management scripts >> > 2. Object store APIs, exposed via both REST and RPC >> > 3. Master service UIs, command line interfaces >> > 4. Pluggable pipeline Integration >> > 5. Ozone File System (Hadoop compatible file system >> implementation, passes all FileSystem contract tests) >> > 6. Corona - a load generator for Ozone. >> > 7. Essential documentation added to Hadoop site. >> > 8. Version specific Ozone Documentation, accessible via service >> UI. >> > 9. Docker support for ozone, which enables faster development >> cycles. >> > >> > >> > To build Ozone and run ozone using docker, please follow >> instructions in this wiki page. https://cwiki.apache.org/confl >> uence/display/HADOOP/Dev+cluster+with+docker. >> Dev cluster with docker - Hadoop - Apache Software Foundation< >> https://cwiki.apache.org/confluence/display/HADOO >> P/Dev+cluster+with+docker> >> cwiki.apache.org >> First, it uses a much more smaller common image which doesn't >> contains Hadoop. Second, the real Hadoop should be built from the source >> and the dist director should be ... >> >> >> >> > >> > >> > We have built a passionate and diverse community to drive this >> feature development. As a team, we have achieved significant progress in >> past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, >> we have resolved almost 400 JIRAs by 20+ contributors/committers from >> different countries and affiliations. We also want to thank the large >> number of community members who were supportive of our efforts and >> contributed ideas and participated in the design of ozone. >> > >> > >> > Please share your thoughts, thanks! >> > >> > >> > -- Weiwei Yang >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > >
