[ 
https://issues.apache.org/jira/browse/HADOOP-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457980#comment-13457980
 ] 

Xianqing Yu commented on HADOOP-8803:
-------------------------------------

Hi Steve,

I am proposing the whole Hadoop (HDFS and Map-Reduce) runnning in public cloud. 
But I feel that what I am doing is the first step. Currently I am focused on 
how to bring fine-grain access control to HDFS and make Map-Reduce framework to 
leverage that. DDoS attack and 0-day exploits are real problem for security 
area, they could be my future work. 0-day exploit is a popular topic in 
computer security, that is also one of the reason so many researchers are 
working on that. I feel that it is beyond the hadoop, and we need to do 
protection outside Hadoop to prevent that.

Setup network protection (iptables, VPN, Hypervisor-based network management 
etc..) is necessary. But the value of my work can not be replaced by those 
methods. For instance, one of my assumption is that one of the datanode or 
tasktracker machine is compromised, and I would try to limit the hacker's 
threat to the whole cluster. One example is that current Hadoop's HDFS only use 
one key to generate block token and verify that, so if the hacker hack one 
machine, she or he can generate block token, thus access the whole cluster. The 
other example is that all tasktrackers and jobtracker share one Map-reduce 
directory on HDFS, there are many important information, such as job token, 
Delegation Token are inside that. So if hacker can hack one of machine, he or 
she would be able to access those content. The third example is that current 
job delegation token is only be used to represent as certain user or job, the 
task process which can get this delegation token can access all content for 
that job or user. It is not fine-grain enough which means the hacker who get 
the Delegation Token can access everything for that user or job. (As Owen and 
Luke mentioned above, some mapreduce job need task process to access more input 
beyond its filesplit range, and my method may interrupt this process, I am 
currently looking into that.) In a word, network protection is good and useful, 
but it can not substitute my work, we are solving different problems.

Thanks for mentioning network billing issues. I think it is very interesting 
point. For the reality, it is one thing need to consider. I didn't see any 
billing data or estimate data yet. But I keep eye on that.

It is a good point for your observation. Indeed hacker can try to get more data 
by showing he or she has more resource and get more jobs from jobtracker, but 
in fact, the machine may execute nothing. However, it is harder for attacker to 
get more data than before, and it is almost impossible for hacker to get all 
data for one single job(because it is jobtracker who distribute tasks). And I 
think it is possible to develop some detect mechanism to find out those 
suspicious activities. (If you don't limit single task's access range, it is 
really hard to distinguish those bad access behaviors from those good ones.) 

If you want to protect the data in compromised datanode, maybe the best way is 
encrypted them as you said. However, considered of potentially huge overhead, I 
doesn't do that in my current work. It could be a future work to think about 
how to do that in the best way. Simply said, I try to limit the damage what 
hackers can do, not completely protect the hadoop.

I feel my work can work very nice in DN decommission scenario. For example, if 
hacker can only has limited of time to attack, in current hadoop, once hacker 
be able to compromise one DN, she or he can start to read everything in this 
HDFS before DN decommission (as mentioned above, all HDFS use one key to 
generate and verify block token). But in my work, the hacker can only read what 
is stored on this DN(other DNs would use different key). 
 
                
> Make Hadoop running more secure public cloud envrionment
> --------------------------------------------------------
>
>                 Key: HADOOP-8803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8803
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, ipc, security
>    Affects Versions: 0.20.204.0
>            Reporter: Xianqing Yu
>              Labels: hadoop
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> I am a Ph.D student in North Carolina State University. I am modifying the 
> Hadoop's code (which including most parts of Hadoop, e.g. JobTracker, 
> TaskTracker, NameNode, DataNode) to achieve better security.
>  
> My major goal is that make Hadoop running more secure in the Cloud 
> environment, especially for public Cloud environment. In order to achieve 
> that, I redesign the currently security mechanism and achieve following 
> proprieties:
> 1. Bring byte-level access control to Hadoop HDFS. Based on 0.20.204, HDFS 
> access control is based on user or block granularity, e.g. HDFS Delegation 
> Token only check if the file can be accessed by certain user or not, Block 
> Token only proof which block or blocks can be accessed. I make Hadoop can do 
> byte-granularity access control, each access party, user or task process can 
> only access the bytes she or he least needed.
> 2. I assume that in the public Cloud environment, only Namenode, secondary 
> Namenode, JobTracker can be trusted. A large number of Datanode and 
> TaskTracker may be compromised due to some of them may be running under less 
> secure environment. So I re-design the secure mechanism to make the damage 
> the hacker can do to be minimized.
>  
> a. Re-design the Block Access Token to solve wildly shared-key problem of 
> HDFS. In original Block Access Token design, all HDFS (Namenode and Datanode) 
> share one master key to generate Block Access Token, if one DataNode is 
> compromised by hacker, the hacker can get the key and generate any  Block 
> Access Token he or she want.
>  
> b. Re-design the HDFS Delegation Token to do fine-grain access control for 
> TaskTracker and Map-Reduce Task process on HDFS. 
>  
> In the Hadoop 0.20.204, all TaskTrackers can use their kerberos credentials 
> to access any files for MapReduce on HDFS. So they have the same privilege as 
> JobTracker to do read or write tokens, copy job file, etc.. However, if one 
> of them is compromised, every critical thing in MapReduce directory (job 
> file, Delegation Token) is exposed to attacker. I solve the problem by making 
> JobTracker to decide which TaskTracker can access which file in MapReduce 
> Directory on HDFS.
>  
> For Task process, once it get HDFS Delegation Token, it can access everything 
> belong to this job or user on HDFS. By my design, it can only access the 
> bytes it needed from HDFS.
>  
> There are some other improvement in the security, such as TaskTracker can not 
> know some information like blockID from the Block Token (because it is 
> encrypted by my way), and HDFS can set up secure channel to send data as a 
> option.
>  
> By those features, Hadoop can run much securely under uncertain environment 
> such as Public Cloud. I already start to test my prototype. I want to know 
> that whether community is interesting about my work? Is that a value work to 
> contribute to production Hadoop?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to