[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13649994#comment-13649994 ] Omkar Vinit Joshi commented on YARN-578: Fixing both comments and throwing different exceptions for different scenarios in ContainerLogsPage. Adding test. Verified it on Secure setup with NativeIO enabled. NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch, YARN-578-20130506.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13650133#comment-13650133 ] Vinod Kumar Vavilapalli commented on YARN-578: -- Okay, I just had an enlightening experience and I realized we need to fix more issues: - LogAggregationService can ignore these permissions and upload sensitive files! Please fix this and write a test to verify that it doesn't happen. - It seems like when logs are deleted, we are using the correct user to delete them. But can you write tests to validate this for two cases (1) when log-aggregation is enabled and (2) when it isn't. NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch, YARN-578-20130506.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13648640#comment-13648640 ] Omkar Vinit Joshi commented on YARN-578: This issue will now only track yarn related changes (ContainerLogsPage). Creating separate jira for mapreduce issue. MAPREDUCE-5208 NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645240#comment-13645240 ] Vinod Kumar Vavilapalli commented on YARN-578: -- Can you use this only for YARN changes i.e. serving logs and open a separate MAPREDUCE ticket for ShuffleHandler? For the YARN changes: - Remove the comment above the code which talks about SecureIOUtils ;) - I think we should separate the exception message to clearly say whether this was an permission-issue or something else. NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13644199#comment-13644199 ] Omkar Vinit Joshi commented on YARN-578: Testing :- This patch I have tested on Secure setup.[ Ubuntu 12.04, single node using kerberos]. NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13643290#comment-13643290 ] Omkar Vinit Joshi commented on YARN-578: Updating the patch. Not sure about the junit test if it is required here. For ContainerLogsPage in case of an IOException not printing the whole exception stack on NodeManager UI; instead logging it. NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Attachments: yarn-578-20130426.patch Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-578) NodeManager should use SecureIOUtils for serving logs and intermediate outputs
[ https://issues.apache.org/jira/browse/YARN-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642524#comment-13642524 ] Omkar Vinit Joshi commented on YARN-578: There are 3 issues related to symlink attacks in serving logs and ShuffleService * Index file (file.out.index) :- [Location - SpillRecord.SpillRecord() - FSDataInputStream ] Here we are directly trying to read from file.out.index file (So the potential problem is that we ShuffleHandler may end up reading files of yarn user or yarn group user. [ yarn:yarn is running nodemanager ] * Map output file (file.out) :- [Location - ShuffleHandler.sendMapOutput() - RandomAccessFile ] Here too we are directly accessing file.out file. * Container Logs :- [Location - ContainerLogsPage.printLogs() - FileInputStream ] Here we are directly accessing container logs as yarn:yarn user. At present SecureIOUtils supports only FileInputStream, so I am adding support for 2 more streams, FSDataInputStream (This is required if you want a stream to be position readable or seekable) and RandomAccessFile. Filing a separate JIRA for this. HADOOP-9511 NodeManager should use SecureIOUtils for serving logs and intermediate outputs -- Key: YARN-578 URL: https://issues.apache.org/jira/browse/YARN-578 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Assignee: Omkar Vinit Joshi Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
