Re: Hadoop QBT job always failing

2020-05-21 Thread Akira Ajisaka
+CC: common-dev Hi Gavin, The job worked as expected in #146 and it helps us to find failed tests and build errors. https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/146/ However, the job itself failed in #147 and #148 by "pipe closed after 0 cycles"

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-21 Thread Eric Yang
Hi Steve, Thank you for sharing the work done for Amazon STS token to work with s3a connector. This works for direct HDFS to S3 bucket interaction. Your statement is also spot on for containers running in YARN has no mechanism to update the triple of session credentials. If I am not mistaken,

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-21 Thread Vipin Rathor
Hi Eric, Thanks for starting this discussion. Kerberos was developed decade before web development becomes popular. > There are some Kerberos limitations which does not work well in Hadoop. > Sure, Kerberos was developed long before the web but it was selected as de facto authentication

[jira] [Created] (HADOOP-17051) add hadoop-azure ITest to fail if all abfs or wasb tests are skipped

2020-05-21 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17051: --- Summary: add hadoop-azure ITest to fail if all abfs or wasb tests are skipped Key: HADOOP-17051 URL: https://issues.apache.org/jira/browse/HADOOP-17051

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-21 Thread Steve Loughran
On Wed, 6 May 2020 at 23:32, Eric Yang wrote: > Hi all, > > > 4. Passing different form of tokens does not work well with cloud provider > security mechanism. For example, passing AWS sts token for S3 bucket. > There is no renewal mechanism, nor good way to identify when the token > would

Branch 2.10 QBT builds

2020-05-21 Thread Jim Brennan
Does anyone know what's up with the qbt builds for branch-2.10? https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch-2.10-java7-linux-x86/692/console Last few have failed with: cd /testptch/hadoop /usr/bin/mvn --batch-mode

[jira] [Resolved] (HADOOP-8143) Change distcp to have -pb on by default

2020-05-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-8143. Resolution: Fixed > Change distcp to have -pb on by default >

[jira] [Reopened] (HADOOP-8143) Change distcp to have -pb on by default

2020-05-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-8143: > Change distcp to have -pb on by default > --- > >

[jira] [Resolved] (HADOOP-15300) distcp -update to WASB and ADL copies up all the files, always

2020-05-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15300. - Resolution: Duplicate > distcp -update to WASB and ADL copies up all the files, always