[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[ https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14188: --- Attachment: HADOOP-14188.04.patch Fix compile error. > Remove the usage of org.mockito.internal.util.reflection.Whitebox > - > > Key: HADOOP-14188 > URL: https://issues.apache.org/jira/browse/HADOOP-14188 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Minor > Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, > HADOOP-14188.03.patch, HADOOP-14188.04.patch > > > org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so > we need to remove the usage to upgrade Mockito. Getter/setter method can be > used instead of this hack. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14218) Replace assertThat with assertTrue in MetricsAsserts
[
https://issues.apache.org/jira/browse/HADOOP-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942816#comment-15942816
]
Hadoop QA commented on HADOOP-14218:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 57s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14218 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860603/HADOOP-14218.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 575d87761382 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 945b006 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11935/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11935/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11935/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Replace assertThat with assertTrue in MetricsAsserts
>
>
> Key: HADOOP-14218
> URL: https://issues.apache.org/jira/browse/HADOOP-14218
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akira Ajisaka
>Assignee: Akir
[jira] [Updated] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Mazur updated HADOOP-13887: Attachment: HADOOP-13897-branch-2-012.patch > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-trunk-011.patch, HADOOP-14171-001.patch > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14245) Use Mockito.when instead of Mockito.stub
Akira Ajisaka created HADOOP-14245: -- Summary: Use Mockito.when instead of Mockito.stub Key: HADOOP-14245 URL: https://issues.apache.org/jira/browse/HADOOP-14245 Project: Hadoop Common Issue Type: Bug Components: test Reporter: Akira Ajisaka Priority: Minor Mockito.stub was removed in Mockito 2. Mockito.when should be used instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14233) Delay construction of PreCondition.check failure message in Configuration#set
[
https://issues.apache.org/jira/browse/HADOOP-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942978#comment-15942978
]
Steve Loughran commented on HADOOP-14233:
-
+1
regarding the suggested logging improvements, every @ debug has traditionally
been guarded by an {{if (LOG.isDebugEnabled()}}; we've been slowly moving to
SLF4J on a file-by-file, package-by-package basis, which offers better dynamic
message construction. We could do more to speed that migration up.
> Delay construction of PreCondition.check failure message in Configuration#set
> -
>
> Key: HADOOP-14233
> URL: https://issues.apache.org/jira/browse/HADOOP-14233
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: HADOOP-14233.1.patch
>
>
> The String in the precondition check is constructed prior to failure
> detection. Since the normal case is no error, we can gain performance by
> delaying the construction of the string until the failure is detected.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14235) S3A Path does not understand colon (:) when globbing
[ https://issues.apache.org/jira/browse/HADOOP-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943020#comment-15943020 ] Steve Loughran commented on HADOOP-14235: - The bad news "one line fixes" are just as much trouble as anything else to get working. The main difference is the amount of review time each line gets is higher. This is the [hadoop-aws testing policy| https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md]. no declared test run: no review. No extra tests: either justify the lack of tests or expect no review. Here I'm not going to worry about it as HADOOP-3257 shows this is a broader problem. This alternative construction trick may work here, but it's only going to delay the problem. For example. it won't do anything for writing data when the destination path has a ":", in, recursive listFiles(path, recursive=true), calls, or anywhere else where the Path/2 constructors are used to build paths. Which is a lot of places in the code. I don't think we can/should be trying to fix this in a stack-trace-by-stack trace approach as it will simply break again the moment someone changes the codepath. Path itself is going to need tuning. That's not impossible; if you look at what goes on there to handle windows paths like "C:/something" you can see what has gone ahead. It's just a significant body of work, which someone who understands that bit of the code (not me!) needs to do. > S3A Path does not understand colon (:) when globbing > > > Key: HADOOP-14235 > URL: https://issues.apache.org/jira/browse/HADOOP-14235 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > S3 paths, colons ":" are valid character in S3 paths. However, the Java URI > class, which is used in the Path class, does not allow it. > This becomes a problem particularly when we are globbing S3 paths. The > globber thinks paths with colons are invalid paths and throws > URISyntaxException. > The reason is we are sharing Globber.java with all other Fs. Some of the > rules for regular Fs are not applicable to S3 just like this colon as an > example. > Same issue is reported here https://issues.apache.org/jira/browse/SPARK-20061 > The good news is I have a one line fix that I am about to send a pull request. > However, for a right fix, we should separate the S3 globber from the > Globber.java as proposed at https://issues.apache.org/jira/browse/HADOOP-13371 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943031#comment-15943031 ] Steve Loughran commented on HADOOP-14217: - Nicholas has pointed out the history of this —looks like it's old and fairly fundamental. Quick case-by-case workarounds aren't the right place to do it. As an example, those globber fixes are at best going to handle the case where the last element in a path contains a ":", code in which a parent path contains a colon, e,g "s3a://bucket/elt1:name/child2 is probably doomed, if not immediately, certainly when later things happen like it gets sent over the wire, deserialized, saved to the list of files to distcp, reloaded, etc. etc. Fixing this is a major piece of work, which will include tests (if HDFS adds ":" support, miniHDFS can be the test FS, otherwise some mock ramfs is needed). Along with the changes to the FS spec, we'd need a contract test for all supporting filesystems to implement, including the object stores, testing all those corner case and operations we can think of. Probably time to do a decent set of globber tests there too, which the contract lacks. I think that could be good, HADOOP-3257 would the be place, and HDFS the team to collaborate with. > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Reporter: Genmao Yu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943035#comment-15943035
]
Hadoop QA commented on HADOOP-13887:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 47s{color}
| {color:red} root-jdk1.8.0_121 with JDK v1.8.0_121 generated 5 new + 884
unchanged - 3 fixed = 889 total (was 887) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 5s{color}
| {color:red} root-jdk1.7.0_121 with JDK v1.7.0_121 generated 4 new + 981
unchanged - 2 fixed = 985 total (was 983) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} root: The patch generated 0 new + 21 unchanged - 2
fixed = 21 total (was 23) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
45s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943039#comment-15943039 ] Steve Loughran commented on HADOOP-13786: - side issue: code uses the traditional part-m-%04d naming pattern. Given that s3 uses the first few chars for its hashing, would it better to have a pattern like "p-%04d", or, to be really daring, put the UUID first. I don't think you'd see the benefits of this unless the output was [almost at the root of the bucket though |https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/] > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch, > s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox
[
https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943044#comment-15943044
]
Hadoop QA commented on HADOOP-14188:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 46 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 24s{color}
| {color:red} root generated 153 new + 777 unchanged - 0 fixed = 930 total (was
777) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} root: The patch generated 0 new + 978 unchanged - 1
fixed = 978 total (was 979) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
6s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
41s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue
[jira] [Commented] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943051#comment-15943051
]
ASF GitHub Bot commented on HADOOP-13371:
-
Github user steveloughran commented on the issue:
https://github.com/apache/hadoop/pull/204
What an s3a globber can do is skip the recursive treewalk of
pseudo-directories as it does the regexp. Instead it could ask for all children
of a path, before doing the filtering. Look at HADOOP-13208 for the details.
This patch is the preamble: a copy of the existing globber, and the initial
performance test which would be the baseline for measuring speedups. Look at
some of the other dir operations to see how we measure that: it's not elapsed
time, but in number of HTTP requests made.
I should warn that this is not trivial to do well. I put the work aside
once I came up in my head of some example directory layouts which would
overload a a naive "ask for all then filter" scheme; any deep& wide tree where
the wildcard was near the top of the tree and very selective would end up
asking for way too much data, and discarding it all. Listing is complex
for now we're doing s3guard, which switches to dynamo DB for consistency as
well as performance. If you do want to improve client performance, this is
somewhere where we would all benefit from you getting involved. I don't see any
of us going near other bits of speedup until after s3guard is in, because
everything will be stamping on the same lines of code and making merging a
pain. And s3guard is probably going to obsolete a lot of the speedup proposed
here on any bucket which has the DDB backing tables.
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943056#comment-15943056
]
Steve Loughran commented on HADOOP-14236:
-
LGTM; let's see what Aaron says.
why are you logging at trace() & not debug. All the rest of s3a logs at the
debug() level, which is the level log4j supports
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14236-HADOOP-13345.000.patch
>
>
> After running integration test {{ITestS3AFileSystemContract}}, I found the
> following items are not cleaned up in DynamoDB:
> {code}
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
> child=subdir
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
> child=file2
> {code}
> At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
> need to be careful when cleaning up test data.
> Then I found it’s a bug(?) in the code of integrating S3Guard with
> S3AFileSystem: for rename we miss sub-directory items to put (dest) and
> delete (src). The reason is that in S3A, we delete those fake directory
> objects if they are not necessary, e.g. non-empty. So when we list the
> objects to rename, the object summaries will only return _file_ objects. This
> has two consequences after rename:
> # there will be left items for src path in metadata store - left-overs will
> confuse {{get(Path)}} which should return null
> # we are not persisting the whole subtree for dest path to metadata store -
> this will break the DynamoDBMetadataStore invariant: _if a path exists, all
> its ancestors will also exist in the table_.
> Existing tests are not complaining about this though. If this is a real bug,
> let’s address it here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943066#comment-15943066
]
ASF GitHub Bot commented on HADOOP-14237:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/207#discussion_r108144389
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SharedInstanceProfileCredentialsProvider.java
---
@@ -58,6 +71,84 @@ public static SharedInstanceProfileCredentialsProvider
getInstance() {
return INSTANCE;
}
+ private AWSCredentials readCredentialsFromHDFS() {
+try {
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(s3crednetialPath)));
+ String accessKey = br.readLine();
+ String secretKey = br.readLine();
+ String token = br.readLine();
+ AWSCredentials credentials;
+ if (StringUtils.isEmpty(accessKey) ||
StringUtils.isEmpty(secretKey)) {
+// if there are no accessKey nor secretKey return null
+return null;
+ } else if (StringUtils.isNotEmpty(token)) {
+credentials = new BasicSessionCredentials(accessKey, secretKey,
token);
+ } else {
+credentials = new BasicAWSCredentials(accessKey, secretKey);
+ }
+ return credentials;
+} catch (Exception e) {
+ return null; // ignore the read errors
+ // throw new AmazonServiceException("Failed reading S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ private void writeCredentialsToHDFS(AWSCredentials credentials) {
+try {
+ // Simulate atomic write by creating a new s3credential file with
random string suffix and rename to s3crednetialPath
+ Path newS3crednetialPath = new Path(s3crednetialPath.toUri() +
RandomStringUtils.randomAlphanumeric(8));
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedWriter br = new BufferedWriter(new
OutputStreamWriter(fs.create(newS3crednetialPath, true)));
+ String accessKey = credentials.getAWSAccessKeyId();
+ String secretKey = credentials.getAWSSecretKey();
+ String token = "";
+ if (credentials instanceof BasicSessionCredentials) {
+token = ((BasicSessionCredentials) credentials).getSessionToken();
+ }
+ br.write(accessKey);
+ br.newLine();
+ br.write(secretKey);
+ br.newLine();
+ br.write(token);
+ br.newLine();
+ br.close();
+ fs.delete(s3crednetialPath, false);
+ fs.rename(newS3crednetialPath, s3crednetialPath);
+} catch (Exception e) {
+ // ignore write errors
+ // throw new AmazonServiceException("Failed writing S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ @Override
+ public AWSCredentials getCredentials() {
+for (int retry = 0; retry < maxRetries; retry++) {
+ try {
+AWSCredentials newCredentials = super.getCredentials();
+// if this new credentials is different from HDFS write back
+if (credentials == null ||
(!newCredentials.getAWSSecretKey().equals(credentials.getAWSSecretKey( {
+ credentials = newCredentials;
+ writeCredentialsToHDFS(credentials);
+}
+break;
+ } catch (Exception e) {
--- End diff --
I't use our normal Retry logic here, consider some sleep + jitter if it
really is caused by throttling
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: co
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943068#comment-15943068
]
ASF GitHub Bot commented on HADOOP-14237:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/207#discussion_r108144627
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SharedInstanceProfileCredentialsProvider.java
---
@@ -58,6 +71,84 @@ public static SharedInstanceProfileCredentialsProvider
getInstance() {
return INSTANCE;
}
+ private AWSCredentials readCredentialsFromHDFS() {
+try {
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(s3crednetialPath)));
+ String accessKey = br.readLine();
+ String secretKey = br.readLine();
+ String token = br.readLine();
+ AWSCredentials credentials;
+ if (StringUtils.isEmpty(accessKey) ||
StringUtils.isEmpty(secretKey)) {
+// if there are no accessKey nor secretKey return null
+return null;
+ } else if (StringUtils.isNotEmpty(token)) {
+credentials = new BasicSessionCredentials(accessKey, secretKey,
token);
+ } else {
+credentials = new BasicAWSCredentials(accessKey, secretKey);
+ }
+ return credentials;
+} catch (Exception e) {
+ return null; // ignore the read errors
+ // throw new AmazonServiceException("Failed reading S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ private void writeCredentialsToHDFS(AWSCredentials credentials) {
+try {
+ // Simulate atomic write by creating a new s3credential file with
random string suffix and rename to s3crednetialPath
+ Path newS3crednetialPath = new Path(s3crednetialPath.toUri() +
RandomStringUtils.randomAlphanumeric(8));
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedWriter br = new BufferedWriter(new
OutputStreamWriter(fs.create(newS3crednetialPath, true)));
+ String accessKey = credentials.getAWSAccessKeyId();
+ String secretKey = credentials.getAWSSecretKey();
+ String token = "";
+ if (credentials instanceof BasicSessionCredentials) {
--- End diff --
I would only allow session credentials to persist, so as to reduce risk of
leakage of persistent secrets
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[
https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943069#comment-15943069
]
ASF GitHub Bot commented on HADOOP-14237:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/207#discussion_r108144687
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SharedInstanceProfileCredentialsProvider.java
---
@@ -58,6 +71,84 @@ public static SharedInstanceProfileCredentialsProvider
getInstance() {
return INSTANCE;
}
+ private AWSCredentials readCredentialsFromHDFS() {
+try {
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(s3crednetialPath)));
+ String accessKey = br.readLine();
+ String secretKey = br.readLine();
+ String token = br.readLine();
+ AWSCredentials credentials;
+ if (StringUtils.isEmpty(accessKey) ||
StringUtils.isEmpty(secretKey)) {
+// if there are no accessKey nor secretKey return null
+return null;
+ } else if (StringUtils.isNotEmpty(token)) {
+credentials = new BasicSessionCredentials(accessKey, secretKey,
token);
+ } else {
+credentials = new BasicAWSCredentials(accessKey, secretKey);
+ }
+ return credentials;
+} catch (Exception e) {
+ return null; // ignore the read errors
+ // throw new AmazonServiceException("Failed reading S3 credentials
from HDFS " + e.getStackTrace());
+}
+ }
+
+ private void writeCredentialsToHDFS(AWSCredentials credentials) {
+try {
+ // Simulate atomic write by creating a new s3credential file with
random string suffix and rename to s3crednetialPath
+ Path newS3crednetialPath = new Path(s3crednetialPath.toUri() +
RandomStringUtils.randomAlphanumeric(8));
+ FileSystem fs = FileSystem.get(new Configuration());
+ BufferedWriter br = new BufferedWriter(new
OutputStreamWriter(fs.create(newS3crednetialPath, true)));
+ String accessKey = credentials.getAWSAccessKeyId();
+ String secretKey = credentials.getAWSSecretKey();
+ String token = "";
+ if (credentials instanceof BasicSessionCredentials) {
+token = ((BasicSessionCredentials) credentials).getSessionToken();
+ }
+ br.write(accessKey);
+ br.newLine();
+ br.write(secretKey);
+ br.newLine();
+ br.write(token);
+ br.newLine();
+ br.close();
+ fs.delete(s3crednetialPath, false);
+ fs.rename(newS3crednetialPath, s3crednetialPath);
+} catch (Exception e) {
+ // ignore write errors
+ // throw new AmazonServiceException("Failed writing S3 credentials
from HDFS " + e.getStackTrace());
--- End diff --
log @ debug at the very least
> S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
> ---
>
> Key: HADOOP-14237
> URL: https://issues.apache.org/jira/browse/HADOOP-14237
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
> Environment: EC2, AWS
>Reporter: Kazuyuki Tanimura
>
> When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails
> getting the instance profile credentials, eventually all jobs on the cluster
> fail. Since a number of S3A clients (all mappers and reducers) try to get the
> credentials, the AWS credential endpoint starts responding 5xx and 4xx error
> codes.
> SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it,
> but it still does not share the credentials with other EC2 nodes / JVM
> processes.
> This issue prevents users from creating Hadoop clusters on EC2
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943071#comment-15943071 ] Steve Loughran commented on HADOOP-14237: - I see: it's extracting the credentials, then saving them to the cluster FS, so that no clients need to hit the IAM infra so much. if it overloads, it reads back from HDFS. If this is to go in, as well as needing per-user temp dir, and all the various tests, maybe even expiry of credentials, this MUST be its own credential provider. This needs to be optional, and now we've added the ability to declare your own providers, that'll be how people use it. Test plan: * split save/load from rest of provider, test independently, including handling some read/write failure conditions. * verify that credentials are saved on successful auth. * maybe using mocking simulate IAM overload > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14239) S3A Retry Multiple S3 Key Deletion
[ https://issues.apache.org/jira/browse/HADOOP-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14239: Affects Version/s: (was: 2.8.1) (was: 3.0.0-alpha2) (was: 3.0.0-alpha1) > S3A Retry Multiple S3 Key Deletion > -- > > Key: HADOOP-14239 > URL: https://issues.apache.org/jira/browse/HADOOP-14239 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When fs.s3a.multiobjectdelete.enable == true, It tries to delete multiple S3 > keys at once. > Although this is a great feature, it becomes problematic when AWS fails > deleting some S3 keys out of the deletion list. The aws-java-sdk internally > retries to delete them, but it does not help because it simply retries the > same list of S3 keys including the successfully deleted ones. In that case, > all successive retries fail deleting previously deleted keys since they do > not exist any more. Eventually it throws an Exception and leads to a job > failure entirely. > Luckily, the AWS API reports which keys it failed to delete. We should retry > only for the keys that failed to be deleted from S3A -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-13363: Attachment: HADOOP-13363.004.patch Attaching a patch to update Dockerfile to download protobuf 3.2.0. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943073#comment-15943073 ] Tsuyoshi Ozawa commented on HADOOP-13363: - Thanks Allen! Updated. @stack unfortunately, not yet. I've also checked discussion on HDFS-11010. Based on the discussion on the jira and here, I would like to start next argument. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943081#comment-15943081 ] Hadoop QA commented on HADOOP-13363: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/11938/console in case of problems. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14239) S3A Retry Multiple S3 Key Deletion
[ https://issues.apache.org/jira/browse/HADOOP-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943084#comment-15943084 ] Steve Loughran commented on HADOOP-14239: - I'm going to mark this up as a dupe of HADOOP-11572, which is where I first sat down to look at this problem, or at least the related one of concurrent object delete failure. I stopped there as I concluded I didn't know enough about how things fail. It may be someone else deleted the object —race condition, that being the likeliest. Or its permission related, auth related, etc etc. Without knowing all the failure modes, I wasn't confident I could implement the right policy. If you have more insight there, that'd be good. As usual, the tests are as important as the production code; the landsat-pds repo the read only one to look at. > S3A Retry Multiple S3 Key Deletion > -- > > Key: HADOOP-14239 > URL: https://issues.apache.org/jira/browse/HADOOP-14239 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When fs.s3a.multiobjectdelete.enable == true, It tries to delete multiple S3 > keys at once. > Although this is a great feature, it becomes problematic when AWS fails > deleting some S3 keys out of the deletion list. The aws-java-sdk internally > retries to delete them, but it does not help because it simply retries the > same list of S3 keys including the successfully deleted ones. In that case, > all successive retries fail deleting previously deleted keys since they do > not exist any more. Eventually it throws an Exception and leads to a job > failure entirely. > Luckily, the AWS API reports which keys it failed to delete. We should retry > only for the keys that failed to be deleted from S3A -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943104#comment-15943104 ] Steve Loughran commented on HADOOP-14237: - +persisted data structure should use the JECKS encrypted credential mechanism, so that it isn't stored in plaintext, even in HDFS. Processes which can access the data would need to be given the shared key and path needed to find and read the data, > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14241) Add ADLS sensitive config keys to default list
[
https://issues.apache.org/jira/browse/HADOOP-14241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943109#comment-15943109
]
Steve Loughran commented on HADOOP-14241:
-
why the change to the regexp? multiline support? Someone who understands the
original code will need to review that
> Add ADLS sensitive config keys to default list
> --
>
> Key: HADOOP-14241
> URL: https://issues.apache.org/jira/browse/HADOOP-14241
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl, security
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14241.001.patch
>
>
> ADLS sensitive credential config keys should be added to the default list for
> {{hadoop.security.sensitive-config-keys}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943151#comment-15943151
]
Hadoop QA commented on HADOOP-13363:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
41s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
11s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
11s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} root: The patch generated 0 new + 27 unchanged - 1
fixed = 27 total (was 28) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
15s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m
15s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
10s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
17s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13363 |
| JIRA Patch URL |
https://issues.ap
[jira] [Updated] (HADOOP-14235) S3A Path does not understand colon (:) when globbing
[ https://issues.apache.org/jira/browse/HADOOP-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14235: Affects Version/s: (was: 3.0.0-alpha2) (was: 3.0.0-alpha1) (was: 2.8.0) > S3A Path does not understand colon (:) when globbing > > > Key: HADOOP-14235 > URL: https://issues.apache.org/jira/browse/HADOOP-14235 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura > > S3 paths, colons ":" are valid character in S3 paths. However, the Java URI > class, which is used in the Path class, does not allow it. > This becomes a problem particularly when we are globbing S3 paths. The > globber thinks paths with colons are invalid paths and throws > URISyntaxException. > The reason is we are sharing Globber.java with all other Fs. Some of the > rules for regular Fs are not applicable to S3 just like this colon as an > example. > Same issue is reported here https://issues.apache.org/jira/browse/SPARK-20061 > The good news is I have a one line fix that I am about to send a pull request. > However, for a right fix, we should separate the S3 globber from the > Globber.java as proposed at https://issues.apache.org/jira/browse/HADOOP-13371 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14235) S3A Path does not understand colon (:) when globbing
[ https://issues.apache.org/jira/browse/HADOOP-14235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14235: Priority: Minor (was: Major) > S3A Path does not understand colon (:) when globbing > > > Key: HADOOP-14235 > URL: https://issues.apache.org/jira/browse/HADOOP-14235 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Priority: Minor > > S3 paths, colons ":" are valid character in S3 paths. However, the Java URI > class, which is used in the Path class, does not allow it. > This becomes a problem particularly when we are globbing S3 paths. The > globber thinks paths with colons are invalid paths and throws > URISyntaxException. > The reason is we are sharing Globber.java with all other Fs. Some of the > rules for regular Fs are not applicable to S3 just like this colon as an > example. > Same issue is reported here https://issues.apache.org/jira/browse/SPARK-20061 > The good news is I have a one line fix that I am about to send a pull request. > However, for a right fix, we should separate the S3 globber from the > Globber.java as proposed at https://issues.apache.org/jira/browse/HADOOP-13371 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943200#comment-15943200 ] Steve Loughran commented on HADOOP-13786: - OK, scheduled a meetup at 10:00 PST, 18:00 BST: https://hortonworks.webex.com/hortonworks/j.php?MTID=mfb8dcba47d84a7f67ee9f72872e47cb4 ; I think that URL should be sufficient to join. > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch, > s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-13363: Attachment: HADOOP-13363.005.patch Trying to fix a problem by lack of execution permission. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943253#comment-15943253 ] Hadoop QA commented on HADOOP-13363: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/11939/console in case of problems. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943482#comment-15943482
]
Hadoop QA commented on HADOOP-13363:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
27s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
24s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 35s{color}
| {color:red} root generated 1232 new + 0 unchanged - 0 fixed = 1232 total (was
0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} root: The patch generated 0 new + 27 unchanged - 1
fixed = 27 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 632 new + 104 unchanged - 0 fixed
= 736 total (was 104) {color} |
| {color:orange}-0{color} | {color:orange} shelldocs {color} | {color:orange}
0m 21s{color} | {color:orange} The patch generated 282 new + 104 unchanged - 0
fixed = 386 total (was 104) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
12s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 9 new + 0
unchanged - 0 fixed = 9 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
51s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason
[jira] [Commented] (HADOOP-14218) Replace assertThat with assertTrue in MetricsAsserts
[
https://issues.apache.org/jira/browse/HADOOP-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943530#comment-15943530
]
Haibo Chen commented on HADOOP-14218:
-
+1 nonbinding
> Replace assertThat with assertTrue in MetricsAsserts
>
>
> Key: HADOOP-14218
> URL: https://issues.apache.org/jira/browse/HADOOP-14218
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HADOOP-14218.01.patch
>
>
> {code:title=MetricsAsserts.java}
> public static void assertCounterGt(String name, long greater,
> MetricsRecordBuilder rb) {
> Assert.assertThat("Bad value for metric " + name, getLongCounter(name,
> rb),
> new GreaterThan(greater));
> }
> {code}
> The following code cannot be compiled with Mockito 2.1+ because it does not
> depend on org.hamcrest.Matcher anymore. We can simply replace this code with
> {{assertTrue(message, getLongCounter() > greater)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14233) Delay construction of PreCondition.check failure message in Configuration#set
[ https://issues.apache.org/jira/browse/HADOOP-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-14233: - Resolution: Fixed Fix Version/s: 3.0.0-alpha3 2.8.1 2.9.0 Target Version/s: 2.9.0, 2.8.1, 3.0.0-alpha3 Status: Resolved (was: Patch Available) Thanks [~ste...@apache.org] and [~hanishakoneru] for the reviews. Committed this trivial patch to trunk, branch-2, and branch-2.8 > Delay construction of PreCondition.check failure message in Configuration#set > - > > Key: HADOOP-14233 > URL: https://issues.apache.org/jira/browse/HADOOP-14233 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14233.1.patch > > > The String in the precondition check is constructed prior to failure > detection. Since the normal case is no error, we can gain performance by > delaying the construction of the string until the failure is detected. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943537#comment-15943537 ] Tsuyoshi Ozawa commented on HADOOP-13363: - protoc fails to generate files? Let me survey for a while. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13926) S3Guard: Improve listLocatedStatus and listFiles
[
https://issues.apache.org/jira/browse/HADOOP-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13926:
---
Priority: Major (was: Minor)
> S3Guard: Improve listLocatedStatus and listFiles
>
>
> Key: HADOOP-13926
> URL: https://issues.apache.org/jira/browse/HADOOP-13926
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Rajesh Balamohan
>Assignee: Steve Loughran
> Attachments: HADOOP-13926.wip.proto.branch-13345.1.patch
>
>
> Need to check if {{listLocatedStatus}} can make use of metastore's
> listChildren feature.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14245) Use Mockito.when instead of Mockito.stub
[ https://issues.apache.org/jira/browse/HADOOP-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14245: --- Issue Type: Test (was: Bug) > Use Mockito.when instead of Mockito.stub > > > Key: HADOOP-14245 > URL: https://issues.apache.org/jira/browse/HADOOP-14245 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Priority: Minor > > Mockito.stub was removed in Mockito 2. Mockito.when should be used instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943685#comment-15943685 ] Andrew Wang commented on HADOOP-13363: -- I see there's been motion on this JIRA, did we ever have the ML discussion? Is source compatibility handled by the shaded client? Did we test wire compatibility? [~chris.douglas] brought up some incompatibilities on HDFS-11010 that I don't think were resolved. I also asked this on HDFS-11010, but what do we gain from upgrading to a 3.x release of PB? > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14241) Add ADLS sensitive config keys to default list
[
https://issues.apache.org/jira/browse/HADOOP-14241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943702#comment-15943702
]
John Zhuge commented on HADOOP-14241:
-
Yes [~steve_l], the change to regexp is to support multi-line value. Since we
are adding more and more entries to the
{{hadoop.security.sensitive-config-keys}} in this JIRA and HADOOP-14243. Comma
separated string on one line is not manageable.
[~sean_impala_9b93], could you please take a look?
> Add ADLS sensitive config keys to default list
> --
>
> Key: HADOOP-14241
> URL: https://issues.apache.org/jira/browse/HADOOP-14241
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl, security
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14241.001.patch
>
>
> ADLS sensitive credential config keys should be added to the default list for
> {{hadoop.security.sensitive-config-keys}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9747: Attachment: HADOOP-9747.trunk.patch This patch incorporates all the open subtasks. Branch-2 patch forthcoming after conflicts resolved. Major highlights: # Frivolous synchronization in common code paths is eliminated. # Login/logout/relogin synchronizes on the subject's private credentials to ensure atomic update relative to each other and SASL authentication. # Supports concurrent UGI instantiation - ctor does not access priv creds that require locking and mutate during relogin. # A login context wrapper "remembers" the login configuration so the class statics for principal and keytab can be eliminated. # Is keytab/ticket is based on the login config, not the current state of the priv creds. #* prevents race during relogin where new instances might think the ugi is not a keytab when logout has occurred and relogin underway. #* ugis created for keytab users after a relogin failure now remember they are keytab based. # Creating ugi from a subject will learn the principal and possibly keytab #* no more "external" ugi hack #* subject based-ugis will relogin, ie. after ipc errors Basic design is: * HadoopLoginContext wraps a "real" LoginContext and remembers the javax-derived HadoopConfiguration which is normally not accessible. * HadoopConfiguration is based on LoginParams which track parameters for login and relogin. * LoginParams for pre-existing subjects are learned prior to login and reused for subsequent relogins. * LoginParams for new subjects are updated post-login. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9747: Target Version/s: 2.8.1 Status: Patch Available (was: Open) > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0-alpha1, 2.0.0-alpha, 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-9749) Remove synchronization for UGI.getCurrentUser
[
https://issues.apache.org/jira/browse/HADOOP-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp resolved HADOOP-9749.
-
Resolution: Duplicate
> Remove synchronization for UGI.getCurrentUser
> -
>
> Key: HADOOP-9749
> URL: https://issues.apache.org/jira/browse/HADOOP-9749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-9749.branch-2.patch, HADOOP-9749.trunk.2.patch,
> HADOOP-9749.trunk.patch, HADOOP-9749.trunk.patch
>
>
> HADOOP-7854 added synchronization to {{getCurrentUser}} due to
> {{ConcurrentModificationExceptions}}. This degrades NN call handler
> performance.
> The problem was not well understood at the time, but it's caused by a
> collision between relogin and {{getCurrentUser}} due to a bug in
> {{Krb5LoginModule}}. Avoiding the collision will allow removal of the
> synchronization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9749) Remove synchronization for UGI.getCurrentUser
[
https://issues.apache.org/jira/browse/HADOOP-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-9749:
Status: Open (was: Patch Available)
Incorporating in rollup HADOOP-9747.
> Remove synchronization for UGI.getCurrentUser
> -
>
> Key: HADOOP-9749
> URL: https://issues.apache.org/jira/browse/HADOOP-9749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0-alpha1, 2.0.0-alpha, 0.23.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-9749.branch-2.patch, HADOOP-9749.trunk.2.patch,
> HADOOP-9749.trunk.patch, HADOOP-9749.trunk.patch
>
>
> HADOOP-7854 added synchronization to {{getCurrentUser}} due to
> {{ConcurrentModificationExceptions}}. This degrades NN call handler
> performance.
> The problem was not well understood at the time, but it's caused by a
> collision between relogin and {{getCurrentUser}} due to a bug in
> {{Krb5LoginModule}}. Avoiding the collision will allow removal of the
> synchronization.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14245) Use Mockito.when instead of Mockito.stub
[ https://issues.apache.org/jira/browse/HADOOP-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou reassigned HADOOP-14245: -- Assignee: Xiaobing Zhou > Use Mockito.when instead of Mockito.stub > > > Key: HADOOP-14245 > URL: https://issues.apache.org/jira/browse/HADOOP-14245 > Project: Hadoop Common > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Xiaobing Zhou >Priority: Minor > > Mockito.stub was removed in Mockito 2. Mockito.when should be used instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14240) Configuration#get return value optimization
[ https://issues.apache.org/jira/browse/HADOOP-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-14240: - Resolution: Fixed Fix Version/s: 3.0.0-alpha3 2.8.1 2.9.0 Status: Resolved (was: Patch Available) Thanks, [~ajisakaa]. I have committed this backwards compatible performance jira to trunk, branch-2, and branch-2.8. > Configuration#get return value optimization > --- > > Key: HADOOP-14240 > URL: https://issues.apache.org/jira/browse/HADOOP-14240 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14240.1.patch, HADOOP-14240.2.patch > > > The string array return value can be more efficiently determined and some > general redundancies can be removed to improve the speed for > Configuration.get. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14218) Replace assertThat with assertTrue in MetricsAsserts
[
https://issues.apache.org/jira/browse/HADOOP-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943760#comment-15943760
]
Arpit Agarwal edited comment on HADOOP-14218 at 3/27/17 6:12 PM:
-
+1
was (Author: arpitagarwal):
+!
> Replace assertThat with assertTrue in MetricsAsserts
>
>
> Key: HADOOP-14218
> URL: https://issues.apache.org/jira/browse/HADOOP-14218
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HADOOP-14218.01.patch
>
>
> {code:title=MetricsAsserts.java}
> public static void assertCounterGt(String name, long greater,
> MetricsRecordBuilder rb) {
> Assert.assertThat("Bad value for metric " + name, getLongCounter(name,
> rb),
> new GreaterThan(greater));
> }
> {code}
> The following code cannot be compiled with Mockito 2.1+ because it does not
> depend on org.hamcrest.Matcher anymore. We can simply replace this code with
> {{assertTrue(message, getLongCounter() > greater)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14218) Replace assertThat with assertTrue in MetricsAsserts
[
https://issues.apache.org/jira/browse/HADOOP-14218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943760#comment-15943760
]
Arpit Agarwal commented on HADOOP-14218:
+!
> Replace assertThat with assertTrue in MetricsAsserts
>
>
> Key: HADOOP-14218
> URL: https://issues.apache.org/jira/browse/HADOOP-14218
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HADOOP-14218.01.patch
>
>
> {code:title=MetricsAsserts.java}
> public static void assertCounterGt(String name, long greater,
> MetricsRecordBuilder rb) {
> Assert.assertThat("Bad value for metric " + name, getLongCounter(name,
> rb),
> new GreaterThan(greater));
> }
> {code}
> The following code cannot be compiled with Mockito 2.1+ because it does not
> depend on org.hamcrest.Matcher anymore. We can simply replace this code with
> {{assertTrue(message, getLongCounter() > greater)}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
Robert Kanter created HADOOP-14246:
--
Summary: Authentication Tokens should use SecureRandom instead of
Random and 256 bit secrets
Key: HADOOP-14246
URL: https://issues.apache.org/jira/browse/HADOOP-14246
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 2.9.0
Reporter: Robert Kanter
Assignee: Robert Kanter
{{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
{{long}} generated by {{Random}} (which is then converted to a {{String}} and
is 160 bits) for secrets.
We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-14246:
---
Attachment: HADOOP-14246.001.patch
The 001 patch:
- Changes {{Random}} to {{SecureRanom}} in {{RandomSignerSecretProvider}} and
{{ZKSignerSecretProvider}}. Unit tests continue to use {{Random}} because we
need to be able to predict the RNG to verify in the tests and {{SecureRandom}}
ignores the seed on linux platforms.
- Changes the length of the secret from 160 bits (a Long converted to a String)
to 256 bits in {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}}.
We luckily store the length of the secret in the data written to ZooKeeper, so
there's no compatibility problems changing the length of the secret.
- Added a unit test to for changing the length of the secret
- Reduced execution time of {{TestRandomSignerSecretProvuder}} from ~50 seconds
to less than 1 second by mocking the rollover scheduling like we already did in
{{TestZKSignerSecretProvider}}
I still need to go and verify in an actual cluster, but here is the patch in
the meantime.
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-14246:
---
Status: Patch Available (was: Open)
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14207) "dfsadmin -refreshCallQueue" command is failing with DecayRpcScheduler
[
https://issues.apache.org/jira/browse/HADOOP-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HADOOP-14207:
Target Version/s: 2.8.1 (was: 2.8.0)
> "dfsadmin -refreshCallQueue" command is failing with DecayRpcScheduler
> --
>
> Key: HADOOP-14207
> URL: https://issues.apache.org/jira/browse/HADOOP-14207
> Project: Hadoop Common
> Issue Type: Bug
> Components: rpc-server
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HADOOP-14207.001.patch, HADOOP-14207.002.patch,
> HADOOP-14207.003.patch
>
>
> {noformat}
> java.lang.RuntimeException: org.apache.hadoop.ipc.DecayRpcScheduler could not
> be constructed.
> at
> org.apache.hadoop.ipc.CallQueueManager.createScheduler(CallQueueManager.java:89)
> at
> org.apache.hadoop.ipc.CallQueueManager.swapQueue(CallQueueManager.java:260)
> at org.apache.hadoop.ipc.Server.refreshCallQueue(Server.java:650)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.refreshCallQueue(NameNodeRpcServer.java:1582)
> at
> org.apache.hadoop.ipc.protocolPB.RefreshCallQueueProtocolServerSideTranslatorPB.refreshCallQueue(RefreshCallQueueProtocolServerSideTranslatorPB.java:49)
> at
> org.apache.hadoop.ipc.proto.RefreshCallQueueProtocolProtos$RefreshCallQueueProtocolService$2.callBlockingMethod(RefreshCallQueueProtocolProtos.java:769)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> DecayRpcSchedulerMetrics2.ipc.65110 already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:144)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:117)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14223) Extend FileStatus#toString() to include details like Erasure Coding and Encryption
[
https://issues.apache.org/jira/browse/HADOOP-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HADOOP-14223:
Attachment: HADOOP-14223.01.patch
Attached v01 patch to extend FileStatus as following.
* added hasAcl() to return the ACL bit set in FsPermission, and on the same
lines of isEncrypted() and isErasureCoded()
* extended toString() to return set or unset details on ACL, encryption and
erasure coding.
* FileSystem spec updated
* Unit tests to verify the newly added details in FileStatus
> Extend FileStatus#toString() to include details like Erasure Coding and
> Encryption
> --
>
> Key: HADOOP-14223
> URL: https://issues.apache.org/jira/browse/HADOOP-14223
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-14223.01.patch
>
>
> HDFS-6843 and HADOOP-13715 have enhanced {{FileStatus}} to include details on
> whether the underlying path is Encrypted and Erasure Coded. The additional
> details are embedded in the FsPermission high order bits. It would be really
> helpful for debugging if FileStatus#toString() returns these new bits details
> along with already existing one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14223) Extend FileStatus#toString() to include details like Erasure Coding and Encryption
[
https://issues.apache.org/jira/browse/HADOOP-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943866#comment-15943866
]
Manoj Govindassamy edited comment on HADOOP-14223 at 3/27/17 7:11 PM:
--
Attached v01 patch to extend FileStatus as following.
* added hasAcl() to return the ACL bit set in FsPermission, and on the same
lines of isEncrypted() and isErasureCoded()
* extended toString() to return set or unset details on ACL, encryption and
erasure coding.
* FileSystem spec updated
* Unit tests to verify the newly added details in FileStatus
[~andrew.wang], [~ste...@apache.org] can you please take a look at the patch ?
was (Author: manojg):
Attached v01 patch to extend FileStatus as following.
* added hasAcl() to return the ACL bit set in FsPermission, and on the same
lines of isEncrypted() and isErasureCoded()
* extended toString() to return set or unset details on ACL, encryption and
erasure coding.
* FileSystem spec updated
* Unit tests to verify the newly added details in FileStatus
> Extend FileStatus#toString() to include details like Erasure Coding and
> Encryption
> --
>
> Key: HADOOP-14223
> URL: https://issues.apache.org/jira/browse/HADOOP-14223
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-14223.01.patch
>
>
> HDFS-6843 and HADOOP-13715 have enhanced {{FileStatus}} to include details on
> whether the underlying path is Encrypted and Erasure Coded. The additional
> details are embedded in the FsPermission high order bits. It would be really
> helpful for debugging if FileStatus#toString() returns these new bits details
> along with already existing one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14223) Extend FileStatus#toString() to include details like Erasure Coding and Encryption
[
https://issues.apache.org/jira/browse/HADOOP-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HADOOP-14223:
Status: Patch Available (was: Open)
> Extend FileStatus#toString() to include details like Erasure Coding and
> Encryption
> --
>
> Key: HADOOP-14223
> URL: https://issues.apache.org/jira/browse/HADOOP-14223
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-14223.01.patch
>
>
> HDFS-6843 and HADOOP-13715 have enhanced {{FileStatus}} to include details on
> whether the underlying path is Encrypted and Erasure Coded. The additional
> details are embedded in the FsPermission high order bits. It would be really
> helpful for debugging if FileStatus#toString() returns these new bits details
> along with already existing one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Open (was: Patch Available) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch, > s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Patch Available (was: Open) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch, > HADOOP-13786-HADOOP-13345-021.patch, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Attachment: HADOOP-13786-HADOOP-13345-021.patch HADOOP-13786 patch 021: unifying data structures and s3 client use This iteration is starting to move towards using the S3AFS writeOoperations, with the first step being: unified JSON persistence across alll committers * New serializable type, MultiplePendingCommits , containing a list of SinglePendingCommit instances; the latter being the things the magic committer saves. * SinglePendingCommit adds fields/operations needed by staging committer. It's a bit ungainly right now; view as an interim step. * Staging committers and tests all moved over to this datatype instead of seralized java stream. Good: debugging, security, validation logic. Bad: JSON serialization overhead. * in the move, switches the various lists that the thread-pooled staging code buids up to being synchronized lists. I think there may have been risk of race conditions there. Other changes * default unique name option == false * tests can handle option of unique vs non-unique filenames * and the partition committer skips the Mapper test. Doesn't make sense. Essentinally: the unique name algorithm doesn't work withy map tasks, as they expect a part-m-/ dir with children explicitly named "index" and "data". Adding unique names under index and data breaks this. I'm still undecided about what the best default value is here, more insight and experimentation needed > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch, > HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch, > HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch, > HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch, > HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch, > HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch, > HADOOP-13786-HADOOP-13345-021.patch, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943911#comment-15943911
]
Hadoop QA commented on HADOOP-14246:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange} hadoop-common-project/hadoop-auth: The patch
generated 1 new + 37 unchanged - 4 fixed = 38 total (was 41) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
38s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14246 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860695/HADOOP-14246.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cf3b377a02d5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db2adf3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11941/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-auth.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11941/testReport/ |
| modules | C: hadoop-common-project/hadoop-auth U:
hadoop-common-project/hadoop-auth |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11941/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
>
[jira] [Commented] (HADOOP-14146) KerberosAuthenticationHandler should authenticate with SPN in AP-REQ
[ https://issues.apache.org/jira/browse/HADOOP-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943916#comment-15943916 ] Daryn Sharp commented on HADOOP-14146: -- bq. One question, maybe we could avoid maintaining the DER parser codes like follows by leveraging the ASN1 and Kerberos library in Apache Kerby? I had quickly looked at Kerby and Apache DS. I didn't use them for a few reasons: The extra dependency burden. Relatively heavy weight to partially decode the ticket - we need just a tiny portion of it. The encoding format is decades old so there shouldn't be a maintainability issue. Thoughts? > KerberosAuthenticationHandler should authenticate with SPN in AP-REQ > > > Key: HADOOP-14146 > URL: https://issues.apache.org/jira/browse/HADOOP-14146 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-14146.1.patch, HADOOP-14146.patch > > > Many attempts (HADOOP-10158, HADOOP-11628, HADOOP-13565) have tried to add > multiple SPN host and/or realm support to spnego authentication. The basic > problem is the server tries to guess and/or brute force what SPN the client > used. The server should just decode the SPN from the AP-REQ. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943780#comment-15943780
]
Robert Kanter edited comment on HADOOP-14246 at 3/27/17 7:53 PM:
-
The 001 patch:
- Changes {{Random}} to {{SecureRanom}} in {{RandomSignerSecretProvider}} and
{{ZKSignerSecretProvider}}. Unit tests continue to use {{Random}} because we
need to be able to predict the RNG to verify in the tests and {{SecureRandom}}
ignores the seed on linux platforms.
- Changes the length of the secret from 160 bits (a Long converted to a String)
to 256 bits in {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}}.
We luckily store the length of the secret in the data written to ZooKeeper, so
there's no compatibility problems changing the length of the secret.
- Added a unit test to for changing the length of the secret
- Reduced execution time of {{TestRandomSignerSecretProvuder}} from ~50 seconds
to less than 1 second by mocking the rollover scheduling like we already did in
{{TestZKSignerSecretProvider}}
I've also verified that this works in an actual cluster.
was (Author: rkanter):
The 001 patch:
- Changes {{Random}} to {{SecureRanom}} in {{RandomSignerSecretProvider}} and
{{ZKSignerSecretProvider}}. Unit tests continue to use {{Random}} because we
need to be able to predict the RNG to verify in the tests and {{SecureRandom}}
ignores the seed on linux platforms.
- Changes the length of the secret from 160 bits (a Long converted to a String)
to 256 bits in {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}}.
We luckily store the length of the secret in the data written to ZooKeeper, so
there's no compatibility problems changing the length of the secret.
- Added a unit test to for changing the length of the secret
- Reduced execution time of {{TestRandomSignerSecretProvuder}} from ~50 seconds
to less than 1 second by mocking the rollover scheduling like we already did in
{{TestZKSignerSecretProvider}}
I still need to go and verify in an actual cluster, but here is the patch in
the meantime.
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[
https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943944#comment-15943944
]
Hadoop QA commented on HADOOP-9747:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 14 new + 118 unchanged - 25 fixed = 132 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
38s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-9747 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860691/HADOOP-9747.trunk.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 01692bb8c8db 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db2adf3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11940/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11940/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11940/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Reduce unnecessary UGI synchronization
> --
>
> Key: HADOOP-9747
> URL: https://issues.apache.org/jira/browse/HADOOP-9747
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
>
[jira] [Commented] (HADOOP-14223) Extend FileStatus#toString() to include details like Erasure Coding and Encryption
[
https://issues.apache.org/jira/browse/HADOOP-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944000#comment-15944000
]
Andrew Wang commented on HADOOP-14223:
--
Hi Manoj, thanks for working on this! Had a few review comments:
* I like adding hasAcl to FileStatus, since an FsPermission seems to
encapsulate just the Unix-style permissions. Seems like we have some
duplication now with the FsPermission getAclBit / getEncryptedBit /
getErausreCodedBit getters, perhaps we should deprecate them? getECBit we can
also annotate as Private. I'm hoping we can clean up these fields/getters in
FsPermission when HDFS-6984 goes in.
* I don't think toString should be required as part of assertErasureCoded in
ContractTestUtils, since as we discussed on HADOOP-13715, toString isn't part
of the public contract for a HCFS. It's fine to have toString asserts in Hadoop
or HDFS-specific tests though.
* Nit in filesystem.md: A given path only has a single ACL composed of multiple
AC entries, so I think "any ACLs" -> "an ACL". Another nit is that Encryption
or Erasure Coded aren't proper nouns, so I'd prefer them to be lower case.
{code:title=TestFileStatus}
if (fileStatus.hasAcl()) {
expected.append("hasAcl=").append(true).append("; ");
} else {
expected.append("hasAcl=").append(false).append("; ");
}
{code}
Could these be made more concise, e.g.
{code}
expected.append("hasAcl=").append(fileStatus.hasAcl()).append("; ");
{code}
> Extend FileStatus#toString() to include details like Erasure Coding and
> Encryption
> --
>
> Key: HADOOP-14223
> URL: https://issues.apache.org/jira/browse/HADOOP-14223
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-14223.01.patch
>
>
> HDFS-6843 and HADOOP-13715 have enhanced {{FileStatus}} to include details on
> whether the underlying path is Encrypted and Erasure Coded. The additional
> details are embedded in the FsPermission high order bits. It would be really
> helpful for debugging if FileStatus#toString() returns these new bits details
> along with already existing one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[
https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944031#comment-15944031
]
Hadoop QA commented on HADOOP-14202:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
56s{color} | {color:red} The patch generated 3 new + 88 unchanged - 12 fixed =
91 total (was 100) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 7s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
31s{color} | {color:green} hadoop-yarn in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
53s{color} | {color:green} hadoop-mapreduce-project in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed TAP tests | hadoop_verify_user_resolves.bats.tap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14202 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860039/HADOOP-14202.01.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 096e90bbc35e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db2adf3 |
| shellcheck | v0.4.5 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/diff-patch-shellcheck.txt
|
| TAP logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/patch-hadoop-common-project_hadoop-common.tap
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn
hadoop-mapreduce-project U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11944/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> fix jsvc/secure user var inconsistencies
>
>
> Key: HADOOP-14202
> URL: https://issues.apache.org/jira/browse/HADOOP-14202
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch
>
>
> Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major
> effort on making the configuration environment varia
[jira] [Commented] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944040#comment-15944040
]
Manoj Govindassamy commented on HADOOP-13715:
-
HDFS-11584 has been filed to track TestErasureCodeBenchmarkThroughput test
issue.
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-13715.01.patch, HADOOP-13715.02.patch,
> HADOOP-13715.03.patch, HADOOP-13715.04.patch, HADOOP-13715.05.patch,
> HADOOP-13715.06.patch
>
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944053#comment-15944053 ] Daryn Sharp commented on HADOOP-9969: - Please attach a current stack trace. Glancing at the code, it should be retrying... > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta, 2.5.0, 2.5.2, 2.6.0, 2.6.1, 2.8.0, 2.7.1, > 2.6.2, 2.6.3 > Environment: IBM JDK7 >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14236:
---
Attachment: HADOOP-14236-HADOOP-13345.001.patch
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14236-HADOOP-13345.000.patch,
> HADOOP-14236-HADOOP-13345.001.patch
>
>
> After running integration test {{ITestS3AFileSystemContract}}, I found the
> following items are not cleaned up in DynamoDB:
> {code}
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
> child=subdir
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
> child=file2
> {code}
> At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
> need to be careful when cleaning up test data.
> Then I found it’s a bug(?) in the code of integrating S3Guard with
> S3AFileSystem: for rename we miss sub-directory items to put (dest) and
> delete (src). The reason is that in S3A, we delete those fake directory
> objects if they are not necessary, e.g. non-empty. So when we list the
> objects to rename, the object summaries will only return _file_ objects. This
> has two consequences after rename:
> # there will be left items for src path in metadata store - left-overs will
> confuse {{get(Path)}} which should return null
> # we are not persisting the whole subtree for dest path to metadata store -
> this will break the DynamoDBMetadataStore invariant: _if a path exists, all
> its ancestors will also exist in the table_.
> Existing tests are not complaining about this though. If this is a real bug,
> let’s address it here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[ https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944074#comment-15944074 ] Allen Wittenauer commented on HADOOP-14202: --- With shellcheck -x enabled, that's a lot better. > fix jsvc/secure user var inconsistencies > > > Key: HADOOP-14202 > URL: https://issues.apache.org/jira/browse/HADOOP-14202 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch > > > Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major > effort on making the configuration environment variables consistent among all > the projects. The vast majority of vars now look like > (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and > HADOOP_PRIVILEGED_NFS_USER. > Additionally, there is > * no generic handling > * no documentation for anyone > * no safety checks to make sure things are defined > In order to fix all of this, we should: > * deprecate the previous vars using the deprecation function, updating the > HDFS documentation that references them > * add generic (command)_(subcommand)_SECURE_USER support > * add some verification for the previously mentioned var > * add some docs to UnixShellGuide.md -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[
https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944074#comment-15944074
]
Allen Wittenauer edited comment on HADOOP-14202 at 3/27/17 9:25 PM:
With shellcheck -x enabled, that's a lot better.
{code}
6 new + 100 unchanged - 0 fixed = 106 total (was 100)
{code}
to
{code}
3 new + 88 unchanged - 12 fixed = 91 total (was 100)
{code}
was (Author: aw):
With shellcheck -x enabled, that's a lot better.
> fix jsvc/secure user var inconsistencies
>
>
> Key: HADOOP-14202
> URL: https://issues.apache.org/jira/browse/HADOOP-14202
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch
>
>
> Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major
> effort on making the configuration environment variables consistent among all
> the projects. The vast majority of vars now look like
> (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and
> HADOOP_PRIVILEGED_NFS_USER.
> Additionally, there is
> * no generic handling
> * no documentation for anyone
> * no safety checks to make sure things are defined
> In order to fix all of this, we should:
> * deprecate the previous vars using the deprecation function, updating the
> HDFS documentation that references them
> * add generic (command)_(subcommand)_SECURE_USER support
> * add some verification for the previously mentioned var
> * add some docs to UnixShellGuide.md
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reassigned HADOOP-14237: -- Assignee: Kazuyuki Tanimura > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[ https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14202: -- Attachment: HADOOP-14202.02.patch -02: * fix shellcheck errors * remove uid test since it doesn't work consistently; leave a skip test telling why > fix jsvc/secure user var inconsistencies > > > Key: HADOOP-14202 > URL: https://issues.apache.org/jira/browse/HADOOP-14202 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch, > HADOOP-14202.02.patch > > > Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major > effort on making the configuration environment variables consistent among all > the projects. The vast majority of vars now look like > (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and > HADOOP_PRIVILEGED_NFS_USER. > Additionally, there is > * no generic handling > * no documentation for anyone > * no safety checks to make sure things are defined > In order to fix all of this, we should: > * deprecate the previous vars using the deprecation function, updating the > HDFS documentation that references them > * add generic (command)_(subcommand)_SECURE_USER support > * add some verification for the previously mentioned var > * add some docs to UnixShellGuide.md -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944099#comment-15944099
]
Hadoop QA commented on HADOOP-14236:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
47s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1
new + 5 unchanged - 2 fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14236 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860732/HADOOP-14236-HADOOP-13345.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6dbdf7f5c39e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / ed15aba |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11945/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11945/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11945/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop
[jira] [Commented] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944143#comment-15944143
]
Mingliang Liu commented on HADOOP-14236:
Tested against us-west-1.
{code}
$ mvn -Dit.test='ITestS3A*,ITestS3Guard*,ITestDynamo*' -Dtest=none -Dscale
-Ds3guard -Ddynamo -q clean verify
Results :
Tests run: 357, Failures: 0, Errors: 0, Skipped: 16
{code}
[~fabbri] I'll hold on commit in 3 days for your review.
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14236-HADOOP-13345.000.patch,
> HADOOP-14236-HADOOP-13345.001.patch
>
>
> After running integration test {{ITestS3AFileSystemContract}}, I found the
> following items are not cleaned up in DynamoDB:
> {code}
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
> child=subdir
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
> child=file2
> {code}
> At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
> need to be careful when cleaning up test data.
> Then I found it’s a bug(?) in the code of integrating S3Guard with
> S3AFileSystem: for rename we miss sub-directory items to put (dest) and
> delete (src). The reason is that in S3A, we delete those fake directory
> objects if they are not necessary, e.g. non-empty. So when we list the
> objects to rename, the object summaries will only return _file_ objects. This
> has two consequences after rename:
> # there will be left items for src path in metadata store - left-overs will
> confuse {{get(Path)}} which should return null
> # we are not persisting the whole subtree for dest path to metadata store -
> this will break the DynamoDBMetadataStore invariant: _if a path exists, all
> its ancestors will also exist in the table_.
> Existing tests are not complaining about this though. If this is a real bug,
> let’s address it here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[
https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944157#comment-15944157
]
Hadoop QA commented on HADOOP-14202:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
54s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 1m
17s{color} | {color:green} The patch generated 0 new + 81 unchanged - 19 fixed
= 81 total (was 100) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
39s{color} | {color:green} hadoop-yarn in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-mapreduce-project in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14202 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860740/HADOOP-14202.02.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 4c757f3e83f2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cd014d5 |
| shellcheck | v0.4.5 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11946/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn
hadoop-mapreduce-project U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11946/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> fix jsvc/secure user var inconsistencies
>
>
> Key: HADOOP-14202
> URL: https://issues.apache.org/jira/browse/HADOOP-14202
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch,
> HADOOP-14202.02.patch
>
>
> Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major
> effort on making the configuration environment variables consistent among all
> the projects. The vast majority of vars now look like
> (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and
> HADOOP_PRIVILEGED_NFS_USER.
> Additionally, there is
> * no generic handling
> * no documentation for anyone
> * no safety checks to make sure things are defined
> In order to fix all of this, we should:
> * deprecate the previous vars using the deprecation function, updating the
> HDFS documen
[jira] [Updated] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
[
https://issues.apache.org/jira/browse/HADOOP-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14227:
---
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-13345
> S3Guard: ITestS3AConcurrentOps is not cleaning up test data
> ---
>
> Key: HADOOP-14227
> URL: https://issues.apache.org/jira/browse/HADOOP-14227
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-14227-HADOOP-13345.000.patch
>
>
> After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in
> DynamoDB. There are two reasons:
> # The {{ITestS3AConcurrentOps::teardown()}} method is not calling super
> teardown() method to clean up the default test directory.
> # The {{auxFs}} is not S3Guard aware even though the {{fs}} to test is.
> That's because the {{auxFs}} is creating a new Configuration object without
> patching in S3Guard options (via {{maybeEnableS3Guard(conf);}}).
> This JIRA is to clean up the data after test.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-10101: Release Note: Guava is updated to version 21.0. In the background of merging this patch into trunk, there is a work, shaded Hadoop client artifacts and minicluster, on HADOOP-11804. hadoop-client has its own Guava which is shaded, so we can update dependency with minimum effect compare to previous HADOOP-11804. See also HADOOP-14238 as related problem. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Fix For: 3.0.0-alpha3 > > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch, > HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.017.patch, > HADOOP-10101.018.patch, HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This issue tries to > update the version to as latest version as possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14223) Extend FileStatus#toString() to include details like Erasure Coding and Encryption
[
https://issues.apache.org/jira/browse/HADOOP-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944182#comment-15944182
]
Hadoop QA commented on HADOOP-14223:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 2 new + 414 unchanged
- 0 fixed = 416 total (was 414) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m
48s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14223 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860705/HADOOP-14223.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e248236094cf 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db2adf3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11942/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11942/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure-datalake U: .
[jira] [Commented] (HADOOP-14104) Client should always ask namenode for kms provider path.
[ https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944191#comment-15944191 ] Yongjun Zhang commented on HADOOP-14104: Thanks [~daryn] a lot for the review and comments. Hi [~rushabh.shah], Wonder if you have chance to look at Daryn's comments? Many thanks! > Client should always ask namenode for kms provider path. > > > Key: HADOOP-14104 > URL: https://issues.apache.org/jira/browse/HADOOP-14104 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HADOOP-14104-trunk.patch, HADOOP-14104-trunk-v1.patch, > HADOOP-14104-trunk-v2.patch > > > According to current implementation of kms provider in client conf, there can > only be one kms. > In multi-cluster environment, if a client is reading encrypted data from > multiple clusters it will only get kms token for local cluster. > Not sure whether the target version is correct or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14104) Client should always ask namenode for kms provider path.
[ https://issues.apache.org/jira/browse/HADOOP-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944191#comment-15944191 ] Yongjun Zhang edited comment on HADOOP-14104 at 3/27/17 11:00 PM: -- Thanks [~daryn] a lot for the review and comments. Hi [~shahrs87], Wonder if you have chance to look at Daryn's comments? Many thanks! was (Author: yzhangal): Thanks [~daryn] a lot for the review and comments. Hi [~rushabh.shah], Wonder if you have chance to look at Daryn's comments? Many thanks! > Client should always ask namenode for kms provider path. > > > Key: HADOOP-14104 > URL: https://issues.apache.org/jira/browse/HADOOP-14104 > Project: Hadoop Common > Issue Type: Improvement > Components: kms >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HADOOP-14104-trunk.patch, HADOOP-14104-trunk-v1.patch, > HADOOP-14104-trunk-v2.patch > > > According to current implementation of kms provider in client conf, there can > only be one kms. > In multi-cluster environment, if a client is reading encrypted data from > multiple clusters it will only get kms token for local cluster. > Not sure whether the target version is correct or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14236:
---
Attachment: HADOOP-14236-HADOOP-13345.002.patch
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14236-HADOOP-13345.000.patch,
> HADOOP-14236-HADOOP-13345.001.patch, HADOOP-14236-HADOOP-13345.002.patch
>
>
> After running integration test {{ITestS3AFileSystemContract}}, I found the
> following items are not cleaned up in DynamoDB:
> {code}
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
> child=subdir
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
> child=file2
> {code}
> At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
> need to be careful when cleaning up test data.
> Then I found it’s a bug(?) in the code of integrating S3Guard with
> S3AFileSystem: for rename we miss sub-directory items to put (dest) and
> delete (src). The reason is that in S3A, we delete those fake directory
> objects if they are not necessary, e.g. non-empty. So when we list the
> objects to rename, the object summaries will only return _file_ objects. This
> has two consequences after rename:
> # there will be left items for src path in metadata store - left-overs will
> confuse {{get(Path)}} which should return null
> # we are not persisting the whole subtree for dest path to metadata store -
> this will break the DynamoDBMetadataStore invariant: _if a path exists, all
> its ancestors will also exist in the table_.
> Existing tests are not complaining about this though. If this is a real bug,
> let’s address it here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[
https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944207#comment-15944207
]
Sean Busbey commented on HADOOP-14202:
--
{code}
2510 # are we expected to be running with privilege?
2511 # if yes, then we need to define all of the priv and daemon stuff
2512 # if not, then we just need to define daemon stuff.
2513 # note that these settings are different between the two
{code}
This phrasing is a bit awkward. What is "these settings" referring to?
> fix jsvc/secure user var inconsistencies
>
>
> Key: HADOOP-14202
> URL: https://issues.apache.org/jira/browse/HADOOP-14202
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch,
> HADOOP-14202.02.patch
>
>
> Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major
> effort on making the configuration environment variables consistent among all
> the projects. The vast majority of vars now look like
> (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and
> HADOOP_PRIVILEGED_NFS_USER.
> Additionally, there is
> * no generic handling
> * no documentation for anyone
> * no safety checks to make sure things are defined
> In order to fix all of this, we should:
> * deprecate the previous vars using the deprecation function, updating the
> HDFS documentation that references them
> * add generic (command)_(subcommand)_SECURE_USER support
> * add some verification for the previously mentioned var
> * add some docs to UnixShellGuide.md
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer
[ https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944209#comment-15944209 ] Allen Wittenauer commented on HADOOP-13363: --- I see two big gains going to at least a newer version: * protoc 2.5.0 doesn't work out of the box on a variety of chipsets and platforms (It was an issue raised on the mailing list a while back, so it is impacting users.) * Ease of contribution: protobuf 2.5.0 is no longer the default protoc that is getting installed on new OS deployments. The first point, IMO, is fairly important. The OOB compilation experience for non-x86, non-Linux is pretty awful and this would be a huge improvement. > Upgrade protobuf from 2.5.0 to something newer > -- > > Key: HADOOP-13363 > URL: https://issues.apache.org/jira/browse/HADOOP-13363 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer > Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, > HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch > > > Standard protobuf 2.5.0 does not work properly on many platforms. (See, for > example, https://gist.github.com/BennettSmith/7111094 ). In order for us to > avoid crazy work arounds in the build environment and the fact that 2.5.0 is > starting to slowly disappear as a standard install-able package for even > Linux/x86, we need to either upgrade or self bundle or something else. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[
https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944217#comment-15944217
]
Allen Wittenauer commented on HADOOP-14202:
---
The intent is to warn people and to make sure they understand what is going on.
I have clearly failed. haha. The big issue is that the if clause needs to be
there for the daemon\_* env vars ("these settings") are getting configured
slightly differently. I guess I should just state that though.
> fix jsvc/secure user var inconsistencies
>
>
> Key: HADOOP-14202
> URL: https://issues.apache.org/jira/browse/HADOOP-14202
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14202.00.patch, HADOOP-14202.01.patch,
> HADOOP-14202.02.patch
>
>
> Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major
> effort on making the configuration environment variables consistent among all
> the projects. The vast majority of vars now look like
> (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and
> HADOOP_PRIVILEGED_NFS_USER.
> Additionally, there is
> * no generic handling
> * no documentation for anyone
> * no safety checks to make sure things are defined
> In order to fix all of this, we should:
> * deprecate the previous vars using the deprecation function, updating the
> HDFS documentation that references them
> * add generic (command)_(subcommand)_SECURE_USER support
> * add some verification for the previously mentioned var
> * add some docs to UnixShellGuide.md
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944230#comment-15944230
]
Hadoop QA commented on HADOOP-14236:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
37s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
37s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1
new + 5 unchanged - 2 fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14236 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860759/HADOOP-14236-HADOOP-13345.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bd7ce193 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / ed15aba |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11947/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11947/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11947/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop
[jira] [Commented] (HADOOP-11794) distcp can copy blocks in parallel
[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944231#comment-15944231
]
Chris Douglas commented on HADOOP-11794:
+1 overall, though the DistCp docs currently claim:
{noformat}
Both the source and the target FileSystem must be DistributedFileSystem
{noformat}
With the relaxed check, this could read "The target FileSystem must support the
{{FileSystem#concat}} operation".
[~omkarksa], you may want to verify the latest patch works for ADLS.
> distcp can copy blocks in parallel
> --
>
> Key: HADOOP-11794
> URL: https://issues.apache.org/jira/browse/HADOOP-11794
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 0.21.0
>Reporter: dhruba borthakur
>Assignee: Yongjun Zhang
> Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch,
> HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch,
> HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch,
> HADOOP-11794.009.patch, HADOOP-11794.010.patch, MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
Mingliang Liu created HADOOP-14247: -- Summary: FileContextMainOperationsBaseTest should clean up test root path Key: HADOOP-14247 URL: https://issues.apache.org/jira/browse/HADOOP-14247 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[ https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-14247: --- Priority: Minor (was: Major) > FileContextMainOperationsBaseTest should clean up test root path > > > Key: HADOOP-14247 > URL: https://issues.apache.org/jira/browse/HADOOP-14247 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[ https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-14247: --- Target Version/s: 2.8.1, 3.0.0-alpha3 (was: 2.8.1) > FileContextMainOperationsBaseTest should clean up test root path > > > Key: HADOOP-14247 > URL: https://issues.apache.org/jira/browse/HADOOP-14247 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0 >Reporter: Mingliang Liu >Assignee: Mingliang Liu > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[ https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-14247: --- Attachment: HADOOP-14247.000.patch > FileContextMainOperationsBaseTest should clean up test root path > > > Key: HADOOP-14247 > URL: https://issues.apache.org/jira/browse/HADOOP-14247 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Minor > Attachments: HADOOP-14247.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[ https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-14247: --- Status: Patch Available (was: Open) > FileContextMainOperationsBaseTest should clean up test root path > > > Key: HADOOP-14247 > URL: https://issues.apache.org/jira/browse/HADOOP-14247 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Minor > Attachments: HADOOP-14247.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[
https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944245#comment-15944245
]
Mingliang Liu commented on HADOOP-14247:
[~ste...@apache.org] can you have a quick look?
> FileContextMainOperationsBaseTest should clean up test root path
>
>
> Key: HADOOP-14247
> URL: https://issues.apache.org/jira/browse/HADOOP-14247
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-14247.000.patch
>
>
> {{fileContextTestHelper}} will create random test root directory for each
> test case, say $RANDOM. After test, we only delete the $RANDOM/test directory
> in tearDown(). This will leave dangling directories under the {{test-dir}}
> directory. For S3A with S3Guard enabled, those useless test directories will
> stay in DynamoDB forever. Let's clear out the test root path, which is
> (generated randomly and) safe to remove.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[
https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14247:
---
Component/s: test
> FileContextMainOperationsBaseTest should clean up test root path
>
>
> Key: HADOOP-14247
> URL: https://issues.apache.org/jira/browse/HADOOP-14247
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-14247.000.patch
>
>
> {{fileContextTestHelper}} will create random test root directory for each
> test case, say $RANDOM. After test, we only delete the $RANDOM/test directory
> in tearDown(). This will leave dangling directories under the {{test-dir}}
> directory. For S3A with S3Guard enabled, those useless test directories will
> stay in DynamoDB forever. Let's clear out the test root path, which is
> (generated randomly and) safe to remove.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
[
https://issues.apache.org/jira/browse/HADOOP-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14247:
---
Description: {{fileContextTestHelper}} will create random test root
directory for each test case, say $RANDOM. After test, we only delete the
$RANDOM/test directory in tearDown(). This will leave dangling directories
under the {{test-dir}} directory. For S3A with S3Guard enabled, those useless
test directories will stay in DynamoDB forever. Let's clear out the test root
path, which is (generated randomly and) safe to remove.
> FileContextMainOperationsBaseTest should clean up test root path
>
>
> Key: HADOOP-14247
> URL: https://issues.apache.org/jira/browse/HADOOP-14247
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, test
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HADOOP-14247.000.patch
>
>
> {{fileContextTestHelper}} will create random test root directory for each
> test case, say $RANDOM. After test, we only delete the $RANDOM/test directory
> in tearDown(). This will leave dangling directories under the {{test-dir}}
> directory. For S3A with S3Guard enabled, those useless test directories will
> stay in DynamoDB forever. Let's clear out the test root path, which is
> (generated randomly and) safe to remove.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[
https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944264#comment-15944264
]
Andrew Wang commented on HADOOP-10101:
--
It would be good to proactively reach out to downstreams who aren't using the
shaded clients about this change. I did looked at CDH, and found these that
seem to consume hadoop-hdfs directly:
{noformat}
-> % ag -l -G pom.xml "hadoop-hdfs" | cut -d/ -f 1 |
sort -u
crunch
flume-ng
hadoop
hbase
hbase-indexer
hive
impala
kite
oozie
pig
sentry
sqoop
{noformat}
> Update guava dependency to the latest version
> -
>
> Key: HADOOP-10101
> URL: https://issues.apache.org/jira/browse/HADOOP-10101
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Rakesh R
>Assignee: Tsuyoshi Ozawa
> Labels: BB2015-05-TBR
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch,
> HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch,
> HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch,
> HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch,
> HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch,
> HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.017.patch,
> HADOOP-10101.018.patch, HADOOP-10101.patch, HADOOP-10101.patch
>
>
> The existing guava version is 11.0.2 which is quite old. This issue tries to
> update the version to as latest version as possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-10101) Update guava dependency to the latest version
[
https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944264#comment-15944264
]
Andrew Wang edited comment on HADOOP-10101 at 3/27/17 11:59 PM:
It would be good to proactively reach out to downstreams who aren't using the
shaded clients about this change. I looked at CDH, and found these that seem to
consume hadoop-hdfs directly:
{noformat}
-> % ag -l -G pom.xml "hadoop-hdfs" | cut -d/ -f 1 |
sort -u
crunch
flume-ng
hadoop
hbase
hbase-indexer
hive
impala
kite
oozie
pig
sentry
sqoop
{noformat}
was (Author: andrew.wang):
It would be good to proactively reach out to downstreams who aren't using the
shaded clients about this change. I did looked at CDH, and found these that
seem to consume hadoop-hdfs directly:
{noformat}
-> % ag -l -G pom.xml "hadoop-hdfs" | cut -d/ -f 1 |
sort -u
crunch
flume-ng
hadoop
hbase
hbase-indexer
hive
impala
kite
oozie
pig
sentry
sqoop
{noformat}
> Update guava dependency to the latest version
> -
>
> Key: HADOOP-10101
> URL: https://issues.apache.org/jira/browse/HADOOP-10101
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Rakesh R
>Assignee: Tsuyoshi Ozawa
> Labels: BB2015-05-TBR
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch,
> HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch,
> HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch,
> HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch,
> HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.014.patch,
> HADOOP-10101.015.patch, HADOOP-10101.016.patch, HADOOP-10101.017.patch,
> HADOOP-10101.018.patch, HADOOP-10101.patch, HADOOP-10101.patch
>
>
> The existing guava version is 11.0.2 which is quite old. This issue tries to
> update the version to as latest version as possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
Mingliang Liu created HADOOP-14248:
--
Summary: Retire SharedInstanceProfileCredentialsProvider after AWS
SDK upgrade
Key: HADOOP-14248
URL: https://issues.apache.org/jira/browse/HADOOP-14248
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.0.0-alpha3
Reporter: Mingliang Liu
Assignee: Mingliang Liu
This is from the discussion in [HADOOP-13050].
So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which
effectively reduces high number of connections to EC2 Instance Metadata Service
caused by InstanceProfileCredentialsProvider. That patch, in order to prevent
the throttling problem, defined new class
{{SharedInstanceProfileCredentialsProvider}} as a subclass of
{{InstanceProfileCredentialsProvider}}, which enforces creation of only a
single instance.
Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
{{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense.
Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully
in trunk branch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
[
https://issues.apache.org/jira/browse/HADOOP-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14248:
---
Attachment: HADOOP-14248.000.patch
> Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
> -
>
> Key: HADOOP-14248
> URL: https://issues.apache.org/jira/browse/HADOOP-14248
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14248.000.patch
>
>
> This is from the discussion in [HADOOP-13050].
> So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which
> effectively reduces high number of connections to EC2 Instance Metadata
> Service caused by InstanceProfileCredentialsProvider. That patch, in order to
> prevent the throttling problem, defined new class
> {{SharedInstanceProfileCredentialsProvider}} as a subclass of
> {{InstanceProfileCredentialsProvider}}, which enforces creation of only a
> single instance.
> Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
> {{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
> singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense.
> Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully
> in trunk branch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
[
https://issues.apache.org/jira/browse/HADOOP-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14248:
---
Status: Patch Available (was: Open)
> Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
> -
>
> Key: HADOOP-14248
> URL: https://issues.apache.org/jira/browse/HADOOP-14248
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14248.000.patch
>
>
> This is from the discussion in [HADOOP-13050].
> So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which
> effectively reduces high number of connections to EC2 Instance Metadata
> Service caused by InstanceProfileCredentialsProvider. That patch, in order to
> prevent the throttling problem, defined new class
> {{SharedInstanceProfileCredentialsProvider}} as a subclass of
> {{InstanceProfileCredentialsProvider}}, which enforces creation of only a
> single instance.
> Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
> {{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
> singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense.
> Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully
> in trunk branch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
[
https://issues.apache.org/jira/browse/HADOOP-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944306#comment-15944306
]
Mingliang Liu commented on HADOOP-14248:
[~ste...@apache.org] and [~cnauroth], can you have a look at this if you find
time? Thanks,
> Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
> -
>
> Key: HADOOP-14248
> URL: https://issues.apache.org/jira/browse/HADOOP-14248
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14248.000.patch
>
>
> This is from the discussion in [HADOOP-13050].
> So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which
> effectively reduces high number of connections to EC2 Instance Metadata
> Service caused by InstanceProfileCredentialsProvider. That patch, in order to
> prevent the throttling problem, defined new class
> {{SharedInstanceProfileCredentialsProvider}} as a subclass of
> {{InstanceProfileCredentialsProvider}}, which enforces creation of only a
> single instance.
> Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
> {{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
> singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense.
> Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully
> in trunk branch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14236:
---
Description:
After running integration test {{ITestS3AFileSystemContract}}, I found the
following items are not cleaned up in DynamoDB:
{code}
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
child=subdir
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
child=file2
{code}
At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
need to be careful when cleaning up test data.
Then I found it’s a bug(?) in the code of integrating S3Guard with
S3AFileSystem: for rename we miss sub-directory items to put (dest) and delete
(src). The reason is that in S3A, we delete those fake directory objects if
they are not necessary, e.g. non-empty. So when we list the objects to rename,
the object summaries will only return _file_ objects. This has two consequences
after rename:
# there will be left items for src path in metadata store - left-overs will
confuse {{get(Path)}} which should return null
# we are not persisting the whole subtree for dest path to metadata store -
this will break the DynamoDBMetadataStore invariant: _if a path exists, all its
ancestors will also exist in the table_.
UPDATE: modified test case {{ITestS3AFileSystemContract::
testRenameDirectoryAsExistingDirectory()}} will fail w/o this patch; if pass w/
this patch. If the test case makes sense, the proposal follows.
Existing tests are not complaining about this though. If this is a real bug,
let’s address it here.
was:
After running integration test {{ITestS3AFileSystemContract}}, I found the
following items are not cleaned up in DynamoDB:
{code}
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
child=subdir
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
child=file2
{code}
At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
need to be careful when cleaning up test data.
Then I found it’s a bug(?) in the code of integrating S3Guard with
S3AFileSystem: for rename we miss sub-directory items to put (dest) and delete
(src). The reason is that in S3A, we delete those fake directory objects if
they are not necessary, e.g. non-empty. So when we list the objects to rename,
the object summaries will only return _file_ objects. This has two consequences
after rename:
# there will be left items for src path in metadata store - left-overs will
confuse {{get(Path)}} which should return null
# we are not persisting the whole subtree for dest path to metadata store -
this will break the DynamoDBMetadataStore invariant: _if a path exists, all its
ancestors will also exist in the table_.
Existing tests are not complaining about this though. If this is a real bug,
let’s address it here.
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14236-HADOOP-13345.000.patch,
> HADOOP-14236-HADOOP-13345.001.patch, HADOOP-14236-HADOOP-13345.002.patch
>
>
> After running integration test {{ITestS3AFileSystemContract}}, I found the
> following items are not cleaned up in DynamoDB:
> {code}
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
> child=subdir
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
> child=file2
> {code}
> At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
> need to be careful when cleaning up test data.
> Then I found it’s a bug(?) in the code of integrating S3Guard with
> S3AFileSystem: for rename we miss sub-directory items to put (dest) and
> delete (src). The reason is that in S3A, we delete those fake directory
> objects if they are not necessary, e.g. non-empty. So when we list the
> objects to rename, the object summaries will only return _file_ objects. This
> has two consequences after rename:
> # there will be left items for src path in metadata store - left-overs will
> confuse {{get(Path)}} which should return null
> # we are not persisting the whole subtree for dest path to metadata store -
> this will break the DynamoDBMetadataStore invariant: _if a path exists, all
> its ancestors will also exist in the table_.
> UPDATE: modified test case {{ITestS3AFileSystemContrac
[jira] [Updated] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
[
https://issues.apache.org/jira/browse/HADOOP-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14236:
---
Description:
After running integration test {{ITestS3AFileSystemContract}}, I found the
following items are not cleaned up in DynamoDB:
{code}
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
child=subdir
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
child=file2
{code}
At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
need to be careful when cleaning up test data.
Then I found it’s a bug(?) in the code of integrating S3Guard with
S3AFileSystem: for rename we miss sub-directory items to put (dest) and delete
(src). The reason is that in S3A, we delete those fake directory objects if
they are not necessary, e.g. non-empty. So when we list the objects to rename,
the object summaries will only return _file_ objects. This has two consequences
after rename:
# there will be left items for src path in metadata store - left-overs will
confuse {{get(Path)}} which should return null
# we are not persisting the whole subtree for dest path to metadata store -
this will break the DynamoDBMetadataStore invariant: _if a path exists, all its
ancestors will also exist in the table_.
UPDATE: modified test case {{ITestS3AFileSystemContract::
testRenameDirectoryAsExistingDirectory()}} will fail w/o this patch; it passes
w/ this patch. If the test case makes sense, the proposal follows.
Existing tests are not complaining about this though. If this is a real bug,
let’s address it here.
was:
After running integration test {{ITestS3AFileSystemContract}}, I found the
following items are not cleaned up in DynamoDB:
{code}
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
child=subdir
parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
child=file2
{code}
At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
need to be careful when cleaning up test data.
Then I found it’s a bug(?) in the code of integrating S3Guard with
S3AFileSystem: for rename we miss sub-directory items to put (dest) and delete
(src). The reason is that in S3A, we delete those fake directory objects if
they are not necessary, e.g. non-empty. So when we list the objects to rename,
the object summaries will only return _file_ objects. This has two consequences
after rename:
# there will be left items for src path in metadata store - left-overs will
confuse {{get(Path)}} which should return null
# we are not persisting the whole subtree for dest path to metadata store -
this will break the DynamoDBMetadataStore invariant: _if a path exists, all its
ancestors will also exist in the table_.
UPDATE: modified test case {{ITestS3AFileSystemContract::
testRenameDirectoryAsExistingDirectory()}} will fail w/o this patch; if pass w/
this patch. If the test case makes sense, the proposal follows.
Existing tests are not complaining about this though. If this is a real bug,
let’s address it here.
> S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
> in metadata store
> ---
>
> Key: HADOOP-14236
> URL: https://issues.apache.org/jira/browse/HADOOP-14236
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-14236-HADOOP-13345.000.patch,
> HADOOP-14236-HADOOP-13345.001.patch, HADOOP-14236-HADOOP-13345.002.patch
>
>
> After running integration test {{ITestS3AFileSystemContract}}, I found the
> following items are not cleaned up in DynamoDB:
> {code}
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir,
> child=subdir
> parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir,
> child=file2
> {code}
> At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we
> need to be careful when cleaning up test data.
> Then I found it’s a bug(?) in the code of integrating S3Guard with
> S3AFileSystem: for rename we miss sub-directory items to put (dest) and
> delete (src). The reason is that in S3A, we delete those fake directory
> objects if they are not necessary, e.g. non-empty. So when we list the
> objects to rename, the object summaries will only return _file_ objects. This
> has two consequences after rename:
> # there will be left items for src path in metadata store - left-overs will
> confuse {{get(Path)}} which should return null
> # we are not persisting the whole subtree for dest pa
[jira] [Updated] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
[
https://issues.apache.org/jira/browse/HADOOP-14248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-14248:
---
Hadoop Flags: Incompatible change
Release Note: SharedInstanceProfileCredentialsProvider is removed after
this change. Users should use InstanceProfileCredentialsProvider provided by
AWS SDK instead, which itself enforces a singleton instance to reduce calls to
AWS EC2 Instance Metadata Service.
> Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
> -
>
> Key: HADOOP-14248
> URL: https://issues.apache.org/jira/browse/HADOOP-14248
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14248.000.patch
>
>
> This is from the discussion in [HADOOP-13050].
> So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which
> effectively reduces high number of connections to EC2 Instance Metadata
> Service caused by InstanceProfileCredentialsProvider. That patch, in order to
> prevent the throttling problem, defined new class
> {{SharedInstanceProfileCredentialsProvider}} as a subclass of
> {{InstanceProfileCredentialsProvider}}, which enforces creation of only a
> single instance.
> Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the
> {{InstanceProfileCredentialsProvider}} in SDK code internally enforces a
> singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense.
> Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully
> in trunk branch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-14237: --- Target Version/s: 2.8.1, 3.0.0-alpha3 Status: Patch Available (was: Open) > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.8.0, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
