[jira] [Commented] (HADOOP-16445) Allow separate custom signing algorithms for S3 and DDB
[ https://issues.apache.org/jira/browse/HADOOP-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891640#comment-16891640 ] Siddharth Seth commented on HADOOP-16445: - bq. github PRs are how we're reviewing patches no. Will move this over to a PR bq. No tests, no review, you know the rules. The patch does have what I think are unit tests. Do you have some specific tests in mind (integration tests?) and pointers on where to add them. A possible test would be to set up a custom signer via the new configs and use it. bq. Which endpoint did you test against, and, for something going anywhere near auth, I'm expecting the SSE-KMS and IAM roles to be tested to, including the role delegation tokens. thanks Not sure why SSE-KMS, IAM roles and delegation tokens need to be tested on this patch (HADOOP-16449 is the one which makes changes to the way authN tokens can be specified). The patch allows for custom signers, and does not change authentication mechanics. > Allow separate custom signing algorithms for S3 and DDB > --- > > Key: HADOOP-16445 > URL: https://issues.apache.org/jira/browse/HADOOP-16445 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Attachments: HADOOP-16445.01.patch, HADOOP-16445.02.patch > > > fs.s3a.signing-algorithm allows overriding the signer. This applies to both > the S3 and DDB clients. Need to be able to specify separate signing algorithm > overrides for S3 and DDB. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16451) Update jackson-databind to 2.9.9.1
[ https://issues.apache.org/jira/browse/HADOOP-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891663#comment-16891663 ] Akira Ajisaka commented on HADOOP-16451: The 002 patch LGTM, +1 pending Jenkins. Thank you, [~smeng]. > Update jackson-databind to 2.9.9.1 > -- > > Key: HADOOP-16451 > URL: https://issues.apache.org/jira/browse/HADOOP-16451 > Project: Hadoop Common > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16451.001.patch, HADOOP-16451.002.patch > > > https://nvd.nist.gov/vuln/detail/CVE-2019-12814 > CVE-2019-12814 flags 2.9.9 as vulnerable. A new version 2.9.9.1 is available. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16454) Share delegation tokens between multiple HttpFS servers
[ https://issues.apache.org/jira/browse/HADOOP-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891666#comment-16891666 ] Akira Ajisaka commented on HADOOP-16454: bq. maybe we can fix this situation by using ZKDelegationTokenSecretManager in HttpFS by setting "httpfs.authentication.zk-dt-secret-manager.enable" to true. We fixed this situation by setting the following parameters: * httpfs.authentication.zk-dt-secret-manager.enable * httpfs.authentication.zk-dt-secret-manager.zkConnectionString If using sasl for ZK authentication, the following parameters are required to be set. * httpfs.authentication.zk-dt-secret-manager.zkAuthType * httpfs.authentication.zk-dt-secret-manager.kerberos.principal" * httpfs.authentication.zk-dt-secret-manager.kerberos.keytab" If there is a service already using ZKDelegationTokenSecretManager (e.g. DFSRouter) and the znodeWorkingPath is default, the following parameter must be changed: * httpfs.authentication.zk-dt-secret-manager.znodeWorkingPath I would like to document how to configure ZKDelegationTokenSecretManager in HttpFS in this issue. Updating the description. > Share delegation tokens between multiple HttpFS servers > --- > > Key: HADOOP-16454 > URL: https://issues.apache.org/jira/browse/HADOOP-16454 > Project: Hadoop Common > Issue Type: New Feature > Components: httpfs > Environment: Kerberized, clients connect to multiple HttpFS servers > via load balancer >Reporter: Akira Ajisaka >Priority: Major > > In our environment, multiple HttpFS servers are deployed for the clients > outside the HDFS cluster. As we are using external load balancer service for > the HttpFS servers, the following situation may happen: > 1. A client authenticates with a HttpFS server and gets a delegation token. > Using the delegation token, the client can access to the NameNode. > 2. In the next session, the client authenticates with another HttpFS server > (via load balancer) using the same delegation token. The client fails to > access because the other HttpFS server does not have the information of the > delegation token. > It would be nice to have a feature like HADOOP-14445 in HttpFS to fix the > situation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16454) Document how to share delegation tokens between multiple HttpFS servers
[ https://issues.apache.org/jira/browse/HADOOP-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reassigned HADOOP-16454: -- Assignee: Akira Ajisaka Priority: Minor (was: Major) Summary: Document how to share delegation tokens between multiple HttpFS servers (was: Share delegation tokens between multiple HttpFS servers) Description: In our environment, multiple HttpFS servers are deployed for the clients outside the HDFS cluster. As we are using external load balancer service for the HttpFS servers, the following situation may happen: 1. A client authenticates with a HttpFS server and gets a delegation token. Using the delegation token, the client can access to the NameNode. 2. In the next session, the client authenticates with another HttpFS server (via load balancer) using the same delegation token. The client fails to access because the other HttpFS server does not have the information of the delegation token. This issue is to document how to fix this situation. was: In our environment, multiple HttpFS servers are deployed for the clients outside the HDFS cluster. As we are using external load balancer service for the HttpFS servers, the following situation may happen: 1. A client authenticates with a HttpFS server and gets a delegation token. Using the delegation token, the client can access to the NameNode. 2. In the next session, the client authenticates with another HttpFS server (via load balancer) using the same delegation token. The client fails to access because the other HttpFS server does not have the information of the delegation token. It would be nice to have a feature like HADOOP-14445 in HttpFS to fix the situation. Component/s: documentation Issue Type: Improvement (was: New Feature) > Document how to share delegation tokens between multiple HttpFS servers > --- > > Key: HADOOP-16454 > URL: https://issues.apache.org/jira/browse/HADOOP-16454 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, httpfs > Environment: Kerberized, clients connect to multiple HttpFS servers > via load balancer >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Minor > > In our environment, multiple HttpFS servers are deployed for the clients > outside the HDFS cluster. As we are using external load balancer service for > the HttpFS servers, the following situation may happen: > 1. A client authenticates with a HttpFS server and gets a delegation token. > Using the delegation token, the client can access to the NameNode. > 2. In the next session, the client authenticates with another HttpFS server > (via load balancer) using the same delegation token. The client fails to > access because the other HttpFS server does not have the information of the > delegation token. > This issue is to document how to fix this situation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16403) Start a new statistical rpc queue and make the Reader's pendingConnection queue runtime-replaceable
[
https://issues.apache.org/jira/browse/HADOOP-16403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891672#comment-16891672
]
Hadoop QA commented on HADOOP-16403:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
52s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
44s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
19s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 19s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 19s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 43s{color} | {color:orange} root: The patch generated 8 new + 547 unchanged
- 7 fixed = 555 total (was 554) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
53s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
47s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
53s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 58s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.0 Server=19.03.0 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HADOOP-16403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975581/HADOOP-16403.003.patch
|
| Optional Tes
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#discussion_r306683097 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PathMetadata.java ## @@ -71,6 +78,7 @@ public PathMetadata(S3AFileStatus fileStatus, Tristate isEmptyDir, boolean this.fileStatus = fileStatus; this.isEmptyDirectory = isEmptyDir; this.isDeleted = isDeleted; +this.setLastUpdated(lastUpdated); Review comment: I'd prefer `Preconditions.checkArgument(lastUpdated >=0 ` instead. It's better fail fast if there's a negative value, we don't need to be that failproof here imo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard:
Filter expired entries and tombstones when lis…
URL: https://github.com/apache/hadoop/pull/1134#discussion_r306683463
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
##
@@ -538,6 +540,47 @@ public void deleteAfterTombstoneExpiryOobCreate() throws
Exception {
}
}
+ /**
+ * Test that a tombstone won't hide an entry after it's expired in the
+ * listing.
+ */
+ @Test
+ public void testRootTombstones() throws Exception {
+long ttl = 10L;
+ITtlTimeProvider mockTimeProvider = mock(ITtlTimeProvider.class);
+when(mockTimeProvider.getMetadataTtl()).thenReturn(ttl);
+when(mockTimeProvider.getNow()).thenReturn(100L);
+ITtlTimeProvider originalTimeProvider = guardedFs.getTtlTimeProvider();
+guardedFs.setTtlTimeProvider(mockTimeProvider);
+
+Path base = path(getMethodName() + UUID.randomUUID());
+Path testFile = new Path(base, "test.file");
+
+try {
+ touch(guardedFs, testFile);
+ ContractTestUtils.assertDeleted(guardedFs, testFile, false);
+
+ touch(rawFS, testFile);
+ awaitFileStatus(rawFS, testFile);
+
+ // the rawFS will include the file=
+ LambdaTestUtils.eventually(5000, 1000, () -> {
+checkListingContainsPath(rawFS, testFile);
+ });
+
+ // it will be hidden because of the tombstone
+ checkListingDoesNotContainPath(guardedFs, testFile);
Review comment:
I'd prefer `Preconditions.checkArgument(lastUpdated >=0`. Better fail fast
if something's wrong. We don't need to be that failproof here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard:
Filter expired entries and tombstones when lis…
URL: https://github.com/apache/hadoop/pull/1134#discussion_r306683463
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
##
@@ -538,6 +540,47 @@ public void deleteAfterTombstoneExpiryOobCreate() throws
Exception {
}
}
+ /**
+ * Test that a tombstone won't hide an entry after it's expired in the
+ * listing.
+ */
+ @Test
+ public void testRootTombstones() throws Exception {
+long ttl = 10L;
+ITtlTimeProvider mockTimeProvider = mock(ITtlTimeProvider.class);
+when(mockTimeProvider.getMetadataTtl()).thenReturn(ttl);
+when(mockTimeProvider.getNow()).thenReturn(100L);
+ITtlTimeProvider originalTimeProvider = guardedFs.getTtlTimeProvider();
+guardedFs.setTtlTimeProvider(mockTimeProvider);
+
+Path base = path(getMethodName() + UUID.randomUUID());
+Path testFile = new Path(base, "test.file");
+
+try {
+ touch(guardedFs, testFile);
+ ContractTestUtils.assertDeleted(guardedFs, testFile, false);
+
+ touch(rawFS, testFile);
+ awaitFileStatus(rawFS, testFile);
+
+ // the rawFS will include the file=
+ LambdaTestUtils.eventually(5000, 1000, () -> {
+checkListingContainsPath(rawFS, testFile);
+ });
+
+ // it will be hidden because of the tombstone
+ checkListingDoesNotContainPath(guardedFs, testFile);
Review comment:
I'd prefer `Preconditions.checkArgument(lastUpdated >=0`. Better fail fast
if something's wrong. We don't need to be that failproof here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard:
Filter expired entries and tombstones when lis…
URL: https://github.com/apache/hadoop/pull/1134#discussion_r306700684
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
##
@@ -538,6 +540,47 @@ public void deleteAfterTombstoneExpiryOobCreate() throws
Exception {
}
}
+ /**
+ * Test that a tombstone won't hide an entry after it's expired in the
+ * listing.
+ */
+ @Test
+ public void testRootTombstones() throws Exception {
+long ttl = 10L;
+ITtlTimeProvider mockTimeProvider = mock(ITtlTimeProvider.class);
+when(mockTimeProvider.getMetadataTtl()).thenReturn(ttl);
+when(mockTimeProvider.getNow()).thenReturn(100L);
+ITtlTimeProvider originalTimeProvider = guardedFs.getTtlTimeProvider();
+guardedFs.setTtlTimeProvider(mockTimeProvider);
+
+Path base = path(getMethodName() + UUID.randomUUID());
+Path testFile = new Path(base, "test.file");
+
+try {
+ touch(guardedFs, testFile);
+ ContractTestUtils.assertDeleted(guardedFs, testFile, false);
+
+ touch(rawFS, testFile);
+ awaitFileStatus(rawFS, testFile);
+
+ // the rawFS will include the file=
+ LambdaTestUtils.eventually(5000, 1000, () -> {
+checkListingContainsPath(rawFS, testFile);
+ });
+
+ // it will be hidden because of the tombstone
+ checkListingDoesNotContainPath(guardedFs, testFile);
Review comment:
ok, I just wanted this as easy to understand and clean as possible. but I
can add more check for sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard:
Filter expired entries and tombstones when lis…
URL: https://github.com/apache/hadoop/pull/1134#discussion_r306701870
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
##
@@ -538,6 +540,47 @@ public void deleteAfterTombstoneExpiryOobCreate() throws
Exception {
}
}
+ /**
+ * Test that a tombstone won't hide an entry after it's expired in the
+ * listing.
+ */
+ @Test
+ public void testRootTombstones() throws Exception {
+long ttl = 10L;
+ITtlTimeProvider mockTimeProvider = mock(ITtlTimeProvider.class);
+when(mockTimeProvider.getMetadataTtl()).thenReturn(ttl);
+when(mockTimeProvider.getNow()).thenReturn(100L);
+ITtlTimeProvider originalTimeProvider = guardedFs.getTtlTimeProvider();
+guardedFs.setTtlTimeProvider(mockTimeProvider);
+
+Path base = path(getMethodName() + UUID.randomUUID());
+Path testFile = new Path(base, "test.file");
+
+try {
+ touch(guardedFs, testFile);
+ ContractTestUtils.assertDeleted(guardedFs, testFile, false);
+
+ touch(rawFS, testFile);
+ awaitFileStatus(rawFS, testFile);
+
+ // the rawFS will include the file=
+ LambdaTestUtils.eventually(5000, 1000, () -> {
+checkListingContainsPath(rawFS, testFile);
+ });
+
+ // it will be hidden because of the tombstone
+ checkListingDoesNotContainPath(guardedFs, testFile);
Review comment:
also we have other tests for getFIleStatus. This is only for listing. I'd
like to keep this test clean. Your test is already testing for that case we
had, I don't want to mix more things into this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lokeshj1703 opened a new pull request #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions
lokeshj1703 opened a new pull request #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions URL: https://github.com/apache/hadoop/pull/1150 ContainerStateMachine should limit number of pending apply transactions in order to avoid excessive heap usage by the pending transactions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard:
Filter expired entries and tombstones when lis…
URL: https://github.com/apache/hadoop/pull/1134#discussion_r306708473
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardTtl.java
##
@@ -286,4 +290,70 @@ public void testCreateParentHasTombstone() throws
Exception {
}
}
+ /**
+ * Test that listing of metadatas is filtered from expired items.
+ */
+ @Test
+ public void testListingFilteredExpiredItems() throws Exception {
+LOG.info("Authoritative mode: {}", authoritative);
+final S3AFileSystem fs = getFileSystem();
+
+long oldTime = 100L;
+long newTime = 110L;
+long ttl = 9L;
+final String basedir = getMethodName();
Review comment:
`[]` chars are not the best in url names. Corrected to string literal
instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306708881
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -462,6 +619,60 @@ long save(File file, FSImageCompression compression)
throws IOException {
}
}
+private void enableSubSectionsIfRequired() {
+ boolean parallelEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_DEFAULT);
+ int inodeThreshold = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_DEFAULT);
+ int targetSections = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_DEFAULT);
+ boolean compressionEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY,
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_DEFAULT);
+
+
+ if (parallelEnabled) {
+if (compressionEnabled) {
+ LOG.warn("Parallel Image loading is not supported when {} is set to"
+
+ " true. Parallel loading will be disabled.",
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY);
+ WRITE_SUB_SECTIONS = false;
+ return;
+}
+if (targetSections <= 0) {
+ LOG.warn("{} is set to {}. It must be greater than zero. Setting to"
+
+ "default of {}",
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY,
+ targetSections,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_DEFAULT);
+ targetSections =
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_DEFAULT;
+}
+if (inodeThreshold <= 0) {
+ LOG.warn("{} is set to {}. It must be greater than zero. Setting to"
+
+ "default of {}",
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY,
+ targetSections,
Review comment:
Correct - well spotted! I have fixed this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#discussion_r306709699 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java ## @@ -309,15 +316,17 @@ public void put(PathMetadata meta, DirListingMetadata parentDirMeta = new DirListingMetadata(parentPath, DirListingMetadata.EMPTY_DIR, false); + parentDirMeta.setLastUpdated(meta.getLastUpdated()); Review comment: if this is not here then the oob tombstone listing expiry test will fail. but I'll try to write a unit test for this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306709830
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -462,6 +619,60 @@ long save(File file, FSImageCompression compression)
throws IOException {
}
}
+private void enableSubSectionsIfRequired() {
+ boolean parallelEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_DEFAULT);
+ int inodeThreshold = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_DEFAULT);
+ int targetSections = conf.getInt(
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY,
+ DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_DEFAULT);
+ boolean compressionEnabled = conf.getBoolean(
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY,
+ DFSConfigKeys.DFS_IMAGE_COMPRESS_DEFAULT);
+
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#discussion_r306729656 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java ## @@ -309,15 +316,17 @@ public void put(PathMetadata meta, DirListingMetadata parentDirMeta = new DirListingMetadata(parentPath, DirListingMetadata.EMPTY_DIR, false); + parentDirMeta.setLastUpdated(meta.getLastUpdated()); Review comment: also, this is only for the DirListingMeta for LocalMetadataStore, and nothing else. We do not persist DirListingMeta in DynamoDB, so the parent directory(/directories) won't get updated if a child is added. Imagine a scenario where we are deep in the dir structure (eg. 20+ directories in), so we should have an update for each parent FileMetadata (we don't store the listings there) whenever a child is added. That's not good imo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on a change in pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#discussion_r306733738 ## File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java ## @@ -309,15 +316,17 @@ public void put(PathMetadata meta, DirListingMetadata parentDirMeta = new DirListingMetadata(parentPath, DirListingMetadata.EMPTY_DIR, false); + parentDirMeta.setLastUpdated(meta.getLastUpdated()); Review comment: I've added `testUpdateParentLastUpdatedOnPutNewParent` to test this in the new PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306736414
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -389,21 +496,26 @@ private void loadErasureCodingSection(InputStream in)
public static final class Saver {
public static final int CHECK_CANCEL_INTERVAL = 4096;
+public static boolean WRITE_SUB_SECTIONS = false;
Review comment:
I agree we should make this change. I was originally copying the pattern
set by CHECK_CANCEL_INTERVAL, but best to avoid the constants I think. I have
made this change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306736876
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
##
@@ -217,33 +273,151 @@ void loadINodeDirectorySection(InputStream in) throws
IOException {
INodeDirectory p = dir.getInode(e.getParent()).asDirectory();
for (long id : e.getChildrenList()) {
INode child = dir.getInode(id);
- addToParent(p, child);
+ if (addToParent(p, child)) {
+if (child.isFile()) {
+ inodeList.add(child);
+}
+if (inodeList.size() >= 1000) {
+ addToCacheAndBlockMap(inodeList);
+ inodeList.clear();
+}
+ }
+
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306737322
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
##
@@ -217,33 +273,151 @@ void loadINodeDirectorySection(InputStream in) throws
IOException {
INodeDirectory p = dir.getInode(e.getParent()).asDirectory();
for (long id : e.getChildrenList()) {
INode child = dir.getInode(id);
- addToParent(p, child);
+ if (addToParent(p, child)) {
+if (child.isFile()) {
+ inodeList.add(child);
+}
+if (inodeList.size() >= 1000) {
+ addToCacheAndBlockMap(inodeList);
+ inodeList.clear();
+}
+ }
+
}
+
for (int refId : e.getRefChildrenList()) {
INodeReference ref = refList.get(refId);
- addToParent(p, ref);
+ if (addToParent(p, ref)) {
+if (ref.isFile()) {
+ inodeList.add(ref);
+}
+if (inodeList.size() >= 1000) {
+ addToCacheAndBlockMap(inodeList);
+ inodeList.clear();
+}
+ }
}
}
+ addToCacheAndBlockMap(inodeList);
+}
+
+private void addToCacheAndBlockMap(ArrayList inodeList) {
+ try {
+cacheNameMapLock.lock();
+for (INode i : inodeList) {
+ dir.cacheName(i);
+}
+ } finally {
+cacheNameMapLock.unlock();
+ }
+
+ try {
+blockMapLock.lock();
+for (INode i : inodeList) {
+ updateBlocksMap(i.asFile(), fsn.getBlockManager());
+}
+ } finally {
+blockMapLock.unlock();
+ }
}
void loadINodeSection(InputStream in, StartupProgress prog,
Step currentStep) throws IOException {
- INodeSection s = INodeSection.parseDelimitedFrom(in);
- fsn.dir.resetLastInodeId(s.getLastInodeId());
- long numInodes = s.getNumInodes();
- LOG.info("Loading " + numInodes + " INodes.");
- prog.setTotal(Phase.LOADING_FSIMAGE, currentStep, numInodes);
+ loadINodeSectionHeader(in, prog, currentStep);
Counter counter = prog.getCounter(Phase.LOADING_FSIMAGE, currentStep);
- for (int i = 0; i < numInodes; ++i) {
+ int totalLoaded = loadINodesInSection(in, counter);
+ LOG.info("Successfully loaded {} inodes", totalLoaded);
+}
+
+private int loadINodesInSection(InputStream in, Counter counter)
+throws IOException {
+ // As the input stream is a LimitInputStream, the reading will stop when
+ // EOF is encountered at the end of the stream.
+ int cntr = 0;
+ while (true) {
INodeSection.INode p = INodeSection.INode.parseDelimitedFrom(in);
+if (p == null) {
+ break;
+}
if (p.getId() == INodeId.ROOT_INODE_ID) {
- loadRootINode(p);
+ synchronized(this) {
Review comment:
The synchronized call does not seem to add any significant overhead in the
single threaded case, so it would make the code cleaner to just leave it in
place, rather than having a branch for parallel and serial. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306739148
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
##
@@ -217,33 +273,151 @@ void loadINodeDirectorySection(InputStream in) throws
IOException {
INodeDirectory p = dir.getInode(e.getParent()).asDirectory();
for (long id : e.getChildrenList()) {
INode child = dir.getInode(id);
- addToParent(p, child);
+ if (addToParent(p, child)) {
+if (child.isFile()) {
+ inodeList.add(child);
+}
+if (inodeList.size() >= 1000) {
+ addToCacheAndBlockMap(inodeList);
+ inodeList.clear();
+}
+ }
+
}
+
for (int refId : e.getRefChildrenList()) {
INodeReference ref = refList.get(refId);
- addToParent(p, ref);
+ if (addToParent(p, ref)) {
+if (ref.isFile()) {
+ inodeList.add(ref);
+}
+if (inodeList.size() >= 1000) {
+ addToCacheAndBlockMap(inodeList);
+ inodeList.clear();
+}
+ }
}
}
+ addToCacheAndBlockMap(inodeList);
+}
+
+private void addToCacheAndBlockMap(ArrayList inodeList) {
+ try {
+cacheNameMapLock.lock();
+for (INode i : inodeList) {
+ dir.cacheName(i);
+}
+ } finally {
+cacheNameMapLock.unlock();
+ }
+
+ try {
+blockMapLock.lock();
+for (INode i : inodeList) {
+ updateBlocksMap(i.asFile(), fsn.getBlockManager());
+}
+ } finally {
+blockMapLock.unlock();
+ }
}
void loadINodeSection(InputStream in, StartupProgress prog,
Step currentStep) throws IOException {
- INodeSection s = INodeSection.parseDelimitedFrom(in);
- fsn.dir.resetLastInodeId(s.getLastInodeId());
- long numInodes = s.getNumInodes();
- LOG.info("Loading " + numInodes + " INodes.");
- prog.setTotal(Phase.LOADING_FSIMAGE, currentStep, numInodes);
+ loadINodeSectionHeader(in, prog, currentStep);
Counter counter = prog.getCounter(Phase.LOADING_FSIMAGE, currentStep);
- for (int i = 0; i < numInodes; ++i) {
+ int totalLoaded = loadINodesInSection(in, counter);
+ LOG.info("Successfully loaded {} inodes", totalLoaded);
+}
+
+private int loadINodesInSection(InputStream in, Counter counter)
+throws IOException {
+ // As the input stream is a LimitInputStream, the reading will stop when
+ // EOF is encountered at the end of the stream.
+ int cntr = 0;
+ while (true) {
INodeSection.INode p = INodeSection.INode.parseDelimitedFrom(in);
+if (p == null) {
+ break;
+}
if (p.getId() == INodeId.ROOT_INODE_ID) {
- loadRootINode(p);
+ synchronized(this) {
+loadRootINode(p);
+ }
} else {
INode n = loadINode(p);
- dir.addToInodeMap(n);
+ synchronized(this) {
+dir.addToInodeMap(n);
+ }
+}
+cntr ++;
+if (counter != null) {
+ counter.increment();
}
-counter.increment();
}
+ return cntr;
+}
+
+
+private void loadINodeSectionHeader(InputStream in, StartupProgress prog,
+Step currentStep) throws IOException {
+ INodeSection s = INodeSection.parseDelimitedFrom(in);
+ fsn.dir.resetLastInodeId(s.getLastInodeId());
+ long numInodes = s.getNumInodes();
+ LOG.info("Loading " + numInodes + " INodes.");
+ prog.setTotal(Phase.LOADING_FSIMAGE, currentStep, numInodes);
+}
+
+void loadINodeSectionInParallel(ExecutorService service,
+ArrayList sections,
+String compressionCodec, StartupProgress prog,
+Step currentStep) throws IOException {
+ LOG.info("Loading the INode section in parallel with {} sub-sections",
+ sections.size());
+ CountDownLatch latch = new CountDownLatch(sections.size());
+ AtomicInteger totalLoaded = new AtomicInteger(0);
+ final CopyOnWriteArrayList exceptions =
+ new CopyOnWriteArrayList<>();
+
+ for (int i=0; i < sections.size(); i++) {
+FileSummary.Section s = sections.get(i);
+InputStream ins = parent.getInputStreamForSection(s, compressionCodec);
+if (i == 0) {
+ // The first inode section has a header which must be processed first
+ loadINodeSectionHeader(ins, prog, currentStep);
+}
+
+service.submit(new Runnable() {
+ public void run() {
+try {
+ totalLoaded.addAndGet(loadINodesInSection(ins, null));
+ prog.setCount(Phase.LO
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306739479
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -255,14 +345,28 @@ public int compare(FileSummary.Section s1,
FileSummary.Section s2) {
case INODE: {
currentStep = new Step(StepType.INODES);
prog.beginStep(Phase.LOADING_FSIMAGE, currentStep);
- inodeLoader.loadINodeSection(in, prog, currentStep);
+ stageSubSections = getSubSectionsOfName(
+ subSections, SectionName.INODE_SUB);
+ if (loadInParallel && (stageSubSections.size() > 0)) {
+inodeLoader.loadINodeSectionInParallel(executorService,
+stageSubSections, summary.getCodec(), prog, currentStep);
+ } else {
+inodeLoader.loadINodeSection(in, prog, currentStep);
+ }
}
break;
case INODE_REFERENCE:
snapshotLoader.loadINodeReferenceSection(in);
break;
case INODE_DIR:
- inodeLoader.loadINodeDirectorySection(in);
+ stageSubSections = getSubSectionsOfName(
+ subSections, SectionName.INODE_DIR_SUB);
+ if (loadInParallel && stageSubSections.size() > 0) {
Review comment:
Yes, I need to figure out how to add some tests for this overall change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve fsimage load time by writing sub-sections to the fsimage index
sodonnel commented on a change in pull request #1028: HDFS-14617 - Improve
fsimage load time by writing sub-sections to the fsimage index
URL: https://github.com/apache/hadoop/pull/1028#discussion_r306740049
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
##
@@ -187,6 +195,73 @@ void load(File file) throws IOException {
}
}
+/**
+ * Given a FSImage FileSummary.section, return a LimitInput stream set to
+ * the starting position of the section and limited to the section length
+ * @param section The FileSummary.Section containing the offset and length
+ * @param compressionCodec The compression codec in use, if any
Review comment:
Looks like I missed the 'return' description. I have added it now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: defaultMetricsHoldsRpcMetrics.png > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: classes.png > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: rpcm.hprof.xz > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16455) ABFS: Implement the access method
Bilahari T H created HADOOP-16455: - Summary: ABFS: Implement the access method Key: HADOOP-16455 URL: https://issues.apache.org/jira/browse/HADOOP-16455 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Bilahari T H Assignee: Bilahari T H Implement the access method -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HADOOP-16435: -- Attachment: related.jxray.png > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891774#comment-16891774 ] Zoltan Haindrich commented on HADOOP-16435: --- [~aajisaka] I've uploaded some more info; including a full heapdump somewhere around the heap gets filled up > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16451) Update jackson-databind to 2.9.9.1
[
https://issues.apache.org/jira/browse/HADOOP-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891782#comment-16891782
]
Hadoop QA commented on HADOOP-16451:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
53m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 59s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-applications-catalog-webapp in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.0 Server=19.03.0 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HADOOP-16451 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975590/HADOOP-16451.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 777fa8d61214 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cf9ff08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16416/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16416/console |
| Powered
[GitHub] [hadoop] bgaborg commented on a change in pull request #1123: HADOOP-16380 S3Guard to determine empty directory status for all non-root directories
bgaborg commented on a change in pull request #1123: HADOOP-16380 S3Guard to
determine empty directory status for all non-root directories
URL: https://github.com/apache/hadoop/pull/1123#discussion_r305812343
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
##
@@ -818,6 +828,22 @@ public static void callQuietly(final Logger log,
return null;
}
+ /**
+ * Get a file status from S3A with the {@code needEmptyDirectoryFlag}
+ * state probed.
+ * This accesses a package-private method in the
+ * S3A filesystem.
+ * @param fs filesystem
+ * @param dir directory
+ * @return a status
+ * @throws IOException
+ */
+ public static S3AFileStatus getStatusWithEmptyDirFlag(
Review comment:
just to remind me that I have a similar method in pr so I have to rebase:
https://github.com/apache/hadoop/pull/1134/files#diff-d946bda31a9bf644846f57b45a89524dR831
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514602153 rebased, fixed based on @steveloughran's review. Tested against ireland, local and dynamo. No unknown issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16062) Add ability to disable Configuration reload registry
[ https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HADOOP-16062: - Assignee: Zoltan Haindrich > Add ability to disable Configuration reload registry > > > Key: HADOOP-16062 > URL: https://issues.apache.org/jira/browse/HADOOP-16062 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HADOOP-16062.01.patch, jxray_registry.png > > > In Hive we see that after an extensive amount of usage there are a lot of > Configuration objects not fully reclaimed because of Configuration's REGISTRY > weak hashmap. > https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321 > !jxray_registry.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16062) Add ability to disable Configuration reload registry
[
https://issues.apache.org/jira/browse/HADOOP-16062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891822#comment-16891822
]
Zoltan Haindrich commented on HADOOP-16062:
---
[~aajisaka] could you please take a look?
note: it seems to me that the {{Configuration.reloadExistingConfigurations}}
functionality is only used to aid some interesting way to "deprecate" some
config keys
while keeping the deprecation disconnected from the main config keys section; I
think this could be probably done differently:
* by making sure that this "deprecation" call is either inside
{{org.apache.hadoop.fs.FileSystem}} implementations (or its imported)
* make sure that before constructing any "configuration" objects the
serviceloader is invoked - to ensure fully loading all FileSystem
implementation; which in turn will run all the static blocks
* what this is missing: probably for runtime loaded jars which contain
filesystem implementations this will not work
> Add ability to disable Configuration reload registry
>
>
> Key: HADOOP-16062
> URL: https://issues.apache.org/jira/browse/HADOOP-16062
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HADOOP-16062.01.patch, jxray_registry.png
>
>
> In Hive we see that after an extensive amount of usage there are a lot of
> Configuration objects not fully reclaimed because of Configuration's REGISTRY
> weak hashmap.
> https://github.com/apache/hadoop/blob/abde1e1f58d5b699e4b8e460cff68e154738169b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L321
> !jxray_registry.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16448) Connection to Hadoop homepage is not secure
[ https://issues.apache.org/jira/browse/HADOOP-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Nemuri updated HADOOP-16448: Status: Patch Available (was: Open) > Connection to Hadoop homepage is not secure > --- > > Key: HADOOP-16448 > URL: https://issues.apache.org/jira/browse/HADOOP-16448 > Project: Hadoop Common > Issue Type: Bug > Components: website >Affects Versions: 2.8.5, 2.7.7, 3.0.3, 2.9.2, 3.1.1, 3.0.2, 3.2.0, 2.7.6, > 2.8.4, 3.0.1, 2.9.1, 3.1.0, 3.0.0, 2.7.5, 2.8.3, 2.8.2, 2.8.1, 2.7.4, 2.6.5, > 2.6.4, 2.9.0, 2.7.3, 2.6.3, 2.6.2, 2.7.2, 2.7.1, 2.8.0, 2.7.0, 2.6.1, 2.6.0, > 2.5.2, 2.5.1, 2.4.1, 2.5.0, 2.4.0, 0.23.11 >Reporter: Kaspar Tint >Assignee: Sandeep Nemuri >Priority: Major > Labels: newbie > Attachments: Screen Shot 2019-07-23 at 9.37.54 AM.png > > > When visiting the [Hadoop > website|https://hadoop.apache.org/docs/r3.2.0/index.html] with the latest > Firefox browser (v 68.0.1) it appears that the website cannot be reached > through secure means by default. > The culprit seems to be the fact that the two header images presented on the > page are loaded in via *HTTP* > !Screen Shot 2019-07-23 at 9.37.54 AM.png!. > These images are located in the respective locations: > http://hadoop.apache.org/images/hadoop-logo.jpg > http://www.apache.org/images/asf_logo_wide.png > These images can be reached also from the following locations: > https://hadoop.apache.org/images/hadoop-logo.jpg > https://www.apache.org/images/asf_logo_wide.png > As one can see, a fix could be made to use a more safe way of including in > the two header pictures to the page. > I feel like I am in danger when reading the Hadoop documentation from the > official Hadoop webpage in a non secure way. Thus I felt the need to open > this ticket and raise the issue in order to have a future where everyone can > learn from Hadoop documentation in a safe and secure way. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions
hadoop-yetus commented on issue #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions URL: https://github.com/apache/hadoop/pull/1150#issuecomment-514635063 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | +1 | mvninstall | 579 | trunk passed | | +1 | compile | 355 | trunk passed | | +1 | checkstyle | 65 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 784 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 142 | trunk passed | | 0 | spotbugs | 413 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 605 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 17 | Maven dependency ordering for patch | | +1 | mvninstall | 551 | the patch passed | | +1 | compile | 366 | the patch passed | | +1 | javac | 366 | the patch passed | | -0 | checkstyle | 37 | hadoop-hdds: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 739 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 144 | the patch passed | | +1 | findbugs | 619 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 284 | hadoop-hdds in the patch passed. | | -1 | unit | 1541 | hadoop-ozone in the patch failed. | | +1 | asflicense | 41 | The patch does not generate ASF License warnings. | | | | 7137 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.TestOzoneConfigurationFields | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1150 | | JIRA Issue | HDDS-1816 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2fa951e5a4a6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cf9ff08 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/1/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/1/testReport/ | | Max. process+thread count | 5292 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service U: hadoop-hdds | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e
[GitHub] [hadoop] nandakumar131 opened a new pull request #1151: HDDS-1853. Fix failing blockade test-cases.
nandakumar131 opened a new pull request #1151: HDDS-1853. Fix failing blockade test-cases. URL: https://github.com/apache/hadoop/pull/1151 This PR is to fix and make sure that all the test-cases in blockade are working. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891915#comment-16891915 ] lqjacklee commented on HADOOP-16435: [~kgyrtkirk] org.apache.hadoop.ipc.metrics.RpcMetrics#shutdown will be called when the server is stopped. so I wonder whether the solution is ok? > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16456) Refactor the S3A codebase into a more maintainable and testable form
Steve Loughran created HADOOP-16456: --- Summary: Refactor the S3A codebase into a more maintainable and testable form Key: HADOOP-16456 URL: https://issues.apache.org/jira/browse/HADOOP-16456 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 3.3.0 Reporter: Steve Loughran The S3A Codebase has got too complex to be maintained. In particular, * the lack of layering in the S3AFileSystem class means that all subcomponents (delegation, dynamo db, block outputstream etc) all get given a back reference and make arbitrary calls in to it. * We can't test in isolation, and while integration tests are the most rigorous testing we can have, they are slow, hard to inject failures into and do not work on isolated parts of code * The code within the S3A FileSystem calls the toplevel API calls internally, so mixing public interface with the implementation details * We are adding context through S3Guard calls for: consistency, performance and recovery; we can't do that without a clean split between that public API and the internals Proposed: # we carefully break up the S3AFileSystem into a layered design # with a "StoreContext" to bind components of the connector to it # and some form of operation context to be passed in with each request to represent the active operation and its state (including that for S3Guard BulkOperations) See [refactoring S3A|https://github.com/steveloughran/engineering-proposals/blob/master/refactoring-s3a.md] I've already started using some of this design in the HADOOP-15183 component, for the addition of those S3Guard bulk operations, and to add a medium-life "RenameOperation". The proposal document reviews that experience and discusses improvements. As noted: this needs to be done with care. We still need to maintain the existing codebase; the more radically we change the code not only do we increase the risk of the changes being wrong, we make backporting that much harder. But we can't sustain the current design -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891922#comment-16891922 ] Zoltan Haindrich commented on HADOOP-16435: --- [~Jack-Lee] yes, it will be unregistered when the server is stopped - if I don't fully understand your concern; could you elaborate? > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16456) Refactor the S3A codebase into a more maintainable and testable form
[ https://issues.apache.org/jira/browse/HADOOP-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-16456: --- Assignee: Steve Loughran > Refactor the S3A codebase into a more maintainable and testable form > > > Key: HADOOP-16456 > URL: https://issues.apache.org/jira/browse/HADOOP-16456 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > The S3A Codebase has got too complex to be maintained. In particular, > * the lack of layering in the S3AFileSystem class means that all > subcomponents (delegation, dynamo db, block outputstream etc) all get given a > back reference and make arbitrary calls in to it. > * We can't test in isolation, and while integration tests are the most > rigorous testing we can have, they are slow, hard to inject failures into and > do not work on isolated parts of code > * The code within the S3A FileSystem calls the toplevel API calls internally, > so mixing public interface with the implementation details > * We are adding context through S3Guard calls for: consistency, performance > and recovery; we can't do that without a clean split between that public API > and the internals > Proposed: > # we carefully break up the S3AFileSystem into a layered design > # with a "StoreContext" to bind components of the connector to it > # and some form of operation context to be passed in with each request to > represent the active operation and its state (including that for S3Guard > BulkOperations) > See [refactoring > S3A|https://github.com/steveloughran/engineering-proposals/blob/master/refactoring-s3a.md] > I've already started using some of this design in the HADOOP-15183 component, > for the addition of those S3Guard bulk operations, and to add a medium-life > "RenameOperation". The proposal document reviews that experience and > discusses improvements. > As noted: this needs to be done with care. We still need to maintain the > existing codebase; the more radically we change the code not only do we > increase the risk of the changes being wrong, we make backporting that much > harder. But we can't sustain the current design -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16435) RpcMetrics should not be retained forever
[ https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891922#comment-16891922 ] Zoltan Haindrich edited comment on HADOOP-16435 at 7/24/19 2:54 PM: [~Jack-Lee] yes, it will be unregistered when the server is stopped - if I don't fully understand your concern; could you be more specific? was (Author: kgyrtkirk): [~Jack-Lee] yes, it will be unregistered when the server is stopped - if I don't fully understand your concern; could you elaborate? > RpcMetrics should not be retained forever > - > > Key: HADOOP-16435 > URL: https://issues.apache.org/jira/browse/HADOOP-16435 > Project: Hadoop Common > Issue Type: Bug > Components: rpc-server >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Critical > Attachments: HADOOP-16435.01.patch, classes.png, > defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz > > > * RpcMetrics objects are registered into > [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101] > * although there is a shutdown() call (which is actually invoked) it doesn't > unregisters itself from the > [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185] > * RpcDetailedMetrics has the same issue > background > * hiveserver2 slowly eats up memory when running simple queries in new > sessions (select 1) > * every session opens a tezsession > * tezsession has rpcmetrics > * with a 150M heap after around 30 session the jvm gets outofmemmory -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16449) Allow an empty credential provider chain, separate chains for S3 and DDB
[ https://issues.apache.org/jira/browse/HADOOP-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891923#comment-16891923 ] Steve Loughran commented on HADOOP-16449: - I have comments, but first can you submit it as a github PR, and state which s3a endpoint you ran against Key point: DDB shouldn't contain any of the logic about how to get its auth chain. Thats something which should come to it via the StoreContext used to help bind it to the rest of the S3A connector. > Allow an empty credential provider chain, separate chains for S3 and DDB > > > Key: HADOOP-16449 > URL: https://issues.apache.org/jira/browse/HADOOP-16449 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > Attachments: HADOOP-16449.01.patch > > > Currently, credentials cannot be empty (falls back to using the default > chain). Credentials for S3 and DDB are always the same. > In some cases it can be useful to use a different credential chain for S3 and > DDB, as well as allow for an empty credential chain. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16245) Enabling SSL within LdapGroupsMapping can break system SSL configs
[
https://issues.apache.org/jira/browse/HADOOP-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-16245:
-
Attachment: HADOOP-16245.004.patch
> Enabling SSL within LdapGroupsMapping can break system SSL configs
> --
>
> Key: HADOOP-16245
> URL: https://issues.apache.org/jira/browse/HADOOP-16245
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, security
>Affects Versions: 2.9.1, 2.8.4, 2.7.6, 3.1.1, 3.0.3
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16245.000.patch, HADOOP-16245.001.patch,
> HADOOP-16245.002.patch, HADOOP-16245.003.patch, HADOOP-16245.004.patch
>
>
> When debugging an issue where one of our server components was unable to
> communicate with other components via SSL, we realized that LdapGroupsMapping
> sets its SSL configurations globally, rather than scoping them to the HTTP
> clients it creates.
> {code:title=LdapGroupsMapping}
> DirContext getDirContext() throws NamingException {
> if (ctx == null) {
> // Set up the initial environment for LDAP connectivity
> Hashtable env = new Hashtable();
> env.put(Context.INITIAL_CONTEXT_FACTORY,
> com.sun.jndi.ldap.LdapCtxFactory.class.getName());
> env.put(Context.PROVIDER_URL, ldapUrl);
> env.put(Context.SECURITY_AUTHENTICATION, "simple");
> // Set up SSL security, if necessary
> if (useSsl) {
> env.put(Context.SECURITY_PROTOCOL, "ssl");
> if (!keystore.isEmpty()) {
> System.setProperty("javax.net.ssl.keyStore", keystore);
> }
> if (!keystorePass.isEmpty()) {
> System.setProperty("javax.net.ssl.keyStorePassword", keystorePass);
> }
> if (!truststore.isEmpty()) {
> System.setProperty("javax.net.ssl.trustStore", truststore);
> }
> if (!truststorePass.isEmpty()) {
> System.setProperty("javax.net.ssl.trustStorePassword",
> truststorePass);
> }
> }
> env.put(Context.SECURITY_PRINCIPAL, bindUser);
> env.put(Context.SECURITY_CREDENTIALS, bindPassword);
> env.put("com.sun.jndi.ldap.connect.timeout",
> conf.get(CONNECTION_TIMEOUT,
> String.valueOf(CONNECTION_TIMEOUT_DEFAULT)));
> env.put("com.sun.jndi.ldap.read.timeout", conf.get(READ_TIMEOUT,
> String.valueOf(READ_TIMEOUT_DEFAULT)));
> ctx = new InitialDirContext(env);
> }
> {code}
> Notice the {{System.setProperty()}} calls, which will change settings
> JVM-wide. This causes issues for other SSL clients, which may rely on the
> default JVM truststore being used. This behavior was initially introduced by
> HADOOP-8121, and extended to include the truststore configurations in
> HADOOP-12862.
> The correct approach is to use a mechanism which is scoped to the LDAP
> requests only. The right approach appears to be to use the
> {{java.naming.ldap.factory.socket}} parameter to set the socket factory to a
> custom SSL socket factory which correctly sets the key and trust store
> parameters. See an example [here|https://stackoverflow.com/a/4615497/4979203].
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16245) Enabling SSL within LdapGroupsMapping can break system SSL configs
[
https://issues.apache.org/jira/browse/HADOOP-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891927#comment-16891927
]
Erik Krogen commented on HADOOP-16245:
--
Attached v004 patch to address checkstyle/whitespace issues.
> Enabling SSL within LdapGroupsMapping can break system SSL configs
> --
>
> Key: HADOOP-16245
> URL: https://issues.apache.org/jira/browse/HADOOP-16245
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, security
>Affects Versions: 2.9.1, 2.8.4, 2.7.6, 3.1.1, 3.0.3
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16245.000.patch, HADOOP-16245.001.patch,
> HADOOP-16245.002.patch, HADOOP-16245.003.patch, HADOOP-16245.004.patch
>
>
> When debugging an issue where one of our server components was unable to
> communicate with other components via SSL, we realized that LdapGroupsMapping
> sets its SSL configurations globally, rather than scoping them to the HTTP
> clients it creates.
> {code:title=LdapGroupsMapping}
> DirContext getDirContext() throws NamingException {
> if (ctx == null) {
> // Set up the initial environment for LDAP connectivity
> Hashtable env = new Hashtable();
> env.put(Context.INITIAL_CONTEXT_FACTORY,
> com.sun.jndi.ldap.LdapCtxFactory.class.getName());
> env.put(Context.PROVIDER_URL, ldapUrl);
> env.put(Context.SECURITY_AUTHENTICATION, "simple");
> // Set up SSL security, if necessary
> if (useSsl) {
> env.put(Context.SECURITY_PROTOCOL, "ssl");
> if (!keystore.isEmpty()) {
> System.setProperty("javax.net.ssl.keyStore", keystore);
> }
> if (!keystorePass.isEmpty()) {
> System.setProperty("javax.net.ssl.keyStorePassword", keystorePass);
> }
> if (!truststore.isEmpty()) {
> System.setProperty("javax.net.ssl.trustStore", truststore);
> }
> if (!truststorePass.isEmpty()) {
> System.setProperty("javax.net.ssl.trustStorePassword",
> truststorePass);
> }
> }
> env.put(Context.SECURITY_PRINCIPAL, bindUser);
> env.put(Context.SECURITY_CREDENTIALS, bindPassword);
> env.put("com.sun.jndi.ldap.connect.timeout",
> conf.get(CONNECTION_TIMEOUT,
> String.valueOf(CONNECTION_TIMEOUT_DEFAULT)));
> env.put("com.sun.jndi.ldap.read.timeout", conf.get(READ_TIMEOUT,
> String.valueOf(READ_TIMEOUT_DEFAULT)));
> ctx = new InitialDirContext(env);
> }
> {code}
> Notice the {{System.setProperty()}} calls, which will change settings
> JVM-wide. This causes issues for other SSL clients, which may rely on the
> default JVM truststore being used. This behavior was initially introduced by
> HADOOP-8121, and extended to include the truststore configurations in
> HADOOP-12862.
> The correct approach is to use a mechanism which is scoped to the LDAP
> requests only. The right approach appears to be to use the
> {{java.naming.ldap.factory.socket}} parameter to set the socket factory to a
> custom SSL socket factory which correctly sets the key and trust store
> parameters. See an example [here|https://stackoverflow.com/a/4615497/4979203].
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514670105 running the tests too (non scale) One thing I want to be sure about is: does `DirListingMetadata.removeExpiredEntriesFromListing()` work if invoked in one thread while another is already doing it. I believe so, because it's using a ConcurrentHashMap and the javaadocs for that seem to imply it. Do you also believe this to be true? nit: now that `PathMetadata.toString()` shows the lastUpdated state, `DDBPathMetadata` should cut it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #1151: HDDS-1853. Fix failing blockade test-cases.
mukul1987 commented on a change in pull request #1151: HDDS-1853. Fix failing blockade test-cases. URL: https://github.com/apache/hadoop/pull/1151#discussion_r306863901 ## File path: hadoop-ozone/dist/src/main/compose/ozoneblockade/docker-config ## @@ -32,9 +32,9 @@ OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1 OZONE-SITE.XML_ozone.scm.pipeline.destroy.timeout=15s OZONE-SITE.XML_hdds.heartbeat.interval=2s OZONE-SITE.XML_hdds.scm.wait.time.after.safemode.exit=30s -OZONE-SITE.XML_hdds.scm.replication.thread.interval=5s -OZONE-SITE.XML_hdds.scm.replication.event.timeout=7s -OZONE-SITE.XML_dfs.ratis.server.failure.duration=25s +OZONE-SITE.XML_hdds.scm.replication.thread.interval=6s Review comment: Can you please explain the need of increasing these values ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16435) RpcMetrics should not be retained forever
[
https://issues.apache.org/jira/browse/HADOOP-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891936#comment-16891936
]
Hadoop QA commented on HADOOP-16435:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-16435 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16435 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16417/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RpcMetrics should not be retained forever
> -
>
> Key: HADOOP-16435
> URL: https://issues.apache.org/jira/browse/HADOOP-16435
> Project: Hadoop Common
> Issue Type: Bug
> Components: rpc-server
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HADOOP-16435.01.patch, classes.png,
> defaultMetricsHoldsRpcMetrics.png, related.jxray.png, rpcm.hprof.xz
>
>
> * RpcMetrics objects are registered into
> [defaultmetricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L101]
> * although there is a shutdown() call (which is actually invoked) it doesn't
> unregisters itself from the
> [metricssystem|https://github.com/apache/hadoop/blob/85d9111a88f94a5e6833cd142272be2c5823e922/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java#L185]
> * RpcDetailedMetrics has the same issue
> background
> * hiveserver2 slowly eats up memory when running simple queries in new
> sessions (select 1)
> * every session opens a tezsession
> * tezsession has rpcmetrics
> * with a 150M heap after around 30 session the jvm gets outofmemmory
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on a change in pull request #1151: HDDS-1853. Fix failing blockade test-cases.
nandakumar131 commented on a change in pull request #1151: HDDS-1853. Fix failing blockade test-cases. URL: https://github.com/apache/hadoop/pull/1151#discussion_r306868976 ## File path: hadoop-ozone/dist/src/main/compose/ozoneblockade/docker-config ## @@ -32,9 +32,9 @@ OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1 OZONE-SITE.XML_ozone.scm.pipeline.destroy.timeout=15s OZONE-SITE.XML_hdds.heartbeat.interval=2s OZONE-SITE.XML_hdds.scm.wait.time.after.safemode.exit=30s -OZONE-SITE.XML_hdds.scm.replication.thread.interval=5s -OZONE-SITE.XML_hdds.scm.replication.event.timeout=7s -OZONE-SITE.XML_dfs.ratis.server.failure.duration=25s +OZONE-SITE.XML_hdds.scm.replication.thread.interval=6s Review comment: In some test-cases we want to write to a pipeline where there are only two nodes in the ring (we cut the connection to the third node). In such cases SCM was destroying the pipeline immediately and the replication manager kicks-in. These values are adjusted such that the client gets enough time to write to a pipeline where there are only two healthy nodes and verify the writes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514676291 I'll add synchronized to `DirListingMetadata.removeExpiredEntriesFromListing()`. It's a little method so I would not bother with more sophisticated locking than that. I'll correct the > PathMetadata.toString() shows the lastUpdated state, DDBPathMetadata should cut it. nit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514683996 OK, +1 pending those changes FWIW, I do think that the iterator code may actually be thread-safe, it's just really hard to parse the javadocs and I do have some distant memories of something like this raising a ConcurrentModificationException for me once ... but those javadocs say "no"; so maybe it was the classic HashTable or something I got burned by. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
bgaborg commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514694409 Thanks for the review @steveloughran, the pr is up to date with those changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
hadoop-yetus commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514696244 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1134 | trunk passed | | +1 | compile | 33 | trunk passed | | +1 | checkstyle | 21 | trunk passed | | +1 | mvnsite | 37 | trunk passed | | +1 | shadedclient | 689 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 27 | trunk passed | | 0 | spotbugs | 61 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 59 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 34 | the patch passed | | +1 | compile | 30 | the patch passed | | +1 | javac | 30 | the patch passed | | -0 | checkstyle | 18 | hadoop-tools/hadoop-aws: The patch generated 1 new + 26 unchanged - 0 fixed = 27 total (was 26) | | +1 | mvnsite | 35 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 742 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | -1 | findbugs | 74 | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 | unit | 283 | hadoop-aws in the patch passed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 3402 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-tools/hadoop-aws | | | Inconsistent synchronization of org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.ttlTimeProvider; locked 75% of time Unsynchronized access at LocalMetadataStore.java:75% of time Unsynchronized access at LocalMetadataStore.java:[line 623] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1134/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1134 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 09333099e4fc 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cf9ff08 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1134/7/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1134/7/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1134/7/testReport/ | | Max. process+thread count | 447 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1134/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
Eric Yang created HADOOP-16457:
--
Summary: Hadoop does not work without Kerberos for simple security
Key: HADOOP-16457
URL: https://issues.apache.org/jira/browse/HADOOP-16457
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Eric Yang
Assignee: Prabhu Joseph
When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
is still setup. This prevents datanode to talk to namenode.
Error message in namenode logs:
{code}
2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
initializers set :
org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
2019-07-24 16:06:26,212 WARN
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
accessible by dn/eyang-5.openstacklo...@example.com
{code}
Errors in datanode log:
{code}
2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
{code}
The logic in HADOOP-16354 always added AuthFilter regardless which http filter
initializer is chosen. This is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-16457:
---
Description:
When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
is still setup. This prevents datanode to talk to namenode.
Error message in namenode logs:
{code}
2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
initializers set :
org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
2019-07-24 16:06:26,212 WARN
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
accessible by dn/eyang-5.openstacklo...@example.com
{code}
Errors in datanode log:
{code}
2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
{code}
The logic in HADOOP-16354 always added AuthFilter regardless security is
enabled or not. This is incorrect. When simple security is chosen and using
StaticUserWebFilter. AutheFilter check should not be required for datanode to
communicate with namenode.
was:
When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
is still setup. This prevents datanode to talk to namenode.
Error message in namenode logs:
{code}
2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
initializers set :
org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
2019-07-24 16:06:26,212 WARN
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
accessible by dn/eyang-5.openstacklo...@example.com
{code}
Errors in datanode log:
{code}
2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
{code}
The logic in HADOOP-16354 always added AuthFilter regardless which http filter
initializer is chosen. This is wrong.
> Hadoop does not work without Kerberos for simple security
> -
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891972#comment-16891972
]
Eric Yang commented on HADOOP-16457:
[~Prabhu Joseph] Sorry, I missed the logic during the review process, DFSUtils
loadSslConfiguration should check if simple security is in use. Otherwise, it
will prevent user from setting up a simple security cluster.
> Hadoop does not work without Kerberos for simple security
> -
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891972#comment-16891972
]
Eric Yang edited comment on HADOOP-16457 at 7/24/19 4:20 PM:
-
[~Prabhu Joseph] Sorry, I missed the logic during the review process, DFSUtils
loadSslConfiguration should check if simple security and StaticUserWebFilter
are in use. Otherwise, it will prevent user from setting up a simple security
cluster.
was (Author: eyang):
[~Prabhu Joseph] Sorry, I missed the logic during the review process, DFSUtils
loadSslConfiguration should check if simple security is in use. Otherwise, it
will prevent user from setting up a simple security cluster.
> Hadoop does not work without Kerberos for simple security
> -
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16448) Connection to Hadoop homepage is not secure
[
https://issues.apache.org/jira/browse/HADOOP-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891997#comment-16891997
]
Hadoop QA commented on HADOOP-16448:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-16448 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16448 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16418/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Connection to Hadoop homepage is not secure
> ---
>
> Key: HADOOP-16448
> URL: https://issues.apache.org/jira/browse/HADOOP-16448
> Project: Hadoop Common
> Issue Type: Bug
> Components: website
>Affects Versions: 0.23.11, 2.4.0, 2.5.0, 2.4.1, 2.5.1, 2.5.2, 2.6.0,
> 2.6.1, 2.7.0, 2.8.0, 2.7.1, 2.7.2, 2.6.2, 2.6.3, 2.7.3, 2.9.0, 2.6.4, 2.6.5,
> 2.7.4, 2.8.1, 2.8.2, 2.8.3, 2.7.5, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6,
> 3.2.0, 3.0.2, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5
>Reporter: Kaspar Tint
>Assignee: Sandeep Nemuri
>Priority: Major
> Labels: newbie
> Attachments: Screen Shot 2019-07-23 at 9.37.54 AM.png
>
>
> When visiting the [Hadoop
> website|https://hadoop.apache.org/docs/r3.2.0/index.html] with the latest
> Firefox browser (v 68.0.1) it appears that the website cannot be reached
> through secure means by default.
> The culprit seems to be the fact that the two header images presented on the
> page are loaded in via *HTTP*
> !Screen Shot 2019-07-23 at 9.37.54 AM.png!.
> These images are located in the respective locations:
> http://hadoop.apache.org/images/hadoop-logo.jpg
> http://www.apache.org/images/asf_logo_wide.png
> These images can be reached also from the following locations:
> https://hadoop.apache.org/images/hadoop-logo.jpg
> https://www.apache.org/images/asf_logo_wide.png
> As one can see, a fix could be made to use a more safe way of including in
> the two header pictures to the page.
> I feel like I am in danger when reading the Hadoop documentation from the
> official Hadoop webpage in a non secure way. Thus I felt the need to open
> this ticket and raise the issue in order to have a future where everyone can
> learn from Hadoop documentation in a safe and secure way.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514716965 LGTM +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
steveloughran commented on issue #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134#issuecomment-514719851 committed to trunk; thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis…
steveloughran closed pull request #1134: HADOOP-16433. S3Guard: Filter expired entries and tombstones when lis… URL: https://github.com/apache/hadoop/pull/1134 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16433) S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren
[
https://issues.apache.org/jira/browse/HADOOP-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16433.
-
Resolution: Fixed
+1, committed to trunk.
Thanks!
> S3Guard: Filter expired entries and tombstones when listing with
> MetadataStore#listChildren
> ---
>
> Key: HADOOP-16433
> URL: https://issues.apache.org/jira/browse/HADOOP-16433
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Blocker
>
> Currently, we don't filter out entries in {{listChildren}} implementations.
> This can cause bugs and inconsistencies, so this should be fixed.
> It can lead to a status where we can't recover from the following:
> {{guarded and raw (OOB op) clients are doing ops to S3}}
> {noformat}
> Guarded: touch /
> Guarded: touch /
> Guarded: rm / {{-> tombstone in MS}}
> RAW: touch //file.ext {{-> file is hidden with a tombstone}}
> Guarded: ls / {{-> only will show up in the listing. }}
> {noformat}
> After we change the following code
> {code:java}
> final List metas = new ArrayList<>();
> for (Item item : items) {
> DDBPathMetadata meta = itemToPathMetadata(item, username);
> metas.add(meta);
> }
> {code}
> to
> {code:java}
> // handle expiry - only add not expired entries to listing.
> if (meta.getLastUpdated() == 0 ||
> !meta.isExpired(ttlTimeProvider.getMetadataTtl(),
> ttlTimeProvider.getNow())) {
> metas.add(meta);
> }
> {code}
> we will filter out expired entries from the listing, so we can recover form
> these kind of OOB ops.
> Note: we have to handle the lastUpdated == 0 case, where the lastUpdated
> field is not filled in!
> Note: this can only be fixed cleanly after HADOOP-16383 is fixed because we
> need to have the TTLtimeProvider in MS to handle this internally.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16458) LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
[
https://issues.apache.org/jira/browse/HADOOP-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892027#comment-16892027
]
Steve Loughran commented on HADOOP-16458:
-
{code}
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
s3a://demo/user/qa/schemaevolution/tests/data/all100k
2019-07-22 14:26:49,833
org.apache.hadoop.mapred.LocatedFileStatusFetcher.getFileStatuses(LocatedFileStatusFetcher.java:155)
2019-07-22 14:26:49,833
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:247)
2019-07-22 14:26:49,834
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
2019-07-22 14:26:49,834
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:524)
2019-07-22 14:26:49,834
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:781)
2019-07-22 14:26:49,834
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
{code}
> LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
> ---
>
> Key: HADOOP-16458
> URL: https://issues.apache.org/jira/browse/HADOOP-16458
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
> Environment: S3 + S3Guard
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is
> using globStatus to find files.
> I'd say "turn s3guard on" except this appears to be the case, and the dataset
> being read is
> over 1h old.
> Which means it is harder than I'd like to blame S3 for what would sound like
> an inconsistency
> We're hampered by the number of debug level statements in the globber code
> being approximately none; there's no debugging to turn on. All we know is
> that globFiles returns null without any explanation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16458) LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
Steve Loughran created HADOOP-16458: --- Summary: LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3 Key: HADOOP-16458 URL: https://issues.apache.org/jira/browse/HADOOP-16458 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Environment: S3 + S3Guard Reporter: Steve Loughran Assignee: Steve Loughran Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is using globStatus to find files. I'd say "turn s3guard on" except this appears to be the case, and the dataset being read is over 1h old. Which means it is harder than I'd like to blame S3 for what would sound like an inconsistency We're hampered by the number of debug level statements in the globber code being approximately none; there's no debugging to turn on. All we know is that globFiles returns null without any explanation. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
Erik Krogen created HADOOP-16459: Summary: Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2 Key: HADOOP-16459 URL: https://issues.apache.org/jira/browse/HADOOP-16459 Project: Hadoop Common Issue Type: Improvement Reporter: Erik Krogen Assignee: Erik Krogen We would like to target pulling HADOOP-16266, an important operability enhancement and prerequisite for HDFS-14403, into branch-2. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 commented on issue #1151: HDDS-1853. Fix failing blockade test-cases.
nandakumar131 commented on issue #1151: HDDS-1853. Fix failing blockade test-cases. URL: https://github.com/apache/hadoop/pull/1151#issuecomment-514748228 Test failures are not related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 merged pull request #1151: HDDS-1853. Fix failing blockade test-cases.
nandakumar131 merged pull request #1151: HDDS-1853. Fix failing blockade test-cases. URL: https://github.com/apache/hadoop/pull/1151 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16433) S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren
[
https://issues.apache.org/jira/browse/HADOOP-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892073#comment-16892073
]
Hudson commented on HADOOP-16433:
-
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16978 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16978/])
HADOOP-16433. S3Guard: Filter expired entries and tombstones when (stevel: rev
7b219778e05a50e33cca75d727e62783322b7f80)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DDBPathMetadata.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestDirListingMetadata.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardTtl.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestLocalMetadataStore.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DirListingMetadata.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PathMetadata.java
> S3Guard: Filter expired entries and tombstones when listing with
> MetadataStore#listChildren
> ---
>
> Key: HADOOP-16433
> URL: https://issues.apache.org/jira/browse/HADOOP-16433
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Blocker
>
> Currently, we don't filter out entries in {{listChildren}} implementations.
> This can cause bugs and inconsistencies, so this should be fixed.
> It can lead to a status where we can't recover from the following:
> {{guarded and raw (OOB op) clients are doing ops to S3}}
> {noformat}
> Guarded: touch /
> Guarded: touch /
> Guarded: rm / {{-> tombstone in MS}}
> RAW: touch //file.ext {{-> file is hidden with a tombstone}}
> Guarded: ls / {{-> only will show up in the listing. }}
> {noformat}
> After we change the following code
> {code:java}
> final List metas = new ArrayList<>();
> for (Item item : items) {
> DDBPathMetadata meta = itemToPathMetadata(item, username);
> metas.add(meta);
> }
> {code}
> to
> {code:java}
> // handle expiry - only add not expired entries to listing.
> if (meta.getLastUpdated() == 0 ||
> !meta.isExpired(ttlTimeProvider.getMetadataTtl(),
> ttlTimeProvider.getNow())) {
> metas.add(meta);
> }
> {code}
> we will filter out expired entries from the listing, so we can recover form
> these kind of OOB ops.
> Note: we have to handle the lastUpdated == 0 case, where the lastUpdated
> field is not filled in!
> Note: this can only be fixed cleanly after HADOOP-16383 is fixed because we
> need to have the TTLtimeProvider in MS to handle this internally.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16458) LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
[
https://issues.apache.org/jira/browse/HADOOP-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892077#comment-16892077
]
Steve Loughran commented on HADOOP-16458:
-
happening when the LocatedFileStatusFetcher gets null back from glob; the fact
that the stack from the raised fault isn't propagated is something I plan to
fix too.
{code}
@Override
public Result call() throws Exception {
Result result = new Result();
FileSystem fs = path.getFileSystem(conf);
result.fs = fs;
FileStatus[] matches = fs.globStatus(path, inputFilter);
if (matches == null) { // no matches
result.addError(new IOException("Input path does not exist: " + path));
// so error raised.
} else if (matches.length == 0) {
result.addError(new IOException("Input Pattern " + path
+ " matches 0 files"));
} else {
result.matchedFileStatuses = matches;
}
return result;
}
{code}
FWIW, I'd actually tighten down the exceptions raised to an FNFE and PathIOE if
I wasn't worried about breaking things.
> LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3
> ---
>
> Key: HADOOP-16458
> URL: https://issues.apache.org/jira/browse/HADOOP-16458
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
> Environment: S3 + S3Guard
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is
> using globStatus to find files.
> I'd say "turn s3guard on" except this appears to be the case, and the dataset
> being read is
> over 1h old.
> Which means it is harder than I'd like to blame S3 for what would sound like
> an inconsistency
> We're hampered by the number of debug level statements in the globber code
> being approximately none; there's no debugging to turn on. All we know is
> that globFiles returns null without any explanation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1151: HDDS-1853. Fix failing blockade test-cases.
hadoop-yetus commented on issue #1151: HDDS-1853. Fix failing blockade test-cases. URL: https://github.com/apache/hadoop/pull/1151#issuecomment-514753413 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 8 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 38 | Maven dependency ordering for branch | | +1 | mvninstall | 594 | trunk passed | | +1 | compile | 371 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | -1 | pylint | 4 | Error running pylint. Please check pylint stderr files. | | +1 | shadedclient | 743 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 169 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | +1 | mvninstall | 552 | the patch passed | | +1 | compile | 377 | the patch passed | | +1 | javac | 377 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | -1 | pylint | 9 | Error running pylint. Please check pylint stderr files. | | +1 | pylint | 9 | There were no new pylint issues. | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 638 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 160 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 284 | hadoop-hdds in the patch passed. | | -1 | unit | 3536 | hadoop-ozone in the patch failed. | | +1 | asflicense | 46 | The patch does not generate ASF License warnings. | | | | 7811 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.hdds.scm.pipeline.TestNodeFailure | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerHandler | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1151/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1151 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs pylint | | uname | Linux e011f9310ef7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cf9ff08 | | Default Java | 1.8.0_212 | | pylint | https://builds.apache.org/job/hadoop-multibranch/job/PR-1151/1/artifact/out/branch-pylint-stderr.txt | | pylint | https://builds.apache.org/job/hadoop-multibranch/job/PR-1151/1/artifact/out/patch-pylint-stderr.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1151/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1151/1/testReport/ | | Max. process+thread count | 3951 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/fault-injection-test/network-tests hadoop-ozone/dist U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1151/1/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 pylint=1.9.2 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.a
[GitHub] [hadoop] hadoop-yetus commented on issue #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions
hadoop-yetus commented on issue #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions URL: https://github.com/apache/hadoop/pull/1150#issuecomment-514754477 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 637 | trunk passed | | +1 | compile | 400 | trunk passed | | +1 | checkstyle | 69 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 875 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 160 | trunk passed | | 0 | spotbugs | 457 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 676 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | +1 | mvninstall | 611 | the patch passed | | +1 | compile | 411 | the patch passed | | +1 | javac | 411 | the patch passed | | -0 | checkstyle | 36 | hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 661 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 147 | the patch passed | | +1 | findbugs | 659 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 324 | hadoop-hdds in the patch passed. | | -1 | unit | 2027 | hadoop-ozone in the patch failed. | | +1 | asflicense | 48 | The patch does not generate ASF License warnings. | | | | 8039 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestOzoneConfigurationFields | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.hdds.scm.pipeline.TestSCMRestart | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1150 | | JIRA Issue | HDDS-1816 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 985262bfb746 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cf9ff08 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/2/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/2/testReport/ | | Max. process+thread count | 4899 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service U: hadoop-hdds | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1150/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892087#comment-16892087
]
Eric Yang commented on HADOOP-16457:
This problem is not related to HADOOP-16354. If
dfs.datanode.kerberos.principal is set in namenode's hdfs-site.xml, then the
ServiceAuthorizationManager expects the datanode username in kerberos principal
format without checking hadoop.security.authentication == simple. The easy
solution is removing dfs.datanode.kerberos.principal config from hdfs-site.xml.
There might be enhancement in this area to make
dfs.datanode.kerberos.principal config less abrupt to simple security setting.
> Hadoop does not work without Kerberos for simple security
> -
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned HADOOP-16457:
--
Assignee: (was: Prabhu Joseph)
> Hadoop does not work without Kerberos for simple security
> -
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Priority: Major
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-16457:
---
Priority: Minor (was: Major)
> Hadoop does not work without Kerberos for simple security
> -
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Priority: Minor
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16457) Hadoop does not work with Kerberos config in hdfs-site.xml for simple security
[
https://issues.apache.org/jira/browse/HADOOP-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-16457:
---
Summary: Hadoop does not work with Kerberos config in hdfs-site.xml for
simple security (was: Hadoop does not work without Kerberos for simple
security)
> Hadoop does not work with Kerberos config in hdfs-site.xml for simple security
> --
>
> Key: HADOOP-16457
> URL: https://issues.apache.org/jira/browse/HADOOP-16457
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Eric Yang
>Priority: Minor
>
> When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
> is still setup. This prevents datanode to talk to namenode.
> Error message in namenode logs:
> {code}
> 2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
> initializers set :
> org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
> 2019-07-24 16:06:26,212 WARN
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
> org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
> accessible by dn/eyang-5.openstacklo...@example.com
> {code}
> Errors in datanode log:
> {code}
> 2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
> {code}
> The logic in HADOOP-16354 always added AuthFilter regardless security is
> enabled or not. This is incorrect. When simple security is chosen and using
> StaticUserWebFilter. AutheFilter check should not be required for datanode
> to communicate with namenode.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16245) Enabling SSL within LdapGroupsMapping can break system SSL configs
[
https://issues.apache.org/jira/browse/HADOOP-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892093#comment-16892093
]
Hadoop QA commented on HADOOP-16245:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
0s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.0 Server=19.03.0 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HADOOP-16245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975665/HADOOP-16245.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d1d93b08ec29 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cf9ff08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16419/testReport/ |
| Max. process+thread count | 1388 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16419/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Enabling SSL withi
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions
bharatviswa504 commented on a change in pull request #1150: HDDS-1816: ContainerStateMachine should limit number of pending apply transactions URL: https://github.com/apache/hadoop/pull/1150#discussion_r306971240 ## File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java ## @@ -175,6 +179,10 @@ public ContainerStateMachine(RaftGroupId gid, ContainerDispatcher dispatcher, final int numContainerOpExecutors = conf.getInt( OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_CONTAINER_OP_EXECUTORS_KEY, OzoneConfigKeys.DFS_CONTAINER_RATIS_NUM_CONTAINER_OP_EXECUTORS_DEFAULT); +int maxPendingApplyTransactions = conf.getInt( + ScmConfigKeys.DFS_CONTAINER_RATIS_STATEMACHINE_MAX_PENDING_APPLY_TRANSACTIONS, + ScmConfigKeys.DFS_CONTAINER_RATIS_STATEMACHINE_MAX_PENDING_APPLY_TRANSACTIONS_DEFAULT); +applyTransactionSemaphore = new Semaphore(maxPendingApplyTransactions); Review comment: Question: Here do we need fair setting, to follow the order? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Description: We would like to target pulling HADOOP-16266, an important operability enhancement and prerequisite for HDFS-14403, into branch-2. It's only present in trunk now so we also need to backport through the 3.x lines. was:We would like to target pulling HADOOP-16266, an important operability enhancement and prerequisite for HDFS-14403, into branch-2. > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 opened a new pull request #1152: HDDS-1817. GetKey fails with IllegalArgumentException.
nandakumar131 opened a new pull request #1152: HDDS-1817. GetKey fails with IllegalArgumentException. URL: https://github.com/apache/hadoop/pull/1152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] nandakumar131 opened a new pull request #1153: HDDS-1855. TestStorageContainerManager#testScmProcessDatanodeHeartbeat is failing.
nandakumar131 opened a new pull request #1153: HDDS-1855. TestStorageContainerManager#testScmProcessDatanodeHeartbeat is failing. URL: https://github.com/apache/hadoop/pull/1153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API.
hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-514788362 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 72 | Maven dependency ordering for branch | | +1 | mvninstall | 576 | trunk passed | | +1 | compile | 352 | trunk passed | | +1 | checkstyle | 63 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 789 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 143 | trunk passed | | 0 | spotbugs | 412 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 605 | trunk passed | | -0 | patch | 451 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | +1 | mvninstall | 533 | the patch passed | | +1 | compile | 367 | the patch passed | | +1 | cc | 367 | the patch passed | | +1 | javac | 367 | the patch passed | | -0 | checkstyle | 32 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -0 | checkstyle | 36 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 628 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 63 | hadoop-hdds generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 | findbugs | 621 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 280 | hadoop-hdds in the patch passed. | | -1 | unit | 176 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 5691 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.ratis.TestOzoneManagerRatisServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux f57c9f61d1b8 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cb69700 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/diff-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/testReport/ | | Max. process+thread count | 1287 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Attachment: HADOOP-16266-branch-3.2.000.patch HADOOP-16266-branch-3.1.000.patch > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-16459:
-
Status: Patch Available (was: Open)
Attached branch-3.2 and branch-3.1 patches, both of which had only very minor
merge conflicts (imports). {{branch-3.0}} has some non-trivial differences; I
am working on a patch now.
> Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the
> RPC layer" to branch-2
>
>
> Key: HADOOP-16459
> URL: https://issues.apache.org/jira/browse/HADOOP-16459
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16266-branch-3.1.000.patch,
> HADOOP-16266-branch-3.2.000.patch
>
>
> We would like to target pulling HADOOP-16266, an important operability
> enhancement and prerequisite for HDFS-14403, into branch-2.
> It's only present in trunk now so we also need to backport through the 3.x
> lines.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16460) ABFS: fix for Sever Name Indication (SNI)
Thomas Marquardt created HADOOP-16460: - Summary: ABFS: fix for Sever Name Indication (SNI) Key: HADOOP-16460 URL: https://issues.apache.org/jira/browse/HADOOP-16460 Project: Hadoop Common Issue Type: Bug Components: fs/azure Affects Versions: 3.1.2 Reporter: Thomas Marquardt Assignee: Vishwajeet Dusane We need to update wildfly-openssl to 1.0.7.Final in ./hadoop-project/pom.xml. ABFS depends on wildfly-openssl for secure sockets due to the performance improvements. The current wildfly-openssl does not support Server Name Indication (SNI). A fix was made in https://github.com/wildfly/wildfly-openssl/issues/59 and there is an official release of wildfly-openssl with the fix ([https://github.com/wildfly/wildfly-openssl/releases/tag/1.0.7.Final)|https://github.com/wildfly/wildfly-openssl/releases/tag/1.0.7.Final).]. The fix has been validated. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hgadre opened a new pull request #1154: [HDDS-1200] Add support for checksum verification in data scrubber
hgadre opened a new pull request #1154: [HDDS-1200] Add support for checksum verification in data scrubber URL: https://github.com/apache/hadoop/pull/1154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Attachment: HADOOP-16266-branch-3.0.000.patch > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HADOOP-16266-branch-3.0.000.patch, > HADOOP-16266-branch-3.1.000.patch, HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 merged pull request #1140: HDDS-1819. Implement S3 Commit MPU request to use Cache and DoubleBuffer.
bharatviswa504 merged pull request #1140: HDDS-1819. Implement S3 Commit MPU request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1140 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1140: HDDS-1819. Implement S3 Commit MPU request to use Cache and DoubleBuffer.
bharatviswa504 commented on issue #1140: HDDS-1819. Implement S3 Commit MPU request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1140#issuecomment-514799385 Thank You @arp7 and @xiaoyuyao for the review. I have committed this to trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 opened a new pull request #1155: HDDS-1842. Implement S3 Abort MPU request to use Cache and DoubleBuffer.
bharatviswa504 opened a new pull request #1155: HDDS-1842. Implement S3 Abort MPU request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1155 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #1155: HDDS-1842. Implement S3 Abort MPU request to use Cache and DoubleBuffer.
arp7 commented on a change in pull request #1155: HDDS-1842. Implement S3 Abort
MPU request to use Cache and DoubleBuffer.
URL: https://github.com/apache/hadoop/pull/1155#discussion_r307023290
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/multipart/S3MultipartUploadAbortResponse.java
##
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.response.s3.multipart;
+
+import org.apache.hadoop.ozone.OmUtils;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmMultipartKeyInfo;
+import org.apache.hadoop.ozone.om.response.OMClientResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.OMResponse;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.PartKeyInfo;
+import org.apache.hadoop.utils.db.BatchOperation;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Response for Multipart Abort Request.
+ */
+public class S3MultipartUploadAbortResponse extends OMClientResponse {
+
+ private String multipartKey;
+ private long timeStamp;
+ private OmMultipartKeyInfo omMultipartKeyInfo;
+
+ public S3MultipartUploadAbortResponse(String multipartKey,
+ long timeStamp,
+ OmMultipartKeyInfo omMultipartKeyInfo,
+ OMResponse omResponse) {
+super(omResponse);
+this.multipartKey = multipartKey;
+this.timeStamp = timeStamp;
+this.omMultipartKeyInfo = omMultipartKeyInfo;
+ }
+
+ @Override
+ public void addToDBBatch(OMMetadataManager omMetadataManager,
+ BatchOperation batchOperation) throws IOException {
+
+if (getOMResponse().getStatus() == OzoneManagerProtocolProtos.Status.OK) {
+
+ // Delete from openKey table and multipart info table.
+ omMetadataManager.getOpenKeyTable().deleteWithBatch(batchOperation,
+ multipartKey);
+ omMetadataManager.getMultipartInfoTable().deleteWithBatch(batchOperation,
Review comment:
Indentation looks wrong perhaps?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892188#comment-16892188 ] Erik Krogen commented on HADOOP-16459: -- Attached branch-3.0 patch. The merge difference was smaller than I expected; no logic changes were necessary. The signatures of some methods that appeared in the vicinity of changes were different, but not in a way that affected this patch. > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HADOOP-16266-branch-3.0.000.patch, > HADOOP-16266-branch-3.1.000.patch, HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892188#comment-16892188
]
Erik Krogen edited comment on HADOOP-16459 at 7/24/19 9:18 PM:
---
Attached branch-3.0 patch. The merge difference was smaller than I expected; no
logic changes were necessary. The signatures of some methods that appeared in
the vicinity of changes were different, but not in a way that affected this
patch.
One notable difference was that {{TestConsistentReadsObserver}} doesn't yet
exist in branch-3.0, so the modifications to that test were excluded. If
HDFS-14573 is finalized before this goes in, those changes should be brought
back.
was (Author: xkrogen):
Attached branch-3.0 patch. The merge difference was smaller than I expected; no
logic changes were necessary. The signatures of some methods that appeared in
the vicinity of changes were different, but not in a way that affected this
patch.
> Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the
> RPC layer" to branch-2
>
>
> Key: HADOOP-16459
> URL: https://issues.apache.org/jira/browse/HADOOP-16459
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16266-branch-3.0.000.patch,
> HADOOP-16266-branch-3.1.000.patch, HADOOP-16266-branch-3.2.000.patch
>
>
> We would like to target pulling HADOOP-16266, an important operability
> enhancement and prerequisite for HDFS-14403, into branch-2.
> It's only present in trunk now so we also need to backport through the 3.x
> lines.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1152: HDDS-1817. GetKey fails with IllegalArgumentException.
hadoop-yetus commented on issue #1152: HDDS-1817. GetKey fails with IllegalArgumentException. URL: https://github.com/apache/hadoop/pull/1152#issuecomment-514807454 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 74 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for branch | | +1 | mvninstall | 616 | trunk passed | | +1 | compile | 373 | trunk passed | | +1 | checkstyle | 77 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 877 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | trunk passed | | 0 | spotbugs | 430 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 622 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for patch | | +1 | mvninstall | 549 | the patch passed | | +1 | compile | 366 | the patch passed | | +1 | javac | 366 | the patch passed | | +1 | checkstyle | 68 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 615 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 147 | the patch passed | | +1 | findbugs | 638 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 340 | hadoop-hdds in the patch passed. | | -1 | unit | 1944 | hadoop-ozone in the patch failed. | | +1 | asflicense | 41 | The patch does not generate ASF License warnings. | | | | 7719 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1152/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1152 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 0bc089c7fcc3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cb69700 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1152/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1152/1/testReport/ | | Max. process+thread count | 4192 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm U: hadoop-hdds | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1152/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892198#comment-16892198 ] Erik Krogen commented on HADOOP-16459: -- I should note that all of the patches attached include the follow-on commit to fix the issue that was discovered after HADOOP-16266 was committed. > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HADOOP-16266-branch-3.0.000.patch, > HADOOP-16266-branch-3.1.000.patch, HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-16459: - Attachment: HADOOP-16266-branch-2.000.patch > Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the > RPC layer" to branch-2 > > > Key: HADOOP-16459 > URL: https://issues.apache.org/jira/browse/HADOOP-16459 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HADOOP-16266-branch-2.000.patch, > HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch, > HADOOP-16266-branch-3.2.000.patch > > > We would like to target pulling HADOOP-16266, an important operability > enhancement and prerequisite for HDFS-14403, into branch-2. > It's only present in trunk now so we also need to backport through the 3.x > lines. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16451) Update jackson-databind to 2.9.9.1
[ https://issues.apache.org/jira/browse/HADOOP-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892215#comment-16892215 ] Siyao Meng commented on HADOOP-16451: - Thanks [~aajisaka] [~jojochuang] for the review! > Update jackson-databind to 2.9.9.1 > -- > > Key: HADOOP-16451 > URL: https://issues.apache.org/jira/browse/HADOOP-16451 > Project: Hadoop Common > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16451.001.patch, HADOOP-16451.002.patch > > > https://nvd.nist.gov/vuln/detail/CVE-2019-12814 > CVE-2019-12814 flags 2.9.9 as vulnerable. A new version 2.9.9.1 is available. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16459) Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the RPC layer" to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-16459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892216#comment-16892216
]
Erik Krogen commented on HADOOP-16459:
--
I've put up a branch-2 patch as well. It has two additional modifications from
the branch-3.0 patch:
* A lamba for {{GenericTestUtils.waitFor()}} is replaced with an anonymous
subclass
* The {{RpcScheduler}} interface can no longer have default methods, since
branch-2 uses Java 7. Unfortunately Java 7 has no way to emulate this behavior,
so if users have a custom {{RpcScheduler}} implementation, it will break with
this change. Our [compatibility
policy|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/InterfaceClassification.html]
states that it is acceptable for us to make this breaking change in a minor
version release since this interface is marked as {{LimitedPrivate}} /
{{Evolving}}.
> Backport [HADOOP-16266] "Add more fine-grained processing time metrics to the
> RPC layer" to branch-2
>
>
> Key: HADOOP-16459
> URL: https://issues.apache.org/jira/browse/HADOOP-16459
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-16266-branch-2.000.patch,
> HADOOP-16266-branch-3.0.000.patch, HADOOP-16266-branch-3.1.000.patch,
> HADOOP-16266-branch-3.2.000.patch
>
>
> We would like to target pulling HADOOP-16266, an important operability
> enhancement and prerequisite for HDFS-14403, into branch-2.
> It's only present in trunk now so we also need to backport through the 3.x
> lines.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #1155: HDDS-1842. Implement S3 Abort MPU request to use Cache and DoubleBuffer.
bharatviswa504 commented on issue #1155: HDDS-1842. Implement S3 Abort MPU request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1155#issuecomment-514813324 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock
Gopal V created HADOOP-16461:
Summary: Regression: FileSystem cache lock parses XML within the
lock
Key: HADOOP-16461
URL: https://issues.apache.org/jira/browse/HADOOP-16461
Project: Hadoop Common
Issue Type: Bug
Components: filecache
Reporter: Gopal V
{code}
fs = createFileSystem(uri, conf);
synchronized (this) { // refetch the lock again
FileSystem oldfs = map.get(key);
if (oldfs != null) { // a file system is created while lock is releasing
fs.close(); // close the new file system
return oldfs; // return the old file system
}
// now insert the new file system into the map
if (map.isEmpty()
&& !ShutdownHookManager.get().isShutdownInProgress()) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
fs.key = key;
map.put(key, fs);
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return fs;
}
{code}
The lock now has a ShutdownHook creation, which ends up doing
{code}
HookEntry(Runnable hook, int priority) {
this(hook, priority,
getShutdownTimeout(new Configuration()),
TIME_UNIT_DEFAULT);
}
{code}
which ends up doing a "new Configuration()" within the locked section.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock
[
https://issues.apache.org/jira/browse/HADOOP-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HADOOP-16461:
-
Description:
{code}
fs = createFileSystem(uri, conf);
synchronized (this) { // refetch the lock again
FileSystem oldfs = map.get(key);
if (oldfs != null) { // a file system is created while lock is releasing
fs.close(); // close the new file system
return oldfs; // return the old file system
}
// now insert the new file system into the map
if (map.isEmpty()
&& !ShutdownHookManager.get().isShutdownInProgress()) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
fs.key = key;
map.put(key, fs);
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return fs;
}
{code}
The lock now has a ShutdownHook creation, which ends up doing
{code}
HookEntry(Runnable hook, int priority) {
this(hook, priority,
getShutdownTimeout(new Configuration()),
TIME_UNIT_DEFAULT);
}
{code}
which ends up doing a "new Configuration()" within the locked section.
This indirectly hurts the cache hit scenarios as well, since if the lock on
this is held, then the other section cannot be entered either.
{code}
I/O Setup 0 State: BLOCKED CPU usage on sample: 6ms
org.apache.hadoop.fs.FileSystem$Cache.getInternal(URI, Configuration,
FileSystem$Cache$Key) FileSystem.java:3345
org.apache.hadoop.fs.FileSystem$Cache.get(URI, Configuration)
FileSystem.java:3320
org.apache.hadoop.fs.FileSystem.get(URI, Configuration) FileSystem.java:479
org.apache.hadoop.fs.FileSystem.getLocal(Configuration) FileSystem.java:435
{code}
slowing down the RawLocalFileSystem when there are other threads creating HDFS
FileSystem objects at the same time.
was:
{code}
fs = createFileSystem(uri, conf);
synchronized (this) { // refetch the lock again
FileSystem oldfs = map.get(key);
if (oldfs != null) { // a file system is created while lock is releasing
fs.close(); // close the new file system
return oldfs; // return the old file system
}
// now insert the new file system into the map
if (map.isEmpty()
&& !ShutdownHookManager.get().isShutdownInProgress()) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
fs.key = key;
map.put(key, fs);
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return fs;
}
{code}
The lock now has a ShutdownHook creation, which ends up doing
{code}
HookEntry(Runnable hook, int priority) {
this(hook, priority,
getShutdownTimeout(new Configuration()),
TIME_UNIT_DEFAULT);
}
{code}
which ends up doing a "new Configuration()" within the locked section.
> Regression: FileSystem cache lock parses XML within the lock
>
>
> Key: HADOOP-16461
> URL: https://issues.apache.org/jira/browse/HADOOP-16461
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
>Reporter: Gopal V
>Priority: Major
>
> {code}
> fs = createFileSystem(uri, conf);
> synchronized (this) { // refetch the lock again
> FileSystem oldfs = map.get(key);
> if (oldfs != null) { // a file system is created while lock is
> releasing
> fs.close(); // close the new file system
> return oldfs; // return the old file system
> }
> // now insert the new file system into the map
> if (map.isEmpty()
> && !ShutdownHookManager.get().isShutdownInProgress()) {
> ShutdownHookManager.get().addShutdownHook(clientFinalizer,
> SHUTDOWN_HOOK_PRIORITY);
> }
> fs.key = key;
> map.put(key, fs);
> if (conf.getBoolean(
> FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
> toAutoClose.add(key);
> }
> return fs;
> }
> {code}
> The lock now has a ShutdownHook creation, which ends up doing
> {code}
> HookEntry(Runnable hook, int priority) {
> this(hook, priority,
> getShutdownTimeout(new Configuration()),
> TIME_UNIT_DEFAULT);
> }
> {code}
> which ends up doing a "new Configuration()" within the locked section.
> This indirectly hurts the cache hit scenarios as well, since if the lock on
> this is held, then the other section cannot be entered either.
> {code}
> I/O Setup 0 State: BLOCKED CPU usage on sample: 6ms
> org.apache.hadoop.fs.FileSystem$Cache.getInternal
