[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585127#comment-17585127
]
ASF GitHub Bot commented on HDFS-16659:
---
ZanderXu commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955628327
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
+EditLogOutputStream stm = qjm.startLogSegment(
+1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
+SettableFuture slowLog = SettableFuture.create();
+Mockito.doReturn(slowLog).when(spies.get(0))
+.sendEdits(eq(1L), eq(11L), eq(1), Mockito.any());
+writeTxns(stm, 1, 10);
+writeTxns(stm, 11, 10);
+writeTxns(stm, 21, 10);
+writeTxns(stm, 31, 10);
+ListeningExecutorService service = MoreExecutors.listeningDecorator(
+Executors.newSingleThreadExecutor());
+Mockito.doAnswer(invocation -> service.submit(
+() -> {
+ ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
+ EditLogFileOutputStream.writeHeader(
+ NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION,
+ new DataOutputStream(byteStream));
+ byteStream.write(createTxnData(21, 20));
+ Thread.sleep(3000);
Review Comment:
Nice suggestion, I learned a lot. Thanks, Sir.
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
+EditLogOutputStream stm = qjm.startLogSegment(
+1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
+SettableFuture slowLog = SettableFuture.create();
+Mockito.doReturn(slowLog).when(spies.get(0))
+.sendEdits(eq(1L), eq(11L), eq(1), Mockito.any());
+writeTxns(stm, 1, 10);
+writeTxns(stm, 11, 10);
+writeTxns(stm, 21, 10);
+writeTxns(stm, 31, 10);
+ListeningExecutorService service = MoreExecutors.listeningDecorator(
+Executors.newSingleThreadExecutor());
+Mockito.doAnswer(invocation -> service.submit(
+() -> {
+ ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
+ EditLogFileOutputStream.writeHeader(
+ NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION,
+ new DataOutputStream(byteStream));
+ byteStream.write(createTxnData(21, 20));
+ Thread.sleep(3000);
+ return GetJournaledEditsResponseProto.newBuilder()
+ .setTxnCount(20)
+ .setEditLog(ByteString.copyFrom(byteStream.toByteArray()))
+ .build();
+})
+).when(spies.get(1)).getJournaledEdits(21,
+QuorumJournalManager.QJM_RPC_MAX_TXNS_DEFAULT);
+
+GetJournaledEditsResponseProto responseProto = spies.get(2)
+.getJournaledEdits(21, 5000).get();
+assertEquals(20, responseProto.getTxnCount());
Review Comment:
remove it.
> JournalNode should throw NewerTxnIdException if SinceTxId is bigger than
> HighestWrittenTxId
> ---
>
> Key: HDFS-16659
> URL: http
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585125#comment-17585125
]
ASF GitHub Bot commented on HDFS-16659:
---
ZanderXu commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955627989
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
Review Comment:
JN0 contains txns 1-10 and miss 11 ~ 40.
> JournalNode should throw NewerTxnIdException if SinceTxId is bigger than
> HighestWrittenTxId
> ---
>
> Key: HDFS-16659
> URL: https://issues.apache.org/jira/browse/HDFS-16659
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JournalNode should throw `CacheMissException` if `sinceTxId` is bigger than
> `highestWrittenTxId` during handling `getJournaledEdits` rpc from NNs.
> Current logic may cause in-progress EditlogTailer cannot replay any Edits
> from JournalNodes in some corner cases, resulting in ObserverNameNode cannot
> handle requests from clients.
> Suppose there are 3 journalNodes, JN0 ~ JN1.
> * JN0 has some abnormal cases when Active Namenode is syncing 10 Edits with
> first txid 11
> * NameNode just ignore the abnormal JN0 and continue to sync Edits to Journal
> 1 and 2
> * JN0 backed to health
> * NameNode continue sync 10 Edits with first txid 21.
> * At this point, there are no Edits 11 ~ 30 in the cache of JN0
> * Observer NameNode try to select EditLogInputStream through
> `getJournaledEdits` with since txId 21
> * Journal 2 has some abnormal cases and caused a slow response
> The expected result is: Response should contain 20 Edits from txId 21 to txId
> 30 from JN1 and JN2. Because Active NameNode successfully write these Edits
> to JN1 and JN2 and failed write these edits to JN0.
> But in the current implementation, the response is [Response(0) from JN0,
> Response(10) from JN1], because there are some abnormal cases in JN2, such
> as GC, bad network, cause a slow response. So the `maxAllowedTxns` will be
> 0, NameNode will not replay any Edits.
> As above, the root case is that JournalNode should throw Miss Cache Exception
> when `sinceTxid` is more than `highestWrittenTxId`.
> And the bug code as blew:
> {code:java}
> if (sinceTxId > getHighestWrittenTxId()) {
> // Requested edits that don't exist yet; short-circuit the cache here
> metrics.rpcEmptyResponses.incr();
> return
> GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585128#comment-17585128
]
ASF GitHub Bot commented on HDFS-16659:
---
ZanderXu commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955628485
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
+EditLogOutputStream stm = qjm.startLogSegment(
+1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
+SettableFuture slowLog = SettableFuture.create();
+Mockito.doReturn(slowLog).when(spies.get(0))
+.sendEdits(eq(1L), eq(11L), eq(1), Mockito.any());
+writeTxns(stm, 1, 10);
+writeTxns(stm, 11, 10);
+writeTxns(stm, 21, 10);
+writeTxns(stm, 31, 10);
+ListeningExecutorService service = MoreExecutors.listeningDecorator(
+Executors.newSingleThreadExecutor());
+Mockito.doAnswer(invocation -> service.submit(
+() -> {
+ ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
+ EditLogFileOutputStream.writeHeader(
+ NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION,
+ new DataOutputStream(byteStream));
+ byteStream.write(createTxnData(21, 20));
+ Thread.sleep(3000);
+ return GetJournaledEditsResponseProto.newBuilder()
+ .setTxnCount(20)
+ .setEditLog(ByteString.copyFrom(byteStream.toByteArray()))
+ .build();
+})
+).when(spies.get(1)).getJournaledEdits(21,
+QuorumJournalManager.QJM_RPC_MAX_TXNS_DEFAULT);
Review Comment:
Nice suggestion. Thanks, Sir.
> JournalNode should throw NewerTxnIdException if SinceTxId is bigger than
> HighestWrittenTxId
> ---
>
> Key: HDFS-16659
> URL: https://issues.apache.org/jira/browse/HDFS-16659
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JournalNode should throw `CacheMissException` if `sinceTxId` is bigger than
> `highestWrittenTxId` during handling `getJournaledEdits` rpc from NNs.
> Current logic may cause in-progress EditlogTailer cannot replay any Edits
> from JournalNodes in some corner cases, resulting in ObserverNameNode cannot
> handle requests from clients.
> Suppose there are 3 journalNodes, JN0 ~ JN1.
> * JN0 has some abnormal cases when Active Namenode is syncing 10 Edits with
> first txid 11
> * NameNode just ignore the abnormal JN0 and continue to sync Edits to Journal
> 1 and 2
> * JN0 backed to health
> * NameNode continue sync 10 Edits with first txid 21.
> * At this point, there are no Edits 11 ~ 30 in the cache of JN0
> * Observer NameNode try to select EditLogInputStream through
> `getJournaledEdits` with since txId 21
> * Journal 2 has some abnormal cases and caused a slow response
> The expected result is: Response should contain 20 Edits from txId 21 to txId
> 30 from JN1 and JN2. Because Active NameNode successfully write these Edits
> to JN1 and JN2 and failed write these edits to JN0.
> But in the current implementation, the response is [Response(0) from JN0,
> Response(10) from JN1], because there are some abnormal cases in JN2, such
> as GC, bad network, cause a slow response. So the `maxAllowedTxns` will be
> 0, NameNode will not replay any Edits.
> As above, the root case is that JournalNode should throw Miss Cache Exception
> when `sinceTxid` is more than `highestWrittenTxId`.
> And the bug code as blew:
> {code:java}
> if (sinceTxId > getHighestWrittenTxId()) {
> // Requested edits that don't exist yet; short-circuit the cache here
> metrics.rpcEmptyResponses.incr();
> return
> GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
> }
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585126#comment-17585126
]
ASF GitHub Bot commented on HDFS-16659:
---
ZanderXu commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955628159
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
+EditLogOutputStream stm = qjm.startLogSegment(
+1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
+SettableFuture slowLog = SettableFuture.create();
+Mockito.doReturn(slowLog).when(spies.get(0))
+.sendEdits(eq(1L), eq(11L), eq(1), Mockito.any());
+writeTxns(stm, 1, 10);
+writeTxns(stm, 11, 10);
+writeTxns(stm, 21, 10);
+writeTxns(stm, 31, 10);
Review Comment:
```
// Successfully write these edits to JN0 ~ JN2
writeTxns(stm, 1, 10);
// Failed write these edits to JN0, but successfully write them to JN1 ~ JN2
writeTxns(stm, 11, 10);
// Successfully write these edits to JN1 ~ JN2
writeTxns(stm, 21, 20);
```
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
+EditLogOutputStream stm = qjm.startLogSegment(
+1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
+SettableFuture slowLog = SettableFuture.create();
+Mockito.doReturn(slowLog).when(spies.get(0))
+.sendEdits(eq(1L), eq(11L), eq(1), Mockito.any());
+writeTxns(stm, 1, 10);
+writeTxns(stm, 11, 10);
+writeTxns(stm, 21, 10);
+writeTxns(stm, 31, 10);
+ListeningExecutorService service = MoreExecutors.listeningDecorator(
+Executors.newSingleThreadExecutor());
Review Comment:
remove it
> JournalNode should throw NewerTxnIdException if SinceTxId is bigger than
> HighestWrittenTxId
> ---
>
> Key: HDFS-16659
> URL: https://issues.apache.org/jira/browse/HDFS-16659
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JournalNode should throw `CacheMissException` if `sinceTxId` is bigger than
> `highestWrittenTxId` during handling `getJournaledEdits` rpc from NNs.
> Current logic may cause in-progress EditlogTailer cannot replay any Edits
> from JournalNodes in some corner cases, resulting in ObserverNameNode cannot
> handle requests from clients.
> Suppose there are 3 journalNodes, JN0 ~ JN1.
> * JN0 has some abnormal cases when Active Namenode is syncing 10 Edits with
> first txid 11
> * NameNode just ignore the abnormal JN0 and continue to sync Edits to Journal
> 1 and 2
> * JN0 backed to health
> * NameNode continue sync 10 Edits with first txid 21.
> * At this point, there are no Edits 11 ~ 30 in the cache of JN0
> * Observer NameNode try to select EditLogInputStream through
> `getJournaledEdits` with since txId 21
> * Journal 2
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585123#comment-17585123
]
ASF GitHub Bot commented on HDFS-16659:
---
ZanderXu commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955627539
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:
##
@@ -750,10 +750,13 @@ public GetJournaledEditsResponseProto
getJournaledEdits(long sinceTxId,
"is a requirement to fetch journaled edits via RPC. Please enable " +
"it via " + DFSConfigKeys.DFS_HA_TAILEDITS_INPROGRESS_KEY);
}
-if (sinceTxId > getHighestWrittenTxId()) {
- // Requested edits that don't exist yet; short-circuit the cache here
- metrics.rpcEmptyResponses.incr();
Review Comment:
recover it.
> JournalNode should throw NewerTxnIdException if SinceTxId is bigger than
> HighestWrittenTxId
> ---
>
> Key: HDFS-16659
> URL: https://issues.apache.org/jira/browse/HDFS-16659
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JournalNode should throw `CacheMissException` if `sinceTxId` is bigger than
> `highestWrittenTxId` during handling `getJournaledEdits` rpc from NNs.
> Current logic may cause in-progress EditlogTailer cannot replay any Edits
> from JournalNodes in some corner cases, resulting in ObserverNameNode cannot
> handle requests from clients.
> Suppose there are 3 journalNodes, JN0 ~ JN1.
> * JN0 has some abnormal cases when Active Namenode is syncing 10 Edits with
> first txid 11
> * NameNode just ignore the abnormal JN0 and continue to sync Edits to Journal
> 1 and 2
> * JN0 backed to health
> * NameNode continue sync 10 Edits with first txid 21.
> * At this point, there are no Edits 11 ~ 30 in the cache of JN0
> * Observer NameNode try to select EditLogInputStream through
> `getJournaledEdits` with since txId 21
> * Journal 2 has some abnormal cases and caused a slow response
> The expected result is: Response should contain 20 Edits from txId 21 to txId
> 30 from JN1 and JN2. Because Active NameNode successfully write these Edits
> to JN1 and JN2 and failed write these edits to JN0.
> But in the current implementation, the response is [Response(0) from JN0,
> Response(10) from JN1], because there are some abnormal cases in JN2, such
> as GC, bad network, cause a slow response. So the `maxAllowedTxns` will be
> 0, NameNode will not replay any Edits.
> As above, the root case is that JournalNode should throw Miss Cache Exception
> when `sinceTxid` is more than `highestWrittenTxId`.
> And the bug code as blew:
> {code:java}
> if (sinceTxId > getHighestWrittenTxId()) {
> // Requested edits that don't exist yet; short-circuit the cache here
> metrics.rpcEmptyResponses.incr();
> return
> GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585124#comment-17585124
]
ASF GitHub Bot commented on HDFS-16659:
---
ZanderXu commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955627637
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:
##
@@ -750,10 +750,13 @@ public GetJournaledEditsResponseProto
getJournaledEdits(long sinceTxId,
"is a requirement to fetch journaled edits via RPC. Please enable " +
"it via " + DFSConfigKeys.DFS_HA_TAILEDITS_INPROGRESS_KEY);
}
-if (sinceTxId > getHighestWrittenTxId()) {
- // Requested edits that don't exist yet; short-circuit the cache here
- metrics.rpcEmptyResponses.incr();
- return
GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
+long highestTxId = getHighestWrittenTxId();
+if (sinceTxId > highestTxId) {
+ // Requested edits that don't exist yet and is newer than highestTxId.
+ throw new NewerTxnIdException(
+ "Highest txn ID available in the journal is %d, but requested txns "
+
+ "staring at %d. Maybe the journal is not healthy, just skip it.",
Review Comment:
fix it.
> JournalNode should throw NewerTxnIdException if SinceTxId is bigger than
> HighestWrittenTxId
> ---
>
> Key: HDFS-16659
> URL: https://issues.apache.org/jira/browse/HDFS-16659
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JournalNode should throw `CacheMissException` if `sinceTxId` is bigger than
> `highestWrittenTxId` during handling `getJournaledEdits` rpc from NNs.
> Current logic may cause in-progress EditlogTailer cannot replay any Edits
> from JournalNodes in some corner cases, resulting in ObserverNameNode cannot
> handle requests from clients.
> Suppose there are 3 journalNodes, JN0 ~ JN1.
> * JN0 has some abnormal cases when Active Namenode is syncing 10 Edits with
> first txid 11
> * NameNode just ignore the abnormal JN0 and continue to sync Edits to Journal
> 1 and 2
> * JN0 backed to health
> * NameNode continue sync 10 Edits with first txid 21.
> * At this point, there are no Edits 11 ~ 30 in the cache of JN0
> * Observer NameNode try to select EditLogInputStream through
> `getJournaledEdits` with since txId 21
> * Journal 2 has some abnormal cases and caused a slow response
> The expected result is: Response should contain 20 Edits from txId 21 to txId
> 30 from JN1 and JN2. Because Active NameNode successfully write these Edits
> to JN1 and JN2 and failed write these edits to JN0.
> But in the current implementation, the response is [Response(0) from JN0,
> Response(10) from JN1], because there are some abnormal cases in JN2, such
> as GC, bad network, cause a slow response. So the `maxAllowedTxns` will be
> 0, NameNode will not replay any Edits.
> As above, the root case is that JournalNode should throw Miss Cache Exception
> when `sinceTxid` is more than `highestWrittenTxId`.
> And the bug code as blew:
> {code:java}
> if (sinceTxId > getHighestWrittenTxId()) {
> // Requested edits that don't exist yet; short-circuit the cache here
> metrics.rpcEmptyResponses.incr();
> return
> GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-2139) Fast copy for HDFS.
[ https://issues.apache.org/jira/browse/HDFS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585081#comment-17585081 ] fanshilun commented on HDFS-2139: - [~ferhui] Thank you very much for your detailed explanation！ Very much looking forward to your completion of this feature！ Thanks again for your contribution!!! [~ferhui] [~xuzq_zander] > Fast copy for HDFS. > --- > > Key: HDFS-2139 > URL: https://issues.apache.org/jira/browse/HDFS-2139 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Rituraj >Priority: Major > Attachments: HDFS-2139-For-2.7.1.patch, HDFS-2139.patch, > HDFS-2139.patch, image-2022-08-11-11-48-17-994.png > > Original Estimate: 168h > Remaining Estimate: 168h > > There is a need to perform fast file copy on HDFS. The fast copy mechanism > for a file works as > follows : > 1) Query metadata for all blocks of the source file. > 2) For each block 'b' of the file, find out its datanode locations. > 3) For each block of the file, add an empty block to the namesystem for > the destination file. > 4) For each location of the block, instruct the datanode to make a local > copy of that block. > 5) Once each datanode has copied over its respective blocks, they > report to the namenode about it. > 6) Wait for all blocks to be copied and exit. > This would speed up the copying process considerably by removing top of > the rack data transfers. > Note : An extra improvement, would be to instruct the datanode to create a > hardlink of the block file if we are copying a block on the same datanode > [~xuzq_zander]Provided a design doc > https://docs.google.com/document/d/1OHdUpQmKD3TZ3xdmQsXNmlXJetn2QFPinMH31Q4BqkI/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-2139) Fast copy for HDFS.
[ https://issues.apache.org/jira/browse/HDFS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585081#comment-17585081 ] fanshilun edited comment on HDFS-2139 at 8/26/22 1:09 AM: -- [~ferhui] Thank you very much for your detailed explanation.Very much looking forward to your completion of this feature. Thanks again for your contribution! [~ferhui] [~xuzq_zander] was (Author: slfan1989): [~ferhui] Thank you very much for your detailed explanation！ Very much looking forward to your completion of this feature！ Thanks again for your contribution!!! [~ferhui] [~xuzq_zander] > Fast copy for HDFS. > --- > > Key: HDFS-2139 > URL: https://issues.apache.org/jira/browse/HDFS-2139 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Rituraj >Priority: Major > Attachments: HDFS-2139-For-2.7.1.patch, HDFS-2139.patch, > HDFS-2139.patch, image-2022-08-11-11-48-17-994.png > > Original Estimate: 168h > Remaining Estimate: 168h > > There is a need to perform fast file copy on HDFS. The fast copy mechanism > for a file works as > follows : > 1) Query metadata for all blocks of the source file. > 2) For each block 'b' of the file, find out its datanode locations. > 3) For each block of the file, add an empty block to the namesystem for > the destination file. > 4) For each location of the block, instruct the datanode to make a local > copy of that block. > 5) Once each datanode has copied over its respective blocks, they > report to the namenode about it. > 6) Wait for all blocks to be copied and exit. > This would speed up the copying process considerably by removing top of > the rack data transfers. > Note : An extra improvement, would be to instruct the datanode to create a > hardlink of the block file if we are copying a block on the same datanode > [~xuzq_zander]Provided a design doc > https://docs.google.com/document/d/1OHdUpQmKD3TZ3xdmQsXNmlXJetn2QFPinMH31Q4BqkI/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16736) Link to Boost library in libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585055#comment-17585055
]
ASF GitHub Bot commented on HDFS-16736:
---
hadoop-yetus commented on PR #4782:
URL: https://github.com/apache/hadoop/pull/4782#issuecomment-1227835831
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 25m 14s | | trunk passed |
| +1 :green_heart: | compile | 4m 15s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 4m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 55m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 3m 59s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 3m 59s | | the patch passed |
| +1 :green_heart: | javac | 3m 59s | | the patch passed |
| +1 :green_heart: | compile | 3m 54s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 54s | | the patch passed |
| +1 :green_heart: | javac | 3m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 48s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 121m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4782 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux 7c7b012e0424 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6424242b0f03c7f155959e7c1a6200a8e5d1c5a9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/testReport/ |
| Max. process+thread count | 751 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Link to Boost library in libhdfspp
> --
>
> Key: HDFS-16736
> URL: https://issues.apache.org/jira/browse/HDFS-16736
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows 10
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
>
> The compilation of HDFS Native Client fails on Windows 10 due to the
> following error -
> {code}
> [exec]
> "H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main
[jira] [Commented] (HDFS-16736) Link to Boost library in libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585024#comment-17585024
]
ASF GitHub Bot commented on HDFS-16736:
---
hadoop-yetus commented on PR #4782:
URL: https://github.com/apache/hadoop/pull/4782#issuecomment-1227754055
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 27m 12s | | trunk passed |
| +1 :green_heart: | compile | 3m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 58m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 3m 24s | | the patch passed |
| +1 :green_heart: | cc | 3m 24s | | the patch passed |
| +1 :green_heart: | javac | 3m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 33m 0s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 152m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4782 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux 89cebebd6bc1 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6424242b0f03c7f155959e7c1a6200a8e5d1c5a9 |
| Default Java | Debian-11.0.16+8-post-Debian-1deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/console |
| versions | git=2.20.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Link to Boost library in libhdfspp
> --
>
> Key: HDFS-16736
> URL: https://issues.apache.org/jira/browse/HDFS-16736
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows 10
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
>
> The compilation of HDFS Native Client fails on Windows 10 due to the
> following error -
> {code}
> [exec]
> "H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj"
> (default target) (105) ->
> [exec] rpc.lib(rpc_engine.obj) : error LNK2019: unresolved external symbol
> "__declspec(dllimport) public: __cdecl
> boost::gregorian::greg_month::greg_month(unsigned short)"
> (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) referenced in function
> "private: static class boost::posix_time::ptime __cdecl
> boost::date_time::microsec_clock boost::posix_time::ptime>::create_time(struct tm * (__cdecl*)(__int64 const
> *,struct tm *))"
> (?create_time@?$microsec_clock@Vptime@posix_time@boost@@@date_time@boost@@CA?AVptime@posix_time@3@P6APEAUtm@@PEB_JPEAU6@@Z@Z)
>
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj]
> [exec] rpc.lib(request.obj) : err
[jira] [Commented] (HDFS-16689) Standby NameNode crashes when transitioning to Active with in-progress tailer
[
https://issues.apache.org/jira/browse/HDFS-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585021#comment-17585021
]
ASF GitHub Bot commented on HDFS-16689:
---
hadoop-yetus commented on PR #4744:
URL: https://github.com/apache/hadoop/pull/4744#issuecomment-1227743672
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 57s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4744/6/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 253 unchanged
- 0 fixed = 254 total (was 253) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 12s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 347m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4744/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4744 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e86124f1fc66 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7f72f67ecf733794ff2df3797c16d03a29c146a9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4744/6/testReport/ |
| Max. process+thread count | 3377 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
[jira] [Commented] (HDFS-16689) Standby NameNode crashes when transitioning to Active with in-progress tailer
[
https://issues.apache.org/jira/browse/HDFS-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584987#comment-17584987
]
ASF GitHub Bot commented on HDFS-16689:
---
xkrogen commented on code in PR #4744:
URL: https://github.com/apache/hadoop/pull/4744#discussion_r955294947
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestNNWithQJM.java:
##
@@ -197,10 +197,9 @@ public void testMismatchedNNIsRejected() throws Exception {
.manageNameDfsDirs(false).format(false).checkExitOnShutdown(false)
.build();
fail("New NN with different namespace should have been rejected");
-} catch (ExitException ee) {
+} catch (IOException ie) {
GenericTestUtils.assertExceptionContains(
- "Unable to start log segment 1: too few journals", ee);
- assertTrue("Didn't terminate properly ", ExitUtil.terminateCalled());
+ "recoverUnfinalizedSegments failed for too many journals", ie);
Review Comment:
I wonder if we should modify the caller to catch the `IOException` and
rethrow as `ExitException` to match previous behavior?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java:
##
@@ -1657,16 +1657,11 @@ synchronized void logEdit(final int length, final
byte[] data) {
/**
* Run recovery on all journals to recover any unclosed segments
*/
- synchronized void recoverUnclosedStreams() {
+ synchronized void recoverUnclosedStreams() throws IOException {
Preconditions.checkState(
state == State.BETWEEN_LOG_SEGMENTS,
"May not recover segments - wrong state: %s", state);
-try {
- journalSet.recoverUnfinalizedSegments();
-} catch (IOException ex) {
- // All journals have failed, it is handled in logSync.
- // TODO: are we sure this is OK?
-}
+journalSet.recoverUnfinalizedSegments();
Review Comment:
This looks right to me as we've been discussing, but I would appreciate
another pair of eyes on it to see if I'm missing anything. @omalley can you
take a look? (see discussion above on why we're making this change)
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java:
##
@@ -299,33 +299,28 @@ public void catchupDuringFailover() throws IOException {
// Important to do tailing as the login user, in case the shared
// edits storage is implemented by a JournalManager that depends
// on security credentials to access the logs (eg QuorumJournalManager).
-SecurityUtil.doAsLoginUser(new PrivilegedExceptionAction() {
- @Override
- public Void run() throws Exception {
-long editsTailed = 0;
-// Fully tail the journal to the end
-do {
- long startTime = timer.monotonicNow();
- try {
-NameNode.getNameNodeMetrics().addEditLogTailInterval(
-startTime - lastLoadTimeMs);
-// It is already under the name system lock and the checkpointer
-// thread is already stopped. No need to acquire any other lock.
-editsTailed = doTailEdits();
- } catch (InterruptedException e) {
-throw new IOException(e);
- } finally {
-NameNode.getNameNodeMetrics().addEditLogTailTime(
-timer.monotonicNow() - startTime);
- }
-} while(editsTailed > 0);
-return null;
+SecurityUtil.doAsLoginUser((PrivilegedExceptionAction) () -> {
+ long startTime = timer.monotonicNow();
+ try {
+NameNode.getNameNodeMetrics().addEditLogTailInterval((startTime -
lastLoadTimeMs));
Review Comment:
why did you remove the do-while loop?
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/SpyQJournalUtil.java:
##
@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.qjournal.client;
+
+import org.apache.hadoop.conf.Configuration;
+import
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProt
[jira] [Commented] (HDFS-16747) some hdfs unit tests failing
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584982#comment-17584982 ] Samrat Deb commented on HDFS-16747: --- I will try to debug and find the root cause of these failures(wheather related to enviroment or intermetient failure ). Though i am using AL2 linux on m5x. machine. Let me try and debug more on these update once there is something i find concrete. Till then i will keep the ticket open ! [~ste...@apache.org] > some hdfs unit tests failing > > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584972#comment-17584972
]
ASF GitHub Bot commented on HDFS-16659:
---
xkrogen commented on code in PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#discussion_r955266389
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:
##
@@ -750,10 +750,13 @@ public GetJournaledEditsResponseProto
getJournaledEdits(long sinceTxId,
"is a requirement to fetch journaled edits via RPC. Please enable " +
"it via " + DFSConfigKeys.DFS_HA_TAILEDITS_INPROGRESS_KEY);
}
-if (sinceTxId > getHighestWrittenTxId()) {
- // Requested edits that don't exist yet; short-circuit the cache here
- metrics.rpcEmptyResponses.incr();
- return
GetJournaledEditsResponseProto.newBuilder().setTxnCount(0).build();
+long highestTxId = getHighestWrittenTxId();
+if (sinceTxId > highestTxId) {
+ // Requested edits that don't exist yet and is newer than highestTxId.
+ throw new NewerTxnIdException(
+ "Highest txn ID available in the journal is %d, but requested txns "
+
+ "staring at %d. Maybe the journal is not healthy, just skip it.",
Review Comment:
typo: `staring` -> `starting`
We expect this to happen more frequently when the JNs are healthy than when
they are unhealthy, right? Maybe we should remove the part about "Maybe the
journal is not healthy" ?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java:
##
@@ -413,5 +413,4 @@ long getCacheMissAmount() {
}
}
-
Review Comment:
Can you undo this whitespace change please
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -53,6 +63,8 @@
import org.apache.hadoop.hdfs.qjournal.MiniJournalCluster;
import org.apache.hadoop.hdfs.qjournal.QJMTestUtil;
import
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.SegmentStateProto;
+import
+
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.GetJournaledEditsResponseProto;
Review Comment:
leave on 1 line, we ignore line length limitation for imports
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
Review Comment:
This test passes even if I revert your production changes from `Journal`,
can you check again?
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1101,6 +1113,59 @@ public void testSelectViaRpcTwoJNsError() throws
Exception {
}
}
+ /**
+ * Test selecting EditLogInputStream after some journalNode jitter.
+ * And the corner case as below:
+ * 1. Journal 0 has some abnormal cases when journaling Edits with start
txId 11.
+ * 2. NameNode just ignore the abnormal journal 0 and continue to write
Edits to Journal 1 and 2.
+ * 3. Journal 0 backed to health.
+ * 4. Observer NameNode try to select EditLogInputStream vis PRC with start
txId 21.
+ * 5. Journal 1 has some abnormal cases caused slow response.
+ *
+ * And the expected selecting result is: Response should contain 20 Edits
from txId 21 to txId 40.
+ * Because there is no Edits from id 21 to 40 in the cache of JournalNode0.
+ */
+ @Test
+ public void testSelectViaRpcAfterJNJitter() throws Exception {
+EditLogOutputStream stm = qjm.startLogSegment(
+1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
+SettableFuture slowLog = SettableFuture.create();
+Mockito.doReturn(slowLog).when(spies.get(0))
+.sendEdits(eq(1L), eq(11L), eq(1), Mockito.any());
+writeTxns(stm, 1, 10);
+writeTxns(stm, 11, 10);
+writeTxns(stm, 21, 10);
+writeTxns(stm, 31, 10);
+ListeningExecutorService service = MoreExecutors.listeningDecorator(
[jira] [Commented] (HDFS-16736) Link to Boost library in libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584971#comment-17584971
]
ASF GitHub Bot commented on HDFS-16736:
---
hadoop-yetus commented on PR #4782:
URL: https://github.com/apache/hadoop/pull/4782#issuecomment-1227615166
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 30s | | trunk passed |
| +1 :green_heart: | compile | 4m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 3m 55s | | the patch passed |
| +1 :green_heart: | cc | 3m 55s | | the patch passed |
| +1 :green_heart: | javac | 3m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 40m 7s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 120m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4782 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux 67b93cf047f2 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6424242b0f03c7f155959e7c1a6200a8e5d1c5a9 |
| Default Java | Red Hat, Inc.-1.8.0_312-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/testReport/ |
| Max. process+thread count | 626 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Link to Boost library in libhdfspp
> --
>
> Key: HDFS-16736
> URL: https://issues.apache.org/jira/browse/HDFS-16736
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows 10
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
>
> The compilation of HDFS Native Client fails on Windows 10 due to the
> following error -
> {code}
> [exec]
> "H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj"
> (default target) (105) ->
> [exec] rpc.lib(rpc_engine.obj) : error LNK2019: unresolved external symbol
> "__declspec(dllimport) public: __cdecl
> boost::gregorian::greg_month::greg_month(unsigned short)"
> (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) referenced in function
> "private: static class boost::posix_time::ptime __cdecl
> boost::date_time::microsec_clock boost::posix_time::ptime>::create_time(struct tm * (__cdecl*)(__int64 const
> *,struct tm *))"
> (?create_time@?$microsec_clock@Vptime@posix_time@boost@@@date_time@boost@@CA?AVptime@posix_time@3@P6APEAUtm@@PEB_JPEAU6@@Z@Z)
>
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\t
[jira] [Resolved] (HDFS-16732) [SBN READ] Avoid get location from observer when the block report is delayed.
[
https://issues.apache.org/jira/browse/HDFS-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen resolved HDFS-16732.
Fix Version/s: 3.4.0
3.3.9
Resolution: Fixed
Merged PR 4756 to trunk and branch-3.3. Thanks [~zhengchenyu]!
> [SBN READ] Avoid get location from observer when the block report is delayed.
> -
>
> Key: HDFS-16732
> URL: https://issues.apache.org/jira/browse/HDFS-16732
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Hive on tez application fail occasionally after observer is enable, log show
> below.
> {code:java}
> 2022-08-18 15:22:06,914 [ERROR] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Vertex Input: namenodeinfo_stg initializer failed,
> vertex=vertex_1660618571916_4839_1_00 [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:329)
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
> at
> com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
> at
> com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:133)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:80)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapred.FileInputFormat.identifyHosts(FileInputFormat.java:748)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplitHostsAndCachedHosts(FileInputFormat.java:714)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:378)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:306)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:408)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:279)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:270)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:254)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> ... 4 more {code}
> As describe in MAPREDUCE-7082, when the block is missing, then will throw
> this exception, but my cluster had no missing block.
> In this example, I found getListing return location information. When block
> report of observer is delayed, will return the block without location.
> HDFS-13924 is introduce to solve this problem, but only consider
> getBlockLocations.
> In observer node, all method which may return location should check whether
> locations is empty or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs
[jira] [Commented] (HDFS-16732) [SBN READ] Avoid get location from observer when the block report is delayed.
[
https://issues.apache.org/jira/browse/HDFS-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584962#comment-17584962
]
ASF GitHub Bot commented on HDFS-16732:
---
xkrogen commented on PR #4756:
URL: https://github.com/apache/hadoop/pull/4756#issuecomment-1227581149
New test is very clean :) Many thanks for the contribution @zhengchenyu !
I've merged this to trunk and branch-3.3.
> [SBN READ] Avoid get location from observer when the block report is delayed.
> -
>
> Key: HDFS-16732
> URL: https://issues.apache.org/jira/browse/HDFS-16732
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Hive on tez application fail occasionally after observer is enable, log show
> below.
> {code:java}
> 2022-08-18 15:22:06,914 [ERROR] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Vertex Input: namenodeinfo_stg initializer failed,
> vertex=vertex_1660618571916_4839_1_00 [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:329)
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
> at
> com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
> at
> com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:133)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:80)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapred.FileInputFormat.identifyHosts(FileInputFormat.java:748)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplitHostsAndCachedHosts(FileInputFormat.java:714)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:378)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:306)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:408)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:279)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:270)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:254)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> ... 4 more {code}
> As describe in MAPREDUCE-7082, when the block is missing, then will throw
> this exception, but my cluster had no missing block.
> In this example, I found getListing return location information. When block
> report of observer is delayed, will return the block without location.
> HDFS-13924 is introduce to solve this problem, but only consider
> getBlockLocations.
> In observer node, all method which may return location should check whether
> locations is empty or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HDFS-16732) [SBN READ] Avoid get location from observer when the block report is delayed.
[
https://issues.apache.org/jira/browse/HDFS-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584955#comment-17584955
]
ASF GitHub Bot commented on HDFS-16732:
---
xkrogen merged PR #4756:
URL: https://github.com/apache/hadoop/pull/4756
> [SBN READ] Avoid get location from observer when the block report is delayed.
> -
>
> Key: HDFS-16732
> URL: https://issues.apache.org/jira/browse/HDFS-16732
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Labels: pull-request-available
>
> Hive on tez application fail occasionally after observer is enable, log show
> below.
> {code:java}
> 2022-08-18 15:22:06,914 [ERROR] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Vertex Input: namenodeinfo_stg initializer failed,
> vertex=vertex_1660618571916_4839_1_00 [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:329)
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056)
> at
> com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)
> at
> com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.afterRanInterruptibly(TrustedListenableFutureTask.java:133)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:80)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapred.FileInputFormat.identifyHosts(FileInputFormat.java:748)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplitHostsAndCachedHosts(FileInputFormat.java:714)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:378)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:306)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:408)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:279)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:270)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:254)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
> ... 4 more {code}
> As describe in MAPREDUCE-7082, when the block is missing, then will throw
> this exception, but my cluster had no missing block.
> In this example, I found getListing return location information. When block
> report of observer is delayed, will return the block without location.
> HDFS-13924 is introduce to solve this problem, but only consider
> getBlockLocations.
> In observer node, all method which may return location should check whether
> locations is empty or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16747) some hdfs unit tests failing
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584949#comment-17584949 ] Steve Loughran commented on HDFS-16747: --- ok, changed the title. probably something related to your setup, or they are intermittent ones which have surfaced. afraid you are going to have to start debugging those tests to work out what they expect and why your deployment doesn't work > some hdfs unit tests failing > > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16747) List of unit test failing in 3.3.3
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16747: -- Affects Version/s: 3.4.0 (was: 3.3.3) > List of unit test failing in 3.3.3 > -- > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16747) some hdfs unit tests failing
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16747: -- Summary: some hdfs unit tests failing (was: List of unit test failing in 3.3.3) > some hdfs unit tests failing > > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16736) Link to Boost library in libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584920#comment-17584920
]
ASF GitHub Bot commented on HDFS-16736:
---
hadoop-yetus commented on PR #4782:
URL: https://github.com/apache/hadoop/pull/4782#issuecomment-1227493834
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 17s | | trunk passed |
| +1 :green_heart: | compile | 4m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 65m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 3m 55s | | the patch passed |
| +1 :green_heart: | cc | 3m 55s | | the patch passed |
| +1 :green_heart: | javac | 3m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 33m 11s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 127m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4782 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets |
| uname | Linux f7a960c17bae 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6424242b0f03c7f155959e7c1a6200a8e5d1c5a9 |
| Default Java | Red Hat, Inc.-1.8.0_342-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4782/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Link to Boost library in libhdfspp
> --
>
> Key: HDFS-16736
> URL: https://issues.apache.org/jira/browse/HDFS-16736
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
> Environment: Windows 10
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
>
> The compilation of HDFS Native Client fails on Windows 10 due to the
> following error -
> {code}
> [exec]
> "H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\tests\logging_test.vcxproj"
> (default target) (105) ->
> [exec] rpc.lib(rpc_engine.obj) : error LNK2019: unresolved external symbol
> "__declspec(dllimport) public: __cdecl
> boost::gregorian::greg_month::greg_month(unsigned short)"
> (__imp_??0greg_month@gregorian@boost@@QEAA@G@Z) referenced in function
> "private: static class boost::posix_time::ptime __cdecl
> boost::date_time::microsec_clock boost::posix_time::ptime>::create_time(struct tm * (__cdecl*)(__int64 const
> *,struct tm *))"
> (?create_time@?$microsec_clock@Vptime@posix_time@boost@@@date_time@boost@@CA?AVptime@posix_time@3@P6APEAUtm@@PEB_JPEAU6@@Z@Z)
>
> [H:\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfspp\te
[jira] [Commented] (HDFS-16706) ViewFS doc points to wrong mount table name
[
https://issues.apache.org/jira/browse/HDFS-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584918#comment-17584918
]
ASF GitHub Bot commented on HDFS-16706:
---
PrabhuJoseph commented on PR #4803:
URL: https://github.com/apache/hadoop/pull/4803#issuecomment-1227474902
There are few other places needs to be changed.
1. Appendix: A Mount Table Configuration Example
>>> there is a definition of the mount table "ClusterX"
>>> fs.viewfs.mounttable.ClusterX
2. ClusterY's root is merged with the root filesystem
fs.viewfs.mounttable.ClusterY.linkMergeSlash
> ViewFS doc points to wrong mount table name
> ---
>
> Key: HDFS-16706
> URL: https://issues.apache.org/jira/browse/HDFS-16706
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Prabhu Joseph
>Assignee: Samrat Deb
>Priority: Minor
> Labels: pull-request-available
>
> ViewFS Doc -
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ViewFs.html
> specifies the view name as *clusterX* where as mount table name as
> *ClusterX*. This will lead to error "ls: ViewFs: Cannot initialize: Empty
> Mount table in config for viewfs://clusterX/"
> {code}
>
> fs.defaultFS
> viewfs://clusterX
>
>
> fs.viewfs.mounttable.ClusterX.link./data
> hdfs://nn1-clusterx.example.com:8020/data
>
> {code}
> The mountable name also has to be same as view name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16732) [SBN READ] Avoid get location from observer when the block report is delayed.
[
https://issues.apache.org/jira/browse/HDFS-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584779#comment-17584779
]
ASF GitHub Bot commented on HDFS-16732:
---
hadoop-yetus commented on PR #4756:
URL: https://github.com/apache/hadoop/pull/4756#issuecomment-1227113493
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 27s | | trunk passed |
| +1 :green_heart: | compile | 1m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 38s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 248m 54s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 369m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4756 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bcf6b87183ae 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bae43f8dba103156707a6d077f13e32dc8a5b551 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/4/testReport/ |
| Max. process+thread count | 3545 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4756/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-16689) Standby NameNode crashes when transitioning to Active with in-progress tailer
[
https://issues.apache.org/jira/browse/HDFS-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584738#comment-17584738
]
ASF GitHub Bot commented on HDFS-16689:
---
hadoop-yetus commented on PR #4744:
URL: https://github.com/apache/hadoop/pull/4744#issuecomment-1227031102
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 3s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 252m 8s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4744/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 362m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.qjournal.TestNNWithQJM |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4744/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4744 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4188a87c263a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7d8b49790968f54bbc943fb3697624a9007d2126 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4744/5/testReport/ |
| Max. process+thread count | 2937 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
[jira] [Commented] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584734#comment-17584734
]
ASF GitHub Bot commented on HDFS-13522:
---
hadoop-yetus commented on PR #4127:
URL: https://github.com/apache/hadoop/pull/4127#issuecomment-1227023297
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 44s | | trunk passed |
| +1 :green_heart: | compile | 25m 58s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 5m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 6s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 6m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 2s | | the patch passed |
| +1 :green_heart: | compile | 24m 29s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| -1 :x: | javac | 24m 29s |
[/results-compile-javac-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4127/31/artifact/out/results-compile-javac-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 1 new + 2847 unchanged - 1
fixed = 2848 total (was 2848) |
| +1 :green_heart: | compile | 21m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 21m 38s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4127/31/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643
unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 18s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4127/31/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 4 new + 157 unchanged - 9 fixed = 161 total (was
166) |
| +1 :green_heart: | mvnsite | 6m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 48s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 5m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 38s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 46s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 342m 14s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 35m 17s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 27s | | The patch does not
generate ASF License warnings. |
| |
[jira] [Commented] (HDFS-16747) List of unit test failing in 3.3.3
[
https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584733#comment-17584733
]
Samrat Deb commented on HDFS-16747:
---
Hi [~ste...@apache.org] !
i was running all these test in cluster, encountered errors .
The branch is synced with `trunk` in remote
Here is some stack trace of it
{code:java}
2022-08-25 14:55:34,561 WARN Server - Log4j
[/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/testValidHttpFSAccess-5/conf/httpfs-log4j.properties]
configuration file not found, using default configuration from classpath
2022-08-25 14:55:34,561 INFO Server -
++
2022-08-25 14:55:34,561 INFO Server - Server [httpfs] starting
2022-08-25 14:55:34,561 INFO Server - Built information:
2022-08-25 14:55:34,561 INFO Server - Version : 3.4.0-SNAPSHOT
2022-08-25 14:55:34,561 INFO Server - Source Repository : REPO NOT AVAIL
2022-08-25 14:55:34,561 INFO Server - Source Revision : REVISION NOT
AVAIL
2022-08-25 14:55:34,561 INFO Server - Built by : dbsamrat
2022-08-25 14:55:34,561 INFO Server - Built timestamp :
2022-08-25T09:03:19+
2022-08-25 14:55:34,561 INFO Server - Runtime information:
2022-08-25 14:55:34,561 INFO Server - Home dir:
/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/testValidHttpFSAccess-5
2022-08-25 14:55:34,561 INFO Server - Config dir:
/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/testValidHttpFSAccess-5/conf
2022-08-25 14:55:34,561 INFO Server - Log dir:
/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/testValidHttpFSAccess-5/log
2022-08-25 14:55:34,561 INFO Server - Temp dir:
/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/testValidHttpFSAccess-5/temp
2022-08-25 14:55:34,563 INFO FileSystemAccessService - Using FileSystemAccess
JARs version [3.4.0-SNAPSHOT]
2022-08-25 14:55:34,567 INFO FileSystemAccessService - Using FileSystemAccess
simple/pseudo authentication, principal [dbsamrat]
2022-08-25 14:55:34,572 INFO Server - Services initialized
2022-08-25 14:55:34,572 INFO Server - Server [httpfs] started!, status [NORMAL]
2022-08-25 14:55:34,572 INFO HttpFSServerWebApp - Connects to Namenode
[file:///]
2022-08-25 14:55:34,572 INFO HttpFSServerWebApp - Initializing
HttpFSServerMetrics
2022-08-25 14:55:34,577 INFO MetricsSystemImpl - HttpFSServer metrics system
started (again)
2022-08-25 14:55:34,577 INFO JvmPauseMonitor - Starting JVM pause monitor
2022-08-25 14:55:34,579 WARN HttpServer2 - Overwriting configuration for
key='signature.secret.file' with
value='${httpfs.config.dir}/httpfs-signature.secret' previous
value='/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-dir/testValidHttpFSAccess-5/conf/secret'
2022-08-25 14:55:34,579 WARN HttpServer2 - Overwriting configuration for
key='type' with value='simple' previous value='kerberos'
Aug 25, 2022 2:55:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.hadoop.fs.http.server
org.apache.hadoop.lib.wsrs
Aug 25, 2022 2:55:34 PM com.sun.jersey.api.core.ScanningResourceConfig
logClasses
INFO: Root resource classes found:
class org.apache.hadoop.fs.http.server.HttpFSServer
Aug 25, 2022 2:55:34 PM com.sun.jersey.api.core.ScanningResourceConfig
logClasses
INFO: Provider classes found:
class org.apache.hadoop.lib.wsrs.JSONProvider
class org.apache.hadoop.fs.http.server.HttpFSExceptionProvider
class org.apache.hadoop.lib.wsrs.JSONMapProvider
class org.apache.hadoop.fs.http.server.HttpFSParametersProvider
Aug 25, 2022 2:55:34 PM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20
PM'
2022-08-25 14:55:34,689 WARN WebAppContext - Failed startup of context
o.e.j.w.WebAppContext@743c6ce4{/webhdfs,file:///Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-classes/webapp/,UNAVAILABLE}{/Users/dbsamrat/code/hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/test-classes/webapp}
java.lang.RuntimeException: Could not read HttpFS signature secret file:
${httpfs.config.dir}/httpfs-signature.secret
at
org.apache.hadoop.fs.http.server.HttpFSAuthenticationFilter.getConfiguration(HttpFSAuthenticationFilter.java:105)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:160)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:181)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140)
a
[jira] [Commented] (HDFS-16747) List of unit test failing in 3.3.3
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584729#comment-17584729 ] Steve Loughran commented on HDFS-16747: --- samrat, are these tests still failing on hadoop trunk/branch-3.3? as those are the branches where all fixed will go in. 3.3.3/3.3.4 are already released > List of unit test failing in 3.3.3 > -- > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.3.3 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16738) Invalid CallerContext caused NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584724#comment-17584724
]
ASF GitHub Bot commented on HDFS-16738:
---
ferhui commented on PR #4791:
URL: https://github.com/apache/hadoop/pull/4791#issuecomment-122742
@ZanderXu Thanks for your contribution. @ashutoshcipher @ayushtkn Thanks for
your reviews! Merged!
> Invalid CallerContext caused NullPointerException
> -
>
> Key: HDFS-16738
> URL: https://issues.apache.org/jira/browse/HDFS-16738
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> {code:java}
> 2022-08-23 11:58:03,258 [FSEditLogAsync] ERROR namenode.FSEditLog
> (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: write op failed
> for required journal (JournalAndStream(mgr=QJM to [127.0.0.1:55779,
> 127.0.0.1:55781, 127.0.0.1:55783], stream=QuorumOutputStream starting at txid
> 1))
> java.lang.NullPointerException
> at org.apache.hadoop.io.UTF8.set(UTF8.java:97)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.writeString(FSImageSerialization.java:361)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$AddCloseOp.writeFields(FSEditLogOp.java:586)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4986)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:158)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:61)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.write(QuorumOutputStream.java:50)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:462)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$200(JournalSet.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:458)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:496)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:311)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:253)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16738) Invalid CallerContext caused NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei resolved HDFS-16738.
Fix Version/s: 3.4.0
Resolution: Fixed
> Invalid CallerContext caused NullPointerException
> -
>
> Key: HDFS-16738
> URL: https://issues.apache.org/jira/browse/HDFS-16738
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> {code:java}
> 2022-08-23 11:58:03,258 [FSEditLogAsync] ERROR namenode.FSEditLog
> (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: write op failed
> for required journal (JournalAndStream(mgr=QJM to [127.0.0.1:55779,
> 127.0.0.1:55781, 127.0.0.1:55783], stream=QuorumOutputStream starting at txid
> 1))
> java.lang.NullPointerException
> at org.apache.hadoop.io.UTF8.set(UTF8.java:97)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.writeString(FSImageSerialization.java:361)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$AddCloseOp.writeFields(FSEditLogOp.java:586)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4986)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:158)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:61)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.write(QuorumOutputStream.java:50)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:462)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$200(JournalSet.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:458)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:496)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:311)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:253)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16738) Invalid CallerContext caused NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584722#comment-17584722
]
ASF GitHub Bot commented on HDFS-16738:
---
ferhui merged PR #4791:
URL: https://github.com/apache/hadoop/pull/4791
> Invalid CallerContext caused NullPointerException
> -
>
> Key: HDFS-16738
> URL: https://issues.apache.org/jira/browse/HDFS-16738
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Critical
> Labels: pull-request-available
>
> {code:java}
> 2022-08-23 11:58:03,258 [FSEditLogAsync] ERROR namenode.FSEditLog
> (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: write op failed
> for required journal (JournalAndStream(mgr=QJM to [127.0.0.1:55779,
> 127.0.0.1:55781, 127.0.0.1:55783], stream=QuorumOutputStream starting at txid
> 1))
> java.lang.NullPointerException
> at org.apache.hadoop.io.UTF8.set(UTF8.java:97)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.writeString(FSImageSerialization.java:361)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$AddCloseOp.writeFields(FSEditLogOp.java:586)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4986)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:158)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:61)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.write(QuorumOutputStream.java:50)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:462)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$200(JournalSet.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:458)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:496)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:311)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:253)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16743) TestSnapshot.testDeletionSnapshotMtime failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] groot updated HDFS-16743: - Summary: TestSnapshot.testDeletionSnapshotMtime failing intermittently (was: TestSnapshot.testDeletionSnapshotMtime Failing) > TestSnapshot.testDeletionSnapshotMtime failing intermittently > - > > Key: HDFS-16743 > URL: https://issues.apache.org/jira/browse/HDFS-16743 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.3.3 >Reporter: groot >Assignee: groot >Priority: Major > > TestSnapshot.testDeletionSnapshotMtime failing with > > |java.lang.AssertionError: Values should be different. Actual: 1661272865709| > |at org.junit.Assert.fail(Assert.java:89)| > |at org.junit.Assert.failEquals(Assert.java:187)| > |at org.junit.Assert.assertNotEquals(Assert.java:201)| > |at org.junit.Assert.assertNotEquals(Assert.java:213)| > |at > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot.testDeletionSnapshotMtime(TestSnapshot.java:508)| > |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| > |at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| > |at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| > |at java.lang.reflect.Method.invoke(Method.java:498)| > |at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| > |at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| > |at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| > |at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| > |at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)| > |at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)| > |at java.util.concurrent.FutureTask.run(FutureTask.java:266)| > |at java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16744) TestReencryption.testCancelFutureThenReencrypt failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] groot updated HDFS-16744: - Summary: TestReencryption.testCancelFutureThenReencrypt failing intermittently (was: TestReencryption.testCancelFutureThenReencrypt Failing) > TestReencryption.testCancelFutureThenReencrypt failing intermittently > - > > Key: HDFS-16744 > URL: https://issues.apache.org/jira/browse/HDFS-16744 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.3.3 >Reporter: groot >Assignee: groot >Priority: Major > > TestReencryption.testCancelFutureThenReencrypt Failing with > > |java.lang.AssertionError: expected:<10> but was:<0>| > |at org.junit.Assert.fail(Assert.java:89)| > |at org.junit.Assert.failNotEquals(Assert.java:835)| > |at org.junit.Assert.assertEquals(Assert.java:647)| > |at org.junit.Assert.assertEquals(Assert.java:633)| > |at > org.apache.hadoop.hdfs.server.namenode.TestReencryption.testCancelFutureThenReencrypt(TestReencryption.java:1576)| > |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| > |at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| > |at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| > |at java.lang.reflect.Method.invoke(Method.java:498)| > |at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| > |at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| > |at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| > |at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| > |at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)| > |at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)| > |at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)| > |at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)| > |at java.util.concurrent.FutureTask.run(FutureTask.java:266)| > |at java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16745) TestReconstructStripedFile.testNNSendsErasureCodingTasks failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] groot updated HDFS-16745: - Summary: TestReconstructStripedFile.testNNSendsErasureCodingTasks failing intermittently (was: TestReconstructStripedFile.testNNSendsErasureCodingTasks failing) > TestReconstructStripedFile.testNNSendsErasureCodingTasks failing > intermittently > --- > > Key: HDFS-16745 > URL: https://issues.apache.org/jira/browse/HDFS-16745 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.3.3 >Reporter: groot >Assignee: groot >Priority: Major > > TestReconstructStripedFile.testNNSendsErasureCodingTasks failing with > > |java.lang.AssertionError: Found 3 timeout pending reconstruction tasks| > |at org.junit.Assert.fail(Assert.java:89)| > |at org.junit.Assert.assertTrue(Assert.java:42)| > |at > org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:529)| > |at > org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:485)| > |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| > |at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| > |at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| > |at java.lang.reflect.Method.invoke(Method.java:498)| > |at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| > |at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| > |at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| > |at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| > |at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)| > |at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)| > |at java.util.concurrent.FutureTask.run(FutureTask.java:266)| > |at java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16747) List of unit test failing in 3.3.3
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb updated HDFS-16747: -- Description: Effected Test class * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos * org.apache.hadoop.hdfs.TestMultipleNNPortQOP * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP * > List of unit test failing in 3.3.3 > -- > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.3.3 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16747) List of unit test failing in 3.3.3
Samrat Deb created HDFS-16747: - Summary: List of unit test failing in 3.3.3 Key: HDFS-16747 URL: https://issues.apache.org/jira/browse/HDFS-16747 Project: Hadoop HDFS Issue Type: Test Affects Versions: 3.3.3 Reporter: Samrat Deb Assignee: Samrat Deb -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16746) TestHttpFSWithKerberos.testInvalidadHttpFSAccess failing
groot created HDFS-16746: Summary: TestHttpFSWithKerberos.testInvalidadHttpFSAccess failing Key: HDFS-16746 URL: https://issues.apache.org/jira/browse/HDFS-16746 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.3.3 Reporter: groot Assignee: groot TestHttpFSWithKerberos.testInvalidadHttpFSAccess failing with |java.lang.AssertionError: expected:<503> but was:<401>| |at org.junit.Assert.fail(Assert.java:89)| |at org.junit.Assert.failNotEquals(Assert.java:835)| |at org.junit.Assert.assertEquals(Assert.java:647)| |at org.junit.Assert.assertEquals(Assert.java:633)| |at org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos.testInvalidadHttpFSAccess(TestHttpFSWithKerberos.java:144)| |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| |at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| |at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| |at java.lang.reflect.Method.invoke(Method.java:498)| |at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| |at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| |at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| |at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| |at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)| |at org.apache.hadoop.test.TestHdfsHelper$HdfsStatement.evaluate(TestHdfsHelper.java:95)| |at org.apache.hadoop.test.TestDirHelper$1.evaluate(TestDirHelper.java:106)| |at org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:42)| |at org.apache.hadoop.test.TestJettyHelper$1.evaluate(TestJettyHelper.java:74)| |at org.apache.hadoop.test.TestDirHelper$1.evaluate(TestDirHelper.java:106)| |at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)| |at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)| |at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)| |at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)| |at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)| |at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)| |at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)| |at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)| |at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)| |at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)| |at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)| |at org.junit.runners.ParentRunner.run(ParentRunner.java:413)| |at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)| |at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)| |at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)| |at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)| |at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)| |at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)| |at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)| |at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)| | | -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16659) JournalNode should throw NewerTxnIdException if SinceTxId is bigger than HighestWrittenTxId
[
https://issues.apache.org/jira/browse/HDFS-16659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584712#comment-17584712
]
ASF GitHub Bot commented on HDFS-16659:
---
hadoop-yetus commented on PR #4560:
URL: https://github.com/apache/hadoop/pull/4560#issuecomment-1226954004
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 42s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 237m 56s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
generate ASF License warnings. |
| | | 347m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4560/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4560 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b6ef6639ef5c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08e0dae7ddf43a22e699a34b21dbe7e32755969c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4560/3/testReport/ |
| Max. process+thread count | 3832 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4560/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[jira] [Created] (HDFS-16745) TestReconstructStripedFile.testNNSendsErasureCodingTasks failing
groot created HDFS-16745: Summary: TestReconstructStripedFile.testNNSendsErasureCodingTasks failing Key: HDFS-16745 URL: https://issues.apache.org/jira/browse/HDFS-16745 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.3.3 Reporter: groot Assignee: groot TestReconstructStripedFile.testNNSendsErasureCodingTasks failing with |java.lang.AssertionError: Found 3 timeout pending reconstruction tasks| |at org.junit.Assert.fail(Assert.java:89)| |at org.junit.Assert.assertTrue(Assert.java:42)| |at org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:529)| |at org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:485)| |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| |at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| |at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| |at java.lang.reflect.Method.invoke(Method.java:498)| |at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| |at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| |at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| |at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| |at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)| |at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)| |at java.util.concurrent.FutureTask.run(FutureTask.java:266)| |at java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16744) TestReencryption.testCancelFutureThenReencrypt Failing
groot created HDFS-16744: Summary: TestReencryption.testCancelFutureThenReencrypt Failing Key: HDFS-16744 URL: https://issues.apache.org/jira/browse/HDFS-16744 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.3.3 Reporter: groot Assignee: groot TestReencryption.testCancelFutureThenReencrypt Failing with |java.lang.AssertionError: expected:<10> but was:<0>| |at org.junit.Assert.fail(Assert.java:89)| |at org.junit.Assert.failNotEquals(Assert.java:835)| |at org.junit.Assert.assertEquals(Assert.java:647)| |at org.junit.Assert.assertEquals(Assert.java:633)| |at org.apache.hadoop.hdfs.server.namenode.TestReencryption.testCancelFutureThenReencrypt(TestReencryption.java:1576)| |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| |at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| |at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| |at java.lang.reflect.Method.invoke(Method.java:498)| |at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| |at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| |at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| |at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| |at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)| |at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)| |at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)| |at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)| |at java.util.concurrent.FutureTask.run(FutureTask.java:266)| |at java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16743) TestSnapshot.testDeletionSnapshotMtime Failing
groot created HDFS-16743: Summary: TestSnapshot.testDeletionSnapshotMtime Failing Key: HDFS-16743 URL: https://issues.apache.org/jira/browse/HDFS-16743 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.3.3 Reporter: groot Assignee: groot TestSnapshot.testDeletionSnapshotMtime failing with |java.lang.AssertionError: Values should be different. Actual: 1661272865709| |at org.junit.Assert.fail(Assert.java:89)| |at org.junit.Assert.failEquals(Assert.java:187)| |at org.junit.Assert.assertNotEquals(Assert.java:201)| |at org.junit.Assert.assertNotEquals(Assert.java:213)| |at org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot.testDeletionSnapshotMtime(TestSnapshot.java:508)| |at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)| |at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)| |at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)| |at java.lang.reflect.Method.invoke(Method.java:498)| |at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)| |at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)| |at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)| |at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)| |at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)| |at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)| |at java.util.concurrent.FutureTask.run(FutureTask.java:266)| |at java.lang.Thread.run(Thread.java:748)| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16738) Invalid CallerContext caused NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584698#comment-17584698
]
ASF GitHub Bot commented on HDFS-16738:
---
hadoop-yetus commented on PR #4791:
URL: https://github.com/apache/hadoop/pull/4791#issuecomment-1226910694
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 243m 2s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 351m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4791/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4791 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b800edc5244c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 171ce66d6d2530eabbfb121cfc06cd36e7d638b9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4791/3/testReport/ |
| Max. process+thread count | 3170 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4791/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584675#comment-17584675
]
ASF GitHub Bot commented on HDFS-13522:
---
hadoop-yetus commented on PR #4311:
URL: https://github.com/apache/hadoop/pull/4311#issuecomment-1226876683
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 51s | | trunk passed |
| +1 :green_heart: | compile | 23m 16s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 53s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 5m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 11m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | | the patch passed |
| +1 :green_heart: | compile | 22m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 39s | | the patch passed |
| +1 :green_heart: | compile | 20m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 20m 34s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 2643
unchanged - 1 fixed = 2644 total (was 2644) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 4s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 4 new + 14 unchanged - 9 fixed = 18 total (was
23) |
| +1 :green_heart: | mvnsite | 6m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 26s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 5m 44s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 12m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 26s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 56s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 236m 53s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 23m 49s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 534m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4311/25/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2b526f6d7f64 4.15.
