[jira] [Assigned] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned HDFS-15677: --- Assignee: Masatake Iwasaki > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15676) TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240554#comment-17240554 ] Masatake Iwasaki commented on HDFS-15676: - The test fails only if I run both testNamenodeMetrics and testGetCachedDatanodeReport. Running {{-Dtest=testNamenodeMetrics#testNamenodeMetrics}} alone has no issue. Since the TestRouterRpcMultiDestination reuses a mini cluster in the test cases, {{testNamenodeMetrics}} could be fixed by addressing {{testGetCachedDatanodeReport}} issue. {noformat} $ for i in `seq 100` ; do echo $i && mvn test -DignoreTestFailure=false -Dtest=TestRouterRpcMultiDestination#testNamenodeMetrics,TestRouterRpcMultiDestination#testGetCachedDatanodeReport || break ; done ... [ERROR] Failures: [ERROR] TestRouterRpcMultiDestination>TestRouterRpc.testNamenodeMetrics:1682 expected:<12> but was:<11> [ERROR] Errors: [ERROR] TestRouterRpcMultiDestination>TestRouterRpc.testGetCachedDatanodeReport:1829 » Timeout [INFO] [ERROR] Tests run: 2, Failures: 1, Errors: 1, Skipped: 0 {noformat} > TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk > > > Key: HDFS-15676 > URL: https://issues.apache.org/jira/browse/HDFS-15676 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in testNamenodeMetrics -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5328) MiniDFSCluster times out at startup
[ https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240597#comment-17240597 ] Matteo Minardi commented on HDFS-5328: -- Any news on this? I get the same error. In the code I still see: {code:java} public void waitClusterUp() throws IOException { int i = 0; if (numDataNodes > 0) { while (!isClusterUp()) { try { LOG.warn("Waiting for the Mini HDFS Cluster to start..."); Thread.sleep(1000); } catch (InterruptedException e) { } if (++i > 10) { final String msg = "Timed out waiting for Mini HDFS Cluster to start"; LOG.error(msg); throw new IOException(msg); } } } } {code} With 10 hardcoded > MiniDFSCluster times out at startup > --- > > Key: HDFS-5328 > URL: https://issues.apache.org/jira/browse/HDFS-5328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1 >Reporter: Alejandro Abdelnur >Priority: Major > Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch > > > It seems MiniDFSCluster startup got slower lately and we are observing the > following intermittent failure on startup in our jenkins boxes: > {code} > java.io.IOException: Timed out waiting for Mini HDFS Cluster to start > at > org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511) > {code} > By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout > (10 1 sec waits). > There are two things we should look at it, make the timeout configurable, and > more important, see what has changed to make the startup slower. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-5328) MiniDFSCluster times out at startup
[ https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240597#comment-17240597 ] Matteo Minardi edited comment on HDFS-5328 at 11/30/20, 9:27 AM: - Any news on this? I get the same error. In the code I still see: {code:java} public void waitClusterUp() throws IOException { int i = 0; if (numDataNodes > 0) { while (!isClusterUp()) { try { LOG.warn("Waiting for the Mini HDFS Cluster to start..."); Thread.sleep(1000); } catch (InterruptedException e) { } if (++i > 10) { final String msg = "Timed out waiting for Mini HDFS Cluster to start"; LOG.error(msg); throw new IOException(msg); } } } } {code} With 10 hardcoded [~rvesse] [~tucu00] was (Author: mino181295): Any news on this? I get the same error. In the code I still see: {code:java} public void waitClusterUp() throws IOException { int i = 0; if (numDataNodes > 0) { while (!isClusterUp()) { try { LOG.warn("Waiting for the Mini HDFS Cluster to start..."); Thread.sleep(1000); } catch (InterruptedException e) { } if (++i > 10) { final String msg = "Timed out waiting for Mini HDFS Cluster to start"; LOG.error(msg); throw new IOException(msg); } } } } {code} With 10 hardcoded [~rvesse] [~tucu00] > MiniDFSCluster times out at startup > --- > > Key: HDFS-5328 > URL: https://issues.apache.org/jira/browse/HDFS-5328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1 >Reporter: Alejandro Abdelnur >Priority: Major > Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch > > > It seems MiniDFSCluster startup got slower lately and we are observing the > following intermittent failure on startup in our jenkins boxes: > {code} > java.io.IOException: Timed out waiting for Mini HDFS Cluster to start > at > org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511) > {code} > By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout > (10 1 sec waits). > There are two things we should look at it, make the timeout configurable, and > more important, see what has changed to make the startup slower. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-5328) MiniDFSCluster times out at startup
[ https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240597#comment-17240597 ] Matteo Minardi edited comment on HDFS-5328 at 11/30/20, 9:27 AM: - Any news on this? I get the same error. In the code I still see: {code:java} public void waitClusterUp() throws IOException { int i = 0; if (numDataNodes > 0) { while (!isClusterUp()) { try { LOG.warn("Waiting for the Mini HDFS Cluster to start..."); Thread.sleep(1000); } catch (InterruptedException e) { } if (++i > 10) { final String msg = "Timed out waiting for Mini HDFS Cluster to start"; LOG.error(msg); throw new IOException(msg); } } } } {code} With 10 hardcoded [~rvesse] [~tucu00] was (Author: mino181295): Any news on this? I get the same error. In the code I still see: {code:java} public void waitClusterUp() throws IOException { int i = 0; if (numDataNodes > 0) { while (!isClusterUp()) { try { LOG.warn("Waiting for the Mini HDFS Cluster to start..."); Thread.sleep(1000); } catch (InterruptedException e) { } if (++i > 10) { final String msg = "Timed out waiting for Mini HDFS Cluster to start"; LOG.error(msg); throw new IOException(msg); } } } } {code} With 10 hardcoded > MiniDFSCluster times out at startup > --- > > Key: HDFS-5328 > URL: https://issues.apache.org/jira/browse/HDFS-5328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1 >Reporter: Alejandro Abdelnur >Priority: Major > Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch > > > It seems MiniDFSCluster startup got slower lately and we are observing the > following intermittent failure on startup in our jenkins boxes: > {code} > java.io.IOException: Timed out waiting for Mini HDFS Cluster to start > at > org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511) > {code} > By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout > (10 1 sec waits). > There are two things we should look at it, make the timeout configurable, and > more important, see what has changed to make the startup slower. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5328) MiniDFSCluster times out at startup
[ https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240599#comment-17240599 ] Hadoop QA commented on HDFS-5328: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color} | {color:red}{color} | {color:red} HDFS-5328 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-5328 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12782266/HDFS-5328.002.patch | | Console output | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/322/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. > MiniDFSCluster times out at startup > --- > > Key: HDFS-5328 > URL: https://issues.apache.org/jira/browse/HDFS-5328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1 >Reporter: Alejandro Abdelnur >Priority: Major > Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch > > > It seems MiniDFSCluster startup got slower lately and we are observing the > following intermittent failure on startup in our jenkins boxes: > {code} > java.io.IOException: Timed out waiting for Mini HDFS Cluster to start > at > org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511) > {code} > By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout > (10 1 sec waits). > There are two things we should look at it, make the timeout configurable, and > more important, see what has changed to make the startup slower. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15163) hdfs-2.10.0-webapps-secondary-status.html miss moment.js
[ https://issues.apache.org/jira/browse/HDFS-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 resolved HDFS-15163. --- Resolution: Fixed > hdfs-2.10.0-webapps-secondary-status.html miss moment.js > > > Key: HDFS-15163 > URL: https://issues.apache.org/jira/browse/HDFS-15163 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.10.0 >Reporter: 谢波 >Priority: Minor > Attachments: 微信截图_20200212183444.png > > Original Estimate: 96h > Remaining Estimate: 96h > > hdfs-2.10.0-webapps-secondary-status.html miss moment.js > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14437) Exception happened when rollEditLog expects empty EditsDoubleBuffer.bufCurrent but not
[ https://issues.apache.org/jira/browse/HDFS-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240663#comment-17240663 ] Xiaoqiao He commented on HDFS-14437: [~Sushma_28] Actually, I also meet the similar case (NameNode crash) long before, it is solved after patch HDFS-13112. For this issue, there is no strict proof and we have not reached the final agreement. Welcome more discussion here. Thanks. > Exception happened when rollEditLog expects empty > EditsDoubleBuffer.bufCurrent but not > - > > Key: HDFS-14437 > URL: https://issues.apache.org/jira/browse/HDFS-14437 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode, qjm >Reporter: angerszhu >Priority: Major > Attachments: HDFS-14437.reproduction.patch, > HDFS-14437.reproductionwithlog.patch, screenshot-1.png > > > For the problem mentioned in https://issues.apache.org/jira/browse/HDFS-10943 > , I have sort the process of write and flush EditLog and some important > function, I found the in the class FSEditLog class, the close() function > will call such process like below: > > {code:java} > waitForSyncToFinish(); > endCurrentLogSegment(true);{code} > since we have gain the object lock in the function close(), so when > waitForSyncToFish() method return, it mean all logSync job has done and all > data in bufReady has been flushed out, and since current thread has the lock > of this object, when call endCurrentLogSegment(), no other thread will gain > the lock so they can't write new editlog into currentBuf. > But when we don't call waitForSyncToFish() before endCurrentLogSegment(), > there may be some autoScheduled logSync()'s flush process is doing, since > this process don't need > synchronization since it has mention in the comment of logSync() method : > > {code:java} > /** > * Sync all modifications done by this thread. > * > * The internal concurrency design of this class is as follows: > * - Log items are written synchronized into an in-memory buffer, > * and each assigned a transaction ID. > * - When a thread (client) would like to sync all of its edits, logSync() > * uses a ThreadLocal transaction ID to determine what edit number must > * be synced to. > * - The isSyncRunning volatile boolean tracks whether a sync is currently > * under progress. > * > * The data is double-buffered within each edit log implementation so that > * in-memory writing can occur in parallel with the on-disk writing. > * > * Each sync occurs in three steps: > * 1. synchronized, it swaps the double buffer and sets the isSyncRunning > * flag. > * 2. unsynchronized, it flushes the data to storage > * 3. synchronized, it resets the flag and notifies anyone waiting on the > * sync. > * > * The lack of synchronization on step 2 allows other threads to continue > * to write into the memory buffer while the sync is in progress. > * Because this step is unsynchronized, actions that need to avoid > * concurrency with sync() should be synchronized and also call > * waitForSyncToFinish() before assuming they are running alone. > */ > public void logSync() { > long syncStart = 0; > // Fetch the transactionId of this thread. > long mytxid = myTransactionId.get().txid; > > boolean sync = false; > try { > EditLogOutputStream logStream = null; > synchronized (this) { > try { > printStatistics(false); > // if somebody is already syncing, then wait > while (mytxid > synctxid && isSyncRunning) { > try { > wait(1000); > } catch (InterruptedException ie) { > } > } > // > // If this transaction was already flushed, then nothing to do > // > if (mytxid <= synctxid) { > numTransactionsBatchedInSync++; > if (metrics != null) { > // Metrics is non-null only when used inside name node > metrics.incrTransactionsBatchedInSync(); > } > return; > } > > // now, this thread will do the sync > syncStart = txid; > isSyncRunning = true; > sync = true; > // swap buffers > try { > if (journalSet.isEmpty()) { > throw new IOException("No journals available to flush"); > } > editLogStream.setReadyToFlush(); > } catch (IOException e) { > final String msg = > "Could not sync enough journals to persistent storage " + > "due to " + e.getMessage() + ". " + > "Unsynced transactions: " + (txid - synctxid); > LOG.fatal(msg, new Exception()); > synchronized(journalSetLock) { >
[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=517873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517873 ] ASF GitHub Bot logged work on HDFS-15677: - Author: ASF GitHub Bot Created on: 30/Nov/20 11:49 Start Date: 30/Nov/20 11:49 Worklog Time Spent: 10m Work Description: iwasakims opened a new pull request #2503: URL: https://github.com/apache/hadoop/pull/2503 https://issues.apache.org/jira/browse/HDFS-15677 Since heartbeatExpireInterval and assertion timeout of DN removal has the same value (5000 ms), the test fails intermittently. Updating the heartbeatExpireInterval and assertion timeout worked for me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 517873) Remaining Estimate: 0h Time Spent: 10m > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15677: -- Labels: pull-request-available (was: ) > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15677: Status: Patch Available (was: Open) > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=517929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517929 ] ASF GitHub Bot logged work on HDFS-15677: - Author: ASF GitHub Bot Created on: 30/Nov/20 13:18 Start Date: 30/Nov/20 13:18 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2503: URL: https://github.com/apache/hadoop/pull/2503#issuecomment-735779663 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 30s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 34s | | trunk passed | | +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 27s | | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | | trunk passed | | +1 :green_heart: | shadedclient | 16m 20s | | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +0 :ok: | spotbugs | 1m 15s | | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 14s | | trunk passed | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | | the patch passed | | +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javac | 0m 33s | | the patch passed | | +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | javac | 0m 29s | | the patch passed | | +1 :green_heart: | checkstyle | 0m 17s | | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 8s | | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | findbugs | 1m 14s | | the patch passed | _ Other Tests _ | | +1 :green_heart: | unit | 8m 34s | | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. | | | | 86m 58s | | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2503/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2503 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f39134cbd1e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 44910b53f38 | | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2503/1/testReport/ | | Max. process+thread count | 3038 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2503/1/console | | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 | | Powered by | Apache Yetus 0.13.0-SNAPSHOT https:
[jira] [Commented] (HDFS-15680) Disable Broken Azure Junits
[ https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240838#comment-17240838 ] Xiaoqiao He commented on HDFS-15680: Thanks [~ayushtkn] for your informations. I backport HADOOP-17325 to branch-3.2 and run failed unit tests mentioned above at local, Both of them are passed. Thanks. > Disable Broken Azure Junits > --- > > Key: HDFS-15680 > URL: https://issues.apache.org/jira/browse/HDFS-15680 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs/azure >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 20m > Remaining Estimate: 0h > > There are 6 test classes have been failing on Yetus for several months. > They contributed to more than 41 failing tests which makes reviewing Yetus > reports every a pain in the neck. Another point is to save the resources and > avoiding utilization of ports, memory, and CPU. > Over the last month, there was some effort to bring the Yetus back to a > stable state. However, there is no progress in addressing Azure failures. > Generally, I do not like to disable failing tests, but for this specific > case, I do not assume that it makes any sense to have 41 failing tests from > one module for several months. Whenever someone finds that those tests are > useful, then they can re-enable the tests on Yetus *_After_* the test is > fixed. > Following a PR, I have to review that my patch does not cause any failures > (include changing error messages in existing tests). A thorough review takes > a considerable amount of time browsing the nightly builds and Github reports. > So, please consider how much time is being spent to review those stack trace > over the last months. > Finally, this is one of the reasons developers tend to ignore the reports, > because it would take too much time to review; and by default, the errors are > considered irrelevant. > CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn] > {code:bash} > hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked >hadoop.fs.azure.TestNativeAzureFileSystemMocked >hadoop.fs.azure.TestBlobMetadata >hadoop.fs.azure.TestNativeAzureFileSystemConcurrency >hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck >hadoop.fs.azure.TestNativeAzureFileSystemContractMocked >hadoop.fs.azure.TestWasbFsck >hadoop.fs.azure.TestOutOfBandAzureBlobOperations > {code} > {code:bash} > org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata > org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata > org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata > org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder > org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMo
[jira] [Work logged] (HDFS-15695) NN should not let the balancer run in safemode
[ https://issues.apache.org/jira/browse/HDFS-15695?focusedWorklogId=518052&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518052 ] ASF GitHub Bot logged work on HDFS-15695: - Author: ASF GitHub Bot Created on: 30/Nov/20 17:51 Start Date: 30/Nov/20 17:51 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2489: URL: https://github.com/apache/hadoop/pull/2489#issuecomment-735941256 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 1m 9s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 34m 51s | | trunk passed | | +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | compile | 1m 10s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 49s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 19s | | trunk passed | | +1 :green_heart: | shadedclient | 19m 11s | | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +0 :ok: | spotbugs | 3m 16s | | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 13s | | trunk passed | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 13s | | the patch passed | | +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javac | 1m 13s | | the patch passed | | +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | javac | 1m 7s | | the patch passed | | +1 :green_heart: | checkstyle | 0m 42s | | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 56 unchanged - 10 fixed = 56 total (was 66) | | +1 :green_heart: | mvnsite | 1m 14s | | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 56s | | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | findbugs | 3m 15s | | the patch passed | _ Other Tests _ | | -1 :x: | unit | 116m 25s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. | | | | 211m 45s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2489 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2393e22da497 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 44910b53f38 | | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibra
[jira] [Work logged] (HDFS-15694) Avoid calling UpdateHeartBeatState inside DataNodeDescriptor
[ https://issues.apache.org/jira/browse/HDFS-15694?focusedWorklogId=518064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518064 ] ASF GitHub Bot logged work on HDFS-15694: - Author: ASF GitHub Bot Created on: 30/Nov/20 18:13 Start Date: 30/Nov/20 18:13 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2487: URL: https://github.com/apache/hadoop/pull/2487#issuecomment-735953319 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 36s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 50s | | trunk passed | | +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 50s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 19s | | trunk passed | | +1 :green_heart: | shadedclient | 17m 32s | | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +0 :ok: | spotbugs | 3m 7s | | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 5s | | trunk passed | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 9s | | the patch passed | | +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javac | 1m 13s | | the patch passed | | +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | javac | 1m 4s | | the patch passed | | +1 :green_heart: | checkstyle | 0m 39s | | the patch passed | | +1 :green_heart: | mvnsite | 1m 9s | | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 37s | | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | | +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | +1 :green_heart: | findbugs | 3m 4s | | the patch passed | _ Other Tests _ | | -1 :x: | unit | 96m 8s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2487/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. | | | | 185m 6s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestRollingUpgrade | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2487/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2487 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1dcdcf3af76c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 44910b53f38 | | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2487/3/testReport/ | | Max. process+thread count | 3969 (vs
[jira] [Created] (HDFS-15700) Protobuf Mesage Incompatibility Detector
junwen yang created HDFS-15700: -- Summary: Protobuf Mesage Incompatibility Detector Key: HDFS-15700 URL: https://issues.apache.org/jira/browse/HDFS-15700 Project: Hadoop HDFS Issue Type: Bug Reporter: junwen yang Attachments: hdfs_proto_incompatibility.txt Regarding the issue -HDFS-9788- caused by the incompatibility of protobuf message, we have created a static checker which keeps track of the proto file change, and detect potential incompatibility: # Add/delete required field, which is the case reported in HBASE-25238. # The tag number of a field has been changed, as described in HDFS-9788. Also, the [protobuf guidelines |https://developers.google.com/protocol-buffers/docs/proto]suggests _each field in the message definition has a *unique number*. These numbers are used to identify your fields in the [message binary format|https://developers.google.com/protocol-buffers/docs/encoding], and should not be changed once your message type is in use_. # A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines in [protobuf official website|https://developers.google.com/protocol-buffers/docs/proto], _*Required Is Forever* You should be very careful about marking fields as {{required}}. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field - old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead._ We have applied our checker on the frequently maintained HDFS versions: rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0, rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0, rel/release-3.3.0, and we found 6 potential problems as attached. The checker is also able to detect the issue describe in -HDFS-9788.- The results reported by our checker got confirmation from developers in HBASE, which can be found here HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15701) Add resolveMountPath API in FileSystem
Uma Maheswara Rao G created HDFS-15701: -- Summary: Add resolveMountPath API in FileSystem Key: HDFS-15701 URL: https://issues.apache.org/jira/browse/HDFS-15701 Project: Hadoop HDFS Issue Type: Sub-task Components: fs, ViewHDFS Affects Versions: 3.4.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently FileSystem has an API resolvePath. To know where the path has mounted, the applications can use that API as the retuned path is from actual target path in the case of mount file systems like ViewFS, ViewFSOverloadScheme or ViewDistributedFileSystem. However, resolvePath does more than what is needed by Apps when they want to know where the path has mounted. It's because resolvePath internally calls "getFileStatus". This additional call is unnecessary when apps just want to where the path mounted. Since we have mounted filesystems available in FS, I think it's good to add resolveMountPath API, which will just do the following. If the fs is mounted fs, then it will resolve it's mount tables and return the actual target path. If the fs is non mounted, then it will simply return the same path. Currently Applications like Hive, Ranger using resolvePath API. ( this is forcing to do additional RPC internally) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=518168&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518168 ] ASF GitHub Bot logged work on HDFS-15677: - Author: ASF GitHub Bot Created on: 30/Nov/20 23:07 Start Date: 30/Nov/20 23:07 Worklog Time Spent: 10m Work Description: iwasakims commented on pull request #2503: URL: https://github.com/apache/hadoop/pull/2503#issuecomment-736109294 Thanks, @goiri. I merged this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518168) Time Spent: 40m (was: 0.5h) > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=518167&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518167 ] ASF GitHub Bot logged work on HDFS-15677: - Author: ASF GitHub Bot Created on: 30/Nov/20 23:07 Start Date: 30/Nov/20 23:07 Worklog Time Spent: 10m Work Description: iwasakims merged pull request #2503: URL: https://github.com/apache/hadoop/pull/2503 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518167) Time Spent: 0.5h (was: 20m) > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-15677: Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk > > > Key: HDFS-15677 > URL: https://issues.apache.org/jira/browse/HDFS-15677 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in > testGetCachedDatanodeReport -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15676) TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk
[ https://issues.apache.org/jira/browse/HDFS-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved HDFS-15676. - Resolution: Duplicate I'm closing this as fixed by HDFS-15677. I will reopen this if the failure of testNameNodeMetrics alone rise again. > TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk > > > Key: HDFS-15676 > URL: https://issues.apache.org/jira/browse/HDFS-15676 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ahmed Hussein >Assignee: Masatake Iwasaki >Priority: Major > > qbt report (Nov 8, 2020, 11:28 AM) shows failures in testNamenodeMetrics -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-14904) Option to let Balancer prefer top used nodes in each iteration
[ https://issues.apache.org/jira/browse/HDFS-14904?focusedWorklogId=518182&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518182 ] ASF GitHub Bot logged work on HDFS-14904: - Author: ASF GitHub Bot Created on: 30/Nov/20 23:56 Start Date: 30/Nov/20 23:56 Worklog Time Spent: 10m Work Description: Jing9 commented on a change in pull request #2483: URL: https://github.com/apache/hadoop/pull/2483#discussion_r532981333 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java ## @@ -435,6 +444,22 @@ private long init(List reports) { return Math.max(overLoadedBytes, underLoadedBytes); } + private void sortOverUtilizedNodes() { +LOG.info("Sorting over-utilized nodes by capacity" + +" to bring down top used datanode capacity faster"); + +if (overUtilized instanceof List) { + List list = (List) overUtilized; + list.sort( + (Source source1, Source source2) -> Review comment: Do we need to consider StorageType (which is associated with Source)? E.g., suppose a DN has 2 storage types, one of which is highly utilized and the other is just above average. Do we want to first schedule the movement for the highly-utilized storage type on this node? ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java ## @@ -435,6 +444,22 @@ private long init(List reports) { return Math.max(overLoadedBytes, underLoadedBytes); } + private void sortOverUtilizedNodes() { +LOG.info("Sorting over-utilized nodes by capacity" + +" to bring down top used datanode capacity faster"); + +if (overUtilized instanceof List) { Review comment: Do we need this "if" statement? Maybe use a Preconditions instead? ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java ## @@ -199,7 +199,10 @@ + "\tWhether to run the balancer during an ongoing HDFS upgrade." + "This is usually not desired since it will not affect used space " + "on over-utilized machines." - + "\n\t[-asService]\tRun as a long running service."; + + "\n\t[-asService]\tRun as a long running service." + + "\n\t[-sortTopNodes]" + + "\tSort over-utilized nodes by capacity to" + + " bring down top used datanode faster."; Review comment: How about describe the parameter option as: "sort datanodes based on the utilization so that highly utilized datanodes get scheduled first"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518182) Time Spent: 40m (was: 0.5h) > Option to let Balancer prefer top used nodes in each iteration > -- > > Key: HDFS-14904 > URL: https://issues.apache.org/jira/browse/HDFS-14904 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Normally the most important purpose for HDFS balancer is to reduce the top > used node to prevent datanode usage from being too high. > Currently, balancer almost randomly picks nodes as sources regardless of > usage, which makes it slow to bring down the top used datanodes in the > cluster, when there are less underutilized nodes in the cluster (consider > expansion). > We can add an option to prefer top used nodes first in each iteration, as > suggested in HDFS-14894 . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15702) Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission
Masatake Iwasaki created HDFS-15702: --- Summary: Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission Key: HDFS-15702 URL: https://issues.apache.org/jira/browse/HDFS-15702 Project: Hadoop HDFS Issue Type: Improvement Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki {noformat} java.lang.AssertionError: expected: but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.hadoop.hdfs.TestDecommission.testAllocAndIBRWhileDecommission(TestDecommission.java:1025) {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15624) Fix the SetQuotaByStorageTypeOp problem after updating hadoop
[ https://issues.apache.org/jira/browse/HDFS-15624?focusedWorklogId=518250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518250 ] ASF GitHub Bot logged work on HDFS-15624: - Author: ASF GitHub Bot Created on: 01/Dec/20 02:53 Start Date: 01/Dec/20 02:53 Worklog Time Spent: 10m Work Description: huangtianhua commented on a change in pull request #2377: URL: https://github.com/apache/hadoop/pull/2377#discussion_r533039713 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java ## @@ -3569,6 +3569,9 @@ void setQuota(String src, long nsQuota, long ssQuota, StorageType type) if (type != null) { requireEffectiveLayoutVersionForFeature(Feature.QUOTA_BY_STORAGE_TYPE); } +if (type == StorageType.NVDIMM) { + requireEffectiveLayoutVersionForFeature(Feature.NVDIMM_SUPPORT); Review comment: Done. Please see code above. Thanks. ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java ## @@ -3569,6 +3569,9 @@ void setQuota(String src, long nsQuota, long ssQuota, StorageType type) if (type != null) { requireEffectiveLayoutVersionForFeature(Feature.QUOTA_BY_STORAGE_TYPE); } +if (type == StorageType.NVDIMM) { + requireEffectiveLayoutVersionForFeature(Feature.NVDIMM_SUPPORT); +} Review comment: Done. Please see code above. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518250) Time Spent: 6h 40m (was: 6.5h) > Fix the SetQuotaByStorageTypeOp problem after updating hadoop > --- > > Key: HDFS-15624 > URL: https://issues.apache.org/jira/browse/HDFS-15624 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.4.0 >Reporter: YaYun Wang >Priority: Major > Labels: pull-request-available, release-blocker > Time Spent: 6h 40m > Remaining Estimate: 0h > > HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum > of StorageType. And, setting the quota by storageType depends on the > ordinal(), therefore, it may cause the setting of quota to be invalid after > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15624) Fix the SetQuotaByStorageTypeOp problem after updating hadoop
[ https://issues.apache.org/jira/browse/HDFS-15624?focusedWorklogId=518251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518251 ] ASF GitHub Bot logged work on HDFS-15624: - Author: ASF GitHub Bot Created on: 01/Dec/20 02:55 Start Date: 01/Dec/20 02:55 Worklog Time Spent: 10m Work Description: huangtianhua commented on a change in pull request #2377: URL: https://github.com/apache/hadoop/pull/2377#discussion_r533040385 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java ## @@ -3569,6 +3569,9 @@ void setQuota(String src, long nsQuota, long ssQuota, StorageType type) if (type != null) { requireEffectiveLayoutVersionForFeature(Feature.QUOTA_BY_STORAGE_TYPE); } +if (type == StorageType.NVDIMM) { + requireEffectiveLayoutVersionForFeature(Feature.NVDIMM_SUPPORT); Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 518251) Time Spent: 6h 50m (was: 6h 40m) > Fix the SetQuotaByStorageTypeOp problem after updating hadoop > --- > > Key: HDFS-15624 > URL: https://issues.apache.org/jira/browse/HDFS-15624 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.4.0 >Reporter: YaYun Wang >Priority: Major > Labels: pull-request-available, release-blocker > Time Spent: 6h 50m > Remaining Estimate: 0h > > HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum > of StorageType. And, setting the quota by storageType depends on the > ordinal(), therefore, it may cause the setting of quota to be invalid after > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org