[jira] [Assigned] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread Masatake Iwasaki (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki reassigned HDFS-15677:
---

Assignee: Masatake Iwasaki

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-15676) TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk

2020-11-30 Thread Masatake Iwasaki (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240554#comment-17240554
 ] 

Masatake Iwasaki commented on HDFS-15676:
-

The test fails only if I run both testNamenodeMetrics and 
testGetCachedDatanodeReport. Running 
{{-Dtest=testNamenodeMetrics#testNamenodeMetrics}} alone has no issue. Since 
the TestRouterRpcMultiDestination reuses a mini cluster in the test cases, 
{{testNamenodeMetrics}} could be fixed by addressing 
{{testGetCachedDatanodeReport}} issue.

{noformat}
$ for i in `seq 100` ; do echo $i && mvn test -DignoreTestFailure=false 
-Dtest=TestRouterRpcMultiDestination#testNamenodeMetrics,TestRouterRpcMultiDestination#testGetCachedDatanodeReport
  || break ; done
...
[ERROR] Failures:
[ERROR]   TestRouterRpcMultiDestination>TestRouterRpc.testNamenodeMetrics:1682 
expected:<12> but was:<11>
[ERROR] Errors:
[ERROR]   
TestRouterRpcMultiDestination>TestRouterRpc.testGetCachedDatanodeReport:1829 » 
Timeout
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 1, Skipped: 0
{noformat}


> TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk
> 
>
> Key: HDFS-15676
> URL: https://issues.apache.org/jira/browse/HDFS-15676
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in testNamenodeMetrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-5328) MiniDFSCluster times out at startup

2020-11-30 Thread Matteo Minardi (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240597#comment-17240597
 ] 

Matteo Minardi commented on HDFS-5328:
--

Any news on this? I get the same error.
In the code I still see:
{code:java}
  public void waitClusterUp() throws IOException {
int i = 0;
if (numDataNodes > 0) {
  while (!isClusterUp()) {
try {
  LOG.warn("Waiting for the Mini HDFS Cluster to start...");
  Thread.sleep(1000);
} catch (InterruptedException e) {
}
if (++i > 10) {
  final String msg = "Timed out waiting for Mini HDFS Cluster to start";
  LOG.error(msg);
  throw new IOException(msg);
}
  }
}
  }
{code}
With 10 hardcoded

 

> MiniDFSCluster times out at startup
> ---
>
> Key: HDFS-5328
> URL: https://issues.apache.org/jira/browse/HDFS-5328
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Major
> Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch
>
>
> It seems MiniDFSCluster startup got slower lately and we are observing the 
> following intermittent failure on startup in our jenkins boxes:
> {code}
> java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511)
> {code}
> By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout 
> (10 1 sec waits).
> There are two things we should look at it, make the timeout configurable, and 
> more important, see what has changed to make the startup slower.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-5328) MiniDFSCluster times out at startup

2020-11-30 Thread Matteo Minardi (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240597#comment-17240597
 ] 

Matteo Minardi edited comment on HDFS-5328 at 11/30/20, 9:27 AM:
-

Any news on this? I get the same error.
 In the code I still see:
{code:java}
  public void waitClusterUp() throws IOException {
int i = 0;
if (numDataNodes > 0) {
  while (!isClusterUp()) {
try {
  LOG.warn("Waiting for the Mini HDFS Cluster to start...");
  Thread.sleep(1000);
} catch (InterruptedException e) {
}
if (++i > 10) {
  final String msg = "Timed out waiting for Mini HDFS Cluster to start";
  LOG.error(msg);
  throw new IOException(msg);
}
  }
}
  }
{code}
With 10 hardcoded
 [~rvesse] [~tucu00]


was (Author: mino181295):
Any news on this? I get the same error.
 In the code I still see:
{code:java}
  public void waitClusterUp() throws IOException {
int i = 0;
if (numDataNodes > 0) {
  while (!isClusterUp()) {
try {
  LOG.warn("Waiting for the Mini HDFS Cluster to start...");
  Thread.sleep(1000);
} catch (InterruptedException e) {
}
if (++i > 10) {
  final String msg = "Timed out waiting for Mini HDFS Cluster to start";
  LOG.error(msg);
  throw new IOException(msg);
}
  }
}
  }
{code}
With 10 hardcoded
[~rvesse] [~tucu00] 

 

> MiniDFSCluster times out at startup
> ---
>
> Key: HDFS-5328
> URL: https://issues.apache.org/jira/browse/HDFS-5328
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Major
> Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch
>
>
> It seems MiniDFSCluster startup got slower lately and we are observing the 
> following intermittent failure on startup in our jenkins boxes:
> {code}
> java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511)
> {code}
> By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout 
> (10 1 sec waits).
> There are two things we should look at it, make the timeout configurable, and 
> more important, see what has changed to make the startup slower.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-5328) MiniDFSCluster times out at startup

2020-11-30 Thread Matteo Minardi (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240597#comment-17240597
 ] 

Matteo Minardi edited comment on HDFS-5328 at 11/30/20, 9:27 AM:
-

Any news on this? I get the same error.
 In the code I still see:
{code:java}
  public void waitClusterUp() throws IOException {
int i = 0;
if (numDataNodes > 0) {
  while (!isClusterUp()) {
try {
  LOG.warn("Waiting for the Mini HDFS Cluster to start...");
  Thread.sleep(1000);
} catch (InterruptedException e) {
}
if (++i > 10) {
  final String msg = "Timed out waiting for Mini HDFS Cluster to start";
  LOG.error(msg);
  throw new IOException(msg);
}
  }
}
  }
{code}
With 10 hardcoded
[~rvesse] [~tucu00] 

 


was (Author: mino181295):
Any news on this? I get the same error.
In the code I still see:
{code:java}
  public void waitClusterUp() throws IOException {
int i = 0;
if (numDataNodes > 0) {
  while (!isClusterUp()) {
try {
  LOG.warn("Waiting for the Mini HDFS Cluster to start...");
  Thread.sleep(1000);
} catch (InterruptedException e) {
}
if (++i > 10) {
  final String msg = "Timed out waiting for Mini HDFS Cluster to start";
  LOG.error(msg);
  throw new IOException(msg);
}
  }
}
  }
{code}
With 10 hardcoded

 

> MiniDFSCluster times out at startup
> ---
>
> Key: HDFS-5328
> URL: https://issues.apache.org/jira/browse/HDFS-5328
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Major
> Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch
>
>
> It seems MiniDFSCluster startup got slower lately and we are observing the 
> following intermittent failure on startup in our jenkins boxes:
> {code}
> java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511)
> {code}
> By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout 
> (10 1 sec waits).
> There are two things we should look at it, make the timeout configurable, and 
> more important, see what has changed to make the startup slower.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-5328) MiniDFSCluster times out at startup

2020-11-30 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240599#comment-17240599
 ] 

Hadoop QA commented on HDFS-5328:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 13s{color} 
| {color:red}{color} | {color:red} HDFS-5328 does not apply to trunk. Rebase 
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for 
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-5328 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12782266/HDFS-5328.002.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/322/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> MiniDFSCluster times out at startup
> ---
>
> Key: HDFS-5328
> URL: https://issues.apache.org/jira/browse/HDFS-5328
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.1.1-beta, 2.7.1, 3.0.0-alpha1
>Reporter: Alejandro Abdelnur
>Priority: Major
> Attachments: HDFS-5328.001.patch, HDFS-5328.002.patch
>
>
> It seems MiniDFSCluster startup got slower lately and we are observing the 
> following intermittent failure on startup in our jenkins boxes:
> {code}
> java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:972)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:665)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:585)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:511)
> {code}
> By looking at the code the {{waitClusterUp()}} has a hardcoded 10secs timeout 
> (10 1 sec waits).
> There are two things we should look at it, make the timeout configurable, and 
> more important, see what has changed to make the startup slower.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-15163) hdfs-2.10.0-webapps-secondary-status.html miss moment.js

2020-11-30 Thread Jira 


 [ 
https://issues.apache.org/jira/browse/HDFS-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

谢波 resolved HDFS-15163.
---
Resolution: Fixed

> hdfs-2.10.0-webapps-secondary-status.html miss moment.js
> 
>
> Key: HDFS-15163
> URL: https://issues.apache.org/jira/browse/HDFS-15163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.10.0
>Reporter: 谢波
>Priority: Minor
> Attachments: 微信截图_20200212183444.png
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> hdfs-2.10.0-webapps-secondary-status.html miss moment.js
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14437) Exception happened when rollEditLog expects empty EditsDoubleBuffer.bufCurrent but not

2020-11-30 Thread Xiaoqiao He (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240663#comment-17240663
 ] 

Xiaoqiao He commented on HDFS-14437:


[~Sushma_28] Actually, I also meet the similar case (NameNode crash) long 
before, it is solved after patch HDFS-13112.
For this issue, there is no strict proof and we have not reached the final 
agreement. Welcome more discussion here. Thanks.

> Exception happened when   rollEditLog expects empty 
> EditsDoubleBuffer.bufCurrent  but not
> -
>
> Key: HDFS-14437
> URL: https://issues.apache.org/jira/browse/HDFS-14437
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ha, namenode, qjm
>Reporter: angerszhu
>Priority: Major
> Attachments: HDFS-14437.reproduction.patch, 
> HDFS-14437.reproductionwithlog.patch, screenshot-1.png
>
>
> For the problem mentioned in https://issues.apache.org/jira/browse/HDFS-10943 
> , I have sort the process of write and flush EditLog and some important 
> function, I found the in the class  FSEditLog class, the close() function 
> will call such process like below:
>  
> {code:java}
> waitForSyncToFinish();
> endCurrentLogSegment(true);{code}
> since we have gain the object lock in the function close(), so when  
> waitForSyncToFish() method return, it mean all logSync job has done and all 
> data in bufReady has been flushed out, and since current thread has the lock 
> of this object, when call endCurrentLogSegment(), no other thread will gain 
> the lock so they can't write new editlog into currentBuf.
> But when we don't call waitForSyncToFish() before endCurrentLogSegment(), 
> there may be some autoScheduled logSync()'s flush process is doing, since 
> this process don't need
> synchronization since it has mention in the comment of logSync() method :
>  
> {code:java}
> /**
>  * Sync all modifications done by this thread.
>  *
>  * The internal concurrency design of this class is as follows:
>  *   - Log items are written synchronized into an in-memory buffer,
>  * and each assigned a transaction ID.
>  *   - When a thread (client) would like to sync all of its edits, logSync()
>  * uses a ThreadLocal transaction ID to determine what edit number must
>  * be synced to.
>  *   - The isSyncRunning volatile boolean tracks whether a sync is currently
>  * under progress.
>  *
>  * The data is double-buffered within each edit log implementation so that
>  * in-memory writing can occur in parallel with the on-disk writing.
>  *
>  * Each sync occurs in three steps:
>  *   1. synchronized, it swaps the double buffer and sets the isSyncRunning
>  *  flag.
>  *   2. unsynchronized, it flushes the data to storage
>  *   3. synchronized, it resets the flag and notifies anyone waiting on the
>  *  sync.
>  *
>  * The lack of synchronization on step 2 allows other threads to continue
>  * to write into the memory buffer while the sync is in progress.
>  * Because this step is unsynchronized, actions that need to avoid
>  * concurrency with sync() should be synchronized and also call
>  * waitForSyncToFinish() before assuming they are running alone.
>  */
> public void logSync() {
>   long syncStart = 0;
>   // Fetch the transactionId of this thread. 
>   long mytxid = myTransactionId.get().txid;
>   
>   boolean sync = false;
>   try {
> EditLogOutputStream logStream = null;
> synchronized (this) {
>   try {
> printStatistics(false);
> // if somebody is already syncing, then wait
> while (mytxid > synctxid && isSyncRunning) {
>   try {
> wait(1000);
>   } catch (InterruptedException ie) {
>   }
> }
> //
> // If this transaction was already flushed, then nothing to do
> //
> if (mytxid <= synctxid) {
>   numTransactionsBatchedInSync++;
>   if (metrics != null) {
> // Metrics is non-null only when used inside name node
> metrics.incrTransactionsBatchedInSync();
>   }
>   return;
> }
>
> // now, this thread will do the sync
> syncStart = txid;
> isSyncRunning = true;
> sync = true;
> // swap buffers
> try {
>   if (journalSet.isEmpty()) {
> throw new IOException("No journals available to flush");
>   }
>   editLogStream.setReadyToFlush();
> } catch (IOException e) {
>   final String msg =
>   "Could not sync enough journals to persistent storage " +
>   "due to " + e.getMessage() + ". " +
>   "Unsynced transactions: " + (txid - synctxid);
>   LOG.fatal(msg, new Exception());
>   synchronized(journalSetLock) {
>

[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=517873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517873
 ]

ASF GitHub Bot logged work on HDFS-15677:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 11:49
Start Date: 30/Nov/20 11:49
Worklog Time Spent: 10m 
  Work Description: iwasakims opened a new pull request #2503:
URL: https://github.com/apache/hadoop/pull/2503


   https://issues.apache.org/jira/browse/HDFS-15677
   
   Since heartbeatExpireInterval and assertion timeout of DN removal has the 
same value (5000 ms), the test fails intermittently. Updating the 
heartbeatExpireInterval and assertion timeout worked for me.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 517873)
Remaining Estimate: 0h
Time Spent: 10m

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDFS-15677:
--
Labels: pull-request-available  (was: )

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread Masatake Iwasaki (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-15677:

Status: Patch Available  (was: Open)

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=517929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-517929
 ]

ASF GitHub Bot logged work on HDFS-15677:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 13:18
Start Date: 30/Nov/20 13:18
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2503:
URL: https://github.com/apache/hadoop/pull/2503#issuecomment-735779663


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 30s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  32m 34s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 41s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |   0m 36s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 27s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 40s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 20s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 40s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   1m 15s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   1m 14s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 29s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |   0m 29s |  |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   0m 17s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no 
whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  15m  8s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   0m 52s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   1m 14s |  |  the patch passed  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   8m 34s |  |  hadoop-hdfs-rbf in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 32s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   |  86m 58s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2503/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2503 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux f39134cbd1e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 44910b53f38 |
   | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2503/1/testReport/ |
   | Max. process+thread count | 3038 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: 
hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2503/1/console |
   | versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
   | Powered by | Apache Yetus 0.13.0-SNAPSHOT https:

[jira] [Commented] (HDFS-15680) Disable Broken Azure Junits

2020-11-30 Thread Xiaoqiao He (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240838#comment-17240838
 ] 

Xiaoqiao He commented on HDFS-15680:


Thanks [~ayushtkn] for your informations. I backport HADOOP-17325 to branch-3.2 
and run failed unit tests mentioned above at local, Both of them are passed. 
Thanks.

> Disable Broken Azure Junits
> ---
>
> Key: HDFS-15680
> URL: https://issues.apache.org/jira/browse/HDFS-15680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> There are 6 test classes have been failing on Yetus for several months. 
> They contributed to more than 41 failing tests which makes reviewing Yetus 
> reports every a pain in the neck. Another point is to save the resources and 
> avoiding utilization of ports, memory, and CPU.
> Over the last month, there was some effort to bring the Yetus back to a 
> stable state. However, there is no progress in addressing Azure failures.
> Generally, I do not like to disable failing tests, but for this specific 
> case, I do not assume that it makes any sense to have 41 failing tests from 
> one module for several months. Whenever someone finds that those tests are 
> useful, then they can re-enable the tests on Yetus *_After_* the test is 
> fixed.
> Following a PR, I have to  review that my patch does not cause any failures 
> (include changing error messages in existing tests). A thorough review takes 
> a considerable amount of time browsing the nightly builds and Github reports.
> So, please consider how much time is being spent to review those stack trace 
> over the last months.
> Finally, this is one of the reasons developers tend to ignore the reports, 
> because it would take too much time to review; and by default, the errors are 
> considered irrelevant.
> CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn]
> {code:bash}
>   hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked 
>hadoop.fs.azure.TestNativeAzureFileSystemMocked 
>hadoop.fs.azure.TestBlobMetadata 
>hadoop.fs.azure.TestNativeAzureFileSystemConcurrency 
>hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck 
>hadoop.fs.azure.TestNativeAzureFileSystemContractMocked 
>hadoop.fs.azure.TestWasbFsck 
>hadoop.fs.azure.TestOutOfBandAzureBlobOperations 
> {code}
> {code:bash}
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMo

[jira] [Work logged] (HDFS-15695) NN should not let the balancer run in safemode

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15695?focusedWorklogId=518052&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518052
 ]

ASF GitHub Bot logged work on HDFS-15695:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 17:51
Start Date: 30/Nov/20 17:51
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2489:
URL: https://github.com/apache/hadoop/pull/2489#issuecomment-735941256


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  9s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  34m 51s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 19s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |   1m 10s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 49s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  19m 11s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 53s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   1m 20s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   3m 16s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   3m 13s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 13s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 13s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |   1m 13s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  7s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |   1m  7s |  |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   0m 42s |  |  
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 56 unchanged - 10 
fixed = 56 total (was 66)  |
   | +1 :green_heart: |  mvnsite  |   1m 14s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no 
whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  16m 56s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   1m 18s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   3m 15s |  |  the patch passed  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 116m 25s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 36s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 211m 45s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | 
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2489 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux 2393e22da497 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 44910b53f38 |
   | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibra

[jira] [Work logged] (HDFS-15694) Avoid calling UpdateHeartBeatState inside DataNodeDescriptor

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15694?focusedWorklogId=518064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518064
 ]

ASF GitHub Bot logged work on HDFS-15694:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 18:13
Start Date: 30/Nov/20 18:13
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2487:
URL: https://github.com/apache/hadoop/pull/2487#issuecomment-735953319


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 36s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |   |   0m  0s | [test4tests](test4tests) |  The patch 
appears to include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  32m 50s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 22s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  compile  |   1m 15s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  checkstyle  |   0m 50s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m 32s |  |  branch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 55s |  |  trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   1m 27s |  |  trunk passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +0 :ok: |  spotbugs  |   3m  7s |  |  Used deprecated FindBugs config; 
considering switching to SpotBugs.  |
   | +1 :green_heart: |  findbugs  |   3m  5s |  |  trunk passed  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m  9s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 13s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javac  |   1m 13s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  javac  |   1m  4s |  |  the patch passed  |
   | +1 :green_heart: |  checkstyle  |   0m 39s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m  9s |  |  the patch passed  |
   | +1 :green_heart: |  whitespace  |   0m  0s |  |  The patch has no 
whitespace issues.  |
   | +1 :green_heart: |  shadedclient  |  14m 37s |  |  patch has no errors 
when building and testing our client artifacts.  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04  |
   | +1 :green_heart: |  javadoc  |   1m 19s |  |  the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01  |
   | +1 :green_heart: |  findbugs  |   3m  4s |  |  the patch passed  |
    _ Other Tests _ |
   | -1 :x: |  unit  |  96m  8s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2487/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 43s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 185m  6s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
   |   | hadoop.hdfs.TestRollingUpgrade |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.40 ServerAPI=1.40 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2487/3/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2487 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux 1dcdcf3af76c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 44910b53f38 |
   | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2487/3/testReport/ |
   | Max. process+thread count | 3969 (vs

[jira] [Created] (HDFS-15700) Protobuf Mesage Incompatibility Detector

2020-11-30 Thread junwen yang (Jira) 
junwen yang created HDFS-15700:
--

 Summary: Protobuf Mesage Incompatibility Detector
 Key: HDFS-15700
 URL: https://issues.apache.org/jira/browse/HDFS-15700
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: junwen yang
 Attachments: hdfs_proto_incompatibility.txt

Regarding the issue  -HDFS-9788- caused by the incompatibility of protobuf 
message, we have created a static checker which keeps track of the proto file 
change, and detect potential incompatibility:
 # Add/delete required field, which is the case reported in HBASE-25238.
 # The tag number of a field has been changed, as described in HDFS-9788. Also, 
the [protobuf guidelines 
|https://developers.google.com/protocol-buffers/docs/proto]suggests _each field 
in the message definition has a *unique number*. These numbers are used to 
identify your fields in the [message binary 
format|https://developers.google.com/protocol-buffers/docs/encoding], and 
should not be changed once your message type is in use_. 
 # A  required field has been changed to optional, or an optional field has 
been changed to required. According to the guidelines in [protobuf official 
website|https://developers.google.com/protocol-buffers/docs/proto], _*Required 
Is Forever* You should be very careful about marking fields as {{required}}. If 
at some point you wish to stop writing or sending a required field, it will be 
problematic to change the field to an optional field - old readers will 
consider messages without this field to be incomplete and may reject or drop 
them unintentionally. You should consider writing application-specific custom 
validation routines for your buffers instead._

We have applied our checker on the frequently maintained HDFS versions: 
rel/release-2.6.4, rel/release-2.7.2, rel/release-2.8.0, rel/release-2.9.0, 
rel/release-2.10.0, rel/release-3.0.0, rel/release-3.1.0, rel/release-3.2.0, 
rel/release-3.3.0, and we found 6 potential problems as attached. The checker 
is also able to detect the issue describe in -HDFS-9788.-

The results reported by our checker got confirmation from developers in HBASE, 
which can be found here HBASE-25340. 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-15701) Add resolveMountPath API in FileSystem

2020-11-30 Thread Uma Maheswara Rao G (Jira) 
Uma Maheswara Rao G created HDFS-15701:
--

 Summary: Add resolveMountPath API in FileSystem
 Key: HDFS-15701
 URL: https://issues.apache.org/jira/browse/HDFS-15701
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: fs, ViewHDFS
Affects Versions: 3.4.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


Currently FileSystem has an API resolvePath. To know where the path has 
mounted, the applications can use that API as the retuned path is from actual 
target path in the case of mount file systems like ViewFS, ViewFSOverloadScheme 
or ViewDistributedFileSystem.

However, resolvePath does more than what is needed by Apps when they want to 
know where the path has mounted. It's because resolvePath internally calls 
"getFileStatus". 

This additional call is unnecessary when apps just want to where the path 
mounted. 

Since we have mounted filesystems available in FS, I think it's good to add 
resolveMountPath API, which will just do the following.

If the fs is mounted fs, then it will resolve it's mount tables and return the 
actual target path. If the fs is non mounted, then it will simply return the 
same path.
Currently Applications like Hive, Ranger using resolvePath API. ( this is 
forcing to do additional RPC internally)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=518168&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518168
 ]

ASF GitHub Bot logged work on HDFS-15677:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 23:07
Start Date: 30/Nov/20 23:07
Worklog Time Spent: 10m 
  Work Description: iwasakims commented on pull request #2503:
URL: https://github.com/apache/hadoop/pull/2503#issuecomment-736109294


   Thanks, @goiri. I merged this.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 518168)
Time Spent: 40m  (was: 0.5h)

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?focusedWorklogId=518167&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518167
 ]

ASF GitHub Bot logged work on HDFS-15677:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 23:07
Start Date: 30/Nov/20 23:07
Worklog Time Spent: 10m 
  Work Description: iwasakims merged pull request #2503:
URL: https://github.com/apache/hadoop/pull/2503


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 518167)
Time Spent: 0.5h  (was: 20m)

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-15677) TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk

2020-11-30 Thread Masatake Iwasaki (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HDFS-15677:

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk
> 
>
> Key: HDFS-15677
> URL: https://issues.apache.org/jira/browse/HDFS-15677
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in 
> testGetCachedDatanodeReport



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-15676) TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk

2020-11-30 Thread Masatake Iwasaki (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki resolved HDFS-15676.
-
Resolution: Duplicate

I'm closing this as fixed by HDFS-15677. I will reopen this if the failure of 
testNameNodeMetrics alone rise again.

> TestRouterRpcMultiDestination#testNamenodeMetrics fails on trunk
> 
>
> Key: HDFS-15676
> URL: https://issues.apache.org/jira/browse/HDFS-15676
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
>
> qbt report (Nov 8, 2020, 11:28 AM) shows failures in testNamenodeMetrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-14904) Option to let Balancer prefer top used nodes in each iteration

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-14904?focusedWorklogId=518182&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518182
 ]

ASF GitHub Bot logged work on HDFS-14904:
-

Author: ASF GitHub Bot
Created on: 30/Nov/20 23:56
Start Date: 30/Nov/20 23:56
Worklog Time Spent: 10m 
  Work Description: Jing9 commented on a change in pull request #2483:
URL: https://github.com/apache/hadoop/pull/2483#discussion_r532981333



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
##
@@ -435,6 +444,22 @@ private long init(List reports) {
 return Math.max(overLoadedBytes, underLoadedBytes);
   }
 
+  private void sortOverUtilizedNodes() {
+LOG.info("Sorting over-utilized nodes by capacity" +
+" to bring down top used datanode capacity faster");
+
+if (overUtilized instanceof List) {
+  List list = (List) overUtilized;
+  list.sort(
+  (Source source1, Source source2) ->

Review comment:
   Do we need to consider StorageType (which is associated with Source)? 
E.g., suppose a DN has 2 storage types, one of which is highly utilized and the 
other is just above average. Do we want to first schedule the movement for the 
highly-utilized storage type on this node? 

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
##
@@ -435,6 +444,22 @@ private long init(List reports) {
 return Math.max(overLoadedBytes, underLoadedBytes);
   }
 
+  private void sortOverUtilizedNodes() {
+LOG.info("Sorting over-utilized nodes by capacity" +
+" to bring down top used datanode capacity faster");
+
+if (overUtilized instanceof List) {

Review comment:
   Do we need this "if" statement? Maybe use a Preconditions instead?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
##
@@ -199,7 +199,10 @@
   + "\tWhether to run the balancer during an ongoing HDFS upgrade."
   + "This is usually not desired since it will not affect used space "
   + "on over-utilized machines."
-  + "\n\t[-asService]\tRun as a long running service.";
+  + "\n\t[-asService]\tRun as a long running service."
+  + "\n\t[-sortTopNodes]"
+  + "\tSort over-utilized nodes by capacity to"
+  + " bring down top used datanode faster.";

Review comment:
   How about describe the parameter option as: "sort datanodes based on the 
utilization so that highly utilized datanodes get scheduled first"?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 518182)
Time Spent: 40m  (was: 0.5h)

> Option to let Balancer prefer top used nodes in each iteration
> --
>
> Key: HDFS-14904
> URL: https://issues.apache.org/jira/browse/HDFS-14904
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover
>Reporter: Leon Gao
>Assignee: Leon Gao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Normally the most important purpose for HDFS balancer is to reduce the top 
> used node to prevent datanode usage from being too high.
> Currently, balancer almost randomly picks nodes as sources regardless of 
> usage, which makes it slow to bring down the top used datanodes in the 
> cluster, when there are less underutilized nodes in the cluster (consider 
> expansion).
> We can add an option to prefer top used nodes first in each iteration, as 
> suggested in HDFS-14894 .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-15702) Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission

2020-11-30 Thread Masatake Iwasaki (Jira) 
Masatake Iwasaki created HDFS-15702:
---

 Summary: Fix intermittent falilure of 
TestDecommission#testAllocAndIBRWhileDecommission
 Key: HDFS-15702
 URL: https://issues.apache.org/jira/browse/HDFS-15702
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


{noformat}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hdfs.TestDecommission.testAllocAndIBRWhileDecommission(TestDecommission.java:1025)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-15624) Fix the SetQuotaByStorageTypeOp problem after updating hadoop

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15624?focusedWorklogId=518250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518250
 ]

ASF GitHub Bot logged work on HDFS-15624:
-

Author: ASF GitHub Bot
Created on: 01/Dec/20 02:53
Start Date: 01/Dec/20 02:53
Worklog Time Spent: 10m 
  Work Description: huangtianhua commented on a change in pull request 
#2377:
URL: https://github.com/apache/hadoop/pull/2377#discussion_r533039713



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
##
@@ -3569,6 +3569,9 @@ void setQuota(String src, long nsQuota, long ssQuota, 
StorageType type)
 if (type != null) {
   requireEffectiveLayoutVersionForFeature(Feature.QUOTA_BY_STORAGE_TYPE);
 }
+if (type == StorageType.NVDIMM) {
+  requireEffectiveLayoutVersionForFeature(Feature.NVDIMM_SUPPORT);

Review comment:
   Done. Please see code above. Thanks.

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
##
@@ -3569,6 +3569,9 @@ void setQuota(String src, long nsQuota, long ssQuota, 
StorageType type)
 if (type != null) {
   requireEffectiveLayoutVersionForFeature(Feature.QUOTA_BY_STORAGE_TYPE);
 }
+if (type == StorageType.NVDIMM) {
+  requireEffectiveLayoutVersionForFeature(Feature.NVDIMM_SUPPORT);
+}

Review comment:
   Done. Please see code above. Thanks.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 518250)
Time Spent: 6h 40m  (was: 6.5h)

>  Fix the SetQuotaByStorageTypeOp problem after updating hadoop 
> ---
>
> Key: HDFS-15624
> URL: https://issues.apache.org/jira/browse/HDFS-15624
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.4.0
>Reporter: YaYun Wang
>Priority: Major
>  Labels: pull-request-available, release-blocker
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum 
> of StorageType. And, setting the quota by storageType depends on the 
> ordinal(), therefore, it may cause the setting of quota to be invalid after 
> upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-15624) Fix the SetQuotaByStorageTypeOp problem after updating hadoop

2020-11-30 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-15624?focusedWorklogId=518251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-518251
 ]

ASF GitHub Bot logged work on HDFS-15624:
-

Author: ASF GitHub Bot
Created on: 01/Dec/20 02:55
Start Date: 01/Dec/20 02:55
Worklog Time Spent: 10m 
  Work Description: huangtianhua commented on a change in pull request 
#2377:
URL: https://github.com/apache/hadoop/pull/2377#discussion_r533040385



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
##
@@ -3569,6 +3569,9 @@ void setQuota(String src, long nsQuota, long ssQuota, 
StorageType type)
 if (type != null) {
   requireEffectiveLayoutVersionForFeature(Feature.QUOTA_BY_STORAGE_TYPE);
 }
+if (type == StorageType.NVDIMM) {
+  requireEffectiveLayoutVersionForFeature(Feature.NVDIMM_SUPPORT);

Review comment:
   Done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 518251)
Time Spent: 6h 50m  (was: 6h 40m)

>  Fix the SetQuotaByStorageTypeOp problem after updating hadoop 
> ---
>
> Key: HDFS-15624
> URL: https://issues.apache.org/jira/browse/HDFS-15624
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.4.0
>Reporter: YaYun Wang
>Priority: Major
>  Labels: pull-request-available, release-blocker
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum 
> of StorageType. And, setting the quota by storageType depends on the 
> ordinal(), therefore, it may cause the setting of quota to be invalid after 
> upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org