[
https://issues.apache.org/jira/browse/HDFS-16179?focusedWorklogId=640588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640588
]
ASF GitHub Bot logged work on HDFS-16179:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Aug/21 06:17
Start Date: 23/Aug/21 06:17
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3313:
URL: https://github.com/apache/hadoop/pull/3313#issuecomment-903476259
> @tomscut Hi, does this WARN log be printed when only writing EC files ?
This WARN logs also appeared in our cluster without writting any files, but not
as many as you said.
> I found that the block in the WARN log belongs to the file written a long
time ago. So, I have some guesses:
>
> * is there a daemon thread calling this method?
> * or other conditions trigger this method?
>
> Here is our 3-hour running log.
>
Thanks @whbing for your comments. I found those logs were printed after
completeFile. Triggered by FSDirWriteFileOp#completeFileInternal().
```
private static boolean completeFileInternal(
FSNamesystem fsn, INodesInPath iip,
String holder, Block last, long fileId)
throws IOException {
(...)
fsn.finalizeINodeFileUnderConstruction(src, pendingFile,
Snapshot.CURRENT_STATE_ID, true);
(...)
return true;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 640588)
Time Spent: 2.5h (was: 2h 20m)
> Update loglevel for BlockManager#chooseExcessRedundancyStriped to avoid too
> much logs
> -------------------------------------------------------------------------------------
>
> Key: HDFS-16179
> URL: https://issues.apache.org/jira/browse/HDFS-16179
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.1.0
> Reporter: tomscut
> Assignee: tomscut
> Priority: Minor
> Labels: pull-request-available
> Attachments: log-count.jpg, logs.jpg
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {code:java}
> private void chooseExcessRedundancyStriped(BlockCollection bc,
> final Collection<DatanodeStorageInfo> nonExcess,
> BlockInfo storedBlock,
> DatanodeDescriptor delNodeHint) {
> ...
> // cardinality of found indicates the expected number of internal blocks
> final int numOfTarget = found.cardinality();
> final BlockStoragePolicy storagePolicy = storagePolicySuite.getPolicy(
> bc.getStoragePolicyID());
> final List<StorageType> excessTypes = storagePolicy.chooseExcess(
> (short) numOfTarget, DatanodeStorageInfo.toStorageTypes(nonExcess));
> if (excessTypes.isEmpty()) {
> LOG.warn("excess types chosen for block {} among storages {} is empty",
> storedBlock, nonExcess);
> return;
> }
> ...
> }
> {code}
>
> IMO, here is just detecting excess StorageType and setting the log level to
> debug has no effect.
>
> We have a cluster that uses the EC policy to store data. The current log
> level is WARN here, and in about 50 minutes, 286,093 logs are printed, which
> can cause other important logs to drown out.
>
> !logs.jpg|width=1167,height=62!
>
> !log-count.jpg|width=760,height=30!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
