[jira] [Work logged] (HDFS-16149) Improve the parameter annotation in FairCallQueue#priorityLevels
[
https://issues.apache.org/jira/browse/HDFS-16149?focusedWorklogId=632761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632761
]
ASF GitHub Bot logged work on HDFS-16149:
-
Author: ASF GitHub Bot
Created on: 03/Aug/21 08:53
Start Date: 03/Aug/21 08:53
Worklog Time Spent: 10m
Work Description: jojochuang merged pull request #3255:
URL: https://github.com/apache/hadoop/pull/3255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 632761)
Time Spent: 40m (was: 0.5h)
> Improve the parameter annotation in FairCallQueue#priorityLevels
>
>
> Key: HDFS-16149
> URL: https://issues.apache.org/jira/browse/HDFS-16149
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The parameter description of FairCallQueue#priorityLevels is missing in
> FairCallQueue.
>/**
> * Create a FairCallQueue.
> * @param capacity the total size of all sub-queues
> * @param ns the prefix to use for configuration
> * @param capacityWeights the weights array for capacity allocation
> * among subqueues
> * @param conf the configuration to read from
> * Notes: Each sub-queue has a capacity of `capacity / numSubqueues`.
> * The first or the highest priority sub-queue has an excess capacity
> * of `capacity% numSubqueues`
> */
>public FairCallQueue(int priorityLevels, int capacity, String ns,
>int[] capacityWeights, Configuration conf) {
> We should be more perfect for them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16149) Improve the parameter annotation in FairCallQueue#priorityLevels
[
https://issues.apache.org/jira/browse/HDFS-16149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved HDFS-16149.
Fix Version/s: 3.4.0
Resolution: Fixed
> Improve the parameter annotation in FairCallQueue#priorityLevels
>
>
> Key: HDFS-16149
> URL: https://issues.apache.org/jira/browse/HDFS-16149
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The parameter description of FairCallQueue#priorityLevels is missing in
> FairCallQueue.
>/**
> * Create a FairCallQueue.
> * @param capacity the total size of all sub-queues
> * @param ns the prefix to use for configuration
> * @param capacityWeights the weights array for capacity allocation
> * among subqueues
> * @param conf the configuration to read from
> * Notes: Each sub-queue has a capacity of `capacity / numSubqueues`.
> * The first or the highest priority sub-queue has an excess capacity
> * of `capacity% numSubqueues`
> */
>public FairCallQueue(int priorityLevels, int capacity, String ns,
>int[] capacityWeights, Configuration conf) {
> We should be more perfect for them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16129) HttpFS signature secret file misusage
[
https://issues.apache.org/jira/browse/HDFS-16129?focusedWorklogId=632823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632823
]
ASF GitHub Bot logged work on HDFS-16129:
-
Author: ASF GitHub Bot
Created on: 03/Aug/21 09:56
Start Date: 03/Aug/21 09:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3209:
URL: https://github.com/apache/hadoop/pull/3209#issuecomment-891708421
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 19s | | trunk passed |
| +1 :green_heart: | compile | 21m 35s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 1s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 44s | | the patch passed |
| +1 :green_heart: | compile | 20m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 28s | | the patch passed |
| +1 :green_heart: | compile | 19m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 48s | | root: The patch generated
0 new + 97 unchanged - 2 fixed = 97 total (was 99) |
| +1 :green_heart: | mvnsite | 3m 1s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 12s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/6/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs.html)
| hadoop-hdfs-project/hadoop-hdfs-httpfs generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 15m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 50s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 38s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | unit | 9m 14s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 216m 12s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-httpfs |
| |
org.apache.hadoop.fs.http.server.HttpFSAuthenticationFilter.CONF_PREFIXES
should be package protected At HttpFSAuthenticationFilter.java: At
HttpFSAuthenticationFilter.java:[line 54] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3209 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclien
[jira] [Comment Edited] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391605#comment-17391605
]
liuyongpan edited comment on HDFS-16147 at 8/3/21, 10:56 AM:
-
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter , y{color}ou can get the
answer.
3、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}note:{color} first you must merge the patch HDFS-14617
You can simply verify as follows：
in class TestFSImageWithSnapshot , method : setUp , add such code:
{code:java}
public void setUp() throws Exception {
conf = new Configuration();
//*add**
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "3");
//**add*
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION)
.build();
cluster.waitActive();
fsn = cluster.getNamesystem();
hdfs = cluster.getFileSystem();
}
{code}
In class TestOfflineImageViewer , method createOriginalFSImage, change as
follow:
{code:java}
class FSImageFormatProtobuf, method loadInternal
case INODE: {
currentStep = new Step(StepType.INODES);
prog.beginStep(Phase.LOADING_FSIMAGE, currentStep);
stageSubSections = getSubSectionsOfName(
subSections, SectionName.INODE_SUB);
// if (loadInParallel && (stageSubSections.size() > 0)) {
//inodeLoader.loadINodeSectionInParallel(executorService,
//stageSubSections, summary.getCodec(), prog, currentStep);
// } else {
//inodeLoader.loadINodeSection(in, prog, currentStep);
// }
inodeLoader.loadINodeSection(in, prog, currentStep);
}
{code}
then run test unit {color:#ffc66d}testSaveLoadImage , you can get the
answer.{color}
{color:#ffc66d}{color:#172b4d}3、{color}
{color}
was (Author: mofei):
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter , y{color}ou can get the
answer.
3、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}note:{color} first you must merge the patch HDFS-14617
You can simply verify as follows：
in class TestFSImageWithSnapshot , method :
[jira] [Comment Edited] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391605#comment-17391605
]
liuyongpan edited comment on HDFS-16147 at 8/3/21, 11:00 AM:
-
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter , y{color}ou can get the
answer.
3、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}note:{color} first you must merge the patch HDFS-14617
You can simply verify as follows：
in class TestFSImageWithSnapshot , method : setUp , add such code:
{code:java}
public void setUp() throws Exception {
conf = new Configuration();
//*add**
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "3");
//**add*
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION)
.build();
cluster.waitActive();
fsn = cluster.getNamesystem();
hdfs = cluster.getFileSystem();
}
{code}
In class TestOfflineImageViewer , method createOriginalFSImage, change as
follow:
{code:java}
class FSImageFormatProtobuf, method loadInternal
case INODE: {
currentStep = new Step(StepType.INODES);
prog.beginStep(Phase.LOADING_FSIMAGE, currentStep);
stageSubSections = getSubSectionsOfName(
subSections, SectionName.INODE_SUB);
// if (loadInParallel && (stageSubSections.size() > 0)) {
//inodeLoader.loadINodeSectionInParallel(executorService,
//stageSubSections, summary.getCodec(), prog, currentStep);
// } else {
//inodeLoader.loadINodeSection(in, prog, currentStep);
// }
inodeLoader.loadINodeSection(in, prog, currentStep);
}
{code}
then run test unit {color:#ffc66d}testSaveLoadImage , you can get the
answer.{color}
{color:#172b4d}3、50% improvement measured against a compressed single threaded
load verses parallel compressed loading. {color}
was (Author: mofei):
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter , y{color}ou can get the
answer.
3、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}note:{color} first you must merge the patch HDFS-1461
[jira] [Comment Edited] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391605#comment-17391605
]
liuyongpan edited comment on HDFS-16147 at 8/3/21, 11:13 AM:
-
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter , y{color}ou can get the
answer.
3、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}note:{color} first you must merge the patch HDFS-14617
You can simply verify as follows：
in class TestFSImageWithSnapshot , method : setUp , add such code:
{code:java}
public void setUp() throws Exception {
conf = new Configuration();
//*add**
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "3");
//**add*
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION)
.build();
cluster.waitActive();
fsn = cluster.getNamesystem();
hdfs = cluster.getFileSystem();
}
{code}
In class TestOfflineImageViewer , method createOriginalFSImage, change as
follow:
{code:java}
class FSImageFormatProtobuf, method loadInternal
case INODE: {
currentStep = new Step(StepType.INODES);
prog.beginStep(Phase.LOADING_FSIMAGE, currentStep);
stageSubSections = getSubSectionsOfName(
subSections, SectionName.INODE_SUB);
// if (loadInParallel && (stageSubSections.size() > 0)) {
//inodeLoader.loadINodeSectionInParallel(executorService,
//stageSubSections, summary.getCodec(), prog, currentStep);
// } else {
//inodeLoader.loadINodeSection(in, prog, currentStep);
// }
inodeLoader.loadINodeSection(in, prog, currentStep);
}
{code}
then run test unit {color:#ffc66d}testSaveLoadImage , you can get the
answer.{color}
{color:#172b4d}3、50% improvement measured against a compressed single threaded
load verses parallel compressed loading. {color}
{color:#172b4d}An FsImage is generated, before compression it is
{color:#172b4d} 27.18M{color} , after compression it is
{color:#172b4d}128M{color}. A simple comparisons were made in table.
{color}
||state||ave loading time||
|{color:#33} compress and parallel{color}| 7.5 sec|
|{color:#33} compress and unparallel{color}| 9.5sec|
|{color:#33} uncompress + parallel{color}| 6.5sec|
* {color:#313131}In fact*，In fact, loading fsimage with uncompress and
parallel will be faster.*{color}
* **
was (Author: mofei):
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSCo
[jira] [Comment Edited] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391605#comment-17391605
]
liuyongpan edited comment on HDFS-16147 at 8/3/21, 11:21 AM:
-
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter , y{color}ou can get the
answer.
3、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}note:{color} first you must merge the patch HDFS-14617
You can simply verify as follows：
in class TestFSImageWithSnapshot , method : setUp , add such code:
{code:java}
public void setUp() throws Exception {
conf = new Configuration();
//*add**
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "3");
//**add*
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION)
.build();
cluster.waitActive();
fsn = cluster.getNamesystem();
hdfs = cluster.getFileSystem();
}
{code}
In class TestOfflineImageViewer , method createOriginalFSImage, change as
follow:
{code:java}
class FSImageFormatProtobuf, method loadInternal
case INODE: {
currentStep = new Step(StepType.INODES);
prog.beginStep(Phase.LOADING_FSIMAGE, currentStep);
stageSubSections = getSubSectionsOfName(
subSections, SectionName.INODE_SUB);
// if (loadInParallel && (stageSubSections.size() > 0)) {
//inodeLoader.loadINodeSectionInParallel(executorService,
//stageSubSections, summary.getCodec(), prog, currentStep);
// } else {
//inodeLoader.loadINodeSection(in, prog, currentStep);
// }
inodeLoader.loadINodeSection(in, prog, currentStep);
}
{code}
then run test unit {color:#ffc66d}testSaveLoadImage , you can get the
answer.{color}
{color:#172b4d}3、50% improvement measured against a compressed single threaded
load verses parallel compressed loading. {color}
{color:#172b4d}An FsImage is generated, before compression it is 27.18M{color}
, after compression it is {color:#172b4d}128M{color}. A simple comparisons
were made in table.
||state||ave loading time||
|compress and parallel|7.5sec|
|compress and unparallel|9.5sec|
|uncompress and parallel|6.5sec|
{color:#313131}In fact loading fsimage with uncompress and parallel will be
faster than compress and parallel. {color}
{color:#313131}As disscussed in HDFS-1435 , compressed fsimage is
necessary.{color}
was (Author: mofei):
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfi
[jira] [Comment Edited] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391605#comment-17391605
]
liuyongpan edited comment on HDFS-16147 at 8/3/21, 11:26 AM:
-
[~sodonnell], I have tested your question carefully, and here is my answer.
1、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code to make a contrast:{color}
{color:#de350b}{color}first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "2");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "2");
{code}
run test unit {color:#ffc66d}testPBDelimitedWriter ,
{color:#172b4d}y{color}{color}{color:#172b4d}ou c{color}an get the answer.
2、If I create a parallel compressed image with this patch, and then try to
load it in a NN without this patch and parallel loading disabled, the NN still
able to load it.
{color:#de350b}*{color} first you must merge the patch HDFS-14617
You can simply verify as follows：
in class TestFSImageWithSnapshot , method : setUp , add such code, to make you
save fsImage with parallel and compressed:
{code:java}
public void setUp() throws Exception {
conf = new Configuration();
//*add**
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_KEY,
"org.apache.hadoop.io.compress.GzipCodec");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_LOAD_KEY, "true");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_INODE_THRESHOLD_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY, "3");
conf.set(DFSConfigKeys.DFS_IMAGE_PARALLEL_THREADS_KEY, "3");
//**add*
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION)
.build();
cluster.waitActive();
fsn = cluster.getNamesystem();
hdfs = cluster.getFileSystem();
}
{code}
In class TestOfflineImageViewer , method createOriginalFSImage, change as
follow, to make it run single thread.
{code:java}
class FSImageFormatProtobuf, method loadInternal
case INODE: {
currentStep = new Step(StepType.INODES);
prog.beginStep(Phase.LOADING_FSIMAGE, currentStep);
stageSubSections = getSubSectionsOfName(
subSections, SectionName.INODE_SUB);
// if (loadInParallel && (stageSubSections.size() > 0)) {
//inodeLoader.loadINodeSectionInParallel(executorService,
//stageSubSections, summary.getCodec(), prog, currentStep);
// } else {
//inodeLoader.loadINodeSection(in, prog, currentStep);
// }
inodeLoader.loadINodeSection(in, prog, currentStep);
}
{code}
then run test unit {color:#ffc66d}testSaveLoadImage , you can get the
answer.{color}
{color:#172b4d}3、50% improvement measured against a compressed single threaded
load verses parallel compressed loading. {color}
{color:#172b4d}An FsImage is generated, before compression it is 27.18M{color}
, after compression it is {color:#172b4d}128M{color}. A simple comparisons
were made in table.
||state||ave loading time||
|compress and parallel|7.5sec|
|compress and unparallel|9.5sec|
|uncompress and parallel|6.5sec|
{color:#313131}In fact loading fsimage with uncompress and parallel will
be faster than compress and parallel. {color}{color:#313131}As disscussed in
HDFS-1435 , compressed fsimage is necessary.{color}
3、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
was (Author: mofei):
[~sodonnell], I have tested your question carefully, and here is my answer.
1、HDFS-16147.002.patch fix the error of test unit at
org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled
2、Upon careful examination, oiv can indeed work normally, and I can't explain
why it works.
You can simply verify as follows：
In class TestOfflineImageViewer , method{color:#172b4d} createOriginalFSImage,
add and remove such code , make a contrast:{color}
{color:#de350b}note:{color} first get my patch HDFS-16147.002.patch !
{code:java}
// turn on both parallelization and compression
conf.setBoolean(DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY, true);
conf.set(DFSConfigKeys.DFS_IMAGE_COMPRESSION_CODEC_
[jira] [Work logged] (HDFS-16143) TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
[
https://issues.apache.org/jira/browse/HDFS-16143?focusedWorklogId=632856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632856
]
ASF GitHub Bot logged work on HDFS-16143:
-
Author: ASF GitHub Bot
Created on: 03/Aug/21 11:51
Start Date: 03/Aug/21 11:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3235:
URL: https://github.com/apache/hadoop/pull/3235#issuecomment-891783059
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 345m 29s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3235/24/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 439m 47s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockScanner |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3235/24/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3235 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux be6f0ffbf5bd 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d5d2db768dae0f44972b249942e71030bd6dfc6f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job
[jira] [Work logged] (HDFS-16129) HttpFS signature secret file misusage
[
https://issues.apache.org/jira/browse/HDFS-16129?focusedWorklogId=632870&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632870
]
ASF GitHub Bot logged work on HDFS-16129:
-
Author: ASF GitHub Bot
Created on: 03/Aug/21 12:11
Start Date: 03/Aug/21 12:11
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3209:
URL: https://github.com/apache/hadoop/pull/3209#issuecomment-891795900
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 1s | | trunk passed |
| +1 :green_heart: | compile | 22m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 5s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 20m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 32s | | the patch passed |
| +1 :green_heart: | compile | 18m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 31s | | root: The patch generated
0 new + 97 unchanged - 2 fixed = 97 total (was 99) |
| +1 :green_heart: | mvnsite | 3m 17s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 52s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 16s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/7/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-httpfs.html)
| hadoop-hdfs-project/hadoop-hdfs-httpfs generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 15m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 19s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | unit | 9m 43s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 219m 39s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-httpfs |
| |
org.apache.hadoop.fs.http.server.HttpFSAuthenticationFilter.CONF_PREFIXES
should be package protected At HttpFSAuthenticationFilter.java: At
HttpFSAuthenticationFilter.java:[line 54] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3209 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclien
[jira] [Work logged] (HDFS-16151) Improve the parameter comments related to ProtobufRpcEngine2#Server()
[
https://issues.apache.org/jira/browse/HDFS-16151?focusedWorklogId=632875&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632875
]
ASF GitHub Bot logged work on HDFS-16151:
-
Author: ASF GitHub Bot
Created on: 03/Aug/21 12:24
Start Date: 03/Aug/21 12:24
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on pull request #3256:
URL: https://github.com/apache/hadoop/pull/3256#issuecomment-891804326
It seems that some exceptions occurred during the UT process, resulting in
the UT not being executed.
Consider re-executing UT.
Then we look at the final execution result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 632875)
Time Spent: 0.5h (was: 20m)
> Improve the parameter comments related to ProtobufRpcEngine2#Server()
> -
>
> Key: HDFS-16151
> URL: https://issues.apache.org/jira/browse/HDFS-16151
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now missing some parameter comments related to ProtobufRpcEngine2#Server(),
> as follows:
> /**
> * Construct an RPC server.
> *
> * @param protocolClass the class of protocol
> * @param protocolImpl the protocolImpl whose methods will be called
> * @param conf the configuration to use
> * @param bindAddress the address to bind on to listen for connection
> * @param port the port to listen for connections on
> * @param numHandlers the number of method handler threads to run
> * @param verbose whether each call should be logged
> * @param portRangeConfig A config parameter that can be used to restrict
> * the range of ports used when port is 0 (an ephemeral port)
> * @param alignmentContext provides server state info on client responses
> */
> public Server(Class protocolClass, Object protocolImpl,
> Configuration conf, String bindAddress, int port, int numHandlers,
> int numReaders, int queueSizePerHandler, boolean verbose,
> SecretManager secretManager,
> String portRangeConfig, AlignmentContext alignmentContext)
> throws IOException {
> super(protocolClass, protocolImpl, conf, bindAddress, port, numHandlers,
> numReaders, queueSizePerHandler, verbose, secretManager,
> portRangeConfig, alignmentContext);
> }
> The description of numReaders, queueSizePerHandler, and secretManager is
> missing here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16146) All three replicas are lost due to not adding a new DataNode in time
[ https://issues.apache.org/jira/browse/HDFS-16146?focusedWorklogId=632965&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632965 ] ASF GitHub Bot logged work on HDFS-16146: - Author: ASF GitHub Bot Created on: 03/Aug/21 14:33 Start Date: 03/Aug/21 14:33 Worklog Time Spent: 10m Work Description: jojochuang commented on pull request #3247: URL: https://github.com/apache/hadoop/pull/3247#issuecomment-891897722 looks good. @Hexiaoqiao please go ahead and merge it. thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 632965) Time Spent: 1h 20m (was: 1h 10m) > All three replicas are lost due to not adding a new DataNode in time > > > Key: HDFS-16146 > URL: https://issues.apache.org/jira/browse/HDFS-16146 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > We have a three-replica file, and all replicas of a block are lost when the > default datanode replacement strategy is used. It happened like this: > 1. addBlock() applies for a new block and successfully connects three > datanodes (dn1, dn2 and dn3) to build a pipeline; > 2. Write data; > 3. dn1 has an error and was kicked out. At this time, the remaining datanodes > in the pipeline > 1, according to the replacement strategy, there is no need > to add a new datanode; > 4. After writing is completed, enter PIPELINE_CLOSE; > 5. dn2 has an error and was kicked out. But because it is already in the > close phase, addDatanode2ExistingPipeline() decides to hand over the task of > transfering the replica to the NameNode. At this time, there is only one > datanode left in the pipeline; > 6. dn3 error, all replicas are lost. > If we add a new datanode in step 5, we can avoid losing all replicas in this > case. I think error in PIPELINE_CLOSE and error in DATA_STREAMING have the > same risk of losing replicas, we should not skip adding a new datanode > during PIPELINE_CLOSE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16129) HttpFS signature secret file misusage
[
https://issues.apache.org/jira/browse/HDFS-16129?focusedWorklogId=632981&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-632981
]
ASF GitHub Bot logged work on HDFS-16129:
-
Author: ASF GitHub Bot
Created on: 03/Aug/21 14:56
Start Date: 03/Aug/21 14:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3209:
URL: https://github.com/apache/hadoop/pull/3209#issuecomment-891916492
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 56s | | trunk passed |
| +1 :green_heart: | compile | 22m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 3s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 39s | | the patch passed |
| +1 :green_heart: | compile | 20m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 30s | | the patch passed |
| +1 :green_heart: | compile | 18m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 32s | | root: The patch generated
0 new + 97 unchanged - 2 fixed = 97 total (was 99) |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 4s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 11s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 44s | | hadoop-kms in the patch passed.
|
| +1 :green_heart: | unit | 8m 51s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 216m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3209/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3209 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux b21da6338d4e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2c497078f45b93822a9b7e2e0c28278d1b585370 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
|
[jira] [Work logged] (HDFS-16146) All three replicas are lost due to not adding a new DataNode in time
[ https://issues.apache.org/jira/browse/HDFS-16146?focusedWorklogId=633028&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633028 ] ASF GitHub Bot logged work on HDFS-16146: - Author: ASF GitHub Bot Created on: 03/Aug/21 16:22 Start Date: 03/Aug/21 16:22 Worklog Time Spent: 10m Work Description: Hexiaoqiao merged pull request #3247: URL: https://github.com/apache/hadoop/pull/3247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 633028) Time Spent: 1.5h (was: 1h 20m) > All three replicas are lost due to not adding a new DataNode in time > > > Key: HDFS-16146 > URL: https://issues.apache.org/jira/browse/HDFS-16146 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > We have a three-replica file, and all replicas of a block are lost when the > default datanode replacement strategy is used. It happened like this: > 1. addBlock() applies for a new block and successfully connects three > datanodes (dn1, dn2 and dn3) to build a pipeline; > 2. Write data; > 3. dn1 has an error and was kicked out. At this time, the remaining datanodes > in the pipeline > 1, according to the replacement strategy, there is no need > to add a new datanode; > 4. After writing is completed, enter PIPELINE_CLOSE; > 5. dn2 has an error and was kicked out. But because it is already in the > close phase, addDatanode2ExistingPipeline() decides to hand over the task of > transfering the replica to the NameNode. At this time, there is only one > datanode left in the pipeline; > 6. dn3 error, all replicas are lost. > If we add a new datanode in step 5, we can avoid losing all replicas in this > case. I think error in PIPELINE_CLOSE and error in DATA_STREAMING have the > same risk of losing replicas, we should not skip adding a new datanode > during PIPELINE_CLOSE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16146) All three replicas are lost due to not adding a new DataNode in time
[ https://issues.apache.org/jira/browse/HDFS-16146?focusedWorklogId=633029&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633029 ] ASF GitHub Bot logged work on HDFS-16146: - Author: ASF GitHub Bot Created on: 03/Aug/21 16:24 Start Date: 03/Aug/21 16:24 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on pull request #3247: URL: https://github.com/apache/hadoop/pull/3247#issuecomment-891986382 Committed to trunk. Thanks @zhangshuyan0 for your works. Thanks @jojochuang for your reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 633029) Time Spent: 1h 40m (was: 1.5h) > All three replicas are lost due to not adding a new DataNode in time > > > Key: HDFS-16146 > URL: https://issues.apache.org/jira/browse/HDFS-16146 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > We have a three-replica file, and all replicas of a block are lost when the > default datanode replacement strategy is used. It happened like this: > 1. addBlock() applies for a new block and successfully connects three > datanodes (dn1, dn2 and dn3) to build a pipeline; > 2. Write data; > 3. dn1 has an error and was kicked out. At this time, the remaining datanodes > in the pipeline > 1, according to the replacement strategy, there is no need > to add a new datanode; > 4. After writing is completed, enter PIPELINE_CLOSE; > 5. dn2 has an error and was kicked out. But because it is already in the > close phase, addDatanode2ExistingPipeline() decides to hand over the task of > transfering the replica to the NameNode. At this time, there is only one > datanode left in the pipeline; > 6. dn3 error, all replicas are lost. > If we add a new datanode in step 5, we can avoid losing all replicas in this > case. I think error in PIPELINE_CLOSE and error in DATA_STREAMING have the > same risk of losing replicas, we should not skip adding a new datanode > during PIPELINE_CLOSE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16146) All three replicas are lost due to not adding a new DataNode in time
[ https://issues.apache.org/jira/browse/HDFS-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-16146: --- Issue Type: Improvement (was: Bug) > All three replicas are lost due to not adding a new DataNode in time > > > Key: HDFS-16146 > URL: https://issues.apache.org/jira/browse/HDFS-16146 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsclient >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We have a three-replica file, and all replicas of a block are lost when the > default datanode replacement strategy is used. It happened like this: > 1. addBlock() applies for a new block and successfully connects three > datanodes (dn1, dn2 and dn3) to build a pipeline; > 2. Write data; > 3. dn1 has an error and was kicked out. At this time, the remaining datanodes > in the pipeline > 1, according to the replacement strategy, there is no need > to add a new datanode; > 4. After writing is completed, enter PIPELINE_CLOSE; > 5. dn2 has an error and was kicked out. But because it is already in the > close phase, addDatanode2ExistingPipeline() decides to hand over the task of > transfering the replica to the NameNode. At this time, there is only one > datanode left in the pipeline; > 6. dn3 error, all replicas are lost. > If we add a new datanode in step 5, we can avoid losing all replicas in this > case. I think error in PIPELINE_CLOSE and error in DATA_STREAMING have the > same risk of losing replicas, we should not skip adding a new datanode > during PIPELINE_CLOSE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16146) All three replicas are lost due to not adding a new DataNode in time
[ https://issues.apache.org/jira/browse/HDFS-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16146. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk. Thanks @zhangshuyan0 for your works. Thanks @jojochuang for your reviews. > All three replicas are lost due to not adding a new DataNode in time > > > Key: HDFS-16146 > URL: https://issues.apache.org/jira/browse/HDFS-16146 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We have a three-replica file, and all replicas of a block are lost when the > default datanode replacement strategy is used. It happened like this: > 1. addBlock() applies for a new block and successfully connects three > datanodes (dn1, dn2 and dn3) to build a pipeline; > 2. Write data; > 3. dn1 has an error and was kicked out. At this time, the remaining datanodes > in the pipeline > 1, according to the replacement strategy, there is no need > to add a new datanode; > 4. After writing is completed, enter PIPELINE_CLOSE; > 5. dn2 has an error and was kicked out. But because it is already in the > close phase, addDatanode2ExistingPipeline() decides to hand over the task of > transfering the replica to the NameNode. At this time, there is only one > datanode left in the pipeline; > 6. dn3 error, all replicas are lost. > If we add a new datanode in step 5, we can avoid losing all replicas in this > case. I think error in PIPELINE_CLOSE and error in DATA_STREAMING have the > same risk of losing replicas, we should not skip adding a new datanode > during PIPELINE_CLOSE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16146) All three replicas are lost due to not adding a new DataNode in time
[ https://issues.apache.org/jira/browse/HDFS-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-16146: --- Component/s: (was: hdfs) (was: datanode) dfsclient > All three replicas are lost due to not adding a new DataNode in time > > > Key: HDFS-16146 > URL: https://issues.apache.org/jira/browse/HDFS-16146 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We have a three-replica file, and all replicas of a block are lost when the > default datanode replacement strategy is used. It happened like this: > 1. addBlock() applies for a new block and successfully connects three > datanodes (dn1, dn2 and dn3) to build a pipeline; > 2. Write data; > 3. dn1 has an error and was kicked out. At this time, the remaining datanodes > in the pipeline > 1, according to the replacement strategy, there is no need > to add a new datanode; > 4. After writing is completed, enter PIPELINE_CLOSE; > 5. dn2 has an error and was kicked out. But because it is already in the > close phase, addDatanode2ExistingPipeline() decides to hand over the task of > transfering the replica to the NameNode. At this time, there is only one > datanode left in the pipeline; > 6. dn3 error, all replicas are lost. > If we add a new datanode in step 5, we can avoid losing all replicas in this > case. I think error in PIPELINE_CLOSE and error in DATA_STREAMING have the > same risk of losing replicas, we should not skip adding a new datanode > during PIPELINE_CLOSE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16055) Quota is not preserved in snapshot INode
[
https://issues.apache.org/jira/browse/HDFS-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-16055:
-
Fix Version/s: 3.3.2
> Quota is not preserved in snapshot INode
>
>
> Key: HDFS-16055
> URL: https://issues.apache.org/jira/browse/HDFS-16055
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Quota feature is not preserved during snapshot creation, this causes
> {{INodeDirectory#metadataEquals}} to ALWAYS return true. Therefore,
> {{snapshotDiff}} will ALWAYS return the snapshot root as modified, even if
> the quota is set before the snapshot creation:
> {code:bash}
> $ hdfs snapshotDiff /diffTest s0 .
> Difference between snapshot s0 and current directory under directory
> /diffTest:
> M .
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16055) Quota is not preserved in snapshot INode
[
https://issues.apache.org/jira/browse/HDFS-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDFS-16055:
-
Fix Version/s: 3.2.3
> Quota is not preserved in snapshot INode
>
>
> Key: HDFS-16055
> URL: https://issues.apache.org/jira/browse/HDFS-16055
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Quota feature is not preserved during snapshot creation, this causes
> {{INodeDirectory#metadataEquals}} to ALWAYS return true. Therefore,
> {{snapshotDiff}} will ALWAYS return the snapshot root as modified, even if
> the quota is set before the snapshot creation:
> {code:bash}
> $ hdfs snapshotDiff /diffTest s0 .
> Difference between snapshot s0 and current directory under directory
> /diffTest:
> M .
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14189) Fix intermittent failure of TestNameNodeMetrics
[
https://issues.apache.org/jira/browse/HDFS-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-14189:
-
Fix Version/s: 3.2.3
Description:
{noformat}
java.lang.AssertionError: Bad value for metric LowRedundancyBlocks expected:<1>
but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
at
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:488)
{noformat}
was:
{noformat}
java.lang.AssertionError: Bad value for metric LowRedundancyBlocks expected:<1>
but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
at
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:488)
{noformat}
Backported to branch-3.2.
> Fix intermittent failure of TestNameNodeMetrics
> ---
>
> Key: HDFS-14189
> URL: https://issues.apache.org/jira/browse/HDFS-14189
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0, 3.2.3
>
> Attachments: HDFS-14189-01.patch
>
>
> {noformat}
> java.lang.AssertionError: Bad value for metric LowRedundancyBlocks
> expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.test.MetricsAsserts.assertGauge(MetricsAsserts.java:189)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics.testCorruptBlock(TestNameNodeMetrics.java:488)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
