[jira] [Commented] (HDFS-8381) Reduce time taken for complete HDFS unit test run
[
https://issues.apache.org/jira/browse/HDFS-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544341#comment-14544341
]
Arpit Agarwal commented on HDFS-8381:
-
Thanks, HDFS-8403 fixes {{TestFileCreation#testOverwriteOpenForWrite}}.
> Reduce time taken for complete HDFS unit test run
> -
>
> Key: HDFS-8381
> URL: https://issues.apache.org/jira/browse/HDFS-8381
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Arpit Agarwal
>
> HDFS unit tests take a long time to run. Our unit tests are more like
> system/integration tests since we spin up a MiniDFSCluster for individual
> test cases. A number of tests have sleeps which further adds to the run time.
> A better option is to use more fine-grained unit tests specific to individual
> classes. I did not find any existing Jiras for this so filing one to track
> this work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8381) Reduce time taken for complete HDFS unit test run
[
https://issues.apache.org/jira/browse/HDFS-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544186#comment-14544186
]
Haohui Mai commented on HDFS-8381:
--
Taking a quick look of Jenkins build, some of the tests takes long time and
might be worth some investigation:
{noformat}
TestFileCreation.testOverwriteOpenForWrite 5 min 0 sec
TestLeaseRecovery2 2 min 54 sec0 0 7
+7 7 +7
TestDatanodeDeath 2 min 36 sec0 0 4
4
TestDFSClientRetries2 min 16 sec0 0 11
11
TestDecommission2 min 7 sec 0 1 14
15
TestEncryptedTransfer 1 min 48 sec0 0 26
+26 26 +26
TestPread 1 min 40 sec0 0 7 +7
7 +7
TestRollingUpgrade 1 min 39 sec0 0 9
+9 9 +9
TestDFSStorageStateRecovery 1 min 3 sec 0 0
3 +3 3 +3
TestFileCreationClient
{noformat}
That contributes around 5% of the running time of the jenkins build.
> Reduce time taken for complete HDFS unit test run
> -
>
> Key: HDFS-8381
> URL: https://issues.apache.org/jira/browse/HDFS-8381
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Arpit Agarwal
>
> HDFS unit tests take a long time to run. Our unit tests are more like
> system/integration tests since we spin up a MiniDFSCluster for individual
> test cases. A number of tests have sleeps which further adds to the run time.
> A better option is to use more fine-grained unit tests specific to individual
> classes. I did not find any existing Jiras for this so filing one to track
> this work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8381) Reduce time taken for complete HDFS unit test run
[ https://issues.apache.org/jira/browse/HDFS-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541241#comment-14541241 ] Sean Busbey commented on HDFS-8381: --- It really helped the HBase builds when we categorized tests by run time and functional component. Then it was much easier to decide what should run for default dev builds and various jenkins jobs. > Reduce time taken for complete HDFS unit test run > - > > Key: HDFS-8381 > URL: https://issues.apache.org/jira/browse/HDFS-8381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Arpit Agarwal > > HDFS unit tests take a long time to run. Our unit tests are more like > system/integration tests since we spin up a MiniDFSCluster for individual > test cases. A number of tests have sleeps which further adds to the run time. > A better option is to use more fine-grained unit tests specific to individual > classes. I did not find any existing Jiras for this so filing one to track > this work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8381) Reduce time taken for complete HDFS unit test run
[ https://issues.apache.org/jira/browse/HDFS-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541044#comment-14541044 ] Allen Wittenauer commented on HDFS-8381: We've talked about enabling parallel tests (mvn -Pparallel-tests) in various test-patch.sh jiras and discussions. Some work was done on making sure that worked a year or two ago (HADOOP-9287), so it might not be that bad to make sure it still works. If it does, we can add that to the mvn command line and see how that helps the time. > Reduce time taken for complete HDFS unit test run > - > > Key: HDFS-8381 > URL: https://issues.apache.org/jira/browse/HDFS-8381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Arpit Agarwal > > HDFS unit tests take a long time to run. Our unit tests are more like > system/integration tests since we spin up a MiniDFSCluster for individual > test cases. A number of tests have sleeps which further adds to the run time. > A better option is to use more fine-grained unit tests specific to individual > classes. I did not find any existing Jiras for this so filing one to track > this work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8381) Reduce time taken for complete HDFS unit test run
[ https://issues.apache.org/jira/browse/HDFS-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541023#comment-14541023 ] Arpit Agarwal commented on HDFS-8381: - A potential approach is identifying the worst offenders and fixing them one at a time. > Reduce time taken for complete HDFS unit test run > - > > Key: HDFS-8381 > URL: https://issues.apache.org/jira/browse/HDFS-8381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: Arpit Agarwal > > HDFS unit tests take a long time to run. Our unit tests are more like > system/integration tests since we spin up a MiniDFSCluster for individual > test cases. A number of tests have sleeps which further adds to the run time. > A better option is to use more fine-grained unit tests specific to individual > classes. I did not find any existing Jiras for this so filing one to track > this work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
