[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9263:
Status: Patch Available (was: Open)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9263:
Status: Open (was: Patch Available)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-9263:
Attachment: HDFS-9263-003.patch
Attaching the Updated patch.
Replaced all occurances of "test.build.data" with GenericTestUtils
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch,
> HDFS-9263-003.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-9263:
-
Target Version/s: 3.0.0, 2.7.3, 2.6.5 (was: 3.0.0, 2.7.3, 2.6.4)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-9263:
-
Target Version/s: 3.0.0, 2.7.3, 2.6.4 (was: 3.0.0, 2.7.3, 2.6.3)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-9263:
-
Status: Patch Available (was: Open)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-9263:
-
Status: Open (was: Patch Available)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated HDFS-9263:
--
Target Version/s: 3.0.0, 2.7.3, 2.6.3 (was: 3.0.0, 2.6.2, 2.7.3)
Targeting 2.6.3 now that 2.6.2 has shipped.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HDFS-9263:
--
Target Version/s: 3.0.0, 2.6.2, 2.7.3
We should backport this to maintenance releases too to keep the test-runs in
sync.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9263:
Attachment: HDFS-9263-002.patch
[~ste...@apache.org], regarding the side discussion on HADOOP-11880, I have
traced the problem to {{TestMiniDFSCluster}}, and the problem only occurs while
running with the HDFS-9263 patch applied. I hope you don't mind, but I'm
attaching a v002 patch with a small modification to fix it.
My only change is in {{GenericTestUtils}}. Prepare to smack forehead. Here is
the patch v001 code:
{code}
public static final String DEFAULT_TEST_DATA_DIR =
"target " + File.pathSeparator + "test" + File.pathSeparator + "data";
{code}
Here is my change in v002:
{code}
public static final String DEFAULT_TEST_DATA_DIR =
"target" + File.separator + "test" + File.separator + "data";
{code}
I removed the extra space character at the end of the "target" string literal,
and I switched from {{File.pathSeparator}} (i.e. classpath separator, ':' on
*nixes) to {{File.separator}} (i.e.file system path separator, '/' on *nixes).
I constantly mix up those 2 myself. I wish they had clearer names.
As to why {{TestMiniDFSCluster}} exposed this, one of the tests in that suite
specifically removes the {{test.build.data}} property to check if the
mini-cluster can still start using defaults. After running that test suite, I
could see it created the funny paths containing spaces and colons. For code
that passes the path through a {{URI}}, it would end up encoding the space to
%20 too.
bq. If we not only consolidate test dir setup, but do it in a way that isolates
it for each test suite, we get that isolation.
I'm on board with the consolidation aspect, but it's still unclear to me that
there is a benefit of adding another random string into the path. I suppose if
the sub-directory was named to match the test suite, then that has some benefit
for post-mortem analysis after a test failure. You could go back and inspect
metadata and blocks, and you'd know that you were looking at files specific to
that test suite.
OTOH, this has the side effect of using many more directories, and they won't
get cleaned up in between runs of different suites. Typically, the data gets
wiped between suite runs, either explicitly via {{FileUtil#fullyDelete}}, or
implicitly via things like a NameNode format. I tried a full test run of
hadoop-hdfs, and then I saw this:
{code}
> du -hs ~/git/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/target/test/data
6.1G
/home/cnauroth/git/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/target/test/data
{code}
That's more disk consumption than I'm used to seeing from a test run. I'm
pretty sure I'd need to reallocate volumes on some of my wimpier VMs to
accommodate this.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch, HDFS-9263-002.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-9263:
---
Assignee: Steve Loughran (was: Jagadesh Kiran N)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-9263:
-
Status: Patch Available (was: Open)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Jagadesh Kiran N
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-9263:
-
Attachment: HDFS-9263-001.patch
This covers -common and a couple of other places too, but it's where HDFS did
the most
# uses of test.build.data have been unified
# many buildups of test dirs now use something random, rather than a hard-coded
path like "dfs". This includes minidfs cluster...which should improve
parallelism on test runs.
# I've kept the bits of code that feed the paths into {{Path}} constructors
independent of the ones that want {{File}} instances. Why? Not sure what
happens in windows and whether the path stuff really wants forward-slash paths
(but then...what happens if maven has set test.build.data to an absolute path
in the local FS?)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Jagadesh Kiran N
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
