[
https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770158#comment-16770158
]
Allen Wittenauer edited comment on HDDS-891 at 2/16/19 5:33 PM:
----------------------------------------------------------------
I already answered your Yetus question: if ozone needs special handling for its
submodules, it needs to go into the hadoop personality. Cutting back on
options just because you think they don't apply to Ozone is almost entirely
incorrect. It's still part of Hadoop and it's still very possible for people
to modify core Hadoop through it. e.g., HDDS-1115, filed by you, does exactly
that.
The project doesn't have enough people working on it anymore to have wildly
different build characteristics. The fewer variances between Hadoop
subprojects the easier it is for the handful of people still working on it.
HADOOP-16035 really drives this point home: there is, without a significant
amount of hacks or\-\-as the PR jobs you've already got setup
demonstrate\-\-massive confusion, only one precommit for github PRs for all
source code sitting in the hadoop tree. Never mind that users running
test-patch locally aren't going to know to use some oddball personality file
that is off to the side.
As to the various bugs introduced by HDDS-146... well, [~anu] should never have
+1'd without a precommit run and it should definitely have not been committed
without it. The shellcheck errors are through the roof. If it had been run or
Ozone was paying attention to the nightly qbt runs, the errors are all laid out
there. By far, the most problematic code is the use of unquoted variables for
command line arguments. TBF, it's a common pitfall that a lot of inexperienced
shell programmers hit, but given that we have tools in our build system to
specifically point these types of errors out, there's little excuse for _new_
code to have this problem.
[In hindsight, I should have incompatibility broken HADOOP_OPTS and friends in
3.x but it is what it is. It's been a bug in Hadoop since Doug copied the
original code from Tomcat or Maven or whatever. Meanwhile, some users are
pretty much required to do a lot of gymnastics to make the system work because
of that long long long standing bug.]
was (Author: aw):
I already answered your Yetus question: if ozone needs special handling for its
submodules, it needs to go into the hadoop personality. Cutting back on
options just because you think they don't apply to Ozone is almost entirely
incorrect. It's still part of Hadoop and it's still very possible for people
to modify core Hadoop through it. e.g., HDDS-1115, filed by you, does exactly
that.
The project doesn't have enough people working on it anymore to have wildly
different build characteristics. The fewer variances between Hadoop
subprojects the easier it is for the handful of people still working on it.
HADOOP-16035 really drives this point home: there is, without a significant
amount of hacks or--as the PR jobs you've already got setup
demonstrate--massive confusion, only one precommit for github PRs for all
source code sitting in the hadoop tree. Never mind that users running
test-patch locally aren't going to know to use some oddball personality file
that is off to the side.
As to the various bugs introduced by HDDS-146... well, [~anu] should never have
+1'd without a precommit run and it should definitely have not been committed
without it. The shellcheck errors are through the roof. If it had been run or
Ozone was paying attention to the nightly qbt runs, the errors are all laid out
there. By far, the most problematic code is the use of unquoted variables for
command line arguments. TBF, it's a common pitfall that a lot of inexperienced
shell programmers hit, but given that we have tools in our build system to
specifically point these types of errors out, there's little excuse for _new_
code to have this problem.
[In hindsight, I should have incompatibility broken HADOOP_OPTS and friends in
3.x but it is what it is. It's been a bug in Hadoop since Doug copied the
original code from Tomcat or Maven or whatever. Meanwhile, some users are
pretty much required to do a lot of gymnastics to make the system work because
of that long long long standing bug.]
> Create customized yetus personality for ozone
> ---------------------------------------------
>
> Key: HDDS-891
> URL: https://issues.apache.org/jira/browse/HDDS-891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Elek, Marton
> Assignee: Elek, Marton
> Priority: Major
>
> Ozone pre commit builds (such as
> https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop
> personality from the yetus personality.
> Yetus personalities are bash scripts which contain personalization for
> specific builds.
> The hadoop personality tries to identify which project should be built and
> use partial build to build only the required subprojects because the full
> build is very time consuming.
> But in Ozone:
> 1.) The build + unit tests are very fast
> 2.) We don't need all the checks (for example the hadoop specific shading
> test)
> 3.) We prefer to do a full build and full unit test for hadoop-ozone and
> hadoop-hdds subrojects (for example the hadoop-ozone integration test always
> should be executed as it contains many generic unit test)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
