[jira] [Created] (HADOOP-13152) Fix dynamic subcommands error on multiple arguments

[Masatake Iwasaki (JIRA)]

Masatake Iwasaki created HADOOP-13152:
-

 Summary: Fix dynamic subcommands error on multiple arguments
 Key: HADOOP-13152
 URL: https://issues.apache.org/jira/browse/HADOOP-13152
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: HADOOP-12930
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

[Masatake Iwasaki]

Hi Allen,

I had a problem when I was testing HADOOP-12930 branch. (built by mvn 
package -Pdist)


Single argument is ok.::

  [iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo
  foo


Error on multiple arguments.  ::

  [iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo bar
  bin/hadoop: line 200: foo bar: syntax error in expression (error 
token is "bar")

  Error: Could not find or load main class

  [iwasakims@centos7 HADOOP-12930]$ bin/hadoop distcp /foo /bar
  bin/hadoop: line 200: /foo /bar: syntax error: operand expected 
(error token is "/foo /bar")

  Error: Could not find or load main class


The fix of HADOOP-13120 seems to be related.::

  diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop 
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop

  index 61fdc2e..0f6982b 100755
  --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
  +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
  @@ -197,6 +197,7 @@ shift
   HADOOP_SUBCMD_ARGS=("$@")

   if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
  +  hadoop_debug "Calling dynamically: 
hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}"

 "hadoop_subcommand_${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
   else
 hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"


Thanks,
Masatake Iwasaki


On 5/15/16 02:33, Allen Wittenauer wrote:

This vote closes in 2 days and the only response has been from a non-committer 
and one of the 137 other committers on the project…. it’d be great if some 
others could take a look.

Thanks!


On May 12, 2016, at 6:07 PM, Andrew Wang  wrote:

+1. I looked at the patches on the branch, wasn't too bad to review. As
Allen said, there's some code movement, assorted other nice doc and shell
fixups.

Found one extra typo, which I added to HADOOP-13129.

Best,
Andrew

On Wed, May 11, 2016 at 1:14 AM, Sean Busbey  wrote:


+1 (non-binding)

reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.

tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.

On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer  wrote:

Hey gang!

I’d like to call a vote to run for 7 days (ending May 16 at

13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
was developed exclusively by me as per the discussion two months ago as a
way to make what would be a rather large patch hopefully easier to review.
The vast majority of the branch is code movement in the same file,
additional license headers, maven assembly hooks for distribution, and
variable renames. Not a whole lot of new code, but a big diff file
none-the-less.

This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’

commands to allow for subcommands to be added or modified at runtime.  This
allows for individual users or entire sites to tweak the execution
environment to suit their local needs.  For example, it has been a practice
for some locations to change the distcp jar out for a custom one.  Using
this functionality, it is possible that the ‘hadoop distcp’ command could
run the local version without overwriting the bundled jar and for existing
documentation (read: results from Internet searches) to work as written
without modification. This has the potential to be a huge win, especially
for:

* advanced end users looking to supplement the Apache

Hadoop experience

* operations teams that may be able to leverage existing

documentation without having to remain local “exception” docs

* development groups wanting an easy way to trial

experimental features

Additionally, this branch includes the following, related

changes:

* Adds the first unit tests for the ‘hadoop’ command
* Adds the infrastructure for hdfs script testing and

the first unit test for the ‘hdfs’ command

* Modifies the hadoop-tools components to be dynamic

rather than hard coded

* Renames the shell profiles for hdfs, mapred, and yarn

to be consistent with other bundled profiles, including the ones introduced
in this branch

Documentation, including a ‘hello world’-style example, is in

the UnixShellGuide markdown file.  (Of course!)

 I am at ApacheCon this week if anyone wants to discuss in-depth.

Thanks!

P.S.,

There are still two open sub-tasks.  These are blocked by other

issues so that we may add unit testing to the shell code in those
respective areas.  I’ll covert to full issues after HADOOP-12930 is closed.


-
To unsubscribe, e-mail: 

[jira] [Created] (HADOOP-13151) Underscores should be escaped in dynamic subcommands document

[Akira AJISAKA (JIRA)]

Akira AJISAKA created HADOOP-13151:
--

 Summary: Underscores should be escaped in dynamic subcommands 
document
 Key: HADOOP-13151
 URL: https://issues.apache.org/jira/browse/HADOOP-13151
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Reporter: Akira AJISAKA
Priority: Minor


{noformat}
(scriptname)_subcommand_(subcommand)
{noformat}
The underscores should be escaped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Compatibility guidelines for toString overrides

[Chris Nauroth]

It sounds like what we're coming around to is that we're comfortable with this 
proposal at the Java API layer, but the potential for impact at the shell 
interface makes it overall uncomfortable to declare anything in the 
compatibility guidelines.  I think that's a fair assessment.

Steve's example of AclEntry#toString is very relevant here.  Here I've been 
arguing that we should -1 patches that rely on toString output for any 
externalized format (serialization, UI, shell, etc.).  I wrote or code reviewed 
all of the HDFS ACL code, and yet this got past me.  Diligence during code 
review is not infallible.  I have assigned HADOOP-13150 to myself to change the 
HDFS ACL commands to avoid dependencies on toString.

Let's try to avoid this in new bits of code and update existing code as we find 
unfortunate dependencies on toString.

--Chris Nauroth

From: Steve Loughran >
Date: Saturday, May 14, 2016 at 1:01 PM
To: Allen Wittenauer >
Cc: Chris Nauroth >, 
"common-dev@hadoop.apache.org" 
>
Subject: Re: Compatibility guidelines for toString overrides


On 14 May 2016, at 18:39, Allen Wittenauer 
> wrote:


On May 12, 2016, at 12:20 PM, Chris Nauroth 
> wrote:

Hello Allen,

The intent is not for this rule to override other compatibility rules,
such as the important CLI output rule.  It's also not intended to give us
free reign to change existing toString() implementations without due
diligence.  If a patch changes an existing toString() implementation that
already goes out to the shell or any other form of external serialization,
then the patch needs to be declined.  (I concur that relying on toString()
like this should never be done, and I'd encourage us to fix any
occurrences we find.)


The problem is that we as a community do a terrible job on following the 
compatibility guidelines when it comes to CLI output.  Because while this maybe 
true:

Instead, the intent is to advertise to Java API consumers that toString()
output may evolve freely, and therefore we recommend against writing Java
code that depends on that output format.

... what's going to happen is people are going to assume that because toString 
is now considered Unstable, all output will be considered Unstable by way of 
mental short cut.  There have been many, many instances over the past year or 
two where even long time PMC members have claimed CLI output was already 
Unstable/outside the scope of the compatibility guidelines and I just see this 
re-inforcing that (incorrect) world view.

HDFS-9732 is a good example of how to handle this.  I didn't explicitly -1
it, but I did call out the CLI compatibility problem and recommend how to
change the patch so that it preserves compatibility.

Does this help address your concerns, or is the full code audit the only
thing that would convince you to lift your -1?

No, because "perception is reality".  If we want to make toString Unstable, 
then we need to be very clear as to which toStrings are Unstable and which 
aren't.  This isn't a universal rule without doing some legwork.


I am painfully coming round to this point of view *on existing classes*


  1.  We need some interface "StableString" which we can tag to say "this is 
stable"; new CLI-ready code can use it.
  2.  We need to see where we use stuff today., by backchaining from uses of 
System.out & System.err, and from TableListing uses especially (which should 
implement StableString itself obviously).
  3.  Or: look at where toString() is subclassed and go from there...there's 
less than 800 such overrides. But here you can't tell where the use of a string 
is going to be. There are obvious ones (DU, DF, others which appear to be 
useful, AclEntry.toString(), but many which look like IDE-generated patterns 
for debug.


There's another tactic, which would be to add some explicit log diagnostics 
method which we offered no guarantees on ... but the challenge then becomes one 
of "how to use efficiently in logging". That could be addressed, equally 
painfully, by having our own log class (extending/mimicing slfj4j and which 
handles that message specially. That would be an absolute PITA to work with. I 
think I'd rather look at our command line use of toString values

Now, what about new bits of code?

[jira] [Created] (HADOOP-13150) Avoid use of toString() in output of HDFS ACL shell commands.

[Chris Nauroth (JIRA)]

Chris Nauroth created HADOOP-13150:
--

 Summary: Avoid use of toString() in output of HDFS ACL shell 
commands.
 Key: HADOOP-13150
 URL: https://issues.apache.org/jira/browse/HADOOP-13150
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Chris Nauroth
Assignee: Chris Nauroth


The HDFS ACL shell commands have at least one usage of the standard Java 
{{toString}} method to generate shell output ({{AclEntry#toString}}).  This 
issue tracks conversion of that code to use methods other than {{toString}}.  
The {{toString}} method is useful primarily for debugging.  It's preferrable to 
use a different method to ensure stable output for public interfaces, such as 
the shell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

[Chris Nauroth]

+1 (binding)

-Tried a dry-run merge of HADOOP-12930 to trunk.
-Successfully built distro on Windows.
-Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
commands through Cygwin.
-Reviewed documentation.

Allen, thank you for the contribution.  Would you please attach a full
patch to HADOOP-12930 to check pre-commit results?

While testing this, I discovered a bug in the distro build for Windows.
Could someone please code review my patch on HADOOP-13149?

--Chris Nauroth




On 5/9/16, 1:26 PM, "Allen Wittenauer"  wrote:

>
>   Hey gang!
>
>   I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>merge the HADOOP-12930 feature branch into trunk. This branch was
>developed exclusively by me as per the discussion two months ago as a way
>to make what would be a rather large patch hopefully easier to review.
>The vast majority of the branch is code movement in the same file,
>additional license headers, maven assembly hooks for distribution, and
>variable renames. Not a whole lot of new code, but a big diff file
>none-the-less.
>
>   This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>to allow for subcommands to be added or modified at runtime.  This allows
>for individual users or entire sites to tweak the execution environment
>to suit their local needs.  For example, it has been a practice for some
>locations to change the distcp jar out for a custom one.  Using this
>functionality, it is possible that the Œhadoop distcp¹ command could run
>the local version without overwriting the bundled jar and for existing
>documentation (read: results from Internet searches) to work as written
>without modification. This has the potential to be a huge win, especially
>for:
>   
>   * advanced end users looking to supplement the Apache Hadoop 
> experience
>   * operations teams that may be able to leverage existing 
> documentation
>without having to remain local ³exception² docs
>   * development groups wanting an easy way to trial experimental 
> features
>
>   Additionally, this branch includes the following, related changes:
>
>   * Adds the first unit tests for the Œhadoop¹ command
>   * Adds the infrastructure for hdfs script testing and the first 
> unit
>test for the Œhdfs¹ command
>   * Modifies the hadoop-tools components to be dynamic rather 
> than hard
>coded
>   * Renames the shell profiles for hdfs, mapred, and yarn to be
>consistent with other bundled profiles, including the ones introduced in
>this branch
>
>   Documentation, including a Œhello world¹-style example, is in the
>UnixShellGuide markdown file.  (Of course!)
>
>I am at ApacheCon this week if anyone wants to discuss in-depth.
>
>   Thanks!
>
>P.S.,
>
>   There are still two open sub-tasks.  These are blocked by other issues
>so that we may add unit testing to the shell code in those respective
>areas.  I¹ll covert to full issues after HADOOP-12930 is closed.
>
>
>-
>To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13149) Windows distro build fails on dist-copynativelibs.

[Chris Nauroth (JIRA)]

Chris Nauroth created HADOOP-13149:
--

 Summary: Windows distro build fails on dist-copynativelibs.
 Key: HADOOP-13149
 URL: https://issues.apache.org/jira/browse/HADOOP-13149
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Blocker


HADOOP-12892 pulled the dist-copynativelibs script into an external file.  The 
call to this script is failing when running a distro build on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13148) TestDistCpViewFs to include IOExceptions in test error reports

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-13148:
---

 Summary: TestDistCpViewFs to include IOExceptions in test error 
reports
 Key: HADOOP-13148
 URL: https://issues.apache.org/jira/browse/HADOOP-13148
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor


Minor test code style: all IOEs are caught and logged @ error, then test failed 
with a simple assert. This will lose all failure data from the XML test 
reports, hiding it in the logs.

Better to not catch the IOEs, just rethrow, and let JUnit handle them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1475

[Apache Jenkins Server]

See 

Changes:

[xgong] YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or

--
[...truncated 5581 lines...]
Running org.apache.hadoop.io.file.tfile.TestCompression
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.447 sec - in 
org.apache.hadoop.io.file.tfile.TestCompression
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.625 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.file.tfile.TestTFileSplit
Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.584 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.61 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.file.tfile.TestTFileComparator2
Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.254 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileComparator2
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.561 sec - in 
org.apache.hadoop.io.TestSequenceFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.file.tfile.TestTFileByteArrays
Running org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison
Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.521 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.file.tfile.TestVLong
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.349 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileSplit
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.158 sec - in 
org.apache.hadoop.io.file.tfile.TestVLong
Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.798 sec - in 
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
Running org.apache.hadoop.io.TestTextNonUTF8
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.251 sec - in 
org.apache.hadoop.io.TestTextNonUTF8
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestArrayWritable
Running org.apache.hadoop.io.erasurecode.rawcoder.TestXORRawCoder
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in 
org.apache.hadoop.io.TestArrayWritable
Running org.apache.hadoop.io.erasurecode.rawcoder.TestRSRawCoderLegacy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.38 sec - in 
org.apache.hadoop.io.erasurecode.rawcoder.TestXORRawCoder
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.erasurecode.rawcoder.TestDummyRawCoder
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.536 sec - in 
org.apache.hadoop.io.erasurecode.rawcoder.TestRSRawCoderLegacy
Running org.apache.hadoop.io.erasurecode.rawcoder.TestRSRawCoder
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.214 sec - in 
org.apache.hadoop.io.erasurecode.rawcoder.TestDummyRawCoder
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping
Running org.apache.hadoop.io.erasurecode.TestECSchema
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec - in 
org.apache.hadoop.io.erasurecode.TestECSchema
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 

Re: Compatibility guidelines for toString overrides

[Steve Loughran]

On 14 May 2016, at 18:39, Allen Wittenauer 
> wrote:


On May 12, 2016, at 12:20 PM, Chris Nauroth 
> wrote:

Hello Allen,

The intent is not for this rule to override other compatibility rules,
such as the important CLI output rule.  It's also not intended to give us
free reign to change existing toString() implementations without due
diligence.  If a patch changes an existing toString() implementation that
already goes out to the shell or any other form of external serialization,
then the patch needs to be declined.  (I concur that relying on toString()
like this should never be done, and I'd encourage us to fix any
occurrences we find.)


The problem is that we as a community do a terrible job on following the 
compatibility guidelines when it comes to CLI output.  Because while this maybe 
true:

Instead, the intent is to advertise to Java API consumers that toString()
output may evolve freely, and therefore we recommend against writing Java
code that depends on that output format.

… what’s going to happen is people are going to assume that because toString is 
now considered Unstable, all output will be considered Unstable by way of 
mental short cut.  There have been many, many instances over the past year or 
two where even long time PMC members have claimed CLI output was already 
Unstable/outside the scope of the compatibility guidelines and I just see this 
re-inforcing that (incorrect) world view.

HDFS-9732 is a good example of how to handle this.  I didn't explicitly -1
it, but I did call out the CLI compatibility problem and recommend how to
change the patch so that it preserves compatibility.

Does this help address your concerns, or is the full code audit the only
thing that would convince you to lift your -1?

No, because “perception is reality”.  If we want to make toString Unstable, 
then we need to be very clear as to which toStrings are Unstable and which 
aren’t.  This isn’t a universal rule without doing some legwork.


I am painfully coming round to this point of view *on existing classes*


  1.  We need some interface "StableString" which we can tag to say "this is 
stable"; new CLI-ready code can use it.
  2.  We need to see where we use stuff today., by backchaining from uses of 
System.out & System.err, and from TableListing uses especially (which should 
implement StableString itself obviously).
  3.  Or: look at where toString() is subclassed and go from there...there's 
less than 800 such overrides. But here you can't tell where the use of a string 
is going to be. There are obvious ones (DU, DF, others which appear to be 
useful, AclEntry.toString(), but many which look like IDE-generated patterns 
for debug.


There's another tactic, which would be to add some explicit log diagnostics 
method which we offered no guarantees on ... but the challenge then becomes one 
of "how to use efficiently in logging". That could be addressed, equally 
painfully, by having our own log class (extending/mimicing slfj4j and which 
handles that message specially. That would be an absolute PITA to work with. I 
think I'd rather look at our command line use of toString values

Now, what about new bits of code?

Re: Compatibility guidelines for toString overrides

[Allen Wittenauer]

> On May 12, 2016, at 12:20 PM, Chris Nauroth  wrote:
> 
> Hello Allen,
> 
> The intent is not for this rule to override other compatibility rules,
> such as the important CLI output rule.  It's also not intended to give us
> free reign to change existing toString() implementations without due
> diligence.  If a patch changes an existing toString() implementation that
> already goes out to the shell or any other form of external serialization,
> then the patch needs to be declined.  (I concur that relying on toString()
> like this should never be done, and I'd encourage us to fix any
> occurrences we find.)


The problem is that we as a community do a terrible job on following 
the compatibility guidelines when it comes to CLI output.  Because while this 
maybe true:

> Instead, the intent is to advertise to Java API consumers that toString()
> output may evolve freely, and therefore we recommend against writing Java
> code that depends on that output format.

… what’s going to happen is people are going to assume that because 
toString is now considered Unstable, all output will be considered Unstable by 
way of mental short cut.  There have been many, many instances over the past 
year or two where even long time PMC members have claimed CLI output was 
already Unstable/outside the scope of the compatibility guidelines and I just 
see this re-inforcing that (incorrect) world view.

> HDFS-9732 is a good example of how to handle this.  I didn't explicitly -1
> it, but I did call out the CLI compatibility problem and recommend how to
> change the patch so that it preserves compatibility.
> 
> Does this help address your concerns, or is the full code audit the only
> thing that would convince you to lift your -1?

No, because “perception is reality”.  If we want to make toString 
Unstable, then we need to be very clear as to which toStrings are Unstable and 
which aren’t.  This isn’t a universal rule without doing some legwork.



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

[Allen Wittenauer]

This vote closes in 2 days and the only response has been from a non-committer 
and one of the 137 other committers on the project…. it’d be great if some 
others could take a look.

Thanks!

> On May 12, 2016, at 6:07 PM, Andrew Wang  wrote:
> 
> +1. I looked at the patches on the branch, wasn't too bad to review. As
> Allen said, there's some code movement, assorted other nice doc and shell
> fixups.
> 
> Found one extra typo, which I added to HADOOP-13129.
> 
> Best,
> Andrew
> 
> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey  wrote:
> 
>> +1 (non-binding)
>> 
>> reviewed everything, filed an additional subtask for a very trivial
>> typo in the docs. should be fine to make a full issue after close and
>> then fix.
>> 
>> tried merging locally, tried running through new shell tests (both
>> with and without bats installed), tried making an example custom
>> command (valid and malformed). everything looks great.
>> 
>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer  wrote:
>>> 
>>>Hey gang!
>>> 
>>>I’d like to call a vote to run for 7 days (ending May 16 at
>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>> was developed exclusively by me as per the discussion two months ago as a
>> way to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>> 
>>>This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
>> commands to allow for subcommands to be added or modified at runtime.  This
>> allows for individual users or entire sites to tweak the execution
>> environment to suit their local needs.  For example, it has been a practice
>> for some locations to change the distcp jar out for a custom one.  Using
>> this functionality, it is possible that the ‘hadoop distcp’ command could
>> run the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>> 
>>>* advanced end users looking to supplement the Apache
>> Hadoop experience
>>>* operations teams that may be able to leverage existing
>> documentation without having to remain local “exception” docs
>>>* development groups wanting an easy way to trial
>> experimental features
>>> 
>>>Additionally, this branch includes the following, related
>> changes:
>>> 
>>>* Adds the first unit tests for the ‘hadoop’ command
>>>* Adds the infrastructure for hdfs script testing and
>> the first unit test for the ‘hdfs’ command
>>>* Modifies the hadoop-tools components to be dynamic
>> rather than hard coded
>>>* Renames the shell profiles for hdfs, mapred, and yarn
>> to be consistent with other bundled profiles, including the ones introduced
>> in this branch
>>> 
>>>Documentation, including a ‘hello world’-style example, is in
>> the UnixShellGuide markdown file.  (Of course!)
>>> 
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>> 
>>>Thanks!
>>> 
>>> P.S.,
>>> 
>>>There are still two open sub-tasks.  These are blocked by other
>> issues so that we may add unit testing to the shell code in those
>> respective areas.  I’ll covert to full issues after HADOOP-12930 is closed.
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>> 
>> 
>> 
>> 
>> --
>> busbey
>> 
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Different JIRA permissions for HADOOP and HDFS

[Allen Wittenauer]

> On May 14, 2016, at 7:07 AM, Zheng, Kai  wrote:
> 
> Hi,
>  
> Noticed this difference but not sure if it’s intended. YARN is similar with 
> HDFS. It’s not convenient. Any clarifying?


Under JIRA, different projects (e.g., HADOOP, YARN, MAPREDUCE, HDFS, 
YETUS, HBASE, ACCUMULO, etc) may have different settings.  At one point in 
time, all of the Hadoop subprojects were under one JIRA project (HADOOP). But 
then a bunch of folks decided they didn’t want to see the other sub projects 
issues so they split them up…. and thus setting the stage for duplicate code 
and operational divergence in the source. 

Since people don’t realize or care that they are separate, people will 
file INFRA tickets or whatever to change “their project” and not the rest. This 
leads to the JIRA projects also diverging… which ultimately drives those of us 
who actually look at the project as a whole bonkers.
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: Different JIRA permissions for HADOOP and HDFS

[Zheng, Kai]

Yeah, kinds of embarrassing. Thanks Ted.

Or simply, with login access some HADOOP or HDFS JIRAs like the following 
below, note the allowed operations are quite different, and most usable 
operations like attaching for HDFS JIRAs are not showing. Wondering they’re 
disabled recently for some reason?
https://issues.apache.org/jira/browse/HADOOP-12782
https://issues.apache.org/jira/browse/HDFS-10285

Regards,
Kai

From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Saturday, May 14, 2016 7:28 AM
To: Zheng, Kai 
Cc: common-dev@hadoop.apache.org
Subject: Re: Different JIRA permissions for HADOOP and HDFS

Looks like you attached some images which didn't go through.

Consider using 3rd party image site.

Cheers

On Sat, May 14, 2016 at 7:07 AM, Zheng, Kai 
> wrote:
Hi,

Noticed this difference but not sure if it’s intended. YARN is similar with 
HDFS. It’s not convenient. Any clarifying? Thanks. -kai

Re: Different JIRA permissions for HADOOP and HDFS

[Ted Yu]

Looks like you attached some images which didn't go through.

Consider using 3rd party image site.

Cheers

On Sat, May 14, 2016 at 7:07 AM, Zheng, Kai  wrote:

> Hi,
>
>
>
> Noticed this difference but not sure if it’s intended. YARN is similar
> with HDFS. It’s not convenient. Any clarifying? Thanks. -kai
>
>
>
>
>
>
>
>

Different JIRA permissions for HADOOP and HDFS

[Zheng, Kai]

Hi,

Noticed this difference but not sure if it's intended. YARN is similar with 
HDFS. It's not convenient. Any clarifying? Thanks. -kai

[cid:image001.png@01D1ADAF.16160940]


[cid:image002.png@01D1ADAF.16160940]

[jira] [Created] (HADOOP-13147) Constructors must not call overrideable methods

[Sebb (JIRA)]

Sebb created HADOOP-13147:
-

 Summary: Constructors must not call overrideable methods
 Key: HADOOP-13147
 URL: https://issues.apache.org/jira/browse/HADOOP-13147
 Project: Hadoop Common
  Issue Type: Bug
 Environment: 
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32C.java
Reporter: Sebb


Constructors must not call overrideable methods.

An object is not guaranteed fully constructed until the constructor exits, so 
the subclass override may not see the fully created parent object.

This applies to:

PureJavaCrc32




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org