Re: node.js and more as dependencies

[Wangda Tan]

@Haihui/@Colin,

Thanks for suggestions, I agree that there're pros and cons of lightweight
framework, Ember and other frameworks like React.js. And since new
frameworks are continuously released, we can spend years to finalize a
framework :).

I would like to revisit choices of frameworks after we have more
contributors. Now it's a side project to several YARN contributors, most of
us are pretty new to JS and understand Ember better than others. I wanna to
get some basic runnable works done before make it projection-ready.

In between, I will try to make code not to be framework-specific as much as
possible to alleviate painful of switching frameworks.

@Steve,

Unit testing for visible part will be hard. First, it changes a lot while
developing, we can spend hours write a test in HtmlUnit, but a few lines of
layout code changes could lead to rewrite entire test.

I think we need tests of invisible part, like how to translate REST
response to models. And smoke tests of visible part, like basic layout,
table rendering, error handling, etc. And manually test is required to
complex UI logic, like SVG chart rendering, etc.

Thanks,
Wangda

On Wed, Mar 2, 2016 at 11:22 AM, Colin P. McCabe  wrote:

> Frameworks are a double-edged sword.  On one hand, developers are
> somewhat more productive when using a framework (at least, a good
> one).  On the other hand, frameworks increase the barrier to entry for
> new contributors, because they have to learn the idioms of the
> framework before they can contribute anything meaningful.
>
> There is also a huge amount of churn in Javascript frameworks.
> There's no guarantee that 2015's framework will still be popular in
> 2017.  On a related note, the "why ember" page you linked to says that
> "Now that Ember has reached 1.0 the code samples below are no longer
> correct and the expressed opinions may no longer be accurate."  Oops.
>
> In HTrace, we went with a more lightweight framework, backbone.js, and
> a small set of libraries like moment.js and jquery.  In general,
> libraries are easier to handle than frameworks, since they have a
> pretty narrowly defined scope.  If you have problems with one you can
> easily swap it out for another, or just write your own functions to do
> what you need.  With frameworks, you don't always have that option.
> (I would point out that the sunburst visualization you linked to is
> part of the d3 library, not the ember framework.  It could just as
> easily be used with dust, backbone, or any other framework as with
> ember.)
>
> Anyway, the choice of framework and library should be up to the
> developers doing the coding and maintenance.  If you think that ember
> will be helpful, then sure, that's fine.
>
> Colin
>
> On Tue, Mar 1, 2016 at 9:55 PM, Haohui Mai  wrote:
> > +1 on adding npm / gulp / mocha to be part of the build dependencies
> > (not runtime). The main benefit is modularity -- for example, in the
> > HDFS UI we manually duplicate the navigation bars and footer. We don't
> > have unit tests for them due to the lack of infrastructure.
> >
> > In my opinion Introducing npm will effectively bridge the gap.
> >
> > However, I'm not entirely convinced by the Ember.js argument -- I
> > understand it might provide better integration with Ambari, but there
> > are clear trends that the industry is moving to a more reactive UI
> > design. I think the decision of using ember.js exclusively might worth
> > revisiting. To me it makes more sense to move both HDFS / YARN towards
> > React and go from there.
> >
> > ~Haohui
> >
> > On Mon, Feb 29, 2016 at 5:14 PM, Wangda Tan  wrote:
> >> Hi Colin,
> >>
> >> Thanks for comment, I think your concerns are all valid but also
> arguable:
> >>
> >> First, YARN UI is different from HDFS UI, it is much more complex:
> >> 1. We have many data models, such as app/container/system/node, etc. UI
> of
> >> HDFS is more like a file explorer.
> >> 2. We plan to add more rich data visualization to YARN UI to make admin
> can
> >> easier identify what happened. For example, using sunburst map
> >>  to render usage breakdown of
> >> cluster/queue/user/app, etc.
> >> 3. We need to get data from different sources with different format. For
> >> example, application's running container information stores at RM and
> >> finished container information stores at Timeline server. We need to get
> >> data from both daemon, normalize these data (because REST API is
> different)
> >> and aggregate them.
> >>
> >> Ember.js could simplify what we should do a lot:
> >> - It has great data store design so we can easily normalize data model
> from
> >> different sources with different formats (adapter)
> >> - It can easily bind data model with view, so any changes to data store
> >> (like application status updated) can trigger page re-rendering without
> any
> >> additional works.
> >> - Besides binding data with view, it can also bind data to other
> computed
> >> properties.

[jira] [Created] (HADOOP-12894) Add yarn.app.mapreduce.am.log.level to mapred-default.xml

[Harsh J (JIRA)]

Harsh J created HADOOP-12894:


 Summary: Add yarn.app.mapreduce.am.log.level to mapred-default.xml
 Key: HADOOP-12894
 URL: https://issues.apache.org/jira/browse/HADOOP-12894
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.9.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Common-trunk #2474

[Apache Jenkins Server]

See 

Changes:

[rohithsharmaks] YARN-4763. RMApps Page crashes with NPE. (Bibin A Chundatt via

--
[...truncated 3818 lines...]
Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating 

Generating 

Generating 

Generating 

Generating 

[INFO] Building jar: 

[INFO] 
[INFO] 
[INFO] Building Apache Hadoop MiniKDC 3.0.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-minikdc ---
[INFO] Deleting 

[INFO] 
[INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-minikdc ---
[INFO] There are 9 errors reported by Checkstyle 6.6 with 
checkstyle/checkstyle.xml ruleset.
[WARNING] Unable to locate Source XRef to link to - DISABLED
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-minikdc ---
[INFO] Executing tasks

main:
[mkdir] Created dir: 

[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
hadoop-minikdc ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-minikdc 
---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ 
hadoop-minikdc ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.637 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.795 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1181

[Apache Jenkins Server]

See 

Changes:

[cnauroth] HADOOP-12717. NPE when trying to rename a directory in Windows Azure

[rohithsharmaks] YARN-4763. RMApps Page crashes with NPE. (Bibin A Chundatt via

--
[...truncated 3842 lines...]
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hadoop-minikdc ---
[INFO] Compiling 2 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-minikdc ---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestMiniKdc
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.155 sec - in 
org.apache.hadoop.minikdc.TestMiniKdc
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.248 sec - in 
org.apache.hadoop.minikdc.TestChangeOrgNameAndDomain

Results :

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-minikdc ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-minikdc 
---
[INFO] 
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ 
hadoop-minikdc ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minikdc ---
[INFO] 
Loading source files for package org.apache.hadoop.minikdc...
Constructing Javadoc information...
Standard Doclet version 1.8.0
Building tree for all the packages and classes...
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating 

Build failed in Jenkins: Hadoop-Common-trunk #2473

[Apache Jenkins Server]

See 

Changes:

[cnauroth] HADOOP-12717. NPE when trying to rename a directory in Windows Azure

--
[...truncated 5119 lines...]
Exception: java.lang.RuntimeException thrown from the UncaughtExceptionHandler 
in thread "Health Monitor for DummyHAService #3"
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.385 sec - in 
org.apache.hadoop.ha.TestHealthMonitorWithDedicatedHealthAddress
Running org.apache.hadoop.ha.TestZKFailoverController
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.954 sec - 
in org.apache.hadoop.ha.TestZKFailoverController
Running org.apache.hadoop.ha.TestZKFailoverControllerStress
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.234 sec - in 
org.apache.hadoop.ha.TestZKFailoverControllerStress
Running org.apache.hadoop.ha.TestActiveStandbyElector
Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.898 sec - in 
org.apache.hadoop.ha.TestActiveStandbyElector
Running org.apache.hadoop.ha.TestSshFenceByTcpPort
Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.743 sec - in 
org.apache.hadoop.ha.TestSshFenceByTcpPort
Running org.apache.hadoop.ha.TestHAAdmin
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.536 sec - in 
org.apache.hadoop.ha.TestHAAdmin
Running org.apache.hadoop.ha.TestFailoverController
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.577 sec - in 
org.apache.hadoop.ha.TestFailoverController
Running org.apache.hadoop.ha.TestShellCommandFencer
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.049 sec - in 
org.apache.hadoop.ha.TestShellCommandFencer
Running org.apache.hadoop.ha.TestNodeFencer
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.871 sec - in 
org.apache.hadoop.ha.TestNodeFencer
Running org.apache.hadoop.jmx.TestJMXJsonServlet
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.164 sec - in 
org.apache.hadoop.jmx.TestJMXJsonServlet
Running org.apache.hadoop.conf.TestConfiguration
Tests run: 62, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.276 sec - in 
org.apache.hadoop.conf.TestConfiguration
Running org.apache.hadoop.conf.TestConfigurationDeprecation
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.732 sec - in 
org.apache.hadoop.conf.TestConfigurationDeprecation
Running org.apache.hadoop.conf.TestConfServlet
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.728 sec - in 
org.apache.hadoop.conf.TestConfServlet
Running org.apache.hadoop.conf.TestConfigurationSubclass
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.437 sec - in 
org.apache.hadoop.conf.TestConfigurationSubclass
Running org.apache.hadoop.conf.TestGetInstances
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.392 sec - in 
org.apache.hadoop.conf.TestGetInstances
Running org.apache.hadoop.conf.TestReconfiguration
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.286 sec - in 
org.apache.hadoop.conf.TestReconfiguration
Running org.apache.hadoop.conf.TestDeprecatedKeys
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.658 sec - in 
org.apache.hadoop.conf.TestDeprecatedKeys
Running org.apache.hadoop.log.TestLogLevel
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.892 sec - in 
org.apache.hadoop.log.TestLogLevel
Running org.apache.hadoop.log.TestLog4Json
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.435 sec - in 
org.apache.hadoop.log.TestLog4Json
Running org.apache.hadoop.cli.TestCLI
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.691 sec - in 
org.apache.hadoop.cli.TestCLI
Running org.apache.hadoop.tracing.TestTraceUtils
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.594 sec - in 
org.apache.hadoop.tracing.TestTraceUtils
Running org.apache.hadoop.util.TestDirectBufferPool
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec - in 
org.apache.hadoop.util.TestDirectBufferPool
Running org.apache.hadoop.util.TestSysInfoLinux
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.333 sec - in 
org.apache.hadoop.util.TestSysInfoLinux
Running org.apache.hadoop.util.TestLightWeightGSet
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.179 sec - in 
org.apache.hadoop.util.TestLightWeightGSet
Running org.apache.hadoop.util.TestHostsFileReader
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.195 sec - in 
org.apache.hadoop.util.TestHostsFileReader
Running org.apache.hadoop.util.hash.TestHash
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.468 sec - in 
org.apache.hadoop.util.hash.TestHash
Running org.apache.hadoop.util.TestPureJavaCrc32
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.595 sec - in 
org.apache.hadoop.util.TestPureJavaCrc32
Running org.apache.hadoop.util.TestNat