Re: Policy on adding timeouts to tests

[Vinod Kumar Vavilapalli]

The other advantage of timeout is early failure - earlier than the uber 10 min 
timeout that seems to exist in the build files. Usually the test-writer has a 
general idea of how long the test is supposed to run and if that doesn't 
happen, we can fail early. Clearly, this involves choosing a reasonable timeout 
so that the test can pass on local machines, different OSes and/or in VMs.

+Vinod

On Apr 15, 2014, at 11:37 AM, Chris Nauroth  wrote:

> +common-dev, hdfs-dev
> 
> My understanding of the current situation is that we had a period where we
> tried to enforce adding timeouts on all new tests in patches, but it caused
> trouble, and now we're back to not requiring it.  Jenkins test-patch isn't
> checking for it anymore.
> 
> I don't think patches are getting rejected for using timeouts though.
> 
> The difficulty is that execution time is quite sensitive to the build
> environment.  (Consider top-of-the-line server hardware used in build
> infrastructure vs. a dev running a VirtualBox VM with 1 dedicated CPU, 2 GB
> RAM and slow virtualized disk.)  When we were enforcing timeouts, it was
> quite common to see follow-up patches tuning up the timeout settings to
> make tests work reliably in a greater variety of environments.  At that
> point, the benefit of using the timeout becomes questionable, because now
> the fast machine is running with the longer timeout too.
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> 
> 
> On Mon, Apr 14, 2014 at 9:41 AM, Karthik Kambatla wrote:
> 
>> Hi folks
>> 
>> Just wanted to check what our policy for adding timeouts to tests is. Do we
>> encourage/discourage using timeouts for tests? If we discourage using
>> timeouts for tests in general, are we okay with adding timeouts for a few
>> tests where we explicitly want the test to fail if it takes longer than a
>> particular amount of time?
>> 
>> Thanks
>> Karthik
>> 
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.


signature.asc
Description: Message signed with OpenPGP using GPGMail

[jira] [Created] (HADOOP-10514) Common side changes to support HDFS extended attributes (HDFS-2006)

[Uma Maheswara Rao G (JIRA)]

Uma Maheswara Rao G created HADOOP-10514:


 Summary: Common side changes to support  HDFS extended attributes 
(HDFS-2006)
 Key: HADOOP-10514
 URL: https://issues.apache.org/jira/browse/HADOOP-10514
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs
Reporter: Uma Maheswara Rao G
Assignee: Yi Liu


This is an umbrella issue for tracking all Hadoop Common changes required to 
support HDFS extended attributes implementation



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Running unit test cases with Kerberos on

[Mohammad Islam]

Hi,
I tried to run a test case using this command from my Linux box:
mvn clean test  -PtestKerberos -Dtest=TestJHSSecurity

And I got the following exception. I know it is  related to setup the principal 
and other kerberos settings.

Can someone please help me  about this? such as what is the mvn command and 
what other settings are required? Do I need to run my own KDC or provide own 
keytab?

Regards,
Mohammad




java.lang.RuntimeException: java.io.IOException: failure to login
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:886)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
    at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)
    at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)
    at java.security.AccessController.doPrivileged(Native Method)
    at 
javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:591)
    at 
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:767)
    at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:744)
    at 
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:617)
    at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:432)
    at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:455)
    at 
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:566)
    at 
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:533)
    at 
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:501)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 
org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:94)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
    at 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:142)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at 
org.apache.hadoop.mapreduce.security.TestJHSSecurity.testDelegationToken(TestJHSSecurity.java:111)

[jira] [Resolved] (HADOOP-8490) Add Configuration to FileSystem cache key

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp resolved HADOOP-8490.
-

Resolution: Won't Fix

Specific issue that prompted the bug was fixed in the NM long ago.

> Add Configuration to FileSystem cache key
> -
>
> Key: HADOOP-8490
> URL: https://issues.apache.org/jira/browse/HADOOP-8490
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 0.23.0, 0.24.0, 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>
> The {{FileSystem#get(URI, Configuration}} does not take the given 
> {{Configuration}} into consideration before returning an existing fs instance 
> from the cache with a possibly different conf.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10513) ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders

[Gopal V (JIRA)]

Gopal V created HADOOP-10513:


 Summary: ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders
 Key: HADOOP-10513
 URL: https://issues.apache.org/jira/browse/HADOOP-10513
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.0
Reporter: Gopal V


Any code which uses custom class-loaders needs to be aware of the 
ReflectionUtils::CONSTRUCTOR_CACHE 

{code}
 /** 
   * Cache of constructors for each class. Pins the classes so they
   * can't be garbage collected until ReflectionUtils can be collected.
   */
  private static final Map, Constructor> CONSTRUCTOR_CACHE = 
new ConcurrentHashMap, Constructor>();
{code}

This is not a problem when using only 1 AppClassLoader.

But in cases where the application uses multiple classloaders (to isolate 
UDFs), this holds onto class references and their associated class loaders by 
ref (& that leaks all the statics).

The clear method for this cache is unfortunately marked only for testing.

{code}
  // methods to support testing
  static void clearCache() {
CONSTRUCTOR_CACHE.clear();
  }
{code}

The cache shows up as the only reference for the class loaders.

!class-loader-leak.png!



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10512) Document usage of node-group layer topology

[Junping Du (JIRA)]

Junping Du created HADOOP-10512:
---

 Summary: Document usage of node-group layer topology
 Key: HADOOP-10512
 URL: https://issues.apache.org/jira/browse/HADOOP-10512
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: documentation
Reporter: Junping Du
Assignee: Junping Du


For work under umbrella of HADOOP-8468, user can enable nodegroup layer between 
node and rack in some situations. We should document it after YARN-18 and 
YARN-19 is figured out.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Passive mode for FTPFileSystem

[Michael Howard]

Please provide feedback as to what other steps I (a newbie) can take to try
to shepherd this (minor) issue through the process.

Thanks,
Michael


On Sun, Mar 30, 2014 at 1:56 PM, Michael Howard <
mich...@newvantagesolutions.com> wrote:

>
> On Wed, Mar 26, 2014 at 6:06 AM, Steve Loughran wrote:
>
>> > I am somewhat puzzled how locale and case sensitivity relate to
>> > FTPFileSystem, but I will take a look and do my best to figure it out.
>> >
>>
>> there's a case sensitive check for the passive configuration option that
>> would fail in Turkey.
>
>
>
> I uploaded a patch to 
> https://issues.apache.org/jira/browse/HADOOP-8602earlier this week to address 
> the concern about Locale case sensitivity.
>
> Please provide open & honest feedback when able.
>
>
> Michael
>
>

Re: DISCUSS: use SLF4J APIs in new modules?

[Eric Baldeschwieler]

+1 * many.  I'd love to see us clean this up.  Getting to agreement on
where we are going would be a huge step forward.

--
Eric14 a.k.a. Eric Baldeschwieler


On Mon, Apr 14, 2014 at 3:33 PM, Steve Loughran wrote:

> On 11 April 2014 18:37, Alejandro Abdelnur  wrote:
>
> > if you dont convert mgs to templates the dont remove the guards, else you
> > create str mgs obj even when not logging.
> >
> >
> that is -we should have static constants? I like to do that in strings
> anyway, because tests can stay in sync with messages that change, though
> once there's template expansion comparison suddenly gets harder
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Policy on adding timeouts to tests

[Andrew Wang]

The timeouts are nice when trying to debug test failures on Jenkins,
because otherwise you just see something like this:

Caused by: java.lang.RuntimeException: The forked VM terminated without
saying properly goodbye. VM crash or System.exit called ?

We still see this today because some tests lack timeouts. I'd encourage
bringing back the test timeout requirement, but always setting a
conservative value (e.g. always 2+ mins). I think the debuggability
improvements are worth it, and we shouldn't need as many "raise the
timeout" JIRAs.

If someone wants to put in some additional effort, it'd be even better to
do what HBase did and categorize our tests into "fast" and "slow" maven
profiles. This would give us a nice way of running the fast subset as a
smoke. Right now, I doubt many devs run the test suite locally since it
takes multiple hours.

Best,
Andrew

On Wed, Apr 16, 2014 at 10:51 AM, Tsuyoshi OZAWA
wrote:

> Hi Karthik,
>
> Some tests with servers like MiniCluster or ZK can never end because
> of unexpected busy loop or something if the tests don't have timeouts.
> It can blocks the other jobs of Jenkins server. Therefore, IMHO, we
> should add timeouts when we write tests with them.
>
> Thanks,
> - Tsuyoshi
>
> On Wed, Apr 16, 2014 at 6:11 PM, Steve Loughran 
> wrote:
> > There's a JIRA somewhere that 's never gone in, to add a timeout rule to
> a
> > base class; this rule gets picked up in that test class and all children
> to
> > specify the timeout
> >
> >   @Rule
> >   public final Timeout testTimeout = new Timeout(TEST_TIMEOUT);
> >
> >
> >1. If we are going to have a timeout everywhere, it should be
> >configurable to different delays. For mavn, that's SystemProperties
> being
> >passed down and extracted.
> >2. We don't want that in every @test method
> >3. so... we should have a AbstractYarnTest, AbstractMapReduce test,
> &c,
> >each picking up the timeout option for their part of the suite
> >4. then cut out all the other timeouts.
> >5. and finally document this somewhere.
> >6. Object store tests need extra-long timeouts, execution time for
> multi
> >GB uploads to S3 and openstack object stores are a function of your
> upload
> >bandwidth, not machine speed
> >
> > -steve
> >
> >
> >
> > On 15 April 2014 21:20, Karthik Kambatla  wrote:
> >
> >> - hwx-hdfs-dev
> >> + hdfs-dev
> >>
> >> Agree with all the points Chris makes.
> >>
> >> I asked this question in the context of a fix that bumps up the timeout
> to
> >> make the test pass on slower machines. If the timeout is not central to
> the
> >> test, is the recommended approach to get rid of the timeout?
> >>
> >>
> >>
> >> On Tue, Apr 15, 2014 at 11:37 AM, Chris Nauroth <
> cnaur...@hortonworks.com
> >> >wrote:
> >>
> >> > +common-dev, hdfs-dev
> >> >
> >> > My understanding of the current situation is that we had a period
> where
> >> we
> >> > tried to enforce adding timeouts on all new tests in patches, but it
> >> caused
> >> > trouble, and now we're back to not requiring it.  Jenkins test-patch
> >> isn't
> >> > checking for it anymore.
> >> >
> >> > I don't think patches are getting rejected for using timeouts though.
> >> >
> >> > The difficulty is that execution time is quite sensitive to the build
> >> > environment.  (Consider top-of-the-line server hardware used in build
> >> > infrastructure vs. a dev running a VirtualBox VM with 1 dedicated
> CPU, 2
> >> GB
> >> > RAM and slow virtualized disk.)  When we were enforcing timeouts, it
> was
> >> > quite common to see follow-up patches tuning up the timeout settings
> to
> >> > make tests work reliably in a greater variety of environments.  At
> that
> >> > point, the benefit of using the timeout becomes questionable, because
> now
> >> > the fast machine is running with the longer timeout too.
> >> >
> >> > Chris Nauroth
> >> > Hortonworks
> >> > http://hortonworks.com/
> >> >
> >> >
> >> >
> >> > On Mon, Apr 14, 2014 at 9:41 AM, Karthik Kambatla  >> > >wrote:
> >> >
> >> > > Hi folks
> >> > >
> >> > > Just wanted to check what our policy for adding timeouts to tests
> is.
> >> Do
> >> > we
> >> > > encourage/discourage using timeouts for tests? If we discourage
> using
> >> > > timeouts for tests in general, are we okay with adding timeouts for
> a
> >> few
> >> > > tests where we explicitly want the test to fail if it takes longer
> >> than a
> >> > > particular amount of time?
> >> > >
> >> > > Thanks
> >> > > Karthik
> >> > >
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> entity
> >> to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictl

Re: Policy on adding timeouts to tests

[Tsuyoshi OZAWA]

Hi Karthik,

Some tests with servers like MiniCluster or ZK can never end because
of unexpected busy loop or something if the tests don't have timeouts.
It can blocks the other jobs of Jenkins server. Therefore, IMHO, we
should add timeouts when we write tests with them.

Thanks,
- Tsuyoshi

On Wed, Apr 16, 2014 at 6:11 PM, Steve Loughran  wrote:
> There's a JIRA somewhere that 's never gone in, to add a timeout rule to a
> base class; this rule gets picked up in that test class and all children to
> specify the timeout
>
>   @Rule
>   public final Timeout testTimeout = new Timeout(TEST_TIMEOUT);
>
>
>1. If we are going to have a timeout everywhere, it should be
>configurable to different delays. For mavn, that's SystemProperties being
>passed down and extracted.
>2. We don't want that in every @test method
>3. so... we should have a AbstractYarnTest, AbstractMapReduce test, &c,
>each picking up the timeout option for their part of the suite
>4. then cut out all the other timeouts.
>5. and finally document this somewhere.
>6. Object store tests need extra-long timeouts, execution time for multi
>GB uploads to S3 and openstack object stores are a function of your upload
>bandwidth, not machine speed
>
> -steve
>
>
>
> On 15 April 2014 21:20, Karthik Kambatla  wrote:
>
>> - hwx-hdfs-dev
>> + hdfs-dev
>>
>> Agree with all the points Chris makes.
>>
>> I asked this question in the context of a fix that bumps up the timeout to
>> make the test pass on slower machines. If the timeout is not central to the
>> test, is the recommended approach to get rid of the timeout?
>>
>>
>>
>> On Tue, Apr 15, 2014 at 11:37 AM, Chris Nauroth > >wrote:
>>
>> > +common-dev, hdfs-dev
>> >
>> > My understanding of the current situation is that we had a period where
>> we
>> > tried to enforce adding timeouts on all new tests in patches, but it
>> caused
>> > trouble, and now we're back to not requiring it.  Jenkins test-patch
>> isn't
>> > checking for it anymore.
>> >
>> > I don't think patches are getting rejected for using timeouts though.
>> >
>> > The difficulty is that execution time is quite sensitive to the build
>> > environment.  (Consider top-of-the-line server hardware used in build
>> > infrastructure vs. a dev running a VirtualBox VM with 1 dedicated CPU, 2
>> GB
>> > RAM and slow virtualized disk.)  When we were enforcing timeouts, it was
>> > quite common to see follow-up patches tuning up the timeout settings to
>> > make tests work reliably in a greater variety of environments.  At that
>> > point, the benefit of using the timeout becomes questionable, because now
>> > the fast machine is running with the longer timeout too.
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Mon, Apr 14, 2014 at 9:41 AM, Karthik Kambatla > > >wrote:
>> >
>> > > Hi folks
>> > >
>> > > Just wanted to check what our policy for adding timeouts to tests is.
>> Do
>> > we
>> > > encourage/discourage using timeouts for tests? If we discourage using
>> > > timeouts for tests in general, are we okay with adding timeouts for a
>> few
>> > > tests where we explicitly want the test to fail if it takes longer
>> than a
>> > > particular amount of time?
>> > >
>> > > Thanks
>> > > Karthik
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.



-- 
- Tsuyoshi

[jira] [Created] (HADOOP-10511) s3n:// incorrectly handles URLs with secret keys that contain a slash

[Daniel Darabos (JIRA)]

Daniel Darabos created HADOOP-10511:
---

 Summary: s3n:// incorrectly handles URLs with secret keys that 
contain a slash
 Key: HADOOP-10511
 URL: https://issues.apache.org/jira/browse/HADOOP-10511
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Daniel Darabos


This is similar to HADOOP-3733, but happens on s3n:// instead of s3://.

Essentially if I have a path like "s3n://key:pass%2fw...@example.com/test", it 
will under certain circumstances be replaced with "s3n://key:pass/test" which 
then causes "Invalid hostname in URI" exceptions.

I have a unit test and a fix for this. I'll make a pull request in a moment.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HADOOP-10510) TestSymlinkLocalFSFileContext tests are failing

[Daniel Darabos (JIRA)]

Daniel Darabos created HADOOP-10510:
---

 Summary: TestSymlinkLocalFSFileContext tests are failing
 Key: HADOOP-10510
 URL: https://issues.apache.org/jira/browse/HADOOP-10510
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
 Environment: Linux
Reporter: Daniel Darabos


Test results:
https://gist.github.com/oza/9965197

This was mentioned on hadoop-common-dev:
http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201404.mbox/%3CCAAD07OKRSmx9VSjmfk1YxyBmnFM8mwZSp%3DizP8yKKwoXYvn3Qg%40mail.gmail.com%3E

Can you suggest a workaround in the meantime? I'd like to send a pull request 
for an unrelated bug, but these failures mean I cannot build hadoop-common to 
test my fix. Thanks.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: Hadoop-Common-trunk #1101

[Apache Jenkins Server]

See 

Changes:

[devaraj] MAPREDUCE-5775. Remove unnecessary job.setNumReduceTasks in 
SleepJob.createJob. Contributed by jhanver chand sharma.

[jianhe] YARN-1934. Fixed a potential NPE in ZKRMStateStore caused by handling 
Disconnected event from ZK. Contributed by Karthik Kambatla.

[cnauroth] HDFS-5409. TestOfflineEditsViewer#testStored fails on Windows due to 
CRLF line endings in editsStored.xml from git checkout. Contributed by Chris 
Nauroth.

[zjshen] YARN-1892. Improved some logs in the scheduler. Contributed by Jian He.

[jeagles] MAPREDUCE-5836. Fix typo in RandomTextWriter (Akira AJISAKA via 
jeagles)

[wheat9] HDFS-6194. Create new tests for ByteRangeInputStream. Contributed by 
Akira Ajisaka.

[daryn] HADOOP-10498. Add support for proxy server. (daryn)

--
[...truncated 60582 lines...]
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 from a zip file
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader 
(parentFirst)
 +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent 
loader (parentFirst)
 +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -> 

Setting project property: test.exclude.pattern -> _
Setting project property: hadoop.assemblies.version -> 3.0.0-SNAPSHOT
Setting project property: test.exclude -> _
Setting project property: distMgmtSnapshotsId -> apache.snapshots.https
Setting project property: project.build.sourceEncoding -> UTF-8
Setting project property: java.security.egd -> file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -> 
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -> 
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version -> 1.7.4
Setting project property: test.build.data -> 

Setting project property: commons-daemon.version -> 1.0.13
Setting project property: hadoop.common.build.dir -> 

Setting project property: testsThreadCount -> 4
Setting project property: maven.test.redirectTestOutputToFile -> true
Setting project property: jdiff.version -> 1.0.9
Setting project property: build.platform -> Linux-i386-32
Setting project property: project.reporting.outputEncoding -> UTF-8
Setting project property: distMgmtStagingName -> Apache Release Distribution 
Repository
Setting project property: protobuf.version -> 2.5.0
Setting project property: failIfNoTests -> false
Setting project property: protoc.path -> ${env.HADOOP_PROTOC_PATH}
Setting project property: jersey.version -> 1.9
Setting project property: distMgmtStagingId -> apache.staging.https
Setting project property: distMgmtSnapshotsName -> Apache Development Snapshot 
Repository
Setting project property: ant.file -> 

[DEBUG] Setting properties with prefix: 
Setting project property: project.groupId -> org.apache.hadoop
Setting project property: project.artifactId -> hadoop-common-project
Setting project property: project.name -> Apache Hadoop Common Project
Setting project property: project.description -> Apache Hadoop Common Project
Setting project property: project.version -> 3.0.0-SNAPSHOT
Setting project property: project.packaging -> pom
Setting project property: project.build.directory -> 

Setting project property: project.build.outputDirectory -> 

Setting project property: project.build.testOutputDirectory -> 

Setting project property: project.build.sourceDirectory -> 

Re: Policy on adding timeouts to tests

[Steve Loughran]

There's a JIRA somewhere that 's never gone in, to add a timeout rule to a
base class; this rule gets picked up in that test class and all children to
specify the timeout

  @Rule
  public final Timeout testTimeout = new Timeout(TEST_TIMEOUT);


   1. If we are going to have a timeout everywhere, it should be
   configurable to different delays. For mavn, that's SystemProperties being
   passed down and extracted.
   2. We don't want that in every @test method
   3. so... we should have a AbstractYarnTest, AbstractMapReduce test, &c,
   each picking up the timeout option for their part of the suite
   4. then cut out all the other timeouts.
   5. and finally document this somewhere.
   6. Object store tests need extra-long timeouts, execution time for multi
   GB uploads to S3 and openstack object stores are a function of your upload
   bandwidth, not machine speed

-steve



On 15 April 2014 21:20, Karthik Kambatla  wrote:

> - hwx-hdfs-dev
> + hdfs-dev
>
> Agree with all the points Chris makes.
>
> I asked this question in the context of a fix that bumps up the timeout to
> make the test pass on slower machines. If the timeout is not central to the
> test, is the recommended approach to get rid of the timeout?
>
>
>
> On Tue, Apr 15, 2014 at 11:37 AM, Chris Nauroth  >wrote:
>
> > +common-dev, hdfs-dev
> >
> > My understanding of the current situation is that we had a period where
> we
> > tried to enforce adding timeouts on all new tests in patches, but it
> caused
> > trouble, and now we're back to not requiring it.  Jenkins test-patch
> isn't
> > checking for it anymore.
> >
> > I don't think patches are getting rejected for using timeouts though.
> >
> > The difficulty is that execution time is quite sensitive to the build
> > environment.  (Consider top-of-the-line server hardware used in build
> > infrastructure vs. a dev running a VirtualBox VM with 1 dedicated CPU, 2
> GB
> > RAM and slow virtualized disk.)  When we were enforcing timeouts, it was
> > quite common to see follow-up patches tuning up the timeout settings to
> > make tests work reliably in a greater variety of environments.  At that
> > point, the benefit of using the timeout becomes questionable, because now
> > the fast machine is running with the longer timeout too.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Mon, Apr 14, 2014 at 9:41 AM, Karthik Kambatla  > >wrote:
> >
> > > Hi folks
> > >
> > > Just wanted to check what our policy for adding timeouts to tests is.
> Do
> > we
> > > encourage/discourage using timeouts for tests? If we discourage using
> > > timeouts for tests in general, are we okay with adding timeouts for a
> few
> > > tests where we explicitly want the test to fail if it takes longer
> than a
> > > particular amount of time?
> > >
> > > Thanks
> > > Karthik
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.