[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342194#comment-15342194
 ] 

Joep Rottinghuis commented on MAPREDUCE-6720:
-

The advantage of using a prefix for MAP_, REDUCE_, and TOTAL_ is that it would 
be easy (and efficient) to pull back only one set if desired. This would be at 
the cost of pulling back all counters (MAP_, REDUCE_, and TOTAL_) for some 
other prefix such as FILE, which would then turn into something like: 
[MAP_FILE_*, REDUCE_FILE*, TOTAL_FILE*].
I think users are still more likely to either pull everything back, or 
specifically only for mappers, reducers or total.

If we want to reduce the write volume we could consider rolling up only the 
total to the flow level. I can see the value of map counters separate from 
reduce counters per job, but I'm not clear what the meaning would be of a sum 
of these across multiple jobs in a flow. Then again, we can just write first 
and ask questions later.

> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: yarn-2928-1st-milestone
> Attachments: Counters For Job.png, Counters for Tasks.png, 
> MAPREDUCE-6720-YARN-2928.01.patch
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341974#comment-15341974
 ] 

Joep Rottinghuis commented on MAPREDUCE-6720:
-

Does this exist in ATS v1, or is it unique to v2?



> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: Counters For Job.png, Counters for Tasks.png
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341250#comment-15341250
 ] 

Joep Rottinghuis commented on MAPREDUCE-6720:
-

Do we yet have a sense where the reducer data is getting dropped? Does
cranking up the debug level help identify this?
Are we even writing in the first place, is the writer doing the right
thing, or is there some bug in the copro (possibly related to compaction)?
IIUC then aggregation requires the application to write its values and if
reducer or mapper values are written separately, then values are
overwritten (discarded as older versions of the same app). It should be the
AMs responsibility to sum the counters for the mapper and the reducer
together and to emit those to the writer.

On Mon, Jun 20, 2016 at 1:11 PM, Varun Saxena (JIRA) 



> Inconsistent values of counters across tasks and job reported to timeline 
> service.
> --
>
> Key: MAPREDUCE-6720
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: applicationmaster
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
> Attachments: Counters For Job.png, Counters for Tasks.png
>
>
> While testing found below issue. For some of the task counters, we do not 
> have consistent values. This is not the case with every counter though.
> Consider the case of counter 
> "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".
> I found that its value for a flow I ran, was 936018 bytes. For the 3 apps 
> associated with this flow run, the values were 312006 bytes each (which 
> equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further 
> down I found though that for one of the apps, the 4 tasks(2 mappers and 2 
> reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes 
> each for the 2 mappers.
> This means the value reported for the app should be (2 * 156003 + 2* 155918) 
> or 623842 bytes but it is only 312006 bytes which indicates that only counter 
> value of mappers is being picked up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5873) Measure bw of a single copy call and display the correct aggregated bw

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155848#comment-14155848
 ] 

Joep Rottinghuis commented on MAPREDUCE-5873:
-

[~jlowe] are you guys seeing this behavior as well ?
We're finding that the current way that the shuffle bandwidth is shows is not 
only useless, but also confusing for our users.
If a mapper has to be re-run it appears as if the bandwidth is a tiny amount.
This new way of showing bandwidth is a much better indication of what is going 
on, and if it is very small, it is actually in indication that there is 
something not right in the network.


 Measure bw of a single copy call and display the correct aggregated bw
 --

 Key: MAPREDUCE-5873
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5873
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Siqi Li
Assignee: Siqi Li
 Attachments: MAPREDUCE-5873.v1.patch, MAPREDUCE-5873.v2.patch, 
 MAPREDUCE-5873.v3.patch


 Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its 
 definition however is confusing because it captures the time where there is 
 no copying because there is a pause between when new wave of map outputs is 
 available.
 current bw is definded as (bytes copied so far) / (total time in the copy 
 phase so far)
 It would be more useful 
 1) to measure bandwidth of a single copy call.
 2) display aggregated bw as long as there is at least one fetcher is in the 
 copy call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-4710) Add peak memory usage counter for each task

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13818021#comment-13818021
 ] 

Joep Rottinghuis commented on MAPREDUCE-4710:
-

Cindy, is your experience that your JVM is giving memory back to the OS to make 
PHYSICAL_MEMORY_BYTES_MAX inaccurate ?

What is the sampling rate to use to detect the max, or is sampling not needed 
here ?




 Add peak memory usage counter for each task
 ---

 Key: MAPREDUCE-4710
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4710
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: task
Affects Versions: 1.0.2
Reporter: Cindy Li
Assignee: Cindy Li
Priority: Minor
  Labels: patch
 Attachments: MAPREDUCE-4710-trunk.patch, mapreduce-4710-v1.0.2.patch, 
 mapreduce-4710.patch, mapreduce4710.patch


 Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which 
 are snapshots of memory usage of that task. They are not sufficient for users 
 to understand peak memory usage by that task, e.g. in order to diagnose task 
 failures, tune job parameters or change application design. This new feature 
 will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and 
 VIRTUAL_MEMORY_BYTES_MAX. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-5502) History link in resource manager is broken for KILLED jobs

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770900#comment-13770900
 ] 

Joep Rottinghuis commented on MAPREDUCE-5502:
-

Thanks Jason, those are some excellent points to consider.

Joep




 History link in resource manager is broken for KILLED jobs
 --

 Key: MAPREDUCE-5502
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5502
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.5-alpha
Reporter: Vrushali C
Assignee: Vrushali C
  Labels: ui

 History link in resource manager is broken for KILLED jobs.
 Seems to happen with jobs with State 'KILLED' and FinalStatus 'KILLED'. If 
 the State is 'FINISHED' and FinalStatus is 'KILLED', then the History link 
 is fine.
 It isn't easy to reproduce the problem since the time at which the app is 
 killed determines the state it ends up in, which is hard to guess. these 
 particular jobs seem to get a Diagnostics message of Application killed by 
 user. where as the other killed jobs get  Kill Job received from client 
 job_1378766187901_0002
 Job received Kill while in RUNNING state. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5502) History link in resource manager is broken for KILLED jobs

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770084#comment-13770084
 ] 

Joep Rottinghuis commented on MAPREDUCE-5502:
-

[~jlowe] Ack that when YARN kills the AM before any resources are allocated / 
anything is run, there is no chance to create any history link.
However, in the case of a regular AM getting killed we should have some code in 
the shut-down hook to update the URL.
These are fine points of difference, but our users could not be bothered with 
those details. We should try to do what we can to provide a good user 
experience when we can.

 History link in resource manager is broken for KILLED jobs
 --

 Key: MAPREDUCE-5502
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5502
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.5-alpha
Reporter: Vrushali C
  Labels: ui

 History link in resource manager is broken for KILLED jobs.
 Seems to happen with jobs with State 'KILLED' and FinalStatus 'KILLED'. If 
 the State is 'FINISHED' and FinalStatus is 'KILLED', then the History link 
 is fine.
 It isn't easy to reproduce the problem since the time at which the app is 
 killed determines the state it ends up in, which is hard to guess. these 
 particular jobs seem to get a Diagnostics message of Application killed by 
 user. where as the other killed jobs get  Kill Job received from client 
 job_1378766187901_0002
 Job received Kill while in RUNNING state. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5186) mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770118#comment-13770118
 ] 

Joep Rottinghuis commented on MAPREDUCE-5186:
-

Any inputs on this ?

 mapreduce.job.max.split.locations causes some splits created by 
 CombineFileInputFormat to fail
 --

 Key: MAPREDUCE-5186
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5186
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Sangjin Lee

 CombineFileInputFormat can easily create splits that can come from many 
 different locations (during the last pass of creating global splits). 
 However, we observe that this often runs afoul of the 
 mapreduce.job.max.split.locations check that's done by JobSplitWriter.
 The default value for mapreduce.job.max.split.locations is 10, and with any 
 decent size cluster, CombineFileInputFormat creates splits that are well 
 above this limit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4939) Faulty getLocations input split passed on blindly to topology script

[Joep Rottinghuis (JIRA)]

Joep Rottinghuis created MAPREDUCE-4939:
---

 Summary: Faulty getLocations input split passed on blindly to 
topology script
 Key: MAPREDUCE-4939
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4939
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.205.0, 2.0.3-alpha
Reporter: Joep Rottinghuis


When users implement a (faulty) InputSplit that returns splits that do not map 
to hostnames, they get passed on to the topology script.

Call stack on 0.20 equivalent code:
o.a.h.net.NetworkTopology.add
o.a.h.mapred.JobTracker.addHostToNodeMapping
o.a.h.mapred.JobTracker.resolveAndAddToTopology
o.a.h.mapred.JobInProgress.createCache
o.a.h.mapred.JobInProgress.initTasks
o.apache.hadoop.mapred.JobTracker.initJob

The CachedDNSToSwitchMapping wraps RawScriptBasedMapping (if a topology script 
is configured).

The arguments (input splits) are simply passed to the topology script as 
arguments without any further checks.
The input split could incorrectly return a comma separated list of hosts, or in 
the worst case something like
{noformat}
topology fake-hostname; rm -rf *
{noformat}

In 2.0 something similar happens in ScriptBasedMapping.runResolveCommand.
That method should check the input values before executing the command.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4939) Faulty getLocations input split passed on blindly to topology script

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551575#comment-13551575
 ] 

Joep Rottinghuis commented on MAPREDUCE-4939:
-

Filed under MAPREDUCE although the most obvious place to fix is under common in 
the place where the runResolveCommand is implemented.

 Faulty getLocations input split passed on blindly to topology script
 

 Key: MAPREDUCE-4939
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4939
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.205.0, 2.0.3-alpha
Reporter: Joep Rottinghuis

 When users implement a (faulty) InputSplit that returns splits that do not 
 map to hostnames, they get passed on to the topology script.
 Call stack on 0.20 equivalent code:
 o.a.h.net.NetworkTopology.add
 o.a.h.mapred.JobTracker.addHostToNodeMapping
 o.a.h.mapred.JobTracker.resolveAndAddToTopology
 o.a.h.mapred.JobInProgress.createCache
 o.a.h.mapred.JobInProgress.initTasks
 o.apache.hadoop.mapred.JobTracker.initJob
 The CachedDNSToSwitchMapping wraps RawScriptBasedMapping (if a topology 
 script is configured).
 The arguments (input splits) are simply passed to the topology script as 
 arguments without any further checks.
 The input split could incorrectly return a comma separated list of hosts, or 
 in the worst case something like
 {noformat}
 topology fake-hostname; rm -rf *
 {noformat}
 In 2.0 something similar happens in ScriptBasedMapping.runResolveCommand.
 That method should check the input values before executing the command.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4939) Faulty getLocations input split passed on blindly to topology script

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551584#comment-13551584
 ] 

Joep Rottinghuis commented on MAPREDUCE-4939:
-

Thinking about it, we may want to consider not caching entries that are mapped 
to unresolved.
Otherwise (faulty) inputSplit implementations can cause an infinite number of 
bad entries to be cached (eventually they would map to 
NetworkTopology.DEFAULT_RACK).

The downside is that if there in fact is a small set of incorrect hosts being 
requested, the topology script would get called over and over again instead of 
getting the cached NetworkTopology.DEFAULT_RACK.

 Faulty getLocations input split passed on blindly to topology script
 

 Key: MAPREDUCE-4939
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4939
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.205.0, 2.0.3-alpha
Reporter: Joep Rottinghuis

 When users implement a (faulty) InputSplit that returns splits that do not 
 map to hostnames, they get passed on to the topology script.
 Call stack on 0.20 equivalent code:
 o.a.h.net.NetworkTopology.add
 o.a.h.mapred.JobTracker.addHostToNodeMapping
 o.a.h.mapred.JobTracker.resolveAndAddToTopology
 o.a.h.mapred.JobInProgress.createCache
 o.a.h.mapred.JobInProgress.initTasks
 o.apache.hadoop.mapred.JobTracker.initJob
 The CachedDNSToSwitchMapping wraps RawScriptBasedMapping (if a topology 
 script is configured).
 The arguments (input splits) are simply passed to the topology script as 
 arguments without any further checks.
 The input split could incorrectly return a comma separated list of hosts, or 
 in the worst case something like
 {noformat}
 topology fake-hostname; rm -rf *
 {noformat}
 In 2.0 something similar happens in ScriptBasedMapping.runResolveCommand.
 That method should check the input values before executing the command.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3039) Make mapreduce use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108801#comment-13108801
 ] 

Joep Rottinghuis commented on MAPREDUCE-3039:
-

Thanks for clarifying and the reference to the existing issue Doug.
That was indeed the case I had in mind.

 Make mapreduce use same version of avro as HBase
 

 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
 contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
 contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-3039-branch-0.22.patch


 HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
 When building HBase on top of hadoop, this should be consistent.
 Moreover, this should be consistent between common, hdfs, and mapreduce.
 Contribs seem to have declared a dependency on avro but are not in fact 
 depending on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3039) Make hadoop-common use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

Make hadoop-common use same version of avro as HBase


 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0


HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
When building HBase on top of hadoop, this should be consistent.
Moreover, this should be consistent between common, hdfs, and mapreduce.

Contribs seem to have declared a dependency on avro but are not in fact 
depending on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3039) Make mapreduce use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-3039:


Summary: Make mapreduce use same version of avro as HBase  (was: Make 
hadoop-common use same version of avro as HBase)

 Make mapreduce use same version of avro as HBase
 

 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
 contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
 contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0


 HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
 When building HBase on top of hadoop, this should be consistent.
 Moreover, this should be consistent between common, hdfs, and mapreduce.
 Contribs seem to have declared a dependency on avro but are not in fact 
 depending on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3039) Make mapreduce use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-3039:


Attachment: MAPREDUCE-3039-branch-0.22.patch

Will need somebody else with Avro experience to look at this please.
In the current patch I have not removed the wrapping of strings (CharSequence) 
in JobSubmittedEvent. The same occurs in about 71 other places.
My concern is that when we insert a Utf8 wrapped string into a map, that when 
other places try to do a get with a string we'll fail to find the entry. String 
or other CharSequence will not equal any o.a.avro.util.Utf8 defines equals:
if (!(o instanceof Utf8)) return false;

In short I think I need to rip out all the Utf8 wrapping going on. Right? 

 Make mapreduce use same version of avro as HBase
 

 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
 contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
 contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-3039-branch-0.22.patch


 HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
 When building HBase on top of hadoop, this should be consistent.
 Moreover, this should be consistent between common, hdfs, and mapreduce.
 Contribs seem to have declared a dependency on avro but are not in fact 
 depending on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3039) Make mapreduce use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-3039:


Status: Patch Available  (was: Open)

 Make mapreduce use same version of avro as HBase
 

 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
 contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
 contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-3039-branch-0.22.patch


 HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
 When building HBase on top of hadoop, this should be consistent.
 Moreover, this should be consistent between common, hdfs, and mapreduce.
 Contribs seem to have declared a dependency on avro but are not in fact 
 depending on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3039) Make mapreduce use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108312#comment-13108312
 ] 

Joep Rottinghuis commented on MAPREDUCE-3039:
-

I think I have the answer. When compiling this on my Ubuntu 11.04 desktop with 
JDK 1.6.0_26, Utf8 is allowed in place of Charsequence.
However, our jenkins server running 1.6.0_16 just spat out a bunch of 
compilation errors:

compile-mapred-classes:
[jsp-compile] 11/09/19 18:20:40 WARN compiler.TldLocationsCache: Internal 
Error: File /WEB-INF/web.xml not found
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/build.xml:380:
 warning: 'includeantruntime' was not set, defaulting to 
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 524 source files to 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/build/classes
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:87:
 incompatible types
[javac] found   : org.apache.avro.util.Utf8
[javac] required: java.lang.CharSequence
[javac] result.name = new Utf8(name);
[javac]   ^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:88:
 incompatible types
[javac] found   : 
org.apache.avro.generic.GenericData.Arrayorg.apache.hadoop.mapreduce.jobhistory.JhCounterGroup
[javac] required: 
java.util.Listorg.apache.hadoop.mapreduce.jobhistory.JhCounterGroup
[javac] result.groups = new GenericData.ArrayJhCounterGroup(0, 
GROUPS);
[javac] ^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:92:
 incompatible types
[javac] found   : org.apache.avro.util.Utf8
[javac] required: java.lang.CharSequence
[javac]   g.name = new Utf8(group.getName());
[javac]^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:93:
 incompatible types
[javac] found   : org.apache.avro.util.Utf8
[javac] required: java.lang.CharSequence
[javac]   g.displayName = new Utf8(group.getDisplayName());
[javac]   ^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:94:
 incompatible types
[javac] found   : 
org.apache.avro.generic.GenericData.Arrayorg.apache.hadoop.mapreduce.jobhistory.JhCounter
[javac] required: 
java.util.Listorg.apache.hadoop.mapreduce.jobhistory.JhCounter
[javac]   g.counts = new GenericData.ArrayJhCounter(group.size(), 
COUNTERS);
[javac]  ^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:97:
 incompatible types
[javac] found   : org.apache.avro.util.Utf8
[javac] required: java.lang.CharSequence
[javac] c.name = new Utf8(counter.getName());
[javac]  ^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java:98:
 incompatible types
[javac] found   : org.apache.avro.util.Utf8
[javac] required: java.lang.CharSequence
[javac] c.displayName = new Utf8(counter.getDisplayName());
[javac] ^
[javac] 
/hadoop01/jenkins/jobs/hadoop-mapreduce-SNAPSHOT/workspace/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptFinishedEvent.java:56:
 incompatible types

etc. etc.

Will remove the UTf8 and submit new patch.

 Make mapreduce use same version of avro as HBase
 

 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
 contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
 contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-3039-branch-0.22.patch


 HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
 When building HBase on top of hadoop, this should be consistent.
 Moreover, this should be consistent between common, hdfs, and mapreduce.
 Contribs seem to have declared a dependency on avro but are not in fact 
 depending on it.

--
This message is automatically generated by JIRA.
For more 

[jira] [Commented] (MAPREDUCE-3039) Make mapreduce use same version of avro as HBase

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13108335#comment-13108335
 ] 

Joep Rottinghuis commented on MAPREDUCE-3039:
-

Ok, I'll take that back.
When heeding my own advice in HADOOP-7646 about clearing out the 
~/.ivy2/repository cache the avro 1.3.2 jar disappeared from the classpath and 
the compilation error went away.

Back to my original question:
Should I remove the new Utf8(String...) code and put straight Strings into the 
map in JobSubmittedEvent?
My concern is that the map is defined as CharSequence and we're putting Utf8's 
in the map. If any client tries to retrieve a key using a string, then they 
will not get the value back as the key of type Utf8 will not equal String or 
Charsequence types.

 Make mapreduce use same version of avro as HBase
 

 Key: MAPREDUCE-3039
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3039
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched, contrib/fair-share, 
 contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, 
 contrib/streaming, jobhistoryserver
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-3039-branch-0.22.patch


 HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
 When building HBase on top of hadoop, this should be consistent.
 Moreover, this should be consistent between common, hdfs, and mapreduce.
 Contribs seem to have declared a dependency on avro but are not in fact 
 depending on it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3025) Contribs not building

[Joep Rottinghuis (JIRA)]

Contribs not building
-

 Key: MAPREDUCE-3025
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3025
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0


Contribs are not getting built.
Snippet from Jenkins:

compile:
[subant] No sub-builds to iterate on


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3025) Contribs not building

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106665#comment-13106665
 ] 

Joep Rottinghuis commented on MAPREDUCE-3025:
-

In addition for mapreduce builds:

https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/70/console

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk/build.xml:875:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk/src/contrib/capacity-scheduler/build.xml:26:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk/src/contrib/build-contrib.xml:83:
 Source resource does not exist: 
/home/jenkins/jenkins-slave/ivy/libraries.properties

 Contribs not building
 -

 Key: MAPREDUCE-3025
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3025
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
 [subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3025) Contribs not building

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-3025:


Attachment: MAPREDUCE-3025-branch-0.22.patch

 Contribs not building
 -

 Key: MAPREDUCE-3025
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3025
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: MAPREDUCE-3025-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
 [subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2972) Running commands from the hadoop-mapreduce-test-*.jar fails with ClassNotFoundException: junit.framework.TestCase

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101283#comment-13101283
 ] 

Joep Rottinghuis commented on MAPREDUCE-2972:
-

That is interesting. We are seeing the same on 0.22 when running in dev mode. 
When things are packaged up and deployed this error does not occur.

 Running commands from the hadoop-mapreduce-test-*.jar fails with  
 ClassNotFoundException: junit.framework.TestCase
 --

 Key: MAPREDUCE-2972
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2972
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Jeffrey Naisbitt
Assignee: Jeffrey Naisbitt

 Running any of the 'hadoop jar hadoop-mapreduce-test-*.jar' commands gives 
 the following exception:
 java.lang.NoClassDefFoundError: junit/framework/TestCase
   at java.lang.ClassLoader.defineClass1(Native Method)
   at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
   at 
 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
   at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
   at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
   at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:300)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
   at 
 org.apache.hadoop.test.MapredTestDriver.init(MapredTestDriver.java:59)
   at 
 org.apache.hadoop.test.MapredTestDriver.init(MapredTestDriver.java:53)
   at 
 org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
 Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
   ... 21 more
 This happens even when just running 'hadoop jar $TEST_JAR' where it should 
 just print the available commands.
 Copying the junit-*.jar from $HADOOP_MAPRED_HOME/lib/ to 
 $HADOOP_COMMON_HOME/share/hadoop/common/lib/ seems to fix the problem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2974) Remove references to yui.yahooapis.com from queueinfo.jsp

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101373#comment-13101373
 ] 

Joep Rottinghuis commented on MAPREDUCE-2974:
-

Both the stylesheet and the javascripts files come witht he following Copyright 
notice:

Copyright (c) 2009, Yahoo! Inc. All rights reserved.
Code licensed under the BSD License:
http://developer.yahoo.net/yui/license.txt
version: 2.7.0



 Remove references to yui.yahooapis.com from queueinfo.jsp
 -

 Key: MAPREDUCE-2974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: scheduler
Affects Versions: 0.22.0
Reporter: Priyo Mustafi

 queueinfo.jsp has references to yui.yahooapis.com which is not accessible 
 from secure networks.  This cause UI not to show up.
 link rel=stylesheet type=text/css 
 href=http://yui.yahooapis.com/combo?2.7.0/build/fonts/fonts-min.css2.7.0/build/grids/grids-min.css2.7.0/build/base/base-min.css2.7.0/build/assets/skins/sam/skin.css;
  
 !-- Combo-handled YUI JS files: -- 
 script type=text/javascript 
 src=http://yui.yahooapis.com/combo?2.7.0/build/utilities/utilities.js2.7.0/build/layout/layout-min.js2.7.0/build/container/container_core-min.js2.7.0/build/menu/menu-min.js2.7.0/build/stylesheet/stylesheet-min.js2.7.0/build/treeview/treeview-min.js;/script
 This needs to be removed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101644#comment-13101644
 ] 

Joep Rottinghuis commented on MAPREDUCE-2940:
-

Build is probably still failing due to HDFS-2315, but I cannot confirm right 
now due to connectivity problems to Jenkins server:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET 
/job/Hadoop-Mapreduce-22-branch/68/.

Reason: Error reading from remote server





Apache/2.2.14 (Ubuntu) Server at builds.apache.org Port 443

 Build fails with ant 1.7.0 but works with 1.8.0
 ---

 Key: MAPREDUCE-2940
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-2940.patch


 contrib builds fail when using Ant 1.7.
 build.xml calls build.xml in contrib, which calls block-forensics build, 
 which in turn uses build-contrib.
 The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2940:


Attachment: MAPREDUCE-2940.patch

 Build fails with ant 1.7.0 but works with 1.8.0
 ---

 Key: MAPREDUCE-2940
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-2940.patch


 contrib builds fail when using Ant 1.7.
 build.xml calls build.xml in contrib, which calls block-forensics build, 
 which in turn uses build-contrib.
 The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2940:


Status: Patch Available  (was: Open)

 Build fails with ant 1.7.0 but works with 1.8.0
 ---

 Key: MAPREDUCE-2940
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-2940.patch


 contrib builds fail when using Ant 1.7.
 build.xml calls build.xml in contrib, which calls block-forensics build, 
 which in turn uses build-contrib.
 The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0

[Joep Rottinghuis (JIRA)]

Build fails with ant 1.7.0 but works with 1.8.0
---

 Key: MAPREDUCE-2940
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0


contrib builds fail when using Ant 1.7.

build.xml calls build.xml in contrib, which calls block-forensics build, which 
in turn uses build-contrib.
The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098621#comment-13098621
 ] 

Joep Rottinghuis commented on MAPREDUCE-2940:
-

Snipped from the build log:

BUILD FAILED
/home/jrottinghuis/git/hadoop-common/mapreduce/build.xml:435: The following 
error occurred while executing this line:
/home/jrottinghuis/git/hadoop-common/mapreduce/src/contrib/build.xml:30: The 
following error occurred while executing this line:
/home/jrottinghuis/git/hadoop-common/mapreduce/src/contrib/block_forensics/build.xml:26:
 The following error occurred while executing this line:
/home/jrottinghuis/git/hadoop-common/mapreduce/src/contrib/build-contrib.xml:82:
 Source resource does not exist: /home/jrottinghuis/git/hadoop-common/mapredu
ce/src/contrib/ivy/libraries.properties

 Build fails with ant 1.7.0 but works with 1.8.0
 ---

 Key: MAPREDUCE-2940
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0


 contrib builds fail when using Ant 1.7.
 build.xml calls build.xml in contrib, which calls block-forensics build, 
 which in turn uses build-contrib.
 The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2901) Build should fail sanely if protoc isn't on PATH

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098346#comment-13098346
 ] 

Joep Rottinghuis commented on MAPREDUCE-2901:
-

Yeah, that is the same error I see with libprotoc 2.3.0:
[INFO] --- exec-maven-plugin:1.2:exec (generate-sources) @ hadoop-yarn-api ---
main:
[INFO] Executed tasks
[INFO] 
[INFO] --- exec-maven-plugin:1.2:exec (generate-sources) @ hadoop-yarn-api ---
yarn_protos.proto:22:8: Option java_generate_equals_and_hash unknown.
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] hadoop-yarn-api ... FAILURE [1.224s]
[INFO] hadoop-yarn-common  SKIPPED
[INFO] hadoop-yarn-server-common . SKIPPED
[INFO] hadoop-yarn-server-nodemanager  SKIPPED
[INFO] hadoop-yarn-server-resourcemanager  SKIPPED
[INFO] hadoop-yarn-server-tests .. SKIPPED
[INFO] hadoop-yarn-server  SKIPPED
[INFO] hadoop-yarn ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1.518s
[INFO] Finished at: Tue Sep 06 12:33:18 PDT 2011
[INFO] Final Memory: 7M/227M
[INFO] 
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec 
(generate-sources) on project hadoop-yarn-api: Command execution failed. 
Process exited with an error: 1(Exit value: 1) - [Help 1]



 Build should fail sanely if protoc isn't on PATH
 

 Key: MAPREDUCE-2901
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2901
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli

 It now fails [ERROR] Failed to execute goal 
 org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project 
 hadoop-yarn-api: Command execution failed. Process exited with an error: 
 1(Exit value: 1) - [Help 1].
 Which doesn't help much.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2901) Build should fail sanely if protoc isn't on PATH

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098360#comment-13098360
 ] 

Joep Rottinghuis commented on MAPREDUCE-2901:
-

After configure make install of protoc the yarn build came through clean.
In my case I uninstalled libprotoc through the synaptic UI.
The default ./configure step did not work and resulted in the following error 
when running protoc --version:
protoc: error while loading shared libraries: libprotobuf.so.7: cannot open 
shared object file: No such file or directory

I did a make unintall, make distclean, ./configure --prefix /usr, make, make 
install
and then things worked fine.

 Build should fail sanely if protoc isn't on PATH
 

 Key: MAPREDUCE-2901
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2901
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli

 It now fails [ERROR] Failed to execute goal 
 org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project 
 hadoop-yarn-api: Command execution failed. Process exited with an error: 
 1(Exit value: 1) - [Help 1].
 Which doesn't help much.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2939) Ant setup on hadoop7 jenkins host

[Joep Rottinghuis (JIRA)]

Ant setup on hadoop7 jenkins host
-

 Key: MAPREDUCE-2939
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2939
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Jenkins: 
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/65/console
Reporter: Joep Rottinghuis
 Fix For: 0.22.0


From the build error it looks like
a) ant is not set up on the machine
b) $ANT_HOME point to the wrong spot



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2939) Ant setup on hadoop7 jenkins host

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098612#comment-13098612
 ] 

Joep Rottinghuis commented on MAPREDUCE-2939:
-

Could somebody with access to the Jenkins host take a look and adjust $ANT_HOME 
or the ant installation there on host hadoop7?
Here is a snippet from the build:

 /homes/hudson/tools/ant/latest/bin/ant -Dversion=2011-08-29_22-31-26 
-Declipse.home=/homes/hudson/tools/eclipse/latest 
-Dfindbugs.home=/homes/hudson/tools/findbugs/latest 
-Dforrest.home=/homes/hudson/tools/forrest/latest -Dcompile.c++=true 
-Dcompile.native=true clean create-c++-configure tar findbugs
nightly/hudsonBuildHadoopNightly.sh: 1: /homes/hudson/tools/ant/latest/bin/ant: 
not found

 Ant setup on hadoop7 jenkins host
 -

 Key: MAPREDUCE-2939
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2939
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: Jenkins: 
 https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/65/console
Reporter: Joep Rottinghuis
 Fix For: 0.22.0


 From the build error it looks like
 a) ant is not set up on the machine
 b) $ANT_HOME point to the wrong spot

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2924) TaskTracker number of failed disks to tolerate should be configurable

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095798#comment-13095798
 ] 

Joep Rottinghuis commented on MAPREDUCE-2924:
-

Would it be possible to keep the TT up. but scale back the # slots available?
Something such as 3 out of 10 discs are bad, so only 7/10th of the slots are 
available?

 TaskTracker number of failed disks to tolerate should be configurable
 -

 Key: MAPREDUCE-2924
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2924
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: tasktracker
Affects Versions: 0.20.204.0
Reporter: Eli Collins

 Like HDFS-1161 but for the TT. The user should be able to configure how many 
 valid disks are needed for operation. Currently the TT will start and accept 
 tasks even if eg only 1 of its 12 disks is working, which leads to poor 
 performance of jobs with tasks that use this machine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2610) Inconsistent API JobClient.getQueueAclsForCurrentUser

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095557#comment-13095557
 ] 

Joep Rottinghuis commented on MAPREDUCE-2610:
-

Don't remember off the top of my head. Will check and get back to you.

 Inconsistent API JobClient.getQueueAclsForCurrentUser
 -

 Key: MAPREDUCE-2610
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2610
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.205.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Attachments: MAPREDUCE-2610.patch


 Client needs access to the current user's queue name.
 Public method JobClient.getQueueAclsForCurrentUser() returns QueueAclsInfo[].
 The QueueAclsInfo class has default access. A public method should not return 
 a package-private class.
 The QueueAclsInfo class, its two constructors, getQueueName, and 
 getOperations methods should be public.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2610) Inconsistent API JobClient.getQueueAclsForCurrentUser

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095575#comment-13095575
 ] 

Joep Rottinghuis commented on MAPREDUCE-2610:
-

No this is not a problem on trunk.
org/apache/hadoop/mapred/QueueAclsInfo.java
is deprecated and the new
org/apache/hadoop/mapreduce/QueueAclsInfo.java
does not suffer from the same problems.

As you know both can be found in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/


 Inconsistent API JobClient.getQueueAclsForCurrentUser
 -

 Key: MAPREDUCE-2610
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2610
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.205.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Attachments: MAPREDUCE-2610.patch


 Client needs access to the current user's queue name.
 Public method JobClient.getQueueAclsForCurrentUser() returns QueueAclsInfo[].
 The QueueAclsInfo class has default access. A public method should not return 
 a package-private class.
 The QueueAclsInfo class, its two constructors, getQueueName, and 
 getOperations methods should be public.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2610) Inconsistent API JobClient.getQueueAclsForCurrentUser

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095578#comment-13095578
 ] 

Joep Rottinghuis commented on MAPREDUCE-2610:
-

This is also not a problem on 0.22, which is the same as trunk in this regard.
Both getQueueName and getOperations are public in the new class.

 Inconsistent API JobClient.getQueueAclsForCurrentUser
 -

 Key: MAPREDUCE-2610
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2610
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.205.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Attachments: MAPREDUCE-2610.patch


 Client needs access to the current user's queue name.
 Public method JobClient.getQueueAclsForCurrentUser() returns QueueAclsInfo[].
 The QueueAclsInfo class has default access. A public method should not return 
 a package-private class.
 The QueueAclsInfo class, its two constructors, getQueueName, and 
 getOperations methods should be public.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2901) Build should fail sanely if protoc isn't on PATH

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095714#comment-13095714
 ] 

Joep Rottinghuis commented on MAPREDUCE-2901:
-

On my Ubuntu desktop (11.04) protobuf-compiler package (2.3.0-4ubuntu2) is 
installed and protoc is on the command line.
I still get the error.

http://wiki.apache.org/hadoop/DevelopingOnTrunkAfter279Merge advices to add 
non-standard installation directory to LD_LIBRARY_PATH.
However, the libprotoc6 package has already put libprotoc.so.6 and 
libprotoc.so.6.0.0 in /usr/lib/. Seems like a standard path to me.

It seems to me that simply checking whether protoc is in the path will not be 
sufficient.

 Build should fail sanely if protoc isn't on PATH
 

 Key: MAPREDUCE-2901
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2901
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli

 It now fails [ERROR] Failed to execute goal 
 org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project 
 hadoop-yarn-api: Command execution failed. Process exited with an error: 
 1(Exit value: 1) - [Help 1].
 Which doesn't help much.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2901) Build should fail sanely if protoc isn't on PATH

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095715#comment-13095715
 ] 

Joep Rottinghuis commented on MAPREDUCE-2901:
-

hadoop-mapreduce-project/hadoop-yarn/README reads that we need to install 
protobuf 2.4.0a or higher.
Does that mean that on my Ubuntu machine I should uninstall the deb packages 
listed in my previous comment and configure/make/install from scratch?

 Build should fail sanely if protoc isn't on PATH
 

 Key: MAPREDUCE-2901
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2901
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli

 It now fails [ERROR] Failed to execute goal 
 org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project 
 hadoop-yarn-api: Command execution failed. Process exited with an error: 
 1(Exit value: 1) - [Help 1].
 Which doesn't help much.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2906) FindBugs OutOfMemoryError

[Joep Rottinghuis (JIRA)]

FindBugs OutOfMemoryError
-

 Key: MAPREDUCE-2906
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2906
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.22.0
 Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 
7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis


When running the findbugs target from Jenkins, I get an OutOfMemory error.
The effort in FindBugs is set to Max which ends up using a lot of memory to 
go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 
MB max.

We can leave the default to 512M, as long as we pass this as an ant parameter 
which can be overwritten in individual cases through -D, or in the 
build.properties file (either basedir, or user's home directory).


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2906) FindBugs OutOfMemoryError

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13093083#comment-13093083
 ] 

Joep Rottinghuis commented on MAPREDUCE-2906:
-

findbugs:
[mkdir] Created dir: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/test/findbugs
 [findbugs] Executing findbugs from ant task
 [findbugs] Running FindBugs...
 [findbugs] Out of memory
 [findbugs] Total memory: 477M
 [findbugs]  free memory: 45M
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/hadoop-mapred-0.22-joep-0.1.jar
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/hadoop-mapred-examples-0.22-joep-0.1.jar
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/hadoop-mapred-tools-0.22-joep-0.1.jar
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/contrib/capacity-scheduler/hadoop-0.22-joep-0.1-capacity-scheduler.jar
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/contrib/streaming/hadoop-0.22-joep-0.1-streaming.jar
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/contrib/gridmix/hadoop-0.22-joep-0.1-gridmix.jar
 [findbugs] Analyzed: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/contrib/mumak/hadoop-0.22-joep-0.1-mumak.jar
 [findbugs]  Aux: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/ivy/lib/Hadoop/common/ant-1.6.5.jar
...cut 40 lines
 [findbugs]  Aux: 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/ivy/lib/Hadoop/common/xmlenc-0.52.jar
 [findbugs] Exception in thread main java.lang.OutOfMemoryError: Java heap 
space
 [findbugs] at java.util.ArrayList.init(ArrayList.java:112)
 [findbugs] at edu.umd.cs.findbugs.ba.Frame.init(Frame.java:110)
 [findbugs] at 
edu.umd.cs.findbugs.ba.constant.ConstantFrame.init(ConstantFrame.java:32)
 [findbugs] at 
edu.umd.cs.findbugs.ba.constant.ConstantAnalysis.createFact(ConstantAnalysis.java:49)
 [findbugs] at 
edu.umd.cs.findbugs.ba.constant.ConstantAnalysis.createFact(ConstantAnalysis.java:38)
 [findbugs] at 
edu.umd.cs.findbugs.ba.BasicAbstractDataflowAnalysis.lookupOrCreateFact(BasicAbstractDataflowAnalysis.java:156)
 [findbugs] at 
edu.umd.cs.findbugs.ba.BasicAbstractDataflowAnalysis.getResultFact(BasicAbstractDataflowAnalysis.java:69)
... cut lines
 [findbugs] at edu.umd.cs.findbugs.FindBugs2.main(FindBugs2.java:1057)
 [findbugs] Java Result: 1
 [findbugs] Output saved to 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/test/findbugs/hadoop-findbugs-report.xml
 [xslt] Processing 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/test/findbugs/hadoop-findbugs-report.xml
 to 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/test/findbugs/hadoop-findbugs-report.html
 [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
 [xslt] : Error! Premature end of file.
 [xslt] : Error! 
com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of 
file.
 [xslt] Failed to process 
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build/test/findbugs/hadoop-findbugs-report.xml

BUILD FAILED
/hadoop01/jenkins/jobs/hadoop-mapreduce-test/workspace/mapreduce/build.xml:943: 
javax.xml.transform.TransformerException: 
javax.xml.transform.TransformerException: 
com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of 
file.
at 
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
at 
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
... cut 15 lines
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Caused by: javax.xml.transform.TransformerException: 
com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of 
file.
at 
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
at 
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
... 20 more
Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: 
Premature end of file.
at 
com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
at 
com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
at 
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
... 21 more

[jira] [Resolved] (MAPREDUCE-2906) FindBugs OutOfMemoryError

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis resolved MAPREDUCE-2906.
-

Resolution: Not A Problem

Even though same entry does not exist in hadoop-common build on 0.20-security, 
or hadoophdfs on 0.22, the mapreduce build actually already has a parameter for 
this: findbugs.heap.size

 FindBugs OutOfMemoryError
 -

 Key: MAPREDUCE-2906
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2906
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.22.0
 Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in 
 Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis

 When running the findbugs target from Jenkins, I get an OutOfMemory error.
 The effort in FindBugs is set to Max which ends up using a lot of memory to 
 go through all the classes. The jvmargs passed to FindBugs is hardcoded to 
 512 MB max.
 We can leave the default to 512M, as long as we pass this as an ant parameter 
 which can be overwritten in individual cases through -D, or in the 
 build.properties file (either basedir, or user's home directory).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-2752) Build does not pass along properties to contrib builds

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis reassigned MAPREDUCE-2752:
---

Assignee: Joep Rottinghuis

 Build does not pass along properties to contrib builds
 --

 Key: MAPREDUCE-2752
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2752
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Minor
 Fix For: 0.22.0

 Attachments: MAPREDUCE-2752.patch


 Subant call to compile contribs do not pass along parameters from parent 
 build.
 Properties such as hadoop-common.version, asfrepo, offline, etc. are not 
 passed along.
 Result is that build not connected to Internet fails, hdfs proxy refuses to 
 build against own recently built common but rather downloads 0.22-SNAPSHOT 
 from apache again.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis reassigned MAPREDUCE-2753:
---

Assignee: Joep Rottinghuis

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Fix For: 0.22.0

 Attachments: MAPREDUCE-2753-1.patch, MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2610) Inconsistent API JobClient.getQueueAclsForCurrentUser

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2610:


Status: Patch Available  (was: Open)

 Inconsistent API JobClient.getQueueAclsForCurrentUser
 -

 Key: MAPREDUCE-2610
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2610
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.205.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
 Attachments: MAPREDUCE-2610.patch


 Client needs access to the current user's queue name.
 Public method JobClient.getQueueAclsForCurrentUser() returns QueueAclsInfo[].
 The QueueAclsInfo class has default access. A public method should not return 
 a package-private class.
 The QueueAclsInfo class, its two constructors, getQueueName, and 
 getOperations methods should be public.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080105#comment-13080105
 ] 

Joep Rottinghuis commented on MAPREDUCE-2779:
-

Patch looks good.
Affects 0.20-security-* branches as well.

FSDataOutputStream.getPos is not thread safe but then again 
DataOutPutStream.size does not seem to be thread safe either.
Even through the DataOutPutStream.write method is synchronized, 
FSDataOutputStrem.write is not synchronized.
This does not seem to be an issue in the current code path because 
createSplitFiles does not expose out.


 JobSplitWriter.java can't handle large job.split file
 -

 Key: MAPREDUCE-2779
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: job submission
Affects Versions: 0.22.0, 0.23.0
Reporter: Ming Ma
 Attachments: MAPREDUCE-2779-trunk.patch


 We use cascading MultiInputFormat. MultiInputFormat sometimes generates big 
 job.split used internally by hadoop, sometimes it can go beyond 2GB.
 In JobSplitWriter.java, the function that generates such file uses 32bit 
 signed integer to compute offset into job.split.
 writeNewSplits
 ...
 int prevCount = out.size();
 ...
 int currCount = out.size();
 writeOldSplits
 ...
   long offset = out.size();
 ...
   int currLen = out.size();

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2752) Build does not pass along properties to contrib builds

[Joep Rottinghuis (JIRA)]

Build does not pass along properties to contrib builds
--

 Key: MAPREDUCE-2752
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2752
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
Priority: Minor


Subant call to compile contribs do not pass along parameters from parent build.
Properties such as hadoop-common.version, asfrepo, offline, etc. are not passed 
along.
Result is that build not connected to Internet fails, hdfs proxy refuses to 
build against own recently built common but rather downloads 0.22-SNAPSHOT from 
apache again.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis


The generated poms inject the version of mapred itself, but hardcode the 
version of hadoop-common they depend on.
When trying to build downstream projects (HBase), then they will require 
hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled against.

When trying to do an offline build this will fail to resolve as another 
hadoop-common has been installed in the local maven repo.
Even during online build, it should compile against the hadoop-common that hdfs 
compiled against.

When versions mismatch one cannot do a coherent build. That is particularly 
problematic when making simultaneous change in hadoop-common and 
hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2753:


Status: Patch Available  (was: Open)

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
 Attachments: MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2753:


Attachment: MAPREDUCE-2753.patch

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
 Attachments: MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2753:


Status: Open  (was: Patch Available)

Wrong patch for this issue.

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
 Attachments: MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2752) Build does not pass along properties to contrib builds

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2752:


Attachment: MAPREDUCE-2752.patch

 Build does not pass along properties to contrib builds
 --

 Key: MAPREDUCE-2752
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2752
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
Priority: Minor
 Attachments: MAPREDUCE-2752.patch


 Subant call to compile contribs do not pass along parameters from parent 
 build.
 Properties such as hadoop-common.version, asfrepo, offline, etc. are not 
 passed along.
 Result is that build not connected to Internet fails, hdfs proxy refuses to 
 build against own recently built common but rather downloads 0.22-SNAPSHOT 
 from apache again.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073072#comment-13073072
 ] 

Joep Rottinghuis commented on MAPREDUCE-2753:
-

Uploaded new patch MAPREDUCE-2753-1.patch

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
 Attachments: MAPREDUCE-2753-1.patch, MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2753:


Status: Patch Available  (was: Open)

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
 Attachments: MAPREDUCE-2753-1.patch, MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2753) Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-2753:


Attachment: MAPREDUCE-2753-1.patch

 Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT
 ---

 Key: MAPREDUCE-2753
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2753
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
 Environment: RHEL 6.1  Ubuntu 11.04; Sun JDK 1.6_016  Sun JDK 
 1.6.0_26; Ant 1.8.2
Reporter: Joep Rottinghuis
 Attachments: MAPREDUCE-2753-1.patch, MAPREDUCE-2753.patch


 The generated poms inject the version of mapred itself, but hardcode the 
 version of hadoop-common they depend on.
 When trying to build downstream projects (HBase), then they will require 
 hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled 
 against.
 When trying to do an offline build this will fail to resolve as another 
 hadoop-common has been installed in the local maven repo.
 Even during online build, it should compile against the hadoop-common that 
 hdfs compiled against.
 When versions mismatch one cannot do a coherent build. That is particularly 
 problematic when making simultaneous change in hadoop-common and 
 hadoop-mapreduce and you want to try this locally before committing each.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2610) Inconsistent API JobClient.getQueueAclsForCurrentUser

[Joep Rottinghuis (JIRA)]

Inconsistent API JobClient.getQueueAclsForCurrentUser
-

 Key: MAPREDUCE-2610
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2610
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.205.0
Reporter: Joep Rottinghuis


Client needs access to the current user's queue name.
Public method JobClient.getQueueAclsForCurrentUser() returns QueueAclsInfo[].
The QueueAclsInfo class has default access. A public method should not return a 
package-private class.

The QueueAclsInfo class, its two constructors, getQueueName, and getOperations 
methods should be public.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2610) Inconsistent API JobClient.getQueueAclsForCurrentUser

[Joep Rottinghuis (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052718#comment-13052718
 ] 

Joep Rottinghuis commented on MAPREDUCE-2610:
-

Note that this class is deprecated on trunk (see MAPREDUCE-777), but its 
replacement in the o.a.h.mapreduce package does have public access to get to 
the queue and operations.

 Inconsistent API JobClient.getQueueAclsForCurrentUser
 -

 Key: MAPREDUCE-2610
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2610
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.205.0
Reporter: Joep Rottinghuis

 Client needs access to the current user's queue name.
 Public method JobClient.getQueueAclsForCurrentUser() returns QueueAclsInfo[].
 The QueueAclsInfo class has default access. A public method should not return 
 a package-private class.
 The QueueAclsInfo class, its two constructors, getQueueName, and 
 getOperations methods should be public.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-1251) c++ utils doesn't compile

[Joep Rottinghuis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated MAPREDUCE-1251:


Affects Version/s: 0.20.205.0

Problem is related to header cleanup in gcc 3.4.
See comment in HADOOP-5611

This should be applied to branch-0.20-security and the 0.20.205 branch

 c++ utils doesn't compile
 -

 Key: MAPREDUCE-1251
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1251
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.1, 0.20.2, 0.20.205.0, 0.21.0, 0.22.0
 Environment: ubuntu karmic 64-bit
Reporter: Eli Collins
Assignee: Eli Collins
 Fix For: 0.20.2

 Attachments: HDFS-790-1.patch, HDFS-790.patch, MR-1251.patch


 c++ utils doesn't compile on ubuntu karmic 64-bit. The latest patch for 
 HADOOP-5611 needs to be applied first.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira