Incorrect number of map tasks when there are multiple input files
-
Key: HADOOP-960
URL: https://issues.apache.org/jira/browse/HADOOP-960
Project: Hadoop
Issue Type: Bug
Aff
[
https://issues.apache.org/jira/browse/HADOOP-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HADOOP-959:
Attachment: renameToWindows.patch
Here is a patch for review.
> TestCheckpoint fails on Wi
[
https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468459
]
Hairong Kuang commented on HADOOP-928:
--
Hi Doug! Yes, wrapping eliminates all the raw methods and thus makes the
[
https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468456
]
Doug Cutting commented on HADOOP-928:
-
> keep all the raw methods in the FileSystem interface
> DistributedFileSy
[
https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468453
]
Hairong Kuang commented on HADOOP-928:
--
Yes, all you said makes sense. But I'd like to make two changes to your
[
https://issues.apache.org/jira/browse/HADOOP-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468452
]
Milind Bhandarkar commented on HADOOP-958:
--
Here is what I see when I build from command line:
compile-core
[
https://issues.apache.org/jira/browse/HADOOP-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Cutting updated HADOOP-884:
Resolution: Fixed
Fix Version/s: 0.11.0
Status: Resolved (was: Patch Available)
[
https://issues.apache.org/jira/browse/HADOOP-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468450
]
Doug Cutting commented on HADOOP-958:
-
Warnings from what? Eclipse? When I build and/or run unit tests from the
TestCheckpoint fails on Windows
---
Key: HADOOP-959
URL: https://issues.apache.org/jira/browse/HADOOP-959
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
A
Building Hadoop results in a lot of warnings
Key: HADOOP-958
URL: https://issues.apache.org/jira/browse/HADOOP-958
Project: Hadoop
Issue Type: Improvement
Reporter: eric baldeschwieler
[
https://issues.apache.org/jira/browse/HADOOP-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468449
]
Runping Qi commented on HADOOP-957:
---
Certainly. That perhaps the most convenient way to do.
> Map/Reduce job with
[
https://issues.apache.org/jira/browse/HADOOP-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468448
]
Doug Cutting commented on HADOOP-957:
-
Could this be done as a subclass of SequenceFileOutputFormat,
RecordOutpu
Map/Reduce job with SequenceFileOutputFormat should be able to add user
specified metadata to the output file
-
Key: HADOOP-957
URL: https://issues.apache.
[
https://issues.apache.org/jira/browse/HADOOP-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468447
]
Doug Cutting commented on HADOOP-956:
-
We could add a 'file' command, like the unix 'file' program, that uses var
[
https://issues.apache.org/jira/browse/HADOOP-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468446
]
Andrew McNabb commented on HADOOP-950:
--
The hanging seems to have been caused by blocked writes to a pipe. I ha
DFSShell/Wen UI should return the metadata of a file
Key: HADOOP-956
URL: https://issues.apache.org/jira/browse/HADOOP-956
Project: Hadoop
Issue Type: New Feature
Reporter: Run
[
https://issues.apache.org/jira/browse/HADOOP-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468438
]
Doug Cutting commented on HADOOP-432:
-
Here's a proposed design:
- change the FS shell's 'rm' command to move fi
Metrics.report() metricValue parameter type should be float, not long
-
Key: HADOOP-955
URL: https://issues.apache.org/jira/browse/HADOOP-955
Project: Hadoop
Issue Type: Imp
[
https://issues.apache.org/jira/browse/HADOOP-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468437
]
Konstantin Shvachko commented on HADOOP-951:
chooseTargets() iterates over the heartbeats list. Looks lik
Metrics should offer complete set of static report methods or none at all
-
Key: HADOOP-954
URL: https://issues.apache.org/jira/browse/HADOOP-954
Project: Hadoop
Issue T
Arkady Borkovsky wrote:
So it is 27*D+13*M vs. 17*D+2*M . With D
[
https://issues.apache.org/jira/browse/HADOOP-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468427
]
Tom White commented on HADOOP-952:
--
I was planning on using my S3 storage - at least until the AMI got too popular
How savings can be much higher than 1/3.
(probably I'm already preaching to the quire)
1. The situation I have in mind looks like this:
-- you have a data set M with 10^11 to 10^12 records that is produced
once in a while and used hundreds of times (before its new version is
generated). This
[
https://issues.apache.org/jira/browse/HADOOP-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468426
]
Doug Cutting commented on HADOOP-952:
-
This would be great to have! Someone would need to donate the S3 storage
huge log files
--
Key: HADOOP-953
URL: https://issues.apache.org/jira/browse/HADOOP-953
Project: Hadoop
Issue Type: Improvement
Affects Versions: 0.10.1
Environment: N/A
Reporter: Andrew McNabb
On our s
[
https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468425
]
Doug Cutting commented on HADOOP-928:
-
> It's hard to for me to see how to remove createRaw and openRaw from
> F
[
https://issues.apache.org/jira/browse/HADOOP-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sameer Paranjpye reassigned HADOOP-432:
---
Assignee: Doug Cutting (was: Wendy Chien)
> support undelete, snapshots, or other me
[
https://issues.apache.org/jira/browse/HADOOP-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468423
]
Hadoop QA commented on HADOOP-884:
--
+1, because
http://issues.apache.org/jira/secure/attachment/12349853/hadoop-884
Create a public (shared) Hadoop EC2 AMI
---
Key: HADOOP-952
URL: https://issues.apache.org/jira/browse/HADOOP-952
Project: Hadoop
Issue Type: Improvement
Components: scripts
Affects Versions:
java.util.ConcurrentModificationException in FSNamesystem.chooseTargets
-
Key: HADOOP-951
URL: https://issues.apache.org/jira/browse/HADOOP-951
Project: Hadoop
Issue T
[
https://issues.apache.org/jira/browse/HADOOP-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468419
]
Andrew McNabb commented on HADOOP-950:
--
I'm doing a test right now. The output of hadoop-streaming is being sen
[
https://issues.apache.org/jira/browse/HADOOP-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468418
]
Owen O'Malley commented on HADOOP-926:
--
The problem with command line switches is that not all applications use
[
https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468417
]
Hairong Kuang commented on HADOOP-928:
--
> The 'raw' FileSystem methods would be removed.
It's hard to for me to
[
https://issues.apache.org/jira/browse/HADOOP-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-884:
-
Attachment: hadoop-884.patch
The attached patch includes the Hadoop EC2 scripts in contrib/ec2/bin. I thi
[
https://issues.apache.org/jira/browse/HADOOP-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HADOOP-884:
-
Status: Patch Available (was: In Progress)
> Create scripts to run Hadoop on Amazon EC2
> --
Job completes but command doesn't return
Key: HADOOP-950
URL: https://issues.apache.org/jira/browse/HADOOP-950
Project: Hadoop
Issue Type: Bug
Components: contrib/streaming
Affects Versi
[
https://issues.apache.org/jira/browse/HADOOP-949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Cutting resolved HADOOP-949.
-
Resolution: Fixed
I just reverted the patch from HADOOP-248. Thanks, Nigel!
> reduce hangs afte
[
https://issues.apache.org/jira/browse/HADOOP-248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Cutting reopened HADOOP-248:
-
I just reverted this, since it was causing things to hang.
> locating map outputs via random probing
[
https://issues.apache.org/jira/browse/HADOOP-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468405
]
Doug Judd commented on HADOOP-946:
--
Ok, I see your reasoning. Go ahead and downgrade this one or remove it
entirel
Nigel Daley wrote:
Currently, when I run checkstyle (with sun's style conventions, which
we use for Hadoop) I get tons of warnings on almost each and every
source file.
...which is perhaps one reason it is too late to use CheckStyle.
We can't use CheckStyle, but not for that reason: it's LGP
[
https://issues.apache.org/jira/browse/HADOOP-948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468402
]
Doug Cutting commented on HADOOP-948:
-
CheckStyle is LGPL, so cannot be included in an Apache project.
I would n
[
https://issues.apache.org/jira/browse/HADOOP-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468399
]
Doug Cutting commented on HADOOP-946:
-
> Seems like you'd need an atomic record append API to handle this.
Using
On a related note, we should really use CheckStyle (http://
checkstyle.sourceforge.net) to enforce the coding styles.
Currently, when I run checkstyle (with sun's style conventions,
which we use for Hadoop) I get tons of warnings on almost each and
every source file.
...which is perhaps on
[
https://issues.apache.org/jira/browse/HADOOP-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468396
]
Nigel Daley commented on HADOOP-949:
Task tracker log:
... [reduce task launches]
2007-01-27 04:35:49,483 INFO o
reduce hangs after applying HADOOP-248 on 20 node cluster
-
Key: HADOOP-949
URL: https://issues.apache.org/jira/browse/HADOOP-949
Project: Hadoop
Issue Type: Bug
Components: m
[
https://issues.apache.org/jira/browse/HADOOP-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468395
]
Doug Cutting commented on HADOOP-941:
-
> It is not clear that Hadoop considers the record io a stable interface t
[
https://issues.apache.org/jira/browse/HADOOP-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Bowen updated HADOOP-948:
---
Attachment: (was: hadoop-948.patch)
> Coding style issues
>
>
>
[
https://issues.apache.org/jira/browse/HADOOP-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Bowen updated HADOOP-948:
---
Attachment: hadoop-948.patch
> Coding style issues
>
>
> Key: H
[
https://issues.apache.org/jira/browse/HADOOP-948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Bowen updated HADOOP-948:
---
Attachment: hadoop-948.patch
> Coding style issues
>
>
> Key: H
[
https://issues.apache.org/jira/browse/HADOOP-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468389
]
Doug Judd commented on HADOOP-946:
--
> Actually, the ability to keep a large number of maps (one per block) and a
>
[
https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sami Siren updated HADOOP-917:
--
Attachment: sequencefile-npe-test.patch
Attached patch reproduces the NPE in trunk version of hadoop:
[
https://issues.apache.org/jira/browse/HADOOP-948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468385
]
Milind Bhandarkar commented on HADOOP-948:
--
On a related note, we should really use CheckStyle
(http://chec
[
https://issues.apache.org/jira/browse/HADOOP-942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi resolved HADOOP-942.
-
Resolution: Duplicate
Dupilcate of HADOOP-632.
> dfsadmin -report returns deadnode as "In Servi
[
https://issues.apache.org/jira/browse/HADOOP-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468380
]
Milind Bhandarkar commented on HADOOP-926:
--
>I will likely leave a config option that retains the current be
[
https://issues.apache.org/jira/browse/HADOOP-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468379
]
Yoram Arnon commented on HADOOP-926:
wouldn't simply running the command in the background provide the current
b
Coding style issues
Key: HADOOP-948
URL: https://issues.apache.org/jira/browse/HADOOP-948
Project: Hadoop
Issue Type: Bug
Components: metrics
Reporter: David Bowen
Priority: Minor
I w
[
https://issues.apache.org/jira/browse/HADOOP-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468378
]
Doug Cutting commented on HADOOP-937:
-
Do you think this change to the protocol is entirely back-compatible, or s
[
https://issues.apache.org/jira/browse/HADOOP-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468377
]
Owen O'Malley commented on HADOOP-926:
--
To be honest, I've used it as a feature too, but it confuses users who d
[
https://issues.apache.org/jira/browse/HADOOP-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468376
]
Milind Bhandarkar commented on HADOOP-941:
--
>Personally, I'm fine with it either way.
Ben,
The way I see
[
https://issues.apache.org/jira/browse/HADOOP-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468375
]
Doug Cutting commented on HADOOP-946:
-
Note that there is a standard workaround for this: have your map function
isReplicationInProgress() is very heavyweight
-
Key: HADOOP-947
URL: https://issues.apache.org/jira/browse/HADOOP-947
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter:
[
https://issues.apache.org/jira/browse/HADOOP-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468374
]
Benjamin Reed commented on HADOOP-941:
--
Using the Hadoop jar would be an extreme cognitive burden. Plunking a
m
Map directly to HDFS or reduce()
Key: HADOOP-946
URL: https://issues.apache.org/jira/browse/HADOOP-946
Project: Hadoop
Issue Type: New Feature
Components: mapred
Environment: all
[
https://issues.apache.org/jira/browse/HADOOP-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hairong Kuang reassigned HADOOP-928:
Assignee: Hairong Kuang (was: Doug Cutting)
> make checksums optional per FileSystem
> ---
Hi. I'm getting some weird errors on the task trackers, which die as a
result. I was wondering if this is a known problem. These are showing
up in 0.10.0; I'm upgrading to 0.10.1 right now, but I thought I'd send
the logs along in case it's not something that's fixed in 0.10.1. I
hope this is h
[
https://issues.apache.org/jira/browse/HADOOP-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468364
]
Doug Cutting commented on HADOOP-926:
-
> I always thought this was a feature ... ;)
Heh. That was my first thou
[
https://issues.apache.org/jira/browse/HADOOP-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468361
]
Doug Cutting commented on HADOOP-941:
-
> I will admit to being one of the motivators of this bug.
So how much of
[
https://issues.apache.org/jira/browse/HADOOP-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HADOOP-934:
-
Attachment: HADOOP-934_20070129_1.patch
Simple fix which catches the MetricsException and just lo
Arkady Borkovsky wrote:
Does this model assume that the size of the output of reduce is similar
to the size of the input?
An important class of applications (mentioned in this thread before)
uses two inputs:
-- M ("master file") -- very large, presorted and not changing from run
to run,
-- D
[
https://issues.apache.org/jira/browse/HADOOP-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468359
]
Arun C Murthy commented on HADOOP-934:
--
Ok, this entails a simple fix; however what are everyone's view on how w
[
https://issues.apache.org/jira/browse/HADOOP-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reassigned HADOOP-934:
Assignee: Arun C Murthy
> TaskTracker sends duplicate status when updating task metrics thr
Doug's calculation shows that the total gain can be only 1/3 (15 are
unavoidable, and taking advantage of largely pre-sorted input reduces
overhead from 12/27 to 3/18, so the maximum total gain is 27->18.)
Does this model assume that the size of the output of reduce is similar
to the size
[
https://issues.apache.org/jira/browse/HADOOP-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468351
]
dhruba borthakur commented on HADOOP-944:
-
I agree.
We also should keep in mind that reducing the number of
[
https://issues.apache.org/jira/browse/HADOOP-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12468247
]
Andrzej Bialecki commented on HADOOP-926:
--
I alaways thought this was a feature ... ;) I would sometimes fo
74 matches
Mail list logo
