[jira] Updated: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-5194: - Release Note: This patch disables usage of setsid and pidfiles on Windows. Status: Patc

[jira] Updated: (HADOOP-4831) Have end to end tests based on MiniMRCluster to verify the correct behaviour of job initialization.

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-4831: -- Status: Open (was: Patch Available) Cancelling this patch, since it hasn't moved in a long ti

[jira] Updated: (HADOOP-4374) JVM should not be killed but given an opportunity to exit gracefully

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-4374: - Release Note: This patch (1) Adds a shutdownHook that does syncLogs sothat logs of the current

[jira] Commented: (HADOOP-4374) JVM should not be killed but given an opportunity to exit gracefully

2009-03-25 Thread Amareshwari Sriramadasu (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689371#action_12689371 ] Amareshwari Sriramadasu commented on HADOOP-4374: - +1 Patch looks fine. >

[jira] Updated: (HADOOP-4374) JVM should not be killed but given an opportunity to exit gracefully

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-4374: - Attachment: HADOOP-4374.v1.1.patch Attaching patch with the suggested changes. > JVM should not

[jira] Commented: (HADOOP-5473) Race condition in command-line kill for a task

2009-03-25 Thread Jothi Padmanabhan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689369#action_12689369 ] Jothi Padmanabhan commented on HADOOP-5473: --- +1 > Race condition in command-lin

[jira] Commented: (HADOOP-4374) JVM should not be killed but given an opportunity to exit gracefully

2009-03-25 Thread Amareshwari Sriramadasu (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689366#action_12689366 ] Amareshwari Sriramadasu commented on HADOOP-4374: - One minor comment: Afte

[jira] Updated: (HADOOP-4374) JVM should not be killed but given an opportunity to exit gracefully

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-4374: - Attachment: HADOOP-4374.v1.patch Attaching new patch removing System.exit from a catch block of

[jira] Commented: (HADOOP-5473) Race condition in command-line kill for a task

2009-03-25 Thread Amareshwari Sriramadasu (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689363#action_12689363 ] Amareshwari Sriramadasu commented on HADOOP-5473: - test-patch result : {no

[jira] Commented: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Vinod K V (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689361#action_12689361 ] Vinod K V commented on HADOOP-5194: --- +1 for the patch from my side. Can the committer pl

[jira] Commented: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689360#action_12689360 ] Ravi Gummadi commented on HADOOP-5194: -- unit tests passed on my linux machine. The

[jira] Updated: (HADOOP-5066) ant binary should not compile docs

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5066: -- Fix Version/s: (was: 0.21.0) 0.20.0 Committed to 0.20 > ant binary sho

[jira] Created: (HADOOP-5575) regarding output dir usage

2009-03-25 Thread girija l (JIRA)
regarding output dir usage -- Key: HADOOP-5575 URL: https://issues.apache.org/jira/browse/HADOOP-5575 Project: Hadoop Core Issue Type: Task Environment: ubuntu hardy Reporter: girija l

[jira] Commented: (HADOOP-5066) ant binary should not compile docs

2009-03-25 Thread Vinod K V (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689352#action_12689352 ] Vinod K V commented on HADOOP-5066: --- bq. .. I'd like to commit this to 0.20 +1 > ant bi

Re: Limit of 64 slots when doing a map-side join

2009-03-25 Thread jason hadoop
Be aware that there is a job run time cost for each additional data set in your join. On the clusters we were working with, 2ghz xeon dell 2950's, each additional data set in the join operator added roughly 30 seconds to the job run time. As a result of that, we would merge data sets in groups of

[jira] Updated: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-5194: - Attachment: HADOOP-5194.v1.patch Vinod, Attaching patch with the changes. > DiskErrorException

[jira] Commented: (HADOOP-5189) Integration with BookKeeper logging system

2009-03-25 Thread Benjamin Reed (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689343#action_12689343 ] Benjamin Reed commented on HADOOP-5189: --- great job luca! konstantin and dhruba can w

[jira] Commented: (HADOOP-4665) Add preemption to the fair scheduler

2009-03-25 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689344#action_12689344 ] Hadoop QA commented on HADOOP-4665: --- -1 overall. Here are the results of testing the la

[jira] Commented: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Vinod K V (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689342#action_12689342 ] Vinod K V commented on HADOOP-5194: --- Patch looks good overall to me. Two minor nits. -

Re: Design for security in Hadoop

2009-03-25 Thread Amandeep Khurana
On Wed, Mar 25, 2009 at 1:43 PM, Kan Zhang wrote: > > > > On 3/25/09 1:04 PM, "Kan Zhang" wrote: > > > > > > > > > On 3/25/09 12:15 PM, "Amandeep Khurana" wrote: > > > >> On Wed, Mar 25, 2009 at 2:49 AM, Doug Cutting > wrote: > >> > >>> Amandeep Khurana wrote: > >>> > 1. The Jira covers o

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689330#action_12689330 ] Jakob Homan commented on HADOOP-5438: - bq. The reason that i have not assigned a short

[jira] Resolved: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved HADOOP-5363. --- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] I committed thi

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread He Yongqiang (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689315#action_12689315 ] He Yongqiang commented on HADOOP-5438: -- The reason that i have not assigned a short m

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689310#action_12689310 ] Konstantin Shvachko commented on HADOOP-5438: - Sounds like a reasonable idea.

[jira] Commented: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689308#action_12689308 ] Chris Douglas commented on HADOOP-5363: --- +1 Looks good {noformat} [exec] +1 ov

[jira] Updated: (HADOOP-4045) Increment checkpoint if we see failures in rollEdits

2009-03-25 Thread Boris Shkolnik (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HADOOP-4045: --- Affects Version/s: (was: 0.19.0) Status: Patch Available (was: Open) > I

[jira] Updated: (HADOOP-4045) Increment checkpoint if we see failures in rollEdits

2009-03-25 Thread Boris Shkolnik (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HADOOP-4045: --- Attachment: HADOOP-4045.patch > Increment checkpoint if we see failures in rollEdits > -

[jira] Updated: (HADOOP-4584) Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode

2009-03-25 Thread Suresh Srinivas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-4584: Attachment: Design.pdf > Slow generation of blockReport at DataNode causes delay of sendin

[jira] Updated: (HADOOP-5561) Javadoc-dev ant target runs out of heap space

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5561: -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status:

[jira] Updated: (HADOOP-5423) It should be posible to specify metadata for the output file produced by SequenceFile.Sorter.sort

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5423: -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status:

[jira] Commented: (HADOOP-5467) Create an offline fsimage image viewer

2009-03-25 Thread dhruba borthakur (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689285#action_12689285 ] dhruba borthakur commented on HADOOP-5467: -- > How about src/tools/o/a/h/hdfs/tool

[jira] Commented: (HADOOP-5467) Create an offline fsimage image viewer

2009-03-25 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689287#action_12689287 ] Hadoop QA commented on HADOOP-5467: --- -1 overall. Here are the results of testing the la

[jira] Updated: (HADOOP-5574) Separate replication setting out from FSNamesystem

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-5574: --- Attachment: 5574_20090325.patch 5574_20090325.patch: introduced a new class

[jira] Updated: (HADOOP-5189) Integration with BookKeeper logging system

2009-03-25 Thread Luca Telloli (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Telloli updated HADOOP-5189: - Attachment: HADOOP-5189.patch I'm posting a new patch for the integration of BookKeeper with HDF

Re: Design for security in Hadoop

2009-03-25 Thread Amandeep Khurana
> > > > > On 3/25/09 12:12 PM, "Amandeep Khurana" wrote: > > >> > >> > >> On 3/20/09 2:47 PM, "Amandeep Khurana" wrote: > >> > >>> > >>> 2. The Jira doesnt have cover the access control aspect of things. As a > >>> client, I can skip talking to the NN and get blocks from the DN > straight > >>> a

[jira] Created: (HADOOP-5574) Separate replication setting out from FSNamesystem

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
Separate replication setting out from FSNamesystem -- Key: HADOOP-5574 URL: https://issues.apache.org/jira/browse/HADOOP-5574 Project: Hadoop Core Issue Type: Improvement Components:

[jira] Resolved: (HADOOP-3662) Misleading log message in computeReplicationWork()

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HADOOP-3662. Resolution: Invalid This issue is no longer valid. Tomas, thanks for work

[jira] Commented: (HADOOP-5467) Create an offline fsimage image viewer

2009-03-25 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689269#action_12689269 ] Jakob Homan commented on HADOOP-5467: - bq. In that case, don't you want this tool to r

[jira] Updated: (HADOOP-5561) Javadoc-dev ant target runs out of heap space

2009-03-25 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HADOOP-5561: Attachment: HADOOP-5561.patch Updated patch to use javadoc.maxmemory property and have all the jav

[jira] Updated: (HADOOP-5561) Javadoc-dev ant target runs out of heap space

2009-03-25 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HADOOP-5561: Status: Patch Available (was: Open) submitting new patch > Javadoc-dev ant target runs out of he

[jira] Commented: (HADOOP-5528) Binary partitioner

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689260#action_12689260 ] Chris Douglas commented on HADOOP-5528: --- * It would make sense for the deprecated pa

[jira] Updated: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread zhiyong zhang (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhiyong zhang updated HADOOP-5363: -- Attachment: HADOOP-5363.patch Make a new ProxyFileForward.java as a subclass of ProxyForwardSe

[jira] Commented: (HADOOP-5573) TestBackupNode sometimes fails

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689249#action_12689249 ] Tsz Wo (Nicholas), SZE commented on HADOOP-5573: The failures can be repro

[jira] Commented: (HADOOP-5573) TestBackupNode sometimes fails

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689248#action_12689248 ] Tsz Wo (Nicholas), SZE commented on HADOOP-5573: Here are more details: -

[jira] Created: (HADOOP-5573) TestBackupNode sometimes fails

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
TestBackupNode sometimes fails -- Key: HADOOP-5573 URL: https://issues.apache.org/jira/browse/HADOOP-5573 Project: Hadoop Core Issue Type: Bug Components: dfs Reporter: Tsz Wo (Nicholas), SZE

[jira] Updated: (HADOOP-5191) After creation and startup of the hadoop namenode on AIX or Solaris, you will only be allowed to connect to the namenode via hostname but not IP.

2009-03-25 Thread Raghu Angadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghu Angadi updated HADOOP-5191: - Resolution: Fixed Release Note: Accessing HDFS with any ip, hostname, or proxy should work

Re: Highly Available HDFS ???

2009-03-25 Thread Dhruba Borthakur
We are running a real-timeish cluster that is configured as two overlapping hdfs clusters. The namenodes run on two different machines but the datanodes run on the same set of slaves machines. (Each slave machine actually runs two datanode instances.) The entire storage space is shared between the

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689237#action_12689237 ] Jakob Homan commented on HADOOP-5438: - You can also use these to combine modes, simila

Re: Design for security in Hadoop

2009-03-25 Thread Kan Zhang
On 3/25/09 1:04 PM, "Kan Zhang" wrote: > > > > On 3/25/09 12:15 PM, "Amandeep Khurana" wrote: > >> On Wed, Mar 25, 2009 at 2:49 AM, Doug Cutting wrote: >> >>> Amandeep Khurana wrote: >>> 1. The Jira covers only authentication using Kerberos. I dont think Kerberos is the

[jira] Commented: (HADOOP-5530) The "Heap Size" in HDFS web ui may not be accurate

2009-03-25 Thread dhruba borthakur (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689235#action_12689235 ] dhruba borthakur commented on HADOOP-5530: -- I have a machine on which the namenod

[jira] Commented: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread zhiyong zhang (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689234#action_12689234 ] zhiyong zhang commented on HADOOP-5363: --- That's a good suggestion. Further, to make

Re: Highly Available HDFS ???

2009-03-25 Thread Konstantin Shvachko
With recently introduced BackupNode HA becomes more feasible. Although it is not done yet and as Sunjay mentioned the plans are not clear in that direction. There is a presentation related to HA on hadoop wiki: http://wiki.apache.org/hadoop/HadoopPresentations http://files.meetup.com/1228907/Had

[jira] Updated: (HADOOP-5571) TupleWritable can return incorrect results if it contains more than 32 values

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5571: -- Assignee: Jingkei Ly Status: Patch Available (was: Open) > TupleWritable can return inc

[jira] Commented: (HADOOP-5530) The "Heap Size" in HDFS web ui may not be accurate

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689232#action_12689232 ] Tsz Wo (Nicholas), SZE commented on HADOOP-5530: I think this is a present

[jira] Updated: (HADOOP-4584) Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode

2009-03-25 Thread Suresh Srinivas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-4584: Attachment: 4584.brthread.3.patch New patch with additional changes: # {{FSDataset.checkAn

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread Jakob Homan (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689229#action_12689229 ] Jakob Homan commented on HADOOP-5438: - The Java-centric way to do C-style bit packing

[jira] Commented: (HADOOP-1491) After successful distcp, couple of checksum error files

2009-03-25 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689228#action_12689228 ] Hudson commented on HADOOP-1491: Integrated in Hadoop-trunk #790 (See [http://hudson.zone

[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave() method

2009-03-25 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689223#action_12689223 ] Hudson commented on HADOOP-5374: Integrated in Hadoop-trunk #790 (See [http://hudson.zone

[jira] Commented: (HADOOP-5557) Two minor problems in TestOverReplicatedBlocks

2009-03-25 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689226#action_12689226 ] Hudson commented on HADOOP-5557: Integrated in Hadoop-trunk #790 (See [http://hudson.zone

[jira] Commented: (HADOOP-4719) The ls shell command documentation is out-dated

2009-03-25 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689225#action_12689225 ] Hudson commented on HADOOP-4719: Integrated in Hadoop-trunk #790 (See [http://hudson.zone

[jira] Commented: (HADOOP-5210) Reduce Task Progress shows > 100% when the total size of map outputs (for a single reducer) is high

2009-03-25 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689224#action_12689224 ] Hudson commented on HADOOP-5210: Integrated in Hadoop-trunk #790 (See [http://hudson.zone

[jira] Commented: (HADOOP-5564) hadoop command uses large JVM heap size

2009-03-25 Thread Hudson (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689227#action_12689227 ] Hudson commented on HADOOP-5564: Integrated in Hadoop-trunk #790 (See [http://hudson.zone

[jira] Commented: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689222#action_12689222 ] Chris Douglas commented on HADOOP-5363: --- For the forwarding servlets, I was thinking

Re: Design for security in Hadoop

2009-03-25 Thread Amandeep Khurana
On Wed, Mar 25, 2009 at 12:23 PM, Kan Zhang wrote: > > > > On 3/25/09 2:49 AM, "Doug Cutting" wrote: > > >> 2. The Jira doesnt have cover the access control aspect of things. As a > >> client, I can skip talking to the NN and get blocks from the DN straight > >> away. There is no way to prevent

[jira] Updated: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5363: -- Status: Open (was: Patch Available) Unfortunately, HADOOP-5390 conflicts with this patch. Wou

[jira] Commented: (HADOOP-5066) ant binary should not compile docs

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689218#action_12689218 ] Chris Douglas commented on HADOOP-5066: --- If nobody objects, I'd like to commit this

[jira] Updated: (HADOOP-5390) hdfsproxy includes duplicate jars in tarball, source in binary tarball

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5390: -- Resolution: Fixed Fix Version/s: 0.20.0 Hadoop Flags: [Reviewed] Status:

Re: Design for security in Hadoop

2009-03-25 Thread Kan Zhang
On 3/25/09 12:15 PM, "Amandeep Khurana" wrote: > On Wed, Mar 25, 2009 at 2:49 AM, Doug Cutting wrote: > >> Amandeep Khurana wrote: >> >>> 1. The Jira covers only authentication using Kerberos. I dont think >>> Kerberos >>> is the best way to do it since I feel the scalability is limited. Al

[jira] Updated: (HADOOP-5565) The job instrumentation API needs to have a method for finalizeJob,

2009-03-25 Thread Chris Douglas (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5565: -- Resolution: Fixed Fix Version/s: 0.20.0 Hadoop Flags: [Reviewed] Status:

Re: Design for security in Hadoop

2009-03-25 Thread Kan Zhang
On 3/25/09 12:12 PM, "Amandeep Khurana" wrote: >> >> >> On 3/20/09 2:47 PM, "Amandeep Khurana" wrote: >> >>> >>> 2. The Jira doesnt have cover the access control aspect of things. As a >>> client, I can skip talking to the NN and get blocks from the DN straight >>> away. There is no way t

Re: Highly Available HDFS ???

2009-03-25 Thread Sanjay Radia
On Mar 25, 2009, at 12:07 PM, Sangmin Lee wrote: Hi all, I am wondering if there is any effort or plans on HA (Highly Available) HDFS out there. Currently, NameNode is single point of failure and recovery requires human intervention. Many (and probably most) users of hadoop are using hd

Re: Design for security in Hadoop

2009-03-25 Thread Kan Zhang
On 3/25/09 2:49 AM, "Doug Cutting" wrote: >> 2. The Jira doesnt have cover the access control aspect of things. As a >> client, I can skip talking to the NN and get blocks from the DN straight >> away. There is no way to prevent it. This paper takes care of that aspect as >> well. > > The int

[jira] Updated: (HADOOP-5531) Disable Chukwa unit tests on branch-0.20

2009-03-25 Thread Robert Chansler (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Chansler updated HADOOP-5531: Component/s: contrib/chukwa > Disable Chukwa unit tests on branch-0.20 > -

Re: Design for security in Hadoop

2009-03-25 Thread Amandeep Khurana
On Wed, Mar 25, 2009 at 2:49 AM, Doug Cutting wrote: > Amandeep Khurana wrote: > >> 1. The Jira covers only authentication using Kerberos. I dont think >> Kerberos >> is the best way to do it since I feel the scalability is limited. All keys >> have to be negotiated by the Kerberos server. >> > >

Re: Design for security in Hadoop

2009-03-25 Thread Amandeep Khurana
> > > On 3/20/09 2:47 PM, "Amandeep Khurana" wrote: > > > > > 2. The Jira doesnt have cover the access control aspect of things. As a > > client, I can skip talking to the NN and get blocks from the DN straight > > away. There is no way to prevent it. This paper takes care of that aspect > as > >

Highly Available HDFS ???

2009-03-25 Thread Sangmin Lee
Hi all, I am wondering if there is any effort or plans on HA (Highly Available) HDFS out there. Currently, NameNode is single point of failure and recovery requires human intervention. In addition, the recovered NameNode may not same as one before the failure. Is there any plans or ongoing effort

[jira] Updated: (HADOOP-5468) Change Hadoop doc menu to sub-menus

2009-03-25 Thread Robert Chansler (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Chansler updated HADOOP-5468: Release Note: Reformatted HTML documentation for Hadoop to use submenus at the left column

[jira] Updated: (HADOOP-5531) Disable Chukwa unit tests on branch-0.20

2009-03-25 Thread Robert Chansler (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Chansler updated HADOOP-5531: Release Note: Disabled Chukwa unit tests for 0.20 branch only. > Disable Chukwa unit tests

[jira] Updated: (HADOOP-1230) Replace parameters with context objects in Mapper, Reducer, Partitioner, InputFormat, and OutputFormat classes

2009-03-25 Thread Robert Chansler (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Chansler updated HADOOP-1230: Release Note: Replaced parameters with context obejcts in Mapper, Reducer, Partitioner, In

Re: Design for security in Hadoop

2009-03-25 Thread Kan Zhang
On 3/20/09 2:47 PM, "Amandeep Khurana" wrote: > > 2. The Jira doesnt have cover the access control aspect of things. As a > client, I can skip talking to the NN and get blocks from the DN straight > away. There is no way to prevent it. This paper takes care of that aspect as > well. > Have

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689189#action_12689189 ] Konstantin Shvachko commented on HADOOP-5438: - Dhruba, this will not compile.

[jira] Updated: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-5194: - Attachment: HADOOP-5194.patch Attaching patch that disables usage of setsid and pidfiles for win

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread dhruba borthakur (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689179#action_12689179 ] dhruba borthakur commented on HADOOP-5438: -- +1 for Konstantin's proposal with a m

[jira] Commented: (HADOOP-5438) Merge FileSystem.create and FileSystem.append

2009-03-25 Thread Konstantin Shvachko (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689178#action_12689178 ] Konstantin Shvachko commented on HADOOP-5438: - You can use exact values, like

[jira] Commented: (HADOOP-5510) logSync() may block NameNode forever.

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689177#action_12689177 ] Tsz Wo (Nicholas), SZE commented on HADOOP-5510: Got a NPE in EditLogBacku

[jira] Commented: (HADOOP-4343) Adding user and service-to-service authentication to Hadoop

2009-03-25 Thread Kan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689174#action_12689174 ] Kan Zhang commented on HADOOP-4343: --- An additional benefit of using Hadoop proprietary d

[jira] Updated: (HADOOP-5194) DiskErrorException in TaskTracker when running a job

2009-03-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-5194: --- Environment: Windows, Cygwin (was: Hudson (Solaris?), Windows, Cygwin) > I

Re: Limit of 64 slots when doing a map-side join

2009-03-25 Thread Jingkei Ly
Yes, we are leaning on the map-side join package quite heavily too - it is an excellent addition to the MapReduce model that's proving really useful. However, while HADOOP-5571 is an immediate problem for us, I can imagine that we will probably be wanting to join over 64 files soon as well, especia

[jira] Commented: (HADOOP-5528) Binary partitioner

2009-03-25 Thread Klaas Bosteels (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689165#action_12689165 ] Klaas Bosteels commented on HADOOP-5528: Any further comments on the latest patch?

[jira] Commented: (HADOOP-5553) Change modifier of SequenceFile.CompressedBytes and SequenceFile.UncompressedBytes from private to public

2009-03-25 Thread He Yongqiang (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689159#action_12689159 ] He Yongqiang commented on HADOOP-5553: -- Thanks, Hong. Skipping unneeded data is indee

[jira] Commented: (HADOOP-5553) Change modifier of SequenceFile.CompressedBytes and SequenceFile.UncompressedBytes from private to public

2009-03-25 Thread Hong Tang (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689155#action_12689155 ] Hong Tang commented on HADOOP-5553: --- bq. Since SequenceFile provides a public methos nex

[jira] Commented: (HADOOP-5363) Proxying for multiple HDFS clusters of different versions

2009-03-25 Thread Hadoop QA (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689145#action_12689145 ] Hadoop QA commented on HADOOP-5363: --- +1 overall. Here are the results of testing the la

Re: Limit of 64 slots when doing a map-side join

2009-03-25 Thread jason hadoop
That code is highly optimized and quite difficult to follow. We have always limited our joins to 31 members and ignored the problem. But I think your jira and fixing it are the correct choices. There is, in my opinion, a decent write up on how to use map side joins in chapter 8 of my book, so I su

[jira] Commented: (HADOOP-4665) Add preemption to the fair scheduler

2009-03-25 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689127#action_12689127 ] Matei Zaharia commented on HADOOP-4665: --- The patch actually makes sure that no job i

[jira] Updated: (HADOOP-4374) JVM should not be killed but given an opportunity to exit gracefully

2009-03-25 Thread Ravi Gummadi (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated HADOOP-4374: - Attachment: HADOOP-4374.patch Attaching patch that has the following: (1) Added a shutdownHook

[jira] Commented: (HADOOP-5572) The map progress value should have a separate phase for doing the final sort.

2009-03-25 Thread Arun C Murthy (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689125#action_12689125 ] Arun C Murthy commented on HADOOP-5572: --- +1 > The map progress value should have a

[jira] Commented: (HADOOP-5572) The map progress value should have a separate phase for doing the final sort.

2009-03-25 Thread Owen O'Malley (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689122#action_12689122 ] Owen O'Malley commented on HADOOP-5572: --- I think we should reserve 33% of the map ta

[jira] Created: (HADOOP-5572) The map progress value should have a separate phase for doing the final sort.

2009-03-25 Thread Owen O'Malley (JIRA)
The map progress value should have a separate phase for doing the final sort. - Key: HADOOP-5572 URL: https://issues.apache.org/jira/browse/HADOOP-5572 Project: Hadoop Core

Limit of 64 slots when doing a map-side join

2009-03-25 Thread Jingkei Ly
Am I right in thinking that the CompositeInputFormat is limited to joining 64 files? I believe this comes about because TupleWritable uses a single long-type instance field in order to maintain a bitset of tuple slots that have been written to - I'm guessing this is for performance reasons, but it

[jira] Updated: (HADOOP-5571) TupleWritable can return incorrect results if it contains more than 32 values

2009-03-25 Thread Jingkei Ly (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingkei Ly updated HADOOP-5571: --- Attachment: HADOOP-5571-1.patch I think the problem is that some of the bit-shift operations in Tupl

  1   2   >