Re: Test failed when compiling Trunk

Street, Tenkasi, 627 811 http://www.declum.com/clement.html -- Todd Lipcon Software Engineer, Cloudera

Re: How to rebuild the shared edits directory

-file-system-hdfs/ http://www.slideshare.net/hortonworks/nn-ha-hadoop-worldfinal-10173419 Thanks, ~Jeff On 5/8/2012 6:49 PM, Todd Lipcon wrote: Hi Jeff, Check out HDFS-3077. We'll probably need the most help when it comes time to do testing. Any testing you can do on the current HA

Re: DFSClient not able to write

used is: 2.0.0 Cheers, Subroto Sanyal -- Todd Lipcon Software Engineer, Cloudera

Re: Namenode starup failure in 2.0.1 if fs.defaultFS not configured.

namenode. Here Namenode is not starting,it's getting shutdown by throwing illegalargumentexception.(Inavlid URI for NameNode address). But in trunk NN is starting successfully for same configuration... Please let me know any changes.. Thanks And Regards Brahma Reddy -- Todd Lipcon

Re: getFileBlockLocations on a newly created and closed file

);      }    } finally {      dfsCluster.shutdown();    }  } } -- Todd Lipcon Software Engineer, Cloudera

Re: Custom Socket Factory

to upgrade my custom socket factory rather than failing back to the old approach. Request some guidance in this direction…. Cheers, Subroto Sanyal -- Todd Lipcon Software Engineer, Cloudera

Re: Hadoop HA

martinus...@gmail.com wrote: Hi, Is there any hadoop HA distribution out there? Thanks. -- Todd Lipcon Software Engineer, Cloudera

Re: Hadoop HA

on our testing results. Furthermore, they are Apache 2 licensed open source, free of vendor lock-in. -Todd -- Todd Lipcon Software Engineer, Cloudera

Re: Unable to start NN after rack assignment attempt

what file it can't read. As always your patience with a noob appreciated; any suggestions to get started again? (I can forget about the rack assignment for now) Thanks. -- Todd Lipcon Software Engineer, Cloudera

Re: How to rebuild the shared edits directory

process again to re-initialize that edits dir. Thanks -Todd -- Todd Lipcon Software Engineer, Cloudera

Re: Does NameNode directory need fine-grained path lock?

level for such application sub-directory to reduce impact each other? It can be used with five(or less) top level to HDFS path with individual lock tree-like structure.      Can anybody provide your feedback or advice? Thanks -Regards Denny Ye -- Todd Lipcon Software Engineer, Cloudera

Re: How to rebuild the shared edits directory

, Todd Lipcon t...@cloudera.com wrote: On Tue, May 8, 2012 at 7:46 AM, Nathaniel Cook nathani...@qualtrics.com wrote: We have be working with an HA hdfs cluster, testing several failover scenarios.  We have a small cluster of 4 machines spun up for testing. We run a namenode on two of the machines

Re: How to rebuild the shared edits directory

complete HA. In others, like the AWS example you mentioned, we need a bit more, and we're working on it. Hang on a bit longer and it will be good to go. -Todd Sorry for the rant.  I just really want to see HDFS become complete HA system without caveats. ~Jeff On 5/8/2012 11:44 AM, Todd Lipcon

Re: How to rebuild the shared edits directory

for was willing spend some resources to help speed this process up, where should be start looking?  I'm sure there are quite a few jiras on these issues. Most of what we've done with the hadoop eco system has been zookeeper and hbase related. Thanks, ~Jeff On 5/8/2012 2:46 PM, Todd Lipcon wrote

Re: Question regarding FSDataOutputStream.close() behavior

utilize lazy replication, though that may just be for bringing underreplicated blocks up to snuff. Any insight would be appreciated. I'm not aware of any change in recent years... what versions are you comparing? -Todd -- Todd Lipcon Software Engineer, Cloudera

Re: Several Inconsistencies between fsck and NN UI reports

? Thanks and Regards Amith -- Todd Lipcon Software Engineer, Cloudera

Re: Question about «dfs.replication.considerLoad» property

, The property «dfs.replication.considerLoad» Decide if chooseTarget considers the target's load or not; but it works both in writing and in reading? or only during writing? Thanks Gianni -- Todd Lipcon Software Engineer, Cloudera

Re: Poll: namespace features

. Could you share your information about your hdfs clusters, if possible? Your information is extremely helpful. Thanks, Thanh -- Todd Lipcon Software Engineer, Cloudera

Re: Blacklisted Tasktracker / AlreadyBeingCreatedException

a default system and the hadoop config is default as well. Any ideas? Thanks in advance, Rafael. -- Todd Lipcon Software Engineer, Cloudera

Re: Notes about HA Name Node

LOS EEUU! http://www.antiterroristas.cu http://justiciaparaloscinco.wordpress.com -- Todd Lipcon Software Engineer, Cloudera

Re: How-to use DFSClient's BlockReader from Java

? No, it's also a private API. If you use the FileSystem API, and you're on the local node, all of these optimizations will happen automatically for you, plus they'll keep working when you upgrade. Even MapReduce just uses the public APIs. -Todd On 9.1.12 9:30 PM, Todd Lipcon t...@cloudera.com

Re: How-to use DFSClient's BlockReader from Java

. Is there an example Java code showing how to access individual blocks of particular file stored on HDFS ? Thanks in advance, David. -- Todd Lipcon Software Engineer, Cloudera

Re: How-to use DFSClient's BlockReader from Java

comments. Regards, David. On 9.1.12 6:59 PM, Todd Lipcon t...@cloudera.com wrote: Hi David, For what it's worth, you should be aware that you're calling internal APIs that have no guarantee of stability between versions. I can practically guarantee that your code will have to be modified

Re: File edit in Hadoop

feature set. -Todd -- Todd Lipcon Software Engineer, Cloudera

Re: Symbolic Links in HDFS

immediately. Before opening any mail and attachments please check them for viruses and defect. -- - -- Todd Lipcon Software Engineer, Cloudera

Re: Sizing help

-- --- Get your facts first, then you can distort them as you please.-- -- --- Get your facts first, then you can distort them as you please.-- -- Todd Lipcon Software Engineer, Cloudera

Re: Using HDFS to store few MB files for file sharing purposes

? Should i compltely drop that idea ? Would the NN accept 500 mounts through FUSE ? Thx brgds -- Todd Lipcon Software Engineer, Cloudera

Re: A question about RPC

work you can be getting done in other tasks at the same time. Although many workloads are IO bound, many are not. For example, if you have a lot of RAM available for HBase, it will become CPU-bound pretty quick as you are primarily hitting cached data. -Todd -- Todd Lipcon Software Engineer

Re: Question about hdfs close * hflush behavior

The actual append() function has some bugs in all of the 0.20 releases, including Cloudera's. The hflush/sync() API is fine to use, but I would recommend against using append(). -Todd -- Todd Lipcon Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera

Re: Question about hdfs close * hflush behavior

2011/9/8 Kanghua151 kanghua...@msn.com: you are so nice，thank you very much：） last question： can i trigger block sync without restart hdfs？ Close the file or have a machine crash :) But no, not really. 发自我的 iPhone 在 2011-9-8，15:00，Todd Lipcon t...@cloudera.com 写道： 2011/9/7 kang hua

Re: cant remove files from tmp

.. ?- ? ? ? ?? input-dir [root@hadoop20 mapred]# -- Marcos Luís Ortíz Valmaseda Software Engineer (UCI) http://marcosluis2186.posterous.com http://twitter.com/marcosluis2186 -- Todd Lipcon Software Engineer, Cloudera

Re: Local block placement policy, request

) to a specific DataNode?  If not, what would be a way to do this?  -Thanks -- Todd Lipcon Software Engineer, Cloudera

Re: HDFS Corruption: How to Troubleshoot or Determine Root Cause?

, too. -- Todd Lipcon Software Engineer, Cloudera

Re: fsimage save file

if it's breadth first or depth first, but the code should tell you) -Todd -- Todd Lipcon Software Engineer, Cloudera

Re: fsimage save file

supergroup 0 2011-05-02 18:47 /user/todd/zebra -Todd Thanh On Mon, May 2, 2011 at 3:51 PM, Todd Lipcon t...@cloudera.com wrote: On Mon, May 2, 2011 at 1:20 PM, Thanh Do than...@cs.wisc.edu wrote: hi all, perhaps this is a dummy question but can anyone tell me that when

Re: using API to determine when async file copy is complete

? That is, can I find out that the async replication has completed? And how many replicas are complete? I’d like to do this via the Java API. Thanks, Martin -- Todd Lipcon Software Engineer, Cloudera

Re: Is DFSInputStream.read(long position,...) designed for multi threaded access?

against an increasingly diverging upstream. Agreed completely. -Todd --- On Wed, 3/2/11, Todd Lipcon t...@cloudera.com wrote: From: Todd Lipcon t...@cloudera.com Subject: Re: Is DFSInputStream.read(long position,...) designed for multi threaded access? To: Jason Rutherglen jason.rutherg

Re: balancing and replication in HDFS

, Jeff -- Todd Lipcon Software Engineer, Cloudera

Re: balancing and replication in HDFS

on different nodes each time teragen is run. Jeff -Original Message- From: Todd Lipcon [mailto:t...@cloudera.com] Sent: Friday, February 25, 2011 3:13 PM To: hdfs-user@hadoop.apache.org Subject: Re: balancing and replication in HDFS Hi Jeff, The output of terasort has

Re: Failed namenode restart, recovering from corrupt edits file?

, Adam Phelps a...@opendns.com wrote: I was going back to see if a bug ticket had been opened against this problem, but am not seeing one. Before I go and open one can anyone let me know if I just failed to find it? - Adam On 1/12/11 1:13 PM, Todd Lipcon wrote: Hi guys, After Friso's issue

Re: Speeding up Data Deletion From Datanodes

is 0.6GB per 1M files, but not sure if that includes a safety factor. -Todd -- Todd Lipcon Software Engineer, Cloudera

Re: Speeding up Data Deletion From Datanodes

documentation for the same regarding tuning up of configurations based on the cluster usage. Thanks Regards, Sravan kumar. -- Todd Lipcon Software Engineer, Cloudera

Re: Failed namenode restart, recovering from corrupt edits file?

) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:313) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:276) at java.lang.Thread.run(Thread.java:662) - Adam -- Todd Lipcon Software Engineer, Cloudera

Re: response time increase??

scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- Todd Lipcon Software Engineer, Cloudera

Re: response time increase??

*completed_time: the final block of data been received response time = 1st_arrival_time = start_time Are you still confirmed that there is NO response_time = O(number of datanode) ? If so, HDFS will be great for our application. Thanks!! -- Kevin Kuei 於 2011/1/6 下午 04:23, Todd Lipcon 提到

Re: filesystem ls not working

, please notify us immediately by e-mail and delete the message and any attachments from your system. -- Todd Lipcon Software Engineer, Cloudera

Re: Trying to rollback for downgrade

some really strange issues with the fair scheduler. Thanks! -- Todd Lipcon Software Engineer, Cloudera

Re: Why datanode does a flush to disk after receiving a packet

onto dataQueue for the new pipeline. -Todd On Wed, Nov 10, 2010 at 11:11 PM, Todd Lipcon t...@cloudera.com wrote: Nope, flush just flushes the java side buffer to the Linux buffer cache -- not all the way to the media. Hsync is the API that will eventually go all the way to disk

Re: HDFS file disappeared

still have no idea how/why that file got purged. :-( Thanks, DR On 11/11/2010 02:13 PM, Todd Lipcon wrote: Given that it's an MR output, my guess is it got moved out of the temporary directory when the job Committed and then was removed as another pass. I'd grep for the containing

Re: Why datanode does a flush to disk after receiving a packet

? Can somebody clarify this for me? Thanks so much. Thanh -- Todd Lipcon Software Engineer, Cloudera

Re: Cloudera CDH3 Beta3 : HDFS sync(), the file length is not right.

something. the seek() and read() operations are both ok. Schubert -- Todd Lipcon Software Engineer, Cloudera

Re: DFSClient swallows InterruptedException

.) This is partially improved in HADOOP-6762 -Todd -- Owen -- Todd Lipcon Software Engineer, Cloudera

Re: How to make command setrepr take effect on data node ?

Zhang -- Eric Sammer phone: +1-917-287-2675 twitter: esammer data: www.cloudera.com -- Best Regards Jeff Zhang -- Eric Sammer phone: +1-917-287-2675 twitter: esammer data: www.cloudera.com -- Best Regards Jeff Zhang -- Todd Lipcon

Re: Is it necessary to cache metadata in client side?

in one node, so if tasktrack cache the metadata, it can reduce the communication with namenode. -- Best Regards Jeff Zhang -- Todd Lipcon Software Engineer, Cloudera

Re: Is it safe enough to write edit log in buffer before writing to disk ?

in the conf file to decide whether use buffer or not? Thanks. -- Best Regards Jeff Zhang -- Todd Lipcon Software Engineer, Cloudera

Re: Lots of Different Kind of Datanode Errors

ending? Any idea of what is going on? Thanks, ~Jeff -- Jeff Whiting Qualtrics Senior Software Engineer je...@qualtrics.com -- Todd Lipcon Software Engineer, Cloudera

Re: Lots of Different Kind of Datanode Errors

/value /property property namedfs.datanode.max.xcievers/name value4096/value /property property namedfs.datanode.handler.count/name value10/value /property /configuration . Todd Lipcon wrote: Hi Jeff, Have you followed the HDFS configuration guide from

Re: Unbalanced Datanode and Lots of Blocks Waiting for Deletion

idea what is going on? Why it isn't sending the delete commands? Or what I need to do or check to solve the problem? Thanks, ~Jeff -- Jeff Whiting Qualtrics Senior Software Engineer je...@qualtrics.com -- Todd Lipcon Software Engineer, Cloudera

Re: Is it possible to recover name node just using the meta data in a new machine

) at org.apache.hadoop.http.HttpServer.start(HttpServer.java:424) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:246) -- Best Regards Jeff Zhang -- Todd Lipcon Software Engineer, Cloudera

Re: Is it possible to recover name node just using the meta data in a new machine

you're trying to bind to a nonlocal IP. -Todd On Fri, May 21, 2010 at 12:39 PM, Todd Lipcon t...@cloudera.com wrote: The new machine apparently has something listening on one of the NN ports. try sudo fuser -n tcp 50070 (will tell you which pid is listening on that port) -Todd

Re: HDFS Read ThroughPut and DISK Read ThroughPut

this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. -- Todd Lipcon Software

Re: Hadoop 0.20.2 namenode inactivity issue

org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 738089 blocks got processed in 305396 msecs 2010-04-27 12:23:10,333 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 738367 blocks got processed in 306389 msecs On 04/27/2010 11:33 AM, Todd Lipcon wrote: Those

Re: response time increases lineraly for each thread!

name UI). Thanks, Gokul -- *From:* Todd Lipcon [mailto:t...@cloudera.com] *Sent:* Tuesday, April 13, 2010 10:54 AM *To:* gok...@huawei.com; hdfs-user@hadoop.apache.org *Cc:* sagar_shu...@persistent.co.in *Subject:* Re: response time increases

Re: append operations

was hoping to implement something myself (using a FileInputStream and FSDataOutputStream) but FSDataOutputStream is not seekable. Any suggestions? Regards, Alan -- thanh -- Todd Lipcon Software Engineer, Cloudera