For those using Hadoop in the social network domain
SOCIAL NETWORK SYSTEMS 2009 (SNS-2009) = Second ACM Workshop on Social Network Systems March 31, EuroSys 2009 Nuremberg, Germany http://www.eecs.harvard.edu/~stein/SocialNets-2009/ OVERVIEW The Second Workshop on Social Network Systems (SNS'08) will gather researchers to discuss novel ideas about computer systems and social networks. Online social networks are among the most popular sites on the Web and continue to grow rapidly. They provide mechanisms to establish identities, share information, and create relationships. The resulting social graph provides a basis for communicating and distributing and locating content. Broadly, the systems issues of social networks include: * How can systems infrastructure be improved for social networks? Infrastructure includes database systems, operating systems, file systems, and storage systems. * How can the social graph be leveraged in computer system design? The social graph encodes trust and common interests. How and to what extent can this encoding be used to improve computer systems? * How can social networks be modeled and characterized? What has been learned from the operation of existing systems? Topics of interest include, but are not limited to: * Security and privacy. * Leveraging the social graph in systems design. * Real-time monitoring and query processing. * Database issues for offline analysis. * Experiences with deployed systems. * Crawlers and other mechanisms for observing social network structure. * Measurement and analysis, including comparative analysis. * Tools for designing and deploying social networks. * Network dynamics, relationships between network links and user behavior. * Benchmarks, modeling, and characterization. * Decentralization: methods for integrating multiple networks. * Application programming interfaces (APIs) for social networks. The papers presented, as well as a summary of the discussion, will be archived electronically. Accepted papers may be subsequently revised, expanded, and submitted to full conferences and journals. ORGANIZERS: Chair: Lex Stein, Facebook Program Committee: Samuel Bernard, LIP6 Meeyoung Cha, MPI-SWS Wei Chen, Microsoft Research Asia Yafei Dai, Peking University Adrienne Felt, UC Berkeley Eran Gabber, Google Bingsheng He, Microsoft Research Asia Anne-Marie Kermarrec, INRIA Peter Key, Microsoft Research Cambridge Chris Lesniewski-Laas, MIT Shiding Lin, Baidu Alan Mislove, MPI-SWS and Rice University Yoann Padioleau, UIUC Peter Pietzuch, Imperial College London Stefan Saroiu, Microsoft Research Redmond Rodrigo Schmidt, Facebook Jacky Shen, Microsoft Research Asia Steven Smaldone, Rutgers Lex Stein, Facebook Jacob Strauss, MIT Nguyen Tran, NYU Edward Wang, Google David Wei, Facebook Geoffrey Werner-Allen, Harvard Eiko Yoneki, University of Cambridge IMPORTANT DATES Paper submissions due:February 2, 2009 Notification to authors: February 16, 2009 Workshop: March 31, 2009 SUBMITTING A PAPER Papers must be received by 23:59 GMT, on January 26, 2009. This is a hard deadline. Submissions should contain six or fewer two-column pages, including all figures and references, using 10-point fonts, standard spacing, and 1-inch margins (we recommend the ACM sig-alternate template, LaTeX template available at http://www.eecs.harvard.edu/~stein/sig-alternate-10pt.cls. Please number pages. All submissions will be electronic, and must be in either PDF format (preferred) or PostScript. Author names and affiliations should appear on the title page. Reviewing will be single-blind. This workshop is sponsored by ACM, ACM SigOps, and EuroSys.
Re: Block placement in HDFS
Fyi - Owen is referring to: https://issues.apache.org/jira/browse/HADOOP-2559 On 11/24/08 10:22 PM, Owen O'Malley [EMAIL PROTECTED] wrote: On Nov 24, 2008, at 8:44 PM, Mahadev Konar wrote: Hi Dennis, I don't think that is possible to do. No, it is not possible. The block placement is determined by HDFS internally (which is local, rack local and off rack). Actually, it was changed in 0.17 or so to be node-local, off-rack, and a second node off rack. -- Owen
Re: hdfs fuse mount and namenode out of memory conditions.
Hi Jason, There's nothing that fuse does that should cause this I don't think. If this is a cat operation, fuse will do this in chunks of 32 K which you can change by mounting with -ordbuffer_size=#bytes. Do you have the stack trace or the NN log while this app is running? Were you doing the equivalent of this app with dfs -cat or something before with no problems? 1.5 M files sounds like it might be close to the limit of a 1.5GB NN? -- pete On 11/17/08 9:29 AM, Jason Venner [EMAIL PROTECTED] wrote: We recently setup a fuse mount using the 18.2 fuse code, against our 18.1 hdfs, which has been running stably for some time. We have about 20 datanodes and 50TB or so in our hdfs. The namenode is running an i686 kernel and has been running with -Xmx1500m. We have 1,492,093 files in our hdfs at present. Since we put up the fuse mount which reads 2 large files at a time, about 6 a second or so, we have had the namenode OOM multiple times. We have upped the memory foot print to the max allowed for a 32bit jvm. What memory size should we be running? Is the fuse mount doing something that is increasing the memory use of the namenode? Happy week all.
Re: FUSE writes in 0.18.1
You know what - writes do not not work at all in 18.1 - sorry my confusion. - just switch to 18.2 fuse-dfs. It can run against 18.1 dfs. Just remember to compile it with -Dlibhdfs.noperms=1 On 11/7/08 12:20 PM, Brian Karlak [EMAIL PROTECTED] wrote: On Nov 7, 2008, at 12:09 PM, Pete Wyckoff wrote: For 0.18.1, cp foo bar where bar does not exist will work. But, if bar exists, it will not.copying over an existing file requires truncate to work and fuse-dfs 0.18.1 does not support it. Pete -- Thanks for the help. I am still having trouble with writes, even to file paths that don't exist on HDFS. For instance: % ls /mnt/hadoop-fuse/user/hadoop jython-wc-3.sh/ test/ % cp ~/hadoop-default.xml /mnt/hadoop-fuse/user/hadoop/test20081108 cp: cannot create regular file `/mnt/hadoop-fuse/user/hadoop/ test20081108': Function not implemented test20081108 does not exist, yet I cannot create it. Nor can I do touch or other writes. Since writes should work in 0.18.1, do you have any suggestions as to where I should start with debugging? Are there any particular options I should make sure I'm using? Thanks, Brian
Re: libhdfs SIGSEGV error
Hi Tomas, I cannot reproduce the problem myself - even writing 100 GB file is fine. I am running on amd64 too with 64 bit jvm. I did have one other question and just wanted to ensure you are compiling libhdfs with -m64 ? I.e., did you edit its Makefile and replace -m32 with -m64? Pete On 11/7/08 5:44 AM, Tamás Szokol [EMAIL PROTECTED] wrote: Hello Pete! Sorry, I forgot to answer to your gdb related question. I ran the tests under the debugger. In case of 100MB buffer it received a SIGSEGV signal at the jni_SetByteArrayRegion function. Any clue what this function does or why it fails? Thank you. Cheers, Tamas --- On Thu, 11/6/08, Pete Wyckoff [EMAIL PROTECTED] wrote: From: Pete Wyckoff [EMAIL PROTECTED] Subject: Re: libhdfs SIGSEGV error To: , [EMAIL PROTECTED]@yahoo.com, core-user@hadoop.apache.org core-user@hadoop.apache.org Date: Thursday, November 6, 2008, 7:20 PM Hi Tamas, Have you tried using the supplied hdfs_write executable includes in the distribution? Also, I didn't understand your comment about using hdfsJniHelper.c - that should be used only by hdfs.c itself. Also, what version of hadoop is this? I haven't seen this problem at least in hadoop 0.17. And have you run this under gdb? -- pete On 11/6/08 10:30 AM, Tamás Szokol [EMAIL PROTECTED] wrote: Hello! I'd like to ask your help in a libhdfs related problem. I'm trying to perform HDFS tests from C by using the libhdfs API. I created a test program, that measures the creation times of 1MB, 10MB, 100MB and 1GB large files. The test runs well for 1 MB and 10 MB but as soon as I reach to 100MB I receive a SIGSEGV error: == # # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7fbead12a32c, pid=6918, tid=140456938362592 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode linux-amd64) # Problematic frame: # V [libjvm.so+0x36d32c] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x00609800): JavaThread main [_thread_in_vm, id=6918, stack(0x7fffb5cc2000,0x7fffb5ec2000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x Registers: RAX=0x0090, RBX=0x00609800, RCX=0x7fbead60d1e0, RDX=0x7fbead447780 RSP=0x7fffb5ec1240, RBP=0x7fffb5ec12c0, RSI=0x, RDI=0x00609800 R8 =0x7fbe7dcff010, R9 =0x0400, R10=0x7fbead610c50, R11=0x7fbea35180c0 R12=0x0630, R13=0x, R14=0x, R15=0x7fffb5ec1250 RIP=0x7fbead12a32c, EFL=0x00010246, CSGSFS=0x0033, ERR=0x0004 TRAPNO=0x000e Top of Stack: (sp=0x7fffb5ec1240) 0x7fffb5ec1240: 00609998 7fbe7dcff010 0x7fffb5ec1250: 00609800 0060a1f0 0x7fffb5ec1260: 0090 7fbead447780 0x7fffb5ec1270: 00609800 00602720 0x7fffb5ec1280: 00609800 00609998 0x7fffb5ec1290: 00609800 7fbe84014220 0x7fffb5ec12a0: 00609998 7fbe7dcff010 0x7fffb5ec12b0: 006f7848 0x7fffb5ec12c0: 0630 7fbeada9ccff 0x7fffb5ec12d0: 006f7830 0x7fffb5ec12e0: 0060a660 0x7fffb5ec12f0: 0021 0x7fffb5ec1300: 7fffb5ec1370 0x7fffb5ec1310: 7fffb5ec14b0 0x7fffb5ec1320: 00400e37 0x7fffb5ec1330: 7fbe7dcff010 0x7fffb5ec1340: 0631 00602720 0x7fffb5ec1350: 063b 7fbe84014220 0x7fffb5ec1360: 006f7840 0021 0x7fffb5ec1370: 7fffb5ec1390 00400cdd 0x7fffb5ec1380: 7fbe7dcff010 001800401126 0x7fffb5ec1390: 7fffb5ec13d0 00400b1b 0x7fffb5ec13a0: 003c 7fffb5ec14b8 0x7fffb5ec13b0: 000100401040 00602280 0x7fffb5ec13c0: 0022 7fbeadec0c00 0x7fffb5ec13d0: 7fbead7561c4 0x7fffb5ec13e0: 004009c0 7fffb5ec14b8 0x7fffb5ec13f0: 0001 00400a78 0x7fffb5ec1400: 7fbeadec0c00 0c1103e05d01fe14 0x7fffb5ec1410: 7fffb5ec14b0 0x7fffb5ec1420: 0x7fffb5ec1430: f3ee68387ac1fe14 f36c590a9e14 Instructions: (pc=0x7fbead12a32c) 0x7fbead12a31c: 5d 90 48 83 7b 08 00 0f 85 5d 01 00 00 45 85 ed 0x7fbead12a32c: 49 8b 36 78 13 45 85 e4 78 0e 47 8d 1c 2c 44 3b Stack: [0x7fffb5cc2000,0x7fffb5ec2000], sp=0x7fffb5ec1240, free space=2044k Native frames: (J
Can FSDataInputStream.read return 0 bytes and if so, what does that mean?
The javadocs says reads up to size bytes. What happens if it returns 0 (presumably an error) or 0 bytes (??) Thanks, pete
Re: Can FSDataInputStream.read return 0 bytes and if so, what does that mean?
Just want to ensure 0 iff EOF or the requested #of bytes was 0. On 11/7/08 6:13 PM, Pete Wyckoff [EMAIL PROTECTED] wrote: The javadocs says reads up to size bytes. What happens if it returns 0 (presumably an error) or 0 bytes (??) Thanks, pete
Re: Status FUSE-Support of HDFS
Have not used it with rsync, but do remember, fuse-dfs will return an EIO for non-sequential writes. No unit test for this yet, but there probably should be :) On 11/4/08 9:07 AM, Robert Krüger [EMAIL PROTECTED] wrote: Thanks! This is good news. So it's fast enough for our purposes if it turns out to be the same order of magnitude on our systems. Have you used this with rsync? If so, any known issues with that (reading or writing)? Thanks in advance, Robert Pete Wyckoff wrote: Reads are 20-30% slower Writes are 33% slower before https://issues.apache.org/jira/browse/HADOOP-3805 - You need a kernel 2.6.26-rc* to test 3805, which I don't have :( These #s are with hadoop 0.17 and the 0.18.2 version of fuse-dfs. -- pete On 11/2/08 6:23 AM, Robert Krüger [EMAIL PROTECTED] wrote: Hi Pete, thanks for the info. That helps a lot. We will probably test it for our use cases then. Did you benchmark throughput when reading writing files through fuse-dfs and compared it to command line tool or API access? Is there a notable difference? Thanks again, Robert Pete Wyckoff wrote: It has come a long way since 0.18 and facebook keeps our (0.17) dfs mounted via fuse and uses that for some operations. There have recently been some problems with fuse-dfs when used in a multithreaded environment, but those have been fixed in 0.18.2 and 0.19. (do not use 0.18 or 0.18.1) The current (known) issues are: 1. Wrong semantics when copying over an existing file - namely it does a delete and then re-creates the file, so ownership/permissions may end up wrong. There is a patch for this. 2. When directories have 10s of thousands of files, performance can be very poor. 3. Posix truncate is supported only for truncating it to 0 size since hdfs doesn't support truncate. 4. Appends are not supported - this is a libhdfs problem and there is a patch for it. It is still a pre-1.0 product for sure, but it has been pretty stable for us. -- pete On 10/31/08 9:08 AM, Robert Krüger [EMAIL PROTECTED] wrote: Hi, could anyone tell me what the current Status of FUSE support for HDFS is? Is this something that can be expected to be usable in a few weeks/months in a production environment? We have been really happy/successful with HDFS in our production system. However, some software we use in our application simply requires an OS-Level file system which currently requires us to do a lot of copying between HDFS and a regular file system for processes which require that software and FUSE support would really eliminate that one disadvantage we have with HDFS. We wouldn't even require the performance of that to be outstanding because just by eliminatimng the copy step, we would greatly increase the thruput of those processes. Thanks for sharing any thoughts on this. Regards, Robert
Re: libhdfs SIGSEGV error
Hi Tamas, Have you tried using the supplied hdfs_write executable includes in the distribution? Also, I didn't understand your comment about using hdfsJniHelper.c - that should be used only by hdfs.c itself. Also, what version of hadoop is this? I haven't seen this problem at least in hadoop 0.17. And have you run this under gdb? -- pete On 11/6/08 10:30 AM, Tamás Szokol [EMAIL PROTECTED] wrote: Hello! I'd like to ask your help in a libhdfs related problem. I'm trying to perform HDFS tests from C by using the libhdfs API. I created a test program, that measures the creation times of 1MB, 10MB, 100MB and 1GB large files. The test runs well for 1 MB and 10 MB but as soon as I reach to 100MB I receive a SIGSEGV error: == # # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7fbead12a32c, pid=6918, tid=140456938362592 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode linux-amd64) # Problematic frame: # V [libjvm.so+0x36d32c] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x00609800): JavaThread main [_thread_in_vm, id=6918, stack(0x7fffb5cc2000,0x7fffb5ec2000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x Registers: RAX=0x0090, RBX=0x00609800, RCX=0x7fbead60d1e0, RDX=0x7fbead447780 RSP=0x7fffb5ec1240, RBP=0x7fffb5ec12c0, RSI=0x, RDI=0x00609800 R8 =0x7fbe7dcff010, R9 =0x0400, R10=0x7fbead610c50, R11=0x7fbea35180c0 R12=0x0630, R13=0x, R14=0x, R15=0x7fffb5ec1250 RIP=0x7fbead12a32c, EFL=0x00010246, CSGSFS=0x0033, ERR=0x0004 TRAPNO=0x000e Top of Stack: (sp=0x7fffb5ec1240) 0x7fffb5ec1240: 00609998 7fbe7dcff010 0x7fffb5ec1250: 00609800 0060a1f0 0x7fffb5ec1260: 0090 7fbead447780 0x7fffb5ec1270: 00609800 00602720 0x7fffb5ec1280: 00609800 00609998 0x7fffb5ec1290: 00609800 7fbe84014220 0x7fffb5ec12a0: 00609998 7fbe7dcff010 0x7fffb5ec12b0: 006f7848 0x7fffb5ec12c0: 0630 7fbeada9ccff 0x7fffb5ec12d0: 006f7830 0x7fffb5ec12e0: 0060a660 0x7fffb5ec12f0: 0021 0x7fffb5ec1300: 7fffb5ec1370 0x7fffb5ec1310: 7fffb5ec14b0 0x7fffb5ec1320: 00400e37 0x7fffb5ec1330: 7fbe7dcff010 0x7fffb5ec1340: 0631 00602720 0x7fffb5ec1350: 063b 7fbe84014220 0x7fffb5ec1360: 006f7840 0021 0x7fffb5ec1370: 7fffb5ec1390 00400cdd 0x7fffb5ec1380: 7fbe7dcff010 001800401126 0x7fffb5ec1390: 7fffb5ec13d0 00400b1b 0x7fffb5ec13a0: 003c 7fffb5ec14b8 0x7fffb5ec13b0: 000100401040 00602280 0x7fffb5ec13c0: 0022 7fbeadec0c00 0x7fffb5ec13d0: 7fbead7561c4 0x7fffb5ec13e0: 004009c0 7fffb5ec14b8 0x7fffb5ec13f0: 0001 00400a78 0x7fffb5ec1400: 7fbeadec0c00 0c1103e05d01fe14 0x7fffb5ec1410: 7fffb5ec14b0 0x7fffb5ec1420: 0x7fffb5ec1430: f3ee68387ac1fe14 f36c590a9e14 Instructions: (pc=0x7fbead12a32c) 0x7fbead12a31c: 5d 90 48 83 7b 08 00 0f 85 5d 01 00 00 45 85 ed 0x7fbead12a32c: 49 8b 36 78 13 45 85 e4 78 0e 47 8d 1c 2c 44 3b Stack: [0x7fffb5cc2000,0x7fffb5ec2000], sp=0x7fffb5ec1240, free space=2044k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x36d32c] --- P R O C E S S --- Java Threads: ( = current thread ) 0x0069f800 JavaThread Low Memory Detector daemon [_thread_blocked, id=6925, stack(0x7fbe8b1b5000,0x7fbe9b1b6000)] 0x0069dc00 JavaThread CompilerThread1 daemon [_thread_blocked, id=6924, stack(0x41c3e000,0x41d3f000)] 0x00692400 JavaThread CompilerThread0 daemon [_thread_blocked, id=6923, stack(0x41b3d000,0x41c3e000)] 0x00690c00 JavaThread Signal Dispatcher daemon [_thread_blocked, id=6922, stack(0x61e4c000,0x71e4d000)] 0x0066cc00 JavaThread Finalizer daemon [_thread_blocked, id=6921, stack(0x51e4b000,0x61e4c000)] 0x0066b800 JavaThread Reference Handler daemon
Re: Status FUSE-Support of HDFS
Reads are 20-30% slower Writes are 33% slower before https://issues.apache.org/jira/browse/HADOOP-3805 - You need a kernel 2.6.26-rc* to test 3805, which I don't have :( These #s are with hadoop 0.17 and the 0.18.2 version of fuse-dfs. -- pete On 11/2/08 6:23 AM, Robert Krüger [EMAIL PROTECTED] wrote: Hi Pete, thanks for the info. That helps a lot. We will probably test it for our use cases then. Did you benchmark throughput when reading writing files through fuse-dfs and compared it to command line tool or API access? Is there a notable difference? Thanks again, Robert Pete Wyckoff wrote: It has come a long way since 0.18 and facebook keeps our (0.17) dfs mounted via fuse and uses that for some operations. There have recently been some problems with fuse-dfs when used in a multithreaded environment, but those have been fixed in 0.18.2 and 0.19. (do not use 0.18 or 0.18.1) The current (known) issues are: 1. Wrong semantics when copying over an existing file - namely it does a delete and then re-creates the file, so ownership/permissions may end up wrong. There is a patch for this. 2. When directories have 10s of thousands of files, performance can be very poor. 3. Posix truncate is supported only for truncating it to 0 size since hdfs doesn't support truncate. 4. Appends are not supported - this is a libhdfs problem and there is a patch for it. It is still a pre-1.0 product for sure, but it has been pretty stable for us. -- pete On 10/31/08 9:08 AM, Robert Krüger [EMAIL PROTECTED] wrote: Hi, could anyone tell me what the current Status of FUSE support for HDFS is? Is this something that can be expected to be usable in a few weeks/months in a production environment? We have been really happy/successful with HDFS in our production system. However, some software we use in our application simply requires an OS-Level file system which currently requires us to do a lot of copying between HDFS and a regular file system for processes which require that software and FUSE support would really eliminate that one disadvantage we have with HDFS. We wouldn't even require the performance of that to be outstanding because just by eliminatimng the copy step, we would greatly increase the thruput of those processes. Thanks for sharing any thoughts on this. Regards, Robert
Re: Status FUSE-Support of HDFS
+1 but since hadoop deals well with such directories currently, fuse-dfs will basically lock up on such directories - this is because ls -color=blah causes a stat on every file in a directory. There is a JIRA open for this but it is a pretty rare case although it has happened to me at facebook. -- pete It's good for a portable application to keep the #of files/directory low by having two levels of directory for storing files -just use a hash operation to determine which dir to store a specific file in. On 11/3/08 4:00 AM, Steve Loughran [EMAIL PROTECTED] wrote: Pete Wyckoff wrote: It has come a long way since 0.18 and facebook keeps our (0.17) dfs mounted via fuse and uses that for some operations. There have recently been some problems with fuse-dfs when used in a multithreaded environment, but those have been fixed in 0.18.2 and 0.19. (do not use 0.18 or 0.18.1) The current (known) issues are: 2. When directories have 10s of thousands of files, performance can be very poor. I've known other filesystems to top out at 64k-1 files per directory, even if they don't slow down. It's good for a portable application to keep the #of files/directory low by having two levels of directory for storing files -just use a hash operation to determine which dir to store a specific file in.
Re: Can anyone recommend me a inter-language data file format?
Protocol buffers, thrift? On 11/3/08 4:07 AM, Steve Loughran [EMAIL PROTECTED] wrote: Zhou, Yunqing wrote: embedded database cannot handle large-scale data, not very efficient I have about 1 billion records. these records should be passed through some modules. I mean a data exchange format similar to XML but more flexible and efficient. JSON CSV erlang-style records (name,value,value,value) RDF-triples in non-XML representations For all of these, you need to test with data that includes things like high unicode characters, single and double quotes, to see how well they get handled. you can actually append with XML by not having opening/closing tags, just stream out the entries to the tail of the file entry.../entry To read this in an XML parser, include it inside another XML file: ?xml version=1.0? !DOCTYPE log [ !ENTITY log SYSTEM log.xml ] file log; /file I've done this for very big files, as long as you aren't trying to load it in-memory to a DOM, things should work -- Steve Loughran http://www.1060.org/blogxter/publish/5 Author: Ant in Action http://antbook.org/
Re: LHadoop Server simple Hadoop input and output
Chukwa also could be used here. On 10/24/08 11:47 AM, Jeff Hammerbacher [EMAIL PROTECTED] wrote: Hey Edward, The application we used at Facebook to transmit new data is open source now and available at http://sourceforge.net/projects/scribeserver/. Later, Jeff On Fri, Oct 24, 2008 at 10:14 AM, Edward Capriolo [EMAIL PROTECTED] wrote: I came up with my line of thinking after reading this article: http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data As a guy that was intrigued by the java coffee cup in 95, that now lives as a data center/noc jock/unix guy. Lets say I look at a log management process from a data center prospective. I know: Syslog is a familiar model (human readable: UDP text) INETD/XINETD is a familiar model (programs that do amazing things with STD IN/STD OUT) Variety of hardware and software I may be supporting an older Solaris 8, windows or Free BSD 5 for example. I want to be able to pipe apache custom log at HDFS, or forward syslog. That is where LHadoop (or something like it) would come into play. I am thinking to even accept raw streams and have the server side use source-host/regex to determine what file the data should go to. I want to stay light on the client side. An application that tails log files and transmits new data is another component to develop and manage. Had anyone had experience with moving this type of data?
aerialization.Deserializer.deserialize method help
This method's signature is
{code}
T deserialize(T);
{code}
But, the RecordReader next method is
{code}
boolean next(K,V);
{code}
So, if the deserialize method does not return the same T (i.e., K or V), how
would this new Object be propagated back thru the RecordReader next method.
It seems the contract on the deserialize method is that it must return the
same T (although the javadocs say may).
Am I missing something? And if not, why isn't the API boolean deserialize(T)
?
Thanks, pete
Ps for things like Thrift, there's no way to re-use the object as there's no
clear method, so if this is the case, I don't see how it would work??
Re: aerialization.Deserializer.deserialize method help
Specifically, line 75 of SequenceFileRecordReader:
boolean remaining = (in.next(key) != null);
Throws out the return value of SequenceFile.next which is the result of
deserialize(obj).
-- pete
On 9/12/08 2:28 PM, Pete Wyckoff [EMAIL PROTECTED] wrote:
What I mean is let's say I plug in a deserializer that always returns a new
Object - in that case, since everything is pass by value, the new object
cannot make its way back to the SequenceFileRecordReader user.
While(sequenceFileRecordReader.next(mykey, myvalue)) {
// do something
}
And then my deserializers one/both looks like:
T deserialize(T obj) {
// ignore obj
return new T(params);
}
Obj would be the key or the value passed in by the user, but since I ignore
it, basically what happens is the deserialized value actually gets thrown
away.
More specifically, it gets thrown away in SequenceFile.Reader I believe.
-- pete
On 9/12/08 2:20 PM, Chris Douglas [EMAIL PROTECTED] wrote:
If you pass in null to the deserializer, it creates a new instance and
returns it; passing in an instance reuses it.
I don't understand the disconnect between Deserializer and the
RecordReader. Does your RecordReader generate instances that only
share a common subtype T? You need separate Deserializers for K and V,
if that's the issue... -C
On Sep 12, 2008, at 2:01 PM, Pete Wyckoff wrote:
This method's signature is
{code}
T deserialize(T);
{code}
But, the RecordReader next method is
{code}
boolean next(K,V);
{code}
So, if the deserialize method does not return the same T (i.e., K or
V), how
would this new Object be propagated back thru the RecordReader next
method.
It seems the contract on the deserialize method is that it must
return the
same T (although the javadocs say may).
Am I missing something? And if not, why isn't the API boolean
deserialize(T)
?
Thanks, pete
Ps for things like Thrift, there's no way to re-use the object as
there's no
clear method, so if this is the case, I don't see how it would work??
Re: aerialization.Deserializer.deserialize method help
Sorry - saw the response after I sent this. But the current javadocs are
wrong and should probably say must return what was passed in.
On 9/12/08 3:02 PM, Pete Wyckoff [EMAIL PROTECTED] wrote:
Specifically, line 75 of SequenceFileRecordReader:
boolean remaining = (in.next(key) != null);
Throws out the return value of SequenceFile.next which is the result of
deserialize(obj).
-- pete
On 9/12/08 2:28 PM, Pete Wyckoff [EMAIL PROTECTED] wrote:
What I mean is let's say I plug in a deserializer that always returns a new
Object - in that case, since everything is pass by value, the new object
cannot make its way back to the SequenceFileRecordReader user.
While(sequenceFileRecordReader.next(mykey, myvalue)) {
// do something
}
And then my deserializers one/both looks like:
T deserialize(T obj) {
// ignore obj
return new T(params);
}
Obj would be the key or the value passed in by the user, but since I ignore
it, basically what happens is the deserialized value actually gets thrown
away.
More specifically, it gets thrown away in SequenceFile.Reader I believe.
-- pete
On 9/12/08 2:20 PM, Chris Douglas [EMAIL PROTECTED] wrote:
If you pass in null to the deserializer, it creates a new instance and
returns it; passing in an instance reuses it.
I don't understand the disconnect between Deserializer and the
RecordReader. Does your RecordReader generate instances that only
share a common subtype T? You need separate Deserializers for K and V,
if that's the issue... -C
On Sep 12, 2008, at 2:01 PM, Pete Wyckoff wrote:
This method's signature is
{code}
T deserialize(T);
{code}
But, the RecordReader next method is
{code}
boolean next(K,V);
{code}
So, if the deserialize method does not return the same T (i.e., K or
V), how
would this new Object be propagated back thru the RecordReader next
method.
It seems the contract on the deserialize method is that it must
return the
same T (although the javadocs say may).
Am I missing something? And if not, why isn't the API boolean
deserialize(T)
?
Thanks, pete
Ps for things like Thrift, there's no way to re-use the object as
there's no
clear method, so if this is the case, I don't see how it would work??
Parameterized deserializers?
If I have a generic Serializer/Deserializers that take some runtime
information to instantiate, how would this work in the current
serializer/deserializer APIs? And depending on this runtime information, may
return different Objects although they may all derive from the same class.
For example, for Thrift, I may have something called a ThriftSerializer that
is general:
{code}
Public class ThriftDeserializerT extends ThriftBase implements
Deserializer {
T deserialize(T);
}
{code}
How would I instantiate this, since the current getDeserializer takes only
the Class but not configuration object.
How would I implement createKey in RecordReader
In other words, I think we need a {code}Class? getClass(); {code} method
in Deserializer() and a {code}Deserializer getDeserializer(Class,
Configuration conf); {code} method in Serializer.java.
Or is there another way to do this?
IF not, I can open a JIRA for implementing parameterized serializers.
Thanks, pete
Re: Parameterized deserializers?
I should mention this is out of the context of SequenceFiles where we get
the class names in the file itself. Here there is some information inserted
into the JobConf that tells me the class of the records in the input file.
-- pete
On 9/12/08 3:26 PM, Pete Wyckoff [EMAIL PROTECTED] wrote:
If I have a generic Serializer/Deserializers that take some runtime
information to instantiate, how would this work in the current
serializer/deserializer APIs? And depending on this runtime information, may
return different Objects although they may all derive from the same class.
For example, for Thrift, I may have something called a ThriftSerializer that
is general:
{code}
Public class ThriftDeserializerT extends ThriftBase implements
Deserializer {
T deserialize(T);
}
{code}
How would I instantiate this, since the current getDeserializer takes only
the Class but not configuration object.
How would I implement createKey in RecordReader
In other words, I think we need a {code}Class? getClass(); {code} method
in Deserializer() and a {code}Deserializer getDeserializer(Class,
Configuration conf); {code} method in Serializer.java.
Or is there another way to do this?
IF not, I can open a JIRA for implementing parameterized serializers.
Thanks, pete
Re: Thinking about retriving DFS metadata from datanodes!!!
You may want to look at hadoop's proposal for snapshotting, where one can take a snapshot's metadata and store it in some disaster resilient place(s) for a rainy day: https://issues.apache.org/jira/browse/HADOOP-3637 On 9/11/08 10:06 AM, Dhruba Borthakur [EMAIL PROTECTED] wrote: My opinion is to not store file-namespace related metadata on the datanodes. When a file is renamed, one has to contact all datanodes to change this new metadata. Worse still, if one renames an entire subdirectory, all blocks that belongs to all files in the subdirectory have to be updated. Similarly, if in future, a file has multiple patches to it (links), a block may belong to two filenames. In the future, if HDFS wants to implement any kind of de-duplication (i.e. if the same block data appears in multiple files, the file system can intelligently keep only one copy of the block).. it will be difficult to do. thanks, dhruba On Wed, Sep 10, 2008 at 7:40 PM, 叶双明 [EMAIL PROTECTED] wrote: Thanks Ari Rabkin! 1. I think the cost is very low, if the block's size is 10m, 1k/10m almost 0.01% of the disk space. 2. Actually, if two of racks lose and replication = 3, it seem that we can't recover all data. But in the situation of losing one rack of two racks and replication =2, we can recover all data. 3. Suppose we recover 87.5% of data. I am not sure whether or not the random 87.5% of the data is usefull for every user. But in the situation of the size of most file is less than block'size, we can recover so much data,.Any recovered data may be valuable for some user. 4. I guess most small companies or organizations just have a cluster with 10-100 nodes, and they can not afford a second HDFS cluster in a different place or SAN. And it is a simple way to I think they would be pleased to ensure data safety for they. 5. We can config to turn on when someone need it, or turn it off otherwise. Glad to discuss with you！ 2008/9/11 Ariel Rabkin [EMAIL PROTECTED] I don't understand this use case. Suppose that you lose half the nodes in the cluster. On average, 12.5% of your blocks were exclusively stored on the half the cluster that's dead. For many (most?) applications, a random 87.5% of the data isn't really useful. Storing metadata in more places would let you turn a dead cluster into a corrupt cluster, but not into a working one. If you need to survive major disasters, you want a second HDFS cluster in a different place. The thing that might be useful to you, if you're worried about simultaneous namenode and secondary NN failure, is to store the edit log and fsimage on a SAN, and get fault tolerance that way. --Ari On Tue, Sep 9, 2008 at 6:38 PM, 叶双明 [EMAIL PROTECTED] wrote: Thanks for paying attention to my tentative idea! What I thought isn't how to store the meradata, but the final (or last) way to recover valuable data in the cluster when something worst (which destroy the metadata in all multiple NameNode) happen. i.e. terrorist attack or natural disasters destroy half of cluster nodes within all NameNode, we can recover as much data as possible by this mechanism, and hava big chance to recover entire data of cluster because fo original replication. Any suggestion is appreciate! 2008/9/10 Pete Wyckoff [EMAIL PROTECTED] +1 - from the perspective of the data nodes, dfs is just a block-level store and is thus much more robust and scalable. On 9/9/08 9:14 AM, Owen O'Malley [EMAIL PROTECTED] wrote: This isn't a very stable direction. You really don't want multiple distinct methods for storing the metadata, because discrepancies are very bad. High Availability (HA) is a very important medium term goal for HDFS, but it will likely be done using multiple NameNodes and ZooKeeper. -- Owen -- Ari Rabkin [EMAIL PROTECTED] UC Berkeley Computer Science Department -- Sorry for my english!! 明 Please help me to correct my english expression and error in syntax
Re: Thinking about retriving DFS metadata from datanodes!!!
+1 - from the perspective of the data nodes, dfs is just a block-level store and is thus much more robust and scalable. On 9/9/08 9:14 AM, Owen O'Malley [EMAIL PROTECTED] wrote: This isn't a very stable direction. You really don't want multiple distinct methods for storing the metadata, because discrepancies are very bad. High Availability (HA) is a very important medium term goal for HDFS, but it will likely be done using multiple NameNodes and ZooKeeper. -- Owen
Re: fuse-dfs
Hi Sebastian. Setting of times doesn¹t work, but ls, rm, rmdir, mkdir, cp, etc etc should work. Things that are not currently supported include: Touch, chown, chmod, permissions in general and obviously random writes for which you would get an IO error. This is what I get on 0.17 for df h: FilesystemSize Used Avail Use% Mounted on fuse XXXT YYYT ZZZT AA% /export/hdfs and the #s are right There is no unit test for df though (doh!), so it¹s quite possible the libhdfs API has changed and fuse-dfs needs to update its code to match the API. I will check that. To be honest, we run on 0.17.1, so other than unit tests, I never run on 0.19 :( -- pete Ps I created: https://issues.apache.org/jira/browse/HADOOP-3928 to track this. On 8/8/08 3:34 AM, Sebastian Vieira [EMAIL PROTECTED] wrote: Hi Pete, From within the 0.19 source i did: ant jar ant metrics.jar ant test-core This resulted in 3 jar files within $HADOOP_HOME/build : [EMAIL PROTECTED] hadoop-0.19]# ls -l build/*.jar -rw-r--r-- 1 root root 2201651 Aug 8 08:26 build/hadoop-0.19.0-dev-core.jar -rw-r--r-- 1 root root 1096699 Aug 8 08:29 build/hadoop-0.19.0-dev-test.jar -rw-r--r-- 1 root root 55695 Aug 8 08:26 build/hadoop-metrics-0.19.0-dev.jar I've added these to be included in the CLASSPATH within the wrapper script: for f in `ls $HADOOP_HOME/build/*.jar`; do export CLASSPATH=$CLASSPATH:$f done This still produced the same error, so (thanks to the more detailed error output your patch provided) i renamed hadoop-0.19.0-dev-core.jar to hadoop-core.jar to match the regexp. Then i figured out that i can't use dfs://master:9000 becaus in hadoop-site.xml i specified that dfs should run on port 54310 (doh!). So i issued this command: ./fuse_dfs_wrapper.sh dfs://master:54310 /mnt/hadoop -d Succes! Even though the output from df -h is .. weird : fuse 512M 0 512M 0% /mnt/hadoop I added some data: for x in `seq 1 25`;do dd if=/dev/zero of=/mnt/hadoop/test-$x.raw bs=1MB count=10 done And now the output from df -h is: fuse 512M -3.4G 3.9G - /mnt/hadoop Note that my HDFS setup now consists of 20 nodes, exporting 15G each, so df is a little confused. Hadoop's status page (dfshealth.jsp) correctly displays the output though, evenly dividing the blocks over all the nodes. What i didn't understand however, is why there's no fuse-dfs in the downloadable tarballs. Am i looking in the wrong place perhaps? Anyway, now that i got things mounted, i come upon the next problem. I can't do much else than dd :) [EMAIL PROTECTED] fuse-dfs]# touch /mnt/hadoop/test.tst touch: setting times of `/mnt/hadoop/test.tst': Function not implemented regards, Sebastian
Re: hdfs question
One way to get all Unix commands to work as is is to mount hdfs as a normal unix filesystem with either fuse-dfs (in contrib) or hdfs-fuse (on google code). Pete On 8/6/08 5:08 PM, Mori Bellamy [EMAIL PROTECTED] wrote: hey all, often i find it would be convenient for me to run conventional unix commands on hdfs, such as using the following to delete the contents of my HDFS hadoop dfs -rm * or moving files from one folder to another: hadoop dfs -mv /path/one/* path/two/ does anyone know of a way to do this?
Re: fuse-dfs
This just means your classpath is not set properly, so when fuse-dfs uses libhdfs to try and connect to your server, it cannot instantiate hadoop objects. I have a JIRA open to improve error messaging when this happens: https://issues.apache.org/jira/browse/HADOOP-3918 If you use the fuse_dfs_wrapper.sh, you should be able to set HADOOP_HOME and it will create the classpath for you. In retrospect, fuse_dfs_wrapper.sh should probably complain and exit if HADOOP_HOME is not set. -- pete On 8/7/08 2:35 PM, Sebastian Vieira [EMAIL PROTECTED] wrote: On Thu, Aug 7, 2008 at 4:25 PM, Pete Wyckoff [EMAIL PROTECTED] wrote: Hi Sebastian, Those 2 things are just warnings and shouldn't cause any problems. What happens when you ls /mnt/hadoop ? [EMAIL PROTECTED] fuse-dfs]# ls /mnt/hadoop ls: /mnt/hadoop: Transport endpoint is not connected Also, this happens when i start fuse-dfs in one terminal, and do a df -h in another: [EMAIL PROTECTED] fuse-dfs]# ./fuse_dfs_wrapper.sh dfs://master:9000 /mnt/hadoop -d port=9000,server=master fuse-dfs didn't recognize /mnt/hadoop,-2 fuse-dfs ignoring option -d unique: 1, opcode: INIT (26), nodeid: 0, insize: 56 INIT: 7.8 flags=0x0003 max_readahead=0x0002 INIT: 7.8 flags=0x0001 max_readahead=0x0002 max_write=0x0010 unique: 1, error: 0 (Success), outsize: 40 unique: 2, opcode: STATFS (17), nodeid: 1, insize: 40 -now i do a df -h in the other term- Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClassInternal(Unknown Source) Then the output from df is: df: `/mnt/hadoop': Software caused connection abort And also what version of fuse-dfs are you using? The handling of options is different in trunk than in the last release. [EMAIL PROTECTED] fuse-dfs]# ./fuse_dfs --version ./fuse_dfs 0.1.0 I did a checkout of the latest svn and compiled using the command you gave in one of your previous mails. You can also look in /var/log/messages. Only one line: Aug 7 20:21:05 master fuse_dfs: mounting dfs://master:9000/ Thanks for your time, Sebastian
Re: fuse-dfs
Hi Sebastian, The problem is that hdfs.so is supposed to be in build/libhdfs but for some reason isn't. Have you tried doing a ant compile-libhdfs -Dlibhdfs=1 ? And then checked if hdfs.so is in build/libhdfs ? Thanks, pete On 8/6/08 5:04 AM, Sebastian Vieira [EMAIL PROTECTED] wrote: Hi, I have installed Hadoop on 20 nodes (data storage) and one master (namenode) to which i want to add data. I have learned that this is possible through a Java API or via the Hadoop shell. However, i would like to mount the HDFS using FUSE and i discovered that there's a contrib/fuse-dfs within the Hadoop tar.gz package. Now i read the README file and noticed that i was unable to compile using this command: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 If i change the line to: ant compile-contrib -Dcompile.c++=1 -Dlibhdfs-fuse=1 It goes a little bit further. It will now start the configure script, but still fails. I've tried alot of different things but i'm unable to compile fuse-dfs. This is a piece of the error i get from ant: compile: [echo] contrib: fuse-dfs -snip- [exec] Making all in src [exec] make[1]: Entering directory `/usr/local/src/hadoop-core-trunk/src/contrib/fuse-dfs/src' [exec] gcc -Wall -O3 -L/usr/local/src/hadoop-core-trunk/build/libhdfs -lhdfs -L/usr/lib -lfuse -L/usr/java/jdk1.6.0_07/jre/lib/i386/server -ljvm -o fuse_dfs fuse_dfs.o [exec] /usr/bin/ld: cannot find -lhdfs [exec] collect2: ld returned 1 exit status [exec] make[1]: *** [fuse_dfs] Error 1 [exec] make[1]: Leaving directory `/usr/local/src/hadoop-core-trunk/src/contrib/fuse-dfs/src' [exec] make: *** [all-recursive] Error 1 BUILD FAILED /usr/local/src/hadoop-core-trunk/build.xml:413: The following error occurred while executing this line: /usr/local/src/hadoop-core-trunk/src/contrib/build.xml:30: The following error occurred while executing this line: /usr/local/src/hadoop-core-trunk/src/contrib/fuse-dfs/build.xml:40: exec returned: 2 Could somebody shed some light on this? thanks, Sebastian.
Re: fuse-dfs
Sorry - I see the problem now: should be: Ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1 Compile-contrib depends on compile-libhdfs which also requires the -Dlibhdfs=1 property to be set. pete On 8/6/08 5:04 AM, Sebastian Vieira [EMAIL PROTECTED] wrote: Hi, I have installed Hadoop on 20 nodes (data storage) and one master (namenode) to which i want to add data. I have learned that this is possible through a Java API or via the Hadoop shell. However, i would like to mount the HDFS using FUSE and i discovered that there's a contrib/fuse-dfs within the Hadoop tar.gz package. Now i read the README file and noticed that i was unable to compile using this command: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 If i change the line to: ant compile-contrib -Dcompile.c++=1 -Dlibhdfs-fuse=1 It goes a little bit further. It will now start the configure script, but still fails. I've tried alot of different things but i'm unable to compile fuse-dfs. This is a piece of the error i get from ant: compile: [echo] contrib: fuse-dfs -snip- [exec] Making all in src [exec] make[1]: Entering directory `/usr/local/src/hadoop-core-trunk/src/contrib/fuse-dfs/src' [exec] gcc -Wall -O3 -L/usr/local/src/hadoop-core-trunk/build/libhdfs -lhdfs -L/usr/lib -lfuse -L/usr/java/jdk1.6.0_07/jre/lib/i386/server -ljvm -o fuse_dfs fuse_dfs.o [exec] /usr/bin/ld: cannot find -lhdfs [exec] collect2: ld returned 1 exit status [exec] make[1]: *** [fuse_dfs] Error 1 [exec] make[1]: Leaving directory `/usr/local/src/hadoop-core-trunk/src/contrib/fuse-dfs/src' [exec] make: *** [all-recursive] Error 1 BUILD FAILED /usr/local/src/hadoop-core-trunk/build.xml:413: The following error occurred while executing this line: /usr/local/src/hadoop-core-trunk/src/contrib/build.xml:30: The following error occurred while executing this line: /usr/local/src/hadoop-core-trunk/src/contrib/fuse-dfs/build.xml:40: exec returned: 2 Could somebody shed some light on this? thanks, Sebastian.
Re: Fuse-j-hadoopfs
Hi Xavier, RE: fuse dfs having facebook specific things in it, I think that the trunk version should be pretty clean. As far as permissions in fuse dfs, the following 2 jiras relate to that, and Craig Macdonald is working on it. https://issues.apache.org/jira/browse/HADOOP-3765 https://issues.apache.org/jira/browse/HADOOP-3536 Let me know if I've missed anything in the wrapper shell script or Makefile. Thanks, pete On 7/15/08 2:53 PM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi everybody, I was wondering if the people that mainting fuse-j-hadoopfs is working to support the HDFS permissions? Another thing, the people who works in fuse_dfs is it possible to clean the scripts fuse_dfs_wrapper.sh and the Makefiles. There is a lot of things from facebook so that will be nice if the scripts are more clean and generic. Thanks for the answer, Xavier
Re: 答复: Problem with key aggregation when number of reduce tasks is more than 1
Yes and as such, we've found better load balancing when the #of reduces is a prime #. Although the string.hashCode isn't great for short strings. On 4/11/08 4:16 AM, Zhang, jian [EMAIL PROTECTED] wrote: Hi, Please read this, you need to implement partitioner. It controls which key is sent to which reducer, if u want to get unique key result, you need to implement partitioner and the compareTO function should work properly. [WIKI] Partitioner Partitioner partitions the key space. Partitioner controls the partitioning of the keys of the intermediate map-outputs. The key (or a subset of the key) is used to derive the partition, typically by a hash function. The total number of partitions is the same as the number of reduce tasks for the job. Hence this controls which of the m reduce tasks the intermediate key (and hence the record) is sent to for reduction. HashPartitioner is the default Partitioner. Best Regards Jian Zhang -邮件原件- 发件人: Harish Mallipeddi [mailto:[EMAIL PROTECTED] 发送时间: 2008年4月11日 19:06 收件人: core-user@hadoop.apache.org 主题: Problem with key aggregation when number of reduce tasks is more than 1 Hi all, I wrote a custom key class (implements WritableComparable) and implemented the compareTo() method inside this class. Everything works fine when I run the m/r job with 1 reduce task (via setNumReduceTasks). Keys are sorted correctly in the output files. But when I increase the number of reduce tasks, keys don't get aggregated properly; same keys seem to end up in separate output files (output/part-0, output/part-1, etc). This should not happen because right before reduce() gets called, all (k,v) pairs from all map outputs with the same 'k' are aggregated and the reduce function just iterates over the values (v1, v2, etc)? Do I need to implement anything else inside my custom key class other than compareTo? I also tried implementing equals() but that didn't help either. Then I came across setOutputKeyComparator(). So I added a custom Comparator class inside the key class and tried setting this on the JobConf object. But that didn't work either. What could be wrong? Cheers,
Experience with Hadoop on Open Solaris
Anyone have experience running a production cluster on Open Solaris? The advantage of course is the availability of ZFS, but I haven't seen much in the way of people on the list mentioning they use Open Solaris. Thanks, pete
Re: HDFS interface
Hi Naama, This JIRA is tracking both the fuse and webdav efforts: https://issues.apache.org/jira/browse/HADOOP-4 -- pete On 3/10/08 11:17 PM, Naama Kraus [EMAIL PROTECTED] wrote: Hi, I'd be interested in information about interfaces to HDFS other then the DFSShell commands. I've seen threads about dfs and fuse, dfs and WebDav. Could anyone provide more details or point me to related resources ? What's the status of these ? Thanks, Naama
Re: How to compile fuse-dfs
But, to be clear, you can do mv, rm, mkdir, rmdir. On 3/11/08 10:24 AM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Thanks Pete. I'll be waiting for 0.17 then
Re: How to compile fuse-dfs
Hi Xavier, If you run ./bootsrap.sh does it not create a Makefile for you? There is a bug in the Makefile that hardcodes it to amd64. I will look at this. What kernel are you using and what HW? --pete On 3/10/08 2:23 PM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi everybody, I'm trying to compile fuse-dfs but I have problems. I don't have a lot of experience with C++. I would like to know: Is it a clear readme file with the instructions to compile, install fuse-dfs? Do I need to replace fuse_dfs.c with the one in fuse-dfs/src/fuse_dfs.c? Do I need to set up different flag if I'm using a i386 or 86 machine? Which one and Where? Which make file do I need to use to compile the code? Thanks Xavier
Re: Does Hadoop Honor Reserved Space?
+1
(obviously :))
On 3/10/08 5:26 PM, Joydeep Sen Sarma [EMAIL PROTECTED] wrote:
I have left some comments behind on the jira.
We could argue over what's the right thing to do (and we will on the
Jira) - but the higher level problem is that this is another case where
backwards compatibility with existing semantics of this option was not
carried over. Neither was there any notification to admins about this
change. The change notes just do not convey the import of this change to
existing deployments (incidentally 1463 was classified as 'Bug Fix' -
not that putting under 'Incompatible Fix' would have helped imho).
Would request the board/committers to consider setting up something
along the lines of:
1. have something better than Change Notes to convey interface changes
2. a field in the JIRA that marks it out as important from interface
change point of view (with notes on what's changing). This could be used
to auto-populate #1
3. Some way of auto-subscribing to bugs that are causing interface
changes (even an email filter on the jira mails would do).
As Hadoop user base keeps growing - and gets used for 'production' tasks
- I think it's absolutely essential that users/admins can keep in tune
with changes that affect their deployments. Otherwise - any organization
other than Yahoo would have tough time upgrading.
(I am new to open-source - but surely this has been solved before?)
Joydeep
-Original Message-
From: Hairong Kuang [mailto:[EMAIL PROTECTED]
Sent: Monday, March 10, 2008 5:17 PM
To: core-user@hadoop.apache.org
Subject: Re: Does Hadoop Honor Reserved Space?
I think you have a misunderstanding of the reserved parameter. As I
commented on hadoop-1463, remember that dfs.du.reserve is the space for
non-dfs usage, including the space for map/reduce, other application, fs
meta-data etc. In your case since /usr already takes 45GB, it far
exceeds
the reserved limit 1G. You should set the reserved space to be 50G.
Hairong
On 3/10/08 4:54 PM, Joydeep Sen Sarma [EMAIL PROTECTED] wrote:
Filed https://issues.apache.org/jira/browse/HADOOP-2991
-Original Message-
From: Joydeep Sen Sarma [mailto:[EMAIL PROTECTED]
Sent: Monday, March 10, 2008 12:56 PM
To: core-user@hadoop.apache.org; core-user@hadoop.apache.org
Cc: Pete Wyckoff
Subject: RE: Does Hadoop Honor Reserved Space?
folks - Jimmy is right - as we have unfortunately hit it as well:
https://issues.apache.org/jira/browse/HADOOP-1463 caused a regression.
we have left some comments on the bug - but can't reopen it.
this is going to be affecting all 0.15 and 0.16 deployments!
-Original Message-
From: Hairong Kuang [mailto:[EMAIL PROTECTED]
Sent: Thu 3/6/2008 2:01 PM
To: core-user@hadoop.apache.org
Subject: Re: Does Hadoop Honor Reserved Space?
In addition to the version, could you please send us a copy of the
datanode
report by running the command bin/hadoop dfsadmin -report?
Thanks,
Hairong
On 3/6/08 11:56 AM, Joydeep Sen Sarma [EMAIL PROTECTED] wrote:
but intermediate data is stored in a different directory from
dfs/data
(something like mapred/local by default i think).
what version are u running?
-Original Message-
From: Ashwinder Ahluwalia on behalf of [EMAIL PROTECTED]
Sent: Thu 3/6/2008 10:14 AM
To: core-user@hadoop.apache.org
Subject: RE: Does Hadoop Honor Reserved Space?
I've run into a similar issue in the past. From what I understand,
this
parameter only controls the HDFS space usage. However, the
intermediate data
in
the map reduce job is stored on the local file system (not HDFS) and
is not
subject to this configuration.
In the past I have used mapred.local.dir.minspacekill and
mapred.local.dir.minspacestart to control the amount of space that is
allowable
for use by this temporary data.
Not sure if that is the best approach though, so I'd love to hear
what
other
people have done. In your case, you have a map-red job that will
consume too
much space (without setting a limit, you didn't have enough disk
capacity for
the job), so looking at mapred.output.compress and
mapred.compress.map.output
might be useful to decrease the job's disk requirements.
--Ash
-Original Message-
From: Jimmy Wan [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 06, 2008 9:56 AM
To: core-user@hadoop.apache.org
Subject: Does Hadoop Honor Reserved Space?
I've got 2 datanodes setup with the following configuration
parameter:
property
namedfs.datanode.du.reserved/name
value429496729600/value
descriptionReserved space in bytes per volume. Always leave this
much
space free for non dfs use.
/description
/property
Both are housed on 800GB volumes, so I thought this would keep about
half
the volume free for non-HDFS usage.
After some long running jobs last night, both disk volumes were
completely
filled. The bulk of the data was in:
${my.hadoop.tmp.dir}/hadoop
Re: changes to compression interfaces in 0.15?
If the API semantics are changing under you, you have to change your code
whether or not the API is pulled or deprecated. Pulling it makes it more
obvious that the user has to change his/her code.
-- pete
On 2/21/08 12:41 PM, Arun C Murthy [EMAIL PROTECTED] wrote:
On Feb 21, 2008, at 12:20 PM, Joydeep Sen Sarma wrote:
To maintain backward compat, we cannot remove old apis - the standard
procedure is to deprecate them for the next release and remove them
in subsequent releases.
you've got to be kidding.
we didn't maintain backwards compatibility. my app broke. Simple
and straightforward. and the old interfaces are not deprecated (to
quote 0.15.3 on a 'deprecated' interface:
You are right, HADOOP-1851 didn't fix it right. I've filed HADOOP-2869.
We do need to be more diligent about listing config changes in
CHANGES.txt for starters, and that point is taken. However, we can't
start pulling out apis without deprecating them first.
Arun
/**
* Set the compression type for sequence files.
* @param job the configuration to modify
* @param val the new compression type (none, block, record)
*/
static public void setCompressionType(Configuration job,
CompressionType val) {
)
I (and i would suspect any average user willing to recompile code)
would much much rather that we broke backwards compatibility
immediately rather than maintain carry over defunct apis that
insidiously break application behavior.
and of course - this does not address the point that the option
strings themselves are depcreated. (remember - people set options
explicitly from xml files and streaming. not everyone goes through
java apis)).
--
as one of my dear professors once said - put ur self in the other
person's shoe. consider that u were in my position and that a
production app suddenly went from consuming 100G to 1TB. and
everything slowed down drastically. and it did not give any sign
that anything was amiss. everything looked golden on the ourside.
what would be ur reaction if u find out after a week that the
system was full and numerous processes had to be re-run? how would
you have figured that was going to happen by looking at the
INCOMPATIBLE section (which btw - i did carefully before sending my
mail).
(fortunately i escaped the worst case - but i think this is a real
call to action)
-Original Message-
From: Arun C Murthy [mailto:[EMAIL PROTECTED]
Sent: Thu 2/21/2008 11:21 AM
To: core-user@hadoop.apache.org
Subject: Re: changes to compression interfaces in 0.15?
Joydeep,
On Feb 20, 2008, at 5:06 PM, Joydeep Sen Sarma wrote:
Hi developers,
In migrating to 0.15 - i am noticing that the compression interfaces
have changed:
- compression type for sequencefile outputs used to be set
by:
SequenceFile.setCompressionType()
- now it seems to be set using:
sequenceFileOutputFormat.setOutputCompressionType()
Yes, we added SequenceFileOutputFormat.setOutputCompressionType and
deprecated the old api. (HADOOP-1851)
The change is for the better - but would it be possible to:
- remove old/dead interfaces. That would have been a
straightforward hint for applications to look for new interfaces.
(hadoop-default.xml also still has setting for old conf variable:
io.seqfile.compression.type)
To maintain backward compat, we cannot remove old apis - the standard
procedure is to deprecate them for the next release and remove them
in subsequent releases.
- if possible - document changed interfaces in the release
notes (there's no way we can find this out by looking at the long
list
of Jiras).
Please look at the INCOMPATIBLE CHANGES section of CHANGES.txt,
HADOOP-1851 is listed there. Admittedly we can do better, but that is
a good place to look for when upgrading to newer releases.
i am not sure how updated the wiki is on the compression stuff (my
responsibility to update it) - but please do consider the impact of
Please use the forrest-based docs (on the hadoop website - e.g.
mapred_tutorial.html) rather than the wiki as the gold-standard. The
reason we moved away from the wiki is precisely this - harder to
maintain docs per release etc.
changing interfaces on existing applications. (maybe we should have a
JIRA tag to mark out bugs that change interfaces).
Again, CHANGES.txt and INCOMPATIBLE CHANGES section for now.
Arun
As always - thanks for all the fish (err .. working code),
Joydeep
