ClientProtocol create、mkdirs 、rename and delete methods are not Idempotent
I think these methods should are idempotent, these methods should be repeated calls to be harmless by same client. Thanks, LiuLei
Re: simple tutorial or procedure for configuring HDFS FEDERATION 0.23.3 in a cluster
A simple search ought to have found this for you. http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html From: Visioner Sadak To: user@hadoop.apache.org Sent: Saturday, October 27, 2012 2:03 AM Subject: Re: simple tutorial or procedure for configuring HDFS FEDERATION 0.23.3 in a cluster any hints friends... On Fri, Oct 19, 2012 at 12:54 PM, Visioner Sadak wrote: Is there any simple links tutorials for installing HDFS federation 0.23.3 in a cluster set up
Re: Cluster wide atomic operations
z
Re: HDFS HA IO Fencing
If you use NSFv4 you should be able to use locks and when a machine dies / fails to renew the lease, the other machine can take over. On Friday, October 26, 2012, Todd Lipcon wrote: > NFS Locks typically last forever if you disconnect abruptly. So they are > not sufficient -- your standby wouldn't be able to take over without manual > intervention to remove the lock. > > If you want to build an unreliable system that might corrupt your data, > you could set up 'shell(/bin/true)' as a second fencing method. But, it's > really a bad idea. There are failure scenarios which could cause split > brain if you do this, and you'd very likely lose data. > > -Todd > > On Fri, Oct 26, 2012 at 1:59 AM, lei liu > > > wrote: > >> We are using NFS for Shared storage, Can we use linux nfslcok service to >> implement IO Fencing ? >> >> >> 2012/10/26 Steve Loughran > 'cvml', 'ste...@hortonworks.com');>> >> >>> >>> >>> On 25 October 2012 14:08, Todd Lipcon >> 'cvml', 't...@cloudera.com');> >>> > wrote: >>> Hi Liu, Locks are not sufficient, because there is no way to enforce a lock in a distributed system without unbounded blocking. What you might be referring to is a lease, but leases are still problematic unless you can put bounds on the speed with which clocks progress on different machines, _and_ have strict guarantees on the way each node's scheduler works. With Linux and Java, the latter is tough. >>> on any OS running in any virtual environment, including EC2, time is >>> entirely unpredictable, just to make things worse. >>> >>> >>> On a single machine you can use file locking as the OS will know that >>> the process is dead and closes the file; other programs can attempt to open >>> the same file with exclusive locking -and, by getting the right failures, >>> know that something else has the file, hence the other process is live. >>> Shared NFS storage you need to mount with softlock set precisely to stop >>> file locks lasting until some lease has expired, because the on-host >>> liveness probes detect failure faster and want to react to it. >>> >>> >>> -Steve >>> >> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera > -- Thanks -balaji -- http://balajin.net/blog/ http://flic.kr/balajijegan
Re: java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.(I)V
Hi, What version/release/distro of Hadoop is this from, 2.x? You need the commons-codec-1.4.jar on your HDFS classpath. Normally, this is available at $HADOOP_PREFIX/lib (1.x based releases) or in $HADOOP_PREFIX/share/hadoop/common/lib/ (2.x based releases). On Sat, Oct 27, 2012 at 10:10 AM, centerqi hu wrote: > hi all > when I access hdfs by webhdfs > curl -i > "http://xx.xxx.xxx.xxx:50070/webhdfs/v1/usr/local/hadoop/log-analyze/tmp/input/prom/?op=GETFILESTATUS&user.name=hadoop"; > Someone encountered the same problem? > > The following error occurred > > org.apache.commons.codec.binary.Base64.(I)V > > RequestURI=/webhdfs/v1/usr/local/hadoop/log-analyze/tmp/input/prom/ > > Caused by: > > java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.(I)V > at > org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:94) > at > org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:54) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:368) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:324) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) > > > -- > cente...@gmail.com|Sam -- Harsh J
Re: HDFS HA IO Fencing
I use NFS V4 to test the java FileLock.
The 192.168.1.233 machine is NFS Server, the nfs configuration are
/home/hdfs.ha/share 192.168.1.221(rw,sync,no_root_squash)
/home/hdfs.ha/share 192.168.1.222(rw,sync,no_root_squash)
in /etc/exports file.
I run below commands to start nfs server:
service nfs start
service nfslock start
The 192.168.1.221 and 192.168.1.222 machines are NFS Client, the nfs
configuration is
192.168.1.223:/home/hdfs.ha/share /home/hdfs.ha/share nfs
rsize=8192,wsize=8192,timeo=14,intr in /etc/fstab file.
I run below commands to start nfs client:
service nfs start
service nfslock start
I write one programm to receive file lock:
public class FileLockTest {
FileLock lock;
public void lock(String path,boolean isShare) throws IOException {
this.lock = tryLock(path,isShare);
if (lock == null) {
String msg = "Cannot lock storage " + path
+ ". The directory is already locked.";
System.out.println(msg);
throw new IOException(msg);
}
}
private FileLock tryLock(String path,boolean isShare) throws
IOException {
boolean deletionHookAdded = false;
File lockF = new File(path);
if (!lockF.exists()) {
lockF.deleteOnExit();
deletionHookAdded = true;
}
RandomAccessFile file = new RandomAccessFile(lockF, "rws");
FileLock res = null;
try {
res = file.getChannel().tryLock(0,Long.MAX_VALUE,isShare);
} catch (OverlappingFileLockException oe) {
file.close();
return null;
} catch (IOException e) {
e.printStackTrace();
file.close();
throw e;
}
if (res != null && !deletionHookAdded) {
// If the file existed prior to our startup, we didn't
// call deleteOnExit above. But since we successfully locked
// the dir, we can take care of cleaning it up.
lockF.deleteOnExit();
}
return res;
}
public static void main(String[] s) {
FileLockTest fileLockTest =new FileLockTest();
try {
fileLockTest.lock(s[0], Boolean.valueOf(s[1]));
Thread.sleep(1000*60*60*1);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I do two test cases.
1. The network is OK
I run " java -cp ./filelock.jar lock.FileLockTest
/home/hdfs.ha/share/test.lock false" command in 192.168.1.221 to hold file
lock, and then I run same command to hold same file lock in 192.168.1.222,
throw below exception:
Cannot lock storage /home/hdfs.ha/share/test.lock. The directory is already
locked.
java.io.IOException: Cannot lock storage /home/hdfs.ha/share/test.lock. The
directory is already locked.
at lock.FileLockTest.lock(FileLockTest.java:18)
at lock.FileLockTest.main(FileLockTest.java:53)
2. machine which hold file lock is diconnected
I run " java -cp ./filelock.jar lock.FileLockTest
/home/hdfs.ha/share/test.lock false" command on 192.168.1.221, then
192.168.1.221 machine is disconnected from network . After three minutes ,
I run the " java -cp ./filelock.jar lock.FileLockTest
/home/hdfs.ha/share/test.lock false" command on 192.168.1.222, that can
hold the file lock.
I use "mount | grep nfs" command to examine the mount nfs directory on
192.168.1.221, the share directory /home/hdfs.ha/share/ is disappear on
192.168.1.221 machine. So I think when the machine is disconnected for
a long time, other machine can receive the same file lock.
Re: FAILED: Hive Internal Error
On 27 October 2012 10:44, sagar nikam wrote: > Respected Sir/Madam, > >> >> this is a hive question. Please don't cross post to general@hadoop or user@hadoop. thanks
Re: FAILED: Hive Internal Error
Respected Sir/Madam, > > > I have installed Hadoop on my ubuntu 12.04 system > i installed HIVE also.for some days it works fine..but one day i directly > shutdown my machine (with closing hive & hadoop) > now i am running some query..it throws error (query like "show > databases","use database_name" works fine) but for below > > hive> select count(*) from cidade; > Error thrown:- > > FAILED: Hive Internal Error: > java.lang.RuntimeException(java.net.ConnectException: Call to localhost/ > 127.0.0.1:54310 failed on connection exception: > java.net.ConnectException: Connection refused) > java.lang.RuntimeException: java.net.ConnectException: Call to localhost/ > 127.0.0.1:54310 failed on connection exception: > java.net.ConnectException: Connection refused > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151) > at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190) > at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:54310failed > on connection exception: java.net.ConnectException: Connection > refused > at org.apache.hadoop.ipc.Client.wrapException(Client.java:767) > at org.apache.hadoop.ipc.Client.call(Client.java:743) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy4.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) > at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:207) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:170) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175) > at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:145) > ... 15 more > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304) > at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:860) > at org.apache.hadoop.ipc.Client.call(Client.java:720) > ... 28 more > > = > > is it that ,some files may damage during shutdown ? > what could be the error ? >
Re: reference architecture
Thank you so much everybody, for the valuable comments. On Saturday, October 27, 2012, Russell Jurney wrote: > Russell Jurney http://datasyndrome.com > > On Oct 25, 2012, at 12:24 PM, "Daniel Käfer" wrote: > >> Hello all, >> >> I'm looking for a reference architecture for hadoop. The only result I >> found is Lambda architecture from Nathan Marz[0]. >> >> With architecture I mean answers to question like: >> - How should I store the data? CSV, Thirft, ProtoBuf > You should use Avro. >> - How should I model the data? ER-Model, Starschema, something new? > You should use document format. >> - normalized or denormalized or both (master data normalized, then >> transformation to denormalized, like ETL) > Demoralized fully, into document format. >> - How should i combine database and HDFS-Files? > Don't. Put everything on HDFS. >> >> Are there any other documented architectures for hadoop? > I really did make an example in my book. It is just one example, but > you wanted answers to questions that always 'depend.' You can check it > out in slides: http://www.slideshare.net/mobile/hortonworks/agile-analytics-applications-on-hadoop >> >> Regards >> Daniel Käfer >> >> >> [0] http://www.manning.com/marz/ just a preprint yet, not completed >> > -- Regards, Mohammad Tariq
Re: reference architecture
Russell Jurney http://datasyndrome.com On Oct 25, 2012, at 12:24 PM, "Daniel Käfer" wrote: > Hello all, > > I'm looking for a reference architecture for hadoop. The only result I > found is Lambda architecture from Nathan Marz[0]. > > With architecture I mean answers to question like: > - How should I store the data? CSV, Thirft, ProtoBuf You should use Avro. > - How should I model the data? ER-Model, Starschema, something new? You should use document format. > - normalized or denormalized or both (master data normalized, then > transformation to denormalized, like ETL) Demoralized fully, into document format. > - How should i combine database and HDFS-Files? Don't. Put everything on HDFS. > > Are there any other documented architectures for hadoop? I really did make an example in my book. It is just one example, but you wanted answers to questions that always 'depend.' You can check it out in slides: http://www.slideshare.net/mobile/hortonworks/agile-analytics-applications-on-hadoop > > Regards > Daniel Käfer > > > [0] http://www.manning.com/marz/ just a preprint yet, not completed >
Re: reference architecture
I define one of these in the book agile data, from O'Reilly. I express opinions on all matters you query us about. But you don't have to take my word for it... It's a reading rainbow! Jordi! Russell Jurney http://datasyndrome.com On Oct 27, 2012, at 1:09 AM, "Daniel Käfer" wrote: > Am Freitag, den 26.10.2012, 18:25 +0100 schrieb Steve Loughran: >> Depends on the amount of data and expected use. If it's transient food >> for the next MR jobs: HDFS > > Thanks for your help >
Re: reference architecture
Am Freitag, den 26.10.2012, 18:25 +0100 schrieb Steve Loughran: > Depends on the amount of data and expected use. If it's transient food > for the next MR jobs: HDFS Thanks for your help
Re: simple tutorial or procedure for configuring HDFS FEDERATION 0.23.3 in a cluster
any hints friends... On Fri, Oct 19, 2012 at 12:54 PM, Visioner Sadak wrote: > Is there any simple links tutorials for installing HDFS federation 0.23.3 > in a cluster set up >
