org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@482d59a3, java.io.IOException: java.io.IOException: No FileSystem for scheme: maprfs
Folk,
I am trying to bulk load the hdfs file into hbase with
LoadIncrementalHFiles loader = new LoadIncrementalHFiles(conf);
loader.doBulkLoad(new Path(args[1]), hTable);
However, i receive exception of java.io.IOException: java.io.IOException:
No FileSystem for scheme: maprfs
Exception in thread "main" java.io.IOException: BulkLoad encountered an
unrecoverable problem
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:331)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:261)
at com.walmartlabs.targeting.mapred.Driver.main(Driver.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=10, exceptions:
Tue Jun 17 21:48:58 PDT 2014,
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@482d59a3,
java.io.IOException: java.io.IOException: No FileSystem for scheme: maprfs
What is the reason for this exception? I did some googling, and tried to
add some config to Hbase configuration:
hbaseConf.set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
hbaseConf.set("fs.file.impl",
org.apache.hadoop.fs.LocalFileSystem.class.getName());
But it does not have any effect. Is this error indicating that I basically
need a hbase mapr client? currently my pom looks like this;
org.apache.hadoop
hadoop-client
1.0.3
org.apache.hadoop
hadoop-core
1.2.1
org.apache.httpcomponents
httpclient
4.1.1
com.google.code.gson
gson
2.2.4
org.apache.hbase
hbase
0.94.6.1
Any idea?
Thanks advance.
Chen
UNSUBSCRIBE
Re: Storing videos in Hdfs
What do you want to achieve with this? I've seen that Hadoop is being used for video analytics, just storing video's metadata, quantity of unique views and that kind of stuff; but I've never seen this use-case. A good example of this is Ooyala, which have been used Hadoop+ Apache Cassandra for this, although they migrated to a Spark/Shark + Cassandra solution. They wrote a whitepaper called "Designing a Scalable Database for Online Video Analytics" and Evan Chan(@evanfchan) did a great talk in the last Cassandra Summit 2013 about how to use Spark/Shark + Cassandra for Real-Time video analytics. -- Marcos Ortiz[1] (@marcosluis2186[2]) http://about.me/marcosortiz[3] On Tuesday, June 17, 2014 06:12:49 PM alajangikish...@gmail.com wrote: > Hi hadoopers, > > What is the best way to store video files in Hdfs? > > Sent from my iPhone [1] http://www.linkedin.com/in/mlortiz [2] http://twitter.com/marcosluis2186 [3] http://about.me/marcosortiz VII Escuela Internacional de Verano en la UCI del 30 de junio al 11 de julio de 2014. Ver www.uci.cu
Storing videos in Hdfs
Hi hadoopers, What is the best way to store video files in Hdfs? Sent from my iPhone
Re: HDFS File Writes & Reads
1. HDFS doesn't allow parallel write 2. HDFS use pipeline to write multiple replicas, so it doesn't take three times more time than a traditional file write 3. HDFS allow parallel read 2014-06-17 19:17 GMT+08:00 Vijaya Narayana Reddy Bhoomi Reddy < vijay.bhoomire...@gmail.com>: > Hi, > > I have a basic question regarding file writes and reads in HDFS. Is the > file write and read process a sequential activity or executed in parallel? > > For example, lets assume that there is a File File1 which constitutes of > three blocks B1, B2 and B3. > > 1. Will the write process write B2 only after B1 is complete and B3 only > after B2 is complete or for a large file with many blocks, can this happen > in parallel? In all the hadoop documentation, I read this to be a > sequential operation. Does that mean for a file of 1TB, it takes three > times more time than a traditional file write? (due to default replication > factor of 3) > 2. Is it similar in the case of read as well? > > Kindly someone please provide some clarity on this... > > Regards > Vijay > -- Best Wishes! Yours, Zesheng
HDFS File Writes & Reads
Hi, I have a basic question regarding file writes and reads in HDFS. Is the file write and read process a sequential activity or executed in parallel? For example, lets assume that there is a File File1 which constitutes of three blocks B1, B2 and B3. 1. Will the write process write B2 only after B1 is complete and B3 only after B2 is complete or for a large file with many blocks, can this happen in parallel? In all the hadoop documentation, I read this to be a sequential operation. Does that mean for a file of 1TB, it takes three times more time than a traditional file write? (due to default replication factor of 3) 2. Is it similar in the case of read as well? Kindly someone please provide some clarity on this... Regards Vijay
hadoop-2.2: build error on FreeBSD
Hello! FreeBSD does not need "-ldl" when linking programs that use dlopen() (dlopen is in libc). Now I am getting the following error trying to compile hadoop-2.2.0 on FreeBSD: [exec] /usr/bin/cc -fPIC -g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -shared -Wl,-soname,libhdfs.so.0.0.0 -o target/usr/local/lib/libhdfs.so.0.0.0 CMakeFiles/hdfs.dir/main/native/libhdfs/exception.c.o CMakeFiles/hdfs.dir/main/native/libhdfs/jni_helper.c.o CMakeFiles/hdfs.dir/main/native/libhdfs/hdfs.c.o /usr/local/openjdk7/jre/lib/amd64/server/libjvm.so -ldl -lpthread -Wl,-rpath,/usr/local/openjdk7/jre/lib/amd64/server [exec] /usr/local/bin/cmake -H/usr/home/mitya/hadoop-2.2.0-src/hadoop-hdfs-project/hadoop-hdfs/src -B/usr/home/mitya/hadoop-2.2.0-src/h/usr/bin/ld: cannot find -ldl Can you please adjust your build system not to use "-ldl" on FreeBSD? Thanks!
