Re: To Store Large Number of Video and Image files
Hi, Any idea about the implementation of Facebook f4? Does it use HBase as the indexer? Regards On Sun, Apr 17, 2016 at 2:36 PM, Ascot Moss wrote: > Hi, > > Yes, the files are immutable. > > Regards > > > On Sun, Apr 17, 2016 at 12:25 PM, Vladimir Rodionov < > vladrodio...@gmail.com> wrote: > >> >> have a project that needs to store large number of image and video >> files, >> >>the file size varies from 10MB to 10GB, the initial number of files will >> be >> >>0.1 billion and would grow over 1 billion, what will be the practical >> >>recommendations to store and view these files? >> >> >> Files are immutable? >> Write small files (less than 1 HDFS block) to large blob (combine them >> into single file), store large files >> directly to HDFS. Keep path index in HBase. >> >> If you need to delete files, mark them as deleted in HBase and run >> periodically GC job to perform real cleaning. >> >> -Vlad >> >> On Sat, Apr 16, 2016 at 7:35 PM, Ted Yu wrote: >> >> > There was HBASE-15370 for backport but it was decided not to backport >> the >> > feature. >> > >> > FYI >> > >> > On Sat, Apr 16, 2016 at 7:26 PM, Ascot Moss >> wrote: >> > >> > > Hi, >> > > >> > > About HBase-11339, >> > > "The size of the MOB data could not be very large, it better to keep >> the >> > > MOB size within 100KB and 10MB. Since MOB cells are written into the >> > > memstore before flushing, large MOB cells stress the memory in region >> > > servers." >> > > >> > > Can this be resolved if we provide more RAM in region servers? for >> > > instances, the servers in the cluster, each has 768GB RAM + 14 x 6T >> HDD. >> > > >> > > regards >> > > >> > > >> > > >> > > On Sun, Apr 17, 2016 at 9:56 AM, Ascot Moss >> > wrote: >> > > >> > > > Thanks Ted! >> > > > >> > > > Just visited HBASE-11339, its status is "resolved" however, it is >> for >> > > > "Fix Version : 2.0.0." >> > > > How to patch it to current HBase stable version (v1.1.4) ? >> > > > >> > > > About Fault Tolerance to DataCenter level, I am thinking HBase >> > > Replication >> > > > method to replicate HBase Tables to another cluster (backup one), is >> > > there >> > > > any real world reference about the replication performance, for >> > instances >> > > > if the bandwidth is 100MB/s? >> > > > >> > > > Regards >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >- >> > > > >> > > > >> > > > On Sun, Apr 17, 2016 at 9:40 AM, Ted Yu >> wrote: >> > > > >> > > >> Have you taken a look at HBASE-11339 (HBase MOB) ? >> > > >> >> > > >> Note: this feature does not handle 10GB objects well. Consider >> store >> > GB >> > > >> image on hdfs. >> > > >> >> > > >> Cheers >> > > >> >> > > >> On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss >> > > wrote: >> > > >> >> > > >> > Hi, >> > > >> > >> > > >> > I have a project that needs to store large number of image and >> video >> > > >> files, >> > > >> > the file size varies from 10MB to 10GB, the initial number of >> files >> > > >> will be >> > > >> > 0.1 billion and would grow over 1 billion, what will be the >> > practical >> > > >> > recommendations to store and view these files? >> > > >> > >> > > >> > >> > > >> > >> > > >> > #1 One cluster, store the HDFS URL in HBase and store the actual >> > file >> > > in >> > > >> > HDFS? (block_size as 128MB and replication factor as 3) >> > > >> > >> > > >> > >> > > >> > #2 One cluster, Store small files in HBase directly and use #1 >> for >> > > large >> > > >> > files? (block_size as 128MB and replication factor as 3) >> > > >> > >> > > >> > >> > > >> > #3 Multiple Hadoop/HBase clusters, each with different block_size >> > > >> settings? >> > > >> > >> > > >> > >> > > >> > e.g. cluster 1 (small): block_size as 128MB and replication >> > > factor >> > > >> as >> > > >> > 3, store all files in HBase if their file size is smaller 128MB >> > > >> > >> > > >> > cluster 2 (large): bigger block_size, say 4GB, >> > replication >> > > >> > factor as 3, store the HDFS URL in HBase and store the actual >> file >> > in >> > > >> HDFS >> > > >> > >> > > >> > >> > > >> > >> > > >> > #4 Use Hadoop Federation for large number of files? >> > > >> > >> > > >> > >> > > >> > About Fault Tolerance, need to consider four types of failures: >> > > driver, >> > > >> > host, rack, and datacenter failures. >> > > >> > >> > > >> > >> > > >> > Regards >> > > >> > >> > > >> >> > > > >> > > > >> > > >> > >> > >
Re: To Store Large Number of Video and Image files
Hi, Yes, the files are immutable. Regards On Sun, Apr 17, 2016 at 12:25 PM, Vladimir Rodionov wrote: > >> have a project that needs to store large number of image and video > files, > >>the file size varies from 10MB to 10GB, the initial number of files will > be > >>0.1 billion and would grow over 1 billion, what will be the practical > >>recommendations to store and view these files? > >> > Files are immutable? > Write small files (less than 1 HDFS block) to large blob (combine them > into single file), store large files > directly to HDFS. Keep path index in HBase. > > If you need to delete files, mark them as deleted in HBase and run > periodically GC job to perform real cleaning. > > -Vlad > > On Sat, Apr 16, 2016 at 7:35 PM, Ted Yu wrote: > > > There was HBASE-15370 for backport but it was decided not to backport the > > feature. > > > > FYI > > > > On Sat, Apr 16, 2016 at 7:26 PM, Ascot Moss > wrote: > > > > > Hi, > > > > > > About HBase-11339, > > > "The size of the MOB data could not be very large, it better to keep > the > > > MOB size within 100KB and 10MB. Since MOB cells are written into the > > > memstore before flushing, large MOB cells stress the memory in region > > > servers." > > > > > > Can this be resolved if we provide more RAM in region servers? for > > > instances, the servers in the cluster, each has 768GB RAM + 14 x 6T > HDD. > > > > > > regards > > > > > > > > > > > > On Sun, Apr 17, 2016 at 9:56 AM, Ascot Moss > > wrote: > > > > > > > Thanks Ted! > > > > > > > > Just visited HBASE-11339, its status is "resolved" however, it is for > > > > "Fix Version : 2.0.0." > > > > How to patch it to current HBase stable version (v1.1.4) ? > > > > > > > > About Fault Tolerance to DataCenter level, I am thinking HBase > > > Replication > > > > method to replicate HBase Tables to another cluster (backup one), is > > > there > > > > any real world reference about the replication performance, for > > instances > > > > if the bandwidth is 100MB/s? > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > > > > > > >- > > > > > > > > > > > > On Sun, Apr 17, 2016 at 9:40 AM, Ted Yu wrote: > > > > > > > >> Have you taken a look at HBASE-11339 (HBase MOB) ? > > > >> > > > >> Note: this feature does not handle 10GB objects well. Consider store > > GB > > > >> image on hdfs. > > > >> > > > >> Cheers > > > >> > > > >> On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss > > > wrote: > > > >> > > > >> > Hi, > > > >> > > > > >> > I have a project that needs to store large number of image and > video > > > >> files, > > > >> > the file size varies from 10MB to 10GB, the initial number of > files > > > >> will be > > > >> > 0.1 billion and would grow over 1 billion, what will be the > > practical > > > >> > recommendations to store and view these files? > > > >> > > > > >> > > > > >> > > > > >> > #1 One cluster, store the HDFS URL in HBase and store the actual > > file > > > in > > > >> > HDFS? (block_size as 128MB and replication factor as 3) > > > >> > > > > >> > > > > >> > #2 One cluster, Store small files in HBase directly and use #1 for > > > large > > > >> > files? (block_size as 128MB and replication factor as 3) > > > >> > > > > >> > > > > >> > #3 Multiple Hadoop/HBase clusters, each with different block_size > > > >> settings? > > > >> > > > > >> > > > > >> > e.g. cluster 1 (small): block_size as 128MB and replication > > > factor > > > >> as > > > >> > 3, store all files in HBase if their file size is smaller 128MB > > > >> > > > > >> > cluster 2 (large): bigger block_size, say 4GB, > > replication > > > >> > factor as 3, store the HDFS URL in HBase and store the actual file > > in > > > >> HDFS > > > >> > > > > >> > > > > >> > > > > >> > #4 Use Hadoop Federation for large number of files? > > > >> > > > > >> > > > > >> > About Fault Tolerance, need to consider four types of failures: > > > driver, > > > >> > host, rack, and datacenter failures. > > > >> > > > > >> > > > > >> > Regards > > > >> > > > > >> > > > > > > > > > > > > > >
Name node log roller destroyed metadata?
Hi all, HBase version: Version 0.94.18, re8f20e1020decc8f106e952f393f0484058068e2, Wed Nov 4 20:57:33 GMT 2015 Amazon EMR We got HBase namenode down recently with the "Namenode is not formatted error" and had to restore everything from backups. Below is what we have in the namenode log file a couple hours before that. Any ideas how can we prevent this from happening in the future? Any ideas what did cause this problem in the first place? Some error in hadoop hdfs? Any ideas how to investigate it further. We would appreciate any help. Full log link: https://www.dropbox.com/s/ynmd9nq8vp3tt98/hadoop-hadoop-namenode-ip-172-31-25-236.log.2016-04-16-23.gz?dl=0 2016-04-16 23:01:37,187 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Rolling edit logs 2016-04-16 23:01:37,187 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Ending log segmen t 1 2016-04-16 23:01:37,188 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Number of transac tions: 2004180 Total time for transactions(ms): 24698 Number of transactions batched in Syncs: 8993 Number of syncs: 1540948 SyncTimes(ms): 4583119 2016-04-16 23:01:37,190 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Finalizi ng edits file /hadoop/dfs-name/current/edits_inprogress_001 -> /hadoop/dfs-name/current/edits_001-2004180 2016-04-16 23:01:37,191 ERROR org.apache.hadoop.hdfs.server.common.Storage (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Error reported on storage directory Storage Directory /hadoop/dfs-name 2016-04-16 23:01:37,191 WARN org.apache.hadoop.hdfs.server.common.Storage (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): About to remove corresponding storage: /hadoop/dfs-name 2016-04-16 23:01:37,191 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Error: finalize log segment 1, 2004180 failed for (journal JournalAndStream(mgr=FileJournalManager(root=/hadoop/dfs-name), stream=null)) java.lang.IllegalStateException: Unable to finalize edits file /hadoop/dfs-name/current/edits_inprogress_001 at org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:137) at org.apache.hadoop.hdfs.server.namenode.JournalSet$3.apply(JournalSet.java:223) at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:359) at org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:218) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1168) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1097) at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1190) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:5376) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4356) at java.lang.Thread.run(Thread.java:745) 2016-04-16 23:01:37,191 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Disabling journal JournalAndStream(mgr=FileJournalManager(root=/hadoop/dfs-name), stream=null) 2016-04-16 23:01:37,191 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Error: finalize log segment 1, 2004180 failed for too many journals 2016-04-16 23:01:37,191 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Starting log segment at 2004181 2016-04-16 23:01:37,192 WARN org.apache.hadoop.hdfs.server.namenode.FileJournalManager (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@3cd78553): Unable to start log segment 2004181 at /hadoop/dfs-name/current/edits_inprogress_2004181: /hadoop/dfs-name/current/edits_inprogress_2004181 (No such file or directory)
Re: Balancing reads and writes
There are separate RPC queues for read and writes in 1.0+ (not sure about 0.98). You need to set sizes of these queues accordingly. -Vlad On Sat, Apr 16, 2016 at 4:23 PM, Kevin Bowling wrote: > Hi, > > Using OpenTSDB 2.2 with its "appends" feature, I see significant impact on > read performance when writes are happening. If a process injects a few > hundred thousand points in batch, the call queues on on the region servers > blow up and until they drain a new read request is basically blocked at the > end of the line. > > Any recommendations for keeping reads balanced vs writes? > > Regards, > Kevin >
Re: To Store Large Number of Video and Image files
>> have a project that needs to store large number of image and video files, >>the file size varies from 10MB to 10GB, the initial number of files will be >>0.1 billion and would grow over 1 billion, what will be the practical >>recommendations to store and view these files? >> Files are immutable? Write small files (less than 1 HDFS block) to large blob (combine them into single file), store large files directly to HDFS. Keep path index in HBase. If you need to delete files, mark them as deleted in HBase and run periodically GC job to perform real cleaning. -Vlad On Sat, Apr 16, 2016 at 7:35 PM, Ted Yu wrote: > There was HBASE-15370 for backport but it was decided not to backport the > feature. > > FYI > > On Sat, Apr 16, 2016 at 7:26 PM, Ascot Moss wrote: > > > Hi, > > > > About HBase-11339, > > "The size of the MOB data could not be very large, it better to keep the > > MOB size within 100KB and 10MB. Since MOB cells are written into the > > memstore before flushing, large MOB cells stress the memory in region > > servers." > > > > Can this be resolved if we provide more RAM in region servers? for > > instances, the servers in the cluster, each has 768GB RAM + 14 x 6T HDD. > > > > regards > > > > > > > > On Sun, Apr 17, 2016 at 9:56 AM, Ascot Moss > wrote: > > > > > Thanks Ted! > > > > > > Just visited HBASE-11339, its status is "resolved" however, it is for > > > "Fix Version : 2.0.0." > > > How to patch it to current HBase stable version (v1.1.4) ? > > > > > > About Fault Tolerance to DataCenter level, I am thinking HBase > > Replication > > > method to replicate HBase Tables to another cluster (backup one), is > > there > > > any real world reference about the replication performance, for > instances > > > if the bandwidth is 100MB/s? > > > > > > Regards > > > > > > > > > > > > > > > > > > > > >- > > > > > > > > > On Sun, Apr 17, 2016 at 9:40 AM, Ted Yu wrote: > > > > > >> Have you taken a look at HBASE-11339 (HBase MOB) ? > > >> > > >> Note: this feature does not handle 10GB objects well. Consider store > GB > > >> image on hdfs. > > >> > > >> Cheers > > >> > > >> On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss > > wrote: > > >> > > >> > Hi, > > >> > > > >> > I have a project that needs to store large number of image and video > > >> files, > > >> > the file size varies from 10MB to 10GB, the initial number of files > > >> will be > > >> > 0.1 billion and would grow over 1 billion, what will be the > practical > > >> > recommendations to store and view these files? > > >> > > > >> > > > >> > > > >> > #1 One cluster, store the HDFS URL in HBase and store the actual > file > > in > > >> > HDFS? (block_size as 128MB and replication factor as 3) > > >> > > > >> > > > >> > #2 One cluster, Store small files in HBase directly and use #1 for > > large > > >> > files? (block_size as 128MB and replication factor as 3) > > >> > > > >> > > > >> > #3 Multiple Hadoop/HBase clusters, each with different block_size > > >> settings? > > >> > > > >> > > > >> > e.g. cluster 1 (small): block_size as 128MB and replication > > factor > > >> as > > >> > 3, store all files in HBase if their file size is smaller 128MB > > >> > > > >> > cluster 2 (large): bigger block_size, say 4GB, > replication > > >> > factor as 3, store the HDFS URL in HBase and store the actual file > in > > >> HDFS > > >> > > > >> > > > >> > > > >> > #4 Use Hadoop Federation for large number of files? > > >> > > > >> > > > >> > About Fault Tolerance, need to consider four types of failures: > > driver, > > >> > host, rack, and datacenter failures. > > >> > > > >> > > > >> > Regards > > >> > > > >> > > > > > > > > >
Get startkeys and endkeys of blocks in HFile
Hi, I'm using HBase 0.98.7 and I want to get startkeys and endkeys of all blocks in a HFile. Is there any way to get them? Thanks, Van
Re: To Store Large Number of Video and Image files
There was HBASE-15370 for backport but it was decided not to backport the feature. FYI On Sat, Apr 16, 2016 at 7:26 PM, Ascot Moss wrote: > Hi, > > About HBase-11339, > "The size of the MOB data could not be very large, it better to keep the > MOB size within 100KB and 10MB. Since MOB cells are written into the > memstore before flushing, large MOB cells stress the memory in region > servers." > > Can this be resolved if we provide more RAM in region servers? for > instances, the servers in the cluster, each has 768GB RAM + 14 x 6T HDD. > > regards > > > > On Sun, Apr 17, 2016 at 9:56 AM, Ascot Moss wrote: > > > Thanks Ted! > > > > Just visited HBASE-11339, its status is "resolved" however, it is for > > "Fix Version : 2.0.0." > > How to patch it to current HBase stable version (v1.1.4) ? > > > > About Fault Tolerance to DataCenter level, I am thinking HBase > Replication > > method to replicate HBase Tables to another cluster (backup one), is > there > > any real world reference about the replication performance, for instances > > if the bandwidth is 100MB/s? > > > > Regards > > > > > > > > > > > > > >- > > > > > > On Sun, Apr 17, 2016 at 9:40 AM, Ted Yu wrote: > > > >> Have you taken a look at HBASE-11339 (HBase MOB) ? > >> > >> Note: this feature does not handle 10GB objects well. Consider store GB > >> image on hdfs. > >> > >> Cheers > >> > >> On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss > wrote: > >> > >> > Hi, > >> > > >> > I have a project that needs to store large number of image and video > >> files, > >> > the file size varies from 10MB to 10GB, the initial number of files > >> will be > >> > 0.1 billion and would grow over 1 billion, what will be the practical > >> > recommendations to store and view these files? > >> > > >> > > >> > > >> > #1 One cluster, store the HDFS URL in HBase and store the actual file > in > >> > HDFS? (block_size as 128MB and replication factor as 3) > >> > > >> > > >> > #2 One cluster, Store small files in HBase directly and use #1 for > large > >> > files? (block_size as 128MB and replication factor as 3) > >> > > >> > > >> > #3 Multiple Hadoop/HBase clusters, each with different block_size > >> settings? > >> > > >> > > >> > e.g. cluster 1 (small): block_size as 128MB and replication > factor > >> as > >> > 3, store all files in HBase if their file size is smaller 128MB > >> > > >> > cluster 2 (large): bigger block_size, say 4GB, replication > >> > factor as 3, store the HDFS URL in HBase and store the actual file in > >> HDFS > >> > > >> > > >> > > >> > #4 Use Hadoop Federation for large number of files? > >> > > >> > > >> > About Fault Tolerance, need to consider four types of failures: > driver, > >> > host, rack, and datacenter failures. > >> > > >> > > >> > Regards > >> > > >> > > > > >
Re: To Store Large Number of Video and Image files
Hi, About HBase-11339, "The size of the MOB data could not be very large, it better to keep the MOB size within 100KB and 10MB. Since MOB cells are written into the memstore before flushing, large MOB cells stress the memory in region servers." Can this be resolved if we provide more RAM in region servers? for instances, the servers in the cluster, each has 768GB RAM + 14 x 6T HDD. regards On Sun, Apr 17, 2016 at 9:56 AM, Ascot Moss wrote: > Thanks Ted! > > Just visited HBASE-11339, its status is "resolved" however, it is for > "Fix Version : 2.0.0." > How to patch it to current HBase stable version (v1.1.4) ? > > About Fault Tolerance to DataCenter level, I am thinking HBase Replication > method to replicate HBase Tables to another cluster (backup one), is there > any real world reference about the replication performance, for instances > if the bandwidth is 100MB/s? > > Regards > > > > > > >- > > > On Sun, Apr 17, 2016 at 9:40 AM, Ted Yu wrote: > >> Have you taken a look at HBASE-11339 (HBase MOB) ? >> >> Note: this feature does not handle 10GB objects well. Consider store GB >> image on hdfs. >> >> Cheers >> >> On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss wrote: >> >> > Hi, >> > >> > I have a project that needs to store large number of image and video >> files, >> > the file size varies from 10MB to 10GB, the initial number of files >> will be >> > 0.1 billion and would grow over 1 billion, what will be the practical >> > recommendations to store and view these files? >> > >> > >> > >> > #1 One cluster, store the HDFS URL in HBase and store the actual file in >> > HDFS? (block_size as 128MB and replication factor as 3) >> > >> > >> > #2 One cluster, Store small files in HBase directly and use #1 for large >> > files? (block_size as 128MB and replication factor as 3) >> > >> > >> > #3 Multiple Hadoop/HBase clusters, each with different block_size >> settings? >> > >> > >> > e.g. cluster 1 (small): block_size as 128MB and replication factor >> as >> > 3, store all files in HBase if their file size is smaller 128MB >> > >> > cluster 2 (large): bigger block_size, say 4GB, replication >> > factor as 3, store the HDFS URL in HBase and store the actual file in >> HDFS >> > >> > >> > >> > #4 Use Hadoop Federation for large number of files? >> > >> > >> > About Fault Tolerance, need to consider four types of failures: driver, >> > host, rack, and datacenter failures. >> > >> > >> > Regards >> > >> > >
Re: To Store Large Number of Video and Image files
Thanks Ted! Just visited HBASE-11339, its status is "resolved" however, it is for "Fix Version : 2.0.0." How to patch it to current HBase stable version (v1.1.4) ? About Fault Tolerance to DataCenter level, I am thinking HBase Replication method to replicate HBase Tables to another cluster (backup one), is there any real world reference about the replication performance, for instances if the bandwidth is 100MB/s? Regards - On Sun, Apr 17, 2016 at 9:40 AM, Ted Yu wrote: > Have you taken a look at HBASE-11339 (HBase MOB) ? > > Note: this feature does not handle 10GB objects well. Consider store GB > image on hdfs. > > Cheers > > On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss wrote: > > > Hi, > > > > I have a project that needs to store large number of image and video > files, > > the file size varies from 10MB to 10GB, the initial number of files will > be > > 0.1 billion and would grow over 1 billion, what will be the practical > > recommendations to store and view these files? > > > > > > > > #1 One cluster, store the HDFS URL in HBase and store the actual file in > > HDFS? (block_size as 128MB and replication factor as 3) > > > > > > #2 One cluster, Store small files in HBase directly and use #1 for large > > files? (block_size as 128MB and replication factor as 3) > > > > > > #3 Multiple Hadoop/HBase clusters, each with different block_size > settings? > > > > > > e.g. cluster 1 (small): block_size as 128MB and replication factor > as > > 3, store all files in HBase if their file size is smaller 128MB > > > > cluster 2 (large): bigger block_size, say 4GB, replication > > factor as 3, store the HDFS URL in HBase and store the actual file in > HDFS > > > > > > > > #4 Use Hadoop Federation for large number of files? > > > > > > About Fault Tolerance, need to consider four types of failures: driver, > > host, rack, and datacenter failures. > > > > > > Regards > > >
Re: To Store Large Number of Video and Image files
Have you taken a look at HBASE-11339 (HBase MOB) ? Note: this feature does not handle 10GB objects well. Consider store GB image on hdfs. Cheers On Sat, Apr 16, 2016 at 6:21 PM, Ascot Moss wrote: > Hi, > > I have a project that needs to store large number of image and video files, > the file size varies from 10MB to 10GB, the initial number of files will be > 0.1 billion and would grow over 1 billion, what will be the practical > recommendations to store and view these files? > > > > #1 One cluster, store the HDFS URL in HBase and store the actual file in > HDFS? (block_size as 128MB and replication factor as 3) > > > #2 One cluster, Store small files in HBase directly and use #1 for large > files? (block_size as 128MB and replication factor as 3) > > > #3 Multiple Hadoop/HBase clusters, each with different block_size settings? > > > e.g. cluster 1 (small): block_size as 128MB and replication factor as > 3, store all files in HBase if their file size is smaller 128MB > > cluster 2 (large): bigger block_size, say 4GB, replication > factor as 3, store the HDFS URL in HBase and store the actual file in HDFS > > > > #4 Use Hadoop Federation for large number of files? > > > About Fault Tolerance, need to consider four types of failures: driver, > host, rack, and datacenter failures. > > > Regards >
Hbase v1 vs 0.98x
Hi, Is there any document about what's new in HBase v1.x vs 9.98.x? Regards
To Store Large Number of Video and Image files
Hi, I have a project that needs to store large number of image and video files, the file size varies from 10MB to 10GB, the initial number of files will be 0.1 billion and would grow over 1 billion, what will be the practical recommendations to store and view these files? #1 One cluster, store the HDFS URL in HBase and store the actual file in HDFS? (block_size as 128MB and replication factor as 3) #2 One cluster, Store small files in HBase directly and use #1 for large files? (block_size as 128MB and replication factor as 3) #3 Multiple Hadoop/HBase clusters, each with different block_size settings? e.g. cluster 1 (small): block_size as 128MB and replication factor as 3, store all files in HBase if their file size is smaller 128MB cluster 2 (large): bigger block_size, say 4GB, replication factor as 3, store the HDFS URL in HBase and store the actual file in HDFS #4 Use Hadoop Federation for large number of files? About Fault Tolerance, need to consider four types of failures: driver, host, rack, and datacenter failures. Regards
Balancing reads and writes
Hi, Using OpenTSDB 2.2 with its "appends" feature, I see significant impact on read performance when writes are happening. If a process injects a few hundred thousand points in batch, the call queues on on the region servers blow up and until they drain a new read request is basically blocked at the end of the line. Any recommendations for keeping reads balanced vs writes? Regards, Kevin
Re: Read HFile from Local file system for studying and testing
Hi there,
I fixed the error by trying different maven dependencies and that solved
the problem in the end.
First, you should not need and, should NOT include the hadoop-core jar.
Here is all the code you need in the end to read HFile from a localfile
system.
And I tested this on a brand new Ubuntu Eclipse installation (virtualbox)
and it worked :)
Java
public class App {
public static void main(String[] args) throws IOException {
Path p = new Path("/home/datafireball/Desktop/testHFile");
Configuration conf = new Configuration();
System.out.println(conf);
FileSystem fs = FileSystem.get(conf);
// FileSystem localfs = FileSystem.newInstanceLocal(conf);
// FileSystem localfs = FileSystem.getLocal(conf).getRawFileSystem();
// FileSystem fs = p.getFileSystem(conf);
// RawLocalFileSystem localfs = new RawLocalFileSystem();
// System.out.println(localfs.toString());
HFile.Reader reader = HFile.createReader(fs, p, new
CacheConfig(conf), conf);
reader.loadFileInfo();
HFileScanner scanner = reader.getScanner(false, false);
scanner.seekTo();
int count = 0;
do {
System.out.println(scanner.getKeyString());
count++;
} while (scanner.next());
System.out.println(count);
reader.close();
}
}
For later reference, if you include hadoop-core, here is the error message,
i really don't know why but here it is.
16/04/16 16:57:03 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Exception in thread "main" java.lang.UnsupportedOperationException:
Not implemented by the DistributedFileSystem FileSystem implementation
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at com.datafireball.readHFile.App.main(App.java:29)
If you include a hadoop-common that is too old, here is the error message
that complaining about missing class:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration$DeprecationDelta
at
org.apache.hadoop.mapreduce.util.ConfigUtil.addDeprecatedKeys(ConfigUtil.java:54)
at
org.apache.hadoop.mapreduce.util.ConfigUtil.loadResources(ConfigUtil.java:42)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:119)
at java.lang.Class.forName0(Native Method)
Best regards,
Bin
On Fri, Apr 15, 2016 at 10:11 PM, Bin Wang wrote:
Hi St.Ack,
>
> localfs = FileSystem.get(conf) triggers another exception and it failed
> right at that step, let alone print out localfs.
>
> 16/04/15 21:54:04 WARN util.NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> Exception in thread "main" java.lang.UnsupportedOperationException: Not
>> implemented by the DistributedFileSystem FileSystem implementation
>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:214)
>> at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2559)
>> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(
>> FileSystem.java:2569)
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2586)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
>> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625
>> )
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
>> at myhbase.MyHbase.main(MyHbase.java:17)
>
> I saw a good stackoverflow
>
> question here regarding the same error however, I am simply trying to use
> localfs instead hadoop so I don't think it is that relevant. Does that mean
> I cannot do anything without hadoop installed on my laptop?
>
> And for the record, I was indeed using an empty configuration simply
> because I donno what other configurations i need to set up.
>
> For anyone who wants to follow along, I also attached the hfile that I was
> working with, so you can try to download and read from localfile system.
>
> (It contains nothing but a few dummy records following an example in the
> book: Hbase the definitive guide)
>
> Meanwhile, I will take a
Re: Re: ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper.
Can you verify that hbase is running by logging onto master node and check the Java processes ? If master is running, can you do a listing of the zookeeper znode (using zkCli) and pastebin the result ? Thanks On Sat, Apr 16, 2016 at 8:14 AM, Eric Gao wrote: > Yes,I have seen your reply.Thanks very much for your kindness. > > This is my hbase-site.xml: > > > hbase.rootdir > hdfs://master:9000/hbase/data > > > hbase.cluster.distributed > true > > > > zookeeper.znode.parent > /hbase > Root ZNode for HBase in ZooKeeper. All of HBase's > ZooKeeper > files that are configured with a relative path will go under this > node. > By default, all of HBase's ZooKeeper file path are configured with a > relative path, so they will all go under this directory unless > changed. > > > > > > hbase.zookeeper.quorum > master,slave1,slave2 > Comma separated list of servers in the ZooKeeper Quorum. For > example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By > default this is set to localhost for local and pseudo-distributed modes of > operation. For a fully-distributed setup, this should be set to a full list > of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh > this is the list of servers which we will start/stop ZooKeeper on. > > > > hbase.zookeeper.property.dataDir > /opt/zookeeper/data > Property from ZooKeeper's config zoo.cfg. The directory where > the snapshot is stored. > > > > This is my hbase-env.sh: > > > [root@master ~]# cat /opt/hbase/conf/hbase-env.sh > # > #/** > # * Licensed to the Apache Software Foundation (ASF) under one > # * or more contributor license agreements. See the NOTICE file > # * distributed with this work for additional information > # * regarding copyright ownership. The ASF licenses this file > # * to you under the Apache License, Version 2.0 (the > # * "License"); you may not use this file except in compliance > # * with the License. You may obtain a copy of the License at > # * > # * http://www.apache.org/licenses/LICENSE-2.0 > # * > # * Unless required by applicable law or agreed to in writing, software > # * distributed under the License is distributed on an "AS IS" BASIS, > # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > implied. > # * See the License for the specific language governing permissions and > # * limitations under the License. > # */ > > # Set environment variables here. > > # This script sets variables multiple times over the course of starting an > hbase process, > # so try to keep things idempotent unless you want to take an even deeper > look > # into the startup scripts (bin/hbase, etc.) > > # The java implementation to use. Java 1.7+ required. > export JAVA_HOME=/usr > > # Extra Java CLASSPATH elements. Optional. > export HBASE_CLASSPATH=/opt/hadoop > > # The maximum amount of heap to use. Default is left to JVM default. > # export HBASE_HEAPSIZE=1G > > # Uncomment below if you intend to use off heap cache. For example, to > allocate 8G of > # offheap, set the value to "8G". > # export HBASE_OFFHEAPSIZE=1G > > # Extra Java runtime options. > # Below are what we set by default. May only work with SUN JVM. > # For more on why as well as other possible settings, > # see http://wiki.apache.org/hadoop/PerformanceTuning > export HBASE_OPTS="-XX:+UseConcMarkSweepGC" > > # Uncomment one of the below three options to enable java garbage > collection logging for the server-side processes. > > # This enables basic gc logging to the .out file. > # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps" > > # This enables basic gc logging to its own file. > # If FILE-PATH is not replaced, the log file(.gc) would still be generated > in the HBASE_LOG_DIR . > # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -Xloggc:" > > # This enables basic GC logging to its own file with automatic log > rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. > # If FILE-PATH is not replaced, the log file(.gc) would still be generated > in the HBASE_LOG_DIR . > # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" > > # Uncomment one of the below three options to enable java garbage > collection logging for the client processes. > > # This enables basic gc logging to the .out file. > # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps" > > # This enables basic gc logging to its own file. > # If FILE-PATH is not replaced, the log file(.gc) would still be generated > in the HBASE_LOG_DIR . > # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -Xloggc:" > > # This enables basic GC logging to its own file with automatic log > rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. > # If FILE-PATH is not replaced, the log file(.gc) would still be generated >
Re: Re: ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper.
Yes,I have seen your reply.Thanks very much for your kindness. This is my hbase-site.xml: hbase.rootdir hdfs://master:9000/hbase/data hbase.cluster.distributed true zookeeper.znode.parent /hbase Root ZNode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed. hbase.zookeeper.quorum master,slave1,slave2 Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". By default this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh this is the list of servers which we will start/stop ZooKeeper on. hbase.zookeeper.property.dataDir /opt/zookeeper/data Property from ZooKeeper's config zoo.cfg. The directory where the snapshot is stored. This is my hbase-env.sh: [root@master ~]# cat /opt/hbase/conf/hbase-env.sh # #/** # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information # * regarding copyright ownership. The ASF licenses this file # * to you under the Apache License, Version 2.0 (the # * "License"); you may not use this file except in compliance # * with the License. You may obtain a copy of the License at # * # * http://www.apache.org/licenses/LICENSE-2.0 # * # * Unless required by applicable law or agreed to in writing, software # * distributed under the License is distributed on an "AS IS" BASIS, # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # * See the License for the specific language governing permissions and # * limitations under the License. # */ # Set environment variables here. # This script sets variables multiple times over the course of starting an hbase process, # so try to keep things idempotent unless you want to take an even deeper look # into the startup scripts (bin/hbase, etc.) # The java implementation to use. Java 1.7+ required. export JAVA_HOME=/usr # Extra Java CLASSPATH elements. Optional. export HBASE_CLASSPATH=/opt/hadoop # The maximum amount of heap to use. Default is left to JVM default. # export HBASE_HEAPSIZE=1G # Uncomment below if you intend to use off heap cache. For example, to allocate 8G of # offheap, set the value to "8G". # export HBASE_OFFHEAPSIZE=1G # Extra Java runtime options. # Below are what we set by default. May only work with SUN JVM. # For more on why as well as other possible settings, # see http://wiki.apache.org/hadoop/PerformanceTuning export HBASE_OPTS="-XX:+UseConcMarkSweepGC" # Uncomment one of the below three options to enable java garbage collection logging for the server-side processes. # This enables basic gc logging to the .out file. # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps" # This enables basic gc logging to its own file. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:" # This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" # Uncomment one of the below three options to enable java garbage collection logging for the client processes. # This enables basic gc logging to the .out file. # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps" # This enables basic gc logging to its own file. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:" # This enables basic GC logging to its own file with automatic log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. # If FILE-PATH is not replaced, the log file(.gc) would still be generated in the HBASE_LOG_DIR . # export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M" # See the package documentation for org.apache.hadoop.hbase.io.hfile for other configurations # needed setting up off-heap block caching. # Uncomment and adjust to enable JMX exporting # See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to c
Re: ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper.
Have you seen my reply ? http://search-hadoop.com/m/q3RTtJHewi1jOgc21 The actual value for zookeeper.znode.parent could be /hbase-secure (just an example). Make sure the correct hbase-site,xml is in classpath for hbase shell. On Sat, Apr 16, 2016 at 7:53 AM, Eric Gao wrote: > Dear expert, > I have encountered a problem,when I run hbase cmd :status it shows: > > hbase(main):001:0> status > 2016-04-16 13:03:02,333 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > 2016-04-16 13:03:02,538 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > 2016-04-16 13:03:02,843 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > 2016-04-16 13:03:03,348 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > 2016-04-16 13:03:04,355 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > 2016-04-16 13:03:06,369 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > 2016-04-16 13:03:10,414 ERROR [main] > client.ConnectionManager$HConnectionImplementation: The node /hbase is not > in ZooKeeper. It should have been written by the master. Check the value > configured in 'zookeeper.znode.parent'. There could be a mismatch with the > one configured in the master. > > How can I solve the problem? > Thanks very much > > > > Eric Gao > Keep on going never give up. > Blog: > http://gaoqiang.blog.chinaunix.net/ > http://gaoqiangdba.blog.163.com/ > > >
ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper.
Dear expert, I have encountered a problem,when I run hbase cmd :status it shows: hbase(main):001:0> status 2016-04-16 13:03:02,333 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2016-04-16 13:03:02,538 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2016-04-16 13:03:02,843 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2016-04-16 13:03:03,348 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2016-04-16 13:03:04,355 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2016-04-16 13:03:06,369 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. 2016-04-16 13:03:10,414 ERROR [main] client.ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master. How can I solve the problem? Thanks very much Eric Gao Keep on going never give up. Blog: http://gaoqiang.blog.chinaunix.net/ http://gaoqiangdba.blog.163.com/
