Esteban Gutierrez created HBASE-14060:
-
Summary: Add FuzzyRowFilter to ParseFilter
Key: HBASE-14060
URL: https://issues.apache.org/jira/browse/HBASE-14060
Project: HBase
Issue Type: Bug
Matteo, thanks for reminding me about verification stage - this is it.
Master, verifying 60K files and regions in a live cluster ... this can
explain 30 minutes.
If even one region splits during a snapshot, snapshot will fail. Its
amazing, that these guys are able to to finish snapshot at all.
-V
yeah, something along that line. but I doubt the problem is RS side.
or the communication between the master and RSs.
in theory the problem may be the verification step where the master
is checking the snapshot. I was just trying to figure out where he is
spending the time
and that "30 minutes to
Matteo, there should be some explanation for 30 min flash_skip snapshot. I
think its should be somewhere in NN/Hdfs. This is a huge cluster and NN
load is extreme, it is probably does not scale well with # DNs and #files
per directory. I presume that NN performance on file operations degrades
when
Manifest per Region, not family.
we couldn't send them back to the master/table to keep compatibility.
60k region on 1200 RS are ~50 manifest per RS that alone should not take
30sec
On Fri, Jul 10, 2015 at 5:21 PM, Vladimir Rodionov
wrote:
> OK, even with 1 manifest file per region (per column
OK, even with 1 manifest file per region (per column family?) - 60K X 4 =
240,000 new files
8000 per minute, 135 per second. That is probably NN limit.
Anyway, the root cause is the same as with reference files during region
split:
HDFS does not do well on file create/open/close/delete.
-Vlad
O
@Vladimir there is no hfile link creation on snapshot. we create 1 manifest
per region
Matteo
On Fri, Jul 10, 2015 at 5:06 PM, Vladimir Rodionov
wrote:
> Being not very familiar with snapshot code, I could speculate only on where
> most time is spent ...
>
> In creating 60K x 4 x K (K is avera
Being not very familiar with snapshot code, I could speculate only on where
most time is spent ...
In creating 60K x 4 x K (K is average # of store files per region) small
HFileLInks? This can be very large # of files.
-Vlad
On Fri, Jul 10, 2015 at 4:57 PM, Matteo Bertozzi
wrote:
> the total
the total time taken by a snapshot should be bounded by the slowest machine.
we send a notification to each RS and each RS execute the snapshot
operation for each region.
can you track down what is slow in your case?
clone has to create a reference for each file, and that is a master
operation, an
Hi Rahul,
Have you identified with it takes those 30 minutes? Is the table balances
correctly across the servers? Form the logs, are you able to identify what
takes that much time?
JM
2015-07-10 18:46 GMT-04:00 rahul gidwani :
> Hi Matteo,
>
> We do SKIP_FLUSH. We have 1200+ regionservers with
Hi Matteo,
We do SKIP_FLUSH. We have 1200+ regionservers with a single table with 60k
regions and 4 column families. It takes around 30 minutes to snapshot this
table using manifests compared to just seconds doing this with hdfs.
Cloning this table takes considerably longer.
For cases where som
Esteban Gutierrez created HBASE-14059:
-
Summary: We should add a RS to the dead servers list if admin
calls fail more than a threshold
Key: HBASE-14059
URL: https://issues.apache.org/jira/browse/HBASE-14059
Abhilash created HBASE-14058:
Summary: Stabilize heap memory tuner
Key: HBASE-14058
URL: https://issues.apache.org/jira/browse/HBASE-14058
Project: HBase
Issue Type: Improvement
Compone
[
https://issues.apache.org/jira/browse/HBASE-14056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-14056.
---
Resolution: Invalid
Please do not file an issue and ask a question on the mailing list. Resolving
as invalid
t;Reporter: Liu Junhong
>Assignee: Liu Junhong
> Fix For: 2.0.0, 0.98.14, 1.3.0
>
> Attachments: 13897-v2.txt,
> HBASE-13897-0.98-20150710-suitable_for_hadoop1.patch,
> HBASE-13897-0.98-20150710.patch, HBASE-13897-0.98.patch,
> HBASE-1
Wellington Chevreuil created HBASE-14057:
Summary: HBase shell user_permission should list super users
defined on hbase-site.xml
Key: HBASE-14057
URL: https://issues.apache.org/jira/browse/HBASE-14057
congcong created HBASE-14056:
Summary: regionserver.HRegion: Failed getting lock in batch put
Key: HBASE-14056
URL: https://issues.apache.org/jira/browse/HBASE-14056
Project: HBase
Issue Type: Bu
[
https://issues.apache.org/jira/browse/HBASE-13387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John resolved HBASE-13387.
Resolution: Fixed
Pushed. Thanks for the quick review Ram.
> Add ByteBufferedCell an exten
[
https://issues.apache.org/jira/browse/HBASE-13387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John reopened HBASE-13387:
My bad.. Need to apply an addendum.
ByteBufferedCell initially was made as Interface and so a
19 matches
Mail list logo
