[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067553#comment-13067553 ] stack commented on HBASE-4010: -- @Ted What lars said. We should close hbase-3503 because this issue fixes it? HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067288#comment-13067288 ] Lars George commented on HBASE-4010: Not sure if this was discussed, but isn't this work what was described in HBASE-3503? Should we close/link this issue? HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13056856#comment-13056856 ] Hudson commented on HBASE-4010: --- Integrated in HBase-TRUNK #1995 (See [https://builds.apache.org/job/HBase-TRUNK/1995/]) HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[
https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053966#comment-13053966
]
Ted Yu commented on HBASE-4010:
---
I did some initial performance test on a cluster with 7 region servers (4GB
heap).
OS is:
{noformat}
Linux ciq.com 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64
x86_64 GNU/Linux
{noformat}
Creating table with 1000 regions took 37 seconds pre-4010
It took 32 seconds with 4010-0.90.txt applied.
HMaster.createTable could be heavily optimized
--
Key: HBASE-4010
URL: https://issues.apache.org/jira/browse/HBASE-4010
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Fix For: 0.92.0
Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt
Looking at the createTable method in HMaster (the one that's private), we
seem to be very inefficient:
- We set the enabled flag for the table for every region (should be done
only once).
- Every time we create a new region we create a new HLog and then close it
(reuse one instead or see if it's really necessary).
- We do one RPC to .META. per region (we should batch put).
This should provide drastic speedups even for those creating tables with just
50 regions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053972#comment-13053972 ] Jean-Daniel Cryans commented on HBASE-4010: --- Can you tell where most of the time is spent? Should we create the HRegions in parallel? HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053983#comment-13053983 ] Ted Yu commented on HBASE-4010: --- I did profile HMaster. The numbers were logged at client side. I have been thinking about using executor service to make HRegion creation more parallel. But I think that would help if there're many many regions when creating the table. I am not sure how common that use case is. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053991#comment-13053991 ] Ted Yu commented on HBASE-4010: --- Do you know how long re-creating a table with a few hundred regions took ? From my test result, the duration can be shortened 13%. If that's still unsatisfactory, shall we address it in another JIRA ? HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053987#comment-13053987 ] Jean-Daniel Cryans commented on HBASE-4010: --- We have a use case where were are re-creating a table with a few hundred regions as part of a MR job, it's currently taking a lot of time so anything that helps is welcome. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054042#comment-13054042 ] Jean-Daniel Cryans commented on HBASE-4010: --- Yes another jira. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054109#comment-13054109 ] Ted Yu commented on HBASE-4010: --- I apologize for presenting incorrect performance data. The 37 second was for a 0.90.3 cluster which had run for a while. The 32 second was for a 0.90.4 cluster started fresh without 4010-0.90.txt applied - I copied the correct hbase-0.90.4-SNAPSHOT.jar but forgot to rename it. So I started the cluster with the hbase-0.90.4-SNAPSHOT.jar which has 4010-0.90.txt applied I created two tables with 1000 regions. The first took 16 seconds and the second took 15 seconds. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053076#comment-13053076 ] Ted Yu commented on HBASE-4010: --- I ran test suite. The failed tests were: TestFSTableDescriptors and TestMergeTable TestTableMapReduce hung. These were consistent with results of build 1980. Committed to TRUNK. Thanks Stack for the review, J-D for the initiative. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053514#comment-13053514 ] Andrew Purtell commented on HBASE-4010: --- Shouldn't this be committed to 0.90 branch too? HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053517#comment-13053517 ] Jean-Daniel Cryans commented on HBASE-4010: --- I think it could... it'd be nice also to have a before/after comparison when trying to create a table with thousands of regions. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053614#comment-13053614 ] Ted Yu commented on HBASE-4010: --- bq. when trying to create a table with thousands of regions. We should do that. But the createTable() call would timeout. Shall we consider HBASE-3904 where I made createTable() synchronous :-) HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053649#comment-13053649 ] Jean-Daniel Cryans commented on HBASE-4010: --- That timeout is configurable IIRC. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-0.90.txt, 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052852#comment-13052852 ] Jean-Daniel Cryans commented on HBASE-4010: --- On the v2 patch: - It seems the new createTable method is mostly copy pasted from the other, refactor? - Same in MetaEditor. - On the batch Put... I would almost prefer not doing multiple batches since in 0.92 we don't even carry the HTD in the HRI meaning that the edits are really small. Is it passing all unit tests (that are currently green)? HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052861#comment-13052861 ] Ted Yu commented on HBASE-4010: --- For item 1, I think you meant createHRegion(). Some variables are computed before HRegion.newHRegion() is called. So I didn't refactor the method. I can refactor addRegionsToMeta() and addRegionToMeta() For batch Put, shall we choose a larger batch size (e.g. 100). Some users would create table with many regions (e.g. 70k). But I don't have strong opinion on this item. I am running unit tests. TestMasterFailover hung in the suite. When I ran it manually, it passed. I started another round of tests. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[
https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052919#comment-13052919
]
Ted Yu commented on HBASE-4010:
---
Interesting, v3 was customized to latest trunk.
TestAdmin used to pass for v2.
Now the tests were fine but JVM didn't shutdown:
{code}
main prio=5 tid=103000800 nid=0x100601000 in Object.wait() [10060]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1210)
- locked 7a6041b98 (a org.apache.hadoop.fs.FileSystem$ClientFinalizer)
at java.lang.Thread.join(Thread.java:1263)
at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
at
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked 7faf8ead8 (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:90)
{code}
HMaster.createTable could be heavily optimized
--
Key: HBASE-4010
URL: https://issues.apache.org/jira/browse/HBASE-4010
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Fix For: 0.92.0
Attachments: 4010-v2.txt, 4010-v3.txt
Looking at the createTable method in HMaster (the one that's private), we
seem to be very inefficient:
- We set the enabled flag for the table for every region (should be done
only once).
- Every time we create a new region we create a new HLog and then close it
(reuse one instead or see if it's really necessary).
- We do one RPC to .META. per region (we should batch put).
This should provide drastic speedups even for those creating tables with just
50 regions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053033#comment-13053033 ] stack commented on HBASE-4010: -- Seems to be a copy/paste prob. with javadoc (You copy from another method but do not amend this new methods javadoc -- or amend the old if this is the method used by 'bootstrap code in the HMaster constructor' Why does this new method in HRegion pass in an hlog instance but not use it? Should we make it so you can pass a null hlog so when doing these bulk creates, we don't bother with the hlog open/close/edit especially when its not necessary. In fact this new method createHRegion looks exactly same as existing method? The addRegionsToMeta looks like nice addition. Don't hardcode this: +final int batchSize = 100; Make it final int batchSize = this.conf.getInt(hbase.master.tablecreate.batchsize, 100); As soon as you commit it, someone else will want to change it (smile). Patch looks good otherwise. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010-v3.txt, 4010-v4.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053047#comment-13053047 ] stack commented on HBASE-4010: -- I was thinking of passing the hlog into HRegion constructor and just having HRegion deal with a null hlog but not important. We can do that in another issue. This patch looks good to me. +1 on commit if tests pass. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4010-v2.txt, 4010-v3.txt, 4010-v5.txt Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4010) HMaster.createTable could be heavily optimized
[ https://issues.apache.org/jira/browse/HBASE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13052263#comment-13052263 ] Ted Yu commented on HBASE-4010: --- HLog depends on regionDir (computed from HRegionInfo) and HRegionInfo itself. The HLog instance is used by the new HRegion. HMaster.createTable could be heavily optimized -- Key: HBASE-4010 URL: https://issues.apache.org/jira/browse/HBASE-4010 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Fix For: 0.92.0 Looking at the createTable method in HMaster (the one that's private), we seem to be very inefficient: - We set the enabled flag for the table for every region (should be done only once). - Every time we create a new region we create a new HLog and then close it (reuse one instead or see if it's really necessary). - We do one RPC to .META. per region (we should batch put). This should provide drastic speedups even for those creating tables with just 50 regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
