[jira] [Updated] (ACCUMULO-2889) Batch metadata table updates for new walogs
[ https://issues.apache.org/jira/browse/ACCUMULO-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Park updated ACCUMULO-2889: Attachment: accumulo-2889_withoutpatch.png accumulo-2889-withpatch.png ACCUMULO-2889.1.patch start-ingest.sh batch_perf_test.sh run_all.sh Results from performance tests: Test design: - Run continuous ingest with 4 ingesters each ingesting 25million entries and then measure time until completion - We varied # of minor compactors and tablets per server (in retrospect, # of minor compactors didn't really matter in these tests, it may have been better to vary # of clients). - Each trial was run 3x and the average was taken. Tests were run on a single node (24 logical cores, 64 GB RAM, 8 drives) ||minc||tablets/server||w/o patch(ms)||w/ patch(ms)||ratio|| |4|32|269790.33|257537.33|0.95458325| |12|32|271124.33|255952|0.94403922| |12|320|355962.67|323737|0.90946896| |24|32|268709|261362.67|0.97266065| |24|320|355182.33|324308.67|0.91307659| I'll try to run this on a multi-node cluster if I can get around to it. > Batch metadata table updates for new walogs > --- > > Key: ACCUMULO-2889 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2889 > Project: Accumulo > Issue Type: Improvement >Affects Versions: 1.5.1, 1.6.0 >Reporter: Jonathan Park >Assignee: Jonathan Park > Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch, > accumulo-2889-withpatch.png, accumulo-2889_withoutpatch.png, > batch_perf_test.sh, run_all.sh, start-ingest.sh > > > Currently, when we update the Metadata table with new loggers, we will update > the metadata for each tablet serially. We could optimize this to instead use > a batchwriter to send all metadata updates for all tablets in a batch. > A few special cases include: > - What if the !METADATA tablet was included in the batch? > - What about the root tablet? > Benefit: > In one of our clusters, we're experiencing particularly slow HDFS operations > leading to large oscillations in ingest performance. We haven't isolated the > cause in HDFS but when we profile the tservers, we noticed that they were > waiting for metadata table operations to complete. This would target the > waiting. > Potential downsides: > Given the existing locking scheme, it looks like we may have to lock a tablet > for slightly longer (we'll lock for the duration of the batch). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (ACCUMULO-2955) Master logs extra warnings about bad table name
[
https://issues.apache.org/jira/browse/ACCUMULO-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046906#comment-14046906
]
Mike Drob commented on ACCUMULO-2955:
-
Yes, you understood perfectly.
> Master logs extra warnings about bad table name
> ---
>
> Key: ACCUMULO-2955
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2955
> Project: Accumulo
> Issue Type: Bug
> Components: master
>Affects Versions: 1.6.0
>Reporter: Mike Drob
>Priority: Minor
> Labels: logging
> Fix For: 1.6.1, 1.7.0
>
>
> {noformat}
> root@accumulo> createtable a.b
> 2014-06-27 07:34:01,338 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Cannot create table in
> non-existent namespace
> root@accumulo> createtable a
> root@accumulo a> renametable a a.b
> 2014-06-27 07:34:45,313 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Cannot move tables to a
> new namespace by renaming. The namespace for a does not match a.b
> root@accumulo a> renametable a .a.b
> 2014-06-27 07:40:09,238 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Table names must only
> contain word characters (letters, digits, and underscores): .a.b
> {noformat}
> The second and third ERROR also shows up on the monitor page as a WARN coming
> from the master. They should behave more like the first one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (ACCUMULO-2955) Master logs extra warnings about bad table name
[
https://issues.apache.org/jira/browse/ACCUMULO-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs updated ACCUMULO-2955:
Fix Version/s: 1.7.0
> Master logs extra warnings about bad table name
> ---
>
> Key: ACCUMULO-2955
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2955
> Project: Accumulo
> Issue Type: Bug
> Components: master
>Affects Versions: 1.6.0
>Reporter: Mike Drob
>Priority: Minor
> Labels: logging
> Fix For: 1.6.1, 1.7.0
>
>
> {noformat}
> root@accumulo> createtable a.b
> 2014-06-27 07:34:01,338 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Cannot create table in
> non-existent namespace
> root@accumulo> createtable a
> root@accumulo a> renametable a a.b
> 2014-06-27 07:34:45,313 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Cannot move tables to a
> new namespace by renaming. The namespace for a does not match a.b
> root@accumulo a> renametable a .a.b
> 2014-06-27 07:40:09,238 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Table names must only
> contain word characters (letters, digits, and underscores): .a.b
> {noformat}
> The second and third ERROR also shows up on the monitor page as a WARN coming
> from the master. They should behave more like the first one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (ACCUMULO-2955) Master logs extra warnings about bad table name
[
https://issues.apache.org/jira/browse/ACCUMULO-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046850#comment-14046850
]
Christopher Tubbs commented on ACCUMULO-2955:
-
I'm a bit confused here. Are you suggesting the first does not have a warning
coming from the master, and the other two do, and that we should remove all
server-side warnings for these?
> Master logs extra warnings about bad table name
> ---
>
> Key: ACCUMULO-2955
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2955
> Project: Accumulo
> Issue Type: Bug
> Components: master
>Affects Versions: 1.6.0
>Reporter: Mike Drob
>Priority: Minor
> Labels: logging
> Fix For: 1.6.1, 1.7.0
>
>
> {noformat}
> root@accumulo> createtable a.b
> 2014-06-27 07:34:01,338 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Cannot create table in
> non-existent namespace
> root@accumulo> createtable a
> root@accumulo a> renametable a a.b
> 2014-06-27 07:34:45,313 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Cannot move tables to a
> new namespace by renaming. The namespace for a does not match a.b
> root@accumulo a> renametable a .a.b
> 2014-06-27 07:40:09,238 [shell.Shell] ERROR:
> org.apache.accumulo.core.client.AccumuloException: Table names must only
> contain word characters (letters, digits, and underscores): .a.b
> {noformat}
> The second and third ERROR also shows up on the monitor page as a WARN coming
> from the master. They should behave more like the first one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
