[
https://issues.apache.org/jira/browse/HDFS-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201118#comment-16201118
]
Chen Liang commented on HDFS-12504:
-----------------------------------
Thanks [~yuanbo] for working on this! v001 patch looks pretty good to me. Just
some minor comments:
1. {{void accept(T item) throws IOException;}}, rename accept to something like
batchConsume?
2. "This class is used to batch operate kv" ==> "This class is used to batch
kv operations"
3. Change the log "Insert to sql container db, for container" to something like
"Insert to sql batch for container", and add some log to {{batchIterateStore}}
such that we can see the progress from log.
Also it would be ideal if we can have some simple benchmark results to see the
performance improvement, I will be looking into this too.
> Ozone: Improve SQLCLI performance
> ---------------------------------
>
> Key: HDFS-12504
> URL: https://issues.apache.org/jira/browse/HDFS-12504
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Reporter: Weiwei Yang
> Assignee: Yuanbo Liu
> Labels: performance
> Attachments: HDFS-12504-HDFS-7240.001.patch
>
>
> In my test, my {{ksm.db}} has *3017660* entries with total size of *128mb*,
> SQLCLI tool runs over *2 hours* but still not finish exporting the DB. This
> is because it iterates each entry and inserts that to another sqllite DB
> file, which is not efficient. We need to improve this to be running more
> efficiently on large DB files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
