[jira] [Comment Edited] (SOLR-11200) provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle

2017-08-09 Thread Nawab Zada Asad iqbal (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120198#comment-16120198
 ] 

Nawab Zada Asad iqbal edited comment on SOLR-11200 at 8/9/17 4:53 PM:
--

[~sarkaramr...@gmail.com] 
I just reviewed your patch, it looks good. For the name, what about 
`enableAutoIOThrottle` ? 

I will test it now and report if I see any issues. 

PS: I edited it after realizing that the long config name is initiating from 
LUCENE code. My previous suggestion was `enableIOThrottle`


was (Author: niqbal):
[~sarkaramr...@gmail.com] 
I just reviewed your patch, it looks good. For the name, what about 
`enableIOThrottle` ? The word `Auto` does not seem necessary. 

I will test it now and report if I see any issues. 

> provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle
> ---
>
> Key: SOLR-11200
> URL: https://issues.apache.org/jira/browse/SOLR-11200
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Nawab Zada Asad iqbal
>Priority: Minor
> Attachments: SOLR-11200.patch, SOLR-11200.patch
>
>
> This config can be useful while bulk indexing. Lucene introduced it 
> https://issues.apache.org/jira/browse/LUCENE-6119 . 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-11200) provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle

2017-08-07 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115348#comment-16115348
 ] 

Amrit Sarkar edited comment on SOLR-11200 at 8/7/17 4:10 PM:
-

Ah! I don't think this will serve our purpose in bulk indexing, logs :: 

{code}
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15, 
ioThrottle=false
2017-08-05 09:14:03.005 INFO  (qtp1205044462-19) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-19]: updateMergeThreads ioThrottle=false 
targetMBPerSec=10240.0 MB/sec
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15, 
ioThrottle=false
2017-08-05 09:15:51.196 INFO  (qtp1205044462-69) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:15:56.711 INFO  (Lucene Merge Thread #0) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][Lucene Merge Thread #0]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:10.752 INFO  (qtp1205044462-17) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-17]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:18.229 INFO  (Lucene Merge Thread #1) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][Lucene Merge Thread #1]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:26.516 INFO  (qtp1205044462-69) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:35.551 INFO  (Lucene Merge Thread #2) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][Lucene Merge Thread #2]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:38.580 INFO  (qtp1205044462-18) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-18]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:49.397 INFO  (Lucene Merge Thread #3) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][Lucene Merge Thread #3]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:56.630 INFO  (qtp1205044462-15) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-15]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
{code}

See the {{targetMBPerSec}} is initialised to {{10gbps}}, but then it falls back 
to default {{20mbps}}, instead of maintaining at 10gbps. Maybe 
{{SolrIndexConfig#buildMergeSchedule}} is not the right place to do it. I will 
look more.


was (Author: sarkaramr...@gmail.com):
Ah! I don't this will serve our purpose in bulk indexing, logs :: 

{code}
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15, 
ioThrottle=false
2017-08-05 09:14:03.005 INFO  (qtp1205044462-19) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-19]: updateMergeThreads ioThrottle=false 
targetMBPerSec=10240.0 MB/sec
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=5, maxMergeCount=15, 
ioThrottle=false
2017-08-05 09:15:51.196 INFO  (qtp1205044462-69) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-69]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:15:56.711 INFO  (Lucene Merge Thread #0) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][Lucene Merge Thread #0]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:10.752 INFO  (qtp1205044462-17) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][qtp1205044462-17]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:18.229 INFO  (Lucene Merge Thread #1) [c:collection1 s:shard1 
r:core_node2 x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream 
[MergeScheduler][Lucene Merge Thread #1]: updateMergeThreads ioThrottle=false 
targetMBPerSec=20.0 MB/sec
2017-08-05 09:16:26.516 INFO  (qtp1205044462-69) [c:collection1 s:shard1 
r:core_node2