[jira] [Closed] (SOLR-4029) Make DirectoryFactory names consistent between Solr and Lucene
[ https://issues.apache.org/jira/browse/SOLR-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar closed SOLR-4029. - Resolution: Invalid Make DirectoryFactory names consistent between Solr and Lucene -- Key: SOLR-4029 URL: https://issues.apache.org/jira/browse/SOLR-4029 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Fix For: 4.1 Attachments: solr-DFhier.txt rename StandardDirectoryFactory to FSDirectory factory. It is much more easier for user reading solr javadoc to understand whats is going there in DirectoryFactory class hierarchy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13527135#comment-13527135 ] Radim Kolar commented on SOLR-4030: --- It was resolved for me. If you need this open your ticket. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13527181#comment-13527181 ] Radim Kolar commented on SOLR-4030: --- You guys had 1 month of time and wasted it. Now we fork and taking our patches with us. Case closed. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13527223#comment-13527223 ] Radim Kolar commented on SOLR-4030: --- Unless you have written permission from us to distribute our work, you are losing court case against us after we prove to court that we are authors of code in question. We never lost such cases. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13527260#comment-13527260 ] Radim Kolar commented on SOLR-4030: --- Yes, i had similar issue with nutch project. Markus Jelsma reuploaded my patch back to JIRA and refused to delete it. Its still in JIRA violating my copyright laws. I decided not to escalate the conflict and go after ASF and harm other projects just because of misbehavior of one person. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Lucene Developers Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
[ https://issues.apache.org/jira/browse/SOLR-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4041: -- Attachment: (was: solr-monitormerge.txt) Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
[ https://issues.apache.org/jira/browse/SOLR-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar resolved SOLR-4041. --- Resolution: Not A Problem Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4030: -- Attachment: (was: solr-mergeratelimit.txt) Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4030: -- Attachment: (was: solr-ratelimit2.txt) Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar resolved SOLR-4030. --- Resolution: Not A Problem Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Priority: Minor Labels: patch Fix For: 4.1, 5.0 add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508813#comment-13508813 ] Radim Kolar commented on SOLR-4030: --- any progress on this? its trivial patch. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4114) Collection API: Allow multiple shards from one collection on the same Solr server
[ https://issues.apache.org/jira/browse/SOLR-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505533#comment-13505533 ] Radim Kolar commented on SOLR-4114: --- could not you do same thing as Elastic Search. Build index with number of shards (initial number is 5). If there is 1 machine in cluster, then all shards are on this machine. If you add more machines, they will move to other machines. It is way simple for administration. Collection API: Allow multiple shards from one collection on the same Solr server - Key: SOLR-4114 URL: https://issues.apache.org/jira/browse/SOLR-4114 Project: Solr Issue Type: New Feature Components: multicore, SolrCloud Affects Versions: 4.0 Environment: Solr 4.0.0 release Reporter: Per Steffensen Assignee: Per Steffensen Labels: collection-api, multicore, shard, shard-allocation Attachments: SOLR-4114.patch, SOLR-4114.patch We should support running multiple shards from one collection on the same Solr server - the run a collection with 8 shards on a 4 Solr server cluster (each Solr server running 2 shards). Performance tests at our side has shown that this is a good idea, and it is also a good idea for easy elasticity later on - it is much easier to move an entire existing shards from one Solr server to another one that just joined the cluter than it is to split an exsiting shard among the Solr that used to run it and the new Solr. See dev mailing list discussion Multiple shards for one collection on the same Solr server -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505790#comment-13505790 ] Radim Kolar commented on SOLR-4030: --- it still merges into trunk cleanly. just do git pull Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505855#comment-13505855 ] Radim Kolar commented on SOLR-4030: --- but it failed to build with 5.x in jenkins. i will take a more detailed look on it Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505866#comment-13505866 ] Radim Kolar commented on SOLR-4030: --- LUCENE-4537 is cause. I will rework it but its not clear if new rate limiter is used for ordinary writes (not just merges) as well. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4030: -- Attachment: solr-ratelimit2.txt Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505892#comment-13505892 ] Radim Kolar commented on SOLR-4030: --- added limits for reading, merging and writing. Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
[ https://issues.apache.org/jira/browse/SOLR-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505203#comment-13505203 ] Radim Kolar commented on SOLR-4041: --- no Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Reporter: Radim Kolar Assignee: Mark Miller Priority: Minor Labels: patch Fix For: 4.1, 5.0 Attachments: solr-monitormerge.txt add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3816) Need a more granular nrt system that is close to a realtime system.
[ https://issues.apache.org/jira/browse/SOLR-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501198#comment-13501198 ] Radim Kolar commented on SOLR-3816: --- What kind of incorrect results it can deliver? For me its okay to live with possibility of broken faceting because i do not use that feature. Need a more granular nrt system that is close to a realtime system. --- Key: SOLR-3816 URL: https://issues.apache.org/jira/browse/SOLR-3816 Project: Solr Issue Type: Improvement Components: clients - java, replication (java), search, SearchComponents - other, SolrCloud, update Affects Versions: 4.0 Reporter: Nagendra Nagarajayya Labels: nrt, realtime, replication, search, solrcloud, update Attachments: alltests_passed_with_realtime_turnedoff.log, SOLR-3816_4.0_branch.patch, SOLR-3816-4.x.trunk.patch, solr-3816-realtime_nrt.patch Need a more granular NRT system that is close to a realtime system. A realtime system should be able to reflect changes to the index as and when docs are added/updated to the index. soft-commit offers NRT and is more realtime friendly than hard commit but is limited by the dependency on the SolrIndexSearcher being closed and reopened and offers a coarse granular NRT. Closing and reopening of the SolrIndexSearcher may impact performance also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1487) Add expungeDelete to SolrJ's SolrServer.commit(..)
[ https://issues.apache.org/jira/browse/SOLR-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13497697#comment-13497697 ] Radim Kolar commented on SOLR-1487: --- workaround to expunge deleted documents UpdateRequest rq = new UpdateRequest(); rq.setAction(UpdateRequest.ACTION.COMMIT, false, false, 100, true); rq.process(solrSvr); Add expungeDelete to SolrJ's SolrServer.commit(..) --- Key: SOLR-1487 URL: https://issues.apache.org/jira/browse/SOLR-1487 Project: Solr Issue Type: Improvement Components: clients - java Affects Versions: 1.3 Environment: N/A Reporter: Jibo John Attachments: expunge-patch.txt Add expungeDelete to SolrJ's SolrServer.commit(..). Currently, this can be done only through updatehandler ( ( curl update -F stream.body=' commit expungeDeletes=true/' )) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
[ https://issues.apache.org/jira/browse/SOLR-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495290#comment-13495290 ] Radim Kolar commented on SOLR-4041: --- yes Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Fix For: 4.1 Attachments: solr-monitormerge.txt add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-4544) possible bug in ConcurrentMergeScheduler.merge(IndexWriter)
Radim Kolar created LUCENE-4544:
---
Summary: possible bug in
ConcurrentMergeScheduler.merge(IndexWriter)
Key: LUCENE-4544
URL: https://issues.apache.org/jira/browse/LUCENE-4544
Project: Lucene - Core
Issue Type: Bug
Components: core/other
Affects Versions: 5.0
Reporter: Radim Kolar
from dev list:
¨i suspect that this code is broken. Lines 331 - 343 in
org.apache.lucene.index.ConcurrentMergeScheduler.merge(IndexWriter)
mergeThreadCount() are currently active merges, they can be at most
maxThreadCount, maxMergeCount is number of queued merges defaulted with
maxThreadCount+2 and it can never be lower then maxThreadCount, which means
that condition in while can never become true.
synchronized(this) {
long startStallTime = 0;
while (mergeThreadCount() = 1+maxMergeCount) {
startStallTime = System.currentTimeMillis();
if (verbose()) {
message(too many merges; stalling...);
}
try {
wait();
} catch (InterruptedException ie) {
throw new ThreadInterruptedException(ie);
}
}
While confusing, I think the code is actually nearly correct... but I
would love to find some simplifications of CMS's logic (it's really
hairy).
It turns out mergeThreadCount() is allowed to go higher than
maxThreadCount; when this happens, Lucene pauses
mergeThreadCount()-maxThreadCount of those merge threads, and resumes
them once threads finish (see updateMergeThreads). Ie, CMS will
accept up to maxMergeCount merges (and launch threads for them), but
will only allow maxThreadCount of those threads to be running at once.
So what that while loop is doing is preventing more than
maxMergeCount+1 threads from starting, and then pausing the incoming
thread to slow down the rate of segment creation (since merging cannot
keep up).
But ... I think the 1+ is wrong ... it seems like it should just be
mergeThreadCount() = maxMergeCount().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4544) possible bug in ConcurrentMergeScheduler.merge(IndexWriter)
[
https://issues.apache.org/jira/browse/LUCENE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13491800#comment-13491800
]
Radim Kolar commented on LUCENE-4544:
-
it can not be made simple by not creating thread for every new segment merge
but to use standard threadpool + queue scheme? Lot of libraries can do it
easily.
possible bug in ConcurrentMergeScheduler.merge(IndexWriter)
Key: LUCENE-4544
URL: https://issues.apache.org/jira/browse/LUCENE-4544
Project: Lucene - Core
Issue Type: Bug
Components: core/other
Affects Versions: 5.0
Reporter: Radim Kolar
Assignee: Michael McCandless
Attachments: LUCENE-4544.patch
from dev list:
¨i suspect that this code is broken. Lines 331 - 343 in
org.apache.lucene.index.ConcurrentMergeScheduler.merge(IndexWriter)
mergeThreadCount() are currently active merges, they can be at most
maxThreadCount, maxMergeCount is number of queued merges defaulted with
maxThreadCount+2 and it can never be lower then maxThreadCount, which means
that condition in while can never become true.
synchronized(this) {
long startStallTime = 0;
while (mergeThreadCount() = 1+maxMergeCount) {
startStallTime = System.currentTimeMillis();
if (verbose()) {
message(too many merges; stalling...);
}
try {
wait();
} catch (InterruptedException ie) {
throw new ThreadInterruptedException(ie);
}
}
While confusing, I think the code is actually nearly correct... but I
would love to find some simplifications of CMS's logic (it's really
hairy).
It turns out mergeThreadCount() is allowed to go higher than
maxThreadCount; when this happens, Lucene pauses
mergeThreadCount()-maxThreadCount of those merge threads, and resumes
them once threads finish (see updateMergeThreads). Ie, CMS will
accept up to maxMergeCount merges (and launch threads for them), but
will only allow maxThreadCount of those threads to be running at once.
So what that while loop is doing is preventing more than
maxMergeCount+1 threads from starting, and then pausing the incoming
thread to slow down the rate of segment creation (since merging cannot
keep up).
But ... I think the 1+ is wrong ... it seems like it should just be
mergeThreadCount() = maxMergeCount().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
Radim Kolar created SOLR-4041: - Summary: Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Affects Versions: 5.0 Reporter: Radim Kolar add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
[ https://issues.apache.org/jira/browse/SOLR-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4041: -- Attachment: solr-monitormerge.txt Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Attachments: solr-monitormerge.txt add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4041) Allow segment merge monitoring in Solr Admin gui
[ https://issues.apache.org/jira/browse/SOLR-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13491980#comment-13491980 ] Radim Kolar commented on SOLR-4041: --- it could be Allow segment merge monitoring in Solr Admin gui Key: SOLR-4041 URL: https://issues.apache.org/jira/browse/SOLR-4041 Project: Solr Issue Type: Improvement Components: web gui Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Fix For: 4.1 Attachments: solr-monitormerge.txt add solrMbean for ConcurrentMergeScheduler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4030: -- Fix Version/s: 4.1 Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Fix For: 4.1 Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-4029) Make DirectoryFactory names consistent between Solr and Lucene
Radim Kolar created SOLR-4029: - Summary: Make DirectoryFactory names consistent between Solr and Lucene Key: SOLR-4029 URL: https://issues.apache.org/jira/browse/SOLR-4029 Project: Solr Issue Type: Improvement Reporter: Radim Kolar rename StandardDirectoryFactory to FSDirectory factory. It is much more easier for user reading solr javadoc to understand whats is going there in DirectoryFactory class hierarchy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4029) Make DirectoryFactory names consistent between Solr and Lucene
[ https://issues.apache.org/jira/browse/SOLR-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4029: -- Attachment: solr-DFhier.txt Make DirectoryFactory names consistent between Solr and Lucene -- Key: SOLR-4029 URL: https://issues.apache.org/jira/browse/SOLR-4029 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Attachments: solr-DFhier.txt rename StandardDirectoryFactory to FSDirectory factory. It is much more easier for user reading solr javadoc to understand whats is going there in DirectoryFactory class hierarchy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4029) Make DirectoryFactory names consistent between Solr and Lucene
[ https://issues.apache.org/jira/browse/SOLR-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4029: -- Labels: patch (was: ) Affects Version/s: 5.0 Make DirectoryFactory names consistent between Solr and Lucene -- Key: SOLR-4029 URL: https://issues.apache.org/jira/browse/SOLR-4029 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Attachments: solr-DFhier.txt rename StandardDirectoryFactory to FSDirectory factory. It is much more easier for user reading solr javadoc to understand whats is going there in DirectoryFactory class hierarchy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-4030) Use Lucene segment merge throttling
Radim Kolar created SOLR-4030: - Summary: Use Lucene segment merge throttling Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4030: -- Attachment: solr-mergeratelimit.txt Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling
[ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-4030: -- Labels: patch (was: ) Use Lucene segment merge throttling --- Key: SOLR-4030 URL: https://issues.apache.org/jira/browse/SOLR-4030 Project: Solr Issue Type: Improvement Affects Versions: 5.0 Reporter: Radim Kolar Labels: patch Attachments: solr-mergeratelimit.txt add argument maxMergeWriteMBPerSec to Solr directory factories. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3816) Need a more granular nrt system that is close to a realtime system.
[ https://issues.apache.org/jira/browse/SOLR-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13489424#comment-13489424 ] Radim Kolar commented on SOLR-3816: --- I will test performance of this as soon i will have enough free diskspace to load 100 GB into solr. Need a more granular nrt system that is close to a realtime system. --- Key: SOLR-3816 URL: https://issues.apache.org/jira/browse/SOLR-3816 Project: Solr Issue Type: Improvement Components: clients - java, replication (java), search, SearchComponents - other, SolrCloud, update Affects Versions: 4.0 Reporter: Nagendra Nagarajayya Labels: nrt, realtime, replication, search, solrcloud, update Attachments: alltests_passed_with_realtime_turnedoff.log, SOLR-3816_4.0_branch.patch, SOLR-3816-4.x.trunk.patch, solr-3816-realtime_nrt.patch Need a more granular NRT system that is close to a realtime system. A realtime system should be able to reflect changes to the index as and when docs are added/updated to the index. soft-commit offers NRT and is more realtime friendly than hard commit but is limited by the dependency on the SolrIndexSearcher being closed and reopened and offers a coarse granular NRT. Closing and reopening of the SolrIndexSearcher may impact performance also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1487) Add expungeDelete to SolrJ's SolrServer.commit(..)
[ https://issues.apache.org/jira/browse/SOLR-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13488059#comment-13488059 ] Radim Kolar commented on SOLR-1487: --- what have SOLR-3938 to do with delete expunge? Add expungeDelete to SolrJ's SolrServer.commit(..) --- Key: SOLR-1487 URL: https://issues.apache.org/jira/browse/SOLR-1487 Project: Solr Issue Type: Improvement Components: clients - java Affects Versions: 1.3 Environment: N/A Reporter: Jibo John Attachments: expunge-patch.txt Add expungeDelete to SolrJ's SolrServer.commit(..). Currently, this can be done only through updatehandler ( ( curl update -F stream.body=' commit expungeDeletes=true/' )) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-1487) Add expungeDelete to SolrJ's SolrServer.commit(..)
[ https://issues.apache.org/jira/browse/SOLR-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Radim Kolar updated SOLR-1487: -- Attachment: expunge-patch.txt added expunge deletes flag to commit Add expungeDelete to SolrJ's SolrServer.commit(..) --- Key: SOLR-1487 URL: https://issues.apache.org/jira/browse/SOLR-1487 Project: Solr Issue Type: Improvement Components: clients - java Affects Versions: 1.3 Environment: N/A Reporter: Jibo John Attachments: expunge-patch.txt Add expungeDelete to SolrJ's SolrServer.commit(..). Currently, this can be done only through updatehandler ( ( curl update -F stream.body=' commit expungeDeletes=true/' )) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3816) Need a more granular nrt system that is close to a realtime system.
[ https://issues.apache.org/jira/browse/SOLR-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486034#comment-13486034 ] Radim Kolar commented on SOLR-3816: --- patch should be against solr trunk. Solr 4.0 branch is for bugfixes only. Need a more granular nrt system that is close to a realtime system. --- Key: SOLR-3816 URL: https://issues.apache.org/jira/browse/SOLR-3816 Project: Solr Issue Type: Improvement Components: clients - java, replication (java), search, SearchComponents - other, SolrCloud, update Affects Versions: 4.0 Reporter: Nagendra Nagarajayya Labels: nrt, realtime, replication, search, solrcloud, update Attachments: SOLR-3816_4.0_branch.patch, solr-3816-realtime_nrt.patch Need a more granular NRT system that is close to a realtime system. A realtime system should be able to reflect changes to the index as and when docs are added/updated to the index. soft-commit offers NRT and is more realtime friendly than hard commit but is limited by the dependency on the SolrIndexSearcher being closed and reopened and offers a coarse granular NRT. Closing and reopening of the SolrIndexSearcher may impact performance also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4226) Efficient compression of small to medium stored fields
[
https://issues.apache.org/jira/browse/LUCENE-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477038#comment-13477038
]
Radim Kolar commented on LUCENE-4226:
-
is there example config provided?
Efficient compression of small to medium stored fields
--
Key: LUCENE-4226
URL: https://issues.apache.org/jira/browse/LUCENE-4226
Project: Lucene - Core
Issue Type: Improvement
Components: core/index
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Trivial
Fix For: 4.1, 5.0
Attachments: CompressionBenchmark.java, CompressionBenchmark.java,
LUCENE-4226.patch, LUCENE-4226.patch, LUCENE-4226.patch, LUCENE-4226.patch,
LUCENE-4226.patch, LUCENE-4226.patch, LUCENE-4226.patch, LUCENE-4226.patch,
SnappyCompressionAlgorithm.java
I've been doing some experiments with stored fields lately. It is very common
for an index with stored fields enabled to have most of its space used by the
.fdt index file. To prevent this .fdt file from growing too much, one option
is to compress stored fields. Although compression works rather well for
large fields, this is not the case for small fields and the compression ratio
can be very close to 100%, even with efficient compression algorithms.
In order to improve the compression ratio for small fields, I've written a
{{StoredFieldsFormat}} that compresses several documents in a single chunk of
data. To see how it behaves in terms of document deserialization speed and
compression ratio, I've run several tests with different index compression
strategies on 100,000 docs from Mike's 1K Wikipedia articles (title and text
were indexed and stored):
- no compression,
- docs compressed with deflate (compression level = 1),
- docs compressed with deflate (compression level = 9),
- docs compressed with Snappy,
- using the compressing {{StoredFieldsFormat}} with deflate (level = 1) and
chunks of 6 docs,
- using the compressing {{StoredFieldsFormat}} with deflate (level = 9) and
chunks of 6 docs,
- using the compressing {{StoredFieldsFormat}} with Snappy and chunks of 6
docs.
For those who don't know Snappy, it is compression algorithm from Google
which has very high compression ratios, but compresses and decompresses data
very quickly.
{noformat}
Format Compression ratio IndexReader.document time
uncompressed 100% 100%
doc/deflate 1 59% 616%
doc/deflate 9 58% 595%
doc/snappy80% 129%
index/deflate 1 49% 966%
index/deflate 9 46% 938%
index/snappy 65% 264%
{noformat}
(doc = doc-level compression, index = index-level compression)
I find it interesting because it allows to trade speed for space (with
deflate, the .fdt file shrinks by a factor of 2, much better than with
doc-level compression). One other interesting thing is that {{index/snappy}}
is almost as compact as {{doc/deflate}} while it is more than 2x faster at
retrieving documents from disk.
These tests have been done on a hot OS cache, which is the worst case for
compressed fields (one can expect better results for formats that have a high
compression ratio since they probably require fewer read/write operations
from disk).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474931#comment-13474931 ] Radim Kolar commented on SOLR-3755: --- Useful theory about rehashing http://en.wikipedia.org/wiki/Consistent_hashing shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-4288) package target depends on svn checkout
Radim Kolar created LUCENE-4288:
---
Summary: package target depends on svn checkout
Key: LUCENE-4288
URL: https://issues.apache.org/jira/browse/LUCENE-4288
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 5.0
Reporter: Radim Kolar
Solr or Lucene builds only if they are checked from SVN, not from git.
package target fails with:
package-src-tgz:
BUILD FAILED
/usr/local/jboss/.jenkins/jobs/Solr/workspace/solr/build.xml:336: The following
error occurred while executing this line:
/usr/local/jboss/.jenkins/jobs/Solr/workspace/lucene/common-build.xml:1522: The
following error occurred while executing this line:
/usr/local/jboss/.jenkins/jobs/Solr/workspace/lucene/common-build.xml:1545:
exec returned: 1
Total time: 50 seconds
Build step 'Invoke Ant' marked build as failure
failing line is:
exec dir=@{directory} executable=${svn.exe}
outputproperty=svn.info failonerror=true
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
