[jira] [Commented] (SOLR-8931) SolrCloud RebalanceShards API
[ https://issues.apache.org/jira/browse/SOLR-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367791#comment-15367791 ] olivier soyez commented on SOLR-8931: - Great news about Bloomreach donated their Rebalance API. So, I think this feature is not needed anymore when SOLR-9241 is committed. > SolrCloud RebalanceShards API > - > > Key: SOLR-8931 > URL: https://issues.apache.org/jira/browse/SOLR-8931 > Project: Solr > Issue Type: Wish > Components: SolrCloud >Reporter: olivier soyez >Priority: Minor > Labels: patch > Fix For: 6.0 > > Attachments: SOLR-8931.patch > > > It would be great to have RebalanceShards action in SolrCloud, such like > described in this post by Suruchi Shah : > "http://engineering.bloomreach.com/solrcloud-rebalance-api/"; > By the way, in order to rebalance shards from a collection with > replicationFactor > 1, one idea could be to split some shards using the > rule-based replica placement. > Since https://issues.apache.org/jira/browse/SOLR-8728 jira issue, splitShard > is using rule-based replica placement (for the "replication" replicas). > As part of a proof of concept, the attached patch introduce a new action to > the collections API, named "REBALANCESHARDS", to rebalance some or all shards > among solrCloud nodes using splitShard. > After each splitShard, a deleteshard of the inactive parent shard is done. > One mandatory parameter: > - collection: the name of the collection > Two parameters: > - deltaMaxFromAverage (default: 20): use to select n shards (<= half of all > shards) to be split, whose number of docs are greater than > deltaMaxFromAverage percent of the average number of docs per shard > - force (default: false): if true, in case of none shards selected with the > deltaMaxFromAverage given, all shards of the collection will be selected to > be split > Use example: > curl > 'http://ip:port/solr/admin/collections?action=REBALANCESHARDS&collection=collection1&async=2&deltaMaxFromAverage=30' > Drawbacks: replicationFactor must be more than one, select shards based on > the average number of docs per shard is not suitable for all cases -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8931) SolrCloud RebalanceShards API
[ https://issues.apache.org/jira/browse/SOLR-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-8931: Attachment: SOLR-8931.patch > SolrCloud RebalanceShards API > - > > Key: SOLR-8931 > URL: https://issues.apache.org/jira/browse/SOLR-8931 > Project: Solr > Issue Type: Wish > Components: SolrCloud >Reporter: olivier soyez >Priority: Minor > Labels: patch > Fix For: trunk > > Attachments: SOLR-8931.patch > > > It would be great to have RebalanceShards action in SolrCloud, such like > described in this post by Suruchi Shah : > "http://engineering.bloomreach.com/solrcloud-rebalance-api/"; > By the way, in order to rebalance shards from a collection with > replicationFactor > 1, one idea could be to split some shards using the > rule-based replica placement. > Since https://issues.apache.org/jira/browse/SOLR-8728 jira issue, splitShard > is using rule-based replica placement (for the "replication" replicas). > As part of a proof of concept, the attached patch introduce a new action to > the collections API, named "REBALANCESHARDS", to rebalance some or all shards > among solrCloud nodes using splitShard. > After each splitShard, a deleteshard of the inactive parent shard is done. > One mandatory parameter: > - collection: the name of the collection > Two parameters: > - deltaMaxFromAverage (default: 20): use to select n shards (<= half of all > shards) to be split, whose number of docs are greater than > deltaMaxFromAverage percent of the average number of docs per shard > - force (default: false): if true, in case of none shards selected with the > deltaMaxFromAverage given, all shards of the collection will be selected to > be split > Use example: > curl > 'http://ip:port/solr/admin/collections?action=REBALANCESHARDS&collection=collection1&async=2&deltaMaxFromAverage=30' > Drawbacks: replicationFactor must be more than one, select shards based on > the average number of docs per shard is not suitable for all cases -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-8931) SolrCloud RebalanceShards API
olivier soyez created SOLR-8931: --- Summary: SolrCloud RebalanceShards API Key: SOLR-8931 URL: https://issues.apache.org/jira/browse/SOLR-8931 Project: Solr Issue Type: Wish Components: SolrCloud Reporter: olivier soyez Priority: Minor Fix For: trunk It would be great to have RebalanceShards action in SolrCloud, such like described in this post by Suruchi Shah : "http://engineering.bloomreach.com/solrcloud-rebalance-api/"; By the way, in order to rebalance shards from a collection with replicationFactor > 1, one idea could be to split some shards using the rule-based replica placement. Since https://issues.apache.org/jira/browse/SOLR-8728 jira issue, splitShard is using rule-based replica placement (for the "replication" replicas). As part of a proof of concept, the attached patch introduce a new action to the collections API, named "REBALANCESHARDS", to rebalance some or all shards among solrCloud nodes using splitShard. After each splitShard, a deleteshard of the inactive parent shard is done. One mandatory parameter: - collection: the name of the collection Two parameters: - deltaMaxFromAverage (default: 20): use to select n shards (<= half of all shards) to be split, whose number of docs are greater than deltaMaxFromAverage percent of the average number of docs per shard - force (default: false): if true, in case of none shards selected with the deltaMaxFromAverage given, all shards of the collection will be selected to be split Use example: curl 'http://ip:port/solr/admin/collections?action=REBALANCESHARDS&collection=collection1&async=2&deltaMaxFromAverage=30' Drawbacks: replicationFactor must be more than one, select shards based on the average number of docs per shard is not suitable for all cases -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270849#comment-14270849 ] olivier soyez commented on SOLR-5316: - Erik: I understand, thank you for your feedback! > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versions: 4.2.1, 4.7 >Reporter: olivier soyez >Priority: Minor > Labels: patch > Fix For: 4.2.1 > > Attachments: SOLR-5316.patch, solr-4.2.1.patchLOTSOFCORES > > > The SOLR-5316.patch is for the lotsofcores use case: a large number of > homogeneous cores with frequent loading/unloading. The new Cores options are: > - defaultLoadOnStartup : true by default > - defaultTransient : false by default > In this way, you can change the default attributes value for all cores. > The lotsofcores use case requires to have defaultLoadOnStartup = false and > defaultTransient = true > - "numBuckets" to create a subdirectory based on a hash on the corename % > numBuckets in the core Datadir, because all cores cannot live in the same > directory > - "auto" with 2 differents values : > 1) createLoad: create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 2) onlyLoad: load the core on the fly on the first incoming request > (update, select), if exist on disk > The auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. All the cores have the same solr config defined in the > solr_home/conf. > - discoverOnDemand : false by default. If enabled, the discovery and loading > of cores at the solr startup are disabled and a new option named > coreDiscovery in the STATUS action is available to discover on demand all the > cores in the coreRootDirectory (only add the cores descriptions) > - noCorePropertiesFile : false by default. If enabled, don't use anymore the > core.properties file, a core will be detected on disk based on the file > segments.gen > The unload command was also modified in order to handle the non loaded > transient cores, to be able to apply for example the deleteIndex, > deleteDataDir options. > I added some Junit tests. > An example of solr.xml: > > schema.xml > 5000 > solr_home/data > solr_home/data > 100 > createLoad > false > true > true > true > > The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision > number 1556554) > --- > The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case, > including some modification : > - by default, all cores have loadOnStartup="false" and transient="true" > attributes > - the create admin command can register a lazy core (to take into account the > transientCacheSize option) > - add transient cores persistency > - handle unload admin command for never launched transient cores (non active > cores) > To improve performance, we use this Solr patched version with the persistence > disabled. > In this way, Solr is working with a solr.xml file without any core entries, > because it's useless in our use case (with the new Auto option for the cores) > The new Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > The Auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903397#comment-13903397 ] olivier soyez commented on SOLR-5316: - I have run "ant precommit" after the changes in SOLR-5316.patch, and it's OK > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versions: 4.2.1, 4.7 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Labels: patch > Fix For: 4.2.1 > > Attachments: SOLR-5316.patch, solr-4.2.1.patchLOTSOFCORES > > > The SOLR-5316.patch is for the lotsofcores use case: a large number of > homogeneous cores with frequent loading/unloading. The new Cores options are: > - defaultLoadOnStartup : true by default > - defaultTransient : false by default > In this way, you can change the default attributes value for all cores. > The lotsofcores use case requires to have defaultLoadOnStartup = false and > defaultTransient = true > - "numBuckets" to create a subdirectory based on a hash on the corename % > numBuckets in the core Datadir, because all cores cannot live in the same > directory > - "auto" with 2 differents values : > 1) createLoad: create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 2) onlyLoad: load the core on the fly on the first incoming request > (update, select), if exist on disk > The auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. All the cores have the same solr config defined in the > solr_home/conf. > - discoverOnDemand : false by default. If enabled, the discovery and loading > of cores at the solr startup are disabled and a new option named > coreDiscovery in the STATUS action is available to discover on demand all the > cores in the coreRootDirectory (only add the cores descriptions) > - noCorePropertiesFile : false by default. If enabled, don't use anymore the > core.properties file, a core will be detected on disk based on the file > segments.gen > The unload command was also modified in order to handle the non loaded > transient cores, to be able to apply for example the deleteIndex, > deleteDataDir options. > I added some Junit tests. > An example of solr.xml: > > schema.xml > 5000 > solr_home/data > solr_home/data > 100 > createLoad > false > true > true > true > > The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision > number 1556554) > --- > The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case, > including some modification : > - by default, all cores have loadOnStartup="false" and transient="true" > attributes > - the create admin command can register a lazy core (to take into account the > transientCacheSize option) > - add transient cores persistency > - handle unload admin command for never launched transient cores (non active > cores) > To improve performance, we use this Solr patched version with the persistence > disabled. > In this way, Solr is working with a solr.xml file without any core entries, > because it's useless in our use case (with the new Auto option for the cores) > The new Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > The Auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5316: Attachment: SOLR-5316.patch > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versions: 4.2.1, 4.7 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Labels: patch > Fix For: 4.2.1 > > Attachments: SOLR-5316.patch, solr-4.2.1.patchLOTSOFCORES > > > The SOLR-5316.patch is for the lotsofcores use case: a large number of > homogeneous cores with frequent loading/unloading. The new Cores options are: > - defaultLoadOnStartup : true by default > - defaultTransient : false by default > In this way, you can change the default attributes value for all cores. > The lotsofcores use case requires to have defaultLoadOnStartup = false and > defaultTransient = true > - "numBuckets" to create a subdirectory based on a hash on the corename % > numBuckets in the core Datadir, because all cores cannot live in the same > directory > - "auto" with 2 differents values : > 1) createLoad: create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 2) onlyLoad: load the core on the fly on the first incoming request > (update, select), if exist on disk > The auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. All the cores have the same solr config defined in the > solr_home/conf. > - discoverOnDemand : false by default. If enabled, the discovery and loading > of cores at the solr startup are disabled and a new option named > coreDiscovery in the STATUS action is available to discover on demand all the > cores in the coreRootDirectory (only add the cores descriptions) > - noCorePropertiesFile : false by default. If enabled, don't use anymore the > core.properties file, a core will be detected on disk based on the file > segments.gen > The unload command was also modified in order to handle the non loaded > transient cores, to be able to apply for example the deleteIndex, > deleteDataDir options. > I added some Junit tests. > An example of solr.xml: > > schema.xml > 5000 > solr_home/data > solr_home/data > 100 > createLoad > false > true > true > true > > The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision > number 1556554) > --- > The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case, > including some modification : > - by default, all cores have loadOnStartup="false" and transient="true" > attributes > - the create admin command can register a lazy core (to take into account the > transientCacheSize option) > - add transient cores persistency > - handle unload admin command for never launched transient cores (non active > cores) > To improve performance, we use this Solr patched version with the persistence > disabled. > In this way, Solr is working with a solr.xml file without any core entries, > because it's useless in our use case (with the new Auto option for the cores) > The new Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > The Auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5316: Description: The SOLR-5316.patch is for the lotsofcores use case: a large number of homogeneous cores with frequent loading/unloading. The new Cores options are: - defaultLoadOnStartup : true by default - defaultTransient : false by default In this way, you can change the default attributes value for all cores. The lotsofcores use case requires to have defaultLoadOnStartup = false and defaultTransient = true - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "auto" with 2 differents values : 1) createLoad: create, if not exist, and load the core on the fly on the first incoming request (update, select). 2) onlyLoad: load the core on the fly on the first incoming request (update, select), if exist on disk The auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. All the cores have the same solr config defined in the solr_home/conf. - discoverOnDemand : false by default. If enabled, the discovery and loading of cores at the solr startup are disabled and a new option named coreDiscovery in the STATUS action is available to discover on demand all the cores in the coreRootDirectory (only add the cores descriptions) - noCorePropertiesFile : false by default. If enabled, don't use anymore the core.properties file, a core will be detected on disk based on the file segments.gen The unload command was also modified in order to handle the non loaded transient cores, to be able to apply for example the deleteIndex, deleteDataDir options. I added some Junit tests. An example of solr.xml: schema.xml 5000 solr_home/data solr_home/data 100 createLoad false true true true The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision number 1556554) --- The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case, including some modification : - by default, all cores have loadOnStartup="false" and transient="true" attributes - the create admin command can register a lazy core (to take into account the transientCacheSize option) - add transient cores persistency - handle unload admin command for never launched transient cores (non active cores) To improve performance, we use this Solr patched version with the persistence disabled. In this way, Solr is working with a solr.xml file without any core entries, because it's useless in our use case (with the new Auto option for the cores) The new Cores options : - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "Auto" with 3 differents values : 1) false : default behaviour 2) createLoad : create, if not exist, and load the core on the fly on the first incoming request (update, select). 3) onlyLoad : load the core on the fly on the first incoming request (update, select), if exist on disk The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. was: The SOLR-5316.patch is for the lotsofcores use case: a large number of homogeneous cores with frequent loading/unloading. The new Cores options are: - defaultLoadOnStartup : true by default - defaultTransient : false by default In this way, you can change the default attributes value for all cores. The lotsofcores use case requires to have defaultLoadOnStartup = false and defaultTransient = true - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "auto" with 2 differents values : 1) createLoad: create, if not exist, and load the core on the fly on the first incoming request (update, select). 2) onlyLoad: load the core on the fly on the first incoming request (update, select), if exist on disk The auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. All the cores have the same solr config defined in the solr_home/conf. - discoverOnDemand : false by default. If enabled, the discovery and loading of cores at the solr startup are disabled and a new option named coreDiscovery in the STATUS action is available to discover on demand all the cores in the coreRootDirectory (only add the cores descriptions) - noCorePropertiesFile : false by default. If enabled, don't use anymore the core.properties file, a core will be detected on disk based on the file segments.gen The unl
[jira] [Updated] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5316: Affects Version/s: 4.7 > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versions: 4.2.1, 4.7 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Labels: patch > Fix For: 4.2.1 > > Attachments: solr-4.2.1.patchLOTSOFCORES > > > The SOLR-5316.patch is for the lotsofcores use case: a large number of > homogeneous cores with frequent loading/unloading. The new Cores options are: > - defaultLoadOnStartup : true by default > - defaultTransient : false by default > In this way, you can change the default attributes value for all cores. > The lotsofcores use case requires to have defaultLoadOnStartup = false and > defaultTransient = true > - "numBuckets" to create a subdirectory based on a hash on the corename % > numBuckets in the core Datadir, because all cores cannot live in the same > directory > - "auto" with 2 differents values : > 1) createLoad: create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 2) onlyLoad: load the core on the fly on the first incoming request > (update, select), if exist on disk > The auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. All the cores have the same solr config defined in the > solr_home/conf. > - discoverOnDemand : false by default. If enabled, the discovery and loading > of cores at the solr startup are disabled and a new option named > coreDiscovery in the STATUS action is available to discover on demand all the > cores in the coreRootDirectory (only add the cores descriptions) > - noCorePropertiesFile : false by default. If enabled, don't use anymore the > core.properties file, a core will be detected on disk based on the file > segments.gen > The unload command was also modified in order to handle the non loaded > transient cores, to be able to apply for example the deleteIndex, > deleteDataDir options. > I added some Junit tests. > An example of solr.xml: > > schema.xml > 5000 > solr_home/data > solr_home/data > 100 > createLoad > false > true > true > true > > The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision > number 1556554) > --- > The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case, > including some modification : > - by default, all cores have loadOnStartup="false" and transient="true" > attributes > - the create admin command can register a lazy core (to take into account the > transientCacheSize option) > - add transient cores persistency > - handle unload admin command for never launched transient cores (non active > cores) > To improve performance, we use this Solr patched version with the persistence > disabled. > In this way, Solr is working with a solr.xml file without any core entries, > because it's useless in our use case (with the new Auto option for the cores) > The new Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > The Auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5316: Description: The SOLR-5316.patch is for the lotsofcores use case: a large number of homogeneous cores with frequent loading/unloading. The new Cores options are: - defaultLoadOnStartup : true by default - defaultTransient : false by default In this way, you can change the default attributes value for all cores. The lotsofcores use case requires to have defaultLoadOnStartup = false and defaultTransient = true - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "auto" with 2 differents values : 1) createLoad: create, if not exist, and load the core on the fly on the first incoming request (update, select). 2) onlyLoad: load the core on the fly on the first incoming request (update, select), if exist on disk The auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. All the cores have the same solr config defined in the solr_home/conf. - discoverOnDemand : false by default. If enabled, the discovery and loading of cores at the solr startup are disabled and a new option named coreDiscovery in the STATUS action is available to discover on demand all the cores in the coreRootDirectory (only add the cores descriptions) - noCorePropertiesFile : false by default. If enabled, don't use anymore the core.properties file, a core will be detected on disk based on the file segments.gen The unload command was also modified in order to handle the non loaded transient cores, to be able to apply for example the deleteIndex, deleteDataDir options. I added some Junit tests. An example of solr.xml: schema.xml 5000 solr_home/data solr_home/data 100 createLoad false true true true The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision number 1556554) --- This patch is for the lotsofcores use case, including some modification : - by default, all cores have loadOnStartup="false" and transient="true" attributes - the create admin command can register a lazy core (to take into account the transientCacheSize option) - add transient cores persistency - handle unload admin command for never launched transient cores (non active cores) To improve performance, we use this Solr patched version with the persistence disabled. In this way, Solr is working with a solr.xml file without any core entries, because it's useless in our use case (with the new Auto option for the cores) The new Cores options : - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "Auto" with 3 differents values : 1) false : default behaviour 2) createLoad : create, if not exist, and load the core on the fly on the first incoming request (update, select). 3) onlyLoad : load the core on the fly on the first incoming request (update, select), if exist on disk The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. was: This patch is for the lotsofcores use case, including some modification : - by default, all cores have loadOnStartup="false" and transient="true" attributes - the create admin command can register a lazy core (to take into account the transientCacheSize option) - add transient cores persistency - handle unload admin command for never launched transient cores (non active cores) To improve performance, we use this Solr patched version with the persistence disabled. In this way, Solr is working with a solr.xml file without any core entries, because it's useless in our use case (with the new Auto option for the cores) The new Cores options : - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "Auto" with 3 differents values : 1) false : default behaviour 2) createLoad : create, if not exist, and load the core on the fly on the first incoming request (update, select). 3) onlyLoad : load the core on the fly on the first incoming request (update, select), if exist on disk The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versi
[jira] [Created] (SOLR-5738) segments.gen missing
olivier soyez created SOLR-5738: --- Summary: segments.gen missing Key: SOLR-5738 URL: https://issues.apache.org/jira/browse/SOLR-5738 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 4.6.1, 4.7 Reporter: olivier soyez Priority: Minor Attachments: SOLR-5738.patch If I add documents to the slave or if slave index gets corrupted, Slave then copies all the files from master to a new index dir. More details : https://wiki.apache.org/solr/SolrReplication#What_if_I_add_documents_to_the_slave_or_if_slave_index_gets_corrupted.3F But the segments.gen file is missing The patch in attachment is for svn solr branch_4X (revision number 1556554) -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5738) segments.gen missing
[ https://issues.apache.org/jira/browse/SOLR-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5738: Attachment: SOLR-5738.patch > segments.gen missing > > > Key: SOLR-5738 > URL: https://issues.apache.org/jira/browse/SOLR-5738 > Project: Solr > Issue Type: Bug > Components: replication (java) >Affects Versions: 4.6.1, 4.7 >Reporter: olivier soyez >Priority: Minor > Labels: patch > Attachments: SOLR-5738.patch > > > If I add documents to the slave or if slave index gets corrupted, > Slave then copies all the files from master to a new index dir. > More details : > https://wiki.apache.org/solr/SolrReplication#What_if_I_add_documents_to_the_slave_or_if_slave_index_gets_corrupted.3F > But the segments.gen file is missing > The patch in attachment is for svn solr branch_4X (revision number 1556554) -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5737) segments.gen missing
olivier soyez created SOLR-5737: --- Summary: segments.gen missing Key: SOLR-5737 URL: https://issues.apache.org/jira/browse/SOLR-5737 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 4.6.1, 4.7 Reporter: olivier soyez Priority: Minor If I add documents to the slave or if slave index gets corrupted, Slave then copies all the files from master to a new index dir. More details : https://wiki.apache.org/solr/SolrReplication#What_if_I_add_documents_to_the_slave_or_if_slave_index_gets_corrupted.3F But the segments.gen file is missing The patch in attachment is for svn solr branch_4X (revision number 1556554) -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5366) Too many close on transient core with unload command
[ https://issues.apache.org/jira/browse/SOLR-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5366: Attachment: SOLR-5366.patch > Too many close on transient core with unload command > > > Key: SOLR-5366 > URL: https://issues.apache.org/jira/browse/SOLR-5366 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.4, 4.6 >Reporter: olivier soyez >Priority: Minor > Labels: patch > Attachments: SOLR-5366.patch > > > Too many close on transient core with unload command, because already closed > by transientcache > To reproduce : > set transientCacheSize=2 and start with no cores > Create 3 cores : > curl > "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; > Execute : > curl "http://ip:port/solr/core1/select?q=*:*"; > curl "http://ip:port/solr/core2/select?q=*:*"; > curl "http://ip:port/solr/core3/select?q=*:*"; > curl "http://ip:port/solr/admin/cores?action=UNLOAD&core=core1"; > And in Solr logs : > ERROR org.apache.solr.core.SolrCore – Too many close [count:-1] on > org.apache.solr.core.SolrCore@20e5ace9. Please report this exception to > solr-u...@lucene.apache.org > The patch in attachment is for svn solr branch_4X (revision number 1526418) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5366) Too many close on transient core with unload command
olivier soyez created SOLR-5366: --- Summary: Too many close on transient core with unload command Key: SOLR-5366 URL: https://issues.apache.org/jira/browse/SOLR-5366 Project: Solr Issue Type: Bug Components: multicore Affects Versions: 4.4, 4.6 Reporter: olivier soyez Priority: Minor Attachments: SOLR-5366.patch Too many close on transient core with unload command, because already closed by transientcache To reproduce : set transientCacheSize=2 and start with no cores Create 3 cores : curl "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; Execute : curl "http://ip:port/solr/core1/select?q=*:*"; curl "http://ip:port/solr/core2/select?q=*:*"; curl "http://ip:port/solr/core3/select?q=*:*"; curl "http://ip:port/solr/admin/cores?action=UNLOAD&core=core1"; And in Solr logs : ERROR org.apache.solr.core.SolrCore – Too many close [count:-1] on org.apache.solr.core.SolrCore@20e5ace9. Please report this exception to solr-u...@lucene.apache.org The patch in attachment is for svn solr branch_4X (revision number 1526418) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5318) create command don't take into account the transient core property
[ https://issues.apache.org/jira/browse/SOLR-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5318: Description: the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) To reproduce : set transientCacheSize=2 and start with no cores Create 3 cores : curl "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; Look at the status : http://ip:port/solr/admin/cores?action=STATUS All cores are still loaded. One core should not be loaded (closed by the transient cache) The patch in attachment is for svn solr branch_4X (revision number 1526418) was: the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) To reproduce : set transientCacheSize=2 and start with no cores Create 3 cores : curl "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; Look at the status : http://ip:port/solr/admin/cores?action=STATUS All cores are still loaded. One core should not be loaded (closed by the transient cache) > create command don't take into account the transient core property > -- > > Key: SOLR-5318 > URL: https://issues.apache.org/jira/browse/SOLR-5318 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.4, 4.6 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-5318.patch > > > the create core admin command don't take into account the transient core > property, when the core is registered (so, the core will be never closed by > the transient core cache) > To reproduce : > set transientCacheSize=2 and start with no cores > Create 3 cores : > curl > "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; > Look at the status : > http://ip:port/solr/admin/cores?action=STATUS > All cores are still loaded. > One core should not be loaded (closed by the transient cache) > The patch in attachment is for svn solr branch_4X (revision number 1526418) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5318) create command don't take into account the transient core property
[ https://issues.apache.org/jira/browse/SOLR-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5318: Affects Version/s: 4.4 > create command don't take into account the transient core property > -- > > Key: SOLR-5318 > URL: https://issues.apache.org/jira/browse/SOLR-5318 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.4, 4.6 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-5318.patch > > > the create core admin command don't take into account the transient core > property, when the core is registered (so, the core will be never closed by > the transient core cache) > To reproduce : > set transientCacheSize=2 and start with no cores > Create 3 cores : > curl > "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; > Look at the status : > http://ip:port/solr/admin/cores?action=STATUS > All cores are still loaded. > One core should not be loaded (closed by the transient cache) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5318) create command don't take into account the transient core property
[ https://issues.apache.org/jira/browse/SOLR-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790169#comment-13790169 ] olivier soyez commented on SOLR-5318: - We are using in production solr 4.2.1, but I also test solr 4.4 and the svn solr branch_4X : same issue I complete the description and the way to reproduce the issue Not correlated with SOLR-4862 > create command don't take into account the transient core property > -- > > Key: SOLR-5318 > URL: https://issues.apache.org/jira/browse/SOLR-5318 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.6 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-5318.patch > > > the create core admin command don't take into account the transient core > property, when the core is registered (so, the core will be never closed by > the transient core cache) > To reproduce : > set transientCacheSize=2 and start with no cores > Create 3 cores : > curl > "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; > Look at the status : > http://ip:port/solr/admin/cores?action=STATUS > All cores are still loaded. > One core should not be loaded (closed by the transient cache) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5318) create command don't take into account the transient core property
[ https://issues.apache.org/jira/browse/SOLR-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5318: Description: the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) To reproduce : set transientCacheSize=2 and start with no cores Create 3 cores : curl "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; Look at the status : http://ip:port/solr/admin/cores?action=STATUS All cores are still loaded. One core should not be loaded (closed by the transient cache) was: the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) To reproduce : set transientCacheSize=2 and start with no cores Create 3 cores : curl "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; Look at the status : http://ip:port/solr/admin/cores?action=STATUS All cores are still loaded. One core should not be loaded (closed by the transient cache). > create command don't take into account the transient core property > -- > > Key: SOLR-5318 > URL: https://issues.apache.org/jira/browse/SOLR-5318 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.6 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-5318.patch > > > the create core admin command don't take into account the transient core > property, when the core is registered (so, the core will be never closed by > the transient core cache) > To reproduce : > set transientCacheSize=2 and start with no cores > Create 3 cores : > curl > "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; > Look at the status : > http://ip:port/solr/admin/cores?action=STATUS > All cores are still loaded. > One core should not be loaded (closed by the transient cache) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5318) create command don't take into account the transient core property
[ https://issues.apache.org/jira/browse/SOLR-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5318: Description: the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) To reproduce : set transientCacheSize=2 and start with no cores Create 3 cores : curl "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; Look at the status : http://ip:port/solr/admin/cores?action=STATUS All cores are still loaded. One core should not be loaded (closed by the transient cache). was: the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) > create command don't take into account the transient core property > -- > > Key: SOLR-5318 > URL: https://issues.apache.org/jira/browse/SOLR-5318 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.6 >Reporter: olivier soyez >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-5318.patch > > > the create core admin command don't take into account the transient core > property, when the core is registered (so, the core will be never closed by > the transient core cache) > To reproduce : > set transientCacheSize=2 and start with no cores > Create 3 cores : > curl > "http://ip:port/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instanceDir_coreX&dataDir=path_to_dataDir_coreX&loadOnStartup=false&transient=true"; > Look at the status : > http://ip:port/solr/admin/cores?action=STATUS > All cores are still loaded. > One core should not be loaded (closed by the transient cache). -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5318) create command don't take into account the transient core property
[ https://issues.apache.org/jira/browse/SOLR-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5318: Attachment: SOLR-5318.patch > create command don't take into account the transient core property > -- > > Key: SOLR-5318 > URL: https://issues.apache.org/jira/browse/SOLR-5318 > Project: Solr > Issue Type: Bug > Components: multicore >Affects Versions: 4.6 >Reporter: olivier soyez >Priority: Minor > Attachments: SOLR-5318.patch > > > the create core admin command don't take into account the transient core > property, when the core is registered (so, the core will be never closed by > the transient core cache) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5318) create command don't take into account the transient core property
olivier soyez created SOLR-5318: --- Summary: create command don't take into account the transient core property Key: SOLR-5318 URL: https://issues.apache.org/jira/browse/SOLR-5318 Project: Solr Issue Type: Bug Components: multicore Affects Versions: 4.6 Reporter: olivier soyez Priority: Minor the create core admin command don't take into account the transient core property, when the core is registered (so, the core will be never closed by the transient core cache) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5316: Description: This patch is for the lotsofcores use case, including some modification : - by default, all cores have loadOnStartup="false" and transient="true" attributes - the create admin command can register a lazy core (to take into account the transientCacheSize option) - add transient cores persistency - handle unload admin command for never launched transient cores (non active cores) To improve performance, we use this Solr patched version with the persistence disabled. In this way, Solr is working with a solr.xml file without any core entries, because it's useless in our use case (with the new Auto option for the cores) The new Cores options : - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "Auto" with 3 differents values : 1) false : default behaviour 2) createLoad : create, if not exist, and load the core on the fly on the first incoming request (update, select). 3) onlyLoad : load the core on the fly on the first incoming request (update, select), if exist on disk The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. was: This patch is for the lotsofcores use case, including some modification : - by default, all cores have loadOnStartup="false" and transient="true" attributes - the create admin command can register a lazy core (to take into account the transientCacheSize option) - add transient cores persistency - handle unload admin command for never launched transient cores (non active cores) The new Cores options : - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "Auto" with 3 differents values : 1) false : default behaviour 2) createLoad : create, if not exist, and load the core on the fly on the first incoming request (update, select). 3) onlyLoad : load the core on the fly on the first incoming request (update, select), if exist on disk The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versions: 4.2.1 >Reporter: olivier soyez >Priority: Minor > Labels: patch > Fix For: 4.2.1 > > Attachments: solr-4.2.1.patchLOTSOFCORES > > > This patch is for the lotsofcores use case, including some modification : > - by default, all cores have loadOnStartup="false" and transient="true" > attributes > - the create admin command can register a lazy core (to take into account the > transientCacheSize option) > - add transient cores persistency > - handle unload admin command for never launched transient cores (non active > cores) > To improve performance, we use this Solr patched version with the persistence > disabled. > In this way, Solr is working with a solr.xml file without any core entries, > because it's useless in our use case (with the new Auto option for the cores) > The new Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > The Auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
olivier soyez created SOLR-5316: --- Summary: Solr 4.2.1 LotsOfCores new options Key: SOLR-5316 URL: https://issues.apache.org/jira/browse/SOLR-5316 Project: Solr Issue Type: Improvement Components: multicore Affects Versions: 4.2.1 Reporter: olivier soyez Priority: Minor Fix For: 4.2.1 Attachments: solr-4.2.1.patchLOTSOFCORES This patch is for the lotsofcores use case, including some modification : - by default, all cores have loadOnStartup="false" and transient="true" attributes - the create admin command can register a lazy core (to take into account the transientCacheSize option) - add transient cores persistency - handle unload admin command for never launched transient cores (non active cores) The new Cores options : - "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory - "Auto" with 3 differents values : 1) false : default behaviour 2) createLoad : create, if not exist, and load the core on the fly on the first incoming request (update, select). 3) onlyLoad : load the core on the fly on the first incoming request (update, select), if exist on disk The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5316) Solr 4.2.1 LotsOfCores new options
[ https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] olivier soyez updated SOLR-5316: Attachment: solr-4.2.1.patchLOTSOFCORES > Solr 4.2.1 LotsOfCores new options > -- > > Key: SOLR-5316 > URL: https://issues.apache.org/jira/browse/SOLR-5316 > Project: Solr > Issue Type: Improvement > Components: multicore >Affects Versions: 4.2.1 >Reporter: olivier soyez >Priority: Minor > Labels: patch > Fix For: 4.2.1 > > Attachments: solr-4.2.1.patchLOTSOFCORES > > > This patch is for the lotsofcores use case, including some modification : > - by default, all cores have loadOnStartup="false" and transient="true" > attributes > - the create admin command can register a lazy core (to take into account the > transientCacheSize option) > - add transient cores persistency > - handle unload admin command for never launched transient cores (non active > cores) > The new Cores options : > - "numBuckets" to create a subdirectory based on a hash on the corename > % numBuckets in the core Datadir, because all cores cannot live in the > same directory > - "Auto" with 3 differents values : > 1) false : default behaviour > 2) createLoad : create, if not exist, and load the core on the fly on > the first incoming request (update, select). > 3) onlyLoad : load the core on the fly on the first incoming request > (update, select), if exist on disk > The Auto option uses an additional cores option named baseDataDir to > automatically generate the dataDir of each core and uses the "numBucket" > option if exists. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1575) loading solr.xml takes a lot of time when there are lot of cores
[ https://issues.apache.org/jira/browse/SOLR-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526534#comment-13526534 ] olivier soyez commented on SOLR-1575: - With trunk solr 5.0, all cores with loadOnStartup="false" in solr.xml : - without any changes in the code : 15K cores server startup in 220115 ms ~ 4 min 50K cores server startup in 2925056 ms ~ 49 min - with the change : 15K cores server startup in 8145 ms ~ 8 sec 50K cores server startup in 17211 ms~ 17 sec This little change is a big boost for the server startup ... > loading solr.xml takes a lot of time when there are lot of cores > > > Key: SOLR-1575 > URL: https://issues.apache.org/jira/browse/SOLR-1575 > Project: Solr > Issue Type: Improvement > Components: multicore >Reporter: Noble Paul >Priority: Minor > Fix For: 1.5 > > > We have around 38K cores which takes up around 25 mins to start the server. > Keep in mind that no cores are started and this is the time taken for loading > the solr.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1575) loading solr.xml takes a lot of time when there are lot of cores
[ https://issues.apache.org/jira/browse/SOLR-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511545#comment-13511545 ] olivier soyez commented on SOLR-1575: - In fact, I used the 4.0 code + SOLR-1293 modified code to test lotsofcores solr 4.0 (to be able to handle 50k cores per server with 1000 lives cores max at any time, shared configs, numBuckets, lazy load cores, efficient start cores, automatic start/stop cores, core creation on the fly, efficient fresh start). It's the reason why I have the loadOnStart="false" option. Anyways, I pull down 5x and without changes, I try to start the server (50K cores with "loadOnStartup=false"), and same issue (very very long time to start ...) After that, I applied the change (line 590 of CoreContainer.java Apache Solr 5.0 trunck). In this way, the node you are working with is detached from the main document. So, it's a huge speed up for the evaluate method used in readProperties function called for each core, especially in case of lots of cores. And now it takes a few seconds to start the server. Thanks, I would following SOLR-4083, because in your specific use case (50k cores), deprecate solr.xml entirely could be a scalability problem (especially fresh start time). > loading solr.xml takes a lot of time when there are lot of cores > > > Key: SOLR-1575 > URL: https://issues.apache.org/jira/browse/SOLR-1575 > Project: Solr > Issue Type: Improvement > Components: multicore >Reporter: Noble Paul >Priority: Minor > Fix For: 1.5 > > > We have around 38K cores which takes up around 25 mins to start the server. > Keep in mind that no cores are started and this is the time taken for loading > the solr.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1575) loading solr.xml takes a lot of time when there are lot of cores
[ https://issues.apache.org/jira/browse/SOLR-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509857#comment-13509857 ] olivier soyez commented on SOLR-1575: - We have around 50K cores with loadOnStart="false" and same issue to start the server. It takes a very long time to start, and it is just the time taken for loading the config file listing the available solr cores : solr.xml. To solve this, in the "load" function, I just replace the line 532 of CoreContainer.java (Apache Solr 4.0 release) from "p.setCoreProperties(readProperties(cfg, node));" to "p.setCoreProperties(readProperties(cfg, node.getParentNode().removeChild(node)));" Now, it takes a few seconds to start the server > loading solr.xml takes a lot of time when there are lot of cores > > > Key: SOLR-1575 > URL: https://issues.apache.org/jira/browse/SOLR-1575 > Project: Solr > Issue Type: Improvement > Components: multicore >Reporter: Noble Paul >Priority: Minor > Fix For: 1.5 > > > We have around 38K cores which takes up around 25 mins to start the server. > Keep in mind that no cores are started and this is the time taken for loading > the solr.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
[
https://issues.apache.org/jira/browse/LUCENE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
olivier soyez closed LUCENE-3119.
-
> Special Character & Hightlighting issues after 3.1.0 update
> ---
>
> Key: LUCENE-3119
> URL: https://issues.apache.org/jira/browse/LUCENE-3119
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/highlighter
>Affects Versions: 3.1
> Environment: ubuntu 10.10, java version "1.6.0_02"
>Reporter: olivier soyez
>Priority: Minor
>
> I have the same issue describe here :
> http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
> Looks like the highlighting code changed. Using the example doc, with 1.4 I
> get :
> http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
> {noformat}
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes: êâîôû"]}}}
> With 3.1, this now looks like :
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes:
> êâîôû"]}}}
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
[
https://issues.apache.org/jira/browse/LUCENE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
olivier soyez resolved LUCENE-3119.
---
Resolution: Not A Problem
> Special Character & Hightlighting issues after 3.1.0 update
> ---
>
> Key: LUCENE-3119
> URL: https://issues.apache.org/jira/browse/LUCENE-3119
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/highlighter
>Affects Versions: 3.1
> Environment: ubuntu 10.10, java version "1.6.0_02"
>Reporter: olivier soyez
>Priority: Minor
>
> I have the same issue describe here :
> http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
> Looks like the highlighting code changed. Using the example doc, with 1.4 I
> get :
> http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
> {noformat}
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes: êâîôû"]}}}
> With 3.1, this now looks like :
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes:
> êâîôû"]}}}
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
[
https://issues.apache.org/jira/browse/LUCENE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036173#comment-13036173
]
olivier soyez commented on LUCENE-3119:
---
Yes, it is just due to HtmlEncoder in solrconfig.xml. Thank you so much, it's
working!
> Special Character & Hightlighting issues after 3.1.0 update
> ---
>
> Key: LUCENE-3119
> URL: https://issues.apache.org/jira/browse/LUCENE-3119
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/highlighter
>Affects Versions: 3.1
> Environment: ubuntu 10.10, java version "1.6.0_02"
>Reporter: olivier soyez
>Priority: Minor
>
> I have the same issue describe here :
> http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
> Looks like the highlighting code changed. Using the example doc, with 1.4 I
> get :
> http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
> {noformat}
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes: êâîôû"]}}}
> With 3.1, this now looks like :
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes:
> êâîôû"]}}}
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
[
https://issues.apache.org/jira/browse/LUCENE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
olivier soyez updated LUCENE-3119:
--
Description:
I have the same issue describe here :
http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
Looks like the highlighting code changed. Using the example doc, with 1.4 I get
:
http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
{noformat}
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes: êâîôû"]}}}
With 3.1, this now looks like :
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes:
êâîôû"]}}}
{noformat}
was:
I have the same issue describe here :
http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
Looks like the highlighting code changed. Using the example doc, with 1.4 I get
:
http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes: êâîôû"]}}}
With 3.1, this now looks like :
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes:
{noformat}
êâîôû"]}}}
{noformat}
> Special Character & Hightlighting issues after 3.1.0 update
> ---
>
> Key: LUCENE-3119
> URL: https://issues.apache.org/jira/browse/LUCENE-3119
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/highlighter
>Affects Versions: 3.1
> Environment: ubuntu 10.10, java version "1.6.0_02"
>Reporter: olivier soyez
>
> I have the same issue describe here :
> http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
> Looks like the highlighting code changed. Using the example doc, with 1.4 I
> get :
> http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
> {noformat}
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes: êâîôû"]}}}
> With 3.1, this now looks like :
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes:
> êâîôû"]}}}
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
[
https://issues.apache.org/jira/browse/LUCENE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
olivier soyez updated LUCENE-3119:
--
Priority: Minor (was: Major)
> Special Character & Hightlighting issues after 3.1.0 update
> ---
>
> Key: LUCENE-3119
> URL: https://issues.apache.org/jira/browse/LUCENE-3119
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/highlighter
>Affects Versions: 3.1
> Environment: ubuntu 10.10, java version "1.6.0_02"
>Reporter: olivier soyez
>Priority: Minor
>
> I have the same issue describe here :
> http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
> Looks like the highlighting code changed. Using the example doc, with 1.4 I
> get :
> http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
> {noformat}
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes: êâîôû"]}}}
> With 3.1, this now looks like :
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes:
> êâîôû"]}}}
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
[
https://issues.apache.org/jira/browse/LUCENE-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
olivier soyez updated LUCENE-3119:
--
Description:
I have the same issue describe here :
http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
Looks like the highlighting code changed. Using the example doc, with 1.4 I get
:
http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes: êâîôû"]}}}
With 3.1, this now looks like :
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes:
{noformat}
êâîôû"]}}}
{noformat}
was:
I have the same issue describe here :
http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
Looks like the highlighting code changed. Using the example doc, with 1.4 I get
:
http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes: êâîôû"]}}}
With 3.1, this now looks like :
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes:
êâîôû"]}}}
> Special Character & Hightlighting issues after 3.1.0 update
> ---
>
> Key: LUCENE-3119
> URL: https://issues.apache.org/jira/browse/LUCENE-3119
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/highlighter
>Affects Versions: 3.1
> Environment: ubuntu 10.10, java version "1.6.0_02"
>Reporter: olivier soyez
>
> I have the same issue describe here :
> http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
> Looks like the highlighting code changed. Using the example doc, with 1.4 I
> get :
> http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes: êâîôû"]}}}
> With 3.1, this now looks like :
> "highlighting":{
> "UTF8TEST":{
> "features":["eaiou with circumflexes:
> {noformat}
> êâîôû"]}}}
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-3119) Special Character & Hightlighting issues after 3.1.0 update
Special Character & Hightlighting issues after 3.1.0 update
---
Key: LUCENE-3119
URL: https://issues.apache.org/jira/browse/LUCENE-3119
Project: Lucene - Java
Issue Type: Bug
Components: modules/highlighter
Affects Versions: 3.1
Environment: ubuntu 10.10, java version "1.6.0_02"
Reporter: olivier soyez
I have the same issue describe here :
http://lucene.472066.n3.nabble.com/Special-Character-amp-Hightlighting-issues-after-3-1-0-update-tc2820405.html#none
Looks like the highlighting code changed. Using the example doc, with 1.4 I get
:
http://localhost:8983/solr/select?q=features:circumflexes&hl=true&hl.fl=features&wt=json&indent=true
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes: êâîôû"]}}}
With 3.1, this now looks like :
"highlighting":{
"UTF8TEST":{
"features":["eaiou with circumflexes:
êâîôû"]}}}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
