[ https://issues.apache.org/jira/browse/SOLR-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erick Erickson updated SOLR-1306: --------------------------------- Attachment: SOLR-1306.patch MUST be applied _after_ SOLR-1028 OK, here's a preliminary cut at this, no tests yet, but I was looking at logging and it seems to be doing what I want, putting up for inspection by the curious... A couple of notes: 1> It turns out that to make this work I needed to incorporate SOLR-4013 and SOLR-3980. I'd appreciate anyone looking at the synchronization I did around the member variable "loadingCores". The intent here is to keep two threads from creating the same core at the same time. 1a> I'm assuming that there is exactly one CoreContainer per JVM. Otherwise I don't understand how any of the synchronization works on the member vars.... 1b> Running a simple test things went all to hell in JMX stuff without the synchronization, apparently the multiple thread problem shows up early and often. 1c> synchronization is always "interesting", so the more eyes the better. 1d> In particular, any good suggestions about bailing out of the sleep loop? Since cores can take quite a while to warm, I'm having a hard time thinking of a good default. I suppose another attribute where the provider is mentioned. There's no reason a custom provider has to be present, so requiring a timeout from the provider doesn't seem workable. 2> I've implemented a trivial CoreDescriptorProvider for a PoC, it's at the bottom. It pre-supposes you have 4 collections, the accompanying Solr.xml is also below. 3> I'm going to put this away for a couple of hours and come back to it with fresh eyes, this copy is purely for the curious/critical... *****sample custom descriptor provider public class TestCoreContainerProvider implements CoreDescriptorProvider { @Override public CoreDescriptor getDescriptor(CoreContainer container, String name) { if (! "collection2".equals(name) && ! "collection3".equals(name) && ! "collection4".equals(name)) return null; CoreDescriptor cd = new CoreDescriptor(container, name, name); // True hack because I know the dirs are the same as the collection. return cd; } @Override public boolean isPersist(String s) { return false; } @Override public Collection<String> getCoreNames() { return new ArrayList<String>(Arrays.asList("collection2", "collection3", "collection4")); } } **** solr.xml. Note no ZK stuff. <solr persistent="false" sharedLib="../../../../../blahblah/out/artifacts/provider_jar"> <cores adminPath="/admin/cores" defaultCoreName="collection1" host="${host:}" hostPort="${jetty.port:}" hostContext="${hostContext:}" zkClientTimeout="${zkClientTimeout:15000}" coreDescriptorProviderClass="blah.TestCoreContainerProvider"> <core name="collection1" instanceDir="collection1" /> </cores> </solr> > Support pluggable persistence/loading of solr.xml details > --------------------------------------------------------- > > Key: SOLR-1306 > URL: https://issues.apache.org/jira/browse/SOLR-1306 > Project: Solr > Issue Type: New Feature > Components: multicore > Reporter: Noble Paul > Assignee: Erick Erickson > Fix For: 4.1 > > Attachments: SOLR-1306.patch > > > Persisting and loading details from one xml is fine if the no:of cores are > small and the no:of cores are few/fixed . If there are 10's of thousands of > cores in a single box adding a new core (with persistent=true) becomes very > expensive because every core creation has to write this huge xml. > Moreover , there is a good chance that the file gets corrupted and all the > cores become unusable . In that case I would prefer it to be stored in a > centralized DB which is backed up/replicated and all the information is > available in a centralized location. > We may need to refactor CoreContainer to have a pluggable implementation > which can load/persist the details . The default implementation should > write/read from/to solr.xml . And the class should be pluggable as follows in > solr.xml > {code:xml} > <solr> > <dataProvider class="com.foo.FooDataProvider" attr1="val1" attr2="val2"/> > </solr> > {code} > There will be a new interface (or abstract class ) called SolrDataProvider > which this class must implement -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org