[ https://issues.apache.org/jira/browse/SOLR-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628399#comment-13628399 ]
Commit Tag Bot commented on SOLR-4658: -------------------------------------- [branch_4x commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1466724 SOLR-4658: - Fix class name shortening issues, for SimilarityFactory's, FieldType's, and analysis factories: persisted class names' original forms are preserved in both shortened form (e.g. 'solr.TextField') and fully qualified form (e.g. 'org.apache.solr.schema.TextField') - luceneMatchVersion on analysis factories is now persisted only when explicitly specified in the original; previously the default value was persisted when it was absent in the original - Add MockTokenFilterFactory > In preparation for dynamic schema modification via REST API, add a "managed" > schema facility > -------------------------------------------------------------------------------------------- > > Key: SOLR-4658 > URL: https://issues.apache.org/jira/browse/SOLR-4658 > Project: Solr > Issue Type: Sub-task > Components: Schema and Analysis > Reporter: Steve Rowe > Assignee: Steve Rowe > Priority: Minor > Fix For: 4.3, 5.0 > > Attachments: SOLR-4658-fix-serialization.patch, SOLR-4658.patch, > SOLR-4658.patch > > > The idea is to have a set of configuration items in {{solrconfig.xml}}: > {code:xml} > <schema managed="true" mutable="true" > managedSchemaResourceName="managed-schema"/> > {code} > It will be a precondition for future dynamic schema modification APIs that > {{mutable="true"}}. {{solrconfig.xml}} parsing will fail if > {{mutable="true"}} but {{managed="false"}}. > When {{managed="true"}}, and the resource named in > {{managedSchemaResourceName}} doesn't exist, Solr will automatically upgrade > the schema to "managed": the non-managed schema resource (typically > {{schema.xml}}) is parsed and then persisted at {{managedSchemaResourceName}} > under {{$solrHome/$collectionOrCore/conf/}}, or on ZooKeeper at > {{/configs/$configName/}}, and the non-managed schema resource is renamed by > appending {{.bak}}, e.g. {{schema.xml.bak}}. > Once the upgrade has taken place, users can get the full schema from the > {{/schema?wt=schema.xml}} REST API, and can use this as the basis for > modifications which can then be used to manually downgrade back to > non-managed schema: put the {{schema.xml}} in place, then add {{<schema > managed="false"/>}} to {{solrconfig.xml}} (or remove the whole {{<schema/>}} > element, since {{managed="false"}} is the default). > If users take no action, then Solr behaves the same as always: the example > {{solrconfig.xml}} will include {{<schema managed="false" ...>}}. > For a discussion of rationale for this feature, see > [~hossman_luc...@fucit.org]'s post to the solr-user mailing list in the > thread "Dynamic schema design: feedback requested" > [http://markmail.org/message/76zj24dru2gkop7b]: > > {quote} > Ignoring for a moment what format is used to persist schema information, I > think it's important to have a conceptual distinction between "data" that > is managed by applications and manipulated by a REST API, and "config" > that is managed by the user and loaded by solr on init -- or via an > explicit "reload config" REST API. > Past experience with how users percieve(d) solr.xml has heavily reinforced > this opinion: on one hand, it's a place users must specify some config > information -- so people wnat to be able to keep it in version control > with other config files. On the other hand it's a "live" data file that > is rewritten by solr when cores are added. (God help you if you want do a > rolling deploy a new version of solr.xml where you've edited some of the > config values while simultenously clients are creating new SolrCores) > As we move forward towards having REST APIs that treat schema information > as "data" that can be manipulated, I anticipate the same types of > confusion, missunderstanding, and grumblings if we try to use the same > pattern of treating the existing schema.xml (or some new schema.json) as a > hybrid configs & data file. "Edit it by hand if you want, the /schema/* > REST API will too!" ... Even assuming we don't make any of the same > technical mistakes that have caused problems with solr.xml round tripping > in hte past (ie: losing comments, reading new config options that we > forget to write back out, etc...) i'm fairly certain there is still going > to be a lot of things that will loook weird and confusing to people. > (XML may bave been designed to be both "human readable & writable" and > "machine readable & writable", but practically speaking it's hard have a > single XML file be "machine and human readable & writable") > I think it would make a lot of sense -- not just in terms of > implementation but also for end user clarity -- to have some simple, > straightforward to understand caveats about maintaining schema > information... > 1) If you want to keep schema information in an authoritative config file > that you can manually edit, then the /schema REST API will be read only. > 2) If you wish to use the /schema REST API for read and write operations, > then schema information will be persisted under the covers in a data store > whose format is an implementation detail just like the index file format. > 3) If you are using a schema config file and you wish to switch to using > the /schema REST API for managing schema information, there is a > tool/command/API you can run to so. > 4) if you are using the /schema REST API for managing schema information, > and you wish to switch to using a schema config file, there is a > tool/command/API you can run to export the schema info if a config file > format. > ...wether of not the "under the covers in a data store" used by the REST > API is JSON, or some binary data, or an XML file just schema.xml w/o > whitespace/comments should be an implementation detail. Likewise is the > question of wether some new config file formats are added -- it shouldn't > matter. > If it's config it's config and the user owns it. > If it's data it's data and the system owns it. > : is the risk they take if they want to manually edit it - it's no > : different than today when you edit the file and do a Core reload or > : something. I think we can improve some validation stuff around that, but > : it doesn't seem like a show stopper to me. > The new risk is multiple "actors" (both the user, and Solr) editing the > file concurrently, and info that might be lost due to Solr reading the > file, manpulating internal state, and then writing the file back out. > Eg: User hand edits may be lost if they happen on disk during Solr's > internal manpulation of data. API edits may be reflected in the internal > state, but lost if the User writes the file directly and then does a core > reload, etc.... > : At a minimum, I think the user should be able to start with a hand > : modified file. Many people *heavily* modify the example schema to fit > : their use case. If you have to start doing that by making 50 rest API > : calls, that's pretty rough. Once you get your schema nice and happy, you > : might script out those rest calls, but initially, it's much > : faster/easier to whack the schema into place in a text editor IMO. > I don't think there is any disagreement about that. The ability to say > "my schema is a config file and i own it" should always exist (remove > it over my dead body) > The question is what trade offs to expect/require for people who would > rather use an API to manipulate these things -- i don't think it's > unreasable to say "if you would like to manipulate the schema using an > API, then you give up the ability to manipulate it as a config file on > disk" > ("if you want the /schema API to drive your car, you have to take your > foot of hte pedals and let go of the steering wheel") > {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org