Update schema.xml without restarting Solr?
Hi, The wiki for Solr talks about the schema.xml, and it seems that changes in this file requires a restart of Solr before they have effect. In the wiki it says: How can I rebuild my index from scratch if I change my schema? The most efficient/complete way is to... 1. Stop your application server 2. Change your schema.xml file 3. Delete the index directory in your data directory 4. Start your application server (Solr will detect that there is no existing index and make a new one) 5. Re-Index your data If the permission scheme of your server does not allow you to manually delete the index directory an alternate technique is... 1. Stop your application server 2. Change your schema.xml file 3. Start your application server 4. Use the "match all docs" query in a delete by query command: *:* 5. Send an command. 6. Re-Index your data Is this really the case? I find that quite strange that you need to restart solr for a change in the schema.xml. The way we plan to use Solr together with a Content Management System is that the authors/editors can create new article/document types when needed, without any need to restart anything. The CMS itself has full support for this. But we need Solr to also support this. Is that possible? Like a simple command, maybe, that would trigger Solr to re-read it's schema.xml file. If this is not possible to do, is it really necessary to restart the entire application server for a change in schema.xml to have effect? Or only the solr webapp? Regards /Jimi
Re: Update schema.xml without restarting Solr?
Hello Team , I can’t change the schema name of an existing index. I want to change the schema name from “schemaV1" to “schemaV2 for one of the existing index curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type: application/json" -d '{"schema":"schemaV2"}' the funny part is curl is not returning anything such either error or success. I expect the next GET to return {"schema":" schemaV2"} but it still returns {"schema":" schemaV1"} I am using solr 4.10.4 version. Have you faced any such issue ?? Regards Lingesh M -- View this message in context: http://lucene.472066.n3.nabble.com/Update-schema-xml-without-restarting-Solr-tp484263p4331225.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Update schema.xml without restarting Solr?
> the funny part is curl is not returning anything such either error or success can you add -v or so to curl, to see http status code at least? On Fri, Apr 21, 2017 at 11:20 AM, Lingeshm wrote: > Hello Team , > > I can’t change the schema name of an existing index. > > I want to change the schema name from “schemaV1" to “schemaV2 for one of > the > existing index > > curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type: > application/json" -d '{"schema":"schemaV2"}' > > the funny part is curl is not returning anything such either error or > success. I expect the next GET to return {"schema":" schemaV2"} but it > still > returns {"schema":" schemaV1"} > > I am using solr 4.10.4 version. > > Have you faced any such issue ?? > > Regards > Lingesh M > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/Update-schema-xml-without-restarting-Solr-tp484263p4331225.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Sincerely yours Mikhail Khludnev
Re: Update schema.xml without restarting Solr?
On 4/21/2017 2:20 AM, Lingeshm wrote: > I can’t change the schema name of an existing index. > > I want to change the schema name from “schemaV1" to “schemaV2 for one of the > existing index > > curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type: > application/json" -d '{"schema":"schemaV2"}' > > the funny part is curl is not returning anything such either error or > success. I expect the next GET to return {"schema":" schemaV2"} but it still > returns {"schema":" schemaV1"} > > I am using solr 4.10.4 version. I have never seen a URL path like "/search/index/my_idx-H" for Solr. Typically any URL path for Solr will start with "/solr". With a 4.x version, you might have changed the context path in the container, in newer versions the admin UI will break if that is changed. If you are attempting to use the Schema API, then your URL path would contain "/schema" somewhere, which is missing. Can you point to a location in the Solr documentation that describes what you are trying to do? I have never seen anything like the JSON that you are sending. The Schema API documentation for version 6.6 (not yet released) doesn't mention any ability to change the name. Such a change would cause no difference in how the schema works even if it were possible -- it's cosmetic, and something somebody who uses Solr will never see. Thanks, Shawn
Re: Update schema.xml without restarting Solr?
I would say that all this points at existence of middle-ware in front of Solr. Therefore, the next action would be to identify the middle-ware and ask this question on _their_ mailing list. Regards, Alex. http://www.solr-start.com/ - Resources for Solr users, new and experienced On 21 April 2017 at 08:38, Shawn Heisey wrote: > On 4/21/2017 2:20 AM, Lingeshm wrote: >> I can’t change the schema name of an existing index. >> >> I want to change the schema name from “schemaV1" to “schemaV2 for one of the >> existing index >> >> curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type: >> application/json" -d '{"schema":"schemaV2"}' >> >> the funny part is curl is not returning anything such either error or >> success. I expect the next GET to return {"schema":" schemaV2"} but it still >> returns {"schema":" schemaV1"} >> >> I am using solr 4.10.4 version. > > I have never seen a URL path like "/search/index/my_idx-H" for Solr. > Typically any URL path for Solr will start with "/solr". With a 4.x > version, you might have changed the context path in the container, in > newer versions the admin UI will break if that is changed. If you are > attempting to use the Schema API, then your URL path would contain > "/schema" somewhere, which is missing. > > Can you point to a location in the Solr documentation that describes > what you are trying to do? > > I have never seen anything like the JSON that you are sending. The > Schema API documentation for version 6.6 (not yet released) doesn't > mention any ability to change the name. Such a change would cause no > difference in how the schema works even if it were possible -- it's > cosmetic, and something somebody who uses Solr will never see. > > Thanks, > Shawn >
Re: Update schema.xml without restarting Solr?
The way we plan to use Solr together with a Content Management System is that the authors/editors can create new article/document types when needed, without any need to restart anything. Perhaps consider using dynamic fields if you need new fields: http://wiki.apache.org/solr/SchemaXml#head-82dba16404c8e3318021320638b669b3a6d780d0 - - - - Do you really need to change the schema? Your CMS will transform from blog software to invoice tracking to forum without restarting? With clever use of dynamic fields this should be possible too. Consider changing the schema the same way you consider changing on for SQL -- you will likely change it and test for a while before deploying. That said, the new MultiCore stuff in trunk allows you to reload a core at runtime: http://wiki.apache.org/solr/MultiCore#head-429a06cb83e1ce7b06857fd03c38d1200c4bcfc1 If this is not possible to do, is it really necessary to restart the entire application server for a change in schema.xml to have effect? Or only the solr webapp? correct, you only need to restart the solr webapp for all settings to get reloaded. ryan
Re: Update schema.xml without restarting Solr?
Quoting Ryan McKinley <[EMAIL PROTECTED]>: The way we plan to use Solr together with a Content Management System is that the authors/editors can create new article/document types when needed, without any need to restart anything. Do you really need to change the schema? Your CMS will transform from blog software to invoice tracking to forum without restarting? Well, it will not transform in that way :) But new article/document types gets created every once in a while. And the CMS in itself has no restart-requirements before they can be used. The only limitation we have is that new or updated presentation jsp pages needs to be transfered to the live server, and this is done by the build (and that involves a tomcat restart). But sometimes we have no need to add/update jsp files, but we still want to input content based on new or updated article types, and index them, without restarting. And we also want to be able to modify existing article types, so that the fields gets indexed in a different way maybe, and reindex all articles/documents while people still can do searches on the site. With clever use of dynamic fields this should be possible too. Is that true also for changes made on how a specific field should be indexed? The wiki example is: What if we want to change this from stored="true" to stored="false" or something similar, without renaming the field? But I must say, that in general I consider type information in the name to be ugly design. What if we have a number field myNumber that we first define as an int (or sint, to use solr semantics), and we call that myNumber_i in solr? If we later feel that we should change this to be a slong, then we need to change the name of the field to nyNumber_f. Consider changing the schema the same way you consider changing on for SQL -- you will likely change it and test for a while before deploying. That said, the new MultiCore stuff in trunk allows you to reload a core at runtime: http://wiki.apache.org/solr/MultiCore#head-429a06cb83e1ce7b06857fd03c38d1200c4bcfc1 Ok, I'll look into that. Thanks. correct, you only need to restart the solr webapp for all settings to get reloaded. Ok, thanks. The wiki was unclear on this one. /Jimi
Re: Update schema.xml without restarting Solr?
[EMAIL PROTECTED] wrote: Quoting Ryan McKinley <[EMAIL PROTECTED]>: The way we plan to use Solr together with a Content Management System is that the authors/editors can create new article/document types when needed, without any need to restart anything. Do you really need to change the schema? Your CMS will transform from blog software to invoice tracking to forum without restarting? Well, it will not transform in that way :) But new article/document types gets created every once in a while. And the CMS in itself has no restart-requirements before they can be used. The only limitation we have is that new or updated presentation jsp pages needs to be transfered to the live server, and this is done by the build (and that involves a tomcat restart). But sometimes we have no need to add/update jsp files, but we still want to input content based on new or updated article types, and index them, without restarting. And we also want to be able to modify existing article types, so that the fields gets indexed in a different way maybe, and reindex all articles/documents while people still can do searches on the site. With clever use of dynamic fields this should be possible too. Is that true also for changes made on how a specific field should be indexed? The wiki example is: What if we want to change this from stored="true" to stored="false" or something similar, without renaming the field? But I must say, that in general I consider type information in the name to be ugly design. What if we have a number field myNumber that we first define as an int (or sint, to use solr semantics), and we call that myNumber_i in solr? If we later feel that we should change this to be a slong, then we need to change the name of the field to nyNumber_f. In general, you need to be very careful when you change the schema without reindexing. Many changes will break all search, some may be just fine. for example, if you change sint to slong anything already indexed as an "sint" will be incompatible with the current settings. Likewise, if you change the "stored" settings, it will only change thing for the incoming documents. I'm not saying its impossible -- it is just something you need to be very careful with because a seemingly simple change can break a lot of things (until you reinidex) That said -- with the MultiCore stuff and SOLR-139, it is possible to build a system that can be dynamically updated and managed. ryan
Re: Update schema.xml without restarting Solr?
Quoting Ryan McKinley <[EMAIL PROTECTED]>: In general, you need to be very careful when you change the schema without reindexing. Many changes will break all search, some may be just fine. for example, if you change sint to slong anything already indexed as an "sint" will be incompatible with the current settings. This example, with changing from sint to slong, was just an example of bad > design in my opinion, not an example of what we need to do without reindexing. Actually, as long as a reindexing doesn't take too long time I don't have any problem with reindexing per say. But I would like it to happen without having the search functionality disabled in the mean time. I guess the MultiCore stuff is the option here. Thanks for your input, Ryan. /Jimi
Re: Update schema.xml without restarting Solr?
Top often requested feature: 1. Make the option on using the "RAMDirectory" to hook in Terracotta( billion(s) of items in an index anyone?..it would be possible using this.) 2. Make the "schema.xml" configurable at runtime, not really sure the best way to address this, because changing the schema would require "re-indexing" the documents. Terracotta: http://www.terracotta.org/ On Tue, Mar 25, 2008 at 11:27 AM, <[EMAIL PROTECTED]> wrote: > Hi, > > The wiki for Solr talks about the schema.xml, and it seems that > changes in this file requires a restart of Solr before they have effect. > > In the wiki it says: > > > How can I rebuild my index from scratch if I change my schema? > > The most efficient/complete way is to... > > 1. Stop your application server > 2. Change your schema.xml file > 3. Delete the index directory in your data directory > 4. Start your application server (Solr will detect that there is > no existing index and make a new one) > 5. Re-Index your data > > If the permission scheme of your server does not allow you to manually > delete the index directory an alternate technique is... > > 1. Stop your application server > 2. Change your schema.xml file > 3. Start your application server > 4. Use the "match all docs" query in a delete by query command: > *:* > 5. Send an command. > 6. Re-Index your data > > > Is this really the case? I find that quite strange that you need to > restart solr for a change in the schema.xml. The way we plan to use > Solr together with a Content Management System is that the > authors/editors can create new article/document types when needed, > without any need to restart anything. The CMS itself has full support > for this. But we need Solr to also support this. Is that possible? > Like a simple command, maybe, that would trigger > Solr to re-read it's schema.xml file. > > If this is not possible to do, is it really necessary to restart the > entire application server for a change in schema.xml to have effect? > Or only the solr webapp? > > Regards > /Jimi > -- Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)
Re: Update schema.xml without restarting Solr?
Quoting Jeryl Cook <[EMAIL PROTECTED]>: 2. Make the "schema.xml" configurable at runtime, not really sure the best way to address this, because changing the schema would require "re-indexing" the documents. Isn't the best way to address this just to leave it to the persons that integrate solr into their system? I mean, if a change in the schema only effects 1% of all documents, then it's a bad idea to reindex them all (at least if the dataset is big). /Jimi
Re: Update schema.xml without restarting Solr?
[EMAIL PROTECTED] wrote: > Quoting Jeryl Cook <[EMAIL PROTECTED]>: > >> 2. Make the "schema.xml" configurable at runtime, not really sure the >> best way to address this, because changing the schema would require >> "re-indexing" the documents. > > Isn't the best way to address this just to leave it to the persons that > integrate solr into their system? I mean, if a change in the schema only > effects 1% of all documents, then it's a bad idea to reindex them all > (at least if the dataset is big). Or if you're adding a new field to the schema (perhaps the most common need for editing schema.xml), you don't need to reindex any documents at all, right? Unless I'm missing something? I suppose if you add a new dynamic field specification that conflicts with existing fields, reindexing is probably a good idea, but if you're doing that... well, I probably don't want to know. Daniel
Re: Update schema.xml without restarting Solr?
Quoting Daniel Papasian <[EMAIL PROTECTED]>: [EMAIL PROTECTED] wrote: Quoting Jeryl Cook <[EMAIL PROTECTED]>: 2. Make the "schema.xml" configurable at runtime, not really sure the best way to address this, because changing the schema would require "re-indexing" the documents. Isn't the best way to address this just to leave it to the persons that integrate solr into their system? I mean, if a change in the schema only effects 1% of all documents, then it's a bad idea to reindex them all (at least if the dataset is big). Or if you're adding a new field to the schema (perhaps the most common need for editing schema.xml), you don't need to reindex any documents at all, right? Unless I'm missing something? Well, it all depends on if that "field" (not solar/lucene field) exists on the already indexed material, but that particular field was never indexed. Lets say that we have a bunch of articles, that has a field "author" that someone decided that it doesn't need to be in the index. But then later he changes his mind, and add the author field to the schema. In this case all articles that has a populated author field should now be reindexed. I suppose if you add a new dynamic field specification that conflicts with existing fields, reindexing is probably a good idea, but if you're doing that... well, I probably don't want to know. I must say that I'm abit confused by these dynamic fields. Can someone tell me if there is any reasonable use of dynamic fields without having the "variable type" (for example i for int/sint) in the name? /Jimi
Re: Update schema.xml without restarting Solr?
Jeryl Cook wrote: Top often requested feature: 1. Make the option on using the "RAMDirectory" to hook in Terracotta( billion(s) of items in an index anyone?..it would be possible using this.) This is noted in: https://issues.apache.org/jira/browse/SOLR-465 Out of cueriosity, any sense of performance with a terracotta index? It seems like it would have to be *substantially* slower. Also, if it is a RAM directly, does it persist? If your looking to support billions of docs, perhaps consider: http://wiki.apache.org/solr/DistributedSearch ryan
Re: Update schema.xml without restarting Solr?
[EMAIL PROTECTED] wrote: Quoting Daniel Papasian <[EMAIL PROTECTED]>: Or if you're adding a new field to the schema (perhaps the most common need for editing schema.xml), you don't need to reindex any documents at all, right? Unless I'm missing something? Well, it all depends on if that "field" (not solar/lucene field) exists on the already indexed material, but that particular field was never indexed. Lets say that we have a bunch of articles, that has a field "author" that someone decided that it doesn't need to be in the index. But then later he changes his mind, and add the author field to the schema. In this case all articles that has a populated author field should now be reindexed. Yeah, I guess the use case I was thinking of was someone who had multiple different types of content in their index (say, articles, events, organizations) and when they added a new content type (book review) if they found the need to add a new field for that content type (say, publisher) that would only be relevant for that type -- as you're adding it before any data that would have it was indexed, I believe you'd be fine making that schema change without reindexing anything. I suppose if you add a new dynamic field specification that conflicts with existing fields, reindexing is probably a good idea, but if you're doing that... well, I probably don't want to know. I must say that I'm abit confused by these dynamic fields. Can someone tell me if there is any reasonable use of dynamic fields without having the "variable type" (for example i for int/sint) in the name? Well, perhaps this is fulfilling your requirement on a technicality, but there's always higher order types... Offhand, I can think of things where you might want to define a dynamic field like *_propername or *_cost and then you'd be able to use fields like author_propername or editor_propername, or book_cost or volume_cost or what have you. Daniel
Re: Update schema.xml without restarting Solr?
Hey Ryan, why do you say a Lucene/Solr index served via Terracotta would be substantially slower? I often wanted to try Terracotta + Lucene, but... time. Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Ryan McKinley <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Wednesday, March 26, 2008 10:52:45 AM Subject: Re: Update schema.xml without restarting Solr? Jeryl Cook wrote: > Top often requested feature: > 1. Make the option on using the "RAMDirectory" to hook in Terracotta( > billion(s) of items in an index anyone?..it would be possible using > this.) This is noted in: https://issues.apache.org/jira/browse/SOLR-465 Out of cueriosity, any sense of performance with a terracotta index? It seems like it would have to be *substantially* slower. Also, if it is a RAM directly, does it persist? If your looking to support billions of docs, perhaps consider: http://wiki.apache.org/solr/DistributedSearch ryan
Re: Update schema.xml without restarting Solr?
just intuition - haven't tried it, so i'd love to be proved wrong. Instrumenting Objects and magically passing them around seems like it would be slower then a tuned approach used in SOLR-303. It looks like they have a lucene example: http://www.terracotta.org/confluence/display/integrations/Lucene Also, i don't understand how terracotta could get lucene past the Integer.MAX_VALUE limit because it does not change the API, it works within it. ryan Otis Gospodnetic wrote: Hey Ryan, why do you say a Lucene/Solr index served via Terracotta would be substantially slower? I often wanted to try Terracotta + Lucene, but... time. Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Ryan McKinley <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Wednesday, March 26, 2008 10:52:45 AM Subject: Re: Update schema.xml without restarting Solr? Jeryl Cook wrote: Top often requested feature: 1. Make the option on using the "RAMDirectory" to hook in Terracotta( billion(s) of items in an index anyone?..it would be possible using this.) This is noted in: https://issues.apache.org/jira/browse/SOLR-465 Out of cueriosity, any sense of performance with a terracotta index? It seems like it would have to be *substantially* slower. Also, if it is a RAM directly, does it persist? If your looking to support billions of docs, perhaps consider: http://wiki.apache.org/solr/DistributedSearch ryan
Re: Update schema.xml without restarting Solr?
On Wed, Mar 26, 2008 at 4:41 PM, Ryan McKinley <[EMAIL PROTECTED]> wrote: > just intuition - haven't tried it, so i'd love to be proved wrong. > Instrumenting Objects and magically passing them around seems like it > would be slower then a tuned approach used in SOLR-303. Yep, that's my sense too. No magic solutions when it comes to scalability. -Yonik
Re: Update schema.xml without restarting Solr?
i wouldn't call Terracotta approach magic(smile)..., it's being used quite a bit in many scalable high performing projects... i personally used Terracotta and Lucene, and it worked but did not try to "cluster" it with multiple terracotta(workers) across nodes , and the Terracotta(master)..just a single box with two tomcat instances... However "talk is cheap", if I have the time over the next few weeks ill make a bench mark test based on the "Terracotta and Lucene", with maybe 3 nodes?and a 1 million documents.. maybe some others can do the same :).. FYI: http://www.terracotta.org/confluence/display/tcforge/Proposal+-+Terracotta+for+Lucene Jeryl Cook On Wed, Mar 26, 2008 at 5:16 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote: > On Wed, Mar 26, 2008 at 4:41 PM, Ryan McKinley <[EMAIL PROTECTED]> wrote: > > just intuition - haven't tried it, so i'd love to be proved wrong. > > Instrumenting Objects and magically passing them around seems like it > > would be slower then a tuned approach used in SOLR-303. > > Yep, that's my sense too. No magic solutions when it comes to scalability. > > -Yonik > -- Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)
Re: Update schema.xml without restarting Solr?
: > Top often requested feature: : > 1. Make the option on using the "RAMDirectory" to hook in Terracotta( : > billion(s) of items in an index anyone?..it would be possible using : > this.) : : This is noted in: https://issues.apache.org/jira/browse/SOLR-465 ...and if people posted comments in the issue saying they tried the patch and it worked well for them (or didn't work well and described their use case and what they think a better API would be) that would help raise the visibility of the visibility of hte issue and increase the likelyhod of geting it commited. API changes, particularly new "plugin hooks" have to be made carefully because once they are released they have to be supported indefinitley. so we need people to really help us test them out while they are still just patches. -Hoss