Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi, excellent, good to hear that you got this resolved! Also, thanks for keeping the list in the loop -- this might help in case anyone else runs into the same problem sometime in the future. So it looks like the initial issue was that you had dc.type listed as a hidden metadata field; then when you changed that, update-discovery-index -b failed for some reason. It's strange that -f worked when -b didn't; I hope this isn't a sign of some deeper problem. However, I'm not sure whether -f will be sufficient when changing the filters; you did after all run -b at some point, even if it only went partway through indexing all your items. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Issue resolved, finally! =) update-discovery-index -f ran just fine and indexed everything. Now all items have the type field and can be filtered correctly. Don't know what happened to update-discovery-index -b to fail, but -f is enough to update the index for a new filter. Maybe an update to the documentation is in order. There it says to run update-discovery-index -b. https://wiki.duraspace.org/display/DSDOC3x/Discovery#Discovery-ConfiguringlistsofsidebarFacetsandsearchFilters Thank you all for helping me with this. Ats, Alcides Carlos de Moraes Neto 2013/8/8 Andrea Schweer schw...@waikato.ac.nz Hi, On 08/08/13 11:55, Alcides Carlos de Moraes Neto wrote: I checked the catalina.out for errors but could not find anything. No, nothing looks suspicious. Very strange. I'll try an update-discovery-index -f tomorrow... Good luck and let us know what happens. It might be interesting to see whether it stops after the same item as last time, assuming it will process the items in the same order as last time (I don't know whether that's the case). cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
As it looks as not asked before... have you reindex your content after have changed the discovery configuration? as you have noted the filter are index under a different SOLR field, in your case type bean id=searchFilterType class=org.dspace.discovery.configuration.DiscoverySearchFilter property name=indexFieldName value=type/ it is derived from your indexFieldName configuration. The metadata are automatically indexed in discovery in the form schema.element.qualifier so it is right that you already have dc.type in your index but if have added the type searchFilter only recently it only affect items added or changed after this change. Hope this help, Andrea Il 07/08/2013 02:29, Andrea Schweer ha scritto: Hi, On 07/08/13 08:45, Alcides Carlos de Moraes Neto wrote: Looking at the solr logs, i see that the discovery filter is passed to SOLR as a parameter fq=type http://www22.senado.leg.br/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*]fq=type:(text) http://www22.senado.leg.br/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*]fq=type:%28text%29 Hm. Discovery has changed between 1.8.2 (which I'm still using) and 3.x and this is getting into areas that I'm not familiar with. That filter query looks for a field with the name type, not dc.type. So I assume only 2189 of your items have that field. curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=type:[*+TO+*]; should verify that (by having numFound of 2189). That fields is created by Discovery based on the search filter settings, so I think you're right in that this has something to do with your custom search filter. However, your filter looks exactly like in the Discovery documentation, so I don't know what's going wrong. Could you try curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=1q=-type:[*+TO+*]+AND+dc.type:[*+TO+*]; This should give you 1 result that has no type field but a dc.type field in the discovery solr index. Then perhaps curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=1q=type:[*+TO+*]+AND+dc.type:[*+TO+*]; to give you an item that has both type and dc.type in the discovery solr index. Then compare these two items and see if there is a pattern. Just a stab in the dark, perhaps there is a bug with discovery indexing for items with more than one value for dc.type? I suspect Kevin would be a good person to help troubleshoot this, but I don't know if he's following the mailing list at the moment. I'm cc'ing him in to increase the chances that he'll see this. cheers, Andrea -- Andrea Bollini Dipartimento Servizi e Soluzioni per l'Amministrazione Universitaria Divisione Ricerca Via dei Tizii, 6 00185 Roma, Italy tel. +39 06 44 486 087 - mob. +39 348 82 77 525 http://www.cineca.it -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
On Wed, Aug 7, 2013 at 8:36 AM, Andrea Bollini a.boll...@cineca.it wrote: The metadata are automatically indexed in discovery in the form schema.element.qualifier so it is right that you already have dc.type in your index but if have added the type searchFilter only recently it only affect items added or changed after this change. So update-discovery-index -f should fix it? Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
yes, this was my guess Il 07/08/2013 08:56, helix84 ha scritto: On Wed, Aug 7, 2013 at 8:36 AM, Andrea Bollini a.boll...@cineca.it wrote: The metadata are automatically indexed in discovery in the form schema.element.qualifier so it is right that you already have dc.type in your index but if have added the type searchFilter only recently it only affect items added or changed after this change. So update-discovery-index -f should fix it? Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Andrea Bollini Dipartimento Servizi e Soluzioni per l'Amministrazione Universitaria Divisione Ricerca Via dei Tizii, 6 00185 Roma, Italy tel. +39 06 44 486 087 - mob. +39 348 82 77 525 http://www.cineca.it -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hello everyone, First of all thanks for helping me out. I did an update-discovery-index -b when I created the new filter, this should work as well as -f, correct? Last night I scheduled a new update-discovery-index -b. I think the problem is that it did not run completely. It ran for only about 4 minutes. I noticed the items it did log as written do work with the filter. I made a pastebin with the section of the log when it starts to index, at 22:00 http://pastebin.com/17PQHLN6 it stops at 22:04 and does not continue. I do not see any exceptions, just a cocoon broken pipe. Should the index rebuild command be run with tomcat disabled? Again thank you everyone for your time. =) Ats, Alcides Carlos de Moraes Neto 2013/8/7 Andrea Bollini a.boll...@cineca.it yes, this was my guess Il 07/08/2013 08:56, helix84 ha scritto: On Wed, Aug 7, 2013 at 8:36 AM, Andrea Bollini a.boll...@cineca.it wrote: The metadata are automatically indexed in discovery in the form schema.element.qualifier so it is right that you already have dc.type in your index but if have added the type searchFilter only recently it only affect items added or changed after this change. So update-discovery-index -f should fix it? Regards, ~~helix84 Compulsory reading: DSpace Mailing List Etiquette https://wiki.duraspace.org/**display/DSPACE/Mailing+List+**Etiquettehttps://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- Andrea Bollini Dipartimento Servizi e Soluzioni per l'Amministrazione Universitaria Divisione Ricerca Via dei Tizii, 6 00185 Roma, Italy tel. +39 06 44 486 087 - mob. +39 348 82 77 525 http://www.cineca.it -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi, On 08/08/13 06:46, Alcides Carlos de Moraes Neto wrote: I think the problem is that it did not run completely. It ran for only about 4 minutes. I noticed the items it did log as written do work with the filter. ...and at least one of them has a repeated dc.type, so that's not the problem either. I made a pastebin with the section of the log when it starts to index, at 22:00 http://pastebin.com/17PQHLN6 it stops at 22:04 and does not continue. I do not see any exceptions, just a cocoon broken pipe. The cocoon broken pipe usually means that a client aborted the download of a file, so this should have nothing to do with the indexing problem. Is there anything useful in your tomcat log file around 22:04? Solr logs to that one, not to the DSpace log file. Should the index rebuild command be run with tomcat disabled? No, tomcat must be running for the discovery reindexing. That's because the reindexer talks to solr, which is a webapp running inside tomcat. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
I checked the catalina.out for errors but could not find anything. I trimmed it, beginning at 22:00 and uploaded it. (SOLR log is really verbose, too big for pastebin) https://dl.dropboxusercontent.com/u/4193365/catalina.out.22h I'll try an update-discovery-index -f tomorrow... Ats, Alcides Carlos de Moraes Neto 2013/8/7 Andrea Schweer schw...@waikato.ac.nz Hi, On 08/08/13 06:46, Alcides Carlos de Moraes Neto wrote: I think the problem is that it did not run completely. It ran for only about 4 minutes. I noticed the items it did log as written do work with the filter. ...and at least one of them has a repeated dc.type, so that's not the problem either. I made a pastebin with the section of the log when it starts to index, at 22:00 http://pastebin.com/17PQHLN6 it stops at 22:04 and does not continue. I do not see any exceptions, just a cocoon broken pipe. The cocoon broken pipe usually means that a client aborted the download of a file, so this should have nothing to do with the indexing problem. Is there anything useful in your tomcat log file around 22:04? Solr logs to that one, not to the DSpace log file. Should the index rebuild command be run with tomcat disabled? No, tomcat must be running for the discovery reindexing. That's because the reindexer talks to solr, which is a webapp running inside tomcat. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi, On 08/08/13 11:55, Alcides Carlos de Moraes Neto wrote: I checked the catalina.out for errors but could not find anything. No, nothing looks suspicious. Very strange. I'll try an update-discovery-index -f tomorrow... Good luck and let us know what happens. It might be interesting to see whether it stops after the same item as last time, assuming it will process the items in the same order as last time (I don't know whether that's the case). cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
No, it's actually the value I expect, 260078 SOLRResponse: response lst name=responseHeader int name=status0/int int name=QTime52/int lst name=params str name=indenttrue/str str name=rows0/str str name=qdc.type:text/str /lst /lst result name=response numFound=260078 start=0/ /response DSpace discovery search: http://www2.senado.leg.br/bdsf/discover?filtertype_1=typefilter_relational_operator_1=equalsfilter_1=text However, if I type ((dc.type:text)) as the main query, the results are closer to what SOLR returns: http://www2.senado.leg.br/bdsf/discover?query=%28%28dc.type%3Atext%29%29 So index is good, rebuilding it won't do any good. Guess the problem is with my custom filter? My discovery filter: bean id=searchFilterType class=org.dspace.discovery.configuration.DiscoverySearchFilter property name=indexFieldName value=type/ property name=metadataFields list valuedc.type/value /list /property /bean But I don't see anything wrong with it. Ats, Alcides Carlos de Moraes Neto 2013/8/5 Andrea Schweer schw...@waikato.ac.nz Hi, On 06/08/13 10:09, Alcides Carlos de Moraes Neto wrote: It seems SOLR has indexed dc.type correctly: response lst name=responseHeader int name=status0/int int name=QTime146/int lst name=params str name=indenttrue/str str name=rows0/str str name=qdc.type:[* TO *]/str /lst /lst result name=response numFound=260514 start=0/ /response This is the exact quantity of items in the repository. Yes, that looks like your discovery index is correct. So either something is going wrong during querying or the metadata values aren't what you expect. What numFound value do you get for this query: curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:text;http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:text Is it 2141 like when you do a search for dc.type=text via the user interface? cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Looking at the solr logs, i see that the discovery filter is passed to SOLR as a parameter fq=type http://www22.senado.leg.br/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*]fq=type:(text) This will only return: result name=response numFound=2189 start=0/ How does filter queries work in SOLR? I'm looking into it but cannot find much. Ats, Alcides Carlos de Moraes Neto 2013/8/6 Alcides Carlos de Moraes Neto alcides.n...@gmail.com No, it's actually the value I expect, 260078 SOLRResponse: response lst name=responseHeader int name=status0/int int name=QTime52/int lst name=params str name=indenttrue/str str name=rows0/str str name=qdc.type:text/str /lst /lst result name=response numFound=260078 start=0/ /response DSpace discovery search: http://www2.senado.leg.br/bdsf/discover?filtertype_1=typefilter_relational_operator_1=equalsfilter_1=text However, if I type ((dc.type:text)) as the main query, the results are closer to what SOLR returns: http://www2.senado.leg.br/bdsf/discover?query=%28%28dc.type%3Atext%29%29 So index is good, rebuilding it won't do any good. Guess the problem is with my custom filter? My discovery filter: bean id=searchFilterType class=org.dspace.discovery.configuration.DiscoverySearchFilter property name=indexFieldName value=type/ property name=metadataFields list valuedc.type/value /list /property /bean But I don't see anything wrong with it. Ats, Alcides Carlos de Moraes Neto 2013/8/5 Andrea Schweer schw...@waikato.ac.nz Hi, On 06/08/13 10:09, Alcides Carlos de Moraes Neto wrote: It seems SOLR has indexed dc.type correctly: response lst name=responseHeader int name=status0/int int name=QTime146/int lst name=params str name=indenttrue/str str name=rows0/str str name=qdc.type:[* TO *]/str /lst /lst result name=response numFound=260514 start=0/ /response This is the exact quantity of items in the repository. Yes, that looks like your discovery index is correct. So either something is going wrong during querying or the metadata values aren't what you expect. What numFound value do you get for this query: curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:text;http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:text Is it 2141 like when you do a search for dc.type=text via the user interface? cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi, On 07/08/13 08:45, Alcides Carlos de Moraes Neto wrote: Looking at the solr logs, i see that the discovery filter is passed to SOLR as a parameter fq=type http://www22.senado.leg.br/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*]fq=type:(text) http://www22.senado.leg.br/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*]fq=type:%28text%29 Hm. Discovery has changed between 1.8.2 (which I'm still using) and 3.x and this is getting into areas that I'm not familiar with. That filter query looks for a field with the name type, not dc.type. So I assume only 2189 of your items have that field. curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=type:[*+TO+*]; should verify that (by having numFound of 2189). That fields is created by Discovery based on the search filter settings, so I think you're right in that this has something to do with your custom search filter. However, your filter looks exactly like in the Discovery documentation, so I don't know what's going wrong. Could you try curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=1q=-type:[*+TO+*]+AND+dc.type:[*+TO+*]; This should give you 1 result that has no type field but a dc.type field in the discovery solr index. Then perhaps curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=1q=type:[*+TO+*]+AND+dc.type:[*+TO+*]; to give you an item that has both type and dc.type in the discovery solr index. Then compare these two items and see if there is a pattern. Just a stab in the dark, perhaps there is a bug with discovery indexing for items with more than one value for dc.type? I suspect Kevin would be a good person to help troubleshoot this, but I don't know if he's following the mailing list at the moment. I'm cc'ing him in to increase the chances that he'll see this. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with 2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Thank you for your response Andrea, Did you by any chance add dc.type to the list of hidden metadata fields? https://github.com/DSpace/DSpace/blob/dspace-3.1/dspace/config/dspace.cfg#L737 metadata.hide.dc.type = false It was indeed set to TRUE. I changed it to FALSE, restarted tomcat, and there was no effect, even after clearing the cocoon cache. I'll try removing the line from dspace.cfg and restarting tomcat again. Should this setting affect the discovery search filters? The purpose of my question was to find out why my new dc.type discovery filter is not working. It's only filtering some items, even though most of the items have the dc.type field. Here is my filter in config/spring/api/discovery.xml bean id=searchFilterType class=org.dspace.discovery.configuration.DiscoverySearchFilter property name=indexFieldName value=type/ property name=metadataFields list valuedc.type/value /list /property /bean It shows and it works partially, with just a few items. For example, a search for dc.type=text returns only 2141 items, but it should return close to 200k http://www2.senado.leg.br/bdsf/discover?filtertype_1=typefilter_relational_operator_1=equalsfilter_1=textfiltertype_2=subjectfilter_relational_operator_2=containsfilter_2=filtertype_3=dateIssuedfilter_relational_operator_3=containsfilter_3=submit_apply_filter=Aplicar -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
About metadata for item types, see latest IRUS report: http://wiki.lib.sun.ac.za/images/c/c6/Irus-item-type-report-Jan2013-v2-0.pdf On 5 August 2013 20:14, Alcides Carlos de Moraes Neto alcides.n...@gmail.com wrote: Thank you for your response Andrea, Did you by any chance add dc.type to the list of hidden metadata fields? https://github.com/DSpace/DSpace/blob/dspace-3.1/dspace/config/dspace.cfg#L737 metadata.hide.dc.type = false It was indeed set to TRUE. I changed it to FALSE, restarted tomcat, and there was no effect, even after clearing the cocoon cache. I'll try removing the line from dspace.cfg and restarting tomcat again. Should this setting affect the discovery search filters? The purpose of my question was to find out why my new dc.type discovery filter is not working. It's only filtering some items, even though most of the items have the dc.type field. Here is my filter in config/spring/api/discovery.xml bean id=searchFilterType class=org.dspace.discovery.configuration.DiscoverySearchFilter property name=indexFieldName value=type/ property name=metadataFields list valuedc.type/value /list /property /bean It shows and it works partially, with just a few items. For example, a search for dc.type=text returns only 2141 items, but it should return close to 200k http://www2.senado.leg.br/bdsf/discover?filtertype_1=typefilter_relational_operator_1=equalsfilter_1=textfiltertype_2=subjectfilter_relational_operator_2=containsfilter_2=filtertype_3=dateIssuedfilter_relational_operator_3=containsfilter_3=submit_apply_filter=Aplicar -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- *Hilton Gibson* Ubuntu Linux Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://za.linkedin.com/in/hiltongibson -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Thank you Hilton, i'll pass this to the librarians responsible. I'm just the 'computer guy'. =) Andrea, It seems removing the line from dspace.cfg resolved the issue. Now both items show dc.type. (This is a bug right? because my line was setting it to FALSE). I'll see now if discovery manages to index it correctly. Ats, Alcides Carlos de Moraes Neto 2013/8/5 Hilton Gibson hilton.gib...@gmail.com About metadata for item types, see latest IRUS report: http://wiki.lib.sun.ac.za/images/c/c6/Irus-item-type-report-Jan2013-v2-0.pdf On 5 August 2013 20:14, Alcides Carlos de Moraes Neto alcides.n...@gmail.com wrote: Thank you for your response Andrea, Did you by any chance add dc.type to the list of hidden metadata fields? https://github.com/DSpace/DSpace/blob/dspace-3.1/dspace/config/dspace.cfg#L737 metadata.hide.dc.type = false It was indeed set to TRUE. I changed it to FALSE, restarted tomcat, and there was no effect, even after clearing the cocoon cache. I'll try removing the line from dspace.cfg and restarting tomcat again. Should this setting affect the discovery search filters? The purpose of my question was to find out why my new dc.type discovery filter is not working. It's only filtering some items, even though most of the items have the dc.type field. Here is my filter in config/spring/api/discovery.xml bean id=searchFilterType class=org.dspace.discovery.configuration.DiscoverySearchFilter property name=indexFieldName value=type/ property name=metadataFields list valuedc.type/value /list /property /bean It shows and it works partially, with just a few items. For example, a search for dc.type=text returns only 2141 items, but it should return close to 200k http://www2.senado.leg.br/bdsf/discover?filtertype_1=typefilter_relational_operator_1=equalsfilter_1=textfiltertype_2=subjectfilter_relational_operator_2=containsfilter_2=filtertype_3=dateIssuedfilter_relational_operator_3=containsfilter_3=submit_apply_filter=Aplicar -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette -- *Hilton Gibson* Ubuntu Linux Systems Administrator JS Gericke Library Room 1025D Stellenbosch University Private Bag X5036 Stellenbosch 7599 South Africa Tel: +27 21 808 4100 | Cell: +27 84 646 4758 http://library.sun.ac.za http://za.linkedin.com/in/hiltongibson -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi, On 06/08/13 07:13, Alcides Carlos de Moraes Neto wrote: It seems removing the line from dspace.cfg resolved the issue. Now both items show dc.type. (This is a bug right? because my line was setting it to FALSE). I think we can argue about this being a bug or not :) The dspace.cfg comments around that setting tell you to set it to true when you want to hide a metadata field. All fields not mentioned are implicitly set to not hidden, so you should never set these lines to false. However, perhaps this should be mentioned somewhere. I'll see now if discovery manages to index it correctly. It looks like this setting should not affect Discovery -- see here for documentation on how to tell Discovery to exclude metadata fields: https://wiki.duraspace.org/display/DSDOC3x/Discovery#Discovery-GeneralDiscoverysettings%28config/modules/discovery.cfg%29 So your Discovery issue may be a separate one. If you have command line access to your server, could you try querying solr directly to check how many items have dc.type indexed at all? This might help determine whether the problem is in the user interface or somewhere deeper. Something like this should work: curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*]; In the output you should get a line similar to this one: result name=response numFound=12345 start=0/ where your numFound should be the close to 200k that you're expecting. cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
2013/8/5 Andrea Schweer schw...@waikato.ac.nz If you have command line access to your server, could you try querying solr directly to check how many items have dc.type indexed at all? This might help determine whether the problem is in the user interface or somewhere deeper. Something like this should work: curl --globoff http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:[*+TO+*] In the output you should get a line similar to this one: result name=response numFound=12345 start=0/ Thank you Andrea. It seems SOLR has indexed dc.type correctly: response lst name=responseHeader int name=status0/int int name=QTime146/int lst name=params str name=indenttrue/str str name=rows0/str str name=qdc.type:[* TO *]/str /lst /lst result name=response numFound=260514 start=0/ /response This is the exact quantity of items in the repository. Ats, Alcides Carlos de Moraes Neto -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi, On 06/08/13 10:09, Alcides Carlos de Moraes Neto wrote: It seems SOLR has indexed dc.type correctly: response lstname="responseHeader" intname="status"0/int intname="QTime"146/int lstname="params" strname="indent"true/str strname="rows"0/str strname="q"dc.type:[* TO *]/str /lst /lst resultname="response"numFound="260514"start="0"/ /response This is the exact quantity of items in the repository. Yes, that looks like your discovery index is correct. So either something is going wrong during querying or the metadata values aren't what you expect. What numFound value do you get for this query: curl --globoff "http://127.0.0.1:8080/solr/search/select?indent=truerows=0q=dc.type:text" Is it 2141 like when you do a search for dc.type=text via the user interface? cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Re: [Dspace-tech] Missing dc.type field in mets.xml, but present in database
Hi there, On 03/08/13 07:47, Alcides Carlos de Moraes Neto wrote: I have this situation in my workplace's dspace installation. Most (if not all) of the items have the dc.type metadata set, but it is only appearing in a few instances. Check out these examples: Shows dc.type: http://www2.senado.leg.br/bdsf/handle/id/190819?show=full Does not show dc.type: http://www2.senado.leg.br/bdsf/handle/id/443620?show=full I'm not seeing dc.type in either item. Is there a command I should run to update this? I ran update-discovery-index -b yesterday, still no metadata showing. The discovery index has nothing to do with item pages. You could try clearing the cocoon cache if you made any metadata changes directly in the database. See https://wiki.duraspace.org/display/DSPACE/TechnicalFaq#TechnicalFaq-ClearingCocoon%28XMLUI%29cache Did you by any chance add dc.type to the list of hidden metadata fields? https://github.com/DSpace/DSpace/blob/dspace-3.1/dspace/config/dspace.cfg#L737 cheers, Andrea -- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
[Dspace-tech] Missing dc.type field in mets.xml, but present in database
I have this situation in my workplace's dspace installation. Most (if not all) of the items have the dc.type metadata set, but it is only appearing in a few instances. Check out these examples: Shows dc.type: http://www2.senado.leg.br/bdsf/handle/id/190819?show=full Does not show dc.type: http://www2.senado.leg.br/bdsf/handle/id/443620?show=full But the last item has the field set in the database. SQL: select m.* from metadatavalue m inner join handle h on m.item_id = h.resource_id where metadata_field_id = 66 and h.handle = 'id/443620'; Result: METADATA_VALUE_ID ITEM_ID METADATA_FIELD_ID TEXT_VALUE TEXT_LANG PLACE AUTHORITY CONFIDENCE 4409886 272094 66 notÃcia de jornal pt_BR 1 -1 4409887 272094 66 text pt_BR 2 -1 Is there a command I should run to update this? I ran update-discovery-index -b yesterday, still no metadata showing. Ats, Alcides Carlos de Moraes Neto It is dangerous to be right when the government is wrong. -Voltaire -- Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711iu=/4140/ostg.clktrk___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
