Good job Mike! ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message----- From: Mike Vogel <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, June 17, 2014 4:19 PM To: "[email protected]" <[email protected]> Subject: RE: Curator UI editing metadata in solr >Problem solved. Updates of data from the curator UI were not working >because of the combination of: >· >The setting of org.apache.oodt.cas.filemgr.metadata.expandProduct=true, >when I set it to false I see the metadata instead of the product data. >Only > somewhat clear from reading the code exactly what this property is >intended to control other than the behavior I encountered. Can anyone >clarify? >· >Filtering out the extra fields that solr is automatically adding as part >of my ingestion pipeline from the updates that the Curator does, e.g., the > _version_, and a timestamp field are automatically added but updates of >them failed. I ignored these fields via a <processor >class="solr.IgnoreFieldUpdateProcessorFactory"> in solrconfig.xml. I¹ll >look into the right way to do that via the overrides > of the metadata set to and from solr in the OODT code. > >From: Mike Vogel > >Sent: Tuesday, June 17, 2014 3:22 PM >To: OODT >Subject: RE: Curator UI editing metadata in solr > > > >I have the fields defined in solr with a namespace of ³CAS.², see below. >My other fields don¹t have a namespace, e.g., ProtocolNumber below. I¹m >not thinking > its some configuration problem with the policy files so unless something >obvious has jumped out at someone you should stop thinking about this >till I make sure it isn¹t something else simple that I¹ve done wrong. > >Is there anything written about how to use the groups concept that is >implemented in the metadata? Is it totally optional or do some features >depend on it? > > <field name="CAS.ProductId" type="string" indexed="true" stored="true" >required="true" multiValued="false" /> > <field name="CAS.ProductName" type="string" indexed="true" >stored="true" required="true" multiValued="false" /> > <field name="CAS.ProductTypeName" type="string" indexed="true" >stored="true" required="true" multiValued="false" /> > <field name="CAS.ProductTypeId" type="string" indexed="true" >stored="true" required="true" multiValued="false" /> > <field name="CAS.ProductReceivedTime" type="date" indexed="true" >stored="true" required="false" multiValued="false" /> > <field name="CAS.ProductStructure" type="string" indexed="true" >stored="true" required="false" multiValued="false" /> > <field name="CAS.ProductTransferStatus" type="string" indexed="true" >stored="true" required="false" multiValued="false" /> > > <field name="CAS.RootReferenceOriginal" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="CAS.RootReferenceDatastore" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="CAS.RootReferenceFileSize" type="long" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="CAS.RootReferenceMimeType" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > > <field name="CAS.ReferenceOriginal" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="CAS.ReferenceDatastore" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="CAS.ReferenceFileSize" type="long" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="CAS.ReferenceMimeType" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > > <field name="ProtocolNumber" type="string" indexed="true" >stored="true" required="false" multiValued="true" /> > <field name="AssayPurpose" type="text_en" indexed="true" stored="true" >required="false" multiValued="true" /> > <field name="Target" type="text_en" indexed="true" stored="true" >required="false" multiValued="true" /> > <field name="Investigator" type="text_en" indexed="true" stored="true" >required="false" multiValued="true" /> > <field name="ProjectDescription" type="text_en" indexed="true" >stored="true" required="false" multiValued="true" /> > >From: Thomas Bennett [mailto:[email protected]] > >Sent: Tuesday, June 17, 2014 8:08 AM >To: OODT >Subject: Re: Curator UI editing metadata in solr > >Hi Mike, > > >Since the solr catalogue persists the CAS namespace protection for core >metadata values, perhaps the problem you are seeing might be caused by >this. > > > >A way for you to test it, would be to, if possible, use the CAS. >namespace for your core metadata values (i.e ProductId, should be >CAS.ProductId, etc). > > > >Question to OODT dev guys - does the curator support this? > > > > >Cheers, > >Tom > > > >------ > > > >PS: Here listed are snippets from three files with important lines >highlighted in red, to point out this issue for discussion. > > > >- filemgr/catalog/solr/Parameters.java > >- filmegr/catalog/solr/SolrCatalog.java > >- filemgr/catalog/solr/DefaultProductSerializer.java > > > >Starting with the definition of Parameters, in >filemgr/catalog/solr/Parameters.java > > > >public class Parameters { > > // the Solr unique identifier field > > public final static String ID = "id"; > > > > >public final static String NS = "CAS."; > > > > public final static String PRODUCT_ID = NS+"ProductId"; > > public final static String PRODUCT_NAME = NS+"ProductName"; > > public final static String PRODUCT_STRUCTURE = >NS+"ProductStructure"; > > public final static String PRODUCT_TRANSFER_STATUS = >NS+"ProductTransferStatus"; > > public final static String PRODUCT_RECEIVED_TIME = >NS+"ProductReceivedTime"; > > public final static String PRODUCT_TYPE_NAME = >NS+"ProductTypeName"; > > public final static String PRODUCT_TYPE_ID = NS+"ProductTypeId"; > > > > >-------------- > > > >Here is the addMedata method in filmegr/catalog/solr/SolrCatalog.java > > > > @Override > > public void addMetadata(Metadata metadata, Product product) >throws CatalogException { > > > > LOG.info("Adding metadata for >product:"+product.getProductName()); > > > > // serialize metadadta to Solr document(s) > > // replace=false i.e. add metadata to existing values > > List<String> docs = >productSerializer.serialize(product.getProductId(), metadata, false); > > > > >-------------- > > > >Looking at the appropriate method signature in >filemgr/catalog/solr/DefaultProductSerializer.java > > > > /** > > * {@inheritDoc} > > */ > > public List<String> serialize(String productId, Metadata >metadata, boolean replace) { > > > > Map<String, List<String>> fields = new HashMap<String, >List<String>>(); > > > > for (String key : metadata.getKeys()) { > > >if (! (key.startsWith(Parameters.NS) // skip metadata keys >starting with reserved namespace > > || >Parameters.PRODUCT_TYPE_NAME.indexOf(key)>=0 // skip 'ProductType' as >already stored as 'CAS.ProductTypeName' > > || >Parameters.PRODUCT_STRUCTURE.indexOf(key)>=0)) { // skip 'ProductType' as >already stored as 'CAS.ProductStructure' > > for (String value : >metadata.getAllMetadata(key)) { > > this.addKeyValueToMap(fields, >key, value); > > } > > } > > } > > > > return this.generateUpdateDocuments(productId, fields, >replace); > > > > } > > > > > > > > >
