Re: Copyto with DIH Interpreting string as MultiValued field on copy
Makes total sense. Thanks to both of your for the clarification! On 8/18/18, 8:03 AM, "Alexandre Rafalovitch" wrote: >Amd part of the issue is that SolrEntityProcessor does not take individual >field definitions. So that part is ignored and instead just 'fl' mapping >is >used as Shawn explained. > >So you could also remap authorText in that definition to an ignored field. >See >https://github.com/apache/lucene-solr/blob/master/solr/example/example-DIH >/solr/solr/conf/solr-data-config.xml > >Regards, >Alex > >On Fri, Aug 17, 2018, 11:50 PM Shawn Heisey, wrote: > >> On 8/17/2018 6:15 PM, Zimmermann, Thomas wrote: >> > I¹m trying to track down an odd issue I¹m seeing when using the >> SolrEntityProcessor to seed some test data from a solr 4.x cluster to a >> solr 7.x cluster. It seems like strings are being interpreted as >> multivalued when passed from a string field to a text field via the >>copyTo >> directive. Any clever ideas how to resolve this? >> >> What's happening is deceptively simple. >> >> In the source system, you're copying from author to authorText. Both >> fields are stored. So if you have "Jeff Hartley" in author, you also >> have "Jeff Hartley" in authorText. So what's happening is that when the >> destination system imports from the source system, it gets "Jeff >> Hartley" in both fields, and then copyField says "put a copy of what's >> in author into authorText" ... and suddenly there are two copies of >> "Jeff Hartley" in authorText. >> >> There are two ways to deal with this: >> >> 1) In the query you're doing with SolrEntityProcessor, add an "fl" >> parameter and list all the fields *except* authorText and any other >> field where this same problem is happening. >> >> 2) Remove the copyField from the schema until after the import from the >> source server is done. >> >> Thanks, >> Shawn >> >>
Re: Copyto with DIH Interpreting string as MultiValued field on copy
Amd part of the issue is that SolrEntityProcessor does not take individual field definitions. So that part is ignored and instead just 'fl' mapping is used as Shawn explained. So you could also remap authorText in that definition to an ignored field. See https://github.com/apache/lucene-solr/blob/master/solr/example/example-DIH/solr/solr/conf/solr-data-config.xml Regards, Alex On Fri, Aug 17, 2018, 11:50 PM Shawn Heisey, wrote: > On 8/17/2018 6:15 PM, Zimmermann, Thomas wrote: > > I’m trying to track down an odd issue I’m seeing when using the > SolrEntityProcessor to seed some test data from a solr 4.x cluster to a > solr 7.x cluster. It seems like strings are being interpreted as > multivalued when passed from a string field to a text field via the copyTo > directive. Any clever ideas how to resolve this? > > What's happening is deceptively simple. > > In the source system, you're copying from author to authorText. Both > fields are stored. So if you have "Jeff Hartley" in author, you also > have "Jeff Hartley" in authorText. So what's happening is that when the > destination system imports from the source system, it gets "Jeff > Hartley" in both fields, and then copyField says "put a copy of what's > in author into authorText" ... and suddenly there are two copies of > "Jeff Hartley" in authorText. > > There are two ways to deal with this: > > 1) In the query you're doing with SolrEntityProcessor, add an "fl" > parameter and list all the fields *except* authorText and any other > field where this same problem is happening. > > 2) Remove the copyField from the schema until after the import from the > source server is done. > > Thanks, > Shawn > >
Re: Copyto with DIH Interpreting string as MultiValued field on copy
On 8/17/2018 6:15 PM, Zimmermann, Thomas wrote: I’m trying to track down an odd issue I’m seeing when using the SolrEntityProcessor to seed some test data from a solr 4.x cluster to a solr 7.x cluster. It seems like strings are being interpreted as multivalued when passed from a string field to a text field via the copyTo directive. Any clever ideas how to resolve this? What's happening is deceptively simple. In the source system, you're copying from author to authorText. Both fields are stored. So if you have "Jeff Hartley" in author, you also have "Jeff Hartley" in authorText. So what's happening is that when the destination system imports from the source system, it gets "Jeff Hartley" in both fields, and then copyField says "put a copy of what's in author into authorText" ... and suddenly there are two copies of "Jeff Hartley" in authorText. There are two ways to deal with this: 1) In the query you're doing with SolrEntityProcessor, add an "fl" parameter and list all the fields *except* authorText and any other field where this same problem is happening. 2) Remove the copyField from the schema until after the import from the source server is done. Thanks, Shawn
Copyto with DIH Interpreting string as MultiValued field on copy
Hi, I’m trying to track down an odd issue I’m seeing when using the SolrEntityProcessor to seed some test data from a solr 4.x cluster to a solr 7.x cluster. It seems like strings are being interpreted as multivalued when passed from a string field to a text field via the copyTo directive. Any clever ideas how to resolve this? Schema: Fields and CopyTo Text fieldtype declaration: DIH Config: http://cluster.solr.eng.techtarget.com/solr/vignette " query="*:*" fl="*,orig_version_l:_version_"> Error: org.apache.solr.common.SolrException: ERROR: [doc=d751e434c69b6210VgnVCM100d01c80aRCRD] Error adding field 'author'='Jeff Hartley' msg=Multiple values encountered for non multiValued copy field authorText: Jeff Hartley at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:203) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:101) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:980) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:971) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:348) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:284) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:234) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:950) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1168) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:633) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80) ~[?:?] at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:258) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:527) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330) ~[?:?] at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233) ~[?:?] at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424) ~[?:?] at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) ~[?:?] at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) ~[?:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172] Caused by: org.apache.solr.common.SolrException: Multiple values encountered for non multiValued copy field authorText: Jeff Hartley at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:180) ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:55:13] ... 22 more
