Hi Julio, I've fixed the bug, can you please replace the exiting TemplateTransformer.java in the SOLR-469.patch and use the attached TemplateTransformer.java file. We'll add the changes to our next patch. Sorry for all the trouble.
On Sat, May 31, 2008 at 10:31 PM, Noble Paul നോബിള് नोब्ळ् <[EMAIL PROTECTED]> wrote: > julio, > Looks like it is a bug. > We can give u a new TemplateTransformer.java which we will incorporate > in the next patch > --Noble > > On Sat, May 31, 2008 at 12:24 AM, Julio Castillo > <[EMAIL PROTECTED]> wrote: >> I'm sorry Shalin, but I still get the same Null Pointer exception. This is >> my complete dataconfig.xml (I remove the parallel entity to narrow down the >> scope of the problem). >> <dataConfig> >> <document name="doc-1"> >> <entity name="vets" pk="id" >> query="select id as idAlias,first_name,last_name FROM vets" >> deltaQuery="SELECT id as idAlias FROM vets WHERE last_modified > >> '${dataimporter.last_index_time}'" >> transformer="TemplateTransformer"> >> <field column="id" name="id" >> template="vets-${vets.idAlias}"/> >> <field column="first_name" name="userName"/> >> <field column="last_name" name="userName"/> >> </entity> >> </document> >> </dataConfig> >> >> Thanks again. >> >> ** julio >> >> -----Original Message----- >> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED] >> Sent: Friday, May 30, 2008 11:38 AM >> To: solr-user@lucene.apache.org >> Subject: Re: How to describe 2 entities in dataConfig for the DataImporter? >> >> The surname is used just as an example of a field. >> >> The NullPointerException is because the same field "id" tries to use it's >> own value in a template. The template cannot contain the same field on which >> it is being applied. I'd suggest that you get the id aliased to another >> name, for example using a query "select id as idAlias from vets" and then >> use: >> <field column="id" template="vets-${vets.idAlias}" /> >> >> That should work, let me know if you face a problem. >> >> On Fri, May 30, 2008 at 10:40 PM, Julio Castillo <[EMAIL PROTECTED]> >> wrote: >>> Thanks for all the leads. >>> I did get however a null pointer exception while implementing it: >>> >>> May 30, 2008 9:57:50 AM >>> org.apache.solr.handler.dataimport.EntityProcessorBase >>> applyTransformer >>> WARNING: transformer threw error >>> java.lang.NullPointerException >>> at >>> org.apache.solr.handler.dataimport.TemplateTransformer.transformRow(Te >>> mplate >>> Transformer.java:55) >>> at >>> org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransforme >>> r(Enti >>> tyProcessorBase.java:186) >>> >>> Looking at the source code, it appears that the resolverMap is null. >>> The resolver returned null given the entityName. >>> >>> Looking at the documentation, there is the reference to a eparent.surname. >>> The example says: >>> <field column="namedesc" template="hello${e.name},${eparent.surname}" >>> /> >>> >>> I'm afraid, I don't know what an eparent.surname is. This is my >>> current dataconfig.xml configuration excerpt: >>> >>> <entity name="vets" pk="id" query="..." deltaQuery="..." >>> transformer="TemplateTransformer"> >>> <field column="id" name="id" template="vets-${vets.id}"/> ... >>> >>> Am I missing a surname? Whatever that may be? >>> >>> Thanks >>> >>> ** julio >>> >>> -----Original Message----- >>> From: Noble Paul ??????? ?????? [mailto:[EMAIL PROTECTED] >>> Sent: Thursday, May 29, 2008 11:10 PM >>> To: solr-user@lucene.apache.org >>> Subject: Re: How to describe 2 entities in dataConfig for the >> DataImporter? >>> >>> Sorry I forgot to mention that. >>> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a990 >>> fb03c4 >>> ff461b3736496a9 >>> --Noble >>> >>> On Fri, May 30, 2008 at 11:37 AM, Shalin Shekhar Mangar >>> <[EMAIL PROTECTED]> wrote: >>>> You need to enable TemplateTransformer for your entity. For example: >>>> <entity name="owners" pk="id" query="...." >>>> transformer="TemplateTransformer"> >>>> >>>> On Fri, May 30, 2008 at 11:31 AM, Julio Castillo >>>> <[EMAIL PROTECTED]> wrote: >>>>> Noble, >>>>> I tried the template setting for the "id" field, but I didn't notice >>>>> any different behavior. I also didn't see where this would be reflected. >>>>> I looked at the fields and the debug output for the dataImporter and >>>>> couldn't see any reference to a modified id name (per the template >>>>> instructions). >>>>> >>>>> The behavior in the end seemed to be the same. Did I miss anything? >>>>> I assume that the <uniqueKey>id</uniqueKey> setting in the >>>>> schema.xml remains the same? >>>>> >>>>> Thanks again >>>>> >>>>> ** julio >>>>> >>>>> -----Original Message----- >>>>> From: Noble Paul ??????? ?????? [mailto:[EMAIL PROTECTED] >>>>> Sent: Thursday, May 29, 2008 9:46 PM >>>>> To: solr-user@lucene.apache.org >>>>> Subject: Re: How to describe 2 entities in dataConfig for the >>> DataImporter? >>>>> >>>>> Consider constructing the id concatenating an extra string for each >>>>> document . You can construct that field using the TeplateTransformer. >>>>> in the entity owners keep the id as >>>>> >>>>> <field column="id" name="id" template="owners-${owners.id}"/> and in >>>>> vets <field column="id" name="id" template="vets-${vets.id}"/> >>>>> >>>>> or anything else which can make it unique >>>>> >>>>> --Noble >>>>> >>>>> On Fri, May 30, 2008 at 10:05 AM, Shalin Shekhar Mangar >>>>> <[EMAIL PROTECTED]> wrote: >>>>>> That will happen only if id is the uniqueKey in Solr and the id >>>>>> coming from both your tables have same values. In that case, they >>>>>> will overwrite each other. You will need a separate uniqueKey (on >>>>>> other than id field). >>>>>> >>>>>> On Fri, May 30, 2008 at 6:34 AM, Julio Castillo >>>>>> <[EMAIL PROTECTED]> >>>>> wrote: >>>>>>> Thanks Shalin, >>>>>>> I tried putting everything under the same document (two different >>>>>>> unrelated entities), and got a bit further. >>>>>>> >>>>>>> My problem now appears to be both of them stepping on each other >>>>>>> due to >>>>> "id" >>>>>>> conflicts. Currently my id is defined in my schema as <field >>>>>>> name="id" type="long" indexed="true" stored="true" >>>>>>> required="true"/> >>>>>>> >>>>>>> Do I have to create a new "id" field? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> ** julio >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED] >>>>>>> Sent: Thursday, May 29, 2008 11:40 AM >>>>>>> To: solr-user@lucene.apache.org >>>>>>> Subject: Re: How to describe 2 entities in dataConfig for the >>>>> DataImporter? >>>>>>> >>>>>>> Hi Julio, >>>>>>> >>>>>>> The first data-config is correct. >>>>>>> >>>>>>> You're running DataImportHandler in debug mode which creates only >>>>>>> the first 10 documents by default. You can also add count=N to >>>>>>> index only the first N documents. But this is intended only for >>>>>>> debugging purposes. If you want to do a full-import just use >>>>>>> /dataimport?command=full-import. By default, a full-import >>>>>>> automatically >>>>> cleans and commits. >>>>>>> >>>>>>> Hope that helps. >>>>>>> >>>>>>> On Thu, May 29, 2008 at 11:42 PM, Julio Castillo >>>>>>> <[EMAIL PROTECTED]> >>>>>>> wrote: >>>>>>>> I have 2 dB tables unrelated to each other that I want to index. >>>>>>>> >>>>>>>> I have tried 2 approaches for specifying them in my >>>>>>>> data-config.xml >>>>> file. >>>>>>>> None of them seem to work (it seems I can only get data for the >>>>>>>> first one listed). >>>>>>>> >>>>>>>> CASE 1) >>>>>>>> <dataConfig> >>>>>>>> <document name="test"> >>>>>>>> <entity name="owners" pk="id" query="...."> >>>>>>>> <field column="id" name="id"/> >>>>>>>> <field column=name" name="userName"/> >>>>>>>> </entity> >>>>>>>> <entity name="vets" pk="id" query="...."> >>>>>>>> <field column="id" name="id"/> >>>>>>>> <field column=name" name="userName"/> >>>>>>>> </entity> >>>>>>>> </document> >>>>>>>> </dataConfig> >>>>>>>> >>>>>>>> CASE 2) >>>>>>>> <dataConfig> >>>>>>>> <document name="test-1"> >>>>>>>> <entity name="owners" pk="id" query="...."> >>>>>>>> <field column="id" name="id"/> >>>>>>>> <field column=name" name="userName"/> >>>>>>>> </entity> >>>>>>>> </document> >>>>>>>> <document name="test-2"> >>>>>>>> <entity name="vets" pk="id" query="...."> >>>>>>>> <field column="id" name="id"/> >>>>>>>> <field column=name" name="userName"/> >>>>>>>> </entity> >>>>>>>> </document> >>>>>>>> </dataConfig> >>>>>>>> >>>>>>>> I ran the dataImporter as follows >>>>>>>> http://localhost:2455/solr/dataimport?command=full-import&debug=o >>>>>>>> n >>>>>>>> &v >>>>>>>> er >>>>>>>> bose=t >>>>>>>> rue&clean=true&commit=true >>>>>>>> >>>>>>>> After running Case 1 the output only shows documents under >>>>> entity:owners. >>>>>>>> And an empty reference to entity:vets (no data). >>>>>>>> >>>>>>>> After running Case 2, the output only shows documents under >>>>> entity:owners. >>>>>>>> No references to entity:vets. >>>>>>>> >>>>>>>> In either case, I have commented out the entity:owners and I do >>>>>>>> see the entity:vets data then. >>>>>>>> >>>>>>>> Thanks for your assistance >>>>>>>> >>>>>>>> Julio Castillo >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Shalin Shekhar Mangar. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Shalin Shekhar Mangar. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --Noble Paul >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Shalin Shekhar Mangar. >>>> >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> >> >> -- >> Regards, >> Shalin Shekhar Mangar. >> >> > > > > -- > --Noble Paul > -- Regards, Shalin Shekhar Mangar.
/** * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package org.apache.solr.handler.dataimport; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.logging.Logger; /** * <p> * A Transformer which can put values into a column by resolving an expression * containing other columns * </p> * * <p> * For example:<br /> * <field column="name" template="${e.lastName}, ${e.firstName} * ${e.middleName}" /> will produce the name by combining values from * lastName, firstName and middleName fields as given in the template * attribute. * </p> * * @author Shalin Shekhar Mangar ([EMAIL PROTECTED]) * @since solr 1.3 */ public class TemplateTransformer implements Transformer { private static final Logger LOG = Logger.getLogger(TemplateTransformer.class.getName()); @SuppressWarnings("unchecked") public Object transformRow(Map<String, Object> row, Context context) { String entityName = context.getEntityAttribute(DataImporter.NAME); VariableResolverImpl resolver = (VariableResolverImpl) context.getVariableResolver(); Map<String, Object> resolverMap = (Map<String, Object>)resolver.resolve(entityName); if (resolverMap == null) resolverMap = new HashMap<String, Object>(); //Clone resolver map Map<String, Object> resolverMapCopy = new HashMap<String, Object>(); //Add current row to the copy of resolver map for (Map.Entry<String, Object> entry : row.entrySet()) resolverMapCopy.put(entry.getKey(), entry.getValue()); //Add this copy to the namespace of the current entity in the resolver resolver.addNamespace(entityName, resolverMapCopy); for (Map<String, String> map : context.getAllEntityFields()) { String expr = map.get(TEMPLATE); if (expr == null) continue; String column = map.get(DataImporter.COLUMN); //Verify if all variables can be resolved or not boolean resolvable = true; List<String> variables = TemplateString.getVariables(expr); for (String v : variables) { if (resolver.resolve(v) == null) { LOG.warning("Unable to resolve variable: " + v + " while parsing expression: " + expr); resolvable = false; } } if (!resolvable) continue; row.put(column, resolver.replaceTokens(expr)); } //Restore the original resolver map resolver.addNamespace(entityName, resolverMap); return row; } public static final String TEMPLATE = "template"; }