Re: having trouble storing large text blob fields - returns binary address in search results

2013-05-18 Thread geeky2
hello

your comment made me think - so i decided to double check myself.

i opened up the schema in squirrel and made sure that the two columns in
question were actually of type TEXT in the schema - check

i went in to the db-config.xml and removed all references to
ClobTransformer, removed the cast directives from the fields as well as the
clob=true on the two fields - i pasted the db-config.xml below for
reference - check

i restarted jboss - thus restarting solr - check

i went in to the solr dataimport admin screen and did a clean import - check

after the import was complete - i queried a part that i knew would have one
of the clob fields - results are pasted below as well - you can see the
binary address in the field.


?xml version=1.0?
result name=response numFound=1 start=0
  doc
str name=accessoryIndicatorN/str
 *   str name=attributes[B@5b372219/str*
str name=availabilityStatusPIA/str
arr name=divProductTypeDesc
  strRefrigerators and Freezers/str
/arr
str name=divProductTypeId0046/str
str name=id12001892,0046,464/str
str name=itemModelDescVALVE, WATER/str
str name=itemModelNo12001892/str
str name=itemModelNoExactMatchStr12001892/str
int name=itemType1/int
str name=otcStockIndicatorY/str
int name=partCnt1/int
str name=partConditionN/str
arr name=plsBrandDesc
  str/
/arr
str name=plsBrandId464/str
str name=productIndicatorN/str
int name=rankNo13/int
float name=sellingPrice53.54/float
str name=sourceOrderNo464 /str
str name=subbedFlagY/str
  /doc
/result








document
entity transformer=TemplateTransformer name=core1-parts
query=select 
summ.*, 
1 as item_type, 
1 as part_cnt, 
'' as brand, 
mst.acy_prt_fl,
mst.dil_tx,
mst.hzd_mtl_typ_cd,
mst.otc_cre_stk_fl,
mst.prd_fl,
mst.prt_cmt_tx,
mst.prt_cnd_cd,
mst.prt_inc_qt,
mst.prt_made_by,
mst.sug_qt,
att.attr_val,
rsr.rsr_val,
case when sub.orb_itm_id is null then 'N' else 'Y' end as
subbed_flag
from 
prtxtps_prt_summ as summ
left outer join prtxtpm_prt_mast as mst on mst.orb_itm_id =
summ.orb_itm_id and mst.prd_gro_id = summ.prd_gro_id and mst.spp_id =
summ.spp_id
left outer join tmpxtpa_prt_attr as att on att.orb_itm_id =
summ.orb_itm_id and att.prd_gro_id = summ.prd_gro_id and att.spp_id =
summ.spp_id 
left outer join tmpxtpr_prt_rsr as rsr on rsr.orb_itm_id =
summ.orb_itm_id and rsr.prd_gro_id = summ.prd_gro_id and rsr.spp_id =
summ.spp_id 
left outer join tmpxtps_prt_sub as sub on sub.orb_itm_id =
summ.orb_itm_id and sub.prd_gro_id = summ.prd_gro_id and sub.spp_id =
summ.spp_id
where 
summ.spp_id = '464' 

field column =id name=id 
template=${core1-parts.orb_itm_id},${core1-parts.prd_gro_id},${core1-parts.spp_id}/
field column=orb_itm_id name=itemModelNo/ 
 
field column=prd_gro_id
name=divProductTypeId/ 
field column=ds_tx 
name=itemModelDesc/ 
field column=spp_id name=plsBrandId/ 
field column=rnk_no name=rankNo/ 
field column=item_type  name=itemType/ 
field column=brand  name=plsBrandDesc/ 
field column=prd_gro_ds
name=divProductTypeDesc/ 
field column=part_cnt   name=partCnt/ 
field column=avail 
name=availabilityStatus/ 
field column=price  name=sellingPrice/ 
field column=prt_son   
name=sourceOrderNo/ 
field column=prt_src_cd name=sourceIdCode/ 
field column=rte_cd
name=sourceRouteCode/ 

field column=acy_prt_fl
name=accessoryIndicator/ 
field column=dil_tx name=disclosure/ 
field column=hzd_mtl_typ_cd
name=hazardousMaterialCode/ 
field column=otc_cre_stk_fl
name=otcStockIndicator/ 
field column=prd_fl
name=productIndicator/ 
field column=prt_cmt_tx name=comment/ 
field column=prt_cnd_cd
name=partCondition/ 
field column=prt_inc_qt name=qtyIncluded/ 
field column=prt_made_byname=madeBy/ 
field column=sug_qt name=suggestedQty/ 

field column=attr_val   name=attributes/ 
field column=rsr_valname=restrictions/ 

field column=subbed_flag

Re: having trouble storing large text blob fields - returns binary address in search results

2013-05-17 Thread Gora Mohanty
On 17 May 2013 00:02, geeky2 gee...@hotmail.com wrote:
[...]
 i have tried setting them up as clob fields - but this is not working (see
 details below)

 i have also tried treating them as plain string fields (removing the
 references to clob in the DIH) - but this does not work either.


 DIH configuration:


   entity transformer=TemplateTransformer,ClobTransformer
 name=core1-parts query=select
 summ.*,
 1 as item_type,
 1 as part_cnt,
 '' as brand,
 ...

  field column=attr_val   name=attributes
 clob=true /
 field column=rsr_valname=restrictions clob=true
 /
[...]

This should work, provided that the SELECT statement is
getting the correct columns from the database. Are you sure
that the entity is enclosing the field in question? Could you
share the entire DIH data configuration file?

You should also try setting sourceColName for the ClobTransformer
field. Have never tried ClobTransformer where the field name differs
from the database column name. Please see:
http://wiki.apache.org/solr/DataImportHandler#ClobTransformer

I am not familiar with Informix, and the final possibility is that
there is some bug with ClobTransformer in translating blobs
from Informix. It should be possible to cajole the database to
convert a blob into a VARCHAR, e.g., something like
  SELECT SUBSTR( attr_val, 0 )
Not sure if the syntax is database dependent.

Regards,
Gora


Re: having trouble storing large text blob fields - returns binary address in search results

2013-05-17 Thread geeky2
Hello Gora,


thank you for the reply - 

i did finally get this to work.  i had to cast the column in the DIH to a
clob - like this.

cast(att.attr_val AS clob) as attr_val,
cast(rsr.rsr_val AS clob) as rsr_val,

once this was done, the ClobTransformer worked.

to my knowledge - this particular use case and the need for the cast is not
documented anywhere.  i checked the solr wiki and searched the threads on
this forum for things like clobtransformer, informix and blob without luck. 
i also did quite a few google searches as well but no luck (but maybe i
missed something ;)

maybe this is just some edge case.  i also realize that informix is not
that common.

i have a question in to the solr developers list - just so i can better
understand what actually is happening, why it was necessary for the cast,
and the limitations / parameters of the ClobTransformer.  

the thread on the developers list is located here:

http://lucene.472066.n3.nabble.com/have-developer-question-about-ClobTransformer-and-DIH-td4064256.html

thx
mark






--
View this message in context: 
http://lucene.472066.n3.nabble.com/having-trouble-storing-large-text-blob-fields-returns-binary-address-in-search-results-tp4063979p4064286.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: having trouble storing large text blob fields - returns binary address in search results

2013-05-17 Thread Gora Mohanty
On 18 May 2013 02:24, geeky2 gee...@hotmail.com wrote:
 Hello Gora,


 thank you for the reply -

 i did finally get this to work.  i had to cast the column in the DIH to a
 clob - like this.

 cast(att.attr_val AS clob) as attr_val,
 cast(rsr.rsr_val AS clob) as rsr_val,

 once this was done, the ClobTransformer worked.
[...]

This would seem to indicate that the column was not a
blob to start with. Did you try to index directly without
the ClobTransformer?

Regards,
Gora