schema field type doesn't work

[Dimitar Ouzounov]

Hi everybody,
I added the following fieldtype in schema.xml :


  fieldtype name=st_numbers class=solr.TextField
 analyzer
 tokenizer class=solr.WhitespaceTokenizerFactory/
 filter class=solr.WordDelimiterFilterFactory catenateAll=1 /
 filter class=solr.LowerCaseFilterFactory/
 /analyzer
  /fieldtype

I want to index two types of strings, for example :

12345678
1234-5678

No matter which of the above strings is stored, I'd like to match it by
using either 12345678 or 1234-5678.
Everything is working fine, except for the case when 12345678 is stored and
I try to match it using
1234-5678. I must be doing something wrong, maybe in the schema. Does anyone
have any suggestions?
Any help would be greatly appreciated.

Re: schema field type doesn't work

[Dimitar Ouzounov]

Thanks a lot ! The analyzer admin tool is indeed useful.

On 3/24/07, Yonik Seeley [EMAIL PROTECTED] wrote:


On 3/24/07, Bertrand Delacretaz [EMAIL PROTECTED] wrote:
 On 3/24/07, Dimitar Ouzounov [EMAIL PROTECTED] wrote:

  ...I must be doing something wrong, maybe in the schema. Does anyone
  have any suggestions?..

 The best way to debug such problems is with the analyzer admin tool:
 http://localhost:8983/solr/admin/analysis.jsp

Yep...
trying the analysis page, one can see that parts of the numbers (not
just the catenation) are also still being generated, messing up the
query.

So if 123-456 is indexed, and you also want to be able to match parts
of that number (like 123), then you need a query analyzer and an index
analyzer for the field type, and turn off generation of parts for the
query analyzer only.

If you don't want to match parts, then a single analyzer for both
query and indexing will do, but explicitly turn off part generation:
   filter class=solr.WordDelimiterFilterFactory
generateWordParts=0 generateNumberParts=0 catenateWords=0
catenateNumbers=0 catenateAll=1/

-Yonik

Re: JVM random crashes

[Dimitar Ouzounov]

It will probably turn out to be a hardware problem - a bad RAM chip. I
removed it and today I will test Solr again to make sure everything is fine.

On 3/5/07, Bill Au [EMAIL PROTECTED] wrote:


Seems like this maybe a JVM bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6500147

http://forum.java.sun.com/thread.jspa?threadID=659990messageID=3876052

Have you tried using a different garbage collector?

Bill

On 3/3/07, Jed Reynolds [EMAIL PROTECTED] wrote:

 Yonik Seeley wrote:
  On 3/3/07, Dimitar Ouzounov [EMAIL PROTECTED] wrote:
  But what hardware problem could it be? Tomorrow I'll make sure that
the
  memory is fine, but nothing
  else comes to my mind.
 
  Memory, motherboard, etc.
  Try http://www.memtest86.com/ to test this.
 
  It may be OS-related - probably a buggy version of
  some library. But which library?
 
  Yep, we've seen that in the past.
  I'd recommend going with OS versions that vendors test with.
  The commercial RHEL or the free clone of it http://www.centos.org/,
  would be my recommendation.
 

 I'm running a lot of CentOS 4.4 myself, on i686 and x86_64 processors.
 I'm testing out Solr on an i686 with JDK 1.5 and I'm running a
 production copy of Nutch on x86_64 JDK 1.5, Tomcat 1.5. It's been rock
 solid.

 From trying to install Java in the past on FC5, I read a lot about how
 you had to be rather careful to make absolutely certain that you had no
 conflicting gjc libs in your path. If this is a production box, I'd got
 with a longer-supported OS than FC6. If the server is only for searching
 and apache, I don't think FC6 will give you any noticeable performance
 boost over CentOS 4.4. FC6's performance enhancements with
 glibc-hash-binding won't affect a JVM.


 Jed

JVM random crashes

[Dimitar Ouzounov]

Hi everybody.
I've been using Solr for more than two months and I was really happy with it
since yesterday. I moved my application
to a faster machine and everything went wrong. I have a PHP script, which
uses libcurl to post records to Solr. It's
sending 100 records at a time. After all 2500 records have been posted, the
script sends a commit/ and optimize/.
I'm running the script several times and the JVM crashes randomly.
Unforunately, I have been unable to reproduce this
bug on another machine. Right now I do not have access to that computer, and
I'm also bothered that I can't reboot it
from a PuTTY terminal. But tomorrow, I'll at least figure out what's going
on with the reboot.

I tried using JDK 1.5.0_05, 1.5.0_10, 1.5.0_11, and 1.6.0. I used the Java
-server mode, tried increasing the heap size, but the JVM
still crashes. I tried Solr 1.1 and two of the last nightly builds but the
problem remains. What's amazing is that there is
no such problem on two other machines that have the same configuration -
FC6, Tomcat 5.5.20, etc. By the way, I edited
Solr's web.xml to password protect solr/update and solr/admin, but I don't
think this causes the problem. I have absolutely
no clue what is going on. Could it be a problem in Solr's schema or
configuration? Or maybe a bug in JVM, or Solr, or Tomcat ?
Any help would be greatly appreciated.

Here are two of the error logs :



#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x061a8681, pid=26358, tid=3051350928
#
# Java VM: Java HotSpot(TM) Client VM (1.6.0-b105 mixed mode, sharing)
# Problematic frame:
# V  [libjvm.so+0x1a8681]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---  T H R E A D  ---

Current thread (0x095a3c00):  VMThread [id=26360]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x2068

Registers:
EAX=0x2010, EBX=0x0400, ECX=0x, EDX=0x0001
ESP=0xb5dfda64, EBP=0xb5dfda68, ESI=0x8e9b0090, EDI=0x8e96e688
EIP=0x061a8681, CR2=0x2068, EFLAGS=0x00010212

Top of Stack: (sp=0xb5dfda64)
0xb5dfda64:   0400 b5dfda98 062e4d39 0400
0xb5dfda74:   8e9b0090 0003 0003 95090da8
0xb5dfda84:   8e1eb710 0001 0003 8e96e684
0xb5dfda94:   908be858 b5dfdac8 061e5330 8e96e684
0xb5dfdaa4:   8e127efc b5dfdac8 062fab3d 8e127ed4
0xb5dfdab4:   b566e405 908be860 90220370 8ea23d74
0xb5dfdac4:   064366b4 b5dfdae8 062e4e4f 908be6e0
0xb5dfdad4:   8e96e678 b5dfdb08 0003 06421560

Instructions: (pc=0x061a8681)
0x061a8671:   74 2c 8b 45 0c ba 01 00 00 00 8b 40 04 83 c0 08
0x061a8681:   8b 40 58 83 e0 07 83 f8 05 74 13 31 d2 49 75 09

Stack: [0xb5d7e000,0xb5dff000),  sp=0xb5dfda64,  free space=510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V  [libjvm.so+0x1a8681]
V  [libjvm.so+0x2e4d39]
V  [libjvm.so+0x1e5330]
V  [libjvm.so+0x2e4e4f]
V  [libjvm.so+0x328db7]
V  [libjvm.so+0x328fef]
V  [libjvm.so+0x328a70]
V  [libjvm.so+0x1d22e7]
V  [libjvm.so+0x1d1a92]
V  [libjvm.so+0x1da23a]
V  [libjvm.so+0x1cf744]
V  [libjvm.so+0x183d45]
V  [libjvm.so+0x1cfacc]
V  [libjvm.so+0x3aed8e]
V  [libjvm.so+0x3c0f17]
V  [libjvm.so+0x3c059c]
V  [libjvm.so+0x3c076a]
V  [libjvm.so+0x3c034f]
V  [libjvm.so+0x3066b3]
C  [libpthread.so.0+0x53db]

VM_Operation (0xb4e66260): generation collection for allocation, mode:
safepoint, requested by thread 0xb566d400


---  P R O C E S S  ---

Java Threads: ( = current thread )
 0xb569d400 JavaThread TP-Monitor daemon [_thread_blocked, id=26401]
 0xb568c400 JavaThread TP-Processor4 daemon [_thread_in_native, id=26400]
 0xb56a0800 JavaThread TP-Processor3 daemon [_thread_blocked, id=26399]
 0xb5672400 JavaThread TP-Processor2 daemon [_thread_blocked, id=26398]
 0xb5671c00 JavaThread TP-Processor1 daemon [_thread_blocked, id=26397]
 0xb5670800 JavaThread http-81-Monitor [_thread_blocked, id=26396]
 0xb566f800 JavaThread http-81-Processor25 daemon [_thread_blocked,
id=26395]
 0xb566e400 JavaThread http-81-Processor24 daemon [_thread_in_native,
id=26394]
 0xb566d400 JavaThread http-81-Processor23 daemon [_thread_blocked,
id=26393]
 0xb566c000 JavaThread http-81-Processor22 daemon [_thread_blocked,
id=26392]
 0xb566b000 JavaThread http-81-Processor21 daemon [_thread_blocked,
id=26391]
 0xb5669c00 JavaThread http-81-Processor20 daemon [_thread_blocked,
id=26390]
 0xb5668c00 JavaThread http-81-Processor19 daemon [_thread_blocked,
id=26389]
 0xb565e000 JavaThread http-81-Processor18 daemon [_thread_blocked,
id=26388]
 0xb565cc00 JavaThread http-81-Processor17 daemon [_thread_blocked,
id=26387]
 0xb565bc00 JavaThread http-81-Processor16 daemon [_thread_blocked,
id=26386]
 0xb565a800 JavaThread http-81-Processor15 daemon [_thread_blocked,
id=26385]
 0xb5659800 JavaThread http-81-Processor14 daemon [_thread_blocked,
id=26384]
 0xb5658400 JavaThread http-81-Processor13 daemon [_thread_blocked,
id=26383]
 0xb5657400 JavaThread 

Re: JVM random crashes

[Dimitar Ouzounov]

But what hardware problem could it be? Tomorrow I'll make sure that the
memory is fine, but nothing
else comes to my mind. It may be OS-related - probably a buggy version of
some library. But which library?

On 3/3/07, Bertrand Delacretaz [EMAIL PROTECTED] wrote:


On 3/3/07, Dimitar Ouzounov [EMAIL PROTECTED] wrote:

 ...I tried using JDK 1.5.0_05, 1.5.0_10, 1.5.0_11, and 1.6.0. I used the
Java
 -server mode, tried increasing the heap size, but the JVM
 still crashes...

Makes me wonder if it's not your new machine that's at fault (hardware
or OS problem maybe?).

-Bertrand