Re: [Zope-dev] Re: Spitter.c Hack
Jason Spisak wrote: > > Zopists, > > I finally got Splitter.c to let me index numbers and 'C++' in a TextIndex. > I have about 50,000 objects in that index, and search performance is nearly > instantaneous still. I am running on a big machine though. If anyone > wants those changes there's really easy. Just mail me directly, since it's > a long file to post. Could you maybe post just the diff for poserity? -- | Casey Duncan | Kaivo, Inc. | [EMAIL PROTECTED] `--> ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Spitter.c Hack
Casey Duncan:
It truely is nothing more than cutting out the two parts that eliminate
single letter words and numbers:
*** Zope-2.2.4-src/lib/python/SearchIndex/Splitter.c
--- Zope-2.2.4-src/lib/python/SearchIndex/Splitter_Old.c
***
*** 169,192
len = PyString_Size(word) - 1;
len = PyString_Size(word);
- /*if(len < 2) Single-letter words are stop words!
- {
- Py_INCREF(Py_None);
- return Py_None;
- } */
-
- /*
- Test whether a word has any letters. */
for (; --len >= 0 && ! isalpha((unsigned char)cword[len]); );
- /*if (len < 0)
- {
- Py_INCREF(Py_None);
- return Py_None;
- }
-
- * If no letters, treat it as a stop word.
- */
Py_INCREF(word);
--- 169,176
All my best,
Jason Spisak
CIO
__ ___ ____
/ // (_)_/_ __/__ / / ___ ___ __ _
/ _ / / __/ -_) / / -_) __/ _ \(_-<_/ __/ _ \/ ' \
/_//_/_/_/ \__/_/ \__/\__/_//_/___(_)__/\___/_/_/_/
6151 West Century Boulevard
Suite 900
Los Angeles, CA 90045
P. 310.665.3444
F. 310.665.3544
Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Spitter.c Hack
Erik,
> [Jason Spisak]
>
> | I am running on a big machine though. If anyone wants those changes
> | there's really easy. Just mail me directly, since it's a long file
> | to post.
>
> Hi. I would be interested in the file :-).
>
Okay, here's the diff. It truely is nothing more than cutting out the two
parts that eliminate single letter words and numbers:
*** Zope-2.2.4-src/lib/python/SearchIndex/Splitter.c
--- Zope-2.2.4-src/lib/python/SearchIndex/Splitter_Old.c
***
*** 169,192
len = PyString_Size(word) - 1;
len = PyString_Size(word);
- /*if(len < 2) Single-letter words are stop words!
- {
- Py_INCREF(Py_None);
- return Py_None;
- } */
-
- /*
- Test whether a word has any letters. */
for (; --len >= 0 && ! isalpha((unsigned char)cword[len]); );
- /*if (len < 0)
- {
- Py_INCREF(Py_None);
- return Py_None;
- }
-
- * If no letters, treat it as a stop word.
- */
Py_INCREF(word);
--- 169,176
> Would you also be willing to share some statistics on how many objects
> you have in how many indexes, and how much time "complex" searches
> take? I do understand if this is not possible, but it'd be appetiated
> if it was possible. :-)
>
> Thanks.
Well, here's the some output of the "Status" tab in the Catalog.
Subtransactions are Disabled
Subtransactions
-
Index Status
* 48205 object are indexed in bobobase_modification_time
* 48205 object are indexed in calendar_date
* 48205 object are indexed in calendar_day
* 48205 object are indexed in call_date
* 48205 object are indexed in curators
* 48205 object are indexed in data
* 48205 object are indexed in id
* 48205 object are indexed in meta_type
* 48205 object are indexed in resume_in
* 48205 object are indexed in status
* 48205 object are indexed in users_calendar
The only TextIndex is the 'data' index though. It is the one that gets
hammered.
Let's see...time stats...hmmm
I put a REQUEST.set with the ZopeTime at the top of the search page and at
the bottom after the 'in' tag for the Catalog.
Search terms are: los and angeles and C++ and MFC and 310
Subtracting the float of the two times I get 1.85400104523 I'm not sure
what that comes out to, I think it's part of a day though because of
DateTime.
The server stats:
Dual Intel 400mhz Xenon w/ 1MB cache each
LVD RAID 5 7200 RPM disk array
1GB RAM
RedHat Linux 6.1 with some kernel updates...
And the best piece of open source software I know: Zope 2.2.4 binary
release
Hope that helps.
All my best,
Jason Spisak
CIO
__ ___ ____
/ // (_)_/_ __/__ / / ___ ___ __ _
/ _ / / __/ -_) / / -_) __/ _ \(_-<_/ __/ _ \/ ' \
/_//_/_/_/ \__/_/ \__/\__/_//_/___(_)__/\___/_/_/_/
6151 West Century Boulevard
Suite 900
Los Angeles, CA 90045
P. 310.665.3444
F. 310.665.3544
Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Spitter.c Hack
> From: "Jason Spisak" <[EMAIL PROTECTED]> wrote: > > Zopists, > > I finally got Splitter.c to let me index numbers and 'C++' in a TextIndex. > I have about 50,000 objects in that index, and search performance is nearly > instantaneous still. I am running on a big machine though. If anyone > wants those changes there's really easy. Just mail me directly, since it's > a long file to post. Can you post a patch, or upload it to your Zope.org member folder and post the link? cvs -q diff -u lib/python/SearchIndex/Splitter.c would do it, if you were working in a CVS sandbox for Zope. Tres. -- === Tres Seaver[EMAIL PROTECTED] Digital Creations "Zope Dealers" http://www.zope.org ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Spitter.c Hack
Tres:
Okay, I uploaded it to my member folder.
http://www.zope.org/Members/jspisak/Splitter/
I wasn't usinga sandbox for this, I just downloaded the source for 2.2.4
Here's the diff -u though:
--- Zope-2.2.4-src/lib/python/SearchIndex/Splitter.cThu Jan 4 10:41:15
2001
+++ Zope-2.2.4-src/lib/python/SearchIndex/Splitter_Old.cFri Jan 5
17:29:43 2001
@@ -169,24 +169,8 @@
len = PyString_Size(word) - 1;
len = PyString_Size(word);
-/*if(len < 2) Single-letter words are stop words!
-{
- Py_INCREF(Py_None);
- return Py_None;
-} */
-
-/*
- Test whether a word has any letters. */
for (; --len >= 0 && ! isalpha((unsigned char)cword[len]); );
-/*if (len < 0)
-{
-Py_INCREF(Py_None);
-return Py_None;
-}
-
- * If no letters, treat it as a stop word.
- */
Py_INCREF(word);
Let me know what else I can do. Did you see my other mails regarding
stats?
> > From: "Jason Spisak" <[EMAIL PROTECTED]> wrote:
> >
> > Zopists,
> >
> > I finally got Splitter.c to let me index numbers and 'C++' in a TextIndex.
> > I have about 50,000 objects in that index, and search performance is nearly
> > instantaneous still. I am running on a big machine though. If anyone
> > wants those changes there's really easy. Just mail me directly, since it's
> > a long file to post.
>
> Can you post a patch, or upload it to your Zope.org member folder
> and post the link?
>
> cvs -q diff -u lib/python/SearchIndex/Splitter.c
>
> would do it, if you were working in a CVS sandbox for Zope.
>
> Tres.
> --
> ===
> Tres Seaver[EMAIL PROTECTED]
> Digital Creations "Zope Dealers" http://www.zope.org
All my best,
Jason Spisak
CIO
__ ___ ____
/ // (_)_/_ __/__ / / ___ ___ __ _
/ _ / / __/ -_) / / -_) __/ _ \(_-<_/ __/ _ \/ ' \
/_//_/_/_/ \__/_/ \__/\__/_//_/___(_)__/\___/_/_/_/
6151 West Century Boulevard
Suite 900
Los Angeles, CA 90045
P. 310.665.3444
F. 310.665.3544
Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
