Re: [Gossip] search going ok

[Earl Hood]

On September 27, 2006 at 17:44, Olly Betts wrote:

> On Mon, Sep 25, 2006 at 03:16:06PM -0700, Jeff Marshall wrote:
> > In major search engines like Google and Yahoo, this relevance ordering 
> > makes the most sense.
> 
> Google Groups and Google News both offer a choice of "date" or
> "relevance" (date being "most recent first").

In my experience, I have found date-based ordering useful when
searching mail.

> Incidentally, Gmane's search offers both these and also "reverse date",
> which shows the oldest matches first - not so useful in general, but
> people requested it and it's easy to do once you've implemented sorting
> by "date".  It lets people find their "first post" if nothing else!

Reverse date sorting is useful, and I use it on occassion on the
various archives I have maintained and search.

I do think such a facility may be useful for a certain-type of user,
but chronology can be considered an intrinsic property of mail/news
archives.

--ewh
-- 
Earl Hood, <[EMAIL PROTECTED]>
Web: 
PGP Public Key: 

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Olly Betts]

On Mon, Sep 25, 2006 at 03:16:06PM -0700, Jeff Marshall wrote:
> In major search engines like Google and Yahoo, this relevance ordering 
> makes the most sense.

Google Groups and Google News both offer a choice of "date" or
"relevance" (date being "most recent first").

> Perhaps in our specialized mailing list searching, freshness matters
> more than relevance?

I think it depends on the user and query.

Incidentally, Gmane's search offers both these and also "reverse date",
which shows the oldest matches first - not so useful in general, but
people requested it and it's easy to do once you've implemented sorting
by "date".  It lets people find their "first post" if nothing else!

Cheers,
Olly

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Jeff Breidenbach]

Hi Mac,

Wow, when you dive in, you dive in deep! By the way, I have
not read the book, but I have looked at the FAQ.


1) It seems as though the Lucene search engine (LSE) deals with
indexes, rather than the email messages. Is that correct?


Ahead of time, Lucene reads email messages to produce an index. For
example, it took a few minutes to read the 12 thousand or so messages
in the Sundial list and create an index. After that, you are totally
correct. When someone searches for "Mac Oglesby" Lucene just consults
the index. Essentially all search software uses this technique because
it is fast.


2) LSE searches fields, and the default field is text. Correct?


Yes, LSE searches fields. We've set things up so the fields are
"subject" "date" "from" and "message". (We stick the message body in
"message"). Additionally there is the "all" field which is a
concatenation of all of the above. We've programmed the "all" field to
be searched by default.

Finally, there is another field where we store the messsage filename
(e.g. "msg00243.html"). I don't want to tell people the name of this
field since it is not useful for searching, and will likely change
anyway. Anyway, we need the filename to create links on the results
page. If this is confusing, just pretend I never wrote this
paragraph. :)


3) Thinking of the Sundial archive, how do I find out what fields are
available? "date" seems to be a field, as does "title" and "text."
What other fields are there?


The fields used by The Mail Archive are listed in the FAQ. My
impression is only a few people will ever want to get this
advanced with their queries.

http://www.mail-archive.com/faq.html#search


Is it possible for me to actually look at some typical indexes?


Sure. Here's the current sundial index, and a tool called "luke" to
inspect it. The tool is oriented towards programmers, but is still
kind of neat. Shows some of what is going on behind the scenes.

http://www.mail-archive.com/sundial_index.zip
http://www.getopt.org/luke/


4) One FAQ dealt with searching within results:
[...]
Can you give me a couple of simple examples of how [BooleanQuery]
might work?


Well, the FAQ is basically saying "hey, programmer! Do you want to
make a user interface like Eudora's search-within-a-search? If so,
here's the Lucene command you should be using." However, Jeff and I
haven't made such a user interface and we aren't using that command.
So this is basically out of you hands. I suppose you can lobby us to
change the user interface, which may or may not be successful.

However! You can get a functional equivalent to "search within a
search" just by making longer and longer query strings. For example:

Mac
Mac AND ebay
Mac AND ebay AND date:2005*
Mac AND ebay AND date:2005 AND sun

By the way, just like the global search engines, we've prgrammed AND
to be implicit. So the following two queries are equivalent.

Mac AND ebay AND date:2005 AND sun
Mac ebay date:2005* sun

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Mac Oglesby]

Jeff,

I've spent some time (several hours, in fact -- I'm a little slow) 
poking about on the Lucene site and its links in an attempt to 
understand more about the search process. I even downloaded and tried 
to read the two sample chapters from "Lucene in Action." That book 
certainly wasn't written for someone like me!


I also tried reading through some of the items on the Lucene FAQ page

http://wiki.apache.org/jakarta-lucene/LuceneFAQ

which left me overwhelmed. I'm sure there's much great stuff at that 
page, but I don't know a lot of the terms used. The answers were 
written for someone with more background than I currently possess.



I have many questions, but will start off with just a few. My 
questions will reveal how little I know about programming at this 
level.


1) It seems as though the Lucene search engine (LSE) deals with 
indexes, rather than the email messages. Is that correct?


2) LSE searches fields, and the default field is text. Correct?

3) Thinking of the Sundial archive, how do I find out what fields are 
available? "date" seems to be a field, as does "title" and "text." 
What other fields are there? Is it possible for me to actually look 
at some typical indexes?


4) One FAQ dealt with searching within results:

http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-f70612c6e4670e7fa2d5aeef4710effc522d85e0

** begin quote **


Can Lucene do a "search within search", so that the second search is 
constrained by the results of the first query?


Yes. There are two primary options:

*

  Use QueryFilter with the previous query as the filter. (you can 
search the mailing list archives for QueryFilter and Doug Cutting's 
recommendations against using it for this purpose)

*

  Combine the previous query with the current query using 
BooleanQuery, using the previous query as required.


The BooleanQuery approach is the recommended one.


** end quote **


Can you give me a couple of simple examples of how this might work?

Thanks for any help.

Mac






I'd be happy to help. If you can put up with my cranky comments, I
can put up with your bumbling efforts to patch the search engine.
;-)


Works for me.

So I tracked down the bug that was keeping most of the sundial
messages out of the search index. One particular message
with a very funny date was gumming up the works. Now that
the problem is resolved, searches should find everything.



___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Jeff Breidenbach]

I'd be happy to help. If you can put up with my cranky comments, I
can put up with your bumbling efforts to patch the search engine.
;-)


Works for me.

So I tracked down the bug that was keeping most of the sundial
messages out of the search index. One particular message
with a very funny date was gumming up the works. Now that
the problem is resolved, searches should find everything.

http://www.mail-archive.com/search?l=sundial%40uni-koeln.de&q=oglesby
http://www.mail-archive.com/search?l=sundial%40uni-koeln.de&q=date%3A200401*

Jeff

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Jeff Marshall]

JM, can you look  into sorting results by newest-first?


I can.  The only downside with sorting results by newest-first is that 
you lose the relevance ordering.  Right now, results are displayed 
according to the search engine's concept of "most relevant" first.


In major search engines like Google and Yahoo, this relevance ordering 
makes the most sense.  Perhaps in our specialized mailing list 
searching, freshness matters more than relevance?  In more advanced 
queries, the relevance ordering probably matters more, but maybe in the 
more common simple searches freshness matters more.


Jeff

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Mac Oglesby]

Jeff,


I really *still* would like to be able to access all of the messages
in the sundial archive by date. I guess I've figured out how to
locate the most recent 10,000, but how to get to the earliest?


Do you find date search useful? For example:

date:1997*





My results from searching the sundial archive:

date:1996*   0
date:1997*   388
date:1998*   1028
date:1999*   1381
date:2000*   4
date:2001*   0
date:2002*   0
date:2003*   0
date:2004*   0
date:2005*   69
date:2006*   0

These results are obviously wrong. The sundial list archive has (or 
did have) over 12,000 messages from Feb 1996 to date. It's possible 
the 1997-99 numbers are correct, but there's still the big problem of 
only getting to see the "first" 100.






Maybe many of your clients need all of the bells and whistles -- I
just need something easy to use.


I think almost everyone wants almost exactly what you do -
something simple and easy to use. We had switch because the
old system was choking on non-English languages. I've never
used (or seen) Eudora, but the idea is to be as simple to
use as the global search engines like Yahoo or Google.




Apropos the Google search engine, I got these results:

"[EMAIL PROTECTED]" 1997*  448
"[EMAIL PROTECTED]" 1998*  707
"[EMAIL PROTECTED]" 1999* 1000
"[EMAIL PROTECTED]" 2000*  707
"[EMAIL PROTECTED]" 2001*  556
"[EMAIL PROTECTED]" 2002* 1000
"[EMAIL PROTECTED]" 2003*  259
"[EMAIL PROTECTED]" 2004*   42
"[EMAIL PROTECTED]" 2005*8
"[EMAIL PROTECTED]" 2006*1

I can't say these results are any more satisfactory!





That said, I did study computer science in college and I understand
that I may be particularly bad at writing documentation for people
with different backgrounds. This is why outside help is extra
appreciated. Anyway, give us some time to look into the things
mentioned above, and then I'd like to take another pass at the
documentation with your assistance. Once all that is in place, we
can get some additional critiques from other Sundial users. Ok?




I'd be happy to help. If you can put up with my cranky comments, I 
can put up with your bumbling efforts to patch the search engine. 
;-)



Best wishes,

Mac





___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Jeff Breidenbach]

You feedback is extremely appreciated, it's the quickest road to
improvement. I can't stress this enough.


My brief tests with "sundial" have left me frustrated. For instance,
a search for "oglesby" used to return over 550 results, 500 of which
I could directly access.


I'll look into this. It should be returning every single match.


can only view 100,


This is on Jeff Marshall's plate.


and they are all from many years ago.


JM, can you look  into sorting results by newest-first?


I really *still* would like to be able to access all of the messages
in the sundial archive by date. I guess I've figured out how to
locate the most recent 10,000, but how to get to the earliest?


Do you find date search useful? For example:

date:1997*


Maybe many of your clients need all of the bells and whistles -- I
just need something easy to use.


I think almost everyone wants almost exactly what you do -
something simple and easy to use. We had switch because the
old system was choking on non-English languages. I've never
used (or seen) Eudora, but the idea is to be as simple to
use as the global search engines like Yahoo or Google.

That said, I did study computer science in college and I understand
that I may be particularly bad at writing documentation for people
with different backgrounds. This is why outside help is extra
appreciated. Anyway, give us some time to look into the things
mentioned above, and then I'd like to take another pass at the
documentation with your assistance. Once all that is in place, we
can get some additional critiques from other Sundial users. Ok?

Cheers,
Jeff

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip

Re: [Gossip] search going ok

[Mac Oglesby]

Hi Jeff,

I don't know what to say. On the one hand I appreciate the work you 
all do in making the Mail Archive available, and for no charge.


On the other hand I still feel the search feature is a weak spot for 
the casual user. You've illustrated the dazzling capabilities of the 
new search engine, but are its features of any use to those of us who 
aren't software engineers?


What I need is a search engine which is *simple* to use, which covers 
*all* of an archive, and returns organized results. The previous 
search engine was inadequate, IMO, but the current engine is a big 
step backwards.


My brief tests with "sundial" have left me frustrated. For instance, 
a search for "oglesby" used to return over 550 results, 500 of which 
I could directly access. Now, I get 162 matches, can only view 100, 
and they are all from many years ago. Similar results when other 
active sundial list users were entered.


Maybe many of your clients need all of the bells and whistles -- I 
just need something easy to use. I love the way Eudora lets me 
quickly search all of the 25,000+ messages on my computer, displays 
all of the results, and then lets me search results, search results, 
etc. until I see what's wanted.


I really *still* would like to be able to access all of the messages 
in the sundial archive by date. I guess I've figured out how to 
locate the most recent 10,000, but how to get to the earliest?


Re: your FAQ -- I understand the need to keep each FAQ entry brief, 
and I appreciate the links to the conversations about the new search 
engine and the link to Lucene. However, some of us mere mortals won't 
live long enough to decipher their instructions. My plea was for a 
simple tutorial. But perhaps Lucene can't do simple things, so 
there's no need for a beginner's tutorial.


Best wishes,

Mac Oglesby

P.S. If it would be of any use to this process, I'd be happy to ask 
sundial list members to send me comments on their use of the new 
search engine. I wouldn't expect many to bother to reply, however.







I am happy to report search is going reasonably well. The user
interface is now localized to all 16 supported languages. Today's
(weekly) update to the search index ran smoothly and the process is
much less computationally expensive than the previous system. There
have been other assorted tweaks here and there.

I don't expect too many users to ever look at the "how do I do
advanced queries" documentation. Nonetheless, I've written a draft and
am looking forward to Mac's assistance in helping make it more
understandable. http://www.mail-archive.com/faq.html#search

If you see any problems, let me know.

Thanks.
Jeff

___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip



___
Discussion list for The Mail Archive
Gossip@jab.org
http://jab.org/cgi-bin/mailman/listinfo/gossip