Re: [HACKERS] Include Lists for Text Search
On Sun, 9 Mar 2008, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: I've coded a small patch to allow CaseSensitive synonyms. Applied with corrections (it'd be good if you at least pretended to test stuff before submitting it). Would a similar parameter be useful for any of the other dictionary types? There are many things desirable to do with dictionaries, for example, say dictionary to return an original word plus it's normal form. Another feature is a not recognize-and-stop dictionaries, but allow filtering dictionary. We have a feeling that a little middleware would help implement this, and CaseSensitive too. Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
On Sun, 2008-03-09 at 23:03 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: I've coded a small patch to allow CaseSensitive synonyms. Applied with corrections (it'd be good if you at least pretended to test stuff before submitting it). It is a frequent accusation of yours that I don't test things, which is incorrect. Defending against that makes me a liar twice in your eyes. If you look more closely at what happens you'll understand that your own rigid expectations are what causes these problems. If you thought at all you'd realise that nobody would be stupid enough to try to sneak untested code into Postgres; all bugs would point directly back to anybody attempting that. That isn't true just of Postgres, its true of any group of people working together on any task, not just software or open source software. As Greg mentions on another thread, not all patches are *intended* to be production quality by their authors. Many patches are shared for the purpose of eliciting general feedback. You yourself encourage a group development approach and specifically punish those people dropping completely finished code into the queue and expecting it to be committed as-is. So people produce patches in various states of readiness, knowing that they may have to produce many versions before it is finally accepted. Grabbing at a piece of code, then shouting unclean, unclean just destroys the feedback process and leaves teamwork in tatters. My arse doesn't need wiping, thanks, nor does my bottom need smacking, nor are you ever likely to catch me telling fibs. If you think so, you're wrong and you should reset. What you will find from me and others, in the past and realistically in the future too, are patches that vary according to how near to completion they are. Not the same thing as completed, yet varying in quality. If they are incomplete it is because of the idea to receive feedback at various points. Some patches need almost none e.g. truncate triggers (1-2 versions), some patches need almost constant feedback e.g. async commit (24+ versions before commit). The existence of an intermediate patch in no way signals laziness, lack of intention to complete or any other failure to appreciate the software development process. If you want people to work on Postgres alongside you, I'd appreciate a software development process that didn't roughly equate to charging at a machine gun trench across a minefield. If you insist on following that you should at least stop wondering why it is that the few people to have made more than a few steps are determined and grim individuals and start thinking about the many skilled people who have chosen non-combatant status, and why. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Simon Riggs wrote: As Greg mentions on another thread, not all patches are *intended* to be production quality by their authors. Many patches are shared for the purpose of eliciting general feedback. You yourself encourage a group development approach and specifically punish those people dropping completely finished code into the queue and expecting it to be committed as-is. If you post a patch that is not intended to be of production quality, it is best to mark it so explicitly. Then nobody can point fingers at you. Also, Bruce would then know not to put it in the queue of patches waiting for application. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
On Mon, 2008-03-10 at 08:24 -0400, Andrew Dunstan wrote: Simon Riggs wrote: As Greg mentions on another thread, not all patches are *intended* to be production quality by their authors. Many patches are shared for the purpose of eliciting general feedback. You yourself encourage a group development approach and specifically punish those people dropping completely finished code into the queue and expecting it to be committed as-is. If you post a patch that is not intended to be of production quality, it is best to mark it so explicitly. Then nobody can point fingers at you. Also, Bruce would then know not to put it in the queue of patches waiting for application. So it can be forgotten about entirely? H. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Andrew Dunstan [EMAIL PROTECTED] writes: I think if you post something marked Work In Progress, there is an implied commitment on your part to post something complete at a later stage. It *wasn't* marked Work In Progress, and Simon went out of his way to cross-post it to -patches, where the thread previously had not been: http://archives.postgresql.org/pgsql-patches/2007-09/msg00150.php I don't think either Bruce or I can be faulted for assuming that it was meant to be applied. In future perhaps I should take it as a given that Simon doesn't expect his patches to be applied? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
On Mon, 2008-03-10 at 09:42 -0400, Andrew Dunstan wrote: I think if you post something marked Work In Progress, there is an implied commitment on your part to post something complete at a later stage. So if it's forgotten you would be the one doing the forgetting. ;-) But if they aren't on a review list, they won't get reviewed, no matter what their status. So everybody has to maintain their own status list and re-submit patches for review monthly until reviewed? I like the idea of marking things WIP, but I think we need a clear system where we agree that multiple statuses exist and that they are described in particular ways. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Simon Riggs wrote: On Mon, 2008-03-10 at 08:24 -0400, Andrew Dunstan wrote: Simon Riggs wrote: As Greg mentions on another thread, not all patches are *intended* to be production quality by their authors. Many patches are shared for the purpose of eliciting general feedback. You yourself encourage a group development approach and specifically punish those people dropping completely finished code into the queue and expecting it to be committed as-is. If you post a patch that is not intended to be of production quality, it is best to mark it so explicitly. Then nobody can point fingers at you. Also, Bruce would then know not to put it in the queue of patches waiting for application. So it can be forgotten about entirely? H. I think if you post something marked Work In Progress, there is an implied commitment on your part to post something complete at a later stage. So if it's forgotten you would be the one doing the forgetting. ;-) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Oleg Bartunov [EMAIL PROTECTED] writes: On Sun, 9 Mar 2008, Tom Lane wrote: Would a similar parameter be useful for any of the other dictionary types? There are many things desirable to do with dictionaries, for example, say dictionary to return an original word plus it's normal form. Another feature is a not recognize-and-stop dictionaries, but allow filtering dictionary. We have a feeling that a little middleware would help implement this, and CaseSensitive too. Hmm, I can see how some middleware would help with folding or not folding the input token, but what about the words coming from the dictionary file (particularly the *output* lexeme)? It's not apparent to me that it's sensible to try to control that from outside the dictionary. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
On Mon, 2008-03-10 at 10:01 -0400, Tom Lane wrote: In future perhaps I should take it as a given that Simon doesn't expect his patches to be applied? I think you should take it as a given that Simon would like to try to work together, sharing ideas and code, without insults and public derision when things don't fit. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Hmm, I can see how some middleware would help with folding or not folding the input token, but what about the words coming from the dictionary file (particularly the *output* lexeme)? It's not apparent to me that it's sensible to try to control that from outside the dictionary. Right now I see an significant advantage of such layer: two possible extension of dictionary (filtering and storing original form) are independent from nature of dictionary. So, instead of modifying of every dictionary we can add some layer, common for all dictionary. With syntax like: CREATE/ALTER TEXT SEARCH DICTIONARY foo (...) WITH ( filtering=on|off, store_original=on|off ); Or per token's type/dictionary pair. -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Right now I see an significant advantage of such layer: two possible extension of dictionary (filtering and storing original form) are One more extension: drop too long words. For example, decrease limit of max length of word to prevent long to be indexed - word with 100 characters is suspiciously long for human input. -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
Simon Riggs [EMAIL PROTECTED] writes: I've coded a small patch to allow CaseSensitive synonyms. Applied with corrections (it'd be good if you at least pretended to test stuff before submitting it). Would a similar parameter be useful for any of the other dictionary types? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Include Lists for Text Search
This has been saved for the 8.4 release: http://momjian.postgresql.org/cgi-bin/pgpatches_hold --- Simon Riggs wrote: On Mon, 2007-09-10 at 10:21 -0400, Tom Lane wrote: Oleg Bartunov [EMAIL PROTECTED] writes: On Mon, 10 Sep 2007, Simon Riggs wrote: Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. ... for 8.4. I've coded a small patch to allow CaseSensitive synonyms. CREATE TEXT SEARCH DICTIONARY my_diction ( TEMPLATE = biglist, DictFile = words, CaseSensitive = true ); -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com [ Attachment, skipping... ] ---(end of broadcast)--- TIP 6: explain analyze is your friend -- Bruce Momjian [EMAIL PROTECTED] http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Include Lists for Text Search
Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. I don't understand what you're proposing. We already have dict_synonym that you can use to accept a simple list of words. But that doesn't change the way GIN and GiST works. ? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Include Lists for Text Search
On Mon, 10 Sep 2007, Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Include Lists for Text Search
On Mon, 2007-09-10 at 12:58 +0100, Heikki Linnakangas wrote: Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. I don't understand what you're proposing. We already have dict_synonym that you can use to accept a simple list of words. How does that allow me to limit the number of words to a known list? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Include Lists for Text Search
On Mon, 10 Sep 2007, Simon Riggs wrote: On Mon, 2007-09-10 at 12:58 +0100, Heikki Linnakangas wrote: Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. I don't understand what you're proposing. We already have dict_synonym that you can use to accept a simple list of words. How does that allow me to limit the number of words to a known list? text search doesn't index unknown words, so if your mapping contains only one dictionary, this dictionary will control what words to index. While dict_synonym is good for not big list I'd write separate dictionary with fast lookup. Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Include Lists for Text Search
On Mon, 10 Sep 2007, Simon Riggs wrote: On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote: On Mon, 10 Sep 2007, Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. So there isn't one yet, but you think it will be easy to write and that we should call it dict_strict? we have dict_synonym already and if your list is not big you'll be happy. Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Include Lists for Text Search
How does that allow me to limit the number of words to a known list? If all dictionaries returns NULL for token the this token will not be indexed at all. -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Include Lists for Text Search
On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote: On Mon, 10 Sep 2007, Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. So there isn't one yet, but you think it will be easy to write and that we should call it dict_strict? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Include Lists for Text Search
There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking GIN doesn't depend strongly on number of words. It has log(N) behaviour for numbers of words because of using B-Tree over words. -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Include Lists for Text Search
On Mon, 2007-09-10 at 16:35 +0400, Oleg Bartunov wrote: On Mon, 10 Sep 2007, Simon Riggs wrote: On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote: On Mon, 10 Sep 2007, Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. So there isn't one yet, but you think it will be easy to write and that we should call it dict_strict? we have dict_synonym already and if your list is not big you'll be happy. So I need to do something like CREATE TEXT SEARCH DICTIONARY my_diction ( template = snowball, synonym = include_only_these_words ); which will then look for a file called include_only_these_words.syn? I would prefer to be able to do something like this CREATE TEXT SEARCH DICTIONARY my_diction ( template = snowball, include = justthese ); ...which makes more sense to anyone reading it and I also want to make the comparison case insensitive. Would it be better to 1. include a new dictionary file (dict_strict, as you suggest) 2. a) allow case sensitivity as another option in dictionaries b) allow include as another word for stoplist, but with the meaning reversed? e.g. CREATE TEXT SEARCH DICTIONARY my_diction ( template = snowball, include = justthese, case_sensitive = true ); -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Include Lists for Text Search
On Mon, 2007-09-10 at 16:48 +0400, Teodor Sigaev wrote: There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking GIN is great, sorry if that sounded negative. GIN doesn't depend strongly on number of words. It has log(N) behaviour for numbers of words because of using B-Tree over words. log(N) in the number of distinct words, but every word you index results in an index insert, so if we index more words than we need then the insert rate will go down. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Include Lists for Text Search
On Mon, 10 Sep 2007, Simon Riggs wrote: On Mon, 2007-09-10 at 16:48 +0400, Teodor Sigaev wrote: There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking GIN is great, sorry if that sounded negative. GIN doesn't depend strongly on number of words. It has log(N) behaviour for numbers of words because of using B-Tree over words. log(N) in the number of distinct words, but every word you index results in an index insert, so if we index more words than we need then the insert rate will go down. yes, there is room to improve support of very long posting lists Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Include Lists for Text Search
On Mon, 10 Sep 2007, Simon Riggs wrote: On Mon, 2007-09-10 at 16:35 +0400, Oleg Bartunov wrote: On Mon, 10 Sep 2007, Simon Riggs wrote: On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote: On Mon, 10 Sep 2007, Simon Riggs wrote: It seems possible to write your own functions to support various possibilities with text search. One of the more common thoughts is to have a list of words that you would like to include, i.e. the opposite of a stop word list. There are clear indications that indexing too many words is a problem for both GIN and GIST. If people already know what they'll be looking for and what they will never be looking for, it seems easier to supply that list up front, rather than hide it behind lots of hand-crafted code. Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. So there isn't one yet, but you think it will be easy to write and that we should call it dict_strict? we have dict_synonym already and if your list is not big you'll be happy. So I need to do something like CREATE TEXT SEARCH DICTIONARY my_diction ( template = snowball, synonym = include_only_these_words ); which will then look for a file called include_only_these_words.syn? I would prefer to be able to do something like this CREATE TEXT SEARCH DICTIONARY my_diction ( template = snowball, include = justthese ); ...which makes more sense to anyone reading it and I also want to make the comparison case insensitive. Would it be better to 1. include a new dictionary file (dict_strict, as you suggest) 2. a) allow case sensitivity as another option in dictionaries b) allow include as another word for stoplist, but with the meaning reversed? e.g. CREATE TEXT SEARCH DICTIONARY my_diction ( template = snowball, include = justthese, case_sensitive = true ); No, you need to write new template, which efficiently works with big lists and support case insensitive comparison. CREATE TEXT SEARCH TEMPLATE biglist ( . ); CREATE TEXT SEARCH DICTIONARY my_diction ( TEMPLATE = biglist, DictFile = words, case_sensitive = true ); Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Include Lists for Text Search
Oleg Bartunov [EMAIL PROTECTED] writes: On Mon, 10 Sep 2007, Simon Riggs wrote: Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. ... for 8.4. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Include Lists for Text Search
On Mon, 10 Sep 2007, Tom Lane wrote: Oleg Bartunov [EMAIL PROTECTED] writes: On Mon, 10 Sep 2007, Simon Riggs wrote: Can we include that functionality now? This could be realized very easyly using dict_strict, which returns only known words, and mapping contains only this dictionary. So, feel free to write it and submit. ... for 8.4. It can be just a contrib module. There are several useful dictionaries we need to port. Regards, Oleg _ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Include Lists for Text Search
On Mon, 2007-09-10 at 10:21 -0400, Tom Lane wrote:
Oleg Bartunov [EMAIL PROTECTED] writes:
On Mon, 10 Sep 2007, Simon Riggs wrote:
Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.
... for 8.4.
I've coded a small patch to allow CaseSensitive synonyms.
CREATE TEXT SEARCH DICTIONARY my_diction (
TEMPLATE = biglist,
DictFile = words,
CaseSensitive = true
);
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
Index: src/backend/tsearch/dict_synonym.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/tsearch/dict_synonym.c,v
retrieving revision 1.4
diff -c -r1.4 dict_synonym.c
*** src/backend/tsearch/dict_synonym.c 25 Aug 2007 02:29:45 - 1.4
--- src/backend/tsearch/dict_synonym.c 10 Sep 2007 15:14:21 -
***
*** 29,34
--- 29,35
typedef struct
{
int len; /* length of syn array */
+ bool case_sensitive;
Syn *syn;
} DictSyn;
***
*** 83,88
--- 84,90
*end = NULL;
int cur = 0;
char *line = NULL;
+ bool case_sensitive = false;
foreach(l, dictoptions)
{
***
*** 95,100
--- 97,107
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg(unrecognized synonym parameter: \%s\,
defel-defname)));
+
+ if (pg_strcasecmp(CaseSensitive, defel-defname) == 0
+ pg_strcasecmp(True, defGetString(defel)) == 0)
+ case_sensitive = true;
+
}
if (!filename)
***
*** 168,173
--- 175,182
d-len = cur;
qsort(d-syn, d-len, sizeof(Syn), compareSyn);
+ d-case_sensitive = case_sensitive;
+
PG_RETURN_POINTER(d);
}
***
*** 180,195
Syn key,
*found;
TSLexeme *res;
/* note: d-len test protects against Solaris bsearch-of-no-items bug */
if (len = 0 || d-len = 0)
PG_RETURN_POINTER(NULL);
! key.in = lowerstr_with_len(in, len);
key.out = NULL;
found = (Syn *) bsearch(key, d-syn, d-len, sizeof(Syn), compareSyn);
! pfree(key.in);
if (!found)
PG_RETURN_POINTER(NULL);
--- 189,214
Syn key,
*found;
TSLexeme *res;
+ bool need_pfree = false;
/* note: d-len test protects against Solaris bsearch-of-no-items bug */
if (len = 0 || d-len = 0)
PG_RETURN_POINTER(NULL);
! if (d-case_sensitive)
! key.in = in;
! else
! {
! key.in = lowerstr_with_len(in, len);
! need_pfree = true;
! }
!
key.out = NULL;
found = (Syn *) bsearch(key, d-syn, d-len, sizeof(Syn), compareSyn);
!
! if (need_pfree)
! pfree(key.in);
if (!found)
PG_RETURN_POINTER(NULL);
---(end of broadcast)---
TIP 6: explain analyze is your friend
