Re: Configure StopWords in full text search without a configuration file?

On Thu, 2022-11-17 at 14:30 +0100, Florian Fuchs wrote: > is it possible to configure a StopWord list without a file in the > configuration-filesystem of postgres? > (StopWord: > https://www.postgresql.org/docs/current/textsearch-dictionaries.html#TEXTSEARCH-STOPWORDS) No, this is impossible. >

Configure StopWords in full text search without a configuration file?

Hello Postgres Community, is it possible to configure a StopWord list without a file in the configuration-filesystem of postgres? (StopWord: https://www.postgresql.org/docs/current/textsearch-dictionaries.html#TEXTSEARCH-STOPWORDS ) By default, the StopWord files are stored at $SHAREDIR/tsearch_

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

On Fri, Mar 4, 2022 at 10:22:11PM +0530, Atri Sharma wrote: > TF/IDF should be pretty simple to implement IMO. > > And no, Solr does not give preference to prior documents.  > > However, Solr allows you to "boost" specific terms, thus creating the > impression of preference.  Postgres can do th

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

TF/IDF should be pretty simple to implement IMO. And no, Solr does not give preference to prior documents. However, Solr allows you to "boost" specific terms, thus creating the impression of preference. On Fri, 4 Mar 2022, 22:15 Bruce Momjian, wrote: > On Fri, Mar 4, 2022 at 11:43:57AM -0500,

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

On Fri, Mar 4, 2022 at 11:43:57AM -0500, Tom Lane wrote: > "Bayer, Samuel" writes: > > One concrete question, I suppose, is: the classic TF/IDF search strategy > > relies on inverse document frequency, which looks across the corpus. I > > can't tell whether that corpus-wide frequency informatio

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

On Fri, Mar 4, 2022 at 11:39:39AM -0500, Bayer, Samuel wrote: > I've tried both ranking functions. I've tried a variety of the > normalization settings. I'm using the standard English language > configuration. Postgres 13. > > I do understand your FTS philosophy - I suppose I'm looking for > guidan

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

"Bayer, Samuel" writes: > One concrete question, I suppose, is: the classic TF/IDF search strategy > relies on inverse document frequency, which looks across the corpus. I can't > tell whether that corpus-wide frequency information is taken into account in > either ranking function. The docume

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

I've tried both ranking functions. I've tried a variety of the normalization settings. I'm using the standard English language configuration. Postgres 13. I do understand your FTS philosophy - I suppose I'm looking for guidance about how best to approximate the search capability in Solr using t

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

Bruce Momjian writes: > On Fri, Mar 4, 2022 at 10:41:16AM -0500, Bayer, Samuel wrote: >> I apologize for not being able to be more specific. > I know it is hard to quantify. Is it possible that Postgres is treating > all the terms equally, while Solr is prioritizing terms that are earlier > in t

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

On Fri, Mar 4, 2022 at 10:41:16AM -0500, Bayer, Samuel wrote: > Example anecdote: the documents I'm searching come with metadata > (e.g., title), which I'm not indexing specially (not a separate field, > just part of the raw text of the document). When I search even for > single terms, and look at

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

can't identify a specific missing function - I'm just observing that I can't make the results as high-quality as Solr. My apologies. Sam On 3/4/22 10:25 AM, Bruce Momjian wrote: On Fri, Mar 4, 2022 at 08:10:48AM -0500, Bayer, Samuel wrote: Hi all - When I have a need for both

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

y apologies. > > Sam > > On 3/4/22 10:25 AM, Bruce Momjian wrote: > > On Fri, Mar 4, 2022 at 08:10:48AM -0500, Bayer, Samuel wrote: > >> Hi all - > >> > >> When I have a need for both sophisticated database querying and > >> full-text search

Re: [EXT] Re: Looking for tips on improving full-text search quality in Postgres

x27;m just observing that I can't make the results as high-quality as Solr. My apologies. Sam On 3/4/22 10:25 AM, Bruce Momjian wrote: On Fri, Mar 4, 2022 at 08:10:48AM -0500, Bayer, Samuel wrote: Hi all - When I have a need for both sophisticated database querying and full-text sea

Re: Looking for tips on improving full-text search quality in Postgres

On Fri, Mar 4, 2022 at 08:10:48AM -0500, Bayer, Samuel wrote: > Hi all - > > When I have a need for both sophisticated database querying and > full-text search, I'd rather not stand up a technology stack with > multiple tools (e.g., Postgres and Apache Solr, or Postgres and &g

Looking for tips on improving full-text search quality in Postgres

Hi all - When I have a need for both sophisticated database querying and full-text search, I'd rather not stand up a technology stack with multiple tools (e.g., Postgres and Apache Solr, or Postgres and ElasticSearch with a zomboDB bridge). So I've been looking at the Postgres

Re: Full text search - wildcard and a stop word

Allan Jardine writes: > => select to_tsquery('all:*'); > NOTICE: text-search query contains only stop words or doesn't contain > lexemes, ignored > to_tsquery > > (1 row) > I get why that is happening - the notification basically details it, but > the wildcard at the end seems to m

Full text search - wildcard and a stop word

Hi all, I'm venturing into full text search in Postgres for the first time, and I'd like to be able to do a search by the start of a word - so I used the `:*` operator. However, this doesn't operate as I'd expect with a stop word - for example, my name is "Allan&quo

Re: Triggers and Full Text Search *

@Ericson I have the script (statements?) by which I created the triggers, but since you asked I do not see them in pga4, from which I manage my postgres. I don't know if this is significant or not. Also, this was originally done quite a while ago, so my memory may be fuzzy. From the text of the sta

Re: Triggers and Full Text Search *

On Tue, 2020-04-21 at 12:24 -0500, Malik Rumi wrote: > More than a year ago, I implemented full text search on one of my sites. > From the beginning, there was one problem (or at least, what I perceive > to be a problem): when I use a script to insert many documents at once, > they

Re: Triggers and Full Text Search *

On 4/21/20 11:21 AM, Malik Rumi wrote: @Ericson, Forgive me for seeming dense, but how does COPY help or hurt here? @Andreas, I had to laugh at your reference to "prose". Would you believe I am actually a published playwright? Long before I started coding, of course. Old habits die hard.

Re: Triggers and Full Text Search *

My apologies - I did not look closely at the manual. Many many years ago (6.xx days I had a similar problem and leapt to answer). Could you post your CREATE TRIGGER statements as well? On Wed, Apr 22, 2020 at 1:21 AM Malik Rumi wrote: > @Ericson, > Forgive me for seeming dense, but how does CO

Re: Triggers and Full Text Search *

@Ericson, Forgive me for seeming dense, but how does COPY help or hurt here? @Andreas, I had to laugh at your reference to "prose". Would you believe I am actually a published playwright? Long before I started coding, of course. Old habits die hard. entry_search_vector_trigger BEG

Re: Triggers and Full Text Search *

On 4/21/20 11:04 AM, Ericson Smith wrote: I think COPY bypasses the triggers. No: https://www.postgresql.org/docs/12/sql-copy.html "COPY FROM will invoke any triggers and check constraints on the destination table. However, it will not invoke rules." Best Regards - Ericson Smith +1 876-3

Re: Triggers and Full Text Search *

I think COPY bypasses the triggers. Best Regards - Ericson Smith +1 876-375-9857 (whatsapp) +1 646-483-3420 (sms) On Wed, Apr 22, 2020 at 12:32 AM Andreas Joseph Krogh wrote: > På tirsdag 21. april 2020 kl. 19:24:10, skrev Malik Rumi < > malik.a.r...@gmail.com>: > > [...] > > I am not (yet) p

Sv: Triggers and Full Text Search *

På tirsdag 21. april 2020 kl. 19:24:10, skrev Malik Rumi < malik.a.r...@gmail.com >: [...] I am not (yet) posting the trigger code because this post is long already, and if your answers are 1) yes, 2) no and 3) triggers often work / fail like this, then there’s

Triggers and Full Text Search *

More than a year ago, I implemented full text search on one of my sites. >From the beginning, there was one problem (or at least, what I perceive to be a problem): when I use a script to insert many documents at once, they do *not* get indexed in fts. If a document is created or inserted one a

Re: full text

Greetings, * Roberto Della Pasqua (roberto.dellapas...@live.com) wrote: > Please sorry because I'm newbie of PGSQL We all were, once upon a time. :) > I need the best performing and overall quality full-text search, can be > possible to have the data stored in pgsql an

Re: full text

de for CreateLuceneIndexFromDb. www.programcreek.com its up to you to determine the jdbc connection attributes from Postgres Buona Fortuna! martini From: Roberto Della Pasqua Sent: Wednesday, April 8, 2020 1:28 PM To: pgsql-gene...@postgresql.org Subject: full text Please sorry b

Re: full text

Roberto, I would look at ZomboDB: https://www.zombodb.com/ . Which is exactly what you suggest. JD On Wed, Apr 8, 2020 at 10:28 AM Roberto Della Pasqua < roberto.dellapas...@live.com> wrote: > Please sorry because I’m newbie of PGSQL > > > > I need the best performing and

full text

Please sorry because I'm newbie of PGSQL I need the best performing and overall quality full-text search, can be possible to have the data stored in pgsql and the index to elasticsearch? Can be in sync between? Thank you Btw. Do you suggest another engine than elastic? Roberto Della P

Re: Replacing Apache Solr with Postgre Full Text Search?

On 26.03.20 17:05, J2eeInside J2eeInside wrote: >> P.S. I need to index .pdf, .html and MS Word .doc/.docx files, is >> there any constraints in Ful Text search regarding those file types? > > - Can you recommend those tools you mention above/any useful resource on how to do that? For PDFs, I

Re: Replacing Apache Solr with Postgre Full Text Search?

I'm currently maintaining a project that uses Apache Solr /Lucene. To be > honest, I wold like to replace Solr with Postgre Full Text Search. However, > there is a huge amount of documents involved - arround 200GB. Wondering, > can Postgre handle this efficiently? > Does anyone h

Re: Replacing Apache Solr with Postgre Full Text Search?

Thanks again. For the end, the finally question: On Thu, Mar 26, 2020 at 4:18 PM Mike Rylander wrote: > On Thu, Mar 26, 2020 at 4:03 AM J2eeInside J2eeInside > wrote: > > > > > P.S. I need to index .pdf, .html and MS Word .doc/.docx files, is there > any constraints in Ful Text search regarding

Sv: Replacing Apache Solr with Postgre Full Text Search?

to eliminate one component from the whole system (if Full text search can replace Solr at all) I see you've gotten some answers but wanted to chime in... We seach in ~15mill. emails and ~10 mill documents (extracted text from Word/PDF etc. using Java-tools), and use PG and FTS (gin, n

Re: Replacing Apache Solr with Postgre Full Text Search?

On Thu, Mar 26, 2020 at 4:03 AM J2eeInside J2eeInside wrote: > > Hi Mike, and thanks for valuable answer! > In short, you think a PG Full Text Search can do the same as Apache Solr? > Can it? I mean, it does today. Whether it would for you depends on your needs and how much ef

Re: Replacing Apache Solr with Postgre Full Text Search?

Hi Mike, and thanks for valuable answer! In short, you think a PG Full Text Search can do the same as Apache Solr? P.S. I need to index .pdf, .html and MS Word .doc/.docx files, is there any constraints in Ful Text search regarding those file types? On Wed, Mar 25, 2020 at 3:36 PM Mike Rylander

Re: Replacing Apache Solr with Postgre Full Text Search?

On Wed, Mar 25, 2020 at 8:37 AM J2eeInside J2eeInside wrote: > > Hi all, > > I hope someone can help/suggest: > I'm currently maintaining a project that uses Apache Solr /Lucene. To be > honest, I wold like to replace Solr with Postgre Full Text Search. However, >

Replacing Apache Solr with Postgre Full Text Search?

Hi all, I hope someone can help/suggest: I'm currently maintaining a project that uses Apache Solr /Lucene. To be honest, I wold like to replace Solr with Postgre Full Text Search. However, there is a huge amount of documents involved - arround 200GB. Wondering, can Postgre handle

RE: Re: Re: Postgres Full Text Search Jsonb Array column does not search for first row

The issue was solved with the approach suggested here.https://stackoverflow.com/questions/59049873/postgres-full-text-search-jsonb-array-column-does-not-show-first-row BR, Dmytro. From: Laurenz AlbeSent: 28 листопада 2019 р. 16:39To: Dmytro Zhluktenko; pgsql-general@lists.postgresql.orgSubject: Re

Re: Re: Postgres Full Text Search Jsonb Array column does not search for first row

On Wed, 2019-11-27 at 11:54 +0200, Dmytro Zhluktenko wrote: > explain (analyze, BUFFERS) > SELECT * > FROM "cp"."Repro" x where cp.make_tsvector(x) @@ 'fir:*'::tsquery > > outputs this query plan: > Bitmap Heap Scan on "Repro" x (cost=12.00..16.26 rows=1 width=72) (actual > time=0.007..0.007 row

RE: Re: Postgres Full Text Search Jsonb Array column does not search for first row

off, then query still does the same result.Also, if you add 10 more entries, it will still fail to find the first one using index. BR, Dmytro. From: Laurenz AlbeSent: 26 листопада 2019 р. 21:13To: Dmytro Zhluktenko; pgsql-general@lists.postgresql.orgSubject: Re: Postgres Full Text Search Json

Re: Postgres Full Text Search Jsonb Array column does not search for first row

On Tue, 2019-11-26 at 13:37 +0200, Dmytro Zhluktenko wrote: > Pgsql is unable to perform indexed full text search onto jsonb column > containing an array when looking for the first row in the table. > > Any ideas why this is happening? > > CREATE OR REPLACE FUNCTION cp.mak

Postgres Full Text Search Jsonb Array column does not search for first row

Hey, Pgsql is unable to perform indexed full text search onto jsonb column containing an array when looking for the first row in the table.Any ideas why this is happening?Table is nothing more than just Id and Foo which is jsonb column.The case is that I have an empty database with predefined gin

Re: pg full text search very slow for Chinese characters

Hi. At Tue, 10 Sep 2019 18:42:26 +0200 (CEST), Andreas Joseph Krogh wrote in > På tirsdag 10. september 2019 kl. 18:21:45, skrev Tom Lane > >: Jimmy Huang writes: > > I tried pg_trgm and my own customized token parser > https://github.com/huangjimmy/pg_cjk_parse

Re: pg full text search very slow for Chinese characters

På tirsdag 10. september 2019 kl. 18:21:45, skrev Tom Lane mailto:t...@sss.pgh.pa.us>>: Jimmy Huang writes: > I tried pg_trgm and my own customized token parser https://github.com/huangjimmy/pg_cjk_parser pg_trgm is going to be fairly useless for indexing text that's mostly multibyte characte

Re: pg full text search very slow for Chinese characters

Jimmy Huang writes: > I tried pg_trgm and my own customized token parser > https://github.com/huangjimmy/pg_cjk_parser pg_trgm is going to be fairly useless for indexing text that's mostly multibyte characters, since its unit of indexable data is just 3 bytes (not characters). I don't know of a

Re: pg full text search very slow for Chinese characters

Lewis 日期: 2019年9月11日 星期三 上午12:11 收件人: 黄 少君 抄送: "pgsql-general@lists.postgresql.org" 主题: Re: pg full text search very slow for Chinese characters >My postgres instance is based on docker image postgres:11 and runs on my >MacBook Pro i7 16GB. How much ram and such did you give t

Re: pg full text search very slow for Chinese characters

On Tue, Sep 10, 2019 at 10:11 AM Michael Lewis wrote: > >My postgres instance is based on docker image postgres:11 and runs on my > MacBook Pro i7 16GB. > > How much ram and such did you give to this vm? > > > >To my surprise, postgres 11 is extremely slow when crea

Re: pg full text search very slow for Chinese characters

>My postgres instance is based on docker image postgres:11 and runs on my MacBook Pro i7 16GB. How much ram and such did you give to this vm? >To my surprise, postgres 11 is extremely slow when creating a full text index. I added a column of tsvector type and tried to create an index o

Re: pg full text search very slow for Chinese characters

fo, preface, text, html, info_html, preface_html} > are extracted from pages.html. The text column of books is a txt version of > the content of html column of table pages. > > > > On average there are 7635 characters (each characters is 3 bytes long > because of utf-8 encodi

pg full text search very slow for Chinese characters

encoding) for text column of table books and I want to add full text search to books(text). I tried pg_trgm and my own customized token parser https://github.com/huangjimmy/pg_cjk_parser To my surprise, postgres 11 is extremely slow when creating a full text index. I added a column of tsvector

PostgreSQL Asian language support for full text search using ICU (and also updating pg_trgm)

Hello, Currently PostgreSQL doesn't support full text search natively for many Asian languages such as Chinese, Japanese and others. These languages are used by a large portion of the population of the world. The two key modules that could be modified to support Asian languages are the

Re: How to use full-text search URL parser to filter query results by domain name?

On Wed, Apr 10, 2019 at 1:58 AM Jess Wren wrote: >-> Parallel Seq Scan on links > (cost=0.00..4554.40 rows=75740 width=112) > >-> Function Scan on ts_parse > (cost=0.00..12.50 rows=5 width=32) >

Re: How to use full-text search URL parser to filter query results by domain name?

On 4/8/19 4:50 AM, Arthur Zakirov wrote: > I think it is normal to use ts_parse(). And I suppose you might use > windows functions. > > For example, you have table links: > > =# create table links (score int, link text); > =# insert into links values >   (1, 'http://www.foo.com/bar'), >   (2, 'http

Re: How to use full-text search URL parser to filter query results by domain name?

On 07.04.2019 07:06, Jess Wren wrote: However, I can't figure out how I would integrate this into the above query to filter out duplicate domains from the results. And because this is the docs for "testing and debugging text search

Re: How to use full-text search URL parser to filter query results by domain name?

On 4/6/19 11:42 PM, haman...@t-online.de wrote: > Hi, > > I have no real idea about solving the complete problem, and would probably try > something with a temp table first. > For extracting the hostname from a url you could use > > select regex_replace('https?://(.*=)/.*', '\\1', url) > > instead

Re: How to use full-text search URL parser to filter query results by domain name?

>> I am trying to understand how to use the full-text search parser for >> URLS and hostnames to filter results from a text field containing URLS >> based on domain, and also how to index text columns for fast >> lookup/matching based on domain. >> >> I h

How to use full-text search URL parser to filter query results by domain name?

I am trying to understand how to use the full-text search parser for URLS and hostnames to filter results from a text field containing URLS based on domain, and also how to index text columns for fast lookup/matching based on domain. I have a PostgreSQL database containing documents and links

Full text search parser dictionary

Hi all, I have tried utilizing the full text searh feature and works as charm, except for one think which I would like to suggest to improve. When a document is split into tokens the parser removes any whitespace or punctuation charactures not otherwise recognized. But some punctuation

RE: Full text search with more than one word

as it is not supporting logical operator search and to_tsquery() is supporting logical operator search but not supporting two word search or phase search. If we need both functionally in full text search is there any other way we can proceed . Can you please help us solve this issue.

Re: Full text search with more than one word

supporting logical operator search but not > supporting two word search or phase search. If we need both functionally > in full text search is there any other way we can proceed . > > > > Can you please help us solve this issue. > Yes, as I said in my original email, check ou

Re: Full text search with more than one word

(please don't top-post on the postgresql mailinglists) On Fri, Jan 18, 2019 at 3:57 PM Ritanjali Majihee < rmaj...@247headhunting.com> wrote: > Hi Magnus, > > > > exact Search keyword is ("Marketing Coordinator" OR "Marketing > Specialist" OR "Marketing Associate" ) AND "Creative Suite". > > > >

RE: Full text search with more than one word

help us solve this issue. Thanks, Ritanjali From: mag...@hagander.net [mailto:mag...@hagander.net] Sent: Friday, January 18, 2019 4:11 PM To: Ritanjali Majihee Cc: pgsql-general General; Ambiger, Mahantesh Subject: Re: Full text search with more than one word On Fri, Jan 18, 2019 at

Re: Full text search with more than one word

arch functionality. We have converted html content to plaintext and > created tokens using to_tsvector. We have implemented full text search on > this token column. > > > > 1. When we search with one word and logical operators, it returns > results as expected. Example: (“Ja

Full text search with more than one word

Hi Team, We have around 20 millions of records in Postgresql database. One of the column is html document stored as Text datatype. We want Use this column in search functionality. We have converted html content to plaintext and created tokens using to_tsvector. We have implemented full text

Re: Full-text Search - Thesaurus relationships

On Wed, Oct 31, 2018 at 10:49:04AM +0100, Laurenz Albe wrote: > Nicolas Paris wrote: > > > > The documentation[1] says thesaurus can include informations of terms > > > > relationships such broader terms, preferred terms ... > > > > I haven't been able to find out how to exploit those relationship

Re: Full-text Search - Thesaurus relationships

Nicolas Paris wrote: > > > The documentation[1] says thesaurus can include informations of terms > > > relationships such broader terms, preferred terms ... > > > I haven't been able to find out how to exploit those relationship in > > > postgres. Is there any keyword to and associated syntax to ma

Re: Full-text Search - Thesaurus relationships

On Wed, Oct 31, 2018 at 07:56:28AM +0100, Laurenz Albe wrote: > > The documentation[1] says thesaurus can include informations of terms > > relationships such broader terms, preferred terms ... > > I haven't been able to find out how to exploit those relationship in > > postgres. Is there any keywo

Re: Full-text Search - Thesaurus relationships

Nicolas Paris wrote: > The documentation[1] says thesaurus can include informations of terms > relationships such broader terms, preferred terms ... > > I haven't been able to find out how to exploit those relationship in > postgres. Is there any keyword to and associated syntax to make use of >

Full-text Search - Thesaurus relationships

Hi, The documentation[1] says thesaurus can include informations of terms relationships such broader terms, preferred terms ... I haven't been able to find out how to exploit those relationship in postgres. Is there any keyword to and associated syntax to make use of them ? Thanks, [1]: https

Re: [GENERAL] Full text search with plain input

Thank you David, I will do something similar. 2018-05-05 3:12 GMT+02:00 David G. Johnston : > On Friday, May 4, 2018, Havasvölgyi Ottó > wrote: > >> >> Now I am thinking about splitting the input text myself to terms, then >> searching and ranking the documents for each term. >> > > Maybe do: r

Re: [GENERAL] Full text search with plain input

On Friday, May 4, 2018, Havasvölgyi Ottó wrote: > > Now I am thinking about splitting the input text myself to terms, then > searching and ranking the documents for each term. > Maybe do: replace(input_text, ' ', ' | ') David J.

Re: [GENERAL] Full text search with plain input

Based on the docs, plainto_tsquery() creates tsquery with the & operator ( https://www.postgresql.org/docs/10/static/functions-textsearch.html), so the @@ operator will not find partial matches. And this way I cannot calculate ts_rank for those partial matches. Now I am thinking about splitting the

Re: [GENERAL] Full text search with plain input

On Friday, May 4, 2018, Havasvölgyi Ottó wrote: > so that it matches (with less rank) even if one of its words match ? > That seems to be what "ts_rank" provides. David J.

[GENERAL] Full text search with plain input

Hi, I would like to use PostgreSQL's FTS functionality. The user would enter a search text, one or more words. There is a suitable plainto_tsquery() function, but it's to strict. It only matches if all words match. What is the best practice to make this kind of search more "forgiving", so that it

Custom ranking function for full text search

are there any examples/documentation for building an alternative ts_rank/ts_rank_cd for full text search?