Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Stephen Davies 

On 20/02/16 16:21, Chris Travers wrote:

A more general way would be to have a function which takes a pdf in and
returns the text.  Mark it immutable.

Then you can index the output of converting that text to a tsvector.

You may want to pull everything into a tsvector column for ease of review, but
functional indexes also make that less important

On Sat, Feb 20, 2016 at 1:10 AM, Stephen Davies > wrote:

On 20/02/16 00:24, Bruce Momjian wrote:

On Fri, Feb 19, 2016 at 02:49:16PM +0100, s d wrote:

On 19 February 2016 at 14:19, Bruce Momjian > wrote:
  > Ah, no. That's not possible
  >
  >
  > ...not possible, Yet.
  >
  > PostgreSQL grows by adding the features people need and
its changing
  rapidly.

  I wonder if PLPerl could be used to extract the words from a 
PDF
  document and create a tsvector column from it.

   I don't know about PLPerl(I'm pretty sure it could be used for
this purpose,
though.).  On the other hand I've written code for this in Python
which should
be easy to adapt for PLPython, if necessary.


Right, so you would write a PL/Perl or PL/Python trigger function that
would populate the tsvector column on every INSERT or UPDATE.

FWIW, I just use pdftotext in my CGI.

--

=
Stephen Davies Consulting P/L Phone: 08-8177 
1595
Adelaide, South Australia.Mobile:040 304 
0583



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org
)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general




--
Best Wishes,
Chris Travers

Efficito:  Hosted Accounting and ERP.  Robust and Flexible.  No vendor lock-in.
http://www.efficito.com/learn_more


I reckon my approach is simpler and easier (given web-based data entry).
I get all the meta data plus the PDF BLOB in one HTML request, get out the 
text and do the insert and all indexing including the tsvector in one PG request.
It also makes is easier to handle BLOB types other than PDF in the same CGI 
script as I just include the extracted text in the PG request.
There are readily callable text extraction utilities similar to pdftotext for 
all BLOB types that I see.


With a function, I would have to have separate functions or an extra BLOB-type 
parameter to the function and separate extraction logic in the function.



--
=
Stephen Davies Consulting P/L Phone: 08-8177 1595
Adelaide, South Australia.Mobile:040 304 0583


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Chris Travers 
A more general way would be to have a function which takes a pdf in and
returns the text.  Mark it immutable.

Then you can index the output of converting that text to a tsvector.

You may want to pull everything into a tsvector column for ease of review,
but functional indexes also make that less important

On Sat, Feb 20, 2016 at 1:10 AM, Stephen Davies  wrote:

> On 20/02/16 00:24, Bruce Momjian wrote:
>
>> On Fri, Feb 19, 2016 at 02:49:16PM +0100, s d wrote:
>>
>>> On 19 February 2016 at 14:19, Bruce Momjian  wrote:
>>>  > Ah, no. That's not possible
>>>  >
>>>  >
>>>  > ...not possible, Yet.
>>>  >
>>>  > PostgreSQL grows by adding the features people need and its
>>> changing
>>>  rapidly.
>>>
>>>  I wonder if PLPerl could be used to extract the words from a PDF
>>>  document and create a tsvector column from it.
>>>
>>>   I don't know about PLPerl(I'm pretty sure it could be used for this
>>> purpose,
>>> though.).  On the other hand I've written code for this in Python which
>>> should
>>> be easy to adapt for PLPython, if necessary.
>>>
>>
>> Right, so you would write a PL/Perl or PL/Python trigger function that
>> would populate the tsvector column on every INSERT or UPDATE.
>>
>> FWIW, I just use pdftotext in my CGI.
>
> --
>
> =
> Stephen Davies Consulting P/L Phone: 08-8177
> 1595
> Adelaide, South Australia.Mobile:040 304
> 0583
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>



-- 
Best Wishes,
Chris Travers

Efficito:  Hosted Accounting and ERP.  Robust and Flexible.  No vendor
lock-in.
http://www.efficito.com/learn_more

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Stephen Davies 

On 20/02/16 00:24, Bruce Momjian wrote:

On Fri, Feb 19, 2016 at 02:49:16PM +0100, s d wrote:

On 19 February 2016 at 14:19, Bruce Momjian  wrote:
 > Ah, no. That's not possible
 >
 >
 > ...not possible, Yet.
 >
 > PostgreSQL grows by adding the features people need and its changing
 rapidly.

 I wonder if PLPerl could be used to extract the words from a PDF
 document and create a tsvector column from it.

  I don't know about PLPerl(I'm pretty sure it could be used for this purpose,
though.).  On the other hand I've written code for this in Python which should
be easy to adapt for PLPython, if necessary.


Right, so you would write a PL/Perl or PL/Python trigger function that
would populate the tsvector column on every INSERT or UPDATE.


FWIW, I just use pdftotext in my CGI.

--
=
Stephen Davies Consulting P/L Phone: 08-8177 1595
Adelaide, South Australia.Mobile:040 304 0583


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Oleg Bartunov 
On Fri, Feb 19, 2016 at 8:28 PM, Josh berkus  wrote:

> On 02/19/2016 05:49 AM, s d wrote:
>
>> On 19 February 2016 at 14:19, Bruce Momjian > > wrote:
>>
>> I wonder if PLPerl could be used to extract the words from a PDF
>> document and create a tsvector column from it.
>>
>>
>>   I don't know about PLPerl(I'm pretty sure it could be used for this
>> purpose, though.).  On the other hand I've written code for this in
>> Python which should be easy to adapt for PLPython, if necessary.
>>
>
> I'd swear someone already built something to do this.  All you need is a
> library which reads PDF and transforms it into text, and then you can FTS
> it.  I know there's a module for OpenOffice docs somewhere as well, but
> heck if I can remember where.
>

I used pdftotext for that.
I think it'd be useful to have extension{s}, which can be used to convert
anything to text. I remember someone indexed chemical formulae,  TeX/LaTeX,
DOC files.


>
> --
> --
> Josh Berkus
> Red Hat OSAS
> (any opinions are my own)
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Josh berkus 

On 02/19/2016 05:49 AM, s d wrote:

On 19 February 2016 at 14:19, Bruce Momjian > wrote:

I wonder if PLPerl could be used to extract the words from a PDF
document and create a tsvector column from it.


  I don't know about PLPerl(I'm pretty sure it could be used for this
purpose, though.).  On the other hand I've written code for this in
Python which should be easy to adapt for PLPython, if necessary.


I'd swear someone already built something to do this.  All you need is a 
library which reads PDF and transforms it into text, and then you can 
FTS it.  I know there's a module for OpenOffice docs somewhere as well, 
but heck if I can remember where.


--
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Daniel Westermann 
>> 
>> I don't know about PLPerl(I'm pretty sure it could be used for this purpose, 
>> though.). On the other hand I've written code for this in Python which 
>> should 
>> be easy to adapt for PLPython, if necessary. 

>Right, so you would write a PL/Perl or PL/Python trigger function that 
>would populate the tsvector column on every INSERT or UPDATE. 

Thanks to all for your input 
Daniel

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Bruce Momjian 
On Fri, Feb 19, 2016 at 02:49:16PM +0100, s d wrote:
> On 19 February 2016 at 14:19, Bruce Momjian  wrote:
> >     Ah, no. That's not possible
> >
> >
> > ...not possible, Yet.
> >
> > PostgreSQL grows by adding the features people need and its changing
> rapidly. 
> 
> I wonder if PLPerl could be used to extract the words from a PDF
> document and create a tsvector column from it.
> 
>  I don't know about PLPerl(I'm pretty sure it could be used for this purpose,
> though.).  On the other hand I've written code for this in Python which should
> be easy to adapt for PLPython, if necessary.

Right, so you would write a PL/Perl or PL/Python trigger function that
would populate the tsvector column on every INSERT or UPDATE.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription +


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread s d 
On 19 February 2016 at 14:19, Bruce Momjian  wrote:

> On Fri, Feb 19, 2016 at 11:53:26AM +, Simon Riggs wrote:
> > On 19 February 2016 at 11:46, Thomas Kellerer 
> wrote:
> >
> > Daniel Westermann schrieb am 19.02.2016 um 12:41:
> >  if I'd need to implement/replace Oracle Text (ww.oracle.com/
> > technetwork/testcontent/index-098492.html).
> > > What choices do I have in PostgreSQL (9.5+) ?
> > >
> > >>Postgres also has a full text search (which I find much easier to
> use
> > than Oracle's):
> > >>
> > >>http://www.postgresql.org/docs/current/static/textsearch.html
> > >
> > > Yes, i have seen this. Can this be used to index and search binary
> > documents, e.g. pdf ?
> >
> > Ah, no. That's not possible
> >
> >
> > ...not possible, Yet.
> >
> > PostgreSQL grows by adding the features people need and its changing
> rapidly.
>
> I wonder if PLPerl could be used to extract the words from a PDF
> document and create a tsvector column from it.
>
>
 I don't know about PLPerl(I'm pretty sure it could be used for this
purpose, though.).  On the other hand I've written code for this in Python
which should be easy to adapt for PLPython, if necessary.



Ezt az e-mailt egy Avast védelemmel rendelkező, vírusmentes számítógépről
küldték.
www.avast.com 
<#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Bruce Momjian 
On Fri, Feb 19, 2016 at 11:53:26AM +, Simon Riggs wrote:
> On 19 February 2016 at 11:46, Thomas Kellerer  wrote:
> 
> Daniel Westermann schrieb am 19.02.2016 um 12:41:
>  if I'd need to implement/replace Oracle Text (ww.oracle.com/
> technetwork/testcontent/index-098492.html).
> > What choices do I have in PostgreSQL (9.5+) ?
> >
> >>Postgres also has a full text search (which I find much easier to use
> than Oracle's):
> >>
> >>http://www.postgresql.org/docs/current/static/textsearch.html
> >
> > Yes, i have seen this. Can this be used to index and search binary
> documents, e.g. pdf ?
> 
> Ah, no. That's not possible
> 
> 
> ...not possible, Yet.
> 
> PostgreSQL grows by adding the features people need and its changing rapidly. 

I wonder if PLPerl could be used to extract the words from a PDF
document and create a tsvector column from it.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Roman grave inscription +


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Simon Riggs 
On 19 February 2016 at 11:46, Thomas Kellerer  wrote:

> Daniel Westermann schrieb am 19.02.2016 um 12:41:
>  if I'd need to implement/replace Oracle Text (
> ww.oracle.com/technetwork/testcontent/index-098492.html).
> > What choices do I have in PostgreSQL (9.5+) ?
> >
> >>Postgres also has a full text search (which I find much easier to use
> than Oracle's):
> >>
> >>http://www.postgresql.org/docs/current/static/textsearch.html
> >
> > Yes, i have seen this. Can this be used to index and search binary
> documents, e.g. pdf ?
>
> Ah, no. That's not possible


...not possible, Yet.

PostgreSQL grows by adding the features people need and its changing
rapidly.

-- 
Simon Riggshttp://www.2ndQuadrant.com/

PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Andreas Joseph Krogh 
På fredag 19. februar 2016 kl. 12:41:49, skrev Daniel Westermann <
daniel.westerm...@dbi-services.com >:
>>Daniel Westermann schrieb am 19.02.2016 um 11:53:
>>> if I'd need to implement/replace Oracle Text 
(ww.oracle.com/technetwork/testcontent/index-098492.html).
  What choices do I have in PostgreSQL (9.5+) ?

 >Postgres also has a full text search (which I find much easier to use than 
Oracle's):
 >
 >http://www.postgresql.org/docs/current/static/textsearch.html

 Yes, i have seen this. Can this be used to index and search binary documents, 
e.g. pdf ?

 
What we do is extract plain-text from PFD/Word etc. clientside in the 
application, and then index that in the database.
Works very well.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com 
www.visena.com 
 


 

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Thomas Kellerer 
Daniel Westermann schrieb am 19.02.2016 um 12:41:
 if I'd need to implement/replace Oracle Text 
 (ww.oracle.com/technetwork/testcontent/index-098492.html). 
> What choices do I have in PostgreSQL (9.5+) ?
> 
>>Postgres also has a full text search (which I find much easier to use than 
>>Oracle's):
>>
>>http://www.postgresql.org/docs/current/static/textsearch.html
> 
> Yes, i have seen this. Can this be used to index and search binary documents, 
> e.g. pdf ?

Ah, no. That's not possible

 



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Daniel Westermann 
>>Daniel Westermann schrieb am 19.02.2016 um 11:53: 
>>> if I'd need to implement/replace Oracle Text 
>>> (ww.oracle.com/technetwork/testcontent/index-098492.html). 
 What choices do I have in PostgreSQL (9.5+) ? 

>Postgres also has a full text search (which I find much easier to use than 
>Oracle's): 
> 
>http://www.postgresql.org/docs/current/static/textsearch.html 

Yes, i have seen this. Can this be used to index and search binary documents, 
e.g. pdf ?

Re: [GENERAL] Replacement for Oracle Text

2016-02-19 Thread Thomas Kellerer 
Daniel Westermann schrieb am 19.02.2016 um 11:53:
> if I'd need to implement/replace Oracle Text 
> (ww.oracle.com/technetwork/testcontent/index-098492.html). 
> What choices do I have in PostgreSQL (9.5+) ?


Postgres also has a full text search (which I find much easier to use than 
Oracle's):

http://www.postgresql.org/docs/current/static/textsearch.html




-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

[GENERAL] Replacement for Oracle Text

2016-02-19 Thread Daniel Westermann 
Hi, 

if I'd need to implement/replace Oracle Text 
(ww.oracle.com/technetwork/testcontent/index-098492.html). What choices do I 
have in PostgreSQL (9.5+) ? 

Regards 
Daniel