Re: [HACKERS] Parser emits mysterious error message for very long tokens

[Kyotaro HORIGUCHI]

Hello, thank you for the opinion.

At Fri, 11 Sep 2015 09:31:30 -0400, Tom Lane  wrote in 
<884.1441978...@sss.pgh.pa.us>
> Kyotaro HORIGUCHI  writes:
> > Hello, this is a problem on an extreme situation.
> > When parser encounters very long tokens, it returns an error
> > message which should be mysterious for users.
> 
> >> $ perl -e "print \"select '\" . \"x\"x(512*1024*1024) . \"'\"" | psql 
> >> postgres
> >> ERROR:  invalid memory alloc request size 1073741824
> 
> I can't get terribly excited about that, because there is not that
> much daylight between there and where the query fails because the
> entire input string exceeds 1GB.  Moreover, anyone who tries working
> with literals in this size range will soon learn not to ;-).  So
> it seems quite an artificial example to me ...
> 
>   regards, tom lane

Yes, so 'an extreme situation'. I wanted to know what to think
about this kind of situation.

Thanks for the comment, I'm confident that fixing it and the
similars in clean way should be too-much for the necessity.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parser emits mysterious error message for very long tokens

[Tom Lane]

Kyotaro HORIGUCHI  writes:
> Hello, this is a problem on an extreme situation.
> When parser encounters very long tokens, it returns an error
> message which should be mysterious for users.

>> $ perl -e "print \"select '\" . \"x\"x(512*1024*1024) . \"'\"" | psql 
>> postgres
>> ERROR:  invalid memory alloc request size 1073741824

I can't get terribly excited about that, because there is not that
much daylight between there and where the query fails because the
entire input string exceeds 1GB.  Moreover, anyone who tries working
with literals in this size range will soon learn not to ;-).  So
it seems quite an artificial example to me ...

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Parser emits mysterious error message for very long tokens

[Kyotaro HORIGUCHI]

Hello, this is a problem on an extreme situation.

When parser encounters very long tokens, it returns an error
message which should be mysterious for users.

> $ perl -e "print \"select '\" . \"x\"x(512*1024*1024) . \"'\"" | psql postgres
> ERROR:  invalid memory alloc request size 1073741824

Since parser repeats repalloc doubling in size starting from 1024
bytes, it practically fails for tokens longer than 512MiB. Most
tokens doesn't touch the limit but text literal may do.

If we don't assume this as a problem, the following discussion
would be useless.


1. Supplying context meessage or something would be needed anyway.

  > ERROR:  invalid memory alloc request size 1073741824
  > DETAIL: maybe encoutered too long token in parsing query


2. Edit the documentaion, or modify the behavior?

 http://www.postgresql.org/docs/devel/static/datatype-character.html 

 >  In any case, the longest possible character string that can be
 >  stored is about 1 GB.

 Maximum possible text length would be about 1GB but it is far
 shorter when parsing text literal.

 I thought it is better to avoid 512MiB limit than adding
 discription about it to documentation.  It could easily be done
 using MaxAllocSize (I don't like this but enlargeStringInfo
 already does this..), or start from (1024 - 1) and repalloc with
 ((last_size + 1) * 2 - 1) would do assuming MaxAllocSize is 2^n
 (this looks unwantedly complex), or simply starts from 1023
 works enough.

 On the other hand, any of these fixes caused "out of memory" of
 receive buffer on my environment. So I think such fix might be
 useless.

Thoughts? Sugestions? or is this totally useless?


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers