Tsunakawa>So the statement-level rollback is newer to users, isn't it?
Technically speaking, the feature was listed in the changelog.
Tsunakawa>Doesn't PgJDBC execute RELEASE after each SQL statement?
It does not.
Tsunakawa>That said, even with RELEASE, the server memory bloat is not
solved.
Tsunakawa> PgJDBC has supported the feature with autosave parameter only
recently
PgJDBC has the implementation for more than a year (REL9.4.1210,
2016-09-07, see https://github.com/pgjdbc/pgjdbc/pull/477 )
Tsunakawa> The point raised in this thread was that that creates
Tsunakawa> too much
>OK, so here's a patch. Review appreciated.
Please correct typo "Write an unsigned integer field (anythign written with
UINT64_FORMAT)". anythign -> anything.
Vladimir
++pgjdbc dev list.
>I am facing unusual connection breakdown problem. Here is the simple code
that I am using to read WAL file:
Does it always fails?
Can you create a test case? For instance, if you file a pull request with
the test, it will get automatically tested across various PG versions,
BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VEVENT
DTSTART:20170616T06Z
DTEND:20170616T07Z
DTSTAMP:20170615T193848Z
ORGANIZER;CN=Vladimir Sitnikov:mailto:sitnikov.vladi...@gmail.com
, 8 мар. 2017 г. в 1:26, Vladimir Sitnikov <sitnikov.vladi...@gmail.com>:
> legrand>when usingversion 42.0.0 with
> legrand> jdbc:postgresql://localhost:5432/postgres?autosave=always
>
> The pitfall there is the value should be written with upper case like
> autosave=A
legrand>when usingversion 42.0.0 with
legrand> jdbc:postgresql://localhost:5432/postgres?autosave=always
The pitfall there is the value should be written with upper case like
autosave=ALWAYS.
I've filed https://github.com/pgjdbc/pgjdbc/issues/769 to improve that at
some point.
Vladimir
Hi,
In pgjdbc we have regular regression testing against "build from
master" PostgreSQL, and recent master builds fail for "statement cancel"
test.
The PostgreSQL as of Mon Oct 17 00:09:39 UTC 2016 was fine, then "statement
cancel" started to fail.
The test executes "select pg_sleep(10)" and
Ildar> Could you please try the patch and tell if it works for you?
I've tested patch6 against recent head. The patch applies with no problems.
The previous case (filter on top of i-o-s) is fixed. Great work.
Here are the test cases and results:
Ildar>The reason why this doesn't work is that '~~' operator (which is a
Ildar>synonym for 'like') isn't supported by operator class for btree. Since
Ildar>the only operators supported by btree are <, <=, =, >=, >, you can use
Ildar>it with queries like:
Ildar>And in 3rd query 'OFFSET' statement
Hi,
I've tried your indexonlypatch5.patch against REL9_6_BETA3.
Here are some results.
TL;DR:
1) <> does not support index-only
scan for index (type, upper(vc) varchar_pattern_ops).
3) <<(... where type=42 offset 0) where upper_vc like '%ABC%'>> does
trigger index-only scan. IOS reduces number
Tatsuo>understanding it always uses unnamed portal even if the SQL is like
"BEGIN" or "COMMIT" (no parameters). They are too often used. Why not
doing like this?
Does it actually work?
The documentation says named portals last till the end of the transaction:
Shay> your analogy breaks down. Of course L2 was invented to improve
performance,
Shay> but that doesn't mean that all caches are the same. More precisely, what I
Shay> find objectionable about your approach is not any caching - it's the
Shay> implicit or automatic preparation of statements. This
Vladimir>> Yes, that is what happens.
Vladimir>> The idea is not to mess with gucs.
Shay:> Wow... That is... insane...
Someone might say that "programming languages that enable side-effects
are insane".
Lots of connection pools work by sharing the connections and it is up
to developer
if he can
Shay> What? I really didn't understand your point here. All the doc is saying is
Shay> that if the driver doesn't support prepared statements, then using them
Please read again. PreparedStatement is the only way to execute statements
in JDBC API. There's no API that allows user to specify "use
Tatsuo>Interesting. What would happen if a user changes some of GUC
parameters? Subsequent session accidentally inherits the changed GUC
parameter?
Yes, that is what happens.
The idea is not to mess with gucs.
Tatsuo>There's nothing wrong with DICARD ALL
Tatsuo>"DISCARD ALL" is perfect for this
Shay>To be honest, the mere idea of having an SQL parser inside my driver
makes me shiver.
Same for me.
However I cannot wait for PostgreSQL 18 that does not need client-side
parsing.
Shay>We did, you just dismissed or ignored them
Please prove me wrong, but I did provide a justified answer to
Tom> while giving something at least passable in the cases
that are broken today.
Would you mind adding an explicit "encoding" field to the error message?
At least it would give clear explanation how to parse that message without
resorting to a guess dance.
The biggest problem is client has no
Victor>It is not a client job to convert encodings.
Of course.
However, there is a vast amount of old PG versions deployed in the wild
that send wrong data to clients.
This indeed makes bad user experience, so it is worth doing 2 things:
1) Implement proper solution in further PostgreSQL
Shay>I don't know much about the Java world, but both pgbouncer and pgpool
(the major pools?)
In Java world, https://github.com/brettwooldridge/HikariCP is a very good
connection pool.
Neither pgbouncer nor pgpool is required.
The architecture is: application <=> HikariCP <=> pgjdbc <=>
Victor>We don't have 190 message catalog translations in the PostgreSQL.
Victor>So problem with encoding for messages is quite limited.
Even though the number of translations is limited, there's a problem when
trying to tell one "one-byte-encoding" from another "one-byte" one.
It would be so
Shay>As I said, an error is going to kill the ongoing transaction, how can
this be solved without application logic?
1) At least, some well-defined error code should be created for that kind
of matter.
2) The driver can use safepoints and autorollback to the good "right before
failure" state in
Shay:
> Prepared statements can have very visible effects apart from the speedup
> they provide (e.g. failure because of schema changes) It's not that these
> effects can't be worked around - they can be - but programmers can be
> surprised by these effects, which can cause difficult-to-diagnose
Shay> it's important to note that query parsing and rewriting isn't an
"inevitable evil".
Ok, I stay corrected. ADO.NET have raw mode in the API. That's interesting.
Let's say "lots of heavily used languages do have their own notion of bind
placeholders".
And for the reset, it is still not that
Stephen> While it may have good results in many cases, it's not accurate to
say that using prepared statements will always be faster than not.
There's no silver bullet. <-- that is accurate, but it is useless for
end-user applications
I've never claimed that "server prepared statement" is a
Stephen>I encourage you to look through the archives
The thing is pl/pgsql suffers from exactly the same problem.
pl/pgsql is not a typical language of choice (e.g. see Tiobe index and
alike), so the probability of running into "prepared statement issues" was
low.
As more languages would use
Robert>But that makes it the job of every driver to implement some
sort of cache, which IMHO isn't a very reasonable position
Let's wait what Shay decides on implementing query cache in npgsql ?
Here's the issue: https://github.com/npgsql/npgsql/issues/1237
He could change his mind when it
Robert Haas:
> but for some reason you can't use prepared statements, for example because
> the queries are dynamically generated and . That case is analogous to -M
> extended, not -M prepared. And -M extended is well-known to be SLOWER
>
I do not buy that "dynamically generated queries defeat
Shay>But here's the more important general point. We're driver developers,
not application developers. I don't really know what performance is "just
fine" for each of my users, and what is not worth optimizing further. Users
may follow best practices, or they may not for various reasons.
Of
Shay>There are many scenarios where connections are very short-lived (think
about webapps where a pooled connection is allocated per-request and reset
in between)
Why the connection is reset in between in the first place?
In pgjdbc we do not reset per-connection statement cache, thus we easily
I'm sorry, we are discussing technical details with no real-life use case
to cover that.
I do not want to suck time for no reason. Please accept my sincere
apologies for not asking the real-life case earlier.
Shay, can you come up with a real-life use case when those "I claim the
statement will
Shay Rojansky :
> Ah, I understand the proposal better now - you're not proposing encoding a
> new message type in an old one, but rather a magic statement name in Parse
> which triggers an optimized processing path in PostgreSQL, that wouldn't go
> through the query cache etc.
>
Shay Rojansky :
> That's definitely a valid point. But do you think it's a strong enough
> argument to avoid ever adding new messages?
>
The point is "adding a message to current v3 protocol is not a backward
compatible change".
The problem with adding new message types is not
Tom Lane :
> FWIW, this thread started on 25-July, less than two weeks ago.
Technically speaking, there was a pgsql-jdbc thread started on May 14:
https://www.postgresql.org/message-id/nh72v6%24582%241%40ger.gmane.org
9.6beta1 was released on May 12
The fact that it wasn't
Shay Rojansky :
>
> That sounds right to me. As you say, the server version is sent early in
> the startup phase, before any queries are sent to the backend, so frontends
> know which server they're communicating with.
>
> We could call this "protocol 3.1" since it doesn't break
Tom Lane :
> [ shrug... ] What do you claim is not machine-readable about
> server_version?
>
0) server_version needs a dance to parse.
For instance, recent "Stamp version 10devel" patch did touch
"server_version" parsing in fe-exec.c:
>
> Sorry, but I don't buy that. I think sending both server_version and
> server_version_num would be silly, and we're certainly not going to stop
> sending server_version.
>
What is wrong with sending machine-readable value?
Vladimir
Tatsuo Ishii :
> Doesn't this patch break an existing behavior of unnamed statements?
> That is, an unnamed statement shall exist until next parse message
> using unnamed statement received. It is possible to use the same
> unnamed statement multiple times in a transaction.
Tom Lane :
> Bruce Momjian writes:
> > On Sun, Jul 31, 2016 at 05:57:12PM -0400, Tom Lane wrote:
> >> In hindsight it seems clear that what a lot of apps want out of extended
> >> protocol is only the ability to send parameter values out-of-line
> instead
>
>
> I really don't get what's problematic with posting a message on a mailing
> list about a potential performance issue, to try to get people's reactions,
> without diving into profiling right away
>
"Benchmark data is a perfect substitute for benchmarking results. Data is
easy to misinterpret,
Shay Rojansky :
> I'm well aware of how the extended protocol works, but it seems odd for a
> 30% increase in processing time to be the result exclusively of processing
> 5 messages instead of just 1 - it doesn't seem like that big a deal
> (although I may be mistaken). I was
>
>
> >> That's a recipe for runaway table bloat; VACUUM can't do much because
> >> there's always some minutes-old transaction hanging around (and SNAPSHOT
> >> TOO OLD doesn't really help, we're talking about minutes here), and
> >> because of all of the indexes HOT isn't effective.
>
Just
Craig>That moves work further away from the DB, which has its own costs,
and isn't something you're likely to be happy with if you're looking at
things like optimising PL/PgSQL with a bytecode compiler. But it's the best
we have right now.
What if JVM was started within a background worker?
Then
> The problem is that, even if I set the reltuples and relpages of my choice, when I run the EXPLAIN clause for a query in which the 'newTable' is involved in (e.g. EXPLAIN SELECT * FROM newTable), I get the
Tom> Not to mention that the whole idea of that being a semantically
Tom> significant property of a name is a monstrous kluge.
You are right here.
Just in case, Marko Kreen says (see [1]) pgbouncer has all the information
required to remap statement names, so he says pgbouncer needs no
Tom>If you think that's not a protocol change, you are mistaken. It
Tom>changes a behavior that's specified in the protocol documentation.
Even if it requires documentation, this particular change will work seamlessly
across existing implementations of v3 protocol.
For instance, it would not
Craig>I really, really doubt you can change this before we do a
protocol version bump.
Technically speaking, the idea of using first bytes of statement name
to convey extra information does not require protocol version bump. It
can be backward and forward compatible.
For instance: if statement
Merlin> All I'm saying is that the use of
Merlin> server side prepared statements is extremely problematic in
Merlin> conjunction with pgbouncer
I've filed https://github.com/pgbouncer/pgbouncer/issues/126 to get
pgbouncer improved in regard to prepared statements.
Vladimir
--
Sent via
Merlin>proposed would allow use of server side prepared statements with JDBC.
It would not. If we discuss end-to-end scenarios in detail, we would end up with
"send full query on each execution" -> lex/gram on each execution kind
of overheads.
That is hardly a proper way of using prepared
Merlin>No one is arguing that that you should send it any every time (at
least -- I hope not).
Well, what is your suggestion exactly?
pgjdbc is NOT using "prepare ..." sql command.
I'm inclined to suppose, it will not use "prepare..." even after your fix.
Merlin>Again, not in pooling scenarios
2016-03-23 16:21 GMT+03:00 Merlin Moncure :
> On Wed, Mar 23, 2016 at 7:27 AM, Craig Ringer wrote:
Craig>> With PREPARE IF NOT EXISTS the client is also paying parse and network
Craig>> overhead for every time you send that statement. Much better
not to
Igal, thanks for the analysis.
Craig>Part of the question for Pg is what exactly we should and should not
be returning.
I think the following might be a good starting point: return set of columns
that would identify the inserted row(s).
E.g. columns of any primary key would do. Columns of any
Robert>Hmm, so in your example, you actually want replanning to be able to
Robert>change the cached plan's result type?
I want backend to cache _several_ plans behind a single "statement name".
I want to treat "prepare...exec...deallocate" dance as an optimization
step for a simple
Why do you want that at the database level?
Do you have end-to-end scenario that benefits from using Lucee?
>I was wondering how difficult it would be to implement a Postgres extension
>that will act as a wrapper around it and will allow to write functions in that
>language?
Have you checked
> If the pl/v8 was easily ported to Windows then I probably wouldn't even try
> to add Lucee,
That is a good question. ChakraCore has been open sourced recently. It
might be easier to build under Windows.
>That seems like a good place to start, thanks
I am not sure you would be able to bind
I want to treat 'prepare' operation as an optimization step, so it is
functionally equivalent to sending a query text.
In other words, I would like backend to track search_path and other parameters
if necessary transparently, creating (caching) different execution plans if
different plans are
Robert>Are you really seeing the same behavior in all versions?
I do not have "pre 9.1" at hand, however all 9.1, 9.2, 9.3, 9.4, and
9.5 are affected.
9.1 just silently executes "old statement" as if search_path was not
modified at all.
9.2, 9.3, 9.4, and 9.5 all fail with "cached plan must not
> I believe, and the conclusion was that
>if you think you need this, you're doing it wrong
So what is the recommended approach to use server-prepared statements
at the client side (I mean at JDBC driver side)?
Currently "prepare, switch search_path, execute" leads to "cached plan
must not
;=BE ParseComplete [null]
<=BE BindComplete [unnamed]
<=BE NoData
<=BE CommandStatus(SET)
<=BE ReadyForQuery(I)
Am I missing something?
[1]: http://www.postgresql.org/message-id/22921.1358876...@sss.pgh.pa.us
[2]: https://github.com/pgjdbc/pgjdbc/issues/496
Vladimir Sitnikov
--
Sen
t time is
spent in standard_ExecutorStart: see [3]
In fact, the time spent in standard_ExecutorStart even exceeds the
time spent in standard_ExecutorRun.
[1]: http://www.postgresql.org/message-id/55130dc8.2070...@redhat.com
[2]: https://github.com/pgjdbc/pgjdbc/pull/491#issuecomment-171780054
[3]: https://g
>I guess you mean there's a transaction surrounding it?
Sure there is a transaction.
I measure the latency from the first Bind message to the ReadyForQuery response.
The database is at localhost.
The flow is as follows (I've use 4 queries in batch for brevity,
however the test above is executed
>the implementation is simply - but it hard to design some really general - it
>is task for UI
Can you please rephrase?
Current design is "if the cost of a generic plan is less than the one
of a custom plan+replan, prefer generic".
I think that is wrong.
"Generic plan" misunderestimates a cost
Note: I do not suggest changing already cached plans yet.
I suggest looking into "6th bind values" when building a cached plan.
In other words, "if first 5 execution do not reveal dependence on bind
values, then cache the generated plan".
>Say you already have a plan which looks like this:
>Now
> The custom plan is *more expensive*;
You compare costs of custom vs generic plans.
I suggest: do not compare costs *at all*.
>I don't know, it's your proposal :-) But it looks like I misunderstood.
It is not.
My suggestion is: build a generic plan (that is the plan that will
return proper
Please, read my suggestion again.
TL;DR: I suggest to create "generic plan" with regard to current bind values.
What's wrong with that approach?
Vladimir
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
>so you don't get to (or want to) have any control over the underlying prepared
>statement.
That is pl/pgsql's problem, isn't it?
In the mean time, user can use different query texts (e.g. by adding
offset 0, offset 0*1, offset 0*2, etc kind of stuff they typically use
to tune queries) to
>A value of -1 could disable generic plans
I do not like the idea.
I've seen dramatic performance improvements from using cached plans.
The numbers are like "20ms to plan vs 1ms to execute" for an often
used OLTP query. Query text is involved (~5-10KiB).
Vladimir
--
Sent via pgsql-hackers
>(1) the generic plan is falsely optimistic
That is my case.
Application is sending most common value on every execution while
backend is optimistic and it things that the app would stop sending
MCVs.
Costs for the plans are OK. However, there is a data skew, so it is
hard to tell what is the
>If plan is ok for one value parameters, then can be pretty bad for following
>parameters.
Happy statements are all alike; every unhappy statement is unhappy in
its own way (see [1]).
If user is sending different kinds of parameters, he is shooting in the foot.
>Albe's proposal can be good
> for one custom plans can be much better than the generic plan, independent of
> cardinalities
So what? I do not suggest dropping custom plans entirely.
I perfectly understand there are cases when better replan every time.
> consider e.g a table with one somewhat common and otherwise just
>This completely screws over PL/PgSQL, among other things.
Can you elaborate a bit?
Vladimir
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
>Basically you're arguing to fix one specific edge case which bugs you
>personally, by creating a lot of others, which don't bug you. Not
>convincing.
It bugs me.
It bugs clients of pgjdbc (e.g. Thomas who started the thread).
Note: support of prepared statements for java applications has just
> Generally using a very small sample
That is another issue. Inventing some other algorithm instead of
current "cache after 5 executions" is another effort.
However, I suggest to "learn" from what client is sending.
You suggest to completely ignore that and just prepare for the case
he/she will
Here's the simplified testcase:
https://gist.github.com/vlsi/df08cbef370b2e86a5c1
It reproduces the problem in both 9.4.4 and 9.5rc1.
It is reproducible via both psql and pgjdbc.
I use a single table, however my production case includes a join of
two tables and the query is like
select ... from
> I don't know if there is a good solution except disabling server prepared
> statements.
Why doesn't backend reuse already existing good plan?
The plan does account for the skew.
Can backend take selectivities from the original bind values?
Vladimir
--
Sent via pgsql-hackers mailing list
VS>>Why doesn't backend reuse already existing good plan?
PS>this logic is driven by plan cost, not by plan execution time.
It do understand that currently PG replans with $1, $2 and uses
default selectivities for that.
What I am asking is to make PG aware of "previously used bind values",
so it
>Have you got numbers showing any actual performance win for postgres_fdw?
For JDBC purposes, it would be nice to have an ability of asking
backend "to stop fetching if produced more than X MiB of response
data".
For small table (4 int4 fields), having decent fetchSize (~1000) makes
result
>and fetch a number of rows that, by its estimation, would fit in the memory
>available
What's wrong with having size limit in the first place? It seems to
make much more sense.
Vladimir
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your
> a global lock would be good enough for a proof of concept that only
evaluates cache hit ratios.
I think emulator can be used to check hit ratios. That way we can see
how different algorithms affect hit ratio.
Is there a set of traces of "buffer load events"? (I did some Google
searches like
s/caffeine/issues/23#issuecomment-161536706
Vladimir Sitnikov
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
There is already a patch for that (ignore/respect nulls in lead/lag):
https://commitfest.postgresql.org/action/patch_view?id=1096
--
Vladimir
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
.
Regards,
Vladimir Sitnikov
The thing is PostgeSQL JDBC driver should be able to parse sql in order to
tell if specific question mark is a bind variable or it is inside string
literal, or it is inside of some comment.
I do not believe JDBC spec forces to treat all ? as bind placeholders.
That's unusable.
Oracle JDBC allows
that would improve N^2 to N)
Regards,
Vladimir Sitnikov
lookup might be better choice.
Regards,
Vladimir Sitnikov
-December there are as many X rows, for
2008-November as many as Y, etc. That could be used for rather accurate
cardinality estimation of between cases, while keeping number of entries
in MCV list small.
Regards,
Vladimir Sitnikov
reasonable accuracy of estimations.I have no idea how to decide between
automatic switch between histograms and MCV. It might sound crazy one could
compute both histograms and MCV and use them both (and pick an average of
two estimations :) )
Regards,
Vladimir Sitnikov
What is the specific difference between what you are talking about and
what scalarineqsel already implements?
Hmm... Northing new. Feel sorry for bothering you. I did not realize
histograms are implemented.
Regards,
Vladimir Sitnikov
sense having several options for explain if that would
result in *different instrumentation *(e.g. explain vs explain analyze).
Regards,
Vladimir Sitnikov
really want that?
Regards,
Vladimir Sitnikov
at a separate
tab. I mean, even before/after plain select (without any explain). That
will show you how the query would behave without any instrumentation.
Regards,
Vladimir Sitnikov
what the sense could be in one session explained into plan_table, while the
other reads that plan.
Does that make sense?
Regards,
Vladimir Sitnikov
,
Vladimir Sitnikov
Vladimir Sitnikov [EMAIL PROTECTED] writes:
I wish there was a way to get the results of explain into some table. I
wish
it was the default output format. That would make life of pgAdmin
easier,
and improve readability even in psql. Do not you think there is something
wrong
several separate queries since optimizer
could use an index scan to get 20 top and seq scan to get the count(*)
Regards,
Vladimir Sitnikov
,
Vladimir Sitnikov
One more point on pg_am: amsearchnull is equal to f however the index
stores and could find nulls perfectly.
Regards,
Vladimir Sitnikov
name,CAST(r AS FLOAT) / c, cd
FROM (SELECT name,
COUNT(*) OVER(ORDER BY salary) as r,
COUNT(*) OVER() AS c,
CUME_DIST() OVER(ORDER BY salary) AS cd
FROM employees
) t;
Sincerely yours,
Vladimir Sitnikov
.
If the patch returns row_number it is wrong since there is no way for
row_number to be a number of rows preceding or peer with R, is there?
Regards,
Vladimir Sitnikov
on the way.
Both testcases reveal defects in index creation.
Regards,
Vladimir Sitnikov
1 - 100 of 109 matches
Mail list logo