> I ran the new Pgpool-1.2.2 and it was a bit faster on
> the TCP but still slower than on UNIX socket. I used
> the same script as before.
>
> TCP Socket (Pgpool 1.2.0)
> --
> 2.39 sec
>
> TCP Socket (Pgpool 1.2.2)
> --
> 0.80 sec
> 0.80 sec
> 0.79 sec
>
> UNIX Socket (Pgpool 1.
I ran the new Pgpool-1.2.2 and it was a bit faster on
the TCP but still slower than on UNIX socket. I used
the same script as before.
TCP Socket (Pgpool 1.2.0)
--
2.39 sec
TCP Socket (Pgpool 1.2.2)
--
0.80 sec
0.80 sec
0.79 sec
UNIX Socket (Pgpool 1.2.2)
---
0.026 sec
0.0
> > Sorry for the confusion because Rod asked a question
> > and I answered too quickly. This is what I mean.
> >
> > 15x Slower:
> > ---
> > Client <--TCP--> PgPool <--UNIX--> PostgreSQL
> > Client <--TCP--> PgPool <--TCP--> PostgreSQL
> >
> > 5x Faster:
> > --
> > Client <--UNIX
> Hi Bruce,
>
> Sorry for the confusion because Rod asked a question
> and I answered too quickly. This is what I mean.
>
> 15x Slower:
> ---
> Client <--TCP--> PgPool <--UNIX--> PostgreSQL
> Client <--TCP--> PgPool <--TCP--> PostgreSQL
>
> 5x Faster:
> --
> Client <--UNIX--> PgP
On Thu, 6 May 2004, Tom Lane wrote:
> sdv mailer <[EMAIL PROTECTED]> writes:
> > The point is pre-forking can *potentially* speed up
> > connections by 5x as shown in this simplistic
> > non-conclusive benchmark.
>
> I think this "benchmark" proves no such thing.
>
> The thing that pgpool is doi
Tom Lane writes:
... too much flushing ...
I agree. I'll bet replacing the pool_write_and_flush() calls in
BinaryRow()
and AsciiRow() with just pool_write(), followed by removing the
fflush() calls at the bottom of those two methods should go a long
way towards fixing things, since the Com
On May 6, 2004, at 1:06 PM, sdv mailer wrote:
I compared against both TCP and UNIX direct
connections. No SSL, no authentication. See benchmark
results posted below again:
I recall your script only connected 20 times - that is not enough to
filter out "noise" in those numbers. Please run it again
Tom,
You're correct about the test measuring a hot backend
and not forking. How much exactly is the "bulk of the
startup" done by cache initialization relative to the
forking? What would be the impact on Win32 knowing
that process creation is twice as slow than on Linux?
James Robinson <[EMAIL PROTECTED]> writes:
> Quick overview of the code for differences in TCP-on-the-frontend code
> is a call to setsockopt(..., TCP_NODELAY, ...) if the connection to the
> frontend is a TCP socket. Could this be producing pseudo-fragmentation,
> resulting in over-the-top cont
sdv mailer <[EMAIL PROTECTED]> writes:
> The point is pre-forking can *potentially* speed up
> connections by 5x as shown in this simplistic
> non-conclusive benchmark.
I think this "benchmark" proves no such thing.
The thing that pgpool is doing is not preforking connections at all, but
re-using
Quick overview of the code for differences in TCP-on-the-frontend code
is a call to setsockopt(..., TCP_NODELAY, ...) if the connection to the
frontend is a TCP socket. Could this be producing pseudo-fragmentation,
resulting in over-the-top context switches? Looks like
pool_process_query() does
Nope. I commented out that block of code at 372 and no
difference.
__
Do you Yahoo!?
Win a $20,000 Career Makeover at Yahoo! HotJobs
http://hotjobs.sweepstakes.yahoo.com/careermakeover
---(end of broadcast)--
I compared against both TCP and UNIX direct
connections. No SSL, no authentication. See benchmark
results posted below again:
Direct
--
0.144 sec. Client <--UNIX--> PG
0.152 sec. Client <--TCP--> PG
5x Faster
-
0.027 sec. Client <--UNIX--> Pgpool <--UNIX--> PG
0.028 sec. Client <--UNI
On May 6, 2004, at 12:19 PM, sdv mailer wrote:
15x Slower:
---
Client <--TCP--> PgPool <--UNIX--> PostgreSQL
Client <--TCP--> PgPool <--TCP--> PostgreSQL
5x Faster:
--
Client <--UNIX--> PgPool <--UNIX--> PostgreSQL
Client <--UNIX--> PgPool <--TCP--> PostgreSQL
If the problem were in
The speedup (UNIX) and slowdown (TCP) are both
compared against normal direct connections from Client
to PostgreSQL. This means with Pgpool (UNIX) it is 5x
faster than normal connections to PostgreSQL. It is
also 15x slower with Pgpool (TCP) compared to normal
connections to PostgreSQL.
My guess i
No SSL. No authentication either. Just friendly
handshakes.
__
Do you Yahoo!?
Win a $20,000 Career Makeover at Yahoo! HotJobs
http://hotjobs.sweepstakes.yahoo.com/careermakeover
---(end of broadcast)-
sdv mailer wrote:
>Hi Bruce,
>
>Sorry for the confusion because Rod asked a question
>and I answered too quickly. This is what I mean.
>
>15x Slower:
>---
>Client <--TCP--> PgPool <--UNIX--> PostgreSQL
>Client <--TCP--> PgPool <--TCP--> PostgreSQL
>
>5x Faster:
>--
>Client <--UNIX-
sdv mailer wrote:
> The speedup (UNIX) and slowdown (TCP) are both
> compared against normal direct connections from Client
> to PostgreSQL. This means with Pgpool (UNIX) it is 5x
> faster than normal connections to PostgreSQL. It is
> also 15x slower with Pgpool (TCP) compared to normal
> connecti
sdv mailer wrote:
> Hi Bruce,
>
> Sorry for the confusion because Rod asked a question
> and I answered too quickly. This is what I mean.
>
> 15x Slower:
> ---
> Client <--TCP--> PgPool <--UNIX--> PostgreSQL
> Client <--TCP--> PgPool <--TCP--> PostgreSQL
>
> 5x Faster:
> --
> Cli
Hi Bruce,
Sorry for the confusion because Rod asked a question
and I answered too quickly. This is what I mean.
15x Slower:
---
Client <--TCP--> PgPool <--UNIX--> PostgreSQL
Client <--TCP--> PgPool <--TCP--> PostgreSQL
5x Faster:
--
Client <--UNIX--> PgPool <--UNIX--> PostgreSQL
sdv mailer wrote:
> Pgpool connects to PostgreSQL on UNIX socket. I also
> ran on TCP socket but there is no significant
> difference if I recall correctly due to the inherent
> nature of connection pooling or pre-fork technology.
> ;-)
I am confused. First you said TCP was slower, but now you s
Pgpool connects to PostgreSQL on UNIX socket. I also
ran on TCP socket but there is no significant
difference if I recall correctly due to the inherent
nature of connection pooling or pre-fork technology.
;-)
--- Rod Taylor <[EMAIL PROTECTED]> wrote:
> > However, when I tried TCP socket, Pgpool
> However, when I tried TCP socket, Pgpool was actually
> slower by 15x !! Perhaps you can clarify why the TCP
> socket is so much slower?
How did you have pgpool configured to connect to the database? Domain
socket or tcpip?
---(end of broadcast)-
> "sdv" == sdv mailer <[EMAIL PROTECTED]> writes:
sdv> Yes, I realize it's a bit old but I just wanted to make a
sdv> small point that forking is slower. It's funny you should ask
sdv> because thread creation on Linux has in fact improved over
sdv> process creation much more in
Ok, I did some benchmark on my Linux box (AMD 1.2Ghz,
256MB, Fedora Core 1 Linux 2.4.20-8) using Pgpool 1.2
and PostgreSQL 7.4. I ran the benchmark script
repeatedly (10+ times each).
I get 5x faster using Pgpool on UNIX socket, which is
encouraging. This shows pre-fork does speed things up
even w
Tatsuo,
I did some benchmark on my Linux box (AMD 1.2Ghz,
256MB, Fedora Core 1 Linux 2.4.20-8) using Pgpool 1.2
and PostgreSQL 7.4. I ran the benchmark script
repeatedly (10+ times each).
I get 5x faster using Pgpool on UNIX socket, which is
encouraging. This shows pre-fork does speed things up.
> > Have you looked at sqlrealy.sourceforge.net? IT looks like it might do
> > what you need.
>
> SQL Relay (and friends) do what I'm looking for in a round about way.
>
> If you put it onto the webservers it would help -- but it would require
> deployment of additional webservers to accommodat
Yes, I realize it's a bit old but I just wanted to
make a small point that forking is slower. It's funny
you should ask because thread creation on Linux has in
fact improved over process creation much more in 2.4
kernel.
Benchmark at IBM shows Linux 2.4 thread creation is
30x faster than process c
sdv mailer said:
> Forking is expensive on many systems. Linux is a bit
> better but still expensive compared to threads. On
> Windows, creating process is much more expensive than
> on Linux. Check this benchmark:
>
> http://cs.nmu.edu/~randy/Research/Papers/Scheduler/understanding.html
>
> Forkin
I don't think I can volunteer on this end as I am
already actively volunteering for another open
project. I was hoping someone could take up on this
since one of the last threads mentionned we don't have
something substantial to present for 7.5 if June 1 is
dateline for code freeze. Pre-fork came t
I'll pretend I didn't see that last comment on
Windows. I wouldn't want to disappoint the users who
are eagerly expecting the Win32 port to complete
including myself. ;-)
Having said that, I think it's more the reason to get
a working pre-fork for Win32. Don't you think so?
--- "scott.marlowe
Forking is expensive on many systems. Linux is a bit
better but still expensive compared to threads. On
Windows, creating process is much more expensive than
on Linux. Check this benchmark:
http://cs.nmu.edu/~randy/Research/Papers/Scheduler/understanding.html
Forking shouldn't be taken lightly as
I've already tried pooling (SQLRelay) and persistent
connection (PHP). They may work for other people but
they do not work for me. I have already separated
static from database driven codes but you can never
balance web server load with database server load.
Pre-fork scales with database load and
Rod Taylor wrote:
As a temporary step these types of things help. But it's still doesn't
really fix the problem of Apache not using real connection pooling.
Rod,
In principle, this should not be enormously hard to do - at least for
Unix where the methods of handing off file handles between pro
> > I cannot tell if mod_pg_pool works across Apache forked backends or is
> > still bound to a single process. They state it is intended for sharing
> > connections across modules, so it is probably still backend specific.
>
> Have you looked at sqlrealy.sourceforge.net? IT looks like it might d
On Wed, 5 May 2004, Rod Taylor wrote:
> > And, of course, most development environments (perl, php, java etc)
> > have their own language specific connection pooling solutions.
>
> Yes, the one for php is what I was thinking of when I made my statement.
> They work on a per backend basis as Apach
> And, of course, most development environments (perl, php, java etc)
> have their own language specific connection pooling solutions.
Yes, the one for php is what I was thinking of when I made my statement.
They work on a per backend basis as Apache does not allow for the type
of communication be
The fact that windows has a heavy process / lightweight thread design
means little to me, since I'll likely never deploy a production postgresql
server on it that needs to handle any serious load.
Yes but Solaris also has a heavy process / lightweight thread design.
J
--
* Steve Atkins ([EMAIL PROTECTED]) wrote:
> On Wed, May 05, 2004 at 02:12:58PM -0400, Rod Taylor wrote:
> >
> > Most of it has been. It's the duty cycle. As stated in another email,
> > only about 20% of the work a script does is database related -- which
> > occurs all at one time. Even when all
On Wed, 5 May 2004, sdv mailer wrote:
> Forking is quite fast on Linux but creating a new
> process is still 10x more expensive than creating a
> thread and is even worse on Win32 platform. CPU load
> goes up because the OS needs to allocate/deallocate
> memory making it difficult to get a steady
On Wed, May 05, 2004 at 02:12:58PM -0400, Rod Taylor wrote:
>
> Most of it has been. It's the duty cycle. As stated in another email,
> only about 20% of the work a script does is database related -- which
> occurs all at one time. Even when all Apache backends are active, a
> large number of conn
On Wednesday 05 May 2004 07:24 am, Rod Taylor wrote:
> > And "preforking" makes this different, how ? Perhaps having a pool of
> > processes ready to be handed a query to a specific database, where you
> > configure N connections to db1, M to db2 etc. still means lots of
> > resource usage. In effe
sdv mailer wrote:
[snip]
Pre-fork will give MySQL one less argument to throw at
PostgreSQL.
I think optimizing is this area will speed up the
general case for everyone rather than optimizing a
feature that affects 10% of the users. On top of that,
it will make a strong marketing case because fork
On Wed, 2004-05-05 at 11:57, Greg Stark wrote:
> Rod Taylor <[EMAIL PROTECTED]> writes:
>
> > Cutting that count down to 10 idlers in total by having PostgreSQL
> > prefork a specific database would make a significant difference.
>
> Well it would be 10 for each database. Since as has been pointe
sdv mailer wrote:
I'm talking about connecting to multiple database
servers on separate machines. Schemas don't apply
here.
How much work would it take to make a pre-fork smart
enough to open different databases on incoming
connection? How much of it can be modeled after
Apache?
I've not used it bu
> Or, you run several seperate Apache webservers. The ones that serve static
> content or don't need database connections don't run with the ones that do.
> And just like each idle Apache process uses memory and other resources,
> each idle PostgreSQL connection does to. So managing the number o
Rod Taylor <[EMAIL PROTECTED]> writes:
> Cutting that count down to 10 idlers in total by having PostgreSQL
> prefork a specific database would make a significant difference.
Well it would be 10 for each database. Since as has been pointed out before
loading the database is most of the delay.
I
Pre-fork does not equal to idle connections! Pre-fork
scales with database load where as persistent
connections scales with webserver load. A web server
that is heavily loaded but not necessarily performing
a lot of database activity will spawn hundreds of idle
database connections using persistent
I'm talking about connecting to multiple database
servers on separate machines. Schemas don't apply
here.
How much work would it take to make a pre-fork smart
enough to open different databases on incoming
connection? How much of it can be modeled after
Apache?
> I know the issue of pre-fork PostgreSQL has been discussed previously.
> Someone mentionned pre-fork can be implemented when schemas become available
> Any chance of that happening for 7.5?
0 chance unless you have a patch ready now.
---(end of broadcast)--
Hi,
I know the issue of pre-fork PostgreSQL has been
discussed previously.
Someone mentionned pre-fork can be implemented when
schemas become available
in PostgreSQL because there will be less of the need
to run multiple
databases.
I think Oracle 7 uses pre-forking and it helps speed
up the start
Hi,
I know the issue of pre-fork PostgreSQL has been discussed previously.
Someone mentionned pre-fork can be implemented when schemas become available
in PostgreSQL because there will be less of the need to run multiple
databases.
I think Oracle 7 uses pre-forking and it helps speed up the start
On Mon, May 03, 2004 at 11:59:45PM -0700, sdv mailer wrote:
>
> Connection pooling (eg. SQLRelay) didn't work either
> because we needed to connect to hundreds of DB servers
> from each web server. Imagine having 200+ open
> connections on the web server and how many more of
> these connections re
> And "preforking" makes this different, how ? Perhaps having a pool of
> processes ready to be handed a query to a specific database, where you
> configure N connections to db1, M to db2 etc. still means lots of resource
> usage. In effect a preforked database server *is* an idle connection, just
sdv mailer wrote:
> We used to run persistent connection until the DB
> servers got maxed out because of too many idle
> connections sucking up all the memory. Web servers run
> different loads than database servers and persistent
> connections are notorious for crashing your DB.
And this translat
We used to run persistent connection until the DB
servers got maxed out because of too many idle
connections sucking up all the memory. Web servers run
different loads than database servers and persistent
connections are notorious for crashing your DB.
Connection pooling (eg. SQLRelay) didn't work
sdv mailer <[EMAIL PROTECTED]> writes:
> Forking consumes a large amount of CPU when you have
> many simultaneous connections and adds up to the
> latency. Particularly MySQL users may think
> PostgreSQL's connection time is much slower because
> these users tend to perform relatively simple queri
Forking consumes a large amount of CPU when you have
many simultaneous connections and adds up to the
latency. Particularly MySQL users may think
PostgreSQL's connection time is much slower because
these users tend to perform relatively simple queries.
In my case, connection pooling and persistent
Paul Ramsey <[EMAIL PROTECTED]> writes:
> ... So the operational benefit of adding the complexity of a
> pre-fork system is not very high.
In particular, most of the connection startup overhead work cannot be
performed until we've identified which database to connect to (since
it largely consists
sdv mailer wrote:
Instead, there's a big need to
create a new connection on
every query and with PostgreSQL needing to fork on
every incoming connection
can be quite slow.
Really? My general experience has beent that forking/connection setup
times are very good with PgSQL. Do not assume your Oracl
I had lots of trouble posting so you may receive this
more than once. My apologies..
--
Hi,
I know the issue of pre-fork PostgreSQL has been
discussed previously.
Someone mentionned pre-fork can be implemented when
schemas become available
in PostgreSQL because there w
61 matches
Mail list logo
