Re: [Zope] ZEO Problem Clients Waiting
On Mon, Aug 07, 2006 at 09:09:56PM +0200, Dieter Maurer wrote:
Gerhard Schmidt wrote at 2006-8-7 15:54 +0200:
...
2006-08-07T14:29:19 INFO ZEO.StorageServer (97002/10.152.64.23:52518)
Transaction
+blocked waiting for storage. Clients waiting: 1.
...
2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.17:54463)
Blocked transaction restarted. Clients waiting: 1
2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.23:52518)
Blocked transaction restarted.
This one was a very quick one only 30 seconds. I have Blocked Transaktion
that ware waiting for more than 2 minutes.
This means that you have very long transactions -- transactions that
take very long to commit.
ZEO cannot commit two transactions for the same storage at the same time.
Therefore, it sets a storage look when a transaction commit begins for
the storage.
If another transaction tries to commit to the same storage, the transaction
is blocked until the first transaction commit completes.
That's your Transaction blocked waiting for storage
When the commit is completed, then a waiting transaction is restarted.
That's your Blocked transaction restarted.
You should try to understand where the huge transactions come from.
Very often, they are caused by poor persistency design (either far
too huge objects or an immense number of tine objects or just some stupidity
(e.g. writing objects unnecessary).
I have benchmarked my Harddisk (which is at the moment an emergency system
because of a hardware failure of the main system) it has 40 MB/sec write
speed and it doesn't show high io load when we have such a hangup. I have
tried to create an object with 50 MB in the storage the ZEO server had no
problem with that. calculating this, there has to be an objekt of CD Image size
to cause the write to take more then 30 sec. But this whould
mean that the Data.fs whould grow at least a 2-3 Gig a Day (we have 5-6
such hangups a day) but it only grows arround 50-100 MB a per Day
(difference befor and after Pack) and real growth is 2-10 MB per Day.
To the number of tiny objects. I have the zeo.log on debug level. Entries
like these seam to be the objekts that are requested to be written.
2006-08-08T07:44:12 DEBUG ZEO.zrpc.Connection(S) (10.152.64.21:50210) calling
storea('\x00\x00\x00\x00\x0057\x85', '\x03gZW\x00\xa0\xa1\xcc', '(...
I don't see a lot of them bevor a hangup occur.
So there are two areas where the Problem could be located. The filesystem
of the host system and in the network between the App-Server and the Zeo.
To understand this can you tell me when a transaction is started and when
its closed. Does the Zeo server wait until all data is recieved bevor the
transaction is started or does the transaktion start when the datatransfer
from the appserver starts.
Bye
Estartu
Gerhard Schmidt| Nick : estartu IRC : Estartu |
Fischbachweg 3 || PGP Public Key
86856 Hiltenfingen | EMail: [EMAIL PROTECTED] | on request
Germany||
pgpRboq3a8VZt.pgp
Description: PGP signature
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO Problem Clients Waiting
Gerhard Schmidt wrote at 2006-8-8 08:06 +0200: ... long time commit locks ... To understand this can you tell me when a transaction is started and when its closed. Does the Zeo server wait until all data is recieved bevor the transaction is started or does the transaktion start when the datatransfer from the appserver starts. The storage lock is set when one of these function is called: vote, abortVersion, commitVersion and undo. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] ZEO Problem Clients Waiting
HI, we have quite a Problem with out Zope/Zeo system. We run at the moment Zope cluster with 10 application servers and a zeo server with Filestorage. Our Data.fs is 3.5 Gig at the moment (after Pack). I runs most of the time without problem. But sometimes things screw up 2006-08-07T14:29:19 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Transaction +blocked waiting for storage. Clients waiting: 1. 2006-08-07T14:29:29 INFO ZEO.StorageServer (97002/10.152.64.18:60205) Transaction +blocked waiting for storage. Clients waiting: 2. 2006-08-07T14:29:39 INFO ZEO.StorageServer (97002/10.152.64.19:55021) Transaction +blocked waiting for storage. Clients waiting: 3. 2006-08-07T14:29:39 INFO ZEO.StorageServer (97002/10.152.64.15:62775) Transaction +blocked waiting for storage. Clients waiting: 4. 2006-08-07T14:29:41 INFO ZEO.StorageServer (97002/10.152.64.22:59407) Transaction +blocked waiting for storage. Clients waiting: 5. 2006-08-07T14:29:41 INFO ZEO.StorageServer (97002/10.152.64.20:51153) Transaction blocked waiting for storage. Clients waiting: 6. 2006-08-07T14:29:41 INFO ZEO.StorageServer (97002/10.152.64.21:52951) Transaction blocked waiting for storage. Clients waiting: 7. 2006-08-07T14:29:41 INFO ZEO.StorageServer (97002/10.152.64.14:53919) Transaction blocked waiting for storage. Clients waiting: 8. 2006-08-07T14:29:41 INFO ZEO.StorageServer (97002/10.152.64.17:54463) Transaction blocked waiting for storage. Clients waiting: 9. 2006-08-07T14:29:47 INFO ZEO.StorageServer (97002/10.152.64.16:53743) Blocked transaction restarted. Clients waiting: 8 2006-08-07T14:29:47 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Blocked transaction restarted. Clients waiting: 7 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Transaction blocked waiting for storage. Clients waiting: 8. 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.18:60205) Blocked transaction restarted. Clients waiting: 7 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.19:55021) Blocked transaction restarted. Clients waiting: 6 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.15:62775) Blocked transaction restarted. Clients waiting: 5 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.22:59407) Blocked transaction restarted. Clients waiting: 4 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.20:51153) Blocked transaction restarted. Clients waiting: 3 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.21:52951) Blocked transaction restarted. Clients waiting: 2 2006-08-07T14:29:48 INFO ZEO.StorageServer (97002/10.152.64.16:53743) Transaction blocked waiting for storage. Clients waiting: 3. 2006-08-07T14:29:49 INFO ZEO.StorageServer (97002/10.152.64.14:53919) Blocked transaction restarted. Clients waiting: 2 2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.17:54463) Blocked transaction restarted. Clients waiting: 1 2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Blocked transaction restarted. This one was a very quick one only 30 seconds. I have Blocked Transaktion that ware waiting for more than 2 minutes. When there a 9 Clients waiting our complete Zopesite comes to a halt until the transaction is restarted. As im not realy deep in the ZODB/ZEO subject. Can anybody pinpoint me where to look for the Problem. Bye Estartu Gerhard Schmidt| Nick : estartu IRC : Estartu | Fischbachweg 3 || PGP Public Key 86856 Hiltenfingen | EMail: [EMAIL PROTECTED] | on request Germany|| pgpngdBFhioow.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] ZEO Problem Clients Waiting
Gerhard Schmidt wrote at 2006-8-7 15:54 +0200: ... 2006-08-07T14:29:19 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Transaction +blocked waiting for storage. Clients waiting: 1. ... 2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.17:54463) Blocked transaction restarted. Clients waiting: 1 2006-08-07T14:29:50 INFO ZEO.StorageServer (97002/10.152.64.23:52518) Blocked transaction restarted. This one was a very quick one only 30 seconds. I have Blocked Transaktion that ware waiting for more than 2 minutes. This means that you have very long transactions -- transactions that take very long to commit. ZEO cannot commit two transactions for the same storage at the same time. Therefore, it sets a storage look when a transaction commit begins for the storage. If another transaction tries to commit to the same storage, the transaction is blocked until the first transaction commit completes. That's your Transaction blocked waiting for storage When the commit is completed, then a waiting transaction is restarted. That's your Blocked transaction restarted. You should try to understand where the huge transactions come from. Very often, they are caused by poor persistency design (either far too huge objects or an immense number of tine objects or just some stupidity (e.g. writing objects unnecessary). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
