Re: [GENERAL] Losing data

2008-06-20 Thread Garry Saddington
On Friday 20 June 2008 05:26, Robert Treat wrote:
 On Thursday 19 June 2008 14:06:38 Garry Saddington wrote:
   In any case, however, if PostgreSQL reported the transaction complete
   and the machine didn't experience any hardware problems (like sudden
   power or disk failure), I would certainly not suspect PostgreSQL as the
   source of the problem.
 
  What has happened to the reports then? I have used this combination of
  Zope and Postgres for 5 years with no problems like this before and we
  have written one complete set of reports on this server in the past 6
  weeks. The problem seems to have started last friday, when reports
  started to go missing.

 Out of curiosity, what is your vacuum strategy?
We back up and vacuum at the same time each day.
garry

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Joshua D. Drake


On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
 I have had a serious loss of data and wondered if anyone could shed any light 
 on what may have happened.
 My users have been writing reports on students. No error messages have been 
 produced and when called back up the reports seem to be present at the time 
 of writing. However, next day they have disappeared, and they do not appear 
 in a pg_dump. They seem to have been kept in memory and never written to 
 disk.
 We are using Zope and connecting to Postgres through psycopg on Centos 5.
 I suspect a hard disk failure but any other ideas would be welcome.
 Would these reports be in the WAL?

If it was hardware related you would know, quickly. This sounds a great
deal more like an application level interaction. Perhaps your zope
application caches things for a while before committing to disk?

Are you familiar with the actual database itself? Have you checked the
database logs for errors? 

Sincerely,

Joshua D. rae




-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Garry Saddington
On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:
 On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
  I have had a serious loss of data and wondered if anyone could shed any
  light on what may have happened.
  My users have been writing reports on students. No error messages have
  been produced and when called back up the reports seem to be present at
  the time of writing. However, next day they have disappeared, and they do
  not appear in a pg_dump. They seem to have been kept in memory and never
  written to disk.
  We are using Zope and connecting to Postgres through psycopg on Centos 5.
  I suspect a hard disk failure but any other ideas would be welcome.
  Would these reports be in the WAL?

 If it was hardware related you would know, quickly. This sounds a great
 deal more like an application level interaction. Perhaps your zope
 application caches things for a while before committing to disk?
Yes I thought of this but once the report is sent to the DB a separate query 
is run to get all of that teacher's reports and these are then displayed on a 
new page. They all appear here but then disappear later. Zope has transaction 
machinery that rolls everything back on an error, so Postgres must have 
indicated a successful write somehow.  I read in a Postgres manual that the 
hard disk may report to the OS that a write has occured when it actually has 
not, is this possible? Oh, and the problem has been intermittant. Another 
thing that happened this morning is that Postgres had today as 18/06/2008 
when in fact it was 19/06/2008 and the OS reported this correctly. Restarting 
postgres sorted it, could this be the problem?
Regards
Garry

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Scott Marlowe
On Thu, Jun 19, 2008 at 9:55 AM, Garry Saddington
[EMAIL PROTECTED] wrote:
 I have had a serious loss of data and wondered if anyone could shed any light
 on what may have happened.
 My users have been writing reports on students. No error messages have been
 produced and when called back up the reports seem to be present at the time
 of writing. However, next day they have disappeared, and they do not appear
 in a pg_dump. They seem to have been kept in memory and never written to
 disk.
 We are using Zope and connecting to Postgres through psycopg on Centos 5.
 I suspect a hard disk failure but any other ideas would be welcome.
 Would these reports be in the WAL?
 regards

Just a guess, but do you have some kind of search engine pointing at
the application?  Could it be hitting a delete link maybe?

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Bill Moran
In response to Garry Saddington [EMAIL PROTECTED]:

 On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:
  On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
   I have had a serious loss of data and wondered if anyone could shed any
   light on what may have happened.
   My users have been writing reports on students. No error messages have
   been produced and when called back up the reports seem to be present at
   the time of writing. However, next day they have disappeared, and they do
   not appear in a pg_dump. They seem to have been kept in memory and never
   written to disk.
   We are using Zope and connecting to Postgres through psycopg on Centos 5.
   I suspect a hard disk failure but any other ideas would be welcome.
   Would these reports be in the WAL?
 
  If it was hardware related you would know, quickly. This sounds a great
  deal more like an application level interaction. Perhaps your zope
  application caches things for a while before committing to disk?
 Yes I thought of this but once the report is sent to the DB a separate query 
 is run to get all of that teacher's reports and these are then displayed on a 
 new page. They all appear here but then disappear later. Zope has transaction 
 machinery that rolls everything back on an error, so Postgres must have 
 indicated a successful write somehow.  I read in a Postgres manual that the 
 hard disk may report to the OS that a write has occured when it actually has 
 not, is this possible?

No.  If that happens you end up with corrupt disks.  The chance of that
going unnoticed by the OS is pretty slim.

 Oh, and the problem has been intermittant. Another 
 thing that happened this morning is that Postgres had today as 18/06/2008 
 when in fact it was 19/06/2008 and the OS reported this correctly. Restarting 
 postgres sorted it, could this be the problem?

Sounds to me like there's something seriously wrong with you OS or your
PostgreSQL install.  What version of PostgreSQL is this?  What OS?

-- 
Bill Moran
Collaborative Fusion Inc.
http://people.collaborativefusion.com/~wmoran/

[EMAIL PROTECTED]
Phone: 412-422-3463x4023

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Alan Hodgson
On Thursday 19 June 2008, Garry Saddington [EMAIL PROTECTED] 
wrote:
 I read in a
 Postgres manual that the hard disk may report to the OS that a write has
 occured when it actually has not, is this possible?

Yeah. But unless the power suddenly turned off that wouldn't cause data 
loss.

 Oh, and the problem 
 has been intermittant. Another thing that happened this morning is that
 Postgres had today as 18/06/2008 when in fact it was 19/06/2008 and the
 OS reported this correctly. Restarting postgres sorted it, could this be
 the problem?

I strongly suspect the problem is between the keyboard and the chair. 

In any case, however, if PostgreSQL reported the transaction complete and 
the machine didn't experience any hardware problems (like sudden power or 
disk failure), I would certainly not suspect PostgreSQL as the source of 
the problem.

-- 
Alan

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Garry Saddington
On Thursday 19 June 2008 18:10, Bill Moran wrote:
 In response to Garry Saddington [EMAIL PROTECTED]:
  On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:
   On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
I have had a serious loss of data and wondered if anyone could shed
any light on what may have happened.
My users have been writing reports on students. No error messages
have been produced and when called back up the reports seem to be
present at the time of writing. However, next day they have
disappeared, and they do not appear in a pg_dump. They seem to have
been kept in memory and never written to disk.
We are using Zope and connecting to Postgres through psycopg on
Centos 5. I suspect a hard disk failure but any other ideas would be
welcome. Would these reports be in the WAL?
  
   If it was hardware related you would know, quickly. This sounds a great
   deal more like an application level interaction. Perhaps your zope
   application caches things for a while before committing to disk?
 
  Yes I thought of this but once the report is sent to the DB a separate
  query is run to get all of that teacher's reports and these are then
  displayed on a new page. They all appear here but then disappear later.
  Zope has transaction machinery that rolls everything back on an error, so
  Postgres must have indicated a successful write somehow.  I read in a
  Postgres manual that the hard disk may report to the OS that a write has
  occured when it actually has not, is this possible?

 No.  If that happens you end up with corrupt disks.  The chance of that
 going unnoticed by the OS is pretty slim.

  Oh, and the problem has been intermittant. Another
  thing that happened this morning is that Postgres had today as 18/06/2008
  when in fact it was 19/06/2008 and the OS reported this correctly.
  Restarting postgres sorted it, could this be the problem?

 Sounds to me like there's something seriously wrong with you OS or your
 PostgreSQL install.  What version of PostgreSQL is this?  What OS?

 --
Centos 5 with the Posgres that comes with it - 8.1
regards
garry

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Adrian Klaver
-- Original message --
From: Garry Saddington [EMAIL PROTECTED]
 On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:
  On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
   I have had a serious loss of data and wondered if anyone could shed any
   light on what may have happened.
   My users have been writing reports on students. No error messages have
   been produced and when called back up the reports seem to be present at
   the time of writing. However, next day they have disappeared, and they do
   not appear in a pg_dump. They seem to have been kept in memory and never
   written to disk.
   We are using Zope and connecting to Postgres through psycopg on Centos 5.
   I suspect a hard disk failure but any other ideas would be welcome.
   Would these reports be in the WAL?
 
  If it was hardware related you would know, quickly. This sounds a great
  deal more like an application level interaction. Perhaps your zope
  application caches things for a while before committing to disk?
 Yes I thought of this but once the report is sent to the DB a separate query 
 is run to get all of that teacher's reports and these are then displayed on a 
 new page. They all appear here but then disappear later. Zope has transaction 
 machinery that rolls everything back on an error, so Postgres must have 
 indicated a successful write somehow.  I read in a Postgres manual that the 
 hard disk may report to the OS that a write has occured when it actually has 
 not, is this possible? Oh, and the problem has been intermittant. Another 
 thing that happened this morning is that Postgres had today as 18/06/2008 
 when in fact it was 19/06/2008 and the OS reported this correctly. Restarting 
 postgres sorted it, could this be the problem?
 Regards
 Garry
 

Seems like a transaction with no commit. Basically along as the session is 
active the data is there but once the session is closed the data does not 
persist.

--
Adrian Klaver
[EMAIL PROTECTED]

 

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Garry Saddington
On Thursday 19 June 2008 18:15, Alan Hodgson wrote:
 On Thursday 19 June 2008, Garry Saddington [EMAIL PROTECTED]

 wrote:
  I read in a
  Postgres manual that the hard disk may report to the OS that a write has
  occured when it actually has not, is this possible?

 Yeah. But unless the power suddenly turned off that wouldn't cause data
 loss.

  Oh, and the problem
  has been intermittant. Another thing that happened this morning is that
  Postgres had today as 18/06/2008 when in fact it was 19/06/2008 and the
  OS reported this correctly. Restarting postgres sorted it, could this be
  the problem?

 I strongly suspect the problem is between the keyboard and the chair.
I'd love to agree, but I have seen this first hand as a user!

 In any case, however, if PostgreSQL reported the transaction complete and
 the machine didn't experience any hardware problems (like sudden power or
 disk failure), I would certainly not suspect PostgreSQL as the source of
 the problem.
What has happened to the reports then? I have used this combination of Zope 
and Postgres for 5 years with no problems like this before and we have 
written one complete set of reports on this server in the past 6 weeks. The 
problem seems to have started last friday, when reports started to go 
missing.
regards
Garry

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Garry Saddington
On Thursday 19 June 2008 18:09, Scott Marlowe wrote:
 On Thu, Jun 19, 2008 at 9:55 AM, Garry Saddington

 [EMAIL PROTECTED] wrote:
  I have had a serious loss of data and wondered if anyone could shed any
  light on what may have happened.
  My users have been writing reports on students. No error messages have
  been produced and when called back up the reports seem to be present at
  the time of writing. However, next day they have disappeared, and they do
  not appear in a pg_dump. They seem to have been kept in memory and never
  written to disk.
  We are using Zope and connecting to Postgres through psycopg on Centos 5.
  I suspect a hard disk failure but any other ideas would be welcome.
  Would these reports be in the WAL?
  regards

 Just a guess, but do you have some kind of search engine pointing at
 the application?  Could it be hitting a delete link maybe?
Don't think so, once a teacher has entered a report there is no way that they 
can delete it.
Regards
Garry

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Geoffrey

Garry Saddington wrote:

On Thursday 19 June 2008 18:15, Alan Hodgson wrote:

On Thursday 19 June 2008, Garry Saddington [EMAIL PROTECTED]

wrote:

I read in a
Postgres manual that the hard disk may report to the OS that a write has
occured when it actually has not, is this possible?

Yeah. But unless the power suddenly turned off that wouldn't cause data
loss.


Oh, and the problem
has been intermittant. Another thing that happened this morning is that
Postgres had today as 18/06/2008 when in fact it was 19/06/2008 and the
OS reported this correctly. Restarting postgres sorted it, could this be
the problem?

I strongly suspect the problem is between the keyboard and the chair.

I'd love to agree, but I have seen this first hand as a user!

In any case, however, if PostgreSQL reported the transaction complete and
the machine didn't experience any hardware problems (like sudden power or
disk failure), I would certainly not suspect PostgreSQL as the source of
the problem.
What has happened to the reports then? I have used this combination of Zope 
and Postgres for 5 years with no problems like this before and we have 
written one complete set of reports on this server in the past 6 weeks. The 
problem seems to have started last friday, when reports started to go 
missing.


What has changed prior to Friday?

--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
 - Benjamin Franklin

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Joshua D. Drake


On Thu, 2008-06-19 at 19:06 +0100, Garry Saddington wrote:
 On Thursday 19 June 2008 18:15, Alan Hodgson wrote:
  On Thursday 19 June 2008, Garry Saddington [EMAIL PROTECTED]

Although I appreciate that this is a funky problem, the problem doesn't
yet exist and we are operating in a diagnostic vacuum.

Can you duplicate the problem now?

Do you have any logs from PostgreSQL that describe the session that
either previously or currently (based on your tests) the problem?

Do those logs actually show ERROR?

What about your apachelog or zope logs? 

Have you turned up the logging in all three components to ensure that
the problem actually does exist?

Sincerely,

Joshua D. Drake


 regards
 Garry
 


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Garry Saddington
On Thursday 19 June 2008 18:52, Adrian Klaver wrote:
 -- Original message --
 From: Garry Saddington [EMAIL PROTECTED]

  On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:
   On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
I have had a serious loss of data and wondered if anyone could shed
any light on what may have happened.
My users have been writing reports on students. No error messages
have been produced and when called back up the reports seem to be
present at the time of writing. However, next day they have
disappeared, and they do not appear in a pg_dump. They seem to have
been kept in memory and never written to disk.
We are using Zope and connecting to Postgres through psycopg on
Centos 5. I suspect a hard disk failure but any other ideas would be
welcome. Would these reports be in the WAL?
  
   If it was hardware related you would know, quickly. This sounds a great
   deal more like an application level interaction. Perhaps your zope
   application caches things for a while before committing to disk?
 
  Yes I thought of this but once the report is sent to the DB a separate
  query is run to get all of that teacher's reports and these are then
  displayed on a new page. They all appear here but then disappear later.
  Zope has transaction machinery that rolls everything back on an error, so
  Postgres must have indicated a successful write somehow.  I read in a
  Postgres manual that the hard disk may report to the OS that a write has
  occured when it actually has not, is this possible? Oh, and the problem
  has been intermittant. Another thing that happened this morning is that
  Postgres had today as 18/06/2008 when in fact it was 19/06/2008 and the
  OS reported this correctly. Restarting postgres sorted it, could this be
  the problem?
  Regards
  Garry

 Seems like a transaction with no commit. Basically along as the session is
 active the data is there but once the session is closed the data does not
 persist.

Makes sense but what is to blame?
Regards
Garry

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Joshua D. Drake


On Thu, 2008-06-19 at 19:12 +0100, Garry Saddington wrote:
 On Thursday 19 June 2008 18:52, Adrian Klaver wrote:
  -- Original message --
  From: Garry Saddington [EMAIL PROTECTED]

  Seems like a transaction with no commit. Basically along as the session is
  active the data is there but once the session is closed the data does not
  persist.
 
 Makes sense but what is to blame?

If this is indeed the problem, it is likely a lack of an explicit commit
to your connect object within zope. In pysocpg2 unless you set your
isolation level to something different it automatically creates a new
transaction for your session so if you don't do conn.commit() everything
you do will be rolled back.

Joshua D. Drake


 Regards
 Garry
 


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Adrian Klaver
 -- Original message --
From: Garry Saddington [EMAIL PROTECTED]
 On Thursday 19 June 2008 18:52, Adrian Klaver wrote:
  -- Original message --
  From: Garry Saddington [EMAIL PROTECTED]
 
   On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:
On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:
 I have had a serious loss of data and wondered if anyone could shed
 any light on what may have happened.
 My users have been writing reports on students. No error messages
 have been produced and when called back up the reports seem to be
 present at the time of writing. However, next day they have
 disappeared, and they do not appear in a pg_dump. They seem to have
 been kept in memory and never written to disk.
 We are using Zope and connecting to Postgres through psycopg on
 Centos 5. I suspect a hard disk failure but any other ideas would be
 welcome. Would these reports be in the WAL?
   
If it was hardware related you would know, quickly. This sounds a great
deal more like an application level interaction. Perhaps your zope
application caches things for a while before committing to disk?
  
   Yes I thought of this but once the report is sent to the DB a separate
   query is run to get all of that teacher's reports and these are then
   displayed on a new page. They all appear here but then disappear later.
   Zope has transaction machinery that rolls everything back on an error, so
   Postgres must have indicated a successful write somehow.  I read in a
   Postgres manual that the hard disk may report to the OS that a write has
   occured when it actually has not, is this possible? Oh, and the problem
   has been intermittant. Another thing that happened this morning is that
   Postgres had today as 18/06/2008 when in fact it was 19/06/2008 and the
   OS reported this correctly. Restarting postgres sorted it, could this be
   the problem?
   Regards
   Garry
 
  Seems like a transaction with no commit. Basically along as the session is
  active the data is there but once the session is closed the data does not
  persist.
 
 Makes sense but what is to blame?

Therein lies the rub. As Josh mentioned in another post you will need turn up 
the logging for the various components and rerun the suspect procedure. I find 
the Postgres logs to most informative as they show what is actually hitting the 
database. Once I see the actual SQL statement I work backwards to see how it 
got there and in that form.

 Regards
 Garry



--
Adrian Klaver
[EMAIL PROTECTED]


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Tom Lane
Garry Saddington [EMAIL PROTECTED] writes:
 Oh, and the problem has been intermittant. Another
 thing that happened this morning is that Postgres had today as 18/06/2008
 when in fact it was 19/06/2008 and the OS reported this correctly.

Two theories about that one:

1. Postgres' timezone setting is far away from where you are.

2. You were in a transaction that had been open since yesterday.
now() and related functions show time of transaction start.


#2 is interesting since forgetting to commit an open transaction might
also explain the data loss complaint.

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Shane Ambler

Adrian Klaver wrote:


Yes I thought of this but once the report is sent to the DB a
separate query is run to get all of that teacher's reports and
these are then displayed on a new page. They all appear here but
then disappear later. Zope has transaction machinery that rolls
everything back on an error, so Postgres must have indicated a
successful write somehow.  I read in a Postgres manual that the 
hard disk may report to the OS that a write has occured when it

actually has not, is this possible? Oh, and the problem has been
intermittant. Another thing that happened this morning is that
Postgres had today as 18/06/2008 when in fact it was 19/06/2008 and
the OS reported this correctly. Restarting postgres sorted it,
could this be the problem? Regards Garry



Seems like a transaction with no commit. Basically along as the
session is active the data is there but once the session is closed
the data does not persist.


This is my thinking.

Once they are entered and then get listed in the new page can another
user see the reports?

Do they log out and then back in and the reports are gone?

If so then maybe the commit step is getting skipped for some reason.

At what stage are things committed? After entry or at user logoff?
Is there some user data updated on logout that gets an error and rolls back?



--

Shane Ambler
pgSQL (at) Sheeky (dot) Biz

Get Sheeky @ http://Sheeky.Biz

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Garry Saddington
On Thursday 19 June 2008 19:03, Joshua D. Drake wrote:
 On Thu, 2008-06-19 at 19:06 +0100, Garry Saddington wrote:
  On Thursday 19 June 2008 18:15, Alan Hodgson wrote:
   On Thursday 19 June 2008, Garry Saddington [EMAIL PROTECTED]

 Although I appreciate that this is a funky problem, the problem doesn't
 yet exist and we are operating in a diagnostic vacuum.

 Can you duplicate the problem now?
The problem is not consistent, some reports are lsot others not.

 Do you have any logs from PostgreSQL that describe the session that
 either previously or currently (based on your tests) the problem?
These are at school so I will look tomorrow


 Do those logs actually show ERROR?
There are no errors relating to reports

 What about your apachelog or zope logs?
Again tomorrow

 Have you turned up the logging in all three components to ensure that
 the problem actually does exist?
Will do this tomorrow
regards and thanks for your efforts.

PS
Could it be somethiing to do with libpq.so.4/5 required by psycopg and will 
the WAL hold data to be replayed?

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Joshua D. Drake


On Thu, 2008-06-19 at 20:29 +0100, Garry Saddington wrote:
 On Thursday 19 June 2008 19:03, Joshua D. Drake wrote:
  On Thu, 2008-06-19 at 19:06 +0100, Garry Saddington wrote:
   On Thursday 19 June 2008 18:15, Alan Hodgson wrote:
On Thursday 19 June 2008, Garry Saddington
 [EMAIL PROTECTED]
 
 PS
 Could it be somethiing to do with libpq.so.4/5 required by psycopg and will 

I think it certainly has something to do with psycopg and the way it
handles commits. Especially based on the description of what you say
above where some work and some don't. My guess (as I mentioned
previously) is there is a bit of code in your report generator that
doesn't call commit correctly.

 the WAL hold data to be replayed?

Only if you are pushing the logs off for standby operations and only if
you commit.

Sincerely,

Joshua D. Drake



 


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Tino Wildenhain

Hi,

Garry Saddington wrote:

On Thursday 19 June 2008 16:55, Joshua D. Drake wrote:

On Thu, 2008-06-19 at 16:55 +0100, Garry Saddington wrote:

...

Yes I thought of this but once the report is sent to the DB a separate query 
is run to get all of that teacher's reports and these are then displayed on a 
new page. They all appear here but then disappear later. Zope has transaction 
machinery that rolls everything back on an error, so Postgres must have 


are you running the report within the same request that made the write?
In this case you would be in the same transaction (and see uncommitted
data) if then an exeption is raised after you display the report
the transaction would be rolled back. You should check your zope logs.

Regards
Tino


smime.p7s
Description: S/MIME Cryptographic Signature


Re: [GENERAL] Losing data

2008-06-19 Thread Robert Treat
On Thursday 19 June 2008 14:06:38 Garry Saddington wrote:
  In any case, however, if PostgreSQL reported the transaction complete and
  the machine didn't experience any hardware problems (like sudden power or
  disk failure), I would certainly not suspect PostgreSQL as the source of
  the problem.

 What has happened to the reports then? I have used this combination of Zope
 and Postgres for 5 years with no problems like this before and we have
 written one complete set of reports on this server in the past 6 weeks. The
 problem seems to have started last friday, when reports started to go
 missing.

Out of curiosity, what is your vacuum strategy? 

-- 
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Losing data

2008-06-19 Thread Tom Lane
Robert Treat [EMAIL PROTECTED] writes:
 On Thursday 19 June 2008 14:06:38 Garry Saddington wrote:
 The problem seems to have started last friday, when reports started to go
 missing.

 Out of curiosity, what is your vacuum strategy? 

If you're thinking transaction ID wraparound, I believe we can rule
that out, because Garry says he's running PG 8.1.  8.1 is not terribly
proactive about preventing wraparound (no forced autovacuums), but it
will squawk loudly about impending wraparound and shut down before it
hits the wall.

My bet is that something was changed in the client-side software last
week that has broken its commit-issuing behavior.

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general