Re: [PERFORM] Hot Standby performance issue

2013-10-20 Thread Kevin Grittner
sparikh spar...@ecotality.com wrote:

 PostgreSQL 9.1.1

You really should apply the fixes for bugs and security
vulnerabilities which are available.  Some of those may address a
relevant performance problem.

http://www.postgresql.org/support/versioning/

 the performance was good in September and somewhere in October it
 started slowing down. I guess it was gradual. There were no code
 change in the application or major change in the data volume.

We really need to see your configuration, along with a description
of the machine.  One common cause for such a slowdown is not having
autovacuum configured to be aggressive enough.  Another possible
cause is a transaction which has been left open for too long.  Look
at pg_stat_activity and pg_prepared_xacts for xact_start or
prepared more than an hour old.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Hot Standby performance issue

2013-10-20 Thread sparikh
Thanks so much Tomas and Kevin for your valuable inputs. I am getting very
good response from this forum and learning so many new stuffs. I will try
all those options and will let you update .


standby_performance_issue.rar
http://postgresql.1045698.n5.nabble.com/file/n5775181/standby_performance_issue.rar
  


On further digging I found from the new relic report that as soon as I
execute query IO spikes immediately (100%). But the same query on primary
executes very fast.

I am not sure if postgres has some utility like what oracle's tkprof or AWR
where I can exactly pin point where exactly the query spends time. I will
try Tomas' suggestions perf and strace.

Below is the query. I also tried to attached rar file  one more time,
hopefully it gets through this time.

SELECT xfer_id, transaction_type, evse_time, transmit_time, error_code,
detail, reported_card_account_number as reported_card_account_number,
event_id,
 event_code,evse_id, batch_id, port, charge_event_id as charge_event_id FROM
(SELECT t.transaction_id::text AS xfer_id, t.transaction_type, e.event_time
AS evse_time,
 t.create_date AS transmit_time, t.error_code::text, '' AS detail,
COALESCE(e.reported_rfid,'N/A') AS reported_card_account_number,
e.event_id::text, e.event_code::text,
 t.evse_id::text, t.batch_id, e.port, COALESCE(e.evse_charge_id,'N/A') AS
charge_event_id 
FROM evse_transaction t, evse_event e , evse_unit u 
WHERE e.transaction_id = t.transaction_id AND t.evse_id = u.evse_id  
AND e.event_code IN
('1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20','21','22','23','24','25','26','27','28','29','30','31','32','33','34','35','36','37','38','39','40','41','42','43','44','45','46','47','48','49'))
  
AND u.evse_id = 1
  AND t.create_date BETWEEN '2013-10-01'::date AND '2013-10-15'::date +
INTERVAL '1 day'
 UNION
 SELECT t.transaction_id::text AS xfer_id, t.transaction_type, t.log_time AS
evse_time, t.create_date AS transmit_time, t.error_code::text, '' AS detail,
 COALESCE(t.reported_card_account_number,'N/A') AS
reported_card_account_number, '' AS event_id, '' AS event_code,
t.evse_id::text,t.batch_id, '' AS port, 'N/A' AS charge_event_id 
FROM evse_transaction t, evse_unit u 
WHERE t.evse_id = u.evse_id AND t.api_error IS NULL 
AND t.transaction_type NOT IN
('DCFCTransactionService','L2TransactionService','EVSEUploadTransactionService','EVSEUploadTransactionService','UploadTransactionService')
  
AND t.transaction_type IN
('DCFCDownloadConfigService','L2DownloadConfigService','EVSEDownloadConfigService','DownloadConfigService','ConfigDownloadService','DCFCUploadConfigService','L2UploadConfigService','EVSEUploadConfigService','UploadConfigService','ConfigUploadService','L2GetAdPackageListService','AdPackageListService','L2GPSService','EVSEGPSService','GPSService','ReportErrorService','EVSEDownloadRevisionService','DCFCCommandService','L2CommandService','CommandService','DCFCErrorService','L2ErrorService','EVSEReportErrorService','ErrorService','DCFCHeartbeatService','L2HeartbeatService','HeartbeatService','DCFCAuthorizeService','L2AuthorizeService','AuthorizeService','DCFCGetAccessListService','L2GetAccessListService','GetAccessListService','DCFCSetAccessService','L2SetAccessService','SetAccessService','DCFCPackageDownloadService','L2PackageDownloadService','PackageDownloadService','DCFCReportInventoryService','L2ReportInventoryService','ReportInventoryService','DCFCTargetVersionService','L2TargetVersionService','TargetVersionService','DCFCPackageListService','L2PackageInfoService','PackageListService','DCFCPackageInfoService','L2PackageInfoService','PackageInfoService','DCFCRegisterService','L2AuthorizeCodeService',
'AuthorizeCodeService') 
 AND u.evse_id = 1  AND t.create_date BETWEEN '2013-10-01'::date AND
'2013-10-15'::date + INTERVAL '1 day' 
UNION
 SELECT ef.fee_id::text AS xfer_id, 'FEE' as transaction_type, ef.event_time
AS evse_time, ef.create_time AS transmit_time, 
'' AS error_code, 'Fee Event' AS detail, COALESCE(ef.card_account_number,
'N/A') AS reported_card_account_number, '' AS event_id, '' AS event_code,
ef.evse_id::text, '' AS batch_id, 
ef.port::text AS port, COALESCE(ef.client_charge_id, 'N/A') AS
charge_event_id 
FROM evse_fee ef LEFT OUTER JOIN evse_unit eu ON eu.evse_id = ef.evse_id 
WHERE ef.evse_id = 1  AND  ef.create_time BETWEEN '2013-10-01'::date AND
'2013-10-15'::date + INTERVAL '1 day'
) x 
ORDER BY transmit_time DESC LIMIT 500

==

Query plan:

Limit  (cost=101950.33..101950.40 rows=30 width=368) (actual
time=18.421..18.421 rows=0 loops=1)
  Output: ((t.transaction_id)::text), t.transaction_type, e.event_time,
t.create_date, t.error_code, (''::text), (COALESCE(e.reported_rfid,
'N/A'::text)), ((e.event_id)::text), ((e.event_code)::text),
((t.evse_id)::text), t.batch_id, e.port, (COALESCE(e.evse_charge_id,
'N/A'::text))
  Buffers: shared hit=5 read=7
  -  Sort  (cost=101950.33..101950.40 rows=30 width=368) (actual

Re: [PERFORM] Hot Standby performance issue

2013-10-20 Thread Tomas Vondra
Hi,

On 20.10.2013 19:58, sparikh wrote:
 Thanks so much Tomas and Kevin for your valuable inputs. I am getting
 very good response from this forum and learning so many new stuffs. I
 will try all those options and will let you update .
 
 
 standby_performance_issue.rar 
 http://postgresql.1045698.n5.nabble.com/file/n5775181/standby_performance_issue.rar

Yup, this time it worked. Anyway, next time please consider posting the
explain plan through explain.depesz.com, it's way more readable than the
plans wrapped when posted inline.

For example this is your plan: http://explain.depesz.com/s/SBVg

However this shows only 18 ms runtime. Is this one of the slow runs? I'm
assuming it's not, because 18ms seems quite fast tome. In that case it's
pretty much useless, because we need to see a plan for one of the slow
queries.

Stupid question - when you say that a query is fast on primary but slow
on standby, are you referring to exactly the same query, including
parameter values?

Or is the query running much longer than the reported 18 ms?

 On further digging I found from the new relic report that as soon as
 I execute query IO spikes immediately (100%). But the same query on
 primary executes very fast.

Well, that might just as well mean that the primary has the data in
filesystem cache, and standby needs to read that from file. If you
repeat the query multiple times, do you still see I/O spike?

How do you use the standby? Is it just sitting there most of the time,
or is it queried about as much as the primary?

BTW when checking the configuration info you've sent, I've noticed this:

  hot_standby_feedback,off

IIRC you've reported the query frequently crashes on the standby because
of replication conflicts. Why don't you set this to on?

 I am not sure if postgres has some utility like what oracle's tkprof
 or AWR where I can exactly pin point where exactly the query spends
 time. I will try Tomas' suggestions perf and strace.

No, at least the community version. But the explain analyze is usually a
good source guide.

Tomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance