Re: [HACKERS] Implementing pg_receivewal --no-sync

2017-10-24 Thread Kuntal Ghosh
waitings'? + [ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ], + 'failure if --synchronous specified without --no-sync'); s/without/with -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com --

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Kuntal Ghosh
uldFree=true for a tuple on a disk page is not sane */ Assert(BufferIsValid(buffer) ? (!shouldFree) : true); For some storage engine, isn't it possible that the buffer is valid and the tuple to be stored is formed in memory (for example, tuple formed from UNDO, in-memory decrypted tuple e

Re: [HACKERS] "inconsistent page found" with checksum and wal_consistency_checking enabled

2017-09-19 Thread Kuntal Ghosh
On Wed, Sep 20, 2017 at 11:05 AM, Michael Paquier wrote: > On Wed, Sep 20, 2017 at 2:26 PM, Kuntal Ghosh > wrote: >> On Wed, Sep 20, 2017 at 10:22 AM, Michael Paquier >> wrote: >>> On Wed, Sep 20, 2017 at 5:23 AM, Ashwin Agrawal wrote: >>>> Current

Re: [HACKERS] "inconsistent page found" with checksum and wal_consistency_checking enabled

2017-09-19 Thread Kuntal Ghosh
If page lsns are different, checksums will be different as well. Anyhow, nice catch and thanks for the patch. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More efficient truncation of pg_stat_activity query strings

2017-09-15 Thread Kuntal Ghosh
5% postgres postgres [.] pgstat_report_activity This shows the significance of this patch in terms of performance improvement of pgstat_report_activity. Is there any other tests I should do for the same? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP patch: distinguish selectivity of < from <= and > from >=

2017-09-13 Thread Kuntal Ghosh
ates for contrib. > Do you want to look at those? > I've performed the regression tests for the same. It passed all the test cases. Also, I've verified the feature implementation using the queries posted by you earlier and some of my own test cases. It is working as expected. --

Re: [HACKERS] pg_stat_wal_write statistics view

2017-09-11 Thread Kuntal Ghosh
On Tue, Sep 12, 2017 at 9:06 AM, Haribabu Kommi wrote: > > > On Tue, Sep 12, 2017 at 2:04 AM, Kuntal Ghosh > wrote: >> >> On Wed, Sep 6, 2017 at 9:16 AM, Haribabu Kommi >> wrote: >> > >> > Attached the latest patch and performance report. >>

Re: [HACKERS] pg_stat_wal_write statistics view

2017-09-11 Thread Kuntal Ghosh
s in XLogWrite() irrespective of the type of backend and update the shared counters at once at the end of the function. Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To ma

Re: [HACKERS] parallelize queries containing initplans

2017-08-09 Thread Kuntal Ghosh
d send all the tuple to upper node, a worst case for parallelism. Probably, this is the reason the optimizer doesn't pick parallel plan for the above case. Just for clarification, do you see any changes in the plan after forcing parallelism(parallel_tuple_cost, parallel_setup_cost, min_paralle

Re: [HACKERS] WIP patch: distinguish selectivity of < from <= and > from >=

2017-07-09 Thread Kuntal Ghosh
On Fri, Jul 7, 2017 at 2:53 AM, Tom Lane wrote: > Kuntal Ghosh writes: Wow. Thank you for the wonderful explanation. >> On Thu, Jul 6, 2017 at 3:45 AM, Tom Lane wrote: > >> if histogram_bounds are assigned as, >> {0,9,19,29,39,49,59,69,79,89,99,109,119,129,13,

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-07 Thread Kuntal Ghosh
On Fri, Jul 7, 2017 at 7:49 AM, Michael Paquier wrote: > On Thu, Jul 6, 2017 at 8:51 PM, Dilip Kumar wrote: >> On Thu, Jul 6, 2017 at 4:18 PM, Kuntal Ghosh >> wrote: >>> But, I'm little concerned/doubt regarding the following part of the code. >>> +/*

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-06 Thread Kuntal Ghosh
On Wed, Jul 5, 2017 at 9:35 AM, Michael Paquier wrote: > On Tue, Jul 4, 2017 at 4:41 PM, Kuntal Ghosh > wrote: >> I've not yet started the patch and it may take some time for me to >> understand and write >> the patch in a correct way. Since, you've almost wri

Re: [HACKERS] WIP patch: distinguish selectivity of < from <= and > from >=

2017-07-06 Thread Kuntal Ghosh
r the first bucket. Do you think my reasoning justifies your concern? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP patch: distinguish selectivity of < from <= and > from >=

2017-07-04 Thread Kuntal Ghosh
On Tue, Jul 4, 2017 at 10:56 PM, Kuntal Ghosh wrote: > On Tue, Jul 4, 2017 at 9:20 PM, Tom Lane wrote: >> Kuntal Ghosh writes: >>> On Tue, Jul 4, 2017 at 9:23 AM, Tom Lane wrote: >>>> ... I have to admit that I've failed to wrap my brain around exactly &g

Re: [HACKERS] WIP patch: distinguish selectivity of < from <= and > from >=

2017-07-04 Thread Kuntal Ghosh
On Tue, Jul 4, 2017 at 9:20 PM, Tom Lane wrote: > Kuntal Ghosh writes: >> On Tue, Jul 4, 2017 at 9:23 AM, Tom Lane wrote: >>> ... I have to admit that I've failed to wrap my brain around exactly >>> why it's correct. The arguments that I've constructed

Re: [HACKERS] WIP patch: distinguish selectivity of < from <= and > from >=

2017-07-04 Thread Kuntal Ghosh
he correction is calculated as = 1 / num_distinct_values = .001. Since, the thousand column of tenk1 is uniformly distributed, this turns out to be the exact selectivity. (rows = .001 * 1000 = 10) Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com --

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-04 Thread Kuntal Ghosh
On Tue, Jul 4, 2017 at 12:44 PM, Michael Paquier wrote: > On Tue, Jul 4, 2017 at 3:25 PM, Kuntal Ghosh > wrote: >> On Mon, Jul 3, 2017 at 6:50 PM, Michael Paquier >> wrote: >>> pg_basebackup/ with fe_recvint64() has its own way to do things, as >>> does

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
#x27;ll do the changes and submit the revised patch. I've added an entry in commitfest for the same. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
L segments are not transferred. Only the WAL data of the the target > data folder is gone through to find all the blocks that have been > touched from the last checkpoint before WAL forked. > > Now, I think that this is broken for relation files higher than 2GB, > see

[HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
COPY fetchchunks, line 2659, column begin: "214800" I guess we've to change the data type to bigint. Also, we need some implementation of ntohl() for 8-byte data types. I've attached a script to reproduce the error and a draft patch. -- Thanks & Regards, Kuntal G

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-23 Thread Kuntal Ghosh
On Fri, Jun 23, 2017 at 3:01 AM, Thomas Munro wrote: > On Thu, Jun 22, 2017 at 4:29 AM, Kuntal Ghosh > wrote: >> On Wed, Jun 21, 2017 at 7:52 PM, Dilip Kumar wrote: >>> On Wed, Jun 21, 2017 at 7:47 PM, Kuntal Ghosh >>> wrote: >>>>> IMHO, It's

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
On Thu, Jun 22, 2017 at 9:48 AM, Michael Paquier wrote: > On Thu, Jun 22, 2017 at 1:29 AM, Kuntal Ghosh > wrote: >> On Wed, Jun 21, 2017 at 7:52 PM, Dilip Kumar wrote: >>> On Wed, Jun 21, 2017 at 7:47 PM, Kuntal Ghosh >>> wrote: >>>>> IMHO, It&#x

Re: [HACKERS] An attempt to reduce WALWriteLock contention

2017-06-21 Thread Kuntal Ghosh
to re-submit the patch. Actually, I'm not sure what I should try next. I would love to get some advice/direction regarding this. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] An attempt to reduce WALWriteLock contention

2017-06-21 Thread Kuntal Ghosh
67193%402ndquadrant.com > Hello Tomas, I'm really sorry for this late reply. I've somehow missed the thread. Actually, I've seen some performance improvement with the CLogControlLock patch. But, then it turned out all the improvements were because of the CLogControlLock patch al

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
On Wed, Jun 21, 2017 at 7:52 PM, Dilip Kumar wrote: > On Wed, Jun 21, 2017 at 7:47 PM, Kuntal Ghosh > wrote: >>> IMHO, It's not a good idea to use DSM call to verify the DSA handle. >>> >> Okay. Is there any particular scenario you've in mind where this m

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
On Wed, Jun 21, 2017 at 7:07 PM, Dilip Kumar wrote: > On Wed, Jun 21, 2017 at 6:50 PM, Kuntal Ghosh > wrote: >> I think we can just check dsm_find_mapping() to check whether the dsm >> handle is already attached. Something like, >> >> } >

Re: [HACKERS] Incorrect documentation about pg_stat_activity

2017-06-21 Thread Kuntal Ghosh
On Wed, Jun 21, 2017 at 6:05 PM, Yugo Nagata wrote: > > Attached is a patch for the documentation fix. > Please attach the patch as well. :-) -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
, } - else + else if(!dsm_find_mapping(AutoVacuumShmem->av_dsa_handle)) { AutoVacuumDSA = dsa_attach(AutoVacuumShmem->av_dsa_handle); dsa_pin_mapping(AutoVacuumDSA); Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://w

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-06-03 Thread Kuntal Ghosh
ate memory). Instead, we use only BackgroundWorkerSlots. Perhaps, this is the reason that backend_type is NULL for parallel workers. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-06-03 Thread Kuntal Ghosh
> concatenation just doesn't sit well with me, especially if it requires > the bgw_name_extra to start with space. > +1. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] "create publication..all tables" ignore 'partition not supported' error

2017-06-01 Thread Kuntal Ghosh
On Thu, Jun 1, 2017 at 6:56 AM, Peter Eisentraut wrote: > On 5/31/17 02:17, Kuntal Ghosh wrote: >> On Wed, May 31, 2017 at 12:58 AM, Masahiko Sawada >> wrote: >>> >>> I'd say we can fix this issue by just changing the query. Attached >>> patch chan

Re: [HACKERS] "create publication..all tables" ignore 'partition not supported' error

2017-05-30 Thread Kuntal Ghosh
urth condition as well. Added to open items. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Create subscription with `create_slot=false` and incorrect slot name

2017-05-22 Thread Kuntal Ghosh
1. Since, slot_name can be provided even when create_slot is set false, it should be validated as well while creating the subscription. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_dump ignoring information_schema tables which used in Create Publication.

2017-05-22 Thread Kuntal Ghosh
On Mon, May 22, 2017 at 5:22 PM, tushar wrote: > On 05/22/2017 05:12 PM, Kuntal Ghosh wrote: >> >> pg_dump ignores anything created under object name "pg_*" or >> "information_schema". > > In this below scenario , I am able to see - pg_dump catch th

Re: [HACKERS] Increasing parallel workers at runtime

2017-05-22 Thread Kuntal Ghosh
e Achilles Heel of query optimization, according to Guy Lohman et al. :)) can always prove the optimizer wrong. In that case, +1 for your suggested approach of dynamically add or kill some workers based on the estimated work left to do. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_dump ignoring information_schema tables which used in Create Publication.

2017-05-22 Thread Kuntal Ghosh
ise PostgreSQL Company > > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] "create publication..all tables" ignore 'partition not supported' error

2017-05-22 Thread Kuntal Ghosh
tion key: RANGE (logdate) > Publications: > "p" > > Publication 'p' has been set against partition table ,which is not > supported. > > -- > regards,tushar > EnterpriseDB https://www.enterprisedb.com/ > The Enterprise PostgreSQL Company > >

Re: [HACKERS] Server Crashes if try to provide slot_name='none' at the time of creating subscription.

2017-05-16 Thread Kuntal Ghosh
Added to open item lists. On Tue, May 16, 2017 at 6:35 AM, Masahiko Sawada wrote: > On Mon, May 15, 2017 at 8:09 PM, Masahiko Sawada > wrote: >> On Mon, May 15, 2017 at 8:06 PM, Kuntal Ghosh >> wrote: >>> On Mon, May 15, 2017 at 4:00 PM, Masahiko Sawada >>&

Re: [HACKERS] Server Crashes if try to provide slot_name='none' at the time of creating subscription.

2017-05-15 Thread Kuntal Ghosh
On Mon, May 15, 2017 at 5:04 PM, Masahiko Sawada wrote: > On Mon, May 15, 2017 at 8:22 PM, Kuntal Ghosh > wrote: >> On Mon, May 15, 2017 at 4:39 PM, Masahiko Sawada >> wrote: >> >> While testing with logical replication, I've found that the server >>

Re: [HACKERS] Server Crashes if try to provide slot_name='none' at the time of creating subscription.

2017-05-15 Thread Kuntal Ghosh
logical decoding found initial starting point at 0/162DF18 DETAIL: Waiting for transactions (approximately 1) older than 560 to end. And, it hangs. Is this an expected behavior? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Server Crashes if try to provide slot_name='none' at the time of creating subscription.

2017-05-15 Thread Kuntal Ghosh
t connect to the publisher: could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? Changing the order of subscription parameter changes the output. I think the modifications should b

Re: [HACKERS] Create publication syntax is not coming properly in pg_dump / pg_dumpall

2017-05-15 Thread Kuntal Ghosh
for reporting. > > Hm, It's a bug of pg_dump. Attached patch should fix both pg_dump and > pg_dumpall. > I've reproduced the bug and the patch fixes the issue for me. Also, tested with different combinations of insert, delete and update. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-11 Thread Kuntal Ghosh
two ways? > For backend_type=background worker, application_name shows the name of the background worker (BackgroundWorker->bgw_name). I think we need some way to distinguish among different background workers. But, application_name may not be the correct field to show the information. -- Th

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-11 Thread Kuntal Ghosh
r the code > you're talking about can be reached in those cases, but I wouldn't bet > against it. I think that for above-mentioned background workers, we follow a different path to call ForgetBackgroundWorker(). CleanupBackgroundWorker() - ReportBackgroundWorkerExit()

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-10 Thread Kuntal Ghosh
ed integer will be near UINT32_MAX. Hence, we need the absolute difference after typecasting the same to integer. This value should be less than the max_parallel_workers upper limit. I've attached both the patches for better accessibility. PFA. -- Thanks & Regards, Kuntal Ghosh Enterpri

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-06 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 6:49 PM, Amit Kapila wrote: > On Wed, Apr 5, 2017 at 12:35 PM, Kuntal Ghosh > wrote: >> On Tue, Apr 4, 2017 at 11:22 PM, Tomas Vondra >>> I'm probably missing something, but I don't quite understand how these >>> values actually su

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 7:45 PM, Robert Haas wrote: > On Wed, Apr 5, 2017 at 10:09 AM, Kuntal Ghosh > wrote: >>> Did you intend to attach that patch to this email? >>> >> Actually, I'm confused how we should ensure (register_count > >> terminate_cou

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 7:31 PM, Robert Haas wrote: > On Wed, Apr 5, 2017 at 6:36 AM, Kuntal Ghosh > wrote: >> Yes. But, as Robert suggested up in the thread, we should not use >> (parallel_register_count = 0) as an alternative to define a bgworker >> crash. Hence, I&#x

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
WorkerArray claims this > is the case. > Agreed on the overflowed case. But, my concern is when an overflow has not yet happened: Suppose, uint parallel_register_count = 1; /* Didn't overflow* / uint parallel_terminate_count = 2; /* Didn't overflow */ Assert(parallel_register_count - par

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 3:07 PM, Tomas Vondra wrote: > > > On 04/05/2017 09:05 AM, Kuntal Ghosh wrote: >> >> AFAICU, during crash recovery, we wait for all non-syslogger children >> to exit, then reset shmem(call BackgroundWorkerShmemInit) and perform >> StartupDat

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
ver unnecessarily while executing q2. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Tue, Apr 4, 2017 at 11:22 PM, Tomas Vondra wrote: > On 04/04/2017 06:52 PM, Robert Haas wrote: >> >> On Mon, Apr 3, 2017 at 6:08 AM, Kuntal Ghosh >> wrote: >>> >>> On Fri, Mar 31, 2017 at 6:50 PM, Robert Haas >>> wrote: >>>> &g

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-03 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 6:50 PM, Robert Haas wrote: > On Thu, Mar 30, 2017 at 4:35 PM, Kuntal Ghosh > wrote: >> 2. the server restarts automatically, initialize >> BackgroundWorkerData->parallel_register_count and >> BackgroundWorkerData->parallel_terminate_count

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 10:40 AM, Beena Emerson wrote: > On 30 Mar 2017 15:10, "Kuntal Ghosh" wrote: >> 03-modify-tools.patch - Makes XLogSegSize into a variable, currently set >> as >> XLOG_SEG_SIZE and modifies the tools to fetch the size instead of using >

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 5:43 AM, Neha Khatri wrote: > > On Fri, Mar 31, 2017 at 8:29 AM, Kuntal Ghosh > wrote: >> >> On Fri, Mar 31, 2017 at 2:05 AM, Kuntal Ghosh >> wrote: >> > >> > 1. Put an Assert(0) in ParallelQueryMain(), start server

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 2:05 AM, Kuntal Ghosh wrote: > > 1. Put an Assert(0) in ParallelQueryMain(), start server and execute > any parallel query. > In LaunchParallelWorkers, you can see >nworkers = n nworkers_launched = n (n>0) > But, all the workers will crash

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Kuntal Ghosh
) >= max_parallel_workers) DO NOT launch any parallel worker. Hence, nworkers = n nworkers_launched = 0. I thought because of my stupid mistake the parallel worker is crashing, so, this is supposed to happen. Sorry for that. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Kuntal Ghosh
db_tests.patch adds tap tests to initialize cluster with different > wal_segment_size and then check the config values. What other tests do you > have in mind? Checking the various tools? Nothing from me to add here. I've nothing to add here for the attached set of patches. On top of the

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-27 Thread Kuntal Ghosh
Thank you Robert for committing the patch. commit fc70a4b0df38bda6a13941f1581f25fbb643c7f3 I've changed the status to Committed. On Mon, Mar 27, 2017 at 6:09 AM, Michael Paquier wrote: > On Sat, Mar 25, 2017 at 5:26 PM, Kuntal Ghosh > wrote: >> On Fri, Mar 24, 2017 at 9:2

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-25 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 9:23 PM, Robert Haas wrote: > On Thu, Mar 23, 2017 at 7:29 AM, Michael Paquier > wrote: >> On Thu, Mar 23, 2017 at 8:19 PM, Kuntal Ghosh >> wrote: >>> Hence, to be consistent with others, bgworker processes

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-24 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 2:17 PM, Kuntal Ghosh wrote: > On Fri, Mar 24, 2017 at 12:35 PM, Craig Ringer wrote: >> On 24 March 2017 at 14:07, Kuntal Ghosh wrote: >>> On Fri, Mar 24, 2017 at 11:28 AM, Kuntal Ghosh >>> wrote: >>>> Hello, >>>> In

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-24 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 12:35 PM, Craig Ringer wrote: > On 24 March 2017 at 14:07, Kuntal Ghosh wrote: >> On Fri, Mar 24, 2017 at 11:28 AM, Kuntal Ghosh >> wrote: >>> Hello, >>> In Windows, if one needs to take a dump in plain text format (this is >>&g

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-23 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 11:28 AM, Kuntal Ghosh wrote: > Hello, > In Windows, if one needs to take a dump in plain text format (this is > the default option, or can be specified using -Fp) with some level of > compression (-Z[0-9]), an output file has to > be specified. Otherwise, i

[HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-23 Thread Kuntal Ghosh
ain-text format, fmt is set to archNull. In that case, the binary mode will not be forced(I think). To fix this, I've attached a patch which adds one extra check in the 'if condition' to check the compression level. PFA. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-23 Thread Kuntal Ghosh
iterMain() walreceiver WalReceiverMain() walsender InitWalSender() Hence, to be consistent with others, bgworker processes can be initialized from BackgroundWorkerInitializeConnectionBy[Oid]. I've attached the updated patches which reflect the above change. PFA. -- Thanks & Regards, Kunt

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Kuntal Ghosh
n main () I think that the problem is in following code: /* parse files as start/end boundaries, extract path if not specified */ if (optind < argc) { + if (!RetrieveXLogSegSize(full_path)) ... } In this case, RetrieveXLogSegSize is conditionally called. So, if the condition is false

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-21 Thread Kuntal Ghosh
7;ve made >> it visible to all user for now. >> >> Please find the updated patches in the attachment. > > Yeah, hiding it may make sense... Modified. > /* The autovacuum launcher is done here */ > if (IsAutoVacuumLauncherProcess()) > + { > return; >

Re: [HACKERS] Two phase commit in ECPG

2017-03-17 Thread Kuntal Ghosh
On Fri, Mar 17, 2017 at 4:34 PM, Masahiko Sawada wrote: > On Fri, Mar 17, 2017 at 12:17 PM, Kuntal Ghosh > wrote: >> On Tue, Mar 14, 2017 at 1:35 AM, Michael Meskes >> wrote: >>>> Previous 002 patch lacked to add describing PREPARE TRANSACTION. >>>>

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-17 Thread Kuntal Ghosh
On Thu, Mar 16, 2017 at 1:18 PM, Michael Paquier wrote: > On Wed, Mar 15, 2017 at 9:14 PM, Kuntal Ghosh > wrote: >> I've attached the updated patches. > > Thanks for the new versions. This begins to look really clear. Thanks again for the review. > Having some acti

Re: [HACKERS] Two phase commit in ECPG

2017-03-17 Thread Kuntal Ghosh
e issue. But, I'm not sure whether this test should be performed by installcheck by default or should only be performed by make installcheck-prepared-txns. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@p

Re: [HACKERS] parallelize queries containing initplans

2017-03-15 Thread Kuntal Ghosh
open up > those cases if required in a separate patch. +1. Unfortunately, this patch doesn't enable parallelism for all possible cases with InitPlans. Our objective is to keep things simple and clean. Still, TPC-H q22 runs 2.5~3 times faster with this patch. -- Thanks & Regards, Kuntal

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-15 Thread Kuntal Ghosh
ELECT pg_stat_get_backend_pid(s.backendid) AS pid, + |pg_stat_get_backend_activity(s.backendid) AS query + | FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s; 16925 | 16927 | 16926 | 16929 | IMHO, this scenario can be easily avoided by fil

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-14 Thread Kuntal Ghosh
pgist, >> +brin, and generic. Only > > Did that, committed this. Also ran pgindent over hash_mask() and > fixed an instance of dubious capitalization. Thanks Robert for the commit. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-14 Thread Kuntal Ghosh
ndex. This is useful for generating a set of currently active client backend ID numbers (from 1 to the number of active client backends). These IDs are used for some pgstat_* functions relevant to client processes, e.g., pg_stat_get_backend_activity, pg_stat_get_backend_client_port etc. Any sugg

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-09 Thread Kuntal Ghosh
er Activity| WalSenderMain | idle | wal sender | | active | client backend Activity| BgWriterMain| idle | writer Activity| CheckpointerMain| idle | checkpointer Activity| WalWriterMain | idle | wa

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-09 Thread Kuntal Ghosh
t; > @@ -248,6 +248,9 @@ BackgroundWriterMain(void) > */ > prev_hibernate = false; > > +/* report walwriter process in the PgBackendStatus array */ > +pgstat_procstart(); > + > > s/walwriter/writer/g Fixed. > Patch 0004 should update monitoring.sgml. A

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-08 Thread Kuntal Ghosh
On Fri, Mar 3, 2017 at 9:44 AM, Amit Kapila wrote: > On Tue, Feb 28, 2017 at 11:06 AM, Kuntal Ghosh > wrote: >> Hello everyone, >> >> I've attached a patch which implements WAL consistency checking for >> hash indexes. This feature is going to be useful f

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-03-05 Thread Kuntal Ghosh
usy system; I'm fairly strongly of the opinion > that you ought to downgrade that by a couple of levels, say to DEBUG3 > or so. +1 I've tested with TPC-H query 18 and it's working fine. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-03-02 Thread Kuntal Ghosh
ould require an extra shift op every time we want to find the next or prev location during a collision. [1] https://www.postgresql.org/message-id/caeepm%3d3rdgjfxw4ckvj0oemya2-34b0qhng1xv0vk7tgpjg...@mail.gmail.com -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 10:53 AM, Andres Freund wrote: > On 2017-03-01 10:47:45 +0530, Kuntal Ghosh wrote: >> if (insertdist > curdist) >> { >> swap the entry to be inserted with the current entry. >> Try to insert the current entry in the same logic. >> } &g

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 9:38 AM, Andres Freund wrote: > Hi, > > On 2017-03-01 09:33:07 +0530, Kuntal Ghosh wrote: >> On Wed, Mar 1, 2017 at 9:19 AM, Andres Freund wrote: >> >> So, I was looking for other alternatives and I've found one called >> >&g

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 9:33 AM, Kuntal Ghosh wrote: > On Wed, Mar 1, 2017 at 9:19 AM, Andres Freund wrote: >> That's without the patch in >> http://archives.postgresql.org/message-id/20161123083351.5vramz52nmdokhzz%40alap3.anarazel.de >> ? With that patch it should co

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 9:19 AM, Andres Freund wrote: > On 2017-03-01 09:13:15 +0530, Kuntal Ghosh wrote: >> On Wed, Mar 1, 2017 at 8:48 AM, Andres Freund wrote: >> >> BTW, another option to consider is lowering the target fillfactor. >> >> IIRC, Kuntal ment

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
or the existing element is less than the current probe length for the element being inserted, swap the two elements and keep going. It leads to a low variance for the chain lengths rather than the last one. Is this approach already considered? -- Thanks & Regards, Kuntal Ghosh Enterprise

[HACKERS] WAL Consistency checking for hash indexes

2017-02-27 Thread Kuntal Ghosh
https://www.postgresql.org/message-id/CAA4eK1%2B%2BP%2BjVZC7MC3xzw5uLCpva9%2BgsBpd3semuWffWAftr5Q%40mail.gmail.com -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com 0001-wal_consistency_checking-for-hash-index.patch Description: application/download -- Sent via pgsq

Re: [HACKERS] increasing the default WAL segment size

2017-02-27 Thread Kuntal Ghosh
On Tue, Feb 28, 2017 at 9:45 AM, Jim Nasby wrote: > On 2/24/17 6:30 AM, Kuntal Ghosh wrote: >> >> * You're considering any WAL file with a power of 2 as valid. Suppose, >> the correct WAL seg size is 64mb. For some reason, the server >> generated a 16mb invalid

Re: [HACKERS] increasing the default WAL segment size

2017-02-24 Thread Kuntal Ghosh
umentation describing the scope and limitations of each approach? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Performance degradation in TPC-H Q18

2017-02-21 Thread Kuntal Ghosh
sh expansion. The problem with the patch is that it triggers a hash expansion even when the filled percentage is pretty low, causing unnecessary memory consumption. Thoughts? [1] https://www.postgresql.org/message-id/20161123083351.5vramz52nmdokhzz%40alap3.anarazel.de 0001-Resize-sim

Re: [HACKERS] Passing query string to workers

2017-02-20 Thread Kuntal Ghosh
xt is a const char* variable. Assigning this const pointer to a non-const pointer violates the rules constant-correctness. So, either you should change query_data as const char*, or as Robert suggested, you can directly use estate->es_sourceText. -- Thanks & Regards, Kuntal Ghosh Enterprise

Re: [HACKERS] Passing query string to workers

2017-02-16 Thread Kuntal Ghosh
ed for parallel workers in pg_stat_activity, log statements and failed-query statements. Moved status to Ready for committer. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your su

Re: [HACKERS] Passing query string to workers

2017-02-16 Thread Kuntal Ghosh
lookup(toc, PARALLEL_KEY_QUERY_TEXT)); Just one lookup is sufficient. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] increasing the default WAL segment size

2017-02-16 Thread Kuntal Ghosh
s a rebase based on the commit 85c11324cabaddcfaf3347df7 (Rename user-facing tools with "xlog" in the name to say "wal"). -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-16 Thread Kuntal Ghosh
NULL, NULL, NULL + }, Maximum value for vacuum_cleanup_index_scale_factor should be 1 instead of 100. As the code indicates, it is certainly not treated as a percentage fraction of relpages. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via p

Re: [HACKERS] Write Ahead Logging for Hash Indexes

2017-02-15 Thread Kuntal Ghosh
On Mon, Feb 13, 2017 at 8:52 PM, Amit Kapila wrote: > As discussed, attached are refactoring patches and a patch to enable > WAL for the hash index on top of them. 0006-Enable-WAL-for-Hash-Indexes.patch needs to be rebased after commit 8da9a226369e9ceec7cef1. -- Thanks & Regards, Ku

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-02-15 Thread Kuntal Ghosh
_backend_idset() AS backendid) AS s; Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com 0001-Infra-to-expose-non-backend-processes-in-pg_stat_get.patch Description: application/download 0002-Expose-stats-for-auxiliary-processes-in-pg_stat_get_.patch Descript

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-10 Thread Kuntal Ghosh
;m a newbie.) +This parameter can only be set anywhere. Oxymoron. :-) -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Passing query string to workers

2017-02-10 Thread Kuntal Ghosh
would be to move pgstat_report_activity in ParallelQueryMain instead of ParallelWorkerMain. Then, you can remove the macro definition from parallel.c. Thoughts? And, the value of the macro seems pretty random to me. IMO, it should be UINT64CONST(0xE007). -- Thanks & Regards, Kunt

Re: [HACKERS] WAL consistency check facility

2017-02-08 Thread Kuntal Ghosh
On Thu, Feb 9, 2017 at 2:26 AM, Robert Haas wrote: > On Wed, Feb 8, 2017 at 1:25 AM, Kuntal Ghosh > wrote: >> Thank you Robert for the review. Please find the updated patch in the >> attachment. > > I have committed this patch after fairly extensive revisions: > Thank

Re: [HACKERS] WAL consistency check facility

2017-02-07 Thread Kuntal Ghosh
On Tue, Jan 31, 2017 at 9:36 PM, Robert Haas wrote: > On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh > wrote: >> I've attached the patch with the modified changes. PFA. > > Can this patch check contrib/bloom? > Added support for generic resource manager type. > +

Re: [HACKERS] WAL consistency check facility

2017-01-31 Thread Kuntal Ghosh
;> things more convenient for the masking functions? If not, could those >> functions at least do something like "Page page = (Page) pagebytes;" >> rather than "Page page_norm = (Page) page;"? > > xlog_internal.h is used as well by frontends, this ma

  1   2   >