Re: [HACKERS] UPDATE of partition key

2017-05-18 Thread Amit Kapila
On Wed, May 17, 2017 at 4:05 PM, Dilip Kumar wrote: > On Wed, May 17, 2017 at 3:15 PM, Amit Kapila wrote: >>> Earlier I thought that option1 is better but later I think that this >>> can complicate the situation as we are firing first BR update then BR >>> delete a

Re: [HACKERS] UPDATE of partition key

2017-05-17 Thread Amit Kapila
On Wed, May 17, 2017 at 5:17 PM, Robert Haas wrote: > On Wed, May 17, 2017 at 6:29 AM, Amit Kapila wrote: >> I think we can do this even without using an additional infomask bit. >> As suggested by Greg up thread, we can set InvalidBlockId in ctid to >> indicate such an

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-17 Thread Amit Kapila
t; commands in a symmetric manner. > +1. Why not similar behavior for any other statements executed in this module by do_sql_command? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPDATE of partition key

2017-05-17 Thread Amit Kapila
On Mon, May 15, 2017 at 5:28 PM, Robert Haas wrote: > On Fri, May 12, 2017 at 3:07 AM, Amit Kapila wrote: >> I agree with you that it might not be straightforward to make it work, >> but now that earliest it can go is v11, do we want to try doing >> something other tha

Re: [HACKERS] UPDATE of partition key

2017-05-17 Thread Amit Kapila
> delete and can change the row multiple time and defining such > behaviour can be complicated. > If we have to go by this theory, then the option you have preferred will still execute BR triggers for both delete and insert, so input row can still be changed twice. -- With Regards, Amit Ka

Re: [HACKERS] Increasing parallel workers at runtime

2017-05-17 Thread Amit Kapila
ay be wrong, but the execution state initialization > routines are written with the assumption that all the workers start > simultaneously? > No such assumptions, workers started later can also join the execution of the query. -- With Regards, Amit Kapila. EnterpriseDB: http://www.en

Re: [HACKERS] Increasing parallel workers at runtime

2017-05-17 Thread Amit Kapila
uential Scan patch and I think some other databases uses some similar technique for this problem. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] bumping HASH_VERSION to 3

2017-05-16 Thread Amit Kapila
On Tue, May 16, 2017 at 5:14 PM, Robert Haas wrote: > On Tue, May 16, 2017 at 7:31 AM, Amit Kapila wrote: > >> I will send an updated patch once we agree on above points. > > Sounds good. > Attached patch addresses all the comments as discussed. -- With Regards, Amit

Re: [HACKERS] NOT NULL constraints on range partition key columns

2017-05-16 Thread Amit Kapila
On Tue, May 16, 2017 at 3:26 PM, Amit Langote wrote: > On 2017/05/16 4:29, Robert Haas wrote: >> On Mon, May 15, 2017 at 9:12 AM, Amit Kapila wrote: >>> Can't we allow NULL to get inserted into the partition (leaf >>> partition) if the user uses the partition name

Re: [HACKERS] bumping HASH_VERSION to 3

2017-05-16 Thread Amit Kapila
27;t this cause the hash indexes to be invalided in the old cluster > rather than the new one? > oops. copy-paste. It passed in my testing because I have not used any different options (like port number) for old or new server. > This might need a visit from pgindent in one or two places, too. > I have run pgindent before sending the previous version, but will again verify the same. I will send an updated patch once we agree on above points. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] NOT NULL constraints on range partition key columns

2017-05-15 Thread Amit Kapila
as well document instead that we don't > currently support routing tuples with NULL values in the partition key > through a range-partitioned table and so NULL values cause error. > Can't we allow NULL to get inserted into the partition (leaf partition) if the user uses the

Re: [HACKERS] [PATCH v2] Progress command to monitor progression of long running SQL queries

2017-05-13 Thread Amit Kapila
just end the query execution. Also, even if it is okay, there doesn't seem to be a way by which a parallel worker can communicate the error back to master backend, rather it will just exit silently which is not right. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2017-05-13 Thread Amit Kapila
perations, I think we hold it over multiple kernel calls. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2017-05-13 Thread Amit Kapila
ql.org/message-id/CAFiTN-tkX6gs-jL8VrPxg6OG9VUAKnObUq7r7pWQqASzdF5OwA%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Hash Functions

2017-05-12 Thread Amit Kapila
think of defining separate portable hash functions which can be used for the purpose of hash partitioning? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPDATE of partition key

2017-05-12 Thread Amit Kapila
On Fri, May 12, 2017 at 10:49 AM, Amit Khandekar wrote: > On 12 May 2017 at 08:30, Amit Kapila wrote: >> On Thu, May 11, 2017 at 5:41 PM, Amit Khandekar >> wrote: > >> If we try to compare it with the non-partitioned update, >> there also it is internally a delet

Re: [HACKERS] UPDATE of partition key

2017-05-12 Thread Amit Kapila
up with something doable without a big architecture change. What is your take on this point now? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPDATE of partition key

2017-05-11 Thread Amit Kapila
On Fri, May 12, 2017 at 9:27 AM, Amit Kapila wrote: > On Thu, May 11, 2017 at 5:45 PM, Amit Khandekar > wrote: >> On 11 May 2017 at 17:24, Amit Kapila wrote: >>> Few comments: >>> 1. >>> Operating directly on partition doesn't allow update to move ro

Re: [HACKERS] UPDATE of partition key

2017-05-11 Thread Amit Kapila
On Thu, May 11, 2017 at 5:45 PM, Amit Khandekar wrote: > On 11 May 2017 at 17:24, Amit Kapila wrote: >> Few comments: >> 1. >> Operating directly on partition doesn't allow update to move row. >> Refer below example: >> create table t1(c1 int) partition b

Re: [HACKERS] UPDATE of partition key

2017-05-11 Thread Amit Kapila
On Thu, May 11, 2017 at 5:41 PM, Amit Khandekar wrote: > On 11 May 2017 at 17:23, Amit Kapila wrote: >> On Fri, Mar 17, 2017 at 4:07 PM, Amit Khandekar >> wrote: >>> On 4 March 2017 at 12:49, Robert Haas wrote: >>>> On Thu, Mar 2, 2017 at 11:53 AM, Amit K

Re: [HACKERS] UPDATE of partition key

2017-05-11 Thread Amit Kapila
e? Basically, if there is no matching partition, then move it to default partition. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPDATE of partition key

2017-05-11 Thread Amit Kapila
ocumentation in the patch indicates that this patch will only fire Update statement level triggers which is odd. Note in docs about Insert On Conflict "Note that with an INSERT with an ON CONFLICT DO UPDATE clause, both INSERT and UPDATE statement level trigger will be fired. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Remove pre-10 compatibility code in hash index

2017-05-09 Thread Amit Kapila
On Wed, May 10, 2017 at 9:17 AM, Robert Haas wrote: > On Tue, May 9, 2017 at 1:41 AM, Amit Kapila wrote: >> Commit ea69a0dead5128c421140dc53fac165ba4af8520 has bumped the hash >> index version and obviates the need for backward compatibility code >

Re: [HACKERS] bumping HASH_VERSION to 3

2017-05-09 Thread Amit Kapila
ade/version_old_8_3.c::old_8_3_invalidate_hash_gin_indexes() > Thanks for the pointer. > I would be happy to restore that code and make it work for PG 10. > Attached patch implements the two points suggested by Tom. I will add this to PG-10 open issues list s

[HACKERS] Remove pre-10 compatibility code in hash index

2017-05-08 Thread Amit Kapila
compatibility code in hash index. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com remove_pre10_compat_hash_index_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] [PROPOSAL] Use SnapshotAny in get_actual_variable_range

2017-05-08 Thread Amit Kapila
actionIdLimitedForOldSnapshots() as that is required to override xmin for table vacuum/pruning purposes. > Maybe we need > to use GetOldestXmin()? It is a costly call as it needs ProcArrayLock, so I don't think it makes sense to use it here. -- With Regards, Amit Kapila. Enterprise

Re: [HACKERS] Atomics for heap_parallelscan_nextpage()

2017-05-08 Thread Amit Kapila
* We don't guarantee any specific ordering in general, though. */ if (scan->rs_syncscan) ss_report_location(scan->rs_rd, page); .. } -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-06 Thread Amit Kapila
27;t seem like a good plan. I'll adjust the patch so >> that it continues to ignore errors in that case. > > Updated patch attached. > Patch looks good to me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] [PROPOSAL] Use SnapshotAny in get_actual_variable_range

2017-05-06 Thread Amit Kapila
xmin should be used there? > I think we can use RecentGlobalDataXmin for non-catalog relations and RecentGlobalXmin for catalog relations (probably a check similar to what we have in heap_page_prune_opt). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-05 Thread Amit Kapila
On Sat, May 6, 2017 at 4:41 AM, Amit Kapila wrote: > On Fri, May 5, 2017 at 9:32 PM, Robert Haas wrote: >> On Fri, May 5, 2017 at 11:15 AM, Amit Kapila wrote: >>> I am not saying to rip those changes. Your most of the mail stresses >>> about the 30-second timeout

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-05 Thread Amit Kapila
On Fri, May 5, 2017 at 9:32 PM, Robert Haas wrote: > On Fri, May 5, 2017 at 11:15 AM, Amit Kapila wrote: >> I am not saying to rip those changes. Your most of the mail stresses >> about the 30-second timeout which you have added in the patch to >> detect timeouts duri

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-05 Thread Amit Kapila
On Fri, May 5, 2017 at 7:44 PM, Robert Haas wrote: > On Fri, May 5, 2017 at 1:01 AM, Amit Kapila wrote: >> I have tried to verify above point and it seems to me in such a >> situation the transaction status will be either PQTRANS_INTRANS or >> PQTRANS_INERROR. I have t

Re: [HACKERS] [PROPOSAL] Use SnapshotAny in get_actual_variable_range

2017-05-05 Thread Amit Kapila
eapTupleSatisfiesNonVacuumable(HeapTuple htup, Snapshot snapshot, + Buffer buffer) +{ + return HeapTupleSatisfiesVacuum(htup, snapshot->xmin, buffer) + != HEAPTUPLE_DEAD; +} + Add comments on top of this function and for the sake of consistency update the file header as well (Summary of visibility f

Re: [HACKERS] Change GetLastImportantRecPtr's definition? (wasSkip checkpoints, archiving on idle systems.)

2017-05-05 Thread Amit Kapila
On Fri, May 5, 2017 at 11:57 AM, Andres Freund wrote: > Hi, > > On 2017-05-05 11:50:12 +0530, Amit Kapila wrote: >> I see that EndPos can be updated in one of the cases after releasing >> the lock (refer below code). Won't that matter? > > I can't see ho

Re: [HACKERS] Change GetLastImportantRecPtr's definition? (wasSkip checkpoints, archiving on idle systems.)

2017-05-04 Thread Amit Kapila
On Fri, May 5, 2017 at 11:43 AM, Andres Freund wrote: > On 2017-05-05 11:04:14 +0530, Amit Kapila wrote: >> On Fri, May 5, 2017 at 6:54 AM, Andres Freund wrote: >> > Hi, >> > >> > On 2016-12-22 19:33:30 +, Andres Freund wrote: >> >> Skip chec

Re: [HACKERS] modeling parallel contention (was: Parallel Append implementation)

2017-05-04 Thread Amit Kapila
seqscan > I think one of the points of scaling/contention is tuple communication. This is what is shown is perf profiles and we (one of my colleagues is working on it) are already working on some ways to improve the same, but I don't think we can get anywhere near to linear scaling b

Re: [HACKERS] Change GetLastImportantRecPtr's definition? (wasSkip checkpoints, archiving on idle systems.)

2017-05-04 Thread Amit Kapila
urns the beginning of said record. > > This is trivially fixable by replacing < with <=. But I wonder if the > better fix would be to redefine GetLastImportantRecPtr() to point to the > end of the record, too? > If you think it is straightforward to note the end of the record, then that sounds

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-04 Thread Amit Kapila
On Thu, May 4, 2017 at 10:40 PM, Robert Haas wrote: > On Thu, May 4, 2017 at 1:04 PM, Amit Kapila wrote: >> On Thu, May 4, 2017 at 10:18 PM, Robert Haas wrote: >>> On Thu, May 4, 2017 at 12:18 PM, Amit Kapila >>> wrote: >>>> As soon as the first co

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-04 Thread Amit Kapila
On Thu, May 4, 2017 at 8:08 PM, Robert Haas wrote: > On Thu, May 4, 2017 at 7:13 AM, Amit Kapila wrote: >> In pgfdw_xact_callback, if the execution of ABORT TRANSACTION fails >> due to any reason then I think it will close the connection. The >> relavant code is: >&g

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-04 Thread Amit Kapila
On Thu, May 4, 2017 at 10:18 PM, Robert Haas wrote: > On Thu, May 4, 2017 at 12:18 PM, Amit Kapila wrote: >> As soon as the first command fails due to timeout, we will set >> 'abort_cleanup_failure' which will make a toplevel transaction to >> abort and also

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-04 Thread Amit Kapila
On Thu, May 4, 2017 at 8:08 PM, Robert Haas wrote: > On Thu, May 4, 2017 at 7:13 AM, Amit Kapila wrote: > >>> - For bonus points, give pgfdw_exec_query() an optional timeout >>> argument, and set it to 30 seconds or so when we're doing abort >>> cleanup.

Re: [HACKERS] statement_timeout is not working as expected with postgres_fdw

2017-05-04 Thread Amit Kapila
ocessing. > > - Change uses of PQexec() in the abort path to use pgfdw_exec_query() > instead. If we exit pgfdw_exec_query() with an error, we'll re-enter > the abort path, but now in_abort_cleanup will be set, so we'll just > drop the connection (and force any outer tran

Re: [HACKERS] [PROPOSAL] Use SnapshotAny in get_actual_variable_range

2017-05-01 Thread Amit Kapila
e case, then how would using SnapshotAny solve this problem. We get the value from index first and then check its visibility in heap, so if time is spent in _bt_checkkeys, why would using a different kind of Snapshot solve the problem? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterpr

Re: [HACKERS] RFC: ALTER SYSTEM [...] COMMENT

2017-05-01 Thread Amit Kapila
a patch for this feature. [1] - https://www.postgresql.org/message-id/CA%2BTgmoZBDLhDexHyTJ%3DH0xZt7-6M%3Dh%2Bv2Xi0Myj7Q37QGZQb4g%40mail.gmail.com [2] - https://www.postgresql.org/message-id/CAA4eK1%2B%3DovYQqYGHcX2OBU3mk%2BhSHjFDpvmrHCos1Vgj8_b6vg%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-30 Thread Amit Kapila
On Mon, May 1, 2017 at 10:23 AM, Tom Lane wrote: > Amit Kapila writes: >> Yeah, that's right. Today, I have spent some time to analyze how and >> where retry logic is required. I think there are two places where we >> need this retry logic, one is if we

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-30 Thread Amit Kapila
logic for handling process exit or alternatively we might want to just try reattach operation in a loop before giving up. Do we want to keep this retry logic for a certain number of times say (10 times) or we want to try indefinitely? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprise

Re: [HACKERS] PG 10 release notes

2017-04-27 Thread Amit Kapila
On Thu, Apr 27, 2017 at 7:53 PM, Marko Tiikkaja wrote: > On Thu, Apr 27, 2017 at 4:13 PM, Bruce Momjian wrote: >> >> On Thu, Apr 27, 2017 at 08:00:28AM +0530, Amit Kapila wrote: >> > > Oh, so non-correlated subqueries can be run in parallel. Yes, that is >> &g

Re: [HACKERS] PG 10 release notes

2017-04-26 Thread Amit Kapila
s, it is unfotunate that the item is in the incompatibility item. I >> wonder if I should split out the need to rebuild the hash indexes and >> keep it there and move this item into the "Index" section. > > Done, items split. >

Re: [HACKERS] PG 10 release notes

2017-04-26 Thread Amit Kapila
On Wed, Apr 26, 2017 at 8:46 PM, Bruce Momjian wrote: > On Wed, Apr 26, 2017 at 07:38:05PM +0530, Amit Kapila wrote: >> >> I have already mentioned the commit id (5e6d8d2b). Text can be "Allow >> >> queries containing subplans to execute in parallel". We

Re: [HACKERS] PG 10 release notes

2017-04-26 Thread Amit Kapila
On Tue, Apr 25, 2017 at 7:19 PM, Bruce Momjian wrote: > On Tue, Apr 25, 2017 at 09:00:45AM +0530, Amit Kapila wrote: >> On Tue, Apr 25, 2017 at 8:35 AM, Bruce Momjian wrote: >> > On Tue, Apr 25, 2017 at 08:30:50AM +0530, Amit Kapila wrote: >> >> On Tue, Apr 25,

Re: [HACKERS] PG 10 release notes

2017-04-25 Thread Amit Kapila
09379a4cbf1fbd847d1cee2293c5e4 is missed? Not sure if this >> requires a separate entry or could be merged with -- Support parallel >> btree index scans. > > This item was merged into the general parallel index scan item: > Okay, but then shouldn't we add Rafia's name as

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-25 Thread Amit Kapila
ns like: improve those latencies --> reduce those latencies such slow standby --> a slow standby -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PG 10 release notes

2017-04-24 Thread Amit Kapila
On Tue, Apr 25, 2017 at 8:35 AM, Bruce Momjian wrote: > On Tue, Apr 25, 2017 at 08:30:50AM +0530, Amit Kapila wrote: >> On Tue, Apr 25, 2017 at 7:01 AM, Bruce Momjian wrote: >> > I have committed the first draft of the Postgres 10 release notes. They >> > are curren

Re: [HACKERS] PG 10 release notes

2017-04-24 Thread Amit Kapila
On Tue, Apr 25, 2017 at 8:30 AM, Amit Kapila wrote: > On Tue, Apr 25, 2017 at 7:01 AM, Bruce Momjian wrote: >> I have committed the first draft of the Postgres 10 release notes. They >> are current as of two days ago, and I will keep them current. Please >> give me

Re: [HACKERS] PG 10 release notes

2017-04-24 Thread Amit Kapila
n many cases. ea69a0dead5128c421140dc53fac165ba4af8520 Expand hash indexes more gradually. I think the above commit needs a separate mention, as this is a really huge step forward to control the size of hash indexes. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Se

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-24 Thread Amit Kapila
On Fri, Apr 21, 2017 at 12:55 AM, Andres Freund wrote: > On 2017-04-20 16:57:03 +0530, Amit Kapila wrote: >> On Wed, Apr 19, 2017 at 9:01 PM, Andres Freund wrote: >> > On 2017-04-19 10:15:31 -0400, Tom Lane wrote: >> >> Amit Kapila writes: >> >> &

Re: [HACKERS] dtrace probes

2017-04-20 Thread Amit Kapila
n PG-10 which has caused this behavior? If not, then I don't think it should be added to open items of PG-10. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-18 Thread Amit Kapila
ct we'd need some retry > logic anyway for that case. > Yeah, that kind of thing can work assuming we don't get conflicts too often, but it could be possible that conflicts are not reported from ASLR enabled environments because of commit 7f3e17b4. [1] - https://blogs.msdn.

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Wed, Apr 19, 2017 at 1:27 AM, Tom Lane wrote: > Amit Kapila writes: >> I have ended up doing something along the lines suggested by you (or >> at least what I have understood from your e-mail). Basically, pass >> the safe-param-ids list to parallel safety function and de

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Mon, Apr 17, 2017 at 10:54 AM, Tom Lane wrote: > Amit Kapila writes: >> On Sun, Apr 16, 2017 at 9:30 PM, Tom Lane wrote: >>> FYI, I have this on my to-look-at list, and expect to fix it before Robert >>> returns from vacation. > >> Let me know if an

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-16 Thread Amit Kapila
whitelist of params for parallel safety, but if you think that is the better way to go, I can give it a try. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Typo in htup_details.h

2017-04-15 Thread Amit Kapila
Attached patch to fix $SUBJECT. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com typo_htup_details.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-12 Thread Amit Kapila
On Wed, Apr 12, 2017 at 10:30 PM, Tom Lane wrote: > Amit Kapila writes: >> On Wed, Apr 12, 2017 at 12:40 AM, Tom Lane wrote: >>> Anyone want to draft a patch for this? > >> Please find patch attached based on above discussion. > > This patch seems fairly incomp

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-12 Thread Amit Kapila
I'm knocking off for the day a bit early today, but I'll have > a look at it tomorrow, unless anyone in the Far East beats me to it. > Thanks for looking into it. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests take a long time

2017-04-12 Thread Amit Kapila
eed it for the sake of testing that operation. Mithun, as you are the original author of these tests, can you please try some of the above optimizations and any others you can think of and see if we can reduce the time for hash index tests? -- With Regards, Amit Kapila. EnterpriseDB: http://www.

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-12 Thread Amit Kapila
On Wed, Apr 12, 2017 at 12:40 AM, Tom Lane wrote: > Amit Kapila writes: > >> However, the worker will >> never execute such a plan as we don't generate a plan where unsafe >> sublan/initplan is referenced in the node passed to the worker. If we >> want to avo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-04-11 Thread Amit Kapila
Amit brought up > where only one of the indexes receive WARM inserts. > 4. Added code to kill wrong index pointers to do online cleanup. > 5. Added code to set a CLEAR pointer to a WARM pointer when we know that the > entire chain is WARM. This should address the workload Dilip ran and

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-11 Thread Amit Kapila
On Wed, Apr 12, 2017 at 12:40 AM, Tom Lane wrote: > Amit Kapila writes: > >> I think there is a possibility of doing ExecInitNode in a parallel >> worker for a parallel-unsafe subplan, because we pass a list of all >> the sublans stored in planned statement. > > It&

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-11 Thread Amit Kapila
rker will never execute such a plan as we don't generate a plan where unsafe sublan/initplan is referenced in the node passed to the worker. If we want to avoid passing parallel-unsafe subplans to workers, then I think we can maintain a list of parallel safe subplans along with subplans in P

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Amit Kapila
gt; some reason? > AFAICU, during crash recovery, we wait for all non-syslogger children > to exit, then reset shmem(call BackgroundWorkerShmemInit) and perform > StartupDataBase. While starting the startup process we check if any > bgworker is scheduled for a restart. > In general, your theory appears right, but can you check how it behaves in standby server because there is a difference in how the startup process behaves during master and standby startup? In master, it stops after recovery whereas in standby it will keep on running to receive WAL. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] increasing the default WAL segment size

2017-04-04 Thread Amit Kapila
t part to allow internal representation change (only). > > No commitment yet to increasing wal-segsize in the way this patch has it. > What part of patch you don't like and do you have any suggestions to improve the same? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterpr

Re: [HACKERS] Page Scan Mode in Hash Index

2017-04-04 Thread Amit Kapila
f. There won't be any case where we will be having pin > on overflow buf at the end of scan. > What if mark the buffer as invalid after releasing the pin? We are already doing it that way in btree code, refer _bt_drop_lock_and_maybe_pin(). I think if we do that way, then we can do what Ro

Re: [HACKERS] Page Scan Mode in Hash Index

2017-04-04 Thread Amit Kapila
so->hashso_split_bucket_buf) + { + so->currPos.prevPage = InvalidBlockNumber; + LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK); + } + else + { + so->currPos.prevPage = (opaque)->hasho_prevblkno; + _hash_relbuf(rel, so->currPos.buf); + } + + so->currPos.nextPage = (opaque)->ha

Re: [HACKERS] [sqlsmith] Failed assertion in _hash_kill_items/MarkBufferDirtyHint

2017-04-01 Thread Amit Kapila
that patch, then it is good to proceed with this fix. [1] - https://commitfest.postgresql.org/13/999/ -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Page Scan Mode in Hash Index

2017-04-01 Thread Amit Kapila
.. + next_buf = _hash_getbuf_with_strategy(rel, blkno, HASH_WRITE, + LH_OVERFLOW_PAGE, + bstrategy); + After this change, you need to modify comments on top of function hashbucketcleanup() and _hash_squeezebucket(). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-31 Thread Amit Kapila
Do you think it make sense to have a separate thread to discuss and get feedback on same as I am not seeing much input on the knobs you are proposing to handle second pass over index? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TPC-H Q20 from 1 hour to 19 hours!

2017-03-31 Thread Amit Kapila
x27;make > check'. > Your patch looks good to me. I have verified some join cases as well where the behaviour is sane after patch. I have also done testing with force_parallel_mode=regress (ran make check-world) and everything seems good. -- With Regards, Amit Kapila. EnterpriseDB:

Re: [HACKERS] Supporting huge pages on Windows

2017-03-31 Thread Amit Kapila
On Fri, Mar 31, 2017 at 8:05 AM, Tsunakawa, Takayuki wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] >> The latest patch looks good to me apart from one Debug message, so I have >> marked it as Ready For Committer. > > Thank you so much! > > >&g

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-30 Thread Amit Kapila
On Thu, Mar 30, 2017 at 5:55 PM, Pavan Deolasee wrote: > > On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila > wrote: >> >> >> >> How have you verified that? Have you checked that in >> heap_prepare_insert it has called toast_insert_or_update() and then >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-30 Thread Amit Kapila
On Thu, Mar 30, 2017 at 4:07 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee >> wrote: >> > >> > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila >> &g

Re: [HACKERS] Supporting huge pages on Windows

2017-03-30 Thread Amit Kapila
similar to what we display in sysv_shmem.c as below: elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m", allocsize); -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] inconsistent page found on STANDBY server

2017-03-30 Thread Amit Kapila
g registered and > therefore, data (xl_hash_update_meta_page) is not completely recorded > into the wal record. > > Fix: > === > Attached patch fixes this issue. > The fix looks good to me. I have scanned the hash index code to see if there is any other similar problem, but

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-29 Thread Amit Kapila
On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila >> wrote: > > Then during recheck, we pass already compressed values to > index_form_tuple().

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-29 Thread Amit Kapila
On Wed, Mar 29, 2017 at 12:51 PM, Mithun Cy wrote: > On Wed, Mar 29, 2017 at 10:12 AM, Amit Kapila wrote: >> Few other comments: >> +/* >> + * This is just a trick to save a division operation. If you look into the >> + * bitmap of 0-based bucket_num 2nd and 3r

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Tue, Mar 28, 2017 at 10:31 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila > wrote: >> >> As asked previously, can you explain me on what basis are you >> considering it robust? The comments on top of datumIsEqual() clearly >> ind

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila wrote: > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > wrote: >> >> On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila >> wrote: >>> >>> For such an heap insert, we will pass >>> the actual valu

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila > wrote: >> >> >> >> For such an heap insert, we will pass >> the actual value of column to index_form_tuple during index insert. >> However

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-28 Thread Amit Kapila
LY_ONE_PHASE)) & \ + SPLITPOINT_PHASE_MASK) This won't work if we change SPLITPOINT_GROUPS_WITH_ONLY_ONE_PHASE to number other than 3. I think you should change it so that it can work with any value of SPLITPOINT_GROUPS_WITH_ONLY_ONE_PHASE. I think you should name this define as SP

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila wrote: > On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee > wrote: >> On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee >> wrote: >>> >>> >>> >>> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >&g

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was wo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was worried

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-27 Thread Amit Kapila
On Tue, Mar 28, 2017 at 10:43 AM, Mithun Cy wrote: > On Mon, Mar 27, 2017 at 11:21 AM, Amit Kapila wrote: > > > As you have said we can solve it if we allocate buckets always in > power-of-2 when we do hash index meta page init. But on other > occasions, when we try to d

Re: [HACKERS] parallelize queries containing initplans

2017-03-27 Thread Amit Kapila
On Thu, Mar 16, 2017 at 2:34 AM, Kuntal Ghosh wrote: > On Tue, Mar 14, 2017 at 3:20 PM, Amit Kapila wrote: >> Based on that idea, I have modified the patch such that it will >> compute the set of initplans Params that are required below gather >> node and store them as bitm

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-27 Thread Amit Kapila
On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> > >> While looking at this problem, it occurred to me that the assumptio

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-27 Thread Amit Kapila
On Mon, Mar 27, 2017 at 11:21 AM, Amit Kapila wrote: > On Sun, Mar 26, 2017 at 11:26 AM, Mithun Cy > wrote: >> Thanks, Amit for the review. >> On Sat, Mar 25, 2017 at 7:03 PM, Amit Kapila wrote: >>> >>> I think one-dimensional patch has fewer places to t

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-26 Thread Amit Kapila
On Sun, Mar 26, 2017 at 11:26 AM, Mithun Cy wrote: > Thanks, Amit for the review. > On Sat, Mar 25, 2017 at 7:03 PM, Amit Kapila wrote: >> >> I think one-dimensional patch has fewer places to touch, so that looks >> better to me. However, I think there is still hard codi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-25 Thread Amit Kapila
On Sat, Mar 25, 2017 at 11:24 PM, Pavan Deolasee wrote: > > On Sat, 25 Mar 2017 at 11:03 PM, Peter Geoghegan wrote: >> >> On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila >> wrote: >> > I am not sure how do you want to binary compare two datums, if you are >>

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-25 Thread Amit Kapila
On Sat, Mar 25, 2017 at 10:13 AM, Mithun Cy wrote: > On Tue, Mar 21, 2017 at 8:16 AM, Amit Kapila wrote: >> Sure, I was telling you based on that. If you are implicitly treating >> it as 2-dimensional array, it might be easier to compute the array >>offsets. > >

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-25 Thread Amit Kapila
t; My vote would be to leave the GUC behaviour as is and add some tests > to select_parallel.sql which exploit setting max_parallel_workers to 0 > for running some tests. > I think force_parallel_mode=regress should test the same thing. -- With Regards, Amit Kapila. EnterpriseDB: http://www.ent

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-03-25 Thread Amit Kapila
On Sat, Mar 25, 2017 at 12:33 PM, Ashutosh Sharma wrote: > On Sat, Mar 25, 2017 at 11:02 AM, Amit Kapila wrote: >> On Thu, Mar 23, 2017 at 11:24 PM, Ashutosh Sharma >> wrote: >>> Hi, >>> >>> On Tue, Feb 7, 2017 at 9:23 AM, Robert Haas wrote: >>

<    1   2   3   4   5   6   7   8   9   10   >