[
https://issues.apache.org/jira/browse/PHOENIX-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493874#comment-16493874
]
ASF GitHub Bot commented on PHOENIX-3534:
-----------------------------------------
GitHub user twdsilva opened a pull request:
https://github.com/apache/phoenix/pull/303
PHOENIX-3534 Support multi region SYSTEM.CATALOG table
This patch adds two new LinkTypes EXCLUDED_COLUMNS (used to represent a
column that has been dropped) and VIEW_INDEX_PARENT_TABLE (used to link an
index on a view to its parent). Views and view indexes no longer store columns
derived from their ancestors in their metadata. When they are resolved the
ancestors are looked up and added to the PTable that is returned to the client
(see combineColumns in MetadataEndpointImpl). The PTable in the server side
metadata cache only stores the columns created in the view/view index and not
derived columns.
We do not propagate metadata changes made to a parent to all its children.
While adding columns to a base table, we no longer lock all the children in
the view hierarchy, we only validate that the columns being added does not
conflict with an existing base table column. We also don't lock children while
dropping a parent table column. When dropping a parent column we also drop any
view indexes that need the column. This patch does not handle the case when
there are concurrent changes (eg. adding a conflicting column or creating a new
view index that requires a parent column that is being dropped). That will be
handled in a follow-up patch.
While dropping a parent table, we don't drop all the child views metadata.
This metadata needs to be cleaned-up (maybe at compaction time?) which will be
handled in a follow-up patch.
There are a few test failures I am working through, which I will fix soon
and update the PR.
@JamesRTaylor can you please review?
FYI @karanmehta93 @ChinmaySKulkarni
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twdsilva/phoenix PHOENIX-3534
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/303.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #303
----
commit a7712e3977830ffe715a2caaa577eb5cdb071e90
Author: rgidwani <rgidwani@...>
Date: 2017-03-17T19:21:56Z
Starting work on splittable System.Catalog
commit 220849fff34211a4b6356200365afe81d639cfc1
Author: rgidwani <rgidwani@...>
Date: 2017-03-20T20:44:54Z
Removed all references to multi-region System.Catalog for
TableViewFinderResult
commit 356cd43e20be0d01c7ef5f97a9b9d26e213a140d
Author: rgidwani <rgidwani@...>
Date: 2017-03-31T23:11:59Z
Create table work, still trying to get rid of all columns
commit e01adb5f1c45d43a684371dcbf39eadbc381f9d2
Author: rgidwani <rgidwani@...>
Date: 2017-04-04T20:33:07Z
Create table and read views work now
commit 19c7ce54dcd1616ca8b8a7078fa8fc738e70f4f4
Author: rgidwani <rgidwani@...>
Date: 2017-04-04T20:35:30Z
Fixed the test - moving on to add drop columns
commit ab20f8da00145cc527c6f78dc7d493df7ac2f1b0
Author: rgidwani <rgidwani@...>
Date: 2017-04-04T22:59:10Z
getting tests and add column to work
commit ec3574453e8fae662a271f4340f82eaf90f52ce2
Author: rgidwani <rgidwani@...>
Date: 2017-04-05T23:02:29Z
Figuring out the delete logic and refactoring the old tests
commit 7d4133034b7167a7919eb9dd4ab19533ae9300ea
Author: rgidwani <rgidwani@...>
Date: 2017-04-11T22:25:32Z
Added proto timestamp and exluded values to pcolumn also took care of
additive case where we take lower timestamp
commit adfc5ce7f9e7f55f801b5b8af5f414fd7ff96c23
Author: rgidwani <rgidwani@...>
Date: 2017-04-28T23:02:27Z
Drop Column Work in Progress, need to figure out how to resolve the combine
logic for excluded columns
commit 24414bd942055769b68943d93e19013777ff7299
Author: rgidwani <rgidwani@...>
Date: 2017-05-01T22:32:35Z
Alter view drop column works!
commit 13b6e520e3f7a81bb257c7fe68bae81ededa6c99
Author: rgidwani <rgidwani@...>
Date: 2017-05-12T22:55:49Z
Drop Cascade and create check completed, need to test
commit 0313ceee0b0a7515323dc0fc402a6eac76786155
Author: rgidwani <rgidwani@...>
Date: 2017-05-15T20:40:14Z
Drop cascade seems to work
commit 590689f3243a388ae528850c3240427ff49a640c
Author: rgidwani <rgidwani@...>
Date: 2017-05-15T21:13:30Z
Phoenix 3534" to "PHOENIX-3534 Support multi region SYSTEM.CATALOG table
commit 96c0570bfcd767ad2342fb2fa22142eceed2b4eb
Author: rgidwani <rgidwani@...>
Date: 2017-05-24T19:32:05Z
Fixing up a few things, resolving read columns for child views doesn't
always seem to work
commit ca64a0b36f5bf33c8ed3169096fce08ac768d695
Author: rgidwani <rgidwani@...>
Date: 2017-05-31T23:00:31Z
Adding in the child view constants
commit ac59c72784aff0478d9a481db5729827e0ba3cce
Author: rgidwani <rgidwani@...>
Date: 2017-05-31T23:00:31Z
Adding in the child view constants
commit 4d0ba3ac71812ed6bf79e383d7a7ab0ab8089287
Author: rgidwani <rgidwani@...>
Date: 2017-06-16T19:06:24Z
Merge branch 'splittableCatalog' into PHOENIX-3534
Conflicts:
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/WhereConstantParser.java
phoenix-core/src/test/java/org/apache/phoenix/coprocessor/MetaDataEndpointImplTest.java
commit c2addd458dbb69542871ce10efc4a041a85d325d
Author: Thomas <tdsilva@...>
Date: 2017-06-19T18:08:13Z
test fixes
commit 91270753ea7c3197eed1694c47cf8691833677ba
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-09-14T21:55:46Z
Merge branch 'master' of https://github.com/apache/phoenix into PHOENIX-3534
commit a4b64f01be6b0e21edab95753267666b3b4b7a39
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-09-15T03:13:36Z
Remove unnessarch txn manager in ConnectionlessQueryServicesImpl
commit f1ea6621355152dcce236412cd5762e76097ffbc
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-09-15T03:15:07Z
Merge branch 'master' into PHOENIX-3534
commit 571e88b1ee3da6e2bc64236e87e6e19130a588d0
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-05T23:15:00Z
Rename link to VIEW_INDEX_PARENT
commit 308d2a89b819736122547febfddf96c7d98801ce
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-06T05:26:05Z
Merge remote-tracking branch 'upstream/master' into PHOENIX-3534
commit f0e96705d3903c55b5449b09d15c4e3f128bf1ed
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-07T02:59:26Z
fixed tests
commit 53fd995361a8a651247bd330ff1f36823c427f27
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-19T20:10:33Z
add missing files
commit dcddc060bc789bd391302289659584c85baf2031
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-20T03:58:45Z
add view index to parent links during upgrade
commit 508aa4bcf7e32d36e8f040bfb06b9de0ff8881d7
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-21T03:25:48Z
fix tests
commit c32dfbbef5887885b9d003128d86bea9cb62327e
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-21T03:56:07Z
fix test failures
commit abd98d92355c52cbc2c4264a70217ad2f46bb7d1
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-21T03:57:25Z
Merge remote-tracking branch 'upstream/master' into PHOENIX-3534
commit 7c32d46dc721b3f0ab44e27521e8df569bead4d9
Author: Thomas D'Silva <tdsilva@...>
Date: 2017-10-26T22:18:13Z
minor
----
> Support multi region SYSTEM.CATALOG table
> -----------------------------------------
>
> Key: PHOENIX-3534
> URL: https://issues.apache.org/jira/browse/PHOENIX-3534
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
> Assignee: Thomas D'Silva
> Priority: Major
> Attachments: PHOENIX-3534-wip.patch
>
>
> Currently Phoenix requires that the SYSTEM.CATALOG table is single region
> based on the server-side row locks being held for operations that impact a
> table and all of it's views. For example, adding/removing a column from a
> base table pushes this change to all views.
> As an alternative to making the SYSTEM.CATALOG transactional (PHOENIX-2431),
> when a new table is created we can do a lazy cleanup of any rows that may be
> left over from a failed DDL call (kudos to [~lhofhansl] for coming up with
> this idea). To implement this efficiently, we'd need to also do PHOENIX-2051
> so that we can efficiently find derived views.
> The implementation would rely on an optimistic concurrency model based on
> checking our sequence numbers for each table/view before/after updating. Each
> table/view row would be individually locked for their change (metadata for a
> view or table cannot span regions due to our split policy), with the sequence
> number being incremented under lock and then returned to the client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
