[jira] [Updated] (PHOENIX-4219) Index gets out of sync on HBase 1.x
[ https://issues.apache.org/jira/browse/PHOENIX-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated PHOENIX-4219: -- Attachment: PHOENIX-4219_test.patch > Index gets out of sync on HBase 1.x > --- > > Key: PHOENIX-4219 > URL: https://issues.apache.org/jira/browse/PHOENIX-4219 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.12.0 >Reporter: Vincent Poon > Attachments: PHOENIX-4219_test.patch > > > When writing batches in parallel with multiple background threads, it seems > the index sometimes gets out of sync. This only happens on the master and > 4.x-HBase-1.2. > The tests pass for 4.x-HBase-0.98 > See the attached test, which writes with 2 background threads with batch size > of 100. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (PHOENIX-4219) Index gets out of sync on HBase 1.x
Vincent Poon created PHOENIX-4219: - Summary: Index gets out of sync on HBase 1.x Key: PHOENIX-4219 URL: https://issues.apache.org/jira/browse/PHOENIX-4219 Project: Phoenix Issue Type: Bug Affects Versions: 4.12.0 Reporter: Vincent Poon When writing batches in parallel with multiple background threads, it seems the index sometimes gets out of sync. This only happens on the master and 4.x-HBase-1.2. The tests pass for 4.x-HBase-0.98 See the attached test, which writes with 2 background threads with batch size of 100. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4198) Remove the need for users to have access to the Phoenix SYSTEM tables to create tables
[ https://issues.apache.org/jira/browse/PHOENIX-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172655#comment-16172655 ] Sergey Soldatov commented on PHOENIX-4198: -- There is an another problem. There is no check for the superuser. So if the table was created by user X and hbase user tries to create an index, the query would fail because we check only for the table permissions (which has only user X, but not hbase). > Remove the need for users to have access to the Phoenix SYSTEM tables to > create tables > -- > > Key: PHOENIX-4198 > URL: https://issues.apache.org/jira/browse/PHOENIX-4198 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: namespaces > Fix For: 4.12.0 > > Attachments: PHOENIX-4198.patch, PHOENIX-4198_v2.patch > > > Problem statement:- > A user who doesn't have access to a table should also not be able to modify > Phoenix Metadata. Currently, every user required to have a write permission > to SYSTEM tables which is a security concern as they can > create/alter/drop/corrupt meta data of any other table without proper access > to the corresponding physical tables. > [~devaraj] recommended a solution as below. > 1. A coprocessor endpoint would be implemented and all write accesses to the > catalog table would have to necessarily go through that. The 'hbase' user > would own that table. Today, there is MetaDataEndpointImpl that's run on the > RS where the catalog is hosted, and that could be enhanced to serve the > purpose we need. > 2. The regionserver hosting the catalog table would do the needful for all > catalog updates - creating the mutations as needed, that is. > 3. The coprocessor endpoint could use Ranger to do necessary authorization > checks before updating the catalog table. So for example, if a user doesn't > have authorization to create a table in a certain namespace, or update the > schema, etc., it can reject such requests outright. Only after successful > validations, does it perform the operations (physical operations to do with > creating the table, and updating the catalog table with the necessary > mutations). > 4. In essence, the code that implements dealing with DDLs, would be hosted in > the catalog table endpoint. The client code would be really thin, and it > would just invoke the endpoint with the necessary info. The additional thing > that needs to be done in the endpoint is the validation of authorization to > prevent unauthorized users from making changes to someone else's > tables/schemas/etc. For example, one should be able to create a view on a > table if he has read access on the base table. That mutation on the catalog > table would be permitted. For changing the schema (adding a new column for > example), the said user would need write permission on the table... etc etc. > Thanks [~elserj] for the write-up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4215) Partial index rebuild never complete after PHOENIX-3525 when rebuild period is configured
[ https://issues.apache.org/jira/browse/PHOENIX-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172630#comment-16172630 ] Rajeshbabu Chintaguntla commented on PHOENIX-4215: -- [~jamestaylor] bq. FYI, I have a test and found and fixed a few issues. Will post patch after I clean it up, Rajeshbabu Chintaguntla. Super. Thanks. bq. To write the test, you'd just create your own Clock class and inject it as the other tests are doing. Just set QueryServices.INDEX_FAILURE_HANDLING_REBUILD_PERIOD in the setup to a reasonable value (not large enough to trigger batches in other tests) and UPSERT some rows with timestamps that will cause batching to occur. You can copy/paste other tests there. In the testcase patch uploaded here PHOENIX-4215_testcase.patch setting period to 1000 milli seconds. With this able to reproduce the issue and meeting all the requirements what ever you are saying. As you already said you have a test that's cool. > Partial index rebuild never complete after PHOENIX-3525 when rebuild period > is configured > - > > Key: PHOENIX-4215 > URL: https://issues.apache.org/jira/browse/PHOENIX-4215 > Project: Phoenix > Issue Type: Bug >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Fix For: 4.12.0 > > Attachments: PHOENIX-4215_testcase.patch > > > Currently the default value of phoenix.index.failure.handling.rebuild.period > is long max. When we configure it some thing like an hour or day then partial > index rebuild never complete and the index is never usable until recreate it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4215) Partial index rebuild never complete after PHOENIX-3525 when rebuild period is configured
[ https://issues.apache.org/jira/browse/PHOENIX-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172605#comment-16172605 ] James Taylor commented on PHOENIX-4215: --- FYI, I have a test and found and fixed a few issues. Will post patch after I clean it up, [~rajeshbabu]. > Partial index rebuild never complete after PHOENIX-3525 when rebuild period > is configured > - > > Key: PHOENIX-4215 > URL: https://issues.apache.org/jira/browse/PHOENIX-4215 > Project: Phoenix > Issue Type: Bug >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Fix For: 4.12.0 > > Attachments: PHOENIX-4215_testcase.patch > > > Currently the default value of phoenix.index.failure.handling.rebuild.period > is long max. When we configure it some thing like an hour or day then partial > index rebuild never complete and the index is never usable until recreate it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172588#comment-16172588 ] Andrew Purtell commented on PHOENIX-672: bq. The HBase API in 1.3 doesn't provide the ability to merge permissions, for example if the initial permission was 'RW' and the user grants 'X' then it overrides the original permission to 'X'. It's also fine to document this behavior. This consideration is orthogonal to syntax. Someone is going to either have to specify a weird (for SQL) string containing "RWX", not just "X", or GRANT READ,WRITE,EXECUTE not just GRANT EXECUTE. > Add GRANT and REVOKE commands using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172586#comment-16172586 ] Andrew Purtell commented on PHOENIX-672: bq. To implement similar behavior, we would have to incur an overhead of reading the permissions and writing it back again. I don't see that as a problem. bq. We decided to stick more closely to HBase in terms of functionality and syntax, HBase shell syntax isn't remotely close to SQL by design, but Phoenix statements should be. Wherever possible if we stick to SQL standards or the syntax of widely used systems users will have less trouble adjusting to Phoenix and we will have fewer user questions or complaints. > Add GRANT and REVOKE commands using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4198) Remove the need for users to have access to the Phoenix SYSTEM tables to create tables
[ https://issues.apache.org/jira/browse/PHOENIX-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172574#comment-16172574 ] Sergey Soldatov commented on PHOENIX-4198: -- [~an...@apache.org] there is a problem in kerberos setup. In hasAccess you are checking for the User.getName value, but hbase acl uses short names, so in the index scenario when user has namespace access we are able to create tables, but unable to create index tables (actually physical table is created, but after that fails during the check that user has access to the parent table). > Remove the need for users to have access to the Phoenix SYSTEM tables to > create tables > -- > > Key: PHOENIX-4198 > URL: https://issues.apache.org/jira/browse/PHOENIX-4198 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: namespaces > Fix For: 4.12.0 > > Attachments: PHOENIX-4198.patch, PHOENIX-4198_v2.patch > > > Problem statement:- > A user who doesn't have access to a table should also not be able to modify > Phoenix Metadata. Currently, every user required to have a write permission > to SYSTEM tables which is a security concern as they can > create/alter/drop/corrupt meta data of any other table without proper access > to the corresponding physical tables. > [~devaraj] recommended a solution as below. > 1. A coprocessor endpoint would be implemented and all write accesses to the > catalog table would have to necessarily go through that. The 'hbase' user > would own that table. Today, there is MetaDataEndpointImpl that's run on the > RS where the catalog is hosted, and that could be enhanced to serve the > purpose we need. > 2. The regionserver hosting the catalog table would do the needful for all > catalog updates - creating the mutations as needed, that is. > 3. The coprocessor endpoint could use Ranger to do necessary authorization > checks before updating the catalog table. So for example, if a user doesn't > have authorization to create a table in a certain namespace, or update the > schema, etc., it can reject such requests outright. Only after successful > validations, does it perform the operations (physical operations to do with > creating the table, and updating the catalog table with the necessary > mutations). > 4. In essence, the code that implements dealing with DDLs, would be hosted in > the catalog table endpoint. The client code would be really thin, and it > would just invoke the endpoint with the necessary info. The additional thing > that needs to be done in the endpoint is the validation of authorization to > prevent unauthorized users from making changes to someone else's > tables/schemas/etc. For example, one should be able to create a view on a > table if he has read access on the base table. That mutation on the catalog > table would be permitted. For changing the schema (adding a new column for > example), the said user would need write permission on the table... etc etc. > Thanks [~elserj] for the write-up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[
https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172511#comment-16172511
]
Hadoop QA commented on PHOENIX-4007:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12887972/PHOENIX-4007_v7.patch
against master branch at commit 3fb104ee56518ca066672e199bff1ed4f4bd9a7e.
ATTACHMENT ID: 12887972
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
++ " ( k INTEGER, c1.a bigint,c2.b bigint CONSTRAINT pk
PRIMARY KEY (k)) GUIDE_POSTS_WIDTH = 0");
++ " ( k INTEGER, c1.a bigint,c2.b bigint CONSTRAINT pk
PRIMARY KEY (k)) GUIDE_POSTS_WIDTH=2000");
++ " (k INTEGER PRIMARY KEY, c1.a bigint, c2.b
bigint) GUIDE_POSTS_WIDTH="
++ " (orgId CHAR(15) NOT NULL, pk2 integer NOT NULL,
c1.a bigint, c2.b bigint CONSTRAINT PK PRIMARY KEY (ORGID, PK2))
MULTI_TENANT=true, GUIDE_POSTS_WIDTH="
+"CLIENT 1-CHUNK 0 ROWS 0 BYTES PARALLEL 1-WAY FULL SCAN OVER "
+ physicalTableName + "\n" +
++ " ( k INTEGER, c1.a bigint,c2.b bigint CONSTRAINT pk
PRIMARY KEY (k)) GUIDE_POSTS_WIDTH=2000");
+String q = "SELECT SUM(GUIDE_POSTS_ROW_COUNT) FROM SYSTEM.STATS
WHERE PHYSICAL_NAME = '" + tableName + "'" + " AND COLUMN_FAMILY = '%s'";
+assertEquals("Number of expected rows in stats table for column
family c1 didn't match", 10, rs.getInt(1));
+assertEquals("Number of expected rows in stats table for column
family c2 didn't match", 10, rs.getInt(1));
+|| (lhsPlan.getEstimatedRowsToScan() == null ||
rhsPlan.getEstimatedRowsToScan() == null)
{color:red}-1 core tests{color}. The patch failed these unit tests:
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ViewIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.QueryWithOffsetIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AggregateIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ArrayIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1460//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1460//console
This message is automatically generated.
> Surface time at which byte/row estimate information was computed in explain
> plan output
> ---
>
> Key: PHOENIX-4007
> URL: https://issues.apache.org/jira/browse/PHOENIX-4007
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
> Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch,
> PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch,
> PHOENIX-4007_v7.patch
>
>
> As part of PHOENIX-3822, we surfaced byte and row estimates for queries in
> explain plan. Since we collect this information through stats collection, it
> would also be helpful to surface when this information was last updated to
> reflect its freshness. We already store last_stats_update_time in
> SYSTEM.STATS. So the task would be essentially surfacing
> last_stats_update_time as another column in the explain plan result set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172493#comment-16172493 ] Ethan Wang edited comment on PHOENIX-672 at 9/19/17 11:26 PM: -- Question, so for GRANT 'user0', 'RX' Will it grant user0 RX for all schemas and tables? +1 [~apurtell] Both MySQL and Postgres grant on SQL commands, such as SELECT, CREATE etc, I assume the "RWXC" in your purpose is about the same [~karanmehta93]? was (Author: aertoria): +1 [~apurtell] Both MySQL and Postgres grant on SQL commands, such as SELECT, CREATE etc, I assume the "RWXC" in your purpose is about the same [~karanmehta93]? > Add GRANT and REVOKE commands using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172493#comment-16172493 ] Ethan Wang commented on PHOENIX-672: +1 [~apurtell] Both MySQL and Postgres grant on SQL commands, such as SELECT, CREATE etc, I assume the "RWXC" in your purpose is about the same [~karanmehta93]? > Add GRANT and REVOKE commands using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172469#comment-16172469 ] Andrew Purtell commented on PHOENIX-672: [~karanmehta93] I prefer we adopt syntax from either Postgres or MySQL. Inventing something new for Phoenix won't serve anyone especially well. Postgres GRANT: https://www.postgresql.org/docs/8.0/static/sql-grant.html REVOKE: https://www.postgresql.org/docs/8.0/static/sql-revoke.html MySQL GRANT: https://dev.mysql.com/doc/refman/5.7/en/grant.html REVOKE: https://dev.mysql.com/doc/refman/5.7/en/revoke.html Adapt one of these with the minimum necessary changes. I like the Postgres syntax better, FWIW > Add GRANT and REVOKE commands using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[
https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172409#comment-16172409
]
Karan Mehta edited comment on PHOENIX-672 at 9/19/17 10:42 PM:
---
The following grammar will be used for {{GRANT}}
{code}GRANT 'userOrGroupName', 'permissionString' ON [TABLE | SCHEMA] 'param1',
'param2', 'param3' {code}
To specify a group instead of a user the first parameter has to started with
"@".
Permission String can contain characters {{RWXCA}} case insensitive.
If all the permissions are general for the user, then the second part is not
needed else the following holds.
Token {{ON}} is required
For the next parameter, if nothing is specified, it defaults to table. For
schema, we need to explicitly use the token {{SCHEMA}}.
For schema, it will be followed by a single parameter which is schema name
For table, it will be followed by Table name and param2/3 will be optional for
Column Family and Column Qualifier
Examples
{code}
GRANT 'user0', 'RX'
GRANT 'user1', 'RWX' ON 'table1'
GRANT '@group2', 'RC' ON 'table2', 'cf1'
GRANT 'user3', 'R' ON SCHEMA 'schema1'
{code}
Similar goes for {{REVOKE}} as well
[~jamestaylor] [~apurtell] Please advice.
[~twdsi...@gmail.com] FYI.
was (Author: karanmehta93):
The following grammar will be used for {{GRANT}}
{code}GRANT 'userOrGroupName', 'permissionString' ON [TABLE | SCHEMA] 'param1',
'param2', 'param3' {code}
To specify a group instead of a user the first parameter has to started with
"@".
Permission String can contain characters {{RWXCA}} case insensitive.
Token {{ON}} is required
For the next parameter, if nothing is specified, it defaults to table. For
schema, we need to explicitly use the token {{SCHEMA}}.
For schema, it will be followed by a single parameter which is schema name
For table, it will be followed by Table name and param2/3 will be optional for
Column Family and Column Qualifier
Examples
{code}
GRANT 'user1', 'RWX' ON 'table1'
GRANT '@group2', 'RC' ON 'table2', 'cf1'
GRANT 'user3', 'R' ON SCHEMA 'schema1'
{code}
Similar goes for {{REVOKE}} as well
[~jamestaylor] [~apurtell] Please advice.
[~twdsi...@gmail.com] FYI.
> Add GRANT and REVOKE commands using HBase AccessController
> --
>
> Key: PHOENIX-672
> URL: https://issues.apache.org/jira/browse/PHOENIX-672
> Project: Phoenix
> Issue Type: Task
>Reporter: James Taylor
>Assignee: Karan Mehta
> Labels: gsoc2016, security
>
> In HBase 0.98, cell-level security will be available. Take a look at
> [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security)
> excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add
> support for security to our SQL grammar.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4212) Disallow DML operations on connections with CURRENT_SCN set - DerivedTableIT
[
https://issues.apache.org/jira/browse/PHOENIX-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172455#comment-16172455
]
Hadoop QA commented on PHOENIX-4212:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12887825/PHOENIX-4212_v3.patch
against master branch at commit 3fb104ee56518ca066672e199bff1ed4f4bd9a7e.
ATTACHMENT ID: 12887825
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+"CREATE INDEX "+dynamicTableName+"_DERIVED_IDX ON
"+dynamicTableName+" (a_byte) INCLUDE (A_STRING, B_STRING)"
+"SERVER AGGREGATE INTO DISTINCT ROWS BY
[\"A_STRING\", \"B_STRING\"]\n" +
+String ddl = "CREATE INDEX IF NOT EXISTS "+tableName+"_DERIVED_IDX
ON "+tableName+" (a_byte) INCLUDE (A_STRING, B_STRING)";
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1462//testReport/
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1462//console
This message is automatically generated.
> Disallow DML operations on connections with CURRENT_SCN set - DerivedTableIT
>
>
> Key: PHOENIX-4212
> URL: https://issues.apache.org/jira/browse/PHOENIX-4212
> Project: Phoenix
> Issue Type: Bug
>Reporter: Ethan Wang
>Assignee: Ethan Wang
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4212.patch, PHOENIX-4212_v2.patch,
> PHOENIX-4212_v3.patch
>
>
> Disallow DML operations on connections with CURRENT_SCN set - DerivedTableIT
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[
https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172409#comment-16172409
]
Karan Mehta commented on PHOENIX-672:
-
The following grammar will be used for {{GRANT}}
{code}GRANT 'userOrGroupName', 'permissionString' ON [TABLE | SCHEMA] 'param1',
'param2', 'param3' {code}
To specify a group instead of a user the first parameter has to started with
"@".
Permission String can contain characters {{RWXCA}} case insensitive.
Token {{ON}} is required
For the next parameter, if nothing is specified, it defaults to table. For
schema, we need to explicitly use the token {{SCHEMA}}.
For schema, it will be followed by a single parameter which is schema name
For table, it will be followed by Table name and param2/3 will be optional for
Column Family and Column Qualifier
Examples
{code}
GRANT 'user1', 'RWX' ON 'table1'
GRANT '@group2', 'RC' ON 'table2', 'cf1'
GRANT 'user3', 'R' ON SCHEMA 'schema1'
{code}
Similar goes for {{REVOKE}} as well
[~jamestaylor] [~apurtell] Please advice.
[~twdsi...@gmail.com] FYI.
> Add GRANT and REVOKE commands using HBase AccessController
> --
>
> Key: PHOENIX-672
> URL: https://issues.apache.org/jira/browse/PHOENIX-672
> Project: Phoenix
> Issue Type: Task
>Reporter: James Taylor
>Assignee: Karan Mehta
> Labels: gsoc2016, security
>
> In HBase 0.98, cell-level security will be available. Take a look at
> [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security)
> excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add
> support for security to our SQL grammar.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] phoenix pull request #273: Add equals verification test
GitHub user TheRealHaui opened a pull request: https://github.com/apache/phoenix/pull/273 Add equals verification test Adds an equals verification Unit Test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/TheRealHaui/phoenix add-equals-verification-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/273.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #273 ---
[jira] [Updated] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[ https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samarth Jain updated PHOENIX-4007: -- Attachment: PHOENIX-4007_v7.patch > Surface time at which byte/row estimate information was computed in explain > plan output > --- > > Key: PHOENIX-4007 > URL: https://issues.apache.org/jira/browse/PHOENIX-4007 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Samarth Jain > Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch, > PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch, > PHOENIX-4007_v7.patch > > > As part of PHOENIX-3822, we surfaced byte and row estimates for queries in > explain plan. Since we collect this information through stats collection, it > would also be helpful to surface when this information was last updated to > reflect its freshness. We already store last_stats_update_time in > SYSTEM.STATS. So the task would be essentially surfacing > last_stats_update_time as another column in the explain plan result set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[ https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samarth Jain updated PHOENIX-4007: -- Attachment: (was: PHOENIX-4007_v7.patch) > Surface time at which byte/row estimate information was computed in explain > plan output > --- > > Key: PHOENIX-4007 > URL: https://issues.apache.org/jira/browse/PHOENIX-4007 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Samarth Jain > Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch, > PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch, > PHOENIX-4007_v7.patch > > > As part of PHOENIX-3822, we surfaced byte and row estimates for queries in > explain plan. Since we collect this information through stats collection, it > would also be helpful to surface when this information was last updated to > reflect its freshness. We already store last_stats_update_time in > SYSTEM.STATS. So the task would be essentially surfacing > last_stats_update_time as another column in the explain plan result set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4198) Remove the need for users to have access to the Phoenix SYSTEM tables to create tables
[ https://issues.apache.org/jira/browse/PHOENIX-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172319#comment-16172319 ] Thomas D'Silva commented on PHOENIX-4198: - bq. Actually, if ACL access is not enabled, we do it with the earlier logic(region.mutateRowsWithLocks) where region server acquiring read locks again during mutation. Not sure why we had required this with a single region of the system.catalog table as we were already taking lock before, does it ensure that all rows are written atomically which Htable.batch will not? are there any consequences of writing set of mutations with HTable.Batch which is not there in region.mutateRowsWithLocks? Since we already acquire the required locks before calling HTable.Batch would that be atomic as well ? If a user has create access on the namespace can he create a view, or does he also need read access on the data table to create the view? I think for creating an index a user should have create access for the schema/namespace. While creating an index users that have read access on the data table should also be granted read access on the index. Users that have write access on the table should be granted write access on the index, and for mutable indexes they should be given execute access on the data table (so that the index metadata can be sent to the server). [~karanmehta93] is working on PHOENIX-672 which can be used to grant users permissions to the index when permissions are granted on the data table. > Remove the need for users to have access to the Phoenix SYSTEM tables to > create tables > -- > > Key: PHOENIX-4198 > URL: https://issues.apache.org/jira/browse/PHOENIX-4198 > Project: Phoenix > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal > Labels: namespaces > Fix For: 4.12.0 > > Attachments: PHOENIX-4198.patch, PHOENIX-4198_v2.patch > > > Problem statement:- > A user who doesn't have access to a table should also not be able to modify > Phoenix Metadata. Currently, every user required to have a write permission > to SYSTEM tables which is a security concern as they can > create/alter/drop/corrupt meta data of any other table without proper access > to the corresponding physical tables. > [~devaraj] recommended a solution as below. > 1. A coprocessor endpoint would be implemented and all write accesses to the > catalog table would have to necessarily go through that. The 'hbase' user > would own that table. Today, there is MetaDataEndpointImpl that's run on the > RS where the catalog is hosted, and that could be enhanced to serve the > purpose we need. > 2. The regionserver hosting the catalog table would do the needful for all > catalog updates - creating the mutations as needed, that is. > 3. The coprocessor endpoint could use Ranger to do necessary authorization > checks before updating the catalog table. So for example, if a user doesn't > have authorization to create a table in a certain namespace, or update the > schema, etc., it can reject such requests outright. Only after successful > validations, does it perform the operations (physical operations to do with > creating the table, and updating the catalog table with the necessary > mutations). > 4. In essence, the code that implements dealing with DDLs, would be hosted in > the catalog table endpoint. The client code would be really thin, and it > would just invoke the endpoint with the necessary info. The additional thing > that needs to be done in the endpoint is the validation of authorization to > prevent unauthorized users from making changes to someone else's > tables/schemas/etc. For example, one should be able to create a view on a > table if he has read access on the base table. That mutation on the catalog > table would be permitted. For changing the schema (adding a new column for > example), the said user would need write permission on the table... etc etc. > Thanks [~elserj] for the write-up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-672) Add GRANT and REVOKE commands using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-672: --- Summary: Add GRANT and REVOKE commands using HBase AccessController (was: Add SQL-ish security features using HBase AccessController) > Add GRANT and REVOKE commands using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (PHOENIX-672) Add SQL-ish security features using HBase AccessController
[ https://issues.apache.org/jira/browse/PHOENIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva reassigned PHOENIX-672: -- Assignee: Karan Mehta (was: Andrew Purtell) > Add SQL-ish security features using HBase AccessController > -- > > Key: PHOENIX-672 > URL: https://issues.apache.org/jira/browse/PHOENIX-672 > Project: Phoenix > Issue Type: Task >Reporter: James Taylor >Assignee: Karan Mehta > Labels: gsoc2016, security > > In HBase 0.98, cell-level security will be available. Take a look at > [this](https://communities.intel.com/community/datastack/blog/2013/10/29/hbase-cell-security) > excellent blog post by @apurtell. Once Phoenix works on 0.96, we should add > support for security to our SQL grammar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[
https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172214#comment-16172214
]
Hadoop QA commented on PHOENIX-4007:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12887951/PHOENIX-4007_v7.patch
against master branch at commit 3fb104ee56518ca066672e199bff1ed4f4bd9a7e.
ATTACHMENT ID: 12887951
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail .
Compilation errors resume:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on
project phoenix-core: Failed to clean project: Failed to delete
/home/jenkins/jenkins-slave/workspace/PreCommit-PHOENIX-Build/phoenix-core/target
-> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :phoenix-core
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1458//console
This message is automatically generated.
> Surface time at which byte/row estimate information was computed in explain
> plan output
> ---
>
> Key: PHOENIX-4007
> URL: https://issues.apache.org/jira/browse/PHOENIX-4007
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Samarth Jain
> Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch,
> PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch,
> PHOENIX-4007_v7.patch
>
>
> As part of PHOENIX-3822, we surfaced byte and row estimates for queries in
> explain plan. Since we collect this information through stats collection, it
> would also be helpful to surface when this information was last updated to
> reflect its freshness. We already store last_stats_update_time in
> SYSTEM.STATS. So the task would be essentially surfacing
> last_stats_update_time as another column in the explain plan result set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[ https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samarth Jain updated PHOENIX-4007: -- Attachment: PHOENIX-4007_v7.patch > Surface time at which byte/row estimate information was computed in explain > plan output > --- > > Key: PHOENIX-4007 > URL: https://issues.apache.org/jira/browse/PHOENIX-4007 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Samarth Jain > Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch, > PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch, > PHOENIX-4007_v7.patch > > > As part of PHOENIX-3822, we surfaced byte and row estimates for queries in > explain plan. Since we collect this information through stats collection, it > would also be helpful to surface when this information was last updated to > reflect its freshness. We already store last_stats_update_time in > SYSTEM.STATS. So the task would be essentially surfacing > last_stats_update_time as another column in the explain plan result set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4138) Create a hard limit on number of indexes per table
[ https://issues.apache.org/jira/browse/PHOENIX-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172184#comment-16172184 ] Andrew Purtell commented on PHOENIX-4138: - lgtm What do you think [~giacomotaylor]? If no objections I can commit > Create a hard limit on number of indexes per table > -- > > Key: PHOENIX-4138 > URL: https://issues.apache.org/jira/browse/PHOENIX-4138 > Project: Phoenix > Issue Type: Bug >Reporter: Rahul Shrivastava >Assignee: churro morales > Attachments: PHOENIX-4138.patch, PHOENIX-4138.v1.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > There should be a config parameter to impose a hard limit on number of > indexes per table. There is a SQL Exception > https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/exception/SQLExceptionCode.java#L260 > , but it gets triggered on the server side > (https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L1589) > . > We need a client side limit that can be configured via Phoenix config > parameter. Something like if user create more than lets say 30 indexes per > table, it would not allow more index creation for the that specific table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4178) Detect failed index write while rebuilder is running with index staying active
[ https://issues.apache.org/jira/browse/PHOENIX-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172159#comment-16172159 ] Geoffrey Jacoby commented on PHOENIX-4178: -- [~jamestaylor] - does this only apply to mutable indexes? > Detect failed index write while rebuilder is running with index staying active > -- > > Key: PHOENIX-4178 > URL: https://issues.apache.org/jira/browse/PHOENIX-4178 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.12.0 > > > If an index write fails while the rebuilder is running, we need to make sure > to run the rebuilder again so we don't need miss any rows. This can only > happen when the index is left active, as otherwise the index will transition > back to disabled and we'll end up rerunning the rebuilder. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[ https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samarth Jain updated PHOENIX-4007: -- Attachment: (was: PHOENIX-4007_v7.patch) > Surface time at which byte/row estimate information was computed in explain > plan output > --- > > Key: PHOENIX-4007 > URL: https://issues.apache.org/jira/browse/PHOENIX-4007 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Samarth Jain > Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch, > PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch > > > As part of PHOENIX-3822, we surfaced byte and row estimates for queries in > explain plan. Since we collect this information through stats collection, it > would also be helpful to surface when this information was last updated to > reflect its freshness. We already store last_stats_update_time in > SYSTEM.STATS. So the task would be essentially surfacing > last_stats_update_time as another column in the explain plan result set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4215) Partial index rebuild never complete after PHOENIX-3525 when rebuild period is configured
[
https://issues.apache.org/jira/browse/PHOENIX-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172155#comment-16172155
]
James Taylor commented on PHOENIX-4215:
---
[~rajeshbabu] - thanks for the info. I have a fix that's part of PHOENIX-4178
(will post patch shortly) in which I go through the regular
MetaDataEndPointImpl.updateIndexState() when setting the
INDEX_DISABLE_TIMESTAMP in this case too. If you could write a test in
PartialIndexRebuilderIT, that'd be appreciated. Manual testing is good, but it
doesn't prevent regressions.
To write the test, you'd just create your own Clock class and inject it as the
other tests are doing. Just set
{{QueryServices.INDEX_FAILURE_HANDLING_REBUILD_PERIOD}} in the setup to a
reasonable value (not large enough to trigger batches in other tests) and
UPSERT some rows with timestamps that will cause batching to occur. You can
copy/paste other tests there.
We've done a lot of work stabilizing mutable indexing and writing tests was a
large part of it. It's not the most sexy work, but it's important. Appreciate
your efforts here.
> Partial index rebuild never complete after PHOENIX-3525 when rebuild period
> is configured
> -
>
> Key: PHOENIX-4215
> URL: https://issues.apache.org/jira/browse/PHOENIX-4215
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4215_testcase.patch
>
>
> Currently the default value of phoenix.index.failure.handling.rebuild.period
> is long max. When we configure it some thing like an hour or day then partial
> index rebuild never complete and the index is never usable until recreate it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (PHOENIX-4007) Surface time at which byte/row estimate information was computed in explain plan output
[ https://issues.apache.org/jira/browse/PHOENIX-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samarth Jain updated PHOENIX-4007: -- Attachment: PHOENIX-4007_v7.patch Updated patch. Now, with this patch, we take care of three scenarios in the following way: 1) We differentiate between stats not being there because collection didn't run or because guide_posts_width was set to 0 - estimates are reported as null 2) When there is no data and stats collection was run - estimates are reported as 0 3) When data is present, but it's size is smaller than guide post width - we now report the actual number of rows and data size instead of reporting them as 0. [~jamestaylor] - please review. > Surface time at which byte/row estimate information was computed in explain > plan output > --- > > Key: PHOENIX-4007 > URL: https://issues.apache.org/jira/browse/PHOENIX-4007 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain >Assignee: Samarth Jain > Attachments: PHOENIX-4007_v1.patch, PHOENIX-4007_v2.patch, > PHOENIX-4007_v3.patch, PHOENIX-4007_v4.patch, PHOENIX-4007_v6.patch, > PHOENIX-4007_v7.patch > > > As part of PHOENIX-3822, we surfaced byte and row estimates for queries in > explain plan. Since we collect this information through stats collection, it > would also be helpful to surface when this information was last updated to > reflect its freshness. We already store last_stats_update_time in > SYSTEM.STATS. So the task would be essentially surfacing > last_stats_update_time as another column in the explain plan result set. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PHOENIX-4178) Detect failed index write while rebuilder is running with index staying active
[ https://issues.apache.org/jira/browse/PHOENIX-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4178: -- Fix Version/s: 4.12.0 > Detect failed index write while rebuilder is running with index staying active > -- > > Key: PHOENIX-4178 > URL: https://issues.apache.org/jira/browse/PHOENIX-4178 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor > Fix For: 4.12.0 > > > If an index write fails while the rebuilder is running, we need to make sure > to run the rebuilder again so we don't need miss any rows. This can only > happen when the index is left active, as otherwise the index will transition > back to disabled and we'll end up rerunning the rebuilder. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4215) Partial index rebuild never complete after PHOENIX-3525 when rebuild period is configured
[
https://issues.apache.org/jira/browse/PHOENIX-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171999#comment-16171999
]
Rajeshbabu Chintaguntla commented on PHOENIX-4215:
--
The reason for the partial index rebuilding not moving forward because
PhoenixRuntime#getTableNoCache always get the index from the server cache but
not scanned from system catalog directly and one more thing is for updating
disable timestamp wit HRegion#checkAndPut which doesn't invalidate the cache
and server so we always see the same disable timestamp for index. Because of
this scanBeginTime and scanEndTime for index rebuild are same in each iterator
that's why we won't move forward and after some time of threshold we go ahead
and disable the index permanently.
{noformat}
PTable indexPTable = PhoenixRuntime.getTableNoCache(conn,
indexTableFullName);
{noformat}
{noformat}
private static boolean updateDisableTimestamp(PhoenixConnection conn,
String indexTableName,
long disabledTimestamp, HTableInterface metaTable, long
expectedDisabledTimestamp) throws IOException {
byte[] indexTableKey =
SchemaUtil.getTableKeyFromFullName(indexTableName);
Put put = new Put(indexTableKey);
put.addColumn(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES,
PLong.INSTANCE.toBytes(disabledTimestamp));
return metaTable.checkAndPut(indexTableKey,
PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES, CompareOp.EQUAL,
PLong.INSTANCE.toBytes(expectedDisabledTimestamp),
put);
}
{noformat}
> Partial index rebuild never complete after PHOENIX-3525 when rebuild period
> is configured
> -
>
> Key: PHOENIX-4215
> URL: https://issues.apache.org/jira/browse/PHOENIX-4215
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 4.12.0
>
> Attachments: PHOENIX-4215_testcase.patch
>
>
> Currently the default value of phoenix.index.failure.handling.rebuild.period
> is long max. When we configure it some thing like an hour or day then partial
> index rebuild never complete and the index is never usable until recreate it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (PHOENIX-4021) Remove CachingHTableFactory
[ https://issues.apache.org/jira/browse/PHOENIX-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171811#comment-16171811 ] Kang Yuan edited comment on PHOENIX-4021 at 9/19/17 2:50 PM: - we backport this patch to phoenix4.11-HBase-0.98 .The performance go down a lot when CachineHTableFactory is disabled, so many HTables were been created and used once.It's waste and lower the performance can we make CachingHTableFactory thread safe instead of removing it was (Author: kirayuan): we backport this patch to phoenix4.11-HBase-0.98 .The performance go down a lot when CachineHTableFactory is disabled, so many HTables were been created and used once.It's wast and lower the performance can we make CachingHTableFactory thread safe instead of removing it > Remove CachingHTableFactory > --- > > Key: PHOENIX-4021 > URL: https://issues.apache.org/jira/browse/PHOENIX-4021 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.11.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Labels: globalMutableSecondaryIndex > Fix For: 4.12.0 > > Attachments: PHOENIX-4021.patch > > > CachingHTableFactory is used as a performance optimization when writing to > global indexes so that HTable instances are cached and later automatically > cleaned up, rather than instantiated each time we write to an index. > This should be removed for two reasons: > 1. It opens us up to race conditions, because HTables aren't threadsafe, but > CachingHTableFactory doesn't guard against two threads both grabbing the same > HTable and using it simultaneously. Since all ops going through a region > share the same IndexWriter and ParallelWriterIndexCommitter, and hence the > same CachingHTableFactory, that means separate operations can both be holding > the same HTable. > 2. According to discussion on PHOENIX-3159, and offline discussions I've had > with [~apurtell], HBase 1.x and above make creating throwaway HTable > instances cheap so the caching is no longer needed. > For 4.x-HBase-1.x and master, we should remove CachingHTableFactory, and for > 4.x-HBase-0.98, we should either get rid of it (if it's not too much of a > perf hit) or at least make it threadsafe. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4021) Remove CachingHTableFactory
[ https://issues.apache.org/jira/browse/PHOENIX-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171811#comment-16171811 ] Kang Yuan commented on PHOENIX-4021: we backport this patch to phoenix4.11-HBase-0.98 .The performance go down a lot when CachineHTableFactory is disabled, so many HTables were been created and used once.It's wast and lower the performance can we make CachingHTableFactory thread safe instead of removing it > Remove CachingHTableFactory > --- > > Key: PHOENIX-4021 > URL: https://issues.apache.org/jira/browse/PHOENIX-4021 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.11.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Labels: globalMutableSecondaryIndex > Fix For: 4.12.0 > > Attachments: PHOENIX-4021.patch > > > CachingHTableFactory is used as a performance optimization when writing to > global indexes so that HTable instances are cached and later automatically > cleaned up, rather than instantiated each time we write to an index. > This should be removed for two reasons: > 1. It opens us up to race conditions, because HTables aren't threadsafe, but > CachingHTableFactory doesn't guard against two threads both grabbing the same > HTable and using it simultaneously. Since all ops going through a region > share the same IndexWriter and ParallelWriterIndexCommitter, and hence the > same CachingHTableFactory, that means separate operations can both be holding > the same HTable. > 2. According to discussion on PHOENIX-3159, and offline discussions I've had > with [~apurtell], HBase 1.x and above make creating throwaway HTable > instances cheap so the caching is no longer needed. > For 4.x-HBase-1.x and master, we should remove CachingHTableFactory, and for > 4.x-HBase-0.98, we should either get rid of it (if it's not too much of a > perf hit) or at least make it threadsafe. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PHOENIX-4027) Mark index as disabled during partial rebuild after configurable amount of time
[
https://issues.apache.org/jira/browse/PHOENIX-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171208#comment-16171208
]
Rajeshbabu Chintaguntla commented on PHOENIX-4027:
--
[~jamestaylor] I ran the tests and they are passing with the addendum. What
about increasing the default threshold to 1 hour(currently 30min) or more than
it because sometimes fixing the HBase inconsistencies might take more time.
Sometimes rebuilding index also take time.
> Mark index as disabled during partial rebuild after configurable amount of
> time
> ---
>
> Key: PHOENIX-4027
> URL: https://issues.apache.org/jira/browse/PHOENIX-4027
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: Samarth Jain
> Fix For: 4.12.0, 4.11.1
>
> Attachments: PHOENIX-4027_addendum_2.patch,
> PHOENIX-4027_addendum.patch, PHOENIX-4027.patch
>
>
> Instead of marking an index as permanently disabled in the partial index
> rebuilder when a failure occurs, we should let it try again up to a
> configurable amount of time. The reason is that the fail-fast approach with
> the lower RPC timeout will continue to cause a failure until the index region
> can be written to. This will allow us to ride out region moves without a long
> RPC time out and thus without holding handler threads for long periods of
> time. We can base the failure on the INDEX_DISABLE_TIMESTAMP value of an
> index as we walk through the scan results here in MetaDataRegionObserver. :
> {code}
> do {
> results.clear();
> hasMore = scanner.next(results);
> if (results.isEmpty()) break;
> Result r = Result.create(results);
> byte[] disabledTimeStamp =
> r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
>
> PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES);
> byte[] indexState =
> r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
> PhoenixDatabaseMetaData.INDEX_STATE_BYTES);
> if (disabledTimeStamp == null || disabledTimeStamp.length
> == 0) {
> continue;
> }
> // TODO: if disabledTimeStamp -
> System.currentTimeMillis() > configurableAmount
> // then disable the index.
> {code}
> I'd propose we allow 30 minutes to get an index back online.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
