[jira] [Updated] (PHOENIX-5837) Table Level Phoenix Metrics Implementation.
[ https://issues.apache.org/jira/browse/PHOENIX-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vikas meka updated PHOENIX-5837: Description: currently GlobalClient Metrics provide aggregated information about the usage of the application in-terms of total no. of reads, total no. of writes, total number of failures, total no.of failures etc. In some scenarios these counters do not provide much useful insights into Phoenix usage pattern. The task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics which would be helpful in analyzing the reads/writes/deletes on a table. (was: currently GlobalClient Metrics provide aggregated information about the usage of the application in-terms of total no. of reads, total no. of writes, total number of failures, total no.of failures etc. In some scenarios these counters do not provide much useful insights into Phoenix usage pattern. The task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics which would be helpful in analyzing the reads/writes/deletes on a table.) > Table Level Phoenix Metrics Implementation. > --- > > Key: PHOENIX-5837 > URL: https://issues.apache.org/jira/browse/PHOENIX-5837 > Project: Phoenix > Issue Type: Task >Reporter: vikas meka >Priority: Major > Labels: metric-collector, metrics > > currently GlobalClient Metrics provide aggregated information about the usage > of the application in-terms of total no. of reads, total no. of writes, total > number of failures, total no.of failures etc. In some scenarios these > counters do not provide much useful insights into Phoenix usage pattern. The > task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics > which would be helpful in analyzing the reads/writes/deletes on a table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5839) CompatUtil#setStopRow semantics problem
[ https://issues.apache.org/jira/browse/PHOENIX-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated PHOENIX-5839: - Description: As noticed by [~gjacoby] , the semantics of the CompatUtil#setStopRow method are incorrect for HBase 1.3. Specifically, we invert the semantics of the *inclusive* flag. Due to the quirks of the HBase 1.3 scan semantics, the resulting behaviour in the specific cases where it used is correct (in fact this behaviour is needed for correct operation), but when used outside this specific use case (single row scan), it would cause problems. Find some other solution that solves the backwards compatibility problem, and does not invert the semantics of the call in the range (not single row) scan case. was: As noticed by [~gjacoby] , the semantics of the CompatUtil#setStopRow method are incorrect for HBase 1.3 Due to the quirks of the HBase 1.3 scan semantics, the resulting behaviour in the specific cases where it used is correct (in fact this behaviour is needed for correct operation), but when used outside this specific use case (single row) scan, it would cause problems. Find some other solution that solves the backwards compatibility problem, and does not invert the semantics of the call in the range (not single row) scan case. > CompatUtil#setStopRow semantics problem > --- > > Key: PHOENIX-5839 > URL: https://issues.apache.org/jira/browse/PHOENIX-5839 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.x >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > As noticed by [~gjacoby] , the semantics of the CompatUtil#setStopRow method > are incorrect for HBase 1.3. Specifically, we invert the semantics of the > *inclusive* flag. > Due to the quirks of the HBase 1.3 scan semantics, the resulting behaviour in > the specific cases where it used is correct (in fact this behaviour is needed > for correct operation), but when used outside this specific use case (single > row scan), it would cause problems. > Find some other solution that solves the backwards compatibility problem, and > does not invert the semantics of the call in the range (not single row) scan > case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5839) CompatUtil#setStopRow semantics problem
Istvan Toth created PHOENIX-5839: Summary: CompatUtil#setStopRow semantics problem Key: PHOENIX-5839 URL: https://issues.apache.org/jira/browse/PHOENIX-5839 Project: Phoenix Issue Type: Bug Components: core Affects Versions: 4.x Reporter: Istvan Toth Assignee: Istvan Toth As noticed by [~gjacoby] , the semantics of the CompatUtil#setStopRow method are incorrect for HBase 1.3 Due to the quirks of the HBase 1.3 scan semantics, the resulting behaviour in the specific cases where it used is correct (in fact this behaviour is needed for correct operation), but when used outside this specific use case (single row) scan, it would cause problems. Find some other solution that solves the backwards compatibility problem, and does not invert the semantics of the call in the range (not single row) scan case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5793) Support parallel init and fast null return for SortMergeJoinPlan.
[ https://issues.apache.org/jira/browse/PHOENIX-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Feng updated PHOENIX-5793: --- Attachment: (was: PHOENIX-5793-v5.patch) > Support parallel init and fast null return for SortMergeJoinPlan. > - > > Key: PHOENIX-5793 > URL: https://issues.apache.org/jira/browse/PHOENIX-5793 > Project: Phoenix > Issue Type: Improvement >Reporter: Chen Feng >Assignee: Chen Feng >Priority: Minor > Attachments: PHOENIX-5793-v2.patch, PHOENIX-5793-v3.patch, > PHOENIX-5793-v4.patch > > > For a join sql like A join B. The implementation of SortMergeJoinPlan > currently inits the two iterators A and B one by one. > By initializing A and B in parallel, we can improve performance in two > aspects. > 1) By overlapping the time in initializing. > 2) If one child query is null, the other child query can be canceled since > the final result must be null. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5838) Add HIstograms for Global Phoenix Metrics and Table level Metrics.
[ https://issues.apache.org/jira/browse/PHOENIX-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vikas meka updated PHOENIX-5838: Labels: metric-collector metrics (was: ) > Add HIstograms for Global Phoenix Metrics and Table level Metrics. > -- > > Key: PHOENIX-5838 > URL: https://issues.apache.org/jira/browse/PHOENIX-5838 > Project: Phoenix > Issue Type: Sub-task >Reporter: vikas meka >Priority: Major > Labels: metric-collector, metrics > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5838) Add HIstograms for Global Phoenix Metrics and Table level Metrics.
vikas meka created PHOENIX-5838: --- Summary: Add HIstograms for Global Phoenix Metrics and Table level Metrics. Key: PHOENIX-5838 URL: https://issues.apache.org/jira/browse/PHOENIX-5838 Project: Phoenix Issue Type: Sub-task Reporter: vikas meka -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5837) Table Level Phoenix Metrics Implementation.
[ https://issues.apache.org/jira/browse/PHOENIX-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vikas meka updated PHOENIX-5837: Labels: metric-collector metrics (was: ) > Table Level Phoenix Metrics Implementation. > --- > > Key: PHOENIX-5837 > URL: https://issues.apache.org/jira/browse/PHOENIX-5837 > Project: Phoenix > Issue Type: Task >Reporter: vikas meka >Priority: Major > Labels: metric-collector, metrics > > currently GlobalClient Metrics provide aggregated information about the usage > of the application in-terms of total no. of reads, total no. of writes, total > number of failures, total no.of failures etc. In some scenarios these > counters do not provide much useful insights into Phoenix usage pattern. > The task is to add TableLevel Phoenix metrics on top of Global Phoenix > metrics which would be helpful in analyzing the reads/writes/deletes on a > table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5837) Table Level Phoenix Metrics Implementation.
vikas meka created PHOENIX-5837: --- Summary: Table Level Phoenix Metrics Implementation. Key: PHOENIX-5837 URL: https://issues.apache.org/jira/browse/PHOENIX-5837 Project: Phoenix Issue Type: Task Reporter: vikas meka currently GlobalClient Metrics provide aggregated information about the usage of the application in-terms of total no. of reads, total no. of writes, total number of failures, total no.of failures etc. In some scenarios these counters do not provide much useful insights into Phoenix usage pattern. The task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics which would be helpful in analyzing the reads/writes/deletes on a table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5432) Refactor LiteralExpression to use the builder pattern
[ https://issues.apache.org/jira/browse/PHOENIX-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christine Feng updated PHOENIX-5432: Description: LiteralExpression is a mess. While it provides newConstant() APIs to build the object, it also provides two public constructors. There are 10 overloaded newConstant() methods and it is unclear which API to use in which case. This should be refactored to use the builder pattern and final member variables. Ideally, getters such as getMaxLength() should be simple member variable accessors and other ad-hoc logic surrounding those variables should be handled correctly when setting their respective values. Two solutions: # -Consolidate the LiteralExpression newConstant() methods down into a single build() method- ** -Pros: easy to use since one build method can create all LiteralExpression objects- ** -Cons: requires adding 'throws SQLException' to a lot of method signatures where it wasn't necessary before, which can be confusing for future developers and potentially cause problems if a SQLException is actually thrown in some of these cases- # -Create two build() methods - one for LiteralExpressions that necessitate deriving value (which could throw SQLExceptions) and ones that don't- ** -Pros: don't need to change any existing method signatures- ** -Cons: requires future developers to know which of the two build methods to use- # Create two Builders for two separate use cases, each with its own build method. ## Pros: less possibility for confusion ## Cons: needs very explicit documentation on which Builder to use in which case NOTE (13 April 2020): Putting this JIRA on hold for a few weeks until I get the bandwidth to come back to it; for reference, was in the process of switching over from my current Builder with two build methods to two distinct Builders, with one build method each, for distinct and ideally well-documented use cases. GOAL: The goal for this Jira is to simplify the LiteralExpression class and make it easier for future contributors to understand and use LiteralExpression objects. ORIGINAL CODE BEHAVIOR: The original LiteralExpression class had a lot of LiteralExpression constructors, as well as newConstant helper methods that modified some set of LiteralExpression attributes before passing the values onto a constructor. All in all, the various constructors and helper methods boiled down to two types of LiteralExpression creation cases: those that necessitate throwing a SQLException (Case A), and those that do not (Case B). PROPOSED REDESIGN: By creating one LiteralExpressionBuilder that handles Case A, and another that handles Case B, and with sufficient documentation, we avoid possible confusion and resultant creation of LiteralExpression objects with incorrect or unintended attributes. was: LiteralExpression is a mess. While it provides newConstant() APIs to build the object, it also provides two public constructors. There are 10 overloaded newConstant() methods and it is unclear which API to use in which case. This should be refactored to use the builder pattern and final member variables. Ideally, getters such as getMaxLength() should be simple member variable accessors and other ad-hoc logic surrounding those variables should be handled correctly when setting their respective values. Two solutions: # -Consolidate the LiteralExpression newConstant() methods down into a single build() method- ** -Pros: easy to use since one build method can create all LiteralExpression objects- ** -Cons: requires adding 'throws SQLException' to a lot of method signatures where it wasn't necessary before, which can be confusing for future developers and potentially cause problems if a SQLException is actually thrown in some of these cases- # -Create two build() methods - one for LiteralExpressions that necessitate deriving value (which could throw SQLExceptions) and ones that don't- ** -Pros: don't need to change any existing method signatures- ** -Cons: requires future developers to know which of the two build methods to use- # Create two Builders for two separate use cases, each with its own build method. ## Pros: less possibility for confusion ## Cons: needs very explicit documentation on which Builder to use in which case NOTE (13 April 2020): Putting this JIRA on hold for a few weeks until I get the bandwidth to come back to it; for reference, was in the process of switching over from my current Builder with two build methods to two distinct Builders, with one build method each, for distinct and ideally well-documented use cases. > Refactor LiteralExpression to use the builder pattern > - > > Key: PHOENIX-5432 > URL: https://issues.apache.org/jira/browse/PHOENIX-5432 > Project: Phoenix > Is
[jira] [Updated] (PHOENIX-5432) Refactor LiteralExpression to use the builder pattern
[ https://issues.apache.org/jira/browse/PHOENIX-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christine Feng updated PHOENIX-5432: Description: LiteralExpression is a mess. While it provides newConstant() APIs to build the object, it also provides two public constructors. There are 10 overloaded newConstant() methods and it is unclear which API to use in which case. This should be refactored to use the builder pattern and final member variables. Ideally, getters such as getMaxLength() should be simple member variable accessors and other ad-hoc logic surrounding those variables should be handled correctly when setting their respective values. Two solutions: # -Consolidate the LiteralExpression newConstant() methods down into a single build() method- ** -Pros: easy to use since one build method can create all LiteralExpression objects- ** -Cons: requires adding 'throws SQLException' to a lot of method signatures where it wasn't necessary before, which can be confusing for future developers and potentially cause problems if a SQLException is actually thrown in some of these cases- # -Create two build() methods - one for LiteralExpressions that necessitate deriving value (which could throw SQLExceptions) and ones that don't- ** -Pros: don't need to change any existing method signatures- ** -Cons: requires future developers to know which of the two build methods to use- # Create two Builders for two separate use cases, each with its own build method. ## Pros: less possibility for confusion ## Cons: needs very explicit documentation on which Builder to use in which case NOTE (13 April 2020): Putting this JIRA on hold for a few weeks until I get the bandwidth to come back to it; for reference, was in the process of switching over from my current Builder with two build methods to two distinct Builders, with one build method each, for distinct and ideally well-documented use cases. was: LiteralExpression is a mess. While it provides newConstant() APIs to build the object, it also provides two public constructors. There are 10 overloaded newConstant() methods and it is unclear which API to use in which case. This should be refactored to use the builder pattern and final member variables. Ideally, getters such as getMaxLength() should be simple member variable accessors and other ad-hoc logic surrounding those variables should be handled correctly when setting their respective values. Two solutions: # -Consolidate the LiteralExpression newConstant() methods down into a single build() method- ** -Pros: easy to use since one build method can create all LiteralExpression objects- ** -Cons: requires adding 'throws SQLException' to a lot of method signatures where it wasn't necessary before, which can be confusing for future developers and potentially cause problems if a SQLException is actually thrown in some of these cases- # Create two build() methods - one for LiteralExpressions that necessitate deriving value (which could throw SQLExceptions) and ones that don't ** Pros: don't need to change any existing method signatures ** Cons: requires future developers to know which of the two build methods to use NOTE: No backward compatibility testing to be done on master branch, waiting on review > Refactor LiteralExpression to use the builder pattern > - > > Key: PHOENIX-5432 > URL: https://issues.apache.org/jira/browse/PHOENIX-5432 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.15.0, 5.1.0 >Reporter: Chinmay Kulkarni >Assignee: Christine Feng >Priority: Major > Attachments: PHOENIX-5432-master-v1.patch, > PHOENIX-5432.master.v10.patch, PHOENIX-5432.master.v11.patch, > PHOENIX-5432.master.v12.patch, PHOENIX-5432.master.v13.patch, > PHOENIX-5432.master.v14.patch, PHOENIX-5432.master.v2.patch, > PHOENIX-5432.master.v3.patch, PHOENIX-5432.master.v4.patch, > PHOENIX-5432.master.v5.patch, PHOENIX-5432.master.v6.patch, > PHOENIX-5432.master.v7.patch, PHOENIX-5432.master.v8.patch, > PHOENIX-5432.master.v9.patch, PHOENIX-5432.patch > > Time Spent: 6h 20m > Remaining Estimate: 0h > > LiteralExpression is a mess. While it provides newConstant() APIs to build > the object, it also provides two public constructors. There are 10 overloaded > newConstant() methods and it is unclear which API to use in which case. > This should be refactored to use the builder pattern and final member > variables. Ideally, getters such as getMaxLength() should be simple member > variable accessors and other ad-hoc logic surrounding those variables should > be handled correctly when setting their respective values. > > Two solutions: > # -Consolidate the LiteralExpression newConstant() methods down into a > si
[jira] [Resolved] (PHOENIX-5836) The first concurrent mutation may not be considered as concurrent
[ https://issues.apache.org/jira/browse/PHOENIX-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kadir OZDEMIR resolved PHOENIX-5836. Resolution: Incomplete > The first concurrent mutation may not be considered as concurrent > - > > Key: PHOENIX-5836 > URL: https://issues.apache.org/jira/browse/PHOENIX-5836 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.14.3 >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Major > Attachments: PHOENIX-5836.4.x.001.patch > > > From the consistent indexing design (PHOENIX-5156) perspective, two or more > pending updates from different batches on the same data row are concurrent if > and only if for all of these updates the data table row state is read from > HBase under the row lock and for none of them the row lock has been acquired > the second time for updating the data table. In other words, all of them are > in the first update phase concurrently. For concurrent updates, the first two > update phases are done but the last update phase is skipped. This means the > data table row will be updated by these updates but the corresponding index > table rows will be left with the unverified status. Then, the read repair > process will repair these unverified index rows during scans. > However, if a concurrent mutation acquires the lock for reading the data row > before any other concurrent mutations on this mutation, and also acquires the > lock for updating the data row before any other mutations, this mutation can > be treated as non-concurrent, and the third update phase can be executed > safely for this mutation. Doing this will reduce the number of unverified > rows due to concurrent mutations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5836) The first concurrent mutation may not be considered as concurrent
Kadir OZDEMIR created PHOENIX-5836: -- Summary: The first concurrent mutation may not be considered as concurrent Key: PHOENIX-5836 URL: https://issues.apache.org/jira/browse/PHOENIX-5836 Project: Phoenix Issue Type: Improvement Affects Versions: 4.14.3, 5.0.0 Reporter: Kadir OZDEMIR Assignee: Kadir OZDEMIR >From the consistent indexing design (PHOENIX-5156) perspective, two or more >pending updates from different batches on the same data row are concurrent if >and only if for all of these updates the data table row state is read from >HBase under the row lock and for none of them the row lock has been acquired >the second time for updating the data table. In other words, all of them are >in the first update phase concurrently. For concurrent updates, the first two >update phases are done but the last update phase is skipped. This means the >data table row will be updated by these updates but the corresponding index >table rows will be left with the unverified status. Then, the read repair >process will repair these unverified index rows during scans. However, if a concurrent mutation acquires the lock for reading the data row before any other concurrent mutations on this mutation, and also acquires the lock for updating the data row before any other mutations, this mutation can be treated as non-concurrent, and the third update phase can be executed safely for this mutation. Doing this will reduce the number of unverified rows due to concurrent mutations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5835) When data columns get TTLed, and run IndexScrutiny or validation we marked them as invalid rows
Gokcen Iskender created PHOENIX-5835: Summary: When data columns get TTLed, and run IndexScrutiny or validation we marked them as invalid rows Key: PHOENIX-5835 URL: https://issues.apache.org/jira/browse/PHOENIX-5835 Project: Phoenix Issue Type: Improvement Reporter: Gokcen Iskender We do index writes as full row writes. This means all columns keep get re-written to index and they have a current timestamp. However, if there is a column that did not get updated for a long time (like Created_By type of columns that don't change) the regular scan that IndexScrutiny uses doesn't return this column even though we might see it in the raw scan. IndexScrutiny mark these rows as invalid because data table had TTL columns and index table have it. Index inline validation also marks these as "Index has extra columns" if the column got TTLed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5834) IndexTool cannot support tablename lowercase
shining created PHOENIX-5834:
Summary: IndexTool cannot support tablename lowercase
Key: PHOENIX-5834
URL: https://issues.apache.org/jira/browse/PHOENIX-5834
Project: Phoenix
Issue Type: Bug
Components: core
Affects Versions: 5.0.0
Reporter: shining
1. Create table in hbase
>> create 'ns:phoenix01','info'
>> put 'ns:phoenix1','row001','info:id',''
>> put 'ns:phoenix1','row001','info:name','manager'
2. Create table and async index in phoenix
>> create schema "ns"
>> create table "ns"."phoenix01"(ROW varchar primary key, "info"."id"
varchar, "info"."name" varchar)column_encoded_bytes=0;
>> create index "phoenix01_index" on "ns"."phoenix01"("info"."name") ASYNC;
3. Use IndexTool build index
hbase org.apache.phoenix.mapreduce.index.IndexTool -s "ns" -dt "phoenix01"
-it "phoenix01_index" -op /indextest
Then i get nothing and return directly.
I check the code and find the error position:
{code:java}
private boolean isValidIndexTable(final Connection connection, final String
masterTable,
final String indexTable) throws SQLException {
final DatabaseMetaData dbMetaData = connection.getMetaData();
final String schemaName =
SchemaUtil.getSchemaNameFromFullName(masterTable);
final String tableName =
SchemaUtil.normalizeIdentifier(SchemaUtil.getTableNameFromFullName(masterTable));
ResultSet rs = null;
try {
rs = dbMetaData.getIndexInfo(null, schemaName, tableName, false,
false);
while (rs.next()) {
final String indexName = rs.getString(6);
if (indexTable.equalsIgnoreCase(indexName)) {
return true;
}
}
} finally {
if (rs != null) {
rs.close();
}
}
return false;
}
{code}
I give the lowcase tablename,but indexTool change it uppercase,the
validIndexTable get null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
