[jira] [Created] (PHOENIX-5999) Have executemany leverage ExecuteBatchRequest
Josh Elser created PHOENIX-5999:
---
Summary: Have executemany leverage ExecuteBatchRequest
Key: PHOENIX-5999
URL: https://issues.apache.org/jira/browse/PHOENIX-5999
Project: Phoenix
Issue Type: Improvement
Components: python
Reporter: Josh Elser
Assignee: Josh Elser
After some testing years ago, I wrote ExecuteBatch bindings for avatica. The
observation was that we spent more time executing the HTTP call and parsing the
tiny protobuf than we did in sending the update to HBase.
ExecuteBatch was a dirt-simple idea in that instead of sending one row's worth
of parameters to bind to a statement, send many row's worth.
e.g. before we would do:
{noformat}
execute(stmt, ['a', 'b']); execute(stmt, ['b', 'c']), ... {noformat}
but with executeBatch we can do
{noformat}
executeBatch(stmt, [['a', 'b'], ['b', 'c'], ...]) {noformat}
and send exactly one http call instead of multiple. Obviously this is a huge
saving.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PHOENIX-5995) Index Rebuild page size is not honored in case of point deletes
[ https://issues.apache.org/jira/browse/PHOENIX-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated PHOENIX-5995: Affects Version/s: 4.15.0 > Index Rebuild page size is not honored in case of point deletes > --- > > Key: PHOENIX-5995 > URL: https://issues.apache.org/jira/browse/PHOENIX-5995 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5995-4.x.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5997) Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths.
[ https://issues.apache.org/jira/browse/PHOENIX-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Wong updated PHOENIX-5997: - Attachment: (was: PHOENIX-5997-4.x.patch) > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > -- > > Key: PHOENIX-5997 > URL: https://issues.apache.org/jira/browse/PHOENIX-5997 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0, 4.16.0 >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Minor > Fix For: 4.16.0 > > Attachments: PHOENIX-5997-4.x.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > Example to follow as well as changed proposed text. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5997) Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths.
[ https://issues.apache.org/jira/browse/PHOENIX-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Wong updated PHOENIX-5997: - Attachment: PHOENIX-5997-4.x.patch > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > -- > > Key: PHOENIX-5997 > URL: https://issues.apache.org/jira/browse/PHOENIX-5997 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0, 4.16.0 >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Minor > Fix For: 4.16.0 > > Attachments: PHOENIX-5997-4.x.patch, PHOENIX-5997-4.x.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > Example to follow as well as changed proposed text. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5998) Paged server side operations
[ https://issues.apache.org/jira/browse/PHOENIX-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kadir OZDEMIR updated PHOENIX-5998: --- Description: Phoenix provides the option of performing upsert select and delete query operations on the client or server side. This is decided by the Phoenix optimizer based on configuration parameters. For the server side option, the table operation (upsert select/delete query) is parallelized such that multiple table regions are scanned and the mutations derived from these scans can also be executed in parallel on the server side. However, currently there is no paging capability and the server side operation can take long enough lead to HBase client timeouts. When this happens, Phoenix can return failure to its applications and the rest of the parallel scans and mutations on the server side can still continue since Phoenix has no mechanism in place to stop these operations before returning failure to applications. This can create unexpected race conditions between these left-over operations and the new operations issued by applications. Putting a limit on the number of rows to be processed within a single RPC call (i.e., the next operation on the scanner) on the server side using a Phoenix level paging is highly desirable and a required step to prevent the possible race conditions. This paging mechanism has been already implemented for index rebuild and verification operations and proven to be effective to prevent timeouts. This paging can be implemented for all server side operations including aggregates, upsert selects, delete queries and so on. > Paged server side operations > - > > Key: PHOENIX-5998 > URL: https://issues.apache.org/jira/browse/PHOENIX-5998 > Project: Phoenix > Issue Type: Improvement > Environment: Phoenix provides the option of performing upsert select > and delete query operations on the client or server side. This is decided by > the Phoenix optimizer based on configuration parameters. For the server side > option, the table operation (upsert select/delete query) is parallelized such > that multiple table regions are scanned and the mutations derived from these > scans can also be executed in parallel on the server side. However, currently > there is no paging capability and the server side operation can take long > enough lead to HBase client timeouts. When this happens, Phoenix can return > failure to its applications and the rest of the parallel scans and mutations > on the server side can still continue since Phoenix has no mechanism in > place to stop these operations before returning failure to applications. This > can create unexpected race conditions between these left-over operations and > the new operations issued by applications. Putting a limit on the number of > rows to be processed within a single RPC call (i.e., the next operation on > the scanner) on the server side using a Phoenix level paging is highly > desirable and a required step to prevent the possible race conditions. This > paging mechanism has been already implemented for index rebuild and > verification operations and proven to be effective to prevent timeouts. This > paging can be implemented for all server side operations including > aggregates, upsert selects, delete queries and so on. >Reporter: Kadir OZDEMIR >Priority: Major > Fix For: 4.x > > > Phoenix provides the option of performing upsert select and delete query > operations on the client or server side. This is decided by the Phoenix > optimizer based on configuration parameters. For the server side option, the > table operation (upsert select/delete query) is parallelized such that > multiple table regions are scanned and the mutations derived from these scans > can also be executed in parallel on the server side. However, currently there > is no paging capability and the server side operation can take long enough > lead to HBase client timeouts. When this happens, Phoenix can return failure > to its applications and the rest of the parallel scans and mutations on the > server side can still continue since Phoenix has no mechanism in place to > stop these operations before returning failure to applications. This can > create unexpected race conditions between these left-over operations and the > new operations issued by applications. Putting a limit on the number of rows > to be processed within a single RPC call (i.e., the next operation on the > scanner) on the server side using a Phoenix level paging is highly desirable > and a required step to prevent the possible race conditions. This paging > mechanism has been already implemented for index rebuild and verification > operations and proven to be effective to prevent timeouts. This paging can be >
[jira] [Updated] (PHOENIX-5986) DropTableWithViewsIT.testDropTableWithChildViews is flapping again
[
https://issues.apache.org/jira/browse/PHOENIX-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5986:
--
Description:
DropTableWithViewsIT.testDropTableWithChildViews is flapping again. I see
[PHOENIX-5074|https://issues.apache.org/jira/browse/PHOENIX-5074] fixed this
test so that the task is run immediately, but somehow this test is still
intermittently failing.
This has caused recent preCommit failures for PHOENIX-5981 (build
[4010|https://builds.apache.org/job/PreCommit-PHOENIX-Build/4010//testReport/])
and
PHOENIX-5984 (build
[4011|https://builds.apache.org/job/PreCommit-PHOENIX-Build/4011//testReport/]).
When I run this test locally, it passes sometimes and fails sometimes. This is
the relevant stack trace:
{code:java}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertFalse(Assert.java:65)
at org.junit.Assert.assertFalse(Assert.java:75)
at
org.apache.phoenix.end2end.DropTableWithViewsIT.testDropTableWithChildViews(DropTableWithViewsIT.java:158)
{code}
Failure is observed in [this
assertion|https://github.com/apache/phoenix/blob/f5ebf2aaa337d0b4b935f10917cee3d99d621c98/phoenix-core/src/it/java/org/apache/phoenix/end2end/DropTableWithViewsIT.java#L158].
I also see that the test sometimes fails because it expects task status to be
COMPLETED, but sees status to be STARTED instead.
was:
DropTableWithViewsIT.testDropTableWithChildViews is flapping again. I see
[PHOENIX-5074|https://issues.apache.org/jira/browse/PHOENIX-5074] fixed this
test so that the task is run immediately, but somehow this test is still
intermittently failing.
This has caused recent preCommit failures for PHOENIX-5981 (build
[4010|https://builds.apache.org/job/PreCommit-PHOENIX-Build/4010//testReport/])
and
PHOENIX-5984 (build
[4011|https://builds.apache.org/job/PreCommit-PHOENIX-Build/4011//testReport/]).
When I run this test locally, it passes sometimes and fails sometimes. This is
the relevant stack trace:
{code:java}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertFalse(Assert.java:65)
at org.junit.Assert.assertFalse(Assert.java:75)
at
org.apache.phoenix.end2end.DropTableWithViewsIT.testDropTableWithChildViews(DropTableWithViewsIT.java:158)
{code}
> DropTableWithViewsIT.testDropTableWithChildViews is flapping again
> --
>
> Key: PHOENIX-5986
> URL: https://issues.apache.org/jira/browse/PHOENIX-5986
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.15.0
>Reporter: Chinmay Kulkarni
>Priority: Major
> Labels: phoenix-hardening, quality-improvement
> Fix For: 4.16.0
>
>
> DropTableWithViewsIT.testDropTableWithChildViews is flapping again. I see
> [PHOENIX-5074|https://issues.apache.org/jira/browse/PHOENIX-5074] fixed this
> test so that the task is run immediately, but somehow this test is still
> intermittently failing.
> This has caused recent preCommit failures for PHOENIX-5981 (build
> [4010|https://builds.apache.org/job/PreCommit-PHOENIX-Build/4010//testReport/])
> and
> PHOENIX-5984 (build
> [4011|https://builds.apache.org/job/PreCommit-PHOENIX-Build/4011//testReport/]).
> When I run this test locally, it passes sometimes and fails sometimes. This
> is the relevant stack trace:
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertFalse(Assert.java:65)
> at org.junit.Assert.assertFalse(Assert.java:75)
> at
> org.apache.phoenix.end2end.DropTableWithViewsIT.testDropTableWithChildViews(DropTableWithViewsIT.java:158)
> {code}
> Failure is observed in [this
> assertion|https://github.com/apache/phoenix/blob/f5ebf2aaa337d0b4b935f10917cee3d99d621c98/phoenix-core/src/it/java/org/apache/phoenix/end2end/DropTableWithViewsIT.java#L158].
> I also see that the test sometimes fails because it expects task status to
> be COMPLETED, but sees status to be STARTED instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (PHOENIX-5998) Paged server side operations
Kadir OZDEMIR created PHOENIX-5998: -- Summary: Paged server side operations Key: PHOENIX-5998 URL: https://issues.apache.org/jira/browse/PHOENIX-5998 Project: Phoenix Issue Type: Improvement Environment: Phoenix provides the option of performing upsert select and delete query operations on the client or server side. This is decided by the Phoenix optimizer based on configuration parameters. For the server side option, the table operation (upsert select/delete query) is parallelized such that multiple table regions are scanned and the mutations derived from these scans can also be executed in parallel on the server side. However, currently there is no paging capability and the server side operation can take long enough lead to HBase client timeouts. When this happens, Phoenix can return failure to its applications and the rest of the parallel scans and mutations on the server side can still continue since Phoenix has no mechanism in place to stop these operations before returning failure to applications. This can create unexpected race conditions between these left-over operations and the new operations issued by applications. Putting a limit on the number of rows to be processed within a single RPC call (i.e., the next operation on the scanner) on the server side using a Phoenix level paging is highly desirable and a required step to prevent the possible race conditions. This paging mechanism has been already implemented for index rebuild and verification operations and proven to be effective to prevent timeouts. This paging can be implemented for all server side operations including aggregates, upsert selects, delete queries and so on. Reporter: Kadir OZDEMIR Fix For: 4.x -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5984) Query timeout counter is not updated in all timeouts cases
[ https://issues.apache.org/jira/browse/PHOENIX-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5984: -- Attachment: PHOENIX-5984-4.x-v2.patch > Query timeout counter is not updated in all timeouts cases > -- > > Key: PHOENIX-5984 > URL: https://issues.apache.org/jira/browse/PHOENIX-5984 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Labels: metrics, phoenix-hardening, quality-improvement > Fix For: 4.16.0 > > Attachments: PHOENIX-5984-4.x-v1.patch, PHOENIX-5984-4.x-v2.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The OverallQueryMetrics query timeout counter and global query timeout > counter are not updated if the query times out before getting the future > corresponding to the scan i.e. if it times out at [this point| > https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1299-L1301]. > We only update relevant metrics if we catch a TimeoutException > [here|https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1361] > if the future.get times out > [here|https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1312]. > We should update the metrics in the first case as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5981) Wrong multiple counting of resultSetTimeMs and wallclockTimeMs in OverallQueryMetrics
[ https://issues.apache.org/jira/browse/PHOENIX-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5981: -- Attachment: PHOENIX-5981-4.x-v2.patch > Wrong multiple counting of resultSetTimeMs and wallclockTimeMs in > OverallQueryMetrics > - > > Key: PHOENIX-5981 > URL: https://issues.apache.org/jira/browse/PHOENIX-5981 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Labels: metrics, phoenix-hardening, quality-improvement > Fix For: 4.16.0 > > Attachments: PHOENIX-5981-4.x-v1.patch, PHOENIX-5981-4.x-v2.patch, > PHOENIX-5981-master-v1.patch > > Time Spent: 20m > Remaining Estimate: 0h > > We update wallClockTimeMS and resultSetTimeMS when we reach the last row in > our ResultSet iterations (see > [this|https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixResultSet.java#L878-L881]). > However, this is also called inside PhoenixResultSet.close(), see > [this|https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixResultSet.java#L214-L215]. > The problem is, these calls in-turn add elapsedTime again > ([endQuery|https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/monitoring/OverAllQueryMetrics.java#L86] > and > [stopResultsetWatch|https://github.com/apache/phoenix/blob/dcc88af8acc2ba8df10d2e9d498ab3646fdf0a78/phoenix-core/src/main/java/org/apache/phoenix/monitoring/OverAllQueryMetrics.java#L95]) > and so these metrics get 2X the value they should. Ideally we should change > the metrics only if the stopWatch was running before calling stopWatch.stop > or else we will count it multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5997) Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths.
[ https://issues.apache.org/jira/browse/PHOENIX-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-5997: --- Fix Version/s: 4.16.0 > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > -- > > Key: PHOENIX-5997 > URL: https://issues.apache.org/jira/browse/PHOENIX-5997 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0, 4.16.0 >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Minor > Fix For: 4.16.0 > > > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > Example to follow as well as changed proposed text. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5997) Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths.
[ https://issues.apache.org/jira/browse/PHOENIX-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-5997: --- Affects Version/s: 4.16.0 4.15.0 > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > -- > > Key: PHOENIX-5997 > URL: https://issues.apache.org/jira/browse/PHOENIX-5997 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0, 4.16.0 >Reporter: Daniel Wong >Assignee: Daniel Wong >Priority: Minor > > Phoenix Explain Plan for Deletes does not clearly differentiate between > server side and client side paths. > Example to follow as well as changed proposed text. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5997) Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths.
Daniel Wong created PHOENIX-5997: Summary: Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths. Key: PHOENIX-5997 URL: https://issues.apache.org/jira/browse/PHOENIX-5997 Project: Phoenix Issue Type: Bug Reporter: Daniel Wong Assignee: Daniel Wong Phoenix Explain Plan for Deletes does not clearly differentiate between server side and client side paths. Example to follow as well as changed proposed text. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (PHOENIX-5789) try to standardize on a JSON library
[ https://issues.apache.org/jira/browse/PHOENIX-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth resolved PHOENIX-5789. -- Resolution: Fixed Pushed the addendum for 4.x > try to standardize on a JSON library > > > Key: PHOENIX-5789 > URL: https://issues.apache.org/jira/browse/PHOENIX-5789 > Project: Phoenix > Issue Type: Improvement > Components: core >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Minor > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5789.4.x.addendum.v1.patch, > PHOENIX-5789.4.x.v1.patch, PHOENIX-5789.4.x.v3.patch, > PHOENIX-5789.4.x.v4.patch, PHOENIX-5789.4.x.v5.patch, > PHOENIX-5789.4.x.v6.patch, PHOENIX-5789.4.x.v7.patch, > PHOENIX-5789.master.addendum.patch, PHOENIX-5789.master.addendum.v2.patch, > PHOENIX-5789.master.addendum.v3.patch, PHOENIX-5789.master.v1.patch, > PHOENIX-5789.master.v2.patch, PHOENIX-5789.master.v3.patch, > PHOENIX-5789.master.v4.patch, Screen Shot 2020-06-17 at 11.45.44 AM.png > > Time Spent: 20m > Remaining Estimate: 0h > > Phoenix uses at least the following JSON libraries: > * gson > * jackson > * jettison > Of these, only the jackson usage is performance critical, as it is used > during bulk loading. > Try to standardize on a single one to reduce dependency hell. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5958) Diverged view created from an older client still sees dropped column data
[ https://issues.apache.org/jira/browse/PHOENIX-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5958: -- Priority: Blocker (was: Major) > Diverged view created from an older client still sees dropped column data > - > > Key: PHOENIX-5958 > URL: https://issues.apache.org/jira/browse/PHOENIX-5958 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Blocker > Fix For: 4.16.0 > > > By "diverged view" I mean creating a view and then dropping one of the > inherited columns from the view. Steps to reproduce: > Start a 4.x server and connect with a pre-4.15 (I tried a 4.14.3) client > # CREATE TABLE IF NOT EXISTS S.T (A INTEGER PRIMARY KEY, B INTEGER); > # CREATE VIEW IF NOT EXISTS S.V (new_col INTEGER) AS SELECT * FROM S.T; > # UPSERT INTO S.T VALUES(1,2); > # ALTER VIEW S.V DROP COLUMN B; > # SELECT * FROM S.T; gives: > |A|B| > |1|2| > # SELECT * FROM S.V; gives: > |B|A|NEW_COL| > |2|1|null| > Though the column 'B' has been dropped from the view. This does not happen > for a 4.x client. > The problem is mostly due to changes introduced by > [PHOENIX-4893|https://issues.apache.org/jira/browse/PHOENIX-4893]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (PHOENIX-5784) Phoenix-connectors doesn't work with phoenix master branch
[ https://issues.apache.org/jira/browse/PHOENIX-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Antal reassigned PHOENIX-5784: -- Assignee: Richard Antal > Phoenix-connectors doesn't work with phoenix master branch > -- > > Key: PHOENIX-5784 > URL: https://issues.apache.org/jira/browse/PHOENIX-5784 > Project: Phoenix > Issue Type: Bug > Components: connectors >Affects Versions: connectors-1.0.0 >Reporter: Istvan Toth >Assignee: Richard Antal >Priority: Major > > phoenix-connectors cannot even be built against the master branch: > mvn clean package -Dphoenix.version=5.1.0-SNAPSHOT -Dhadoop-two.version=3.0.0 > -Dhbase.version=2.2.1 -fn > fails for about half of the modules. > Most of these problems are simple API changes, have some kind of solution in > the pre-split master branch, the question is how to make them coexist in a > repo/branch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5996) IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly delete index row when a delete and put mutation with the same timestamp
[
https://issues.apache.org/jira/browse/PHOENIX-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated PHOENIX-5996:
--
Description:
With PHOENIX-5748,
{{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}} is responsible
for generating index table mutations for rebuild.
In the processing of data table mutations list, there can be a delete and put
mutation with the same timestamp. If so, the delete and put are processed
together in one iteration. First, the delete mutation is applied on the put
mutation and current row state, and then the modified put mutation is processed.
But when the {{modified put mutation}} is empty , even the current row state is
not empty after the delete mutation is applied, the whole index row is
deleted, just as following line 1191 in
{{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}}:
{code:java}
1189 } else {
1190if (currentDataRowState != null) {
1191Mutation del =
indexMaintainer.buildRowDeleteMutation(indexRowKeyForCurrentDataRow,
1192IndexMaintainer.DeleteType.ALL_VERSIONS,
ts);
1193indexMutations.add(del);
1194// For the next iteration of the for loop
1195currentDataRowState = null;
1196indexRowKeyForCurrentDataRow = null;
1197}
1198 }
{code}
I think above logical is wrong, when the current row state is not empty after
the delete mutation is applied, we can not delete the whole index row, but
instead we should reuse the logical of applying a delete mutation on current
row state. I wrote a unit test in {{PrepareIndexMutationsForRebuildTest}} to
produce the case:
{code:java}
@Test
public void testPutDeleteOnSameTimeStampAndPutNullifiedByDelete() throws
Exception {
SetupInfo info = setup(
TABLE_NAME,
INDEX_NAME,
"ROW_KEY VARCHAR, CF1.C1 VARCHAR, CF2.C2 VARCHAR",
"CF2.C2",
"ROW_KEY",
"");
Put dataPut = new Put(Bytes.toBytes(ROW_KEY));
addCellToPutMutation(
dataPut,
Bytes.toBytes("CF2"),
Bytes.toBytes("C2"),
1,
Bytes.toBytes("v2"));
addEmptyColumnToDataPutMutation(dataPut, info.pDataTable, 1);
addCellToPutMutation(
dataPut,
Bytes.toBytes("CF1"),
Bytes.toBytes("C1"),
2,
Bytes.toBytes("v1"));
addEmptyColumnToDataPutMutation(dataPut, info.pDataTable, 2);
Delete dataDel = new Delete(Bytes.toBytes(ROW_KEY));
addCellToDelMutation(
dataDel,
Bytes.toBytes("CF1"),
null,
2,
KeyValue.Type.DeleteFamily);
List actualIndexMutations =
IndexRebuildRegionScanner.prepareIndexMutationsForRebuild(
info.indexMaintainer,
dataPut,
dataDel);
List expectedIndexMutations = new ArrayList<>();
byte[] idxKeyBytes = generateIndexRowKey("v2");
Put idxPut1 = new Put(idxKeyBytes);
addEmptyColumnToIndexPutMutation(idxPut1, info.indexMaintainer, 1);
expectedIndexMutations.add(idxPut1);
Put idxPut2 = new Put(idxKeyBytes);
addEmptyColumnToIndexPutMutation(idxPut2, info.indexMaintainer, 2);
expectedIndexMutations.add(idxPut2);
assertEqualMutationList(expectedIndexMutations, actualIndexMutations);
}
{code}
was:
With PHOENIX-5748,
{{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}} is responsible
for generating index table mutations for rebuild.
In the processing of data table mutations list, there can be a delete and put
mutation with the same timestamp. If so, the delete and put are processed
together in one iteration. First, the delete mutation is applied on the put
mutation and current row state, and then the modified put mutation is processed.
But when the {{modified put mutation}} is empty , even the current row state is
not empty after the delete mutation is applied, the whole index row is
deleted, just as following line 1191 in
{{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}}:
{code:java}
1189 } else {
1190if (currentDataRowState != null) {
1191Mutation del =
indexMaintainer.buildRowDeleteMutation(indexRowKeyForCurrentDataRow,
1192IndexMaintainer.DeleteType.ALL_VERSIONS,
ts);
1193indexMutations.add(del);
1194// For the next iteration of the for loop
1195currentDataRowState = null;
1196
[jira] [Updated] (PHOENIX-5996) IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly delete index row when a delete and put mutation with the same timestamp
[
https://issues.apache.org/jira/browse/PHOENIX-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated PHOENIX-5996:
--
Description:
With PHOENIX-5748,
{{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}} is responsible
for generating index table mutations for rebuild.
In the processing of data table mutations list, there can be a delete and put
mutation with the same timestamp. If so, the delete and put are processed
together in one iteration. First, the delete mutation is applied on the put
mutation and current row state, and then the modified put mutation is processed.
But when the {{modified put mutation}} is empty , even the current row state is
not empty after the delete mutation is applied, the whole index row is
deleted, just as following line 1191 in
{{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}}:
{code:java}
1189 } else {
1190if (currentDataRowState != null) {
1191Mutation del =
indexMaintainer.buildRowDeleteMutation(indexRowKeyForCurrentDataRow,
1192IndexMaintainer.DeleteType.ALL_VERSIONS,
ts);
1193indexMutations.add(del);
1194// For the next iteration of the for loop
1195currentDataRowState = null;
1196indexRowKeyForCurrentDataRow = null;
1197}
1198 }
{code}
I think above logical is wrong, when the current row state is not empty after
the delete mutation is applied, we can not delete the whole index row, but
instead we should reuse the logical of processing a delete mutation. I wrote a
unit test in {{PrepareIndexMutationsForRebuildTest}} to produce the case:
{code:java}
@Test
public void testPutDeleteOnSameTimeStampAndPutNullifiedByDelete() throws
Exception {
SetupInfo info = setup(
TABLE_NAME,
INDEX_NAME,
"ROW_KEY VARCHAR, CF1.C1 VARCHAR, CF2.C2 VARCHAR",
"CF2.C2",
"ROW_KEY",
"");
Put dataPut = new Put(Bytes.toBytes(ROW_KEY));
addCellToPutMutation(
dataPut,
Bytes.toBytes("CF2"),
Bytes.toBytes("C2"),
1,
Bytes.toBytes("v2"));
addEmptyColumnToDataPutMutation(dataPut, info.pDataTable, 1);
addCellToPutMutation(
dataPut,
Bytes.toBytes("CF1"),
Bytes.toBytes("C1"),
2,
Bytes.toBytes("v1"));
addEmptyColumnToDataPutMutation(dataPut, info.pDataTable, 2);
Delete dataDel = new Delete(Bytes.toBytes(ROW_KEY));
addCellToDelMutation(
dataDel,
Bytes.toBytes("CF1"),
null,
2,
KeyValue.Type.DeleteFamily);
List actualIndexMutations =
IndexRebuildRegionScanner.prepareIndexMutationsForRebuild(
info.indexMaintainer,
dataPut,
dataDel);
List expectedIndexMutations = new ArrayList<>();
byte[] idxKeyBytes = generateIndexRowKey("v2");
Put idxPut1 = new Put(idxKeyBytes);
addEmptyColumnToIndexPutMutation(idxPut1, info.indexMaintainer, 1);
expectedIndexMutations.add(idxPut1);
Put idxPut2 = new Put(idxKeyBytes);
addEmptyColumnToIndexPutMutation(idxPut2, info.indexMaintainer, 2);
expectedIndexMutations.add(idxPut2);
assertEqualMutationList(expectedIndexMutations, actualIndexMutations);
}
{code}
was:With PHOENIX-5748, IndexRebuildRegionScanner is responsible for
generating index table mutations
> IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly
> delete index row when a delete and put mutation with the same timestamp
> -
>
> Key: PHOENIX-5996
> URL: https://issues.apache.org/jira/browse/PHOENIX-5996
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.1.0, 4.16.0
>Reporter: chenglei
>Priority: Major
>
> With PHOENIX-5748,
> {{IndexRebuildRegionScanner.prepareIndexMutationsForRebuild}} is responsible
> for generating index table mutations for rebuild.
> In the processing of data table mutations list, there can be a delete and put
> mutation with the same timestamp. If so, the delete and put are processed
> together in one iteration. First, the delete mutation is applied on the put
> mutation and current row state, and then the modified put mutation is
> processed.
> But when the {{modified put mutation}} is empty , even the current row state
> is not empty after
[jira] [Updated] (PHOENIX-5996) IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly delete index row when a delete and put mutation with the same timestamp
[ https://issues.apache.org/jira/browse/PHOENIX-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenglei updated PHOENIX-5996: -- Description: With PHOENIX-5748, IndexRebuildRegionScanner is responsible for generating index table mutations > IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly > delete index row when a delete and put mutation with the same timestamp > - > > Key: PHOENIX-5996 > URL: https://issues.apache.org/jira/browse/PHOENIX-5996 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.1.0, 4.16.0 >Reporter: chenglei >Priority: Major > > With PHOENIX-5748, IndexRebuildRegionScanner is responsible for generating > index table mutations -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5996) IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly delete index row when a delete and put mutation with the same timestamp
chenglei created PHOENIX-5996: - Summary: IndexRebuildRegionScanner.prepareIndexMutationsForRebuild may incorrectly delete index row when a delete and put mutation with the same timestamp Key: PHOENIX-5996 URL: https://issues.apache.org/jira/browse/PHOENIX-5996 Project: Phoenix Issue Type: Bug Affects Versions: 5.1.0, 4.16.0 Reporter: chenglei -- This message was sent by Atlassian Jira (v8.3.4#803005)
