[jira] [Commented] (PHOENIX-4519) Index rebuild MR jobs not created for "alter index rebuild async" rebuilds

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331824#comment-16331824
 ] 

Ankit Singhal commented on PHOENIX-4519:


bq. Have these been incorporated in the MR partial index rebuild code path? 
Yes. IndexTool was extended with an option for a partial rebuild. 
https://issues.apache.org/jira/browse/PHOENIX-2890

bq. Do we have unit tests for the MR partial index rebuilder? Can we run the 
PartialIndexRebuilderIT tests using the MR-based rebuilder?
Yes, IndexToolForPartialBuildIT is for partial index rebuilder only

bq. Has the documentation been updated here: 
https://phoenix.apache.org/secondary_indexing.html?
Not yet, I regret the same. I have some documentation pending for 2-3 features 
(Phoenix ACLs is another). Let me try to update them all this weekend.

bq. Can we do some refactoring so that the code can be shared between 
MetaDataRegionObserver and the MR partial index rebuilder?
yes, we can do that. there is a lot of overlap between 
PhoenixIndexPartialBuildMapper and MetaDataRegionObserver

bq. How does the MR based partial index rebuilder handle clearing the 
INDEX_DISABLE_TIMESTAMP? For example, what if one of the map tasks fails while 
others succeed?
We mark the index as active in a single reducer when all map tasks are 
completed.

bq. Does the MR base rebuilder wait until all the index regions are back online 
before starting?
some Map will not progress until all the regions are up or may fail eventually 
after some retries.

bq. Are all the various rebuild options supported by the MR partial rebuilder 
(leaving the index active while rebuilding, allowing this option to be 
configured on a per table basis)?
I'm not aware of this option so probably not supported. In MR partial 
rebuilder, Index will be inactive until the rebuild completes, it will keep on 
accepting the writes but not serve any queries. 







> Index rebuild MR jobs not created for "alter index rebuild async" rebuilds
> --
>
> Key: PHOENIX-4519
> URL: https://issues.apache.org/jira/browse/PHOENIX-4519
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
>
> It seems we have two ASYNC flags for index rebuilds:
> ASYNC_CREATED_DATE - when an index is created async
> ASYNC_REBUILD_TIMESTAMP - created by "alter index ... rebuild async"
> The PhoenixMRJobSubmitter only submits MR jobs for the former.  We should 
> also submit jobs for the latter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4378) Unable to set KEEP_DELETED_CELLS to true on RS scanner

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331804#comment-16331804
 ] 

Ankit Singhal commented on PHOENIX-4378:


[~jamestaylor], raised HBASE-19826 for the same.

> Unable to set KEEP_DELETED_CELLS to true on RS scanner
> --
>
> Key: PHOENIX-4378
> URL: https://issues.apache.org/jira/browse/PHOENIX-4378
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0.0
>
>
> [~jamestaylor], 
> It seems we may need to fix PHOENIX-4277 differently for HBase 2.0 as we can 
> only update TTL and maxVersions now in preStoreScannerOpen and cannot return 
> a new StoreScanner with updated scanInfo.
> for reference:
> [1]https://issues.apache.org/jira/browse/PHOENIX-4318?focusedCommentId=16249943=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16249943



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (PHOENIX-4378) Unable to set KEEP_DELETED_CELLS to true on RS scanner

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal reassigned PHOENIX-4378:
--

Assignee: Ankit Singhal

> Unable to set KEEP_DELETED_CELLS to true on RS scanner
> --
>
> Key: PHOENIX-4378
> URL: https://issues.apache.org/jira/browse/PHOENIX-4378
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0.0
>
>
> [~jamestaylor], 
> It seems we may need to fix PHOENIX-4277 differently for HBase 2.0 as we can 
> only update TTL and maxVersions now in preStoreScannerOpen and cannot return 
> a new StoreScanner with updated scanInfo.
> for reference:
> [1]https://issues.apache.org/jira/browse/PHOENIX-4318?focusedCommentId=16249943=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16249943



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331775#comment-16331775
 ] 

Ankit Singhal commented on PHOENIX-4537:


bq. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
before a client first connects
A simple fix could be to skip 
UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes if fullTableName is a 
SYSTEM table as we are not expecting indexes on a system table.







> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4534) upsert/delete/upsert for the same row corrupts the indexes

[Sergey Soldatov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331767#comment-16331767
 ] 

Sergey Soldatov commented on PHOENIX-4534:
--

[~rajeshbabu] at first glance that would work. Possible we may want to get a 
patch for master branch and let jenkins check whether any of UT/IT fails?

> upsert/delete/upsert for the same row corrupts the indexes
> --
>
> Key: PHOENIX-4534
> URL: https://issues.apache.org/jira/browse/PHOENIX-4534
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Romil Choksi
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
>  Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4534.patch
>
>
> If we delete and upsert again the same row, the corresponding index has a 
> null value. 
> {noformat}
> 0: jdbc:phoenix:> create table a (id integer primary key, f float);
> No rows affected (2.272 seconds)
> 0: jdbc:phoenix:> create index i1 on a (f);
> No rows affected (5.769 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.021 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F  | :ID  |
> +--+--+
> | 0.5  | 1|
> +--+--+
> 1 row selected (0.016 seconds)
> 0: jdbc:phoenix:> delete from a where id = 1;
> 1 row affected (0.009 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +--+--+
> | 0:F  | :ID  |
> +--+--+
> +--+--+
> No rows selected (0.015 seconds)
> 0: jdbc:phoenix:> upsert into a values (1,0.5);
> 1 row affected (0.008 seconds)
> 0: jdbc:phoenix:> select * from i1;
> +---+--+
> |  0:F  | :ID  |
> +---+--+
> | null  | 1|
> +---+--+
> 1 row selected (0.013 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331649#comment-16331649
 ] 

James Taylor commented on PHOENIX-4539:
---

[~poornachandra] - any idea why our transaction tests pass using HBase 1.4 
without having a compat module? Is it because we're using 
InMemoryTxSystemClient in our unit tests (see TephraTransactionContext)? Should 
we be doing something different?

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331646#comment-16331646
 ] 

James Taylor commented on PHOENIX-4531:
---

Nice work, [~vincentpoon]. Thanks for tracking that down. Patch looks good. 
Minor nit: if it's not too difficult, would be good to have a 
QueryOptimizerTest that with an unhinted DELETE statement that favors the data 
table over the index table.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Vincent Poon (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331630#comment-16331630
 ] 

Vincent Poon commented on PHOENIX-4531:
---

[~jamestaylor] The hint was not using the data table because the optimizer 
chooses the smaller table (fewest non-pk columns), which is generally the 
index.  It also turns out that the logic was flipped for interpreting the hint.

I attached a v3 patch that has your patch plus:
 * -Changes to QueryOptimizer that resolves the above.  I only compare table 
size for queries that don't have a DATA_OVER_INDEX hint.  The rest of the logic 
stays the same such that a WHERE clause would cause the index to still be used.
 * Added a test in QueryOptimizerTest
 * Added my test in PartialIndexRebuilderIT to test that the index gets 
disabled properly when there's a write failure to the index when issuing 
deletes.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331628#comment-16331628
 ] 

Andrew Purtell commented on PHOENIX-4539:
-

Hmm.. Transactional tests not really testing what's intended?

I'll try to spend some time on this next week.

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Vincent Poon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vincent Poon updated PHOENIX-4531:
--
Attachment: PHOENIX-4531.v3.master.patch

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, 
> PHOENIX-4531.v3.master.patch, PHOENIX-4531_v1.patch, PHOENIX-4531_v2.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331618#comment-16331618
 ] 

James Taylor commented on PHOENIX-4539:
---

These failures don’t look related to transactions, but it’s a good point that 
the transaction related test will fail too until a Tephra release. Not sure why 
we’re not seeing those failures too.

Not sure what the best course of action is wrt reverting versus keeping. We’d 
need to fix before releasing, of course. If we keep, there’s some overhead in 
keeping the branch in sync, but I wouldn’t expect conflicts between branches.

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331581#comment-16331581
 ] 

Andrew Purtell commented on PHOENIX-4539:
-

Some of these may be due to Tephra not supporting 1.4? I filed TEPHRA-278 for 
that. It has been merged, but there isn't a release including it yet.

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331516#comment-16331516
 ] 

Andrew Purtell commented on PHOENIX-4539:
-

I don't have the bandwidth. Do you want to revert back to 1.3 and ignore 1.4 
for a while? Or push forward and fix tests as we have bandwidth?

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331485#comment-16331485
 ] 

James Taylor commented on PHOENIX-4539:
---

Please investigate, [~lhofhansl] and/or [~apurtell].

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4539:
--
Environment:(was: See 
https://builds.apache.org/job/Phoenix-master/1914. This failures are 
reproducible locally as well.)

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment:  
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4539:
--
Description: See https://builds.apache.org/job/Phoenix-master/1914. This 
failures are reproducible locally as well.

> Unit tests failures on move of master to HBase 1.4
> --
>
> Key: PHOENIX-4539
> URL: https://issues.apache.org/jira/browse/PHOENIX-4539
> Project: Phoenix
>  Issue Type: Bug
> Environment: See https://builds.apache.org/job/Phoenix-master/1914. 
> This failures are reproducible locally as well.
>Reporter: James Taylor
>Priority: Major
>
> See https://builds.apache.org/job/Phoenix-master/1914. This failures are 
> reproducible locally as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4539) Unit tests failures on move of master to HBase 1.4

[James Taylor (JIRA)]

James Taylor created PHOENIX-4539:
-

 Summary: Unit tests failures on move of master to HBase 1.4
 Key: PHOENIX-4539
 URL: https://issues.apache.org/jira/browse/PHOENIX-4539
 Project: Phoenix
  Issue Type: Bug
 Environment: See https://builds.apache.org/job/Phoenix-master/1914. 
This failures are reproducible locally as well.
Reporter: James Taylor






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

pre-commit Jenkins build isn't running - is this related to mast moved to HBase 1.4?

[James Taylor]

The pre-commit Jenkins job doesn't appear to be running. Do you think this
might be related to moving master to HBase 1.4? Another potential culprit
might be PHOENIX-4538.

WDYT, Lars?

Thanks,
James

[jira] [Updated] (PHOENIX-4538) Phoenix/Spark unit tests fail to compile

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4538:
--
Description: 
The phoenix-spark unit test fail to compile with these two commits for 
PHOENIX-4466: 
- 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
- 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d

If I revert those, the test compile and pass (FYI, I'm on a Mac, so not sure if 
this a contributing factor). This may be the reason our pre-commit Jenkins is 
busted too.

  was:
The phoenix-spark unit test fail to compile with these two commits for 
PHOENIX-4466: 
- 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
- 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d

If I revert those, the test compile and pass. I believe this is probably the 
reason our precommit Jenkins is busted too.


> Phoenix/Spark unit tests fail to compile
> 
>
> Key: PHOENIX-4538
> URL: https://issues.apache.org/jira/browse/PHOENIX-4538
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
>
> The phoenix-spark unit test fail to compile with these two commits for 
> PHOENIX-4466: 
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d
> If I revert those, the test compile and pass (FYI, I'm on a Mac, so not sure 
> if this a contributing factor). This may be the reason our pre-commit Jenkins 
> is busted too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4538) Phoenix/Spark unit tests fail to compile

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4538:
--
Summary: Phoenix/Spark unit tests fail to compile  (was: Phoenix/Spark unit 
tests broken)

> Phoenix/Spark unit tests fail to compile
> 
>
> Key: PHOENIX-4538
> URL: https://issues.apache.org/jira/browse/PHOENIX-4538
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
>
> The phoenix-spark unit test fail to compile with these two commits for 
> PHOENIX-4466: 
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d
> If I revert those, the test compile and pass. I believe this is probably the 
> reason our precommit Jenkins is busted too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4538) Phoenix/Spark unit tests broken

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331471#comment-16331471
 ] 

James Taylor commented on PHOENIX-4538:
---

[~elserj] - can you take a look at this please?

> Phoenix/Spark unit tests broken
> ---
>
> Key: PHOENIX-4538
> URL: https://issues.apache.org/jira/browse/PHOENIX-4538
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
>
> The phoenix-spark unit test fail to compile with these two commits for 
> PHOENIX-4466: 
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d
> If I revert those, the test compile and pass. I believe this is probably the 
> reason our precommit Jenkins is busted too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4538) Phoenix/Spark unit tests broken

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4538:
--
Description: 
The phoenix-spark unit test fail to compile with these two commits for 
PHOENIX-4466: 
- 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
- 
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d

If I revert those, the test compile and pass. I believe this is probably the 
reason our precommit Jenkins is busted too.

> Phoenix/Spark unit tests broken
> ---
>
> Key: PHOENIX-4538
> URL: https://issues.apache.org/jira/browse/PHOENIX-4538
> Project: Phoenix
>  Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
>
> The phoenix-spark unit test fail to compile with these two commits for 
> PHOENIX-4466: 
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=2136b002c37db478ffea11233f9ebb80276d2594
> - 
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commit;h=34693843abe4490b54fbd30512bf7d98d0f59c0d
> If I revert those, the test compile and pass. I believe this is probably the 
> reason our precommit Jenkins is busted too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331460#comment-16331460
 ] 

James Taylor commented on PHOENIX-4531:
---

The {{runOnServer}} variable gets set to false if the index table is chosen as 
the best table to drive the delete from. If it was kept as true, then the data 
table would be used. If you have a WHERE clause in the delete that turns into a 
range scan when the index is used, this is certainly going to be better than 
doing a full table scan using the data table. The case I'm not sure about is if 
there's no where clause (or the same number of rows would be scanned for the 
index or data table). I'm not sure which is better (but probably using the data 
table since as you mention the parallelization will be better). 

I think It needs further investigation as to why the hint didn't cause the data 
table to be chosen as the best plan.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, PHOENIX-4531_v1.patch, 
> PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Vincent Poon (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331433#comment-16331433
 ] 

Vincent Poon commented on PHOENIX-4531:
---

[~jamestaylor] My understanding was that we would could run it on server side 
because we trigger a scan over the data table, which then goes through 
UngroupedAggregateRegionObserver, which then issues mutates against the data 
table, which then causes the index updates to be issued via 
postBatchMutateIndispensably

That appears to be the case, because if you look at ServerSelectDeleteMutation, 
it uses the dataPlan, not the optimized queryPlan (which would use the index).  
When I try my patch which makes runOnServer=true, it does the right thing.

I'm not sure which approach is the best, but I thought we wanted to do it 
server-side for better parallelization, by issuing the delete scans in parallel 
with each table region concurrently deleting its own rows.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, PHOENIX-4531_v1.patch, 
> PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4538) Phoenix/Spark unit tests broken

[James Taylor (JIRA)]

James Taylor created PHOENIX-4538:
-

 Summary: Phoenix/Spark unit tests broken
 Key: PHOENIX-4538
 URL: https://issues.apache.org/jira/browse/PHOENIX-4538
 Project: Phoenix
  Issue Type: Bug
Reporter: James Taylor






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331400#comment-16331400
 ] 

James Taylor commented on PHOENIX-4537:
---

[~karanmehta93], [~tdsilva]

> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Sergey Soldatov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated PHOENIX-4537:
-
Description: 
[~sergey.soldatov] has been doing some great digging around a test failure 
we've been seeing at $dayjob. The situation goes like this.

0. Run some arbitrary load
1. Stop HBase
2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} and 
{{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
3. Start HBase
4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
before a client first connects

When the RegionServer initiates the compaction, it will end up running 
{{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
Phoenix connection. While the RegionServer won't upgrade system tables, it 
*will* try to migrate them into the schema mapped variants (e.g. SYSTEM.CATALOG 
to SYSTEM:CATALOG).

However, one of the first steps in the schema migration is to disable the 
SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled until 
the region is CLOSED, and the region cannot be CLOSED until the compaction is 
finished. *deadlock*

The "obvious" fix is to avoid RegionServers from triggering system table 
migrations, but Sergey and [~elserj] both think that this will end badly 
(RegionServers falling over because they expect the tables to be migrated and 
they aren't).

Thoughts? [~ankit.singhal], [~jamestaylor], any others?

  was:
[~sergey.soldatov] has been doing some great digging around a test failure 
we've been seeing at $dayjob. The situation goes like this.

0. Run some arbitrary load
1. Stop HBase
2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} and 
{{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
3. Start HBase
4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
before a client first connects

When the RegionServer initiates the compaction, it will end up running 
{{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
Phoenix connection. While the RegionServer won't upgrade system tables, it 
*will* try to migrate them into the schema mapped variants (e.g. SYSTEM.CATALOG 
to SYSTEM:CATALOG).

However, one of the first steps in the schema migration is to disable the 
SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled until 
the region is CLOSED, and the region cannot be CLOSED until the compaction is 
finished. *deadlock*

The "obvious" fix is to avoid RegionServers from triggering system table 
migrations, but Sergey and I both think that this will end badly (RegionServers 
falling over because they expect the tables to be migrated and they aren't).

Thoughts? [~ankit.singhal], [~jamestaylor], any others?


> RegionServer initiating compaction can trigger schema migration and deadlock 
> the system
> ---
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Romil Choksi
>Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure 
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} 
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running 
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
> Phoenix connection. While the RegionServer won't upgrade system tables, it 
> *will* try to migrate them into the schema mapped variants (e.g. 
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the 
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled 
> until the region is CLOSED, and the region cannot be CLOSED until the 
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table 
> migrations, but Sergey and [~elserj] both think that this will end badly 
> (RegionServers falling over because they expect the tables to be migrated and 
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system

[Josh Elser (JIRA)]

Josh Elser created PHOENIX-4537:
---

 Summary: RegionServer initiating compaction can trigger schema 
migration and deadlock the system
 Key: PHOENIX-4537
 URL: https://issues.apache.org/jira/browse/PHOENIX-4537
 Project: Phoenix
  Issue Type: Bug
Reporter: Romil Choksi
 Fix For: 5.0.0, 4.14.0


[~sergey.soldatov] has been doing some great digging around a test failure 
we've been seeing at $dayjob. The situation goes like this.

0. Run some arbitrary load
1. Stop HBase
2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} and 
{{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
3. Start HBase
4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run 
before a client first connects

When the RegionServer initiates the compaction, it will end up running 
{{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a 
Phoenix connection. While the RegionServer won't upgrade system tables, it 
*will* try to migrate them into the schema mapped variants (e.g. SYSTEM.CATALOG 
to SYSTEM:CATALOG).

However, one of the first steps in the schema migration is to disable the 
SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled until 
the region is CLOSED, and the region cannot be CLOSED until the compaction is 
finished. *deadlock*

The "obvious" fix is to avoid RegionServers from triggering system table 
migrations, but Sergey and I both think that this will end badly (RegionServers 
falling over because they expect the tables to be migrated and they aren't).

Thoughts? [~ankit.singhal], [~jamestaylor], any others?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4536) Change getWAL usage due HBASE-19751

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331240#comment-16331240
 ] 

Josh Elser commented on PHOENIX-4536:
-

+1

> Change getWAL usage due HBASE-19751
> ---
>
> Key: PHOENIX-4536
> URL: https://issues.apache.org/jira/browse/PHOENIX-4536
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4536.patch
>
>
> in HBASE-19751 WALFactory#getWAL signature has been changed from using byte[] 
> to RegionInfo. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4536) Change getWAL usage due HBASE-19751

[Sergey Soldatov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated PHOENIX-4536:
-
Attachment: PHOENIX-4536.patch

> Change getWAL usage due HBASE-19751
> ---
>
> Key: PHOENIX-4536
> URL: https://issues.apache.org/jira/browse/PHOENIX-4536
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0
>
> Attachments: PHOENIX-4536.patch
>
>
> in HBASE-19751 WALFactory#getWAL signature has been changed from using byte[] 
> to RegionInfo. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4535) Index in array data type is inconsistent.

[Sergey Soldatov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated PHOENIX-4535:
-
Labels: HBase-2.0  (was: )

> Index in array data type is inconsistent. 
> --
>
> Key: PHOENIX-4535
> URL: https://issues.apache.org/jira/browse/PHOENIX-4535
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0, 4.13.0
>Reporter: Romil Choksi
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0
>
>
> We allow using zero indexes for elements in array column and returns null by 
> accident. 
> A simple test case to reproduce:
> {noformat}
> Properties props = PropertiesUtil.deepCopy(TEST_PROPERTIES);
> Connection conn = DriverManager.getConnection(getUrl(), props);
> conn.createStatement().execute("create table A(ID INTEGER PRIMARY 
> KEY, array_id VARCHAR[])");
> conn.createStatement().execute("upsert into A values (1, 
> ARRAY['test','test2','test3'])");
> conn.commit();
> ResultSet rs = conn.createStatement().executeQuery("select 
> array_id[0], array_id[1] from A");
> while(rs.next()) {
> System.out.println(rs.getString(1));
> System.out.println(rs.getString(2));
> }
> {noformat}
> The result for 4.x branches would be {null, 'test'} and it would fail with an 
> exception for 5.x.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4536) Change getWAL usage due HBASE-19751

[Sergey Soldatov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated PHOENIX-4536:
-
Labels: HBase-2.0  (was: )

> Change getWAL usage due HBASE-19751
> ---
>
> Key: PHOENIX-4536
> URL: https://issues.apache.org/jira/browse/PHOENIX-4536
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0
>
>
> in HBASE-19751 WALFactory#getWAL signature has been changed from using byte[] 
> to RegionInfo. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (PHOENIX-4536) Change getWAL usage due HBASE-19751

[Sergey Soldatov (JIRA)]

Sergey Soldatov created PHOENIX-4536:


 Summary: Change getWAL usage due HBASE-19751
 Key: PHOENIX-4536
 URL: https://issues.apache.org/jira/browse/PHOENIX-4536
 Project: Phoenix
  Issue Type: Bug
Affects Versions: 5.0
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
 Fix For: 5.0


in HBASE-19751 WALFactory#getWAL signature has been changed from using byte[] 
to RegionInfo. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 5.0.0-beta release before month's end?

[James Taylor]

Awesome! That's great work!!

On Thu, Jan 18, 2018 at 1:00 PM, Josh Elser  wrote:

> Hah, funny you should ask! I was just thinking that I should send out a
> note.
>
> * Rajeshbabu and Sergey are trying to track down a nasty issue that the
> IndexScrutiny tool has caught where there are dangling index records
> (PHOENIX-4534)
> * Rajeshbabu is also looking into some local index failures
> * Ankit has done some testing of the phoenix-hive integration, I think
> phoenix-spark is also looking OK
>
> Last I chatted with folks, they were seeing >90% pass rate of the test
> suite which seems pretty good to me. My plan was to work on a beta release
> soon now that we have a beta from HBase. Things are generally functional as
> of now -- I think getting it into the hands of folks to get more people
> poking it would be great.
>
> Maybe rc0 next week?
>
> - Josh
>
>
> On 1/18/18 1:11 PM, James Taylor wrote:
>
>> How are things looking with the 5.0.0 alpha/beta on HBase 2.x?
>>
>> On Thu, Jan 4, 2018 at 10:57 AM, Josh Elser  wrote:
>>
>> Good point. Perhaps "alpha" would be a better label?
>>>
>>> IIUC, the issue is that we need the HBase release, and then a Tephra
>>> release, and then we can get Tephra fixed for Phoenix5. Perhaps Ankit can
>>> provide some more color to the situation.
>>>
>>>
>>> On 1/4/18 12:07 PM, Nick Dimiduk wrote:
>>>
>>> Isn't Tephra integration mandatory for transaction support? What happens
 to
 a user who has TRANSACTIONAL=true tables when they upgrade? This can't
 really fail gracefully. I guess transaction support is still marked
 'beta',
 but still, this would be a regression of functionality in "base
 Phoenix".

 On Thu, Jan 4, 2018 at 8:34 AM Josh Elser  wrote:

 Talked to Rajeshbabu and Ankit offline this morning.

>
> Sounds like there are a few integration points which are still lacking:
>
> * phoenix-hive: PHOENIX-4423
> * phoenix-spark: untested (probably broken against newest Spark)
> * phoenix-kafka: untested (probably broken against newest Kafka -- see
> PHOENIX-4515 PHOENIX-4516)
> * Tephra integration: Needs a new release of Tephra with some fixes
> Ankit helped with.
>
> I plan to not consider these 5.0.0-alpha/beta release blockers, we'll
> just call those out which we don't get tested/fixed.
>
> On 1/2/18 1:08 PM, Josh Elser wrote:
>
> Happy New Year folks!
>>
>> I'd like to test the waters: what do people think about trying to get
>> a
>> 5.0.0 "beta" release out to the community before the end of January?
>>
>> HBase is doing the same right now with 2.0.0. My thinking is that if
>> things are stable "enough", getting a base for people to use a 5.0
>> Phoenix release more easily, we can catch more bugs and get a better
>> product out the door.
>>
>> Thoughts/concerns? I'm happy to RM.
>>
>> - Josh
>>
>>
>
>

>>

Re: [DISCUSS] 5.0.0-beta release before month's end?

[Josh Elser]

Hah, funny you should ask! I was just thinking that I should send out a 
note.


* Rajeshbabu and Sergey are trying to track down a nasty issue that the 
IndexScrutiny tool has caught where there are dangling index records 
(PHOENIX-4534)

* Rajeshbabu is also looking into some local index failures
* Ankit has done some testing of the phoenix-hive integration, I think 
phoenix-spark is also looking OK


Last I chatted with folks, they were seeing >90% pass rate of the test 
suite which seems pretty good to me. My plan was to work on a beta 
release soon now that we have a beta from HBase. Things are generally 
functional as of now -- I think getting it into the hands of folks to 
get more people poking it would be great.


Maybe rc0 next week?

- Josh

On 1/18/18 1:11 PM, James Taylor wrote:

How are things looking with the 5.0.0 alpha/beta on HBase 2.x?

On Thu, Jan 4, 2018 at 10:57 AM, Josh Elser  wrote:


Good point. Perhaps "alpha" would be a better label?

IIUC, the issue is that we need the HBase release, and then a Tephra
release, and then we can get Tephra fixed for Phoenix5. Perhaps Ankit can
provide some more color to the situation.


On 1/4/18 12:07 PM, Nick Dimiduk wrote:


Isn't Tephra integration mandatory for transaction support? What happens
to
a user who has TRANSACTIONAL=true tables when they upgrade? This can't
really fail gracefully. I guess transaction support is still marked
'beta',
but still, this would be a regression of functionality in "base Phoenix".

On Thu, Jan 4, 2018 at 8:34 AM Josh Elser  wrote:

Talked to Rajeshbabu and Ankit offline this morning.


Sounds like there are a few integration points which are still lacking:

* phoenix-hive: PHOENIX-4423
* phoenix-spark: untested (probably broken against newest Spark)
* phoenix-kafka: untested (probably broken against newest Kafka -- see
PHOENIX-4515 PHOENIX-4516)
* Tephra integration: Needs a new release of Tephra with some fixes
Ankit helped with.

I plan to not consider these 5.0.0-alpha/beta release blockers, we'll
just call those out which we don't get tested/fixed.

On 1/2/18 1:08 PM, Josh Elser wrote:


Happy New Year folks!

I'd like to test the waters: what do people think about trying to get a
5.0.0 "beta" release out to the community before the end of January?

HBase is doing the same right now with 2.0.0. My thinking is that if
things are stable "enough", getting a base for people to use a 5.0
Phoenix release more easily, we can catch more bugs and get a better
product out the door.

Thoughts/concerns? I'm happy to RM.

- Josh

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331217#comment-16331217
 ] 

James Taylor commented on PHOENIX-4531:
---

Attached v2 patch with more comments and some cleanup to code that determines 
the update count returned to the client. Looks like our pre-commit is busted 
(or on vacation). I suspect it's due to the move to HBase 1.4 on master (but it 
could be coincidence).

Please review, [~vincentpoon].

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, PHOENIX-4531_v1.patch, 
> PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Taylor updated PHOENIX-4531:
--
Attachment: PHOENIX-4531_v2.patch

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, PHOENIX-4531_v1.patch, 
> PHOENIX-4531_v2.patch, PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4378) Unable to set KEEP_DELETED_CELLS to true on RS scanner

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331199#comment-16331199
 ] 

James Taylor commented on PHOENIX-4378:
---

Ping [~an...@apache.org] - please file and/or point us to an HBase JIRA for 
this regression.

> Unable to set KEEP_DELETED_CELLS to true on RS scanner
> --
>
> Key: PHOENIX-4378
> URL: https://issues.apache.org/jira/browse/PHOENIX-4378
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Ankit Singhal
>Priority: Major
>  Labels: HBase-2.0
> Fix For: 5.0.0
>
>
> [~jamestaylor], 
> It seems we may need to fix PHOENIX-4277 differently for HBase 2.0 as we can 
> only update TTL and maxVersions now in preStoreScannerOpen and cannot return 
> a new StoreScanner with updated scanInfo.
> for reference:
> [1]https://issues.apache.org/jira/browse/PHOENIX-4318?focusedCommentId=16249943=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16249943



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[James Taylor (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331138#comment-16331138
 ] 

James Taylor commented on PHOENIX-4531:
---

If the scan run for the delete is over an index table, we can't (currently) run 
that on the server side because we don't have the code that'll translate from 
the index row key to the data row key on the server side. In theory, if the 
index is local or global and mutable, we could run it on the server side if 
this code was there.

We do attempt to favor using the data table over the index table if all things 
are equal from the optimizers POV. That's what this code a little above in 
DeleteCompiler should do:
{code}
if (runOnServer && 
!delete.getHint().hasHint(Hint.USE_INDEX_OVER_DATA_TABLE)) {
select = SelectStatement.create(select, HintNode.create(hint, 
Hint.USE_DATA_OVER_INDEX_TABLE));
}
{code}
I'd expect that hint to cause the data table, not the index table to be used if 
there's no WHERE clause. Perhaps this isn't working as expected?

It's not entirely clear what the best choice is, though. If the index table is 
much smaller in size than the data table, is it better to execute the scan 
against the index table, pull back over the row key and issue the deletes from 
the client? Or is it better to execute the delete on the server side (though 
it'd be a cross RS call)?

WDYT? Maybe if this is pursued it should be a separate JIRA, though?

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, PHOENIX-4531_v1.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4531) Delete on a table with a global mutable index can issue client-side deletes against the index

[Vincent Poon (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331091#comment-16331091
 ] 

Vincent Poon commented on PHOENIX-4531:
---

[~jamestaylor] your patch works in terms of preventing any mutations against 
the index, but if I issue a "DELETE FROM data_table", it uses uses 
ClientSelectDeleteMutationPlan, instead of ServerSelectDeleteMutationPlan.  Is 
that what we want?

You're right that we have the initialization of runOnServer that checks for 
hasImmutableIndexes, but again, after the initialization, we run:
{code:java}
runOnServer &= queryPlans.get(0).getTableRef().getTable().getType() != 
PTableType.INDEX; {code}
So if the query optimizer came back with a plan to use the index table, even 
for mutable indexes, runOnServer becomes false.

> Delete on a table with a global mutable index can issue client-side deletes 
> against the index
> -
>
> Key: PHOENIX-4531
> URL: https://issues.apache.org/jira/browse/PHOENIX-4531
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 4.13.0
> Environment:  
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>Priority: Major
> Attachments: PHOENIX-4531.v1.master.patch, PHOENIX-4531_v1.patch, 
> PartialIndexRebuilderIT.java
>
>
> For a table with a global mutable index, I found the following result in 
> client-side deletes against both the data table and index table.
> "DELETE FROM data_table" 
> "DELETE FROM data_table WHERE indexed_col='v'"
> We only need the delete to be issued against the data table, because
> 1) It's redundant since a delete against the index will be issued on the 
> server side when we process the delete of the data table row
> 2) Deletes issued from the client-side won't have the index failure policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 5.0.0-beta release before month's end?

[James Taylor]

How are things looking with the 5.0.0 alpha/beta on HBase 2.x?

On Thu, Jan 4, 2018 at 10:57 AM, Josh Elser  wrote:

> Good point. Perhaps "alpha" would be a better label?
>
> IIUC, the issue is that we need the HBase release, and then a Tephra
> release, and then we can get Tephra fixed for Phoenix5. Perhaps Ankit can
> provide some more color to the situation.
>
>
> On 1/4/18 12:07 PM, Nick Dimiduk wrote:
>
>> Isn't Tephra integration mandatory for transaction support? What happens
>> to
>> a user who has TRANSACTIONAL=true tables when they upgrade? This can't
>> really fail gracefully. I guess transaction support is still marked
>> 'beta',
>> but still, this would be a regression of functionality in "base Phoenix".
>>
>> On Thu, Jan 4, 2018 at 8:34 AM Josh Elser  wrote:
>>
>> Talked to Rajeshbabu and Ankit offline this morning.
>>>
>>> Sounds like there are a few integration points which are still lacking:
>>>
>>> * phoenix-hive: PHOENIX-4423
>>> * phoenix-spark: untested (probably broken against newest Spark)
>>> * phoenix-kafka: untested (probably broken against newest Kafka -- see
>>> PHOENIX-4515 PHOENIX-4516)
>>> * Tephra integration: Needs a new release of Tephra with some fixes
>>> Ankit helped with.
>>>
>>> I plan to not consider these 5.0.0-alpha/beta release blockers, we'll
>>> just call those out which we don't get tested/fixed.
>>>
>>> On 1/2/18 1:08 PM, Josh Elser wrote:
>>>
 Happy New Year folks!

 I'd like to test the waters: what do people think about trying to get a
 5.0.0 "beta" release out to the community before the end of January?

 HBase is doing the same right now with 2.0.0. My thinking is that if
 things are stable "enough", getting a base for people to use a 5.0
 Phoenix release more easily, we can catch more bugs and get a better
 product out the door.

 Thoughts/concerns? I'm happy to RM.

 - Josh

>>>
>>>
>>