[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472825#comment-16472825
]
Xu Cang commented on PHOENIX-4726:
--
uploaded new patch. It addressed Vincent's comment.
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.4.patch, PHOENIX-4726.patch.1,
> PHOENIX-4726.patch.2, PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated PHOENIX-4726:
-
Attachment: PHOENIX-4726.4.patch
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.4.patch, PHOENIX-4726.patch.1,
> PHOENIX-4726.patch.2, PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
[
https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472824#comment-16472824
]
Hudson commented on PHOENIX-4685:
-
FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1887 (See
[https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1887/])
PHOENIX-4685 Properly handle connection caching for Phoenix inside (rajeshbabu:
rev 829f3fcc82c6a00910db5b21b85aea90f9f0afbf)
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
> Properly handle connection caching for Phoenix inside RegionServers
> ---
>
> Key: PHOENIX-4685
> URL: https://issues.apache.org/jira/browse/PHOENIX-4685
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch,
> PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch,
> PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where
> unable to create native threads. But it's working fine with 4.7.x branches.
> Found many threads created for meta lookup and shared threads and no space to
> create threads. This is happening even with short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN
> [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>
[jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
[
https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472794#comment-16472794
]
Hudson commented on PHOENIX-4685:
-
ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #126 (See
[https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/126/])
PHOENIX-4685 Properly handle connection caching for Phoenix inside (rajeshbabu:
rev 39b92bf9e8d9cae46b1fa230d91ac04a8e49e629)
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
> Properly handle connection caching for Phoenix inside RegionServers
> ---
>
> Key: PHOENIX-4685
> URL: https://issues.apache.org/jira/browse/PHOENIX-4685
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch,
> PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch,
> PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where
> unable to create native threads. But it's working fine with 4.7.x branches.
> Found many threads created for meta lookup and shared threads and no space to
> create threads. This is happening even with short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN
> [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>
[jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[
https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472710#comment-16472710
]
Ankit Singhal commented on PHOENIX-4701:
{quote}I suspect it's for the write metrics. That'd be good to add in a follow
on JIRA.
{quote}
Yes, currently we don't store write metrics or any metric only collected at a
connection level.
{quote}Do you think we have enough testing to commit this?
{quote}
I can test it over the weekend and commit if you are not planning the RC today.
bq. probably better to use EnvironmentEdgeManager.currentTimeMillis() instead
of System.currentTimeMillis() here and elsewhere. Will make testing easier.
Sure, I'll make that change.
> Write client-side metrics asynchronously to SYSTEM.LOG
> --
>
> Key: PHOENIX-4701
> URL: https://issues.apache.org/jira/browse/PHOENIX-4701
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
> Fix For: 4.15.0
>
> Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch,
> PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch
>
>
> Rather than inventing a new, different set of client-side metrics to persist,
> we should just persist our [client
> metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The
> metrics captures all the same information as your QueryLogInfo (and much
> more), rolls all the information up to a single set of metrics for each
> Phoenix statement (aggregating/merging parallel scans, etc), and can emits a
> single log line (which could be written in a single upsert statement). At
> SFDC, we emit this information into a file system log in a layer above (and
> use Splunk to produce nifty dashboard for monitoring), but this could easily
> be emitted directly in Phoenix and go through your asynchronous write path
> (and then use Phoenix queries to produce the same kind of dashboards). The
> only piece would be to add the concept of a log level to each metric to
> enable statically controlling which metrics are output.
> With this approach, the SYSTEM.LOG table could be declared immutable and use
> our dense storage format with a single byte for column encoding and get a
> 3-5x perf gain. This would also open the door for users to potentially add
> secondary indexes on the table. See schema identified in the wip2 patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[
https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472690#comment-16472690
]
James Taylor commented on PHOENIX-4701:
---
One minor thing, [~an...@apache.org], probably better to use
EnvironmentEdgeManager.currentTimeMillis() instead of
System.currentTimeMillis() here and elsewhere. Will make testing easier.
{code:java}
private QueryLogger(PhoenixConnection connection) {
this.queryId = UUID.randomUUID().toString();
this.queryDisruptor = connection.getQueryServices().getQueryDisruptor();
- this.startTime = System.currentTimeMillis();
logLevel = connection.getLogLevel();
+ log(QueryLogInfo.QUERY_ID_I, queryId);
+ log(QueryLogInfo.START_TIME_I, System.currentTimeMillis());
{code}
> Write client-side metrics asynchronously to SYSTEM.LOG
> --
>
> Key: PHOENIX-4701
> URL: https://issues.apache.org/jira/browse/PHOENIX-4701
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
> Fix For: 4.15.0
>
> Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch,
> PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch
>
>
> Rather than inventing a new, different set of client-side metrics to persist,
> we should just persist our [client
> metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The
> metrics captures all the same information as your QueryLogInfo (and much
> more), rolls all the information up to a single set of metrics for each
> Phoenix statement (aggregating/merging parallel scans, etc), and can emits a
> single log line (which could be written in a single upsert statement). At
> SFDC, we emit this information into a file system log in a layer above (and
> use Splunk to produce nifty dashboard for monitoring), but this could easily
> be emitted directly in Phoenix and go through your asynchronous write path
> (and then use Phoenix queries to produce the same kind of dashboards). The
> only piece would be to add the concept of a log level to each metric to
> enable statically controlling which metrics are output.
> With this approach, the SYSTEM.LOG table could be declared immutable and use
> our dense storage format with a single byte for column encoding and get a
> 3-5x perf gain. This would also open the door for users to potentially add
> secondary indexes on the table. See schema identified in the wip2 patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472667#comment-16472667
]
Xu Cang commented on PHOENIX-4726:
--
"Never mind - I see now you're using a dynamic column. Is that the way we want
it to be? "
– Xu: Yes, we had 2 similar timestamps already as dynamic columns.
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472668#comment-16472668
]
Xu Cang commented on PHOENIX-4726:
--
"[~xucang] Can you use EnvironmentEdgeManager.currentTimeMillis() instead of
System.currentTimeMillis() ?
Thanks!"
--Xu: will do.
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4734) SQL Query with an RVC expression lexographically higher than all values in an OR clause causes query to blow up
[ https://issues.apache.org/jira/browse/PHOENIX-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472646#comment-16472646 ] Hudson commented on PHOENIX-4734: - ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1886 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1886/]) PHOENIX-4734 SQL Query with an RVC expression lexographically higher (tdsilva: rev faacf046998b89115a72c968ba4458820be91e64) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/RowValueConstructorIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java > SQL Query with an RVC expression lexographically higher than all values in an > OR clause causes query to blow up > --- > > Key: PHOENIX-4734 > URL: https://issues.apache.org/jira/browse/PHOENIX-4734 > Project: Phoenix > Issue Type: Bug >Reporter: Jan Fernando >Assignee: Thomas D'Silva >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4734.patch, SqlInClauseIssueIT.java > > > See Attached unit test for repro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4734) SQL Query with an RVC expression lexographically higher than all values in an OR clause causes query to blow up
[ https://issues.apache.org/jira/browse/PHOENIX-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472555#comment-16472555 ] Hudson commented on PHOENIX-4734: - FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1883 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1883/]) PHOENIX-4734 SQL Query with an RVC expression lexographically higher (tdsilva: rev 0c8349e3c01a9f1e18e2ebd565833bd448271d9e) * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/ScanRanges.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/RowValueConstructorIT.java > SQL Query with an RVC expression lexographically higher than all values in an > OR clause causes query to blow up > --- > > Key: PHOENIX-4734 > URL: https://issues.apache.org/jira/browse/PHOENIX-4734 > Project: Phoenix > Issue Type: Bug >Reporter: Jan Fernando >Assignee: Thomas D'Silva >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4734.patch, SqlInClauseIssueIT.java > > > See Attached unit test for repro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
[
https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472556#comment-16472556
]
Hudson commented on PHOENIX-4685:
-
FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1883 (See
[https://builds.apache.org/job/PreCommit-PHOENIX-Build/1883/])
PHOENIX-4685 Properly handle connection caching for Phoenix inside (rajeshbabu:
rev 56f109603cec93f3904366d4bb23415981947ae0)
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
> Properly handle connection caching for Phoenix inside RegionServers
> ---
>
> Key: PHOENIX-4685
> URL: https://issues.apache.org/jira/browse/PHOENIX-4685
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch,
> PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch,
> PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where
> unable to create native threads. But it's working fine with 4.7.x branches.
> Found many threads created for meta lookup and shared threads and no space to
> create threads. This is happening even with short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN
> [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>
[jira] [Commented] (PHOENIX-4733) NPE while running sql through file using psql
[
https://issues.apache.org/jira/browse/PHOENIX-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472557#comment-16472557
]
Hudson commented on PHOENIX-4733:
-
FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1883 (See
[https://builds.apache.org/job/PreCommit-PHOENIX-Build/1883/])
PHOENIX-4733 NPE while running sql through file using psql (ankitsinghal59: rev
f31c6cdb27fae6b263ab18c84626c37ad48d80e7)
* (edit) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerUtil.java
> NPE while running sql through file using psql
> -
>
> Key: PHOENIX-4733
> URL: https://issues.apache.org/jira/browse/PHOENIX-4733
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.14.0, 5.0.0
>Reporter: Srikanth Janardhan
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4733.patch
>
>
> {code:java}
> cat /tmp/test.sql
> CREATE TABLE IF NOT EXISTS QETEST (ID INTEGER NOT NULL PRIMARY KEY, A
> VARCHAR, B INTEGER);
> upsert into QETEST VALUES(1,'A',10);
> upsert into QETEST VALUES(2,'B',1000);
> upsert into QETEST VALUES(3,'A',20);
> upsert into QETEST VALUES(4,'A',100);
> upsert into QETEST VALUES(5,'B',9000);
> SELECT A||'_GROUP' AS GRP,SUM(B)||'_RESULT' AS SUM FROM QETEST GROUP BY A;
> DROP TABLE QETEST;{code}
> bin/psql.py localhost /tmp/test.sql
> {code:java}
> no rows upserted
> Time: 0.858 sec(s)
> 1 row upserted
> Time: 0.04 sec(s)
> 1 row upserted
> Time: 0.004 sec(s)
> 1 row upserted
> Time: 0.006 sec(s)
> 1 row upserted
> Time: 0.004 sec(s)
> 1 row upserted
> Time: 0.004 sec(s)
> java.lang.NullPointerException: null value in entry: QUERY_I=null
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:235)
> at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:144)
> at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:182)
> at
> org.apache.phoenix.log.QueryLoggerUtil.getInitialDetails(QueryLoggerUtil.java:50)
> at
> org.apache.phoenix.log.QueryLoggerUtil.logInitialDetails(QueryLoggerUtil.java:36)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.createQueryLogger(PhoenixStatement.java:1783)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
> at
> org.apache.phoenix.jdbc.PhoenixConnection.executeStatements(PhoenixConnection.java:468)
> at
> org.apache.phoenix.util.PhoenixRuntime.executeStatements(PhoenixRuntime.java:348)
> at org.apache.phoenix.util.PhoenixRuntime.main(PhoenixRuntime.java:295){code}
> FYI [~jamestaylor] , if you see it a blocker for 4.14.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472531#comment-16472531
]
Vincent Poon commented on PHOENIX-4726:
---
Sure, will do
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472527#comment-16472527
]
James Taylor commented on PHOENIX-4726:
---
FYI, we're getting ready to roll the next RC for 4.14. Would you mind checking
this in, [~vincentpoon] (assuming it's a good one to have in there)?
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472525#comment-16472525
]
James Taylor commented on PHOENIX-4726:
---
Never mind - I see now you're using a dynamic column. Is that the way we want
it to be?
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[
https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472522#comment-16472522
]
James Taylor commented on PHOENIX-4701:
---
{quote}Not sure why PhoenixRuntime.resetMetrics(rs) is required as every query
has it's own instance of ReadMetricQueue and OverAllQueryMetrics
{quote}
I suspect it's for the write metrics. That'd be good to add in a follow on JIRA.
Do you think we have enough testing to commit this?
> Write client-side metrics asynchronously to SYSTEM.LOG
> --
>
> Key: PHOENIX-4701
> URL: https://issues.apache.org/jira/browse/PHOENIX-4701
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
> Fix For: 4.15.0
>
> Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch,
> PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch
>
>
> Rather than inventing a new, different set of client-side metrics to persist,
> we should just persist our [client
> metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The
> metrics captures all the same information as your QueryLogInfo (and much
> more), rolls all the information up to a single set of metrics for each
> Phoenix statement (aggregating/merging parallel scans, etc), and can emits a
> single log line (which could be written in a single upsert statement). At
> SFDC, we emit this information into a file system log in a layer above (and
> use Splunk to produce nifty dashboard for monitoring), but this could easily
> be emitted directly in Phoenix and go through your asynchronous write path
> (and then use Phoenix queries to produce the same kind of dashboards). The
> only piece would be to add the concept of a log level to each metric to
> enable statically controlling which metrics are output.
> With this approach, the SYSTEM.LOG table could be declared immutable and use
> our dense storage format with a single byte for column encoding and get a
> 3-5x perf gain. This would also open the door for users to potentially add
> secondary indexes on the table. See schema identified in the wip2 patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472523#comment-16472523
]
Vincent Poon commented on PHOENIX-4726:
---
[~jamestaylor] this is a dynamic column - is it still required to do
addColumnIfNotExists for that ?
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472518#comment-16472518
]
James Taylor commented on PHOENIX-4726:
---
[~xucang]. Thanks for the patch. You need to handle the upgrade path too.
That's what handles the incremental schema changes to SYSTEM.CATALOG when you
upgrade to a new minor release. See
ConnectionQueryServicesImpl.upgradeSystemCatalogIfRequired(). You'll need to
add another addColumnIfNotExists call here:
{code:java}
if (currentServerSideTableTimeStamp <
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_14_0) {
metaConnection = addColumnsIfNotExists(
metaConnection,
PhoenixDatabaseMetaData.SYSTEM_CATALOG,
MetaDataProtocol.MIN_SYSTEM_TABLE_TIMESTAMP_4_14_0,
PhoenixDatabaseMetaData.TRANSACTION_PROVIDER + " "
+ PTinyint.INSTANCE.getSqlTypeName());{code}
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[
https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472497#comment-16472497
]
Ankit Singhal commented on PHOENIX-4701:
{quote}Do the test in QueryLoggerIT pass?
{quote}
Yes, it was passing with wip and current patch both
{quote}
If you're ok with it, I think we should get this into 4.14. We can mark it as
beta while we get feedback. It can be disabled completely, correct
{quote}
Yes, I think we should have this in 4.14 to avoid upgrade for log table in
later releases. Beta should be fine as it can be disabled completely.
{quote}
Did you see the LoggingPhoenixConnection and others like
LoggingPhoenixResultSet and see how they manage the lifecycle of metrics?
{quote}
Yes I checked that, Not sure why PhoenixRuntime.resetMetrics(rs) is required as
every query has it's own instance of ReadMetricQueue and OverAllQueryMetrics.
> Write client-side metrics asynchronously to SYSTEM.LOG
> --
>
> Key: PHOENIX-4701
> URL: https://issues.apache.org/jira/browse/PHOENIX-4701
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
>Assignee: James Taylor
>Priority: Major
> Fix For: 4.15.0
>
> Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch,
> PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch
>
>
> Rather than inventing a new, different set of client-side metrics to persist,
> we should just persist our [client
> metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The
> metrics captures all the same information as your QueryLogInfo (and much
> more), rolls all the information up to a single set of metrics for each
> Phoenix statement (aggregating/merging parallel scans, etc), and can emits a
> single log line (which could be written in a single upsert statement). At
> SFDC, we emit this information into a file system log in a layer above (and
> use Splunk to produce nifty dashboard for monitoring), but this could easily
> be emitted directly in Phoenix and go through your asynchronous write path
> (and then use Phoenix queries to produce the same kind of dashboards). The
> only piece would be to add the concept of a log level to each metric to
> enable statically controlling which metrics are output.
> With this approach, the SYSTEM.LOG table could be declared immutable and use
> our dense storage format with a single byte for column encoding and get a
> 3-5x perf gain. This would also open the door for users to potentially add
> secondary indexes on the table. See schema identified in the wip2 patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472481#comment-16472481
]
Vincent Poon commented on PHOENIX-4726:
---
[~xucang] Can you use EnvironmentEdgeManager.currentTimeMillis() instead of
System.currentTimeMillis() ?
Thanks!
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701_master.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4726) save index build timestamp -- for SYNC case only.
[
https://issues.apache.org/jira/browse/PHOENIX-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472474#comment-16472474
]
Vincent Poon commented on PHOENIX-4726:
---
+1 [~xucang] , thanks for the patch
> save index build timestamp -- for SYNC case only.
> -
>
> Key: PHOENIX-4726
> URL: https://issues.apache.org/jira/browse/PHOENIX-4726
> Project: Phoenix
> Issue Type: Improvement
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Attachments: PHOENIX-4726.patch.1, PHOENIX-4726.patch.2,
> PHOENIX-4726.patch.3
>
>
> save index build timestamp, similar to ASYNC_REBUILD_TIMESTAMP, or
> ASYNC_CREATED_DATE
> ("SYNC_INDEX_CREATED_DATE" is my proposed name for SYNC case.)
>
> Check IndexUtil.java for related code.
> The reason this can be useful is: We saw a case index state stuck in 'b' for
> quite some long time. And without a timestamp to indicate where it started,
> it's hard to tell if this is a legit running task or stuck...
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472471#comment-16472471 ] James Taylor commented on PHOENIX-4701: --- Some specific questions: * Did you see the LoggingPhoenixConnection and others like LoggingPhoenixResultSet and see how they manage the lifecycle of metrics? * Any need to call PhoenixRuntime.resetMetrics(rs) as they do? > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_wip1.patch, > PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_wip1.patch, > PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472406#comment-16472406 ] James Taylor commented on PHOENIX-4701: --- [~samarthjain] - any chance you could give this a quick look over? > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, > PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4734) SQL Query with an RVC expression lexographically higher than all values in an OR clause causes query to blow up
[ https://issues.apache.org/jira/browse/PHOENIX-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-4734: Fix Version/s: 5.0.0 4.14.0 > SQL Query with an RVC expression lexographically higher than all values in an > OR clause causes query to blow up > --- > > Key: PHOENIX-4734 > URL: https://issues.apache.org/jira/browse/PHOENIX-4734 > Project: Phoenix > Issue Type: Bug >Reporter: Jan Fernando >Assignee: Thomas D'Silva >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4734.patch, SqlInClauseIssueIT.java > > > See Attached unit test for repro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
[
https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472003#comment-16472003
]
James Taylor commented on PHOENIX-4685:
---
+1. Thanks for the fix, [~rajeshbabu].
> Properly handle connection caching for Phoenix inside RegionServers
> ---
>
> Key: PHOENIX-4685
> URL: https://issues.apache.org/jira/browse/PHOENIX-4685
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch,
> PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch,
> PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where
> unable to create native threads. But it's working fine with 4.7.x branches.
> Found many threads created for meta lookup and shared threads and no space to
> create threads. This is happening even with short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN
> [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40)
> at
[jira] [Commented] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
[
https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471981#comment-16471981
]
Rajeshbabu Chintaguntla commented on PHOENIX-4685:
--
[~jamestaylor] uploaded addendum fixes it.
bq. Rajeshbabu Chintaguntla I'm mostly fine with the solution in this patch,
but one potential problem I see is that in some places, we have overridden the
environment's config using DelegateRegionCoprocessEnvironment. So the first
caller to
ConnectionFactory.getConnection(ConnectionType.DEFAULT_SERVER_CONNECTION, env)
might be passing in a DelegateRegionCoprocessorEnvironment with a modified
config, and the next called to
getConnnection(ConnectionType.DEFAULT_SERVER_CONNECTION) would be using that
modified config.
No it should not happen. We just use default configuration every time. We are
modifying the configurations at one place.
> Properly handle connection caching for Phoenix inside RegionServers
> ---
>
> Key: PHOENIX-4685
> URL: https://issues.apache.org/jira/browse/PHOENIX-4685
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch,
> PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch,
> PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where
> unable to create native threads. But it's working fine with 4.7.x branches.
> Found many threads created for meta lookup and shared threads and no space to
> create threads. This is happening even with short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN
> [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by:
[jira] [Updated] (PHOENIX-4685) Properly handle connection caching for Phoenix inside RegionServers
[
https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajeshbabu Chintaguntla updated PHOENIX-4685:
-
Attachment: PHOENIX-4685_addendum.patch
> Properly handle connection caching for Phoenix inside RegionServers
> ---
>
> Key: PHOENIX-4685
> URL: https://issues.apache.org/jira/browse/PHOENIX-4685
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4685.patch, PHOENIX-4685_5.x-HBase-2.0.patch,
> PHOENIX-4685_addendum.patch, PHOENIX-4685_jstack, PHOENIX-4685_v2.patch,
> PHOENIX-4685_v3.patch, PHOENIX-4685_v4.patch, PHOENIX-4685_v5.patch
>
>
> Currently trying to write data to indexed table failing with OOME where
> unable to create native threads. But it's working fine with 4.7.x branches.
> Found many threads created for meta lookup and shared threads and no space to
> create threads. This is happening even with short circuit writes enabled.
> {noformat}
> 2018-04-08 13:06:04,747 WARN
> [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: handleFailure failed
> java.io.IOException: java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217)
> at
> org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40)
> at
REMINDER: Apache EU Roadshow 2018 schedule announced!
Hello Apache Supporters and Enthusiasts This is a reminder that the schedule for the Apache EU Roadshow 2018 in Berlin has been announced. http://apachecon.com/euroadshow18/schedule.html Please note that we will not be running an ApacheCon in Europe this year which means that this Apache EU Roadshow will be the main Apache event in Europe for 2018. The Apache EU Roadshow tracks take place on the 13th and 14th June 2018, and will feature 28 sessions across the following themes; Apache Tomcat, IoT , Cloud Technologies, Microservices and Apache Httpd Server. Please note that the Apache EU Roadshow is co-located with FOSS Backstage and their schedule (https://foss-backstage.de/sessions) includes many Apache related sessions such as Incubator, Apache Way, Open Source Governance, Legal, Trademarks as well as a full range community related presentations and panel discussions. One single registration gives you access to both events - the Apache EU Roadshow and FOSS Backstage. Registration includes catering (breakfast & lunch both days) and also an attendee evening event. And if you want to have a project meet-up, hack or simply spend time and relax in our on-site Apache Lounge between sessions, then you are more than welcome. We look forward to seeing you in Berlin! Thanks Sharan Foga, VP Apache Community Development PLEASE NOTE: You are receiving this message because you are subscribed to a user@ or dev@ list of one or more Apache Software Foundation projects.
[jira] [Comment Edited] (PHOENIX-3955) Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync between the physical data table and index tables
[
https://issues.apache.org/jira/browse/PHOENIX-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471710#comment-16471710
]
Chinmay Kulkarni edited comment on PHOENIX-3955 at 5/11/18 9:58 AM:
Hey [~jamestaylor], [~samarthjain] [~tdsilva]
Here are some points on achieving this along with some questions I have:
Let's take a simple example. Say I create the base data table with the
following query:
{code:sql}
CREATE TABLE IF NOT EXISTS z_base_table (
id INTEGER not null primary key, CF1.host VARCHAR(10),flag BOOLEAN)
TTL=12,CF1.KEEP_DELETED_CELLS='true',REPLICATION_SCOPE='1';
{code}
We have the following paths to consider:
1. Create Index code path:
* *Case1: We create the data table with specific column families and there is
no default CF*:
In this case, the global index table's default CF and the CFs corresponding to
all local indexes should have default values for REPLICATION_SCOPE and
KEEP_DELETED_CELLS as they do now, BUT they should inherit the TTL property
from the non-local index CFs. In this case, it should be sufficient to check
any non-local index CF's TTL since they are enforced to all be the same.
* *Case2: The data table has a default CF*:
In this case, the global index table's default CF and the CFs corresponding to
all local indexes should inherit REPLICATION_SCOPE, KEEP_DELETED_CELLS and the
TTL property from the data table's default CF.
* *Question 1*: If we create an index with its own properties, say something
like:
{code:sql}
CREATE INDEX diff_properties_z_index ON z_base_table(host)
TTL=5000,KEEP_DELETED_CELLS='true';
{code}
We override the data table properties making the index tables and data table
properties out of sync. This JIRA might set expectations that these properties
are always in sync between index tables and the data table, so should we
disallow this henceforth? At the very least we may want to log that the index
table and data table properties will be out of sync after executing this query.
* *Question 1.1*: Given the above situation, if we later on alter the data
table, should we blindly also alter the properties of the index tables (given
that we want them to be in sync), or only alter index table properties in case
they are equivalent to the data table properties?
* "Create index code path" changes should be achievable by changes in
_CQSI.generateTableDescriptor_ before we apply specific properties of the index
tables themselves.
2. Alter table set code path:
* Here we can keep track of properties to be applied to
_QueryConstants.ALL_FAMILY_PROPERTIES_KEY_ and not to specific CFs. In case we
are changing TTL, REPLICATION_SCOPE or KEEP_DELETED_CELLS for all families, we
will alter the properties for index table CFs as well.
* *Case 1: Global Index Tables:*
We can have _CQSI.separateAndValidateProperties_ return a _Map>_ and then later store all
tabledescs and call _sendHBaseMetaData_() with this list of changes (which will
now include GLOBAL index table changes as well).
* *Case 2: Local Indexes:*
Can we simply change the column descriptor for the local index CF for the data
table? I'm not sure if this makes sense, but feel free to throw some light on
this case.
* *Question 2:*: If I create a local index on a column not belonging to the
default CF like:
{code:sql}
CREATE LOCAL INDEX cf_specific_z_index ON z_base_table(host);
{code}
then shouldn't the local index be using a CF of "L#CF1" instead of the default
"L#0"? In sqlline, when I do _select * from cf_specific_z_index;_, I see the
column as _CF1:Host_, but when I _desc 'z_base_table'_ in HBase shell, I see
the cf name to be "L#0".
* *Question 3:* How do we handle the case of multiple local indexes created on
the same table? If I run the following:
{code:sql}
CREATE LOCAL INDEX local_z_index1 ON z_base_table(host)
TTL=,KEEP_DELETED_CELLS='true';
CREATE LOCAL INDEX local_z_index2 ON z_base_table(flag)
TTL=,KEEP_DELETED_CELLS='false';
{code}
The actual HBase metadata change only reflects the last statement, since both
local indexes map to the 'L#0' column family. Please let me know if this is
handled at the Phoenix layer and I'm missing something.
was (Author: ckulkarni):
Hey [~jamestaylor], [~samarthjain] [~tdsilva]
Here are some points on achieving this along with some questions I have:
Let's take a simple example. Say I create the base data table with the
following query:
{code:sql}
CREATE TABLE IF NOT EXISTS z_base_table (
id INTEGER not null primary key, CF1.host VARCHAR(10),flag BOOLEAN)
TTL=12,CF1.KEEP_DELETED_CELLS='true',REPLICATION_SCOPE='1';
{code}
We have the following paths to consider:
1. Create Index code path:
* *Case1: We create the data table with specific column families and there is
no default CF*:
In this case, the global index table's default CF and the CFs corresponding to
all local indexes should have
[jira] [Commented] (PHOENIX-3955) Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync between the physical data table and index tables
[
https://issues.apache.org/jira/browse/PHOENIX-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471710#comment-16471710
]
Chinmay Kulkarni commented on PHOENIX-3955:
---
Hey [~jamestaylor], [~samarthjain] [~tdsilva]
Here are some points on achieving this along with some questions I have:
Let's take a simple example. Say I create the base data table with the
following query:
{code:sql}
CREATE TABLE IF NOT EXISTS z_base_table (
id INTEGER not null primary key, CF1.host VARCHAR(10),flag BOOLEAN)
TTL=12,CF1.KEEP_DELETED_CELLS='true',REPLICATION_SCOPE='1';
{code}
We have the following paths to consider:
1. Create Index code path:
* *Case1: We create the data table with specific column families and there is
no default CF*:
In this case, the global index table's default CF and the CFs corresponding to
all local indexes should have default values for REPLICATION_SCOPE and
KEEP_DELETED_CELLS as they do now, BUT they should inherit the TTL property
from the non-local index CFs. In this case, it should be sufficient to check
any non-local index CF's TTL since they are enforced to all be the same.
* *Case2: The data table has a default CF*:
In this case, the global index table's default CF and the CFs corresponding to
all local indexes should inherit REPLICATION_SCOPE, KEEP_DELETED_CELLS and the
TTL property from the data table's default CF.
* *Question 1*: If we create an index with its own properties, say something
like:
{code:sql}
CREATE INDEX diff_properties_z_index ON z_base_table(host)
TTL=5000,KEEP_DELETED_CELLS='true';
{code}
We override the data table properties making the index tables and data table
properties out of sync. This JIRA might set expectations that these properties
are always in sync between index tables and the data table, so should we
disallow this henceforth? At the very least we may want to log that the index
table and data table properties will be out of sync after executing this query.
* *Question 1.1*: Given the above situation, if we later on alter the data
table, should we blindly also alter the properties of the index tables (given
that we want them to be in sync), or only alter index table properties in case
they are equivalent to the data table properties?
* "Create index code path" changes should be achievable by changes in
_CQSI.generateTableDescriptor_ before we apply specific properties of the index
tables themselves.
2. Alter table set code path:
* Here we can keep track of properties to be applied to
_QueryConstants.ALL_FAMILY_PROPERTIES_KEY_ and not to specific CFs. In case we
are changing TTL, REPLICATION_SCOPE or KEEP_DELETED_CELLS for all families, we
will alter the properties for index table CFs as well.
* *Case 1: Global Index Tables:*
We can have _CQSI.separateAndValidateProperties_ return a _Map>_ and then later store all tabledescs and
call _sendHBaseMetaData_() with this list of changes (which will now include
GLOBAL index table changes as well).
* *Case 2: Local Indexes:*
Can we simply change the column descriptor for the local index CF for the data
table? I'm not sure if this makes sense, but feel free to throw some light on
this case.
* *Question 2:*: If I create a local index on a CF specific column like:
{code:sql}
CREATE LOCAL INDEX cf_specific_z_index ON z_base_table(host);
{code}
then shouldn't the local index be using a CF of "L#CF1" instead of the default
"L#0"? In sqlline, when I do _select * from cf_specific_z_index;_, I see the
column as _CF1:Host_, but when I _desc 'z_base_table'_ in HBase shell, I see
the cf name to be "L#0".
* *Question 3:* How do we handle the case of multiple local indexes created on
the same table? If I run the following:
{code:sql}
CREATE LOCAL INDEX local_z_index1 ON z_base_table(host)
TTL=,KEEP_DELETED_CELLS='true';
CREATE LOCAL INDEX local_z_index2 ON z_base_table(flag)
TTL=,KEEP_DELETED_CELLS='false';
{code}
The actual HBase metadata change only reflects the last statement, since both
local indexes map to the 'L#0' column family. Please let me know if this is
handled at the Phoenix layer and I'm missing something.
> Ensure KEEP_DELETED_CELLS, REPLICATION_SCOPE, and TTL properties stay in sync
> between the physical data table and index tables
> --
>
> Key: PHOENIX-3955
> URL: https://issues.apache.org/jira/browse/PHOENIX-3955
> Project: Phoenix
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> We need to make sure that indexes inherit the REPLICATION_SCOPE,
> KEEP_DELETED_CELLS and TTL properties from the base table. Otherwise we can
> run into situations where the data was removed (or not removed) from the data
> table but was
