[jira] [Updated] (PHOENIX-4820) Optimize OrderBy for ClientAggregatePlan
[
https://issues.apache.org/jira/browse/PHOENIX-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated PHOENIX-4820:
--
Attachment: (was: PHOENIX-4820-4.x-HBase-1.4-v2.patch)
> Optimize OrderBy for ClientAggregatePlan
>
>
> Key: PHOENIX-4820
> URL: https://issues.apache.org/jira/browse/PHOENIX-4820
> Project: Phoenix
> Issue Type: Improvement
>Affects Versions: 4.14.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 4.15.0
>
>
> Given a table
> {code}
> create table test (
>pk1 varchar not null ,
>pk2 varchar not null,
>pk3 varchar not null,
> v1 varchar,
> v2 varchar,
> CONSTRAINT TEST_PK PRIMARY KEY (
> pk1,
> pk2,
> pk3 ))
> {code}
> for following sql :
> {code}
> select a.ak3
> from (select substr(pk1,1,1) ak1,substr(pk2,1,1) ak2,substr(pk3,1,1)
> ak3,substr(v1,1,1) av1,substr(v2,1,1) av2 from test order by pk2,pk3 limit
> 10) a group by a.ak3,a.av1 order by a.ak3,a.av1
> {code}
> Intuitively, the above OrderBy statement {{order by a.ak3,a.av1}} should be
> compiled out because it match the group by statement, but in fact it is not.
> The problem is caused by the {{QueryCompiler.compileSingleQuery}} and
> {{QueryCompiler.compileSingleFlatQuery}},for
> {{QueryCompiler.compileSingleQuery}} method,because the inner query has order
> by, so in line 520, local variable {{isInRowKeyOrder}} is false:
> {code}
> 519context.setCurrentTable(tableRef);
> 520boolean isInRowKeyOrder = innerPlan.getGroupBy() ==
> GroupBy.EMPTY_GROUP_BY && innerPlan.getOrderBy() == OrderBy.EMPTY_ORDER_BY;
> {code}
> In {{QueryCompiler.compileSingleFlatQuery}},when {{OrderByCompiler.compile}}
> method is invoked, the last parameter {{isInRowKeyOrder}} is false:
> {code}
> 562OrderBy orderBy = OrderByCompiler.compile(context, select,
> groupBy, limit, offset, projector,
> 563groupBy == GroupBy.EMPTY_GROUP_BY ?
> innerPlanTupleProjector : null, isInRowKeyOrder);
> {code}
> So in following line 156 for {{OrderByCompiler.compile}},even though the
> {{tracker.isOrderPreserving}} is true, the OrderBy statement could not be
> compiled out.
> {code}
> 156 if (isInRowKeyOrder && tracker.isOrderPreserving()) {
> {code}
> In my opinion, with GroupBy, in following line 563 for
> {{QueryCompiler.compileSingleFlatQuery}} method, when we call
> {{OrderByCompiler.compile}} method, we no need to conside the
> {{isInRowKeyOrder}}, just like the previous parameter {{tupleProjector}}
> does.
> {code}
> 562OrderBy orderBy = OrderByCompiler.compile(context, select,
> groupBy, limit, offset, projector,
> 563groupBy == GroupBy.EMPTY_GROUP_BY ?
> innerPlanTupleProjector : null, isInRowKeyOrder);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4820) Optimize OrderBy for ClientAggregatePlan
[
https://issues.apache.org/jira/browse/PHOENIX-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated PHOENIX-4820:
--
Attachment: (was: PHOENIX-4820-4.x-HBase-1.4.patch)
> Optimize OrderBy for ClientAggregatePlan
>
>
> Key: PHOENIX-4820
> URL: https://issues.apache.org/jira/browse/PHOENIX-4820
> Project: Phoenix
> Issue Type: Improvement
>Affects Versions: 4.14.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 4.15.0
>
>
> Given a table
> {code}
> create table test (
>pk1 varchar not null ,
>pk2 varchar not null,
>pk3 varchar not null,
> v1 varchar,
> v2 varchar,
> CONSTRAINT TEST_PK PRIMARY KEY (
> pk1,
> pk2,
> pk3 ))
> {code}
> for following sql :
> {code}
> select a.ak3
> from (select substr(pk1,1,1) ak1,substr(pk2,1,1) ak2,substr(pk3,1,1)
> ak3,substr(v1,1,1) av1,substr(v2,1,1) av2 from test order by pk2,pk3 limit
> 10) a group by a.ak3,a.av1 order by a.ak3,a.av1
> {code}
> Intuitively, the above OrderBy statement {{order by a.ak3,a.av1}} should be
> compiled out because it match the group by statement, but in fact it is not.
> The problem is caused by the {{QueryCompiler.compileSingleQuery}} and
> {{QueryCompiler.compileSingleFlatQuery}},for
> {{QueryCompiler.compileSingleQuery}} method,because the inner query has order
> by, so in line 520, local variable {{isInRowKeyOrder}} is false:
> {code}
> 519context.setCurrentTable(tableRef);
> 520boolean isInRowKeyOrder = innerPlan.getGroupBy() ==
> GroupBy.EMPTY_GROUP_BY && innerPlan.getOrderBy() == OrderBy.EMPTY_ORDER_BY;
> {code}
> In {{QueryCompiler.compileSingleFlatQuery}},when {{OrderByCompiler.compile}}
> method is invoked, the last parameter {{isInRowKeyOrder}} is false:
> {code}
> 562OrderBy orderBy = OrderByCompiler.compile(context, select,
> groupBy, limit, offset, projector,
> 563groupBy == GroupBy.EMPTY_GROUP_BY ?
> innerPlanTupleProjector : null, isInRowKeyOrder);
> {code}
> So in following line 156 for {{OrderByCompiler.compile}},even though the
> {{tracker.isOrderPreserving}} is true, the OrderBy statement could not be
> compiled out.
> {code}
> 156 if (isInRowKeyOrder && tracker.isOrderPreserving()) {
> {code}
> In my opinion, with GroupBy, in following line 563 for
> {{QueryCompiler.compileSingleFlatQuery}} method, when we call
> {{OrderByCompiler.compile}} method, we no need to conside the
> {{isInRowKeyOrder}}, just like the previous parameter {{tupleProjector}}
> does.
> {code}
> 562OrderBy orderBy = OrderByCompiler.compile(context, select,
> groupBy, limit, offset, projector,
> 563groupBy == GroupBy.EMPTY_GROUP_BY ?
> innerPlanTupleProjector : null, isInRowKeyOrder);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4820) Optimize OrderBy for ClientAggregatePlan
[
https://issues.apache.org/jira/browse/PHOENIX-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated PHOENIX-4820:
--
Attachment: PHOENIX-4820-4.x-HBase-1.4.patch
> Optimize OrderBy for ClientAggregatePlan
>
>
> Key: PHOENIX-4820
> URL: https://issues.apache.org/jira/browse/PHOENIX-4820
> Project: Phoenix
> Issue Type: Improvement
>Affects Versions: 4.14.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 4.15.0
>
> Attachments: PHOENIX-4820-4.x-HBase-1.4.patch
>
>
> Given a table
> {code}
> create table test (
>pk1 varchar not null ,
>pk2 varchar not null,
>pk3 varchar not null,
> v1 varchar,
> v2 varchar,
> CONSTRAINT TEST_PK PRIMARY KEY (
> pk1,
> pk2,
> pk3 ))
> {code}
> for following sql :
> {code}
> select a.ak3
> from (select substr(pk1,1,1) ak1,substr(pk2,1,1) ak2,substr(pk3,1,1)
> ak3,substr(v1,1,1) av1,substr(v2,1,1) av2 from test order by pk2,pk3 limit
> 10) a group by a.ak3,a.av1 order by a.ak3,a.av1
> {code}
> Intuitively, the above OrderBy statement {{order by a.ak3,a.av1}} should be
> compiled out because it match the group by statement, but in fact it is not.
> The problem is caused by the {{QueryCompiler.compileSingleQuery}} and
> {{QueryCompiler.compileSingleFlatQuery}},for
> {{QueryCompiler.compileSingleQuery}} method,because the inner query has order
> by, so in line 520, local variable {{isInRowKeyOrder}} is false:
> {code}
> 519context.setCurrentTable(tableRef);
> 520boolean isInRowKeyOrder = innerPlan.getGroupBy() ==
> GroupBy.EMPTY_GROUP_BY && innerPlan.getOrderBy() == OrderBy.EMPTY_ORDER_BY;
> {code}
> In {{QueryCompiler.compileSingleFlatQuery}},when {{OrderByCompiler.compile}}
> method is invoked, the last parameter {{isInRowKeyOrder}} is false:
> {code}
> 562OrderBy orderBy = OrderByCompiler.compile(context, select,
> groupBy, limit, offset, projector,
> 563groupBy == GroupBy.EMPTY_GROUP_BY ?
> innerPlanTupleProjector : null, isInRowKeyOrder);
> {code}
> So in following line 156 for {{OrderByCompiler.compile}},even though the
> {{tracker.isOrderPreserving}} is true, the OrderBy statement could not be
> compiled out.
> {code}
> 156 if (isInRowKeyOrder && tracker.isOrderPreserving()) {
> {code}
> In my opinion, with GroupBy, in following line 563 for
> {{QueryCompiler.compileSingleFlatQuery}} method, when we call
> {{OrderByCompiler.compile}} method, we no need to conside the
> {{isInRowKeyOrder}}, just like the previous parameter {{tupleProjector}}
> does.
> {code}
> 562OrderBy orderBy = OrderByCompiler.compile(context, select,
> groupBy, limit, offset, projector,
> 563groupBy == GroupBy.EMPTY_GROUP_BY ?
> innerPlanTupleProjector : null, isInRowKeyOrder);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] phoenix pull request #319: PHOENIX-4820
Github user comnetwork closed the pull request at: https://github.com/apache/phoenix/pull/319 ---
[GitHub] phoenix issue #315: PHOENIX-3655 Global Phoenix Client Metrics for PQS
Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/315 > I think the only question pending in this PR is if we want JVMMetrics or not, which are crucial from PQS POV, otherwise not really for a regular client. Agreed. What about just making a new configuration key in QueryServices/QueryServicesOptions for including JVM metrics in client-metrics which defaults to `false`? Then, have PQS always set this to be `true` via code. Going to pull this down right now and look at it too out of curiosity. ---
[GitHub] phoenix pull request #318: Fixed Spelling.
Github user asfgit closed the pull request at: https://github.com/apache/phoenix/pull/318 ---
[jira] [Assigned] (PHOENIX-4829) Fixed Spelling
[ https://issues.apache.org/jira/browse/PHOENIX-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser reassigned PHOENIX-4829: --- Assignee: Jimmy Casey > Fixed Spelling > -- > > Key: PHOENIX-4829 > URL: https://issues.apache.org/jira/browse/PHOENIX-4829 > Project: Phoenix > Issue Type: Improvement >Reporter: Jimmy Casey >Assignee: Jimmy Casey >Priority: Trivial > Fix For: 4.15.0, 5.1.0 > > > https://github.com/apache/phoenix/pull/318 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4829) Fixed Spelling
[ https://issues.apache.org/jira/browse/PHOENIX-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated PHOENIX-4829: Fix Version/s: 5.1.0 4.15.0 > Fixed Spelling > -- > > Key: PHOENIX-4829 > URL: https://issues.apache.org/jira/browse/PHOENIX-4829 > Project: Phoenix > Issue Type: Improvement >Reporter: Jimmy Casey >Assignee: Jimmy Casey >Priority: Trivial > Fix For: 4.15.0, 5.1.0 > > > https://github.com/apache/phoenix/pull/318 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (PHOENIX-4829) Fixed Spelling
[ https://issues.apache.org/jira/browse/PHOENIX-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved PHOENIX-4829. - Resolution: Fixed Thanks for the patch, Jimmy. I've added you as a contributor so you can self-assign issues in the future! > Fixed Spelling > -- > > Key: PHOENIX-4829 > URL: https://issues.apache.org/jira/browse/PHOENIX-4829 > Project: Phoenix > Issue Type: Improvement >Reporter: Jimmy Casey >Assignee: Jimmy Casey >Priority: Trivial > Fix For: 4.15.0, 5.1.0 > > > https://github.com/apache/phoenix/pull/318 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix issue #315: PHOENIX-3655 Global Phoenix Client Metrics for PQS
Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/315 Pulling this into `master`, it looks like there are some commons-configuration issues that will need to be worked out for GlobalPhoenixMetricsTestSink.java ---
[GitHub] phoenix issue #315: PHOENIX-3655 Global Phoenix Client Metrics for PQS
Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/315 > it looks like there are some commons-configuration issues that will need to be worked out for GlobalPhoenixMetricsTestSink.java Just needs to be switched to the `org.apache.commons.configuration2` package. Also, GlobalPhoenixMetricsTestSink has a lot of unused imports. Would be good to make a pass over your changes for those. ---
[GitHub] phoenix issue #315: PHOENIX-3655 Global Phoenix Client Metrics for PQS
Github user karanmehta93 commented on the issue: https://github.com/apache/phoenix/pull/315 Thanks @joshelser will update the PR soon. We have immediate requirements for HBase-1.x versions here. That is why I mentioned that at the start of PR [itself](https://issues.apache.org/jira/browse/PHOENIX-3655?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=16559438#comment-16559438) that it might be tricky to port these patches. You can try it out with 4.x-HBase-1.4 branch, which is where my current PR is based on. Let me know if you think anything else is required here. ---
[DRAFT] August2018 board report
Given no further input, here's the draft board report. If no concerns, I'll be sending this out sooner than later. --- ## Description: - Apache Phoenix enables SQL-based OLTP and operational analytics for Apache Hadoop ## Issues: - There are no issues requiring board attention at this time ## Activity: - Two major release: 4.14.0 and 5.0.0. The former target HBase-0.98 and 1.x major release lines, the latter targeted HBase-2.0. - We've decided to stop releases versions of Phoenix that support HBase 0.98 and 1.1 releases as the HBase project has made a similar decision on creating more releases of those versions. - Created a new mailing list, issues@phoenix, to reduce overall traffic to dev@phoenix. - Held PhoenixCon on June 18th in San Jose, California, USA concurrently with HBaseCon (same day, same space). We had roughly 150 people in attendance at the events, with 10 Phoenix-specific talks given. Feedback was overall positive. Recordings are available at https://phoenix.apache.org/phoenixcon-archives.html for those who were unable to physically attend the event. - We've added two new PMC members and one new committer since our last report. ## Health report: - Things largely remain the same for the project. As HBase continues to evolve into newer versions, it requires us to do the same in Phoenix. We have continued activity in the project and I believe things look OK for the future. - Continuing to support and grow new contributors is a challenge. ## PMC changes: - Currently 27 PMC members. - New PMC members: - Pedro Boado was added to the PMC on Thu Jun 07 2018 - Vincent Poon was added to the PMC on Thu Jun 07 2018 ## Committer base changes: - Currently 36 committers. - Ohad Shaham was added as a committer on Fri Jun 08 2018 ## Releases: - 4.14.0 was released on Fri Jun 08 2018 - 5.0.0 was released on Sat Jul 14 2018 ## Mailing list activity: - List activity is largely consistent with previous months. There is a slight down-tick in volume to users@phoenix.
Re: [DISCUSS] reduce notifications to phoenix-dev list
Thomas -- we should be good on the mailing list being available. Can we flip Jira over to use this new list? On 2018/07/30 16:22:07, Josh Elser wrote: > Done. I can't seem to find a way to figure out who the moderators of the > currents lists are, so I put down myself, James (sorry ^.^), and Thomas ;) On 2018/07/26 17:39:49, "Thomas D'Silva" wrote: > Josh, > > Only apache members or pmc chairs can create a new mailing list ( > https://selfserve.apache.org/mail.html). > Can you create iss...@phoenix.apache.org? > > Thanks, > Thomas > > On Wed, Jul 25, 2018 at 9:20 AM, Josh Elser wrote: > > > I think this would be a nice thing to do. > > > > > > On 7/24/18 8:17 PM, Thomas D'Silva wrote: > > > >> Do folks think we need an issues mailing list similar to HBase that will > >> receive notifications about comments? > >> > >> On Tue, Jul 24, 2018 at 5:10 PM, Thomas D'Silva > >> wrote: > >> > >> Thanks Josh, I filed an INFRA Jira and updated the notification scheme. > >>> dev@phoenix.apache.org will no longer receive notifications when an > >>> issue > >>> is commented on or a comment is edited/deleted. > >>> > >>> On Thu, Jul 12, 2018 at 1:41 PM, Josh Elser > >>> wrote: > >>> > >>> I think INFRA keeps these locked down. Just requires an INFRA JIRA issue, > I'd guess asking them to change it. > > > On 7/12/18 1:53 PM, Thomas D'Silva wrote: > > Josh, > > > > Do you know how to edit the notification scheme here > > https://issues.apache.org/jira/plugins/servlet/project-confi > > g/PHOENIX/notifications > > ? > > I don't see an edit button. > > > > Thanks, > > Thomas > > > > On Thu, Jul 12, 2018 at 7:09 AM, James Taylor > > wrote: > > > > +1 > > > >> > >> On Thu, Jul 12, 2018 at 5:51 AM Josh Elser > >> wrote: > >> > >> Fine by me. > >> > >>> > >>> On 7/11/18 9:50 PM, Thomas D'Silva wrote: > >>> > >>> I think we should reduce the number of notifications that are sent to > > the > >>> > >> > >> phoenix-dev list. > >>> > The notification scheme we use is located here : > > https://issues.apache.org/ > >>> > >>> jira/plugins/servlet/project-config/PHOENIX/notifications > > Maybe we don't need to notify the dev list when an issue is updated, > > or a > >>> > >> > >> comment is created/edited/deleted. People watching a particular issue > >>> > > would > >>> > >>> still see all the changes. > This would be similar to how other projects like HBase and Calcite > > operate. > >>> > >>> > Thanks, > Thomas > > > > >>> > >> > > > >>> > >> >
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r206968534
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
Lets keep it as client since they represent client in the end, even though
it PQS that is instantiating it.
---
[jira] [Created] (PHOENIX-4832) Add Canary Test Tool for Phoenix Query Server
Ashutosh Parekh created PHOENIX-4832: Summary: Add Canary Test Tool for Phoenix Query Server Key: PHOENIX-4832 URL: https://issues.apache.org/jira/browse/PHOENIX-4832 Project: Phoenix Issue Type: Improvement Reporter: Ashutosh Parekh A suggested improvement is to add a Canary Test tool to the Phoenix Query Server. It will execute a set of Basic Tests (CRUD) against a PQS end-point and report on the proper functioning and testing results. A configurable Log Sink can help to publish the results as required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r206971126
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
How will you differentiate this "client" (PQS) from other clients?
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r206973402
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
I think its hard to differentiate, but my concern is why do we need to
differentiate. We should get that info from the application JVM itself right,
whether its PQS or some app server using Phoenix Client.
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r206974865
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
> why do we need to differentiate
As an operator, I'm going to expect much different characteristics coming
from client applications than I am from PQS. Since PQS is a SPOF for
(potentially) many users, I would expect a higher level of regularity from PQS
than I would a client's app.
> We should get that info from the application JVM itself right
I'm sure there's something you can apply to find a PQS metric compared to
other clients, but I am just thinking that it would be good to make this as
easy as possible. I think watching PQS metrics would be a common use-case, not
the exception.
---
Re: [DISCUSS] reduce notifications to phoenix-dev list
Sure I will file an INFRA ticket to change the behavior to send an email to the issues list when an issue is commented on or issue comment is edited or deleted. Should we also redirect the ASF GitHub Bot emails to the issues list instead of the dev list? On Wed, Aug 1, 2018 at 10:20 AM, Josh Elser wrote: > Thomas -- we should be good on the mailing list being available. > > Can we flip Jira over to use this new list? > > On 2018/07/30 16:22:07, Josh Elser wrote: > Done. > I can't seem to find a way to figure out who the moderators of the > > currents lists are, so I put down myself, James (sorry ^.^), and Thomas ;) > > >> On 2018/07/26 17:39:49, "Thomas D'Silva" >> wrote: > Josh, >> > > Only apache members or pmc chairs can create a new mailing list ( >> > https://selfserve.apache.org/mail.html). >> > Can you create iss...@phoenix.apache.org? >> > > Thanks, >> > Thomas >> > > On Wed, Jul 25, 2018 at 9:20 AM, Josh Elser >> wrote: >> > > > I think this would be a nice thing to do. >> > > >> > > >> > > On 7/24/18 8:17 PM, Thomas D'Silva wrote: >> > > >> > >> Do folks think we need an issues mailing list similar to HBase that >> will >> > >> receive notifications about comments? >> > >> >> > >> On Tue, Jul 24, 2018 at 5:10 PM, Thomas D'Silva < >> tdsi...@salesforce.com> >> > >> wrote: >> > >> >> > >> Thanks Josh, I filed an INFRA Jira and updated the notification >> scheme. >> > >>> dev@phoenix.apache.org will no longer receive notifications when an >> > >>> issue >> > >>> is commented on or a comment is edited/deleted. >> > >>> >> > >>> On Thu, Jul 12, 2018 at 1:41 PM, Josh Elser >> > >>> wrote: >> > >>> >> > >>> I think INFRA keeps these locked down. Just requires an INFRA JIRA >> issue, >> > I'd guess asking them to change it. >> > >> > >> > On 7/12/18 1:53 PM, Thomas D'Silva wrote: >> > >> > Josh, >> > > >> > > Do you know how to edit the notification scheme here >> > > https://issues.apache.org/jira/plugins/servlet/project-confi >> > > g/PHOENIX/notifications >> > > ? >> > > I don't see an edit button. >> > > >> > > Thanks, >> > > Thomas >> > > >> > > On Thu, Jul 12, 2018 at 7:09 AM, James Taylor < >> jamestay...@apache.org> >> > > wrote: >> > > >> > > +1 >> > > >> > >> >> > >> On Thu, Jul 12, 2018 at 5:51 AM Josh Elser > > >> > >> wrote: >> > >> >> > >> Fine by me. >> > >> >> > >>> >> > >>> On 7/11/18 9:50 PM, Thomas D'Silva wrote: >> > >>> >> > >>> I think we should reduce the number of notifications that are >> sent to >> > >> > the >> > >>> >> > >> >> > >> phoenix-dev list. >> > >>> >> > The notification scheme we use is located here : >> > >> > https://issues.apache.org/ >> > >>> >> > >>> jira/plugins/servlet/project-config/PHOENIX/notifications >> > >> > Maybe we don't need to notify the dev list when an issue is >> updated, >> > >> > or a >> > >>> >> > >> >> > >> comment is created/edited/deleted. People watching a particular >> issue >> > >>> >> > >> > would >> > >>> >> > >>> still see all the changes. >> > This would be similar to how other projects like HBase and >> Calcite >> > >> > operate. >> > >>> >> > >>> >> > Thanks, >> > Thomas >> > >> > >> > >> > >>> >> > >> >> > > >> > >>> >> > >> >> > >> >
[GitHub] phoenix issue #315: PHOENIX-3655 Global Phoenix Client Metrics for PQS
Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/315 Just used JVisualVM to look at these metrics via the MBeans directly. Worked just fine. I think naming/convention when being collected from a distributed environment needs to be thought about more though. ---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207001759
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
I am sorry, I didn't you on this one. Could you elaborate?
Also, I realized that stopping JVM metrics from Phoenix Level is not an
option here. Since phoenix embeds hbase-client which instantiates JVM Metrics
by default. Checkout this stack trace.
`
init:57, BaseSourceImpl$DefaultMetricsSystemInitializer
(org.apache.hadoop.hbase.metrics)
:112, BaseSourceImpl (org.apache.hadoop.hbase.metrics)
:56, MetricsZooKeeperSourceImpl (org.apache.hadoop.hbase.zookeeper)
:51, MetricsZooKeeperSourceImpl (org.apache.hadoop.hbase.zookeeper)
newInstance0:-1, NativeConstructorAccessorImpl (sun.reflect)
newInstance:62, NativeConstructorAccessorImpl (sun.reflect)
newInstance:45, DelegatingConstructorAccessorImpl (sun.reflect)
newInstance:423, Constructor (java.lang.reflect)
newInstance:442, Class (java.lang)
nextService:380, ServiceLoader$LazyIterator (java.util)
next:404, ServiceLoader$LazyIterator (java.util)
next:480, ServiceLoader$1 (java.util)
getInstance:59, CompatibilitySingletonFactory (org.apache.hadoop.hbase)
:38, MetricsZooKeeper (org.apache.hadoop.hbase.zookeeper)
:130, RecoverableZooKeeper (org.apache.hadoop.hbase.zookeeper)
connect:143, ZKUtil (org.apache.hadoop.hbase.zookeeper)
:181, ZooKeeperWatcher (org.apache.hadoop.hbase.zookeeper)
:155, ZooKeeperWatcher (org.apache.hadoop.hbase.zookeeper)
:43, ZooKeeperKeepAliveConnection (org.apache.hadoop.hbase.client)
getKeepAliveZooKeeperWatcher:1737,
ConnectionManager$HConnectionImplementation (org.apache.hadoop.hbase.client)
getClusterId:104, ZooKeeperRegistry (org.apache.hadoop.hbase.client)
retrieveClusterId:945, ConnectionManager$HConnectionImplementation
(org.apache.hadoop.hbase.client)
:721, ConnectionManager$HConnectionImplementation
(org.apache.hadoop.hbase.client)
newInstance0:-1, NativeConstructorAccessorImpl (sun.reflect)
newInstance:62, NativeConstructorAccessorImpl (sun.
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207003788
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
> I am sorry, I didn't you on this one. Could you elaborate?
You're using openTSDB to collect all of this data and your use case is to
"understand how PQS runs". To do this, you will need to know what hosts that
PQS is running on to just look at PQS metrics. I was suggesting that having a
unique name (or a tag) would make an operator's life much more simple -- it
would be clear (and presumably easier to filter on) to find just PQS metrics.
For example, the difference of issuing a filter "host in pqs_server1,
pqs_server2, pqs_server3" as opposed to just "tag=PQS", or similar.
Now that I'm thinking about this, leaving this as {{Phoenix,sub=CLIENT}}
may be OK and you can instead just configure the push of metrics data to Hadoop
Metrics2 to include a "tag" that indicates this is from PQS. That would make me
happy.
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207049490
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
So if I understand this correctly, I will include a new property which
takes a csv of tags as input. Later on, I can split by comma and add those tags
to `HBaseMetrics2HadoopMetricsAdapter#snapshotAllMetrics()`. @joshelser
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207051474
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
I don't have a strong opinion on how you expose this to let PQS say "I am
PQS" (e.g. whether CSV, list, whatever).
But yeah, modifying `snapshotAllMetrics` to include a Hadoop Metrics2
MetricsTag should do the trick.
---
Re: [DISCUSS] reduce notifications to phoenix-dev list
Sounds reasonable to me. On 8/1/18 1:59 PM, Thomas D'Silva wrote: Sure I will file an INFRA ticket to change the behavior to send an email to the issues list when an issue is commented on or issue comment is edited or deleted. Should we also redirect the ASF GitHub Bot emails to the issues list instead of the dev list? On Wed, Aug 1, 2018 at 10:20 AM, Josh Elser wrote: Thomas -- we should be good on the mailing list being available. Can we flip Jira over to use this new list? On 2018/07/30 16:22:07, Josh Elser wrote: > Done. I can't seem to find a way to figure out who the moderators of the > currents lists are, so I put down myself, James (sorry ^.^), and Thomas ;) On 2018/07/26 17:39:49, "Thomas D'Silva" wrote: > Josh, Only apache members or pmc chairs can create a new mailing list ( https://selfserve.apache.org/mail.html). Can you create iss...@phoenix.apache.org? Thanks, Thomas On Wed, Jul 25, 2018 at 9:20 AM, Josh Elser wrote: I think this would be a nice thing to do. On 7/24/18 8:17 PM, Thomas D'Silva wrote: Do folks think we need an issues mailing list similar to HBase that will receive notifications about comments? On Tue, Jul 24, 2018 at 5:10 PM, Thomas D'Silva < tdsi...@salesforce.com> wrote: Thanks Josh, I filed an INFRA Jira and updated the notification scheme. dev@phoenix.apache.org will no longer receive notifications when an issue is commented on or a comment is edited/deleted. On Thu, Jul 12, 2018 at 1:41 PM, Josh Elser wrote: I think INFRA keeps these locked down. Just requires an INFRA JIRA issue, I'd guess asking them to change it. On 7/12/18 1:53 PM, Thomas D'Silva wrote: Josh, Do you know how to edit the notification scheme here https://issues.apache.org/jira/plugins/servlet/project-confi g/PHOENIX/notifications ? I don't see an edit button. Thanks, Thomas On Thu, Jul 12, 2018 at 7:09 AM, James Taylor < jamestay...@apache.org> wrote: +1 On Thu, Jul 12, 2018 at 5:51 AM Josh Elser wrote: Fine by me. On 7/11/18 9:50 PM, Thomas D'Silva wrote: I think we should reduce the number of notifications that are sent to the phoenix-dev list. The notification scheme we use is located here : https://issues.apache.org/ jira/plugins/servlet/project-config/PHOENIX/notifications Maybe we don't need to notify the dev list when an issue is updated, or a comment is created/edited/deleted. People watching a particular issue would still see all the changes. This would be similar to how other projects like HBase and Calcite operate. Thanks, Thomas
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user xcangCRM commented on a diff in the pull request: https://github.com/apache/phoenix/pull/315#discussion_r207053124 --- Diff: phoenix-core/src/test/resources/hadoop-metrics2.properties --- @@ -32,10 +32,9 @@ #[prefix].[source|sink].[instance].[options] # See javadoc of package-info.java for org.apache.hadoop.metrics2 for detail - -# Don't attempt to start jmx mbeans for all sources. -# For right now, all metrics are exported to a phoenix table -*.source.start_mbeans=false +phoenix.source.start_mbeans=true +phoenix.sink.sink0.class=org.apache.phoenix.monitoring.GlobalPhoenixMetricsTestSink --- End diff -- I think it usually starts with sink 1 . (?) at least ni HBase. ---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user xcangCRM commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207053000
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("Phoenix");
+JvmMetrics.initSingleton("Phoenix", "");
+}
+
+public static GlobalMetricRegistriesAdapter getInstance() {
+if (INSTANCE == null) {
+INSTANCE = new GlobalMetricRegistriesAdapter();
+}
+return INSTANCE;
+}
+
+public void registerMetricRegistry(MetricRegistry registry) {
+if (registry == null) {
+LOG.warn("Registry cannot be registered with Hadoop Metrics 2
since it is null.");
+return;
+}
+
+HBaseMetrics2HadoopMetricsAdapter adapter = new
HBaseMetrics2HadoopMetricsAdapter(registry);
+registerToDefaultMetricsSystem(registry, adapter);
+}
+
+private void registerToDefaultMetricsSystem(MetricRegistry registry,
HBaseMetrics2HadoopMetricsAdapter adapter) {
+MetricRegistryInfo info = registry.getMetricRegistryInfo();
+LOG.info("Registering " + info.getMetricsJmxContext() + " " +
info.getMetricsDescription() + " into DefaultMetricsSystem");
+
DefaultMetricsSystem.instance().register(info.getMetricsJmxContext(),
info.getMetricsDescription(), adapter);
+}
+
+/**
+ * Class to convert HBase Metric Objects to Hadoop Metrics2 Metric
Objects
+ */
+private static class HBaseMetrics2HadoopMetricsAdapter implements
MetricsSource {
+private static final Log LOG =
LogFactory.getLog(HBaseMetrics2HadoopMetricsAdapter.class);
+private final MetricRegistry registry;
+private HBaseMetrics2HadoopMetricsAdapter(MetricRegistry registry)
{
+this.registry = registry;
+}
+
+public void snapshotAllMetrics(MetricRegistry metricRegistry,
MetricsCollector collector) {
+MetricRegistryInfo info =
metricRegistry.getMetricRegistryInfo();
+MetricsRecordBuilder builder =
collector.addRecord(Interns.info(info.getMetricsName(),
info.getMetricsDescription()));
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user xcangCRM commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207052688
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("Phoenix");
+JvmMetrics.initSingleton("Phoenix", "");
+}
+
+public static GlobalMetricRegistriesAdapter getInstance() {
+if (INSTANCE == null) {
--- End diff --
nit: do you need double checking here?
if (INSTANCE == null) {
synchronized(this) {
if(INSTANCE == null) {
...
}
}
}
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207054938
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("HBase");
+JvmMetrics.initSingleton("HBase", "");
--- End diff --
Cool. Thanks!
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request: https://github.com/apache/phoenix/pull/315#discussion_r207054877 --- Diff: phoenix-core/src/test/resources/hadoop-metrics2.properties --- @@ -32,10 +32,9 @@ #[prefix].[source|sink].[instance].[options] # See javadoc of package-info.java for org.apache.hadoop.metrics2 for detail - -# Don't attempt to start jmx mbeans for all sources. -# For right now, all metrics are exported to a phoenix table -*.source.start_mbeans=false +phoenix.source.start_mbeans=true +phoenix.sink.sink0.class=org.apache.phoenix.monitoring.GlobalPhoenixMetricsTestSink --- End diff -- It is just an identifier and can be anything. ---
Re: [DISCUSS] reduce notifications to phoenix-dev list
+1, it's alot of noise On Wed, Aug 1, 2018, 6:29 PM Josh Elser wrote: > Sounds reasonable to me. > > On 8/1/18 1:59 PM, Thomas D'Silva wrote: > > Sure I will file an INFRA ticket to change the behavior to send an email > to > > the issues list when an issue is commented on or issue comment is edited > or > > deleted. > > Should we also redirect the ASF GitHub Bot emails to the issues list > > instead of the dev list? > > > > > > > > > > On Wed, Aug 1, 2018 at 10:20 AM, Josh Elser > wrote: > > > >> Thomas -- we should be good on the mailing list being available. > >> > >> Can we flip Jira over to use this new list? > >> > >> On 2018/07/30 16:22:07, Josh Elser wrote: > > Done. > >> I can't seem to find a way to figure out who the moderators of the > > >> currents lists are, so I put down myself, James (sorry ^.^), and Thomas > ;) > >> > >> > >>> On 2018/07/26 17:39:49, "Thomas D'Silva" > >>> wrote: > Josh, > > Only apache members or pmc chairs can create a new mailing list ( > https://selfserve.apache.org/mail.html). > Can you create iss...@phoenix.apache.org? > > Thanks, > Thomas > > On Wed, Jul 25, 2018 at 9:20 AM, Josh Elser > >>> wrote: > >> I think this would be a nice thing to do. > > > > > > On 7/24/18 8:17 PM, Thomas D'Silva wrote: > > > >> Do folks think we need an issues mailing list similar to HBase that > >>> will > >> receive notifications about comments? > >> > >> On Tue, Jul 24, 2018 at 5:10 PM, Thomas D'Silva < > >>> tdsi...@salesforce.com> > >> wrote: > >> > >> Thanks Josh, I filed an INFRA Jira and updated the notification > >>> scheme. > >>> dev@phoenix.apache.org will no longer receive notifications when > an > >>> issue > >>> is commented on or a comment is edited/deleted. > >>> > >>> On Thu, Jul 12, 2018 at 1:41 PM, Josh Elser > >>> wrote: > >>> > >>> I think INFRA keeps these locked down. Just requires an INFRA JIRA > >>> issue, > I'd guess asking them to change it. > > > On 7/12/18 1:53 PM, Thomas D'Silva wrote: > > Josh, > > > > Do you know how to edit the notification scheme here > > https://issues.apache.org/jira/plugins/servlet/project-confi > > g/PHOENIX/notifications > > ? > > I don't see an edit button. > > > > Thanks, > > Thomas > > > > On Thu, Jul 12, 2018 at 7:09 AM, James Taylor < > >>> jamestay...@apache.org> > > wrote: > > > > +1 > > > >> > >> On Thu, Jul 12, 2018 at 5:51 AM Josh Elser < > josh.el...@gmail.com > > >> wrote: > >> > >> Fine by me. > >> > >>> > >>> On 7/11/18 9:50 PM, Thomas D'Silva wrote: > >>> > >>> I think we should reduce the number of notifications that are > >>> sent to > > the > >>> > >> > >> phoenix-dev list. > >>> > The notification scheme we use is located here : > > https://issues.apache.org/ > >>> > >>> jira/plugins/servlet/project-config/PHOENIX/notifications > > Maybe we don't need to notify the dev list when an issue is > >>> updated, > > or a > >>> > >> > >> comment is created/edited/deleted. People watching a particular > >>> issue > >>> > > would > >>> > >>> still see all the changes. > This would be similar to how other projects like HBase and > >>> Calcite > > operate. > >>> > >>> > Thanks, > Thomas > > > > >>> > >> > > > >>> > >> > > >>> > >> > > >
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207060270
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("Phoenix");
+JvmMetrics.initSingleton("Phoenix", "");
+}
+
+public static GlobalMetricRegistriesAdapter getInstance() {
+if (INSTANCE == null) {
--- End diff --
As of now, this method is called from static block in
`GlobalClientMetrics`, which can happen only once, so we dont have any race
conditions here.
I have two options here, add a lock here or move this initialization to
static block of this class. What do you suggest?
---
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207060201
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("Phoenix");
+JvmMetrics.initSingleton("Phoenix", "");
+}
+
+public static GlobalMetricRegistriesAdapter getInstance() {
+if (INSTANCE == null) {
+INSTANCE = new GlobalMetricRegistriesAdapter();
+}
+return INSTANCE;
+}
+
+public void registerMetricRegistry(MetricRegistry registry) {
+if (registry == null) {
+LOG.warn("Registry cannot be registered with Hadoop Metrics 2
since it is null.");
+return;
+}
+
+HBaseMetrics2HadoopMetricsAdapter adapter = new
HBaseMetrics2HadoopMetricsAdapter(registry);
+registerToDefaultMetricsSystem(registry, adapter);
+}
+
+private void registerToDefaultMetricsSystem(MetricRegistry registry,
HBaseMetrics2HadoopMetricsAdapter adapter) {
+MetricRegistryInfo info = registry.getMetricRegistryInfo();
+LOG.info("Registering " + info.getMetricsJmxContext() + " " +
info.getMetricsDescription() + " into DefaultMetricsSystem");
+
DefaultMetricsSystem.instance().register(info.getMetricsJmxContext(),
info.getMetricsDescription(), adapter);
+}
+
+/**
+ * Class to convert HBase Metric Objects to Hadoop Metrics2 Metric
Objects
+ */
+private static class HBaseMetrics2HadoopMetricsAdapter implements
MetricsSource {
+private static final Log LOG =
LogFactory.getLog(HBaseMetrics2HadoopMetricsAdapter.class);
+private final MetricRegistry registry;
+private HBaseMetrics2HadoopMetricsAdapter(MetricRegistry registry)
{
+this.registry = registry;
+}
+
+public void snapshotAllMetrics(MetricRegistry metricRegistry,
MetricsCollector collector) {
+MetricRegistryInfo info =
metricRegistry.getMetricRegistryInfo();
+MetricsRecordBuilder builder =
collector.addRecord(Interns.info(info.getMetricsName(),
info.getMetricsDescription()))
[GitHub] phoenix pull request #315: PHOENIX-3655 Global Phoenix Client Metrics for PQ...
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/315#discussion_r207063036
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/monitoring/GlobalMetricRegistriesAdapter.java
---
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.monitoring;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Gauge;
+import org.apache.hadoop.hbase.metrics.Histogram;
+import org.apache.hadoop.hbase.metrics.Meter;
+import org.apache.hadoop.hbase.metrics.Metric;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.metrics.MetricRegistryInfo;
+import org.apache.hadoop.hbase.metrics.Timer;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.metrics2.lib.MutableHistogram;
+import org.apache.hadoop.metrics2.source.JvmMetrics;
+
+/**
+ * Contents mostly copied from GlobalMetricRegistriesAdapter class from
hbase-hadoop2-compat
+ * The adapter attaches HBase's MetricRegistry to Hadoop's
DefaultMetricsSystem
+ * Doesn't handle dynamic attach/detach of registries
+ */
+public class GlobalMetricRegistriesAdapter {
+
+private static final Log LOG =
LogFactory.getLog(GlobalMetricRegistriesAdapter.class);
+private static GlobalMetricRegistriesAdapter INSTANCE;
+
+private GlobalMetricRegistriesAdapter() {
+DefaultMetricsSystem.initialize("Phoenix");
+JvmMetrics.initSingleton("Phoenix", "");
+}
+
+public static GlobalMetricRegistriesAdapter getInstance() {
+if (INSTANCE == null) {
--- End diff --
I think I am moving away from lazy initialization all together. No need of
premature optimization here I believe.
---
Re: [DISCUSS] reduce notifications to phoenix-dev list
I filed https://issues.apache.org/jira/browse/INFRA-16862 and https://issues.apache.org/jira/browse/INFRA-16863. On Wed, Aug 1, 2018 at 3:53 PM, Artem Ervits wrote: > +1, it's alot of noise > > On Wed, Aug 1, 2018, 6:29 PM Josh Elser wrote: > > > Sounds reasonable to me. > > > > On 8/1/18 1:59 PM, Thomas D'Silva wrote: > > > Sure I will file an INFRA ticket to change the behavior to send an > email > > to > > > the issues list when an issue is commented on or issue comment is > edited > > or > > > deleted. > > > Should we also redirect the ASF GitHub Bot emails to the issues list > > > instead of the dev list? > > > > > > > > > > > > > > > On Wed, Aug 1, 2018 at 10:20 AM, Josh Elser > > wrote: > > > > > >> Thomas -- we should be good on the mailing list being available. > > >> > > >> Can we flip Jira over to use this new list? > > >> > > >> On 2018/07/30 16:22:07, Josh Elser wrote: > > > Done. > > >> I can't seem to find a way to figure out who the moderators of the > > > >> currents lists are, so I put down myself, James (sorry ^.^), and > Thomas > > ;) > > >> > > >> > > >>> On 2018/07/26 17:39:49, "Thomas D'Silva" > > >>> wrote: > Josh, > > > Only apache members or pmc chairs can create a new mailing list ( > > https://selfserve.apache.org/mail.html). > > Can you create iss...@phoenix.apache.org? > > > Thanks, > > Thomas > > > On Wed, Jul 25, 2018 at 9:20 AM, Josh Elser > > >>> wrote: > > >> I think this would be a nice thing to do. > > > > > > > > > On 7/24/18 8:17 PM, Thomas D'Silva wrote: > > > > > >> Do folks think we need an issues mailing list similar to HBase > that > > >>> will > > >> receive notifications about comments? > > >> > > >> On Tue, Jul 24, 2018 at 5:10 PM, Thomas D'Silva < > > >>> tdsi...@salesforce.com> > > >> wrote: > > >> > > >> Thanks Josh, I filed an INFRA Jira and updated the notification > > >>> scheme. > > >>> dev@phoenix.apache.org will no longer receive notifications when > > an > > >>> issue > > >>> is commented on or a comment is edited/deleted. > > >>> > > >>> On Thu, Jul 12, 2018 at 1:41 PM, Josh Elser < > josh.el...@gmail.com> > > >>> wrote: > > >>> > > >>> I think INFRA keeps these locked down. Just requires an INFRA > JIRA > > >>> issue, > > I'd guess asking them to change it. > > > > > > On 7/12/18 1:53 PM, Thomas D'Silva wrote: > > > > Josh, > > > > > > Do you know how to edit the notification scheme here > > > https://issues.apache.org/jira/plugins/servlet/project-confi > > > g/PHOENIX/notifications > > > ? > > > I don't see an edit button. > > > > > > Thanks, > > > Thomas > > > > > > On Thu, Jul 12, 2018 at 7:09 AM, James Taylor < > > >>> jamestay...@apache.org> > > > wrote: > > > > > > +1 > > > > > >> > > >> On Thu, Jul 12, 2018 at 5:51 AM Josh Elser < > > josh.el...@gmail.com > > > > >> wrote: > > >> > > >> Fine by me. > > >> > > >>> > > >>> On 7/11/18 9:50 PM, Thomas D'Silva wrote: > > >>> > > >>> I think we should reduce the number of notifications that are > > >>> sent to > > > > the > > >>> > > >> > > >> phoenix-dev list. > > >>> > > The notification scheme we use is located here : > > > > https://issues.apache.org/ > > >>> > > >>> jira/plugins/servlet/project-config/PHOENIX/notifications > > > > Maybe we don't need to notify the dev list when an issue is > > >>> updated, > > > > or a > > >>> > > >> > > >> comment is created/edited/deleted. People watching a > particular > > >>> issue > > >>> > > > > would > > >>> > > >>> still see all the changes. > > This would be similar to how other projects like HBase and > > >>> Calcite > > > > operate. > > >>> > > >>> > > Thanks, > > Thomas > > > > > > > > >>> > > >> > > > > > >>> > > >> > > > > >>> > > >> > > > > > >
[GitHub] phoenix issue #313: PHOENIX-4799 Write cells using checkAndMutate to prevent...
Github user twdsilva commented on the issue: https://github.com/apache/phoenix/pull/313 Testing iss...@phoenix.apache.org. ---
[jira] [Updated] (PHOENIX-4476) Range scan used for point lookups if filter is not in order of primary keys
[
https://issues.apache.org/jira/browse/PHOENIX-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated PHOENIX-4476:
-
Comment: was deleted
(was: testing comment. (to test devlist notification fix requested by Thomas))
> Range scan used for point lookups if filter is not in order of primary keys
> ---
>
> Key: PHOENIX-4476
> URL: https://issues.apache.org/jira/browse/PHOENIX-4476
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.13.1
>Reporter: Mujtaba Chohan
>Assignee: Xu Cang
>Priority: Major
> Labels: SFDC
> Attachments: PHOENIX-4476-workInProgress.001.patch
>
>
> {noformat}
> DROP TABLE TEST;
> CREATE TABLE IF NOT EXISTS TEST (
> PK1 CHAR(1) NOT NULL,
> PK2 VARCHAR NOT NULL,
> PK3 VARCHAR NOT NULL,
> PK4 UNSIGNED_LONG NOT NULL,
> PK5 VARCHAR NOT NULL,
> V1 VARCHAR,
> V2 VARCHAR,
> V3 UNSIGNED_LONG
> CONSTRAINT state_pk PRIMARY KEY (
> PK1,
> PK2,
> PK3,
> PK4,
> PK5
> )
> );
> // Incorrect explain plan with un-ordered PKs
> EXPLAIN SELECT V1 FROM TEST WHERE (PK1, PK5, PK2, PK3, PK4) IN (('A', 'E',
> 'N', 'T', 3), ('A', 'Y', 'G', 'T', 4));
> +--+--+--+-+
> | PLAN | EST_BYTES_READ
> | EST_ROWS_READ | |
> +--+--+--+-+
> | CLIENT 1-CHUNK PARALLEL 1-WAY ROUND ROBIN RANGE SCAN OVER TEST ['A'] | null
> | null |
> | SERVER FILTER BY (PK1, PK5, PK2, PK3, PK4) IN
> ([65,69,0,78,0,84,0,0,0,0,0,0,0,0,3],[65,89,0,71,0,84,0,0,0,0,0,0,0,0,4]) |
> null |
> +--+--+--+-+
> // Correct explain plan with PKs in order
> EXPLAIN SELECT V1 FROM TEST WHERE (PK1,PK2,PK3,PK4,PK5) IN (('A', 'E', 'N',3,
> 'T'),('A', 'Y', 'G', 4, 'T'));
> +--+--+--+-+
> | PLAN | EST_BYTES_READ
> | EST_ROWS_READ | |
> +--+--+--+-+
> | CLIENT 1-CHUNK 2 ROWS 712 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP ON
> 2 KEYS OVER TEST | 712 | |
> +--+--+--+-+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4751) Support client-side hash aggregation with SORT_MERGE_JOIN
[ https://issues.apache.org/jira/browse/PHOENIX-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerald Sangudi updated PHOENIX-4751: Attachment: 0001-PHOENIX-4751-Implement-client-side-hash-aggre.master.patch > Support client-side hash aggregation with SORT_MERGE_JOIN > - > > Key: PHOENIX-4751 > URL: https://issues.apache.org/jira/browse/PHOENIX-4751 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.14.0, 4.13.1 >Reporter: Gerald Sangudi >Assignee: Gerald Sangudi >Priority: Major > Attachments: > 0001-PHOENIX-4751-Add-HASH_AGGREGATE-hint.4.x-HBase-1.4.patch, > 0001-PHOENIX-4751-Implement-client-side-has.4.x-HBase-1.4.patch, > 0001-PHOENIX-4751-Implement-client-side-hash-aggre.master.patch, > 0002-PHOENIX-4751-Begin-implementation-of-c.4.x-HBase-1.4.patch, > 0003-PHOENIX-4751-Generated-aggregated-resu.4.x-HBase-1.4.patch, > 0004-PHOENIX-4751-Sort-results-of-client-ha.4.x-HBase-1.4.patch, > 0005-PHOENIX-4751-Add-integration-test-for-.4.x-HBase-1.4.patch, > 0006-PHOENIX-4751-Fix-and-run-integration-t.4.x-HBase-1.4.patch, > 0007-PHOENIX-4751-Add-integration-test-for-.4.x-HBase-1.4.patch, > 0008-PHOENIX-4751-Verify-EXPLAIN-plan-for-b.4.x-HBase-1.4.patch, > 0009-PHOENIX-4751-Standardize-null-checks-a.4.x-HBase-1.4.patch, > 0010-PHOENIX-4751-Abort-when-client-aggrega.4.x-HBase-1.4.patch, > 0011-PHOENIX-4751-Use-Phoenix-memory-mgmt-t.4.x-HBase-1.4.patch, > 0012-PHOENIX-4751-Remove-extra-memory-limit.4.x-HBase-1.4.patch, > 0013-PHOENIX-4751-Sort-only-when-necessary.4.x-HBase-1.4.patch, > 0014-PHOENIX-4751-Sort-only-when-necessary-.4.x-HBase-1.4.patch, > 0015-PHOENIX-4751-Show-client-hash-aggregat.4.x-HBase-1.4.patch, > 0016-PHOENIX-4751-Handle-reverse-sort-add-c.4.x-HBase-1.4.patch > > > A GROUP BY that follows a SORT_MERGE_JOIN should be able to use hash > aggregation in some cases, for improved performance. > When a GROUP BY follows a SORT_MERGE_JOIN, the GROUP BY does not use hash > aggregation. It instead performs a CLIENT SORT followed by a CLIENT > AGGREGATE. The performance can be improved if (a) the GROUP BY output does > not need to be sorted, and (b) the GROUP BY input is large enough and has low > cardinality. > The hash aggregation can initially be a hint. Here is an example from Phoenix > 4.13.1 that would benefit from hash aggregation if the GROUP BY input is > large with low cardinality. > CREATE TABLE unsalted ( > keyA BIGINT NOT NULL, > keyB BIGINT NOT NULL, > val SMALLINT, > CONSTRAINT pk PRIMARY KEY (keyA, keyB) > ); > EXPLAIN > SELECT /*+ USE_SORT_MERGE_JOIN */ > t1.val v1, t2.val v2, COUNT(\*) c > FROM unsalted t1 JOIN unsalted t2 > ON (t1.keyA = t2.keyA) > GROUP BY t1.val, t2.val; > > +-+++--+ > |PLAN|EST_BYTES_READ|EST_ROWS_READ| | > +-+++--+ > |SORT-MERGE-JOIN (INNER) TABLES|null|null| | > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER UNSALTED|null|null| | > |AND|null|null| | > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER UNSALTED|null|null| | > |CLIENT SORTED BY [TO_DECIMAL(T1.VAL), T2.VAL]|null|null| | > |CLIENT AGGREGATE INTO DISTINCT ROWS BY [T1.VAL, T2.VAL]|null|null| | > +-+++--+ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4476) Range scan used for point lookups if filter is not in order of primary keys
[
https://issues.apache.org/jira/browse/PHOENIX-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated PHOENIX-4476:
-
Attachment: PHOENIX-4476-4.x-HBase-1.3.002.patch
> Range scan used for point lookups if filter is not in order of primary keys
> ---
>
> Key: PHOENIX-4476
> URL: https://issues.apache.org/jira/browse/PHOENIX-4476
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.13.1
>Reporter: Mujtaba Chohan
>Assignee: Xu Cang
>Priority: Major
> Labels: SFDC
> Attachments: PHOENIX-4476-4.x-HBase-1.3.002.patch,
> PHOENIX-4476-workInProgress.001.patch
>
>
> {noformat}
> DROP TABLE TEST;
> CREATE TABLE IF NOT EXISTS TEST (
> PK1 CHAR(1) NOT NULL,
> PK2 VARCHAR NOT NULL,
> PK3 VARCHAR NOT NULL,
> PK4 UNSIGNED_LONG NOT NULL,
> PK5 VARCHAR NOT NULL,
> V1 VARCHAR,
> V2 VARCHAR,
> V3 UNSIGNED_LONG
> CONSTRAINT state_pk PRIMARY KEY (
> PK1,
> PK2,
> PK3,
> PK4,
> PK5
> )
> );
> // Incorrect explain plan with un-ordered PKs
> EXPLAIN SELECT V1 FROM TEST WHERE (PK1, PK5, PK2, PK3, PK4) IN (('A', 'E',
> 'N', 'T', 3), ('A', 'Y', 'G', 'T', 4));
> +--+--+--+-+
> | PLAN | EST_BYTES_READ
> | EST_ROWS_READ | |
> +--+--+--+-+
> | CLIENT 1-CHUNK PARALLEL 1-WAY ROUND ROBIN RANGE SCAN OVER TEST ['A'] | null
> | null |
> | SERVER FILTER BY (PK1, PK5, PK2, PK3, PK4) IN
> ([65,69,0,78,0,84,0,0,0,0,0,0,0,0,3],[65,89,0,71,0,84,0,0,0,0,0,0,0,0,4]) |
> null |
> +--+--+--+-+
> // Correct explain plan with PKs in order
> EXPLAIN SELECT V1 FROM TEST WHERE (PK1,PK2,PK3,PK4,PK5) IN (('A', 'E', 'N',3,
> 'T'),('A', 'Y', 'G', 4, 'T'));
> +--+--+--+-+
> | PLAN | EST_BYTES_READ
> | EST_ROWS_READ | |
> +--+--+--+-+
> | CLIENT 1-CHUNK 2 ROWS 712 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP ON
> 2 KEYS OVER TEST | 712 | |
> +--+--+--+-+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-4476) Range scan used for point lookups if filter is not in order of primary keys
[
https://issues.apache.org/jira/browse/PHOENIX-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated PHOENIX-4476:
-
Attachment: (was: PHOENIX-4476-workInProgress.001.patch)
> Range scan used for point lookups if filter is not in order of primary keys
> ---
>
> Key: PHOENIX-4476
> URL: https://issues.apache.org/jira/browse/PHOENIX-4476
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.13.1
>Reporter: Mujtaba Chohan
>Assignee: Xu Cang
>Priority: Major
> Labels: SFDC
> Attachments: PHOENIX-4476-4.x-HBase-1.3.002.patch
>
>
> {noformat}
> DROP TABLE TEST;
> CREATE TABLE IF NOT EXISTS TEST (
> PK1 CHAR(1) NOT NULL,
> PK2 VARCHAR NOT NULL,
> PK3 VARCHAR NOT NULL,
> PK4 UNSIGNED_LONG NOT NULL,
> PK5 VARCHAR NOT NULL,
> V1 VARCHAR,
> V2 VARCHAR,
> V3 UNSIGNED_LONG
> CONSTRAINT state_pk PRIMARY KEY (
> PK1,
> PK2,
> PK3,
> PK4,
> PK5
> )
> );
> // Incorrect explain plan with un-ordered PKs
> EXPLAIN SELECT V1 FROM TEST WHERE (PK1, PK5, PK2, PK3, PK4) IN (('A', 'E',
> 'N', 'T', 3), ('A', 'Y', 'G', 'T', 4));
> +--+--+--+-+
> | PLAN | EST_BYTES_READ
> | EST_ROWS_READ | |
> +--+--+--+-+
> | CLIENT 1-CHUNK PARALLEL 1-WAY ROUND ROBIN RANGE SCAN OVER TEST ['A'] | null
> | null |
> | SERVER FILTER BY (PK1, PK5, PK2, PK3, PK4) IN
> ([65,69,0,78,0,84,0,0,0,0,0,0,0,0,3],[65,89,0,71,0,84,0,0,0,0,0,0,0,0,4]) |
> null |
> +--+--+--+-+
> // Correct explain plan with PKs in order
> EXPLAIN SELECT V1 FROM TEST WHERE (PK1,PK2,PK3,PK4,PK5) IN (('A', 'E', 'N',3,
> 'T'),('A', 'Y', 'G', 4, 'T'));
> +--+--+--+-+
> | PLAN | EST_BYTES_READ
> | EST_ROWS_READ | |
> +--+--+--+-+
> | CLIENT 1-CHUNK 2 ROWS 712 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP ON
> 2 KEYS OVER TEST | 712 | |
> +--+--+--+-+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DRAFT] August2018 board report
+1 On Wed, Aug 1, 2018 at 10:16 AM Josh Elser wrote: > Given no further input, here's the draft board report. If no concerns, > I'll be sending this out sooner than later. > > --- > > ## Description: > - Apache Phoenix enables SQL-based OLTP and operational analytics for > Apache Hadoop > > ## Issues: > - There are no issues requiring board attention at this time > > ## Activity: > - Two major release: 4.14.0 and 5.0.0. The former target HBase-0.98 > and 1.x major release lines, the latter targeted HBase-2.0. > - We've decided to stop releases versions of Phoenix that support > HBase 0.98 and 1.1 releases as the HBase project has made a similar > decision on creating more releases of those versions. > - Created a new mailing list, issues@phoenix, to reduce overall > traffic to dev@phoenix. > - Held PhoenixCon on June 18th in San Jose, California, USA > concurrently with HBaseCon (same day, same space). We had roughly 150 > people in attendance at the events, with 10 Phoenix-specific talks > given. Feedback was overall positive. Recordings are available at > https://phoenix.apache.org/phoenixcon-archives.html for those who were > unable to physically attend the event. > - We've added two new PMC members and one new committer since our last > report. > > ## Health report: > - Things largely remain the same for the project. As HBase continues > to evolve into newer versions, it requires us to do the same in Phoenix. > We have continued activity in the project and I believe things look OK > for the future. > - Continuing to support and grow new contributors is a challenge. > > ## PMC changes: > - Currently 27 PMC members. > - New PMC members: > - Pedro Boado was added to the PMC on Thu Jun 07 2018 > - Vincent Poon was added to the PMC on Thu Jun 07 2018 > > ## Committer base changes: > - Currently 36 committers. > - Ohad Shaham was added as a committer on Fri Jun 08 2018 > > ## Releases: > - 4.14.0 was released on Fri Jun 08 2018 > - 5.0.0 was released on Sat Jul 14 2018 > > ## Mailing list activity: > - List activity is largely consistent with previous months. There is a > slight down-tick in volume to users@phoenix. > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
[GitHub] phoenix issue #315: PHOENIX-3655 Global Phoenix Client Metrics for PQS
Github user karanmehta93 commented on the issue: https://github.com/apache/phoenix/pull/315 Thanks @xcangCRM and @joshelser for the review. I have addressed your review comments and added tag support to differentiate between various types of client when publishing metrics to JMX. ---
[jira] [Created] (PHOENIX-4833) Function Undefined when using nested sub query
Dinesh created PHOENIX-4833: --- Summary: Function Undefined when using nested sub query Key: PHOENIX-4833 URL: https://issues.apache.org/jira/browse/PHOENIX-4833 Project: Phoenix Issue Type: Bug Affects Versions: 4.13.1 Environment: Hadoop 2.9.0 Hbase 1.3.2 Phoenix 4.13.1-Hbase1.3 Reporter: Dinesh Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> select usagekb(strstatscount) from (SELECT to_char(coalesce(ROUND(SUM(avgusage)/SUM(users)),0),'#') AS strstatscount FROM (SELECT SUM(DOWNLOADDATA)/COUNT(DISTINCT(OFFLOADDATE)) AS avgusage,COUNT(DISTINCT(IMEI)) as users FROM JIOANDSF.TBLIMEIPLMNWISEOFFLOAD WHERE OFFLOADCOUNT > 0 GROUP BY IMEI,OFFLOADDATE) foo) ; Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:476) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1742) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) FROM (SELECT to_char(SUM(DOWNLOADDATA),'#') AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 ) ; +--+ | USAGEINKB(TO_CHAR(SUM(A.DOWNLOADDATA))) | +--+ | 7.82GB | +--+ 1 row selected (0.033 seconds) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4833) Function Undefined when using nested sub query
[ https://issues.apache.org/jira/browse/PHOENIX-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh updated PHOENIX-4833: Description: Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) as dd from (select to_char(SUM(strstatscount),'#') from (SELECT SUM(DOWNLOADDATA)AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA )) ; Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:476) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1742) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) FROM (SELECT to_char(SUM(DOWNLOADDATA),'#') AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 ) ; +--+ |USAGEINKB(TO_CHAR(SUM(A.DOWNLOADDATA))) | +--+ |7.82GB | +--+ 1 row selected (0.033 seconds) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> was: Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> select usagekb(strstatscount) from (SELECT to_char(coalesce(ROUND(SUM(avgusage)/SUM(users)),0),'#') AS strstatscount FROM (SELECT SUM(DOWNLOADDATA)/COUNT(DISTINCT(OFFLOADDATE)) AS avgusage,COUNT(DISTINCT(IMEI)) as users FROM JIOANDSF.TBLIMEIPLMNWISEOFFLOAD WHERE OFFLOADCOUNT > 0 GROUP BY IMEI,OFFLOADDATE) foo) ; Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStateme
[jira] [Updated] (PHOENIX-4833) Function Undefined when using nested sub query
[ https://issues.apache.org/jira/browse/PHOENIX-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated PHOENIX-4833: - Description: Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) as dd from (select to_char(SUM(strstatscount),'#') from (SELECT SUM(DOWNLOADDATA)AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA )) ; Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:476) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1742) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) FROM (SELECT to_char(SUM(DOWNLOADDATA),'#') AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 ) ; +--+ |USAGEINKB(TO_CHAR(SUM(A.DOWNLOADDATA))) | +--+ |7.82GB | +--+ 1 row selected (0.033 seconds) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> was: Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) as dd from (select to_char(SUM(strstatscount),'#') from (SELECT SUM(DOWNLOADDATA)AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA )) ; Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:476) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442) at org
[jira] [Updated] (PHOENIX-4833) Function Undefined when using nested sub query
[ https://issues.apache.org/jira/browse/PHOENIX-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh updated PHOENIX-4833: Description: Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscounts) as usage from (select to_char(avg(strstatscount),'#') as strstatscounts from (SELECT SUM(DOWNLOADDATA) AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 )); Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:476) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1742) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:813) at sqlline.SqlLine.begin(SqlLine.java:686) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:291) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) as usage FROM (SELECT to_char(SUM(DOWNLOADDATA),'#') AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 ) ; +-+ | USAGE | +-+ | 7.82GB | +-+ 1 row selected (0.053 seconds) 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscounts) as usage from (select to_char(strstatscount,'#') as strstatscounts from (SELECT SUM(DOWNLOADDATA) AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 )) ; +-+ | USAGE | +-+ | 7.82GB | +-+ 1 row selected (0.041 seconds) was: Function Undefined when using nested sub query 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) as dd from (select to_char(SUM(strstatscount),'#') from (SELECT SUM(DOWNLOADDATA)AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA )) ; Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB (state=42F01,code=6001) org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) at org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) at org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) at org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) at org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) at org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) at org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) at org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157)
[jira] [Updated] (PHOENIX-4833) UDF Function Undefined when using nested sub query with Aggregation
[ https://issues.apache.org/jira/browse/PHOENIX-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh updated PHOENIX-4833: Summary: UDF Function Undefined when using nested sub query with Aggregation (was: Function Undefined when using nested sub query) > UDF Function Undefined when using nested sub query with Aggregation > --- > > Key: PHOENIX-4833 > URL: https://issues.apache.org/jira/browse/PHOENIX-4833 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.13.1 > Environment: Hadoop 2.9.0 > Hbase 1.3.2 > Phoenix 4.13.1-Hbase1.3 >Reporter: Dinesh >Priority: Major > > Function Undefined when using nested sub query > > 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscounts) > as usage from (select to_char(avg(strstatscount),'#') as strstatscounts from > (SELECT SUM(DOWNLOADDATA) AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA > HAVING COUNT(*) > 0 )); > Error: ERROR 6001 (42F01): Function undefined. functionName=USAGEKB > (state=42F01,code=6001) > org.apache.phoenix.schema.FunctionNotFoundException: ERROR 6001 (42F01): > Function undefined. functionName=USAGEKB > at > org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.resolveFunction(FromCompiler.java:725) > at > org.apache.phoenix.compile.ExpressionCompiler.visitLeave(ExpressionCompiler.java:327) > at > org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:696) > at > org.apache.phoenix.compile.ProjectionCompiler$SelectClauseVisitor.visitLeave(ProjectionCompiler.java:585) > at > org.apache.phoenix.parse.FunctionParseNode.accept(FunctionParseNode.java:86) > at > org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:412) > at > org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:561) > at > org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:522) > at > org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:202) > at > org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:476) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:442) > at > org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:300) > at > org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:290) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:289) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1742) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:813) > at sqlline.SqlLine.begin(SqlLine.java:686) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:291) > > > > 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscount) > as usage FROM (SELECT to_char(SUM(DOWNLOADDATA),'#') AS strstatscount FROM > JIOANDSF.TBLRAWOFFLOADDATA HAVING COUNT(*) > 0 ) ; > +-+ > | USAGE | > +-+ > | 7.82GB | > +-+ > 1 row selected (0.053 seconds) > > > > 0: jdbc:phoenix:scipnode1,scipnode2,scipnode3> SELECT usagekb(strstatscounts) > as usage from (select to_char(strstatscount,'#') as strstatscounts from > (SELECT SUM(DOWNLOADDATA) AS strstatscount FROM JIOANDSF.TBLRAWOFFLOADDATA > HAVING COUNT(*) > 0 )) ; > +-+ > | USAGE | > +-+ > | 7.82GB | > +-+ > 1 row selected (0.041 seconds) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
