[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497467#comment-16497467 ] Lars Hofhansl commented on PHOENIX-2715: Thanks [~an...@apache.org], and sorry for swooping in late and complaining. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491324#comment-16491324 ] Hudson commented on PHOENIX-2715: - ABORTED: Integrated in Jenkins build PreCommit-PHOENIX-Build #1887 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1887/]) PHOENIX-2715 Query Log (addendum 2) (ankitsinghal59: rev 8d80d5a987376dfde822ea60be93618682a45d9c) * (edit) bin/hbase-site.xml > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490441#comment-16490441 ] Hudson commented on PHOENIX-2715: - ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #143 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/143/]) PHOENIX-2715 Query Log (addendum 2) (ankitsinghal59: rev 81f887584e3e4c86e487e0c3ed8f4729b1283f1f) * (edit) bin/hbase-site.xml > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490257#comment-16490257 ] Ankit Singhal commented on PHOENIX-2715: bq.Next RC coming tomorrow am. Pushed the change now. (y) > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490252#comment-16490252 ] James Taylor commented on PHOENIX-2715: --- Yes, please [~ankit.singhal]. Next RC coming tomorrow am. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490233#comment-16490233
]
Ankit Singhal commented on PHOENIX-2715:
bq. Looking at the code I see the default is OFF, so something is not working
right, unless I misunderstood something.
By mistake, I have committed bin/hbase-site.xml with logging.level to DEBUG.
Let me remove that.
{code}
https://github.com/apache/phoenix/blob/master/bin/hbase-site.xml
{code}
bq. Everything but OFF is a bad default, IMHO.
Any suggestions, It will be an easy change before I commit addendum for
hbase-site.xml?
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch,
> PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490153#comment-16490153 ] Lars Hofhansl commented on PHOENIX-2715: The default is supposed to the OFF, right? I just true with a vanilla build from Phoenix' master branch and I see SYSTEM.LOG is being written to. Looking at the code I see the default *is* OFF, so something is not working right, unless I misunderstood something. Everything but OFF is a bad default, IMHO. [~an...@apache.org] > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445408#comment-16445408 ] Hudson commented on PHOENIX-2715: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1864 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1864/]) PHOENIX-2715 Query Log (addendum) (ankitsinghal59: rev 751043d8f6aa2f9aa54035b58255462456cfb196) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/SystemCatalogCreationOnConnectionIT.java > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445335#comment-16445335 ] Ankit Singhal commented on PHOENIX-2715: Thanks [~apurtell] for notifying the failure. I have pushed the test fix now. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438255#comment-16438255 ] James Taylor commented on PHOENIX-2715: --- Thanks, [~an...@apache.org]! I'll start a new thread on dev to clear up any confusion. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438251#comment-16438251 ] Ankit Singhal commented on PHOENIX-2715: bq. Actually, it was backported to 1.1 (thanks!), but just not 0.98. I thought we discussed to not port features in 0.98 but not sure if that was agreed upon. Nevertheless, I have now committed this to 0.98 as well. https://lists.apache.org/thread.html/%3ccajsqkibyxgce_axfnpkym-awvvrcg1i7wat6ir4xfzqnsjq...@mail.gmail.com%3E > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437996#comment-16437996 ] James Taylor commented on PHOENIX-2715: --- Actually, it was backported to 1.1 (thanks!), but just not 0.98. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437898#comment-16437898 ] James Taylor commented on PHOENIX-2715: --- [~an...@apache.org] - would it be possible to get this into 1.1 and 0.98 branches too? Merges are becoming increasingly harder without this there. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434134#comment-16434134 ] Hudson commented on PHOENIX-2715: - FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1828 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1828/]) PHOENIX-2715 Query Log (addendum) Fix old com.lmax.disruptor dependency (ankitsinghal59: rev d48518bc2ddea3f4c6cd9ef5a6c2504b65255da2) * (edit) phoenix-core/pom.xml * (edit) phoenix-flume/pom.xml * (edit) phoenix-kafka/pom.xml > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433996#comment-16433996 ] Hudson commented on PHOENIX-2715: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #88 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/88/]) PHOENIX-2715 Query Log (addendum) Fix old com.lmax.disruptor dependency (ankitsinghal59: rev a9c38bcd250e24ddab42c41e64b20cd57c806ba1) * (edit) phoenix-core/pom.xml * (edit) phoenix-kafka/pom.xml * (edit) phoenix-flume/pom.xml > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432373#comment-16432373 ] Hudson commented on PHOENIX-2715: - FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1826 (See [https://builds.apache.org/job/PreCommit-PHOENIX-Build/1826/]) PHOENIX-2715 Query Log (Ankit Singhal) (ankitsinghal59: rev 0bc0368455ae03de2f3f2cabd72de06e5cb90cbb) * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ScanMetricsHolder.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogger.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/LogWriter.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/LogLevel.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryDatabaseMetaDataIT.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogState.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerDefaultExceptionHandler.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/TenantSpecificTablesDDLIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ReadMetricQueue.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java * (edit) phoenix-core/pom.xml * (edit) pom.xml * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/StatementContext.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogDetailsEventHandler.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogInfo.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/RingBufferEventTranslator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixResultSet.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerDisruptor.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryLoggerIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/ScanningResultIterator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixPreparedStatement.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/RingBufferEvent.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/TableLogWriter.java PHOENIX-2715 Query Log (addendum) (ankitsinghal59: rev ed636616138cbb8db4b71c9cf8a1b2fa7b503aa0) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/MigrateSystemTablesToSystemNamespaceIT.java > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432125#comment-16432125 ] Hudson commented on PHOENIX-2715: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #86 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/86/]) PHOENIX-2715 Query Log (addendum) (ankitsinghal59: rev 768bff97e28fa5def199c8303b84fc71ca93df9a) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/MigrateSystemTablesToSystemNamespaceIT.java > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431995#comment-16431995 ] Hudson commented on PHOENIX-2715: - FAILURE: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #85 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/85/]) PHOENIX-2715 Query Log (Ankit Singhal) (ankitsinghal59: rev 6468abb005fa6068bb6e5d6d77572eff9291bee2) * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDatabaseMetaData.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/RingBufferEvent.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixPreparedStatement.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixResultSet.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogState.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerDefaultExceptionHandler.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogDetailsEventHandler.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/TableLogWriter.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/LogWriter.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogInfo.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerUtil.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServices.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/RingBufferEventTranslator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/iterate/ScanningResultIterator.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLoggerDisruptor.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ScanMetricsHolder.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/TenantSpecificTablesDDLIT.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/QueryLogger.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/StatementContext.java * (edit) pom.xml * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryDatabaseMetaDataIT.java * (add) phoenix-core/src/main/java/org/apache/phoenix/log/LogLevel.java * (add) phoenix-core/src/it/java/org/apache/phoenix/end2end/QueryLoggerIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataProtocol.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/monitoring/ReadMetricQueue.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java * (edit) phoenix-core/pom.xml > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431812#comment-16431812 ] Ankit Singhal commented on PHOENIX-2715: Thank you guys for the review. Committed in master, 5.x and 4.x branches. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431614#comment-16431614 ] Josh Elser commented on PHOENIX-2715: - +1 this is fantastic. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431090#comment-16431090 ] Andrew Purtell commented on PHOENIX-2715: - Awesome, thanks so much > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430121#comment-16430121 ] Ankit Singhal commented on PHOENIX-2715: Thanks [~apurtell], these were great suggestions and are easy to implement. The latest patch now has all these changes incorporated. bq. I'm noticing that I get ~4 entries in system.log for every one query I run in sqlline against a user table. Yes, it will happen with sqlline only as it queries meta table for primary key and column information. bq.Pruning out system table queries (or maybe having an option to prune them) would be a nice-to-have follow-on – my guess is that it's hard to identify these queries. I fixed this in the latest patch, but it may not cover the complex queries which are EXPLICITLY running on SYSTEM table which include join or derived tables or bind node. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch, PHOENIX-2715_master_V2.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430012#comment-16430012 ] Josh Elser commented on PHOENIX-2715: - Great! I got this working locally with your updated patch. Thanks again, Ankit! One more thing: did we discuss at some point about not logging system table queries? I'm noticing that I get ~4 entries in system.log for every one query I run in sqlline against a user table. I think making a quick pass to ensure some good Interfaces exist should be done before an initial commit. It would be nice to see the sampling but I think that could be done follow-on (not sure if Andrew feels strongly one way or the other). Pruning out system table queries (or maybe having an option to prune them) would be a nice-to-have follow-on – my guess is that it's hard to identify these queries. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429938#comment-16429938 ] Andrew Purtell commented on PHOENIX-2715: - Cool. Perhaps an operator can configure a default sampling probability in site configuration but it could be overridden on a connection basis with a connection string parameter. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch, > PHOENIX-2715_master_V1.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429927#comment-16429927
]
Josh Elser commented on PHOENIX-2715:
-
Thanks for the updated patch (on the weekend too)! Will test-drive it too.
{quote} * If not an alternate implementation of LogWriter, at least a better
factoring. Make LogWriter abstract or an interface. That should be quickly
accomplished.{quote}
I think this is a good suggestion for an initial implementation. Since we're
starting out fresh, putting the abstraction points in there now would help. A
LogWriter interface would be great.
{quote}Can we have a knob for probabilistic sampling? This is really easy to
implement. Add one config parameter, a float or double, one that can ideally be
changed dynamically.
{quote}
I like this too. Great suggestion on implementation detail too, Andrew :)
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch,
> PHOENIX-2715_master_V1.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429814#comment-16429814
]
Andrew Purtell commented on PHOENIX-2715:
-
Random thoughts on trying to use this in a production setting
* Cool to have a LogWriter that puts the log into a table, so the log itself
can be queried. Powerful. How about a LogWriter that just emits to Java logging
as well. HBase+Phoenix systems throw off a ton of this type of logging, so we
already need a solution for managing it, for which query log would just be a
new subset. Many places may want their log search solution to be based on
something else (Splunk, Elastic, Solr, etc.)
* If not an alternate implementation of LogWriter, at least a better
factoring. Make LogWriter abstract or an interface. That should be quickly
accomplished.
* What happens if query logging becomes too expensive? We can turn it all the
way on and all the way off. Can we have a knob for probabilistic sampling? This
is really easy to implement. Add one config parameter, a float or double, one
that can ideally be changed dynamically. Call it something like
QUERY_LOG_SAMPLE_RATE (not a great name but whatever) In the code where you go
to do the query logging, add a conditional \{{if
(ThreadLocalRandom.getCurrent().getDouble() <=
getConfig(QUERY_LOG_SAMPLE_RATE))}} . Easy. So if logging 100% of queries is
too expensive (at QUERY_LOG_SAMPLE_RATE = 1.0), we can try logging 50% of them
(at QUERY_LOG_SAMPLE_RATE = 0.5), or 10% of them (at QUERY_LOG_SAMPLE_RATE =
0.1), or 1% of them (at QUERY_LOG_SAMPLE_RATE = 0.01).
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch,
> PHOENIX-2715_master_V1.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429751#comment-16429751
]
Ankit Singhal commented on PHOENIX-2715:
Thanks [~elserj] for testing it.
bq. I see the SYSTEM.LOG table was created, but the table is not getting
propagated with any information. Maybe it's related to just using sqlline? I've
been waiting a bit to see if the results are just delayed, and that doesn't
seem to be the case. I can see that the QueryLogger is created and then stopped
before I get a prompt. Additionally, I can see that there is no QueryLogger*
thread running in the sqlline jvm
My bad, while porting patch for master , I moved disruptor from connection to
connection query services so that we will not create it for every connection,
but close() was happening in connection only, so the disruptor is getting
shutdown for any connection close. I fixed it in the latest patch.
bq. maybe I need to enable request metrics for it to
work?(QueryServices.COLLECT_REQUEST_LEVEL_METRICS)
this needs to be set if we need scan metrics to be logged.(SCAN_METRICS_JSON
column)
{code}
SCAN_METRICS_JSON
[{"scan":{"loadColumnFamiliesOnDemand":null,"filter":"COLUMN_FAMILY IS
NULL","startRow":"\\x00\\x00SYSTEM.LOG\\x00\\x01","stopRow":"\\x00\\x00SYSTEM.LOG\\x01","batch":-1,"cacheBlocks":true,"totalColumns":7,"maxResultSize":2097152,"families":{"0":["COLUMN_SIZE","DATA_TYPE","KEY_SEQ","PK_NAME"]},"caching":2147483647,"maxVersions":1,"timeRange":[0,9223372036854775807]},
"scanMetrics":{"BYTES_IN_RESULTS":0,"MILLIS_BETWEEN_NEXTS":2,"REGIONS_SCANNED":1,"REMOTE_RPC_CALLS":0,"BYTES_IN_REMOTE_RESULTS":0,"ROWS_SCANNED":0,"ROWS_FILTERED":0,"RPC_RETRIES":0,"RPC_CALLS":1,"NOT_SERVING_REGION_EXCEPTION":0,"REMOTE_RPC_RETRIES":0}}]
{code}
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch,
> PHOENIX-2715_master_V1.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429618#comment-16429618
]
Josh Elser commented on PHOENIX-2715:
-
I pulled down the Master patch and tried that out locally. I see the SYSTEM.LOG
table was created, but the table is not getting propagated with any
information. I didn't set any extra configuration – looking at the IT, maybe I
need to enable request metrics for it to work?
({{QueryServices.COLLECT_REQUEST_LEVEL_METRICS}})
Maybe it's related to just using sqlline? I've been waiting a bit to see if the
results are just delayed, and that doesn't seem to be the case. I can see that
the QueryLogger is created and then stopped before I get a prompt.
Additionally, I can see that there is no {{QueryLogger*}} thread running in the
sqlline jvm. I'll try to play around with this some more tmrw.
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428876#comment-16428876 ] Josh Elser commented on PHOENIX-2715: - Just getting back (sorry I didn't get to try this out before stepping out of the office for a few days). No further comments from me at this point, I think a final patch would be good. I'll plan to do some testing this weekend to make up for my time off and making ya wait. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424366#comment-16424366 ] Ankit Singhal commented on PHOENIX-2715: Thanks [~apurtell] for the review, will change the log level of bind parameters to TRACE and update document with the correct logging parameters. [~apurtell]/[~sergey.soldatov], Any more comments or should I upload a final patch with last discussed changes? > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423434#comment-16423434 ] Andrew Purtell commented on PHOENIX-2715: - When I said separate log line I meant separate log level. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423433#comment-16423433 ] Andrew Purtell commented on PHOENIX-2715: - Thank you. My mistake for not seeing the earlier comment. Didn't see it in the patch I looked at, but may have missed a different log level. Veto rescinded. Separate log line would work. We don't need a server side configuration. Compliance is usually a cooperative effort undertaken by the whole organization. As long as it is possible to apply the controls, and there is a documented process to confirm the controls are in place (eg correctly configured log levels), and there is documentation the process was followed, this should be good. Certainly for a first version of the feature. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423121#comment-16423121
]
Andrew Purtell commented on PHOENIX-2715:
-
Since my query is being ignored, I am -1 on this patch until we discuss it.
This is a binding veto.
Please respond:
{quote}Is it possible to set a flag or a configuration variable so bind
parameter logging can be left out?
{quote}
Reason I ask is the query log feature will otherwise not be useful in many
environments with compliance requirements for personally identifiable
information (commonly abbreviated as PII). Assuming a coder can isolate PII to
bind variables, which is a reasonable tradeoff, It doesn't seem hard to do
this. So why not at least consider it? Seems easy to make logging of bind
parameter information configurable.
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422079#comment-16422079 ] Ankit Singhal commented on PHOENIX-2715: bq. Wondering if we want this to be INFO, or even OFF, by default. Not sure.. I think we can keep it OFF as sometimes a client will not have access to SYSTEM.LOG and they don't want to see there logs getting flood with such exceptions. User can enable it when they really need to debug their own queries and Admin can make it ON on PQS server(for audit or any other need) where it makes more sense as all client queries will be logged. bq. Wondering if this should be DEBUG. I would guess that, in most cases, we can understand the intent of a query without the parameters being explicitly present? bq. Is it possible to set a flag or a configuration variable so bind parameter logging can be left out? We can mark it with LogLevel.TRACE so that it will be logged if TRACE is enabled. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420665#comment-16420665 ] Andrew Purtell commented on PHOENIX-2715: - Is it possible to set a flag or a configuration variable so bind parameter logging can be left out? > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[
https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419840#comment-16419840
]
Josh Elser commented on PHOENIX-2715:
-
{code:java}
+public static final String DEFAULT_LOGGING_LEVEL = "DEBUG";{code}
Wondering if we want this to be INFO, or even OFF, by default. Not sure..
{code:java}
+BIND_PARAMETERS_I(BIND_PARAMETERS,QueryLogState.STARTED,
LogLevel.INFO,PVarchar.INSTANCE),{code}
Wondering if this should be DEBUG. I would guess that, in most cases, we can
understand the intent of a query without the parameters being explicitly
present?
Going to try this out locally.
> Query Log
> -
>
> Key: PHOENIX-2715
> URL: https://issues.apache.org/jira/browse/PHOENIX-2715
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Nick Dimiduk
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch
>
>
> One useful feature of other database systems is the query log. It allows the
> DBA to review the queries run, who's run them, time taken, This serves
> both as an audit and also as a source of "ground truth" for performance
> optimization. For instance, which columns should be indexed. It may also
> serve as the foundation for automated performance recommendations/actions.
> What queries are being run is the first piece. Have this data tied into
> tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very
> useful.
> This might take the form of clients writing data to a new system table, but
> other implementation suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418603#comment-16418603 ] Ankit Singhal commented on PHOENIX-2715: Thanks [~sergey.soldatov] for taking a look. bq. Could you please adopt to master branch? Have now attached a patch for master as well. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch, PHOENIX-2715_master.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418488#comment-16418488 ] Sergey Soldatov commented on PHOENIX-2715: -- [~an...@apache.org] At first glance it looks reasonable. But it would be nice to test it in real env. Could you please adopt to master branch? I tried it with HBase branch-2, but it seems that something is wrong with my env and HBase itself doesn't work properly ( looking into it whether it was caused by some recent commits or it's a problem on my side) > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Assignee: Ankit Singhal >Priority: Major > Attachments: PHOENIX-2715.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417376#comment-16417376 ] Ankit Singhal commented on PHOENIX-2715: [~elserj], [~sergey.soldatov],[~chrajeshbab...@gmail.com] can anyone help me with the review of the attached patch. I tried to cover as much as we discussed in the [thread|https://lists.apache.org/thread.html/89dd45a990a5f163d89433d672ca47c75e8ea86c1b21ba67d2471826@%3Cdev.phoenix.apache.org%3E] and documented [here|https://s.apache.org/phoenix-query-log]. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Priority: Major > Attachments: PHOENIX-2715.patch > > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384223#comment-16384223 ] Josh Elser commented on PHOENIX-2715: - Nick kindly pointed me to this one. There's a discussion on the mailing list presently with (what hopes to be) a concrete plan. https://lists.apache.org/thread.html/89dd45a990a5f163d89433d672ca47c75e8ea86c1b21ba67d2471826@%3Cdev.phoenix.apache.org%3E > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk >Priority: Major > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170670#comment-15170670 ] James Taylor commented on PHOENIX-2715: --- Good point about the gap if sampling is on with tracing, [~apurtell]. Client-side metrics alone doesn't solve this either, but I neglected to mention how we get at this information today at SFDC. We use client-side metrics to gather a bunch of useful metrics and produce a log line per statement (including the text of the SQL). Then we use Splunk to produce dashboards that show aggregated and summarized information across all use cases for the cluster. But without something like Spunk, there's a gap, so this JIRA would indeed fill that. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170537#comment-15170537 ] Andrew Purtell commented on PHOENIX-2715: - No unfortunately tracing does not provide the necessary detail, at least when sampling is in effect, and certainly metrics do not. You don't want every statement recorded for those use cases anyway. So there is still value in this as I said > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170480#comment-15170480 ] James Taylor commented on PHOENIX-2715: --- FYI, we have two mechanisms that can do this already: - Client-side metrics collection (documentation coming soon) - this is what we use to drive our very nice internal dashboards build by [~jfernando_sfdc]. - Tracing - https://phoenix.apache.org/tracing.html > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2715) Query Log
[ https://issues.apache.org/jira/browse/PHOENIX-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170476#comment-15170476 ] Andrew Purtell commented on PHOENIX-2715: - Great idea. It can also be useful for post hoc characterization of production workload for load generation tools that simulate production conditions (perhaps scaled) for dev and staging environments. Lots of possibilities. If you're willing to also consider this use case, then we should think about how sensitive information in queries can be masked or substituted with fake data, to avoid security and compliance issues in some environments. > Query Log > - > > Key: PHOENIX-2715 > URL: https://issues.apache.org/jira/browse/PHOENIX-2715 > Project: Phoenix > Issue Type: New Feature >Reporter: Nick Dimiduk > > One useful feature of other database systems is the query log. It allows the > DBA to review the queries run, who's run them, time taken, This serves > both as an audit and also as a source of "ground truth" for performance > optimization. For instance, which columns should be indexed. It may also > serve as the foundation for automated performance recommendations/actions. > What queries are being run is the first piece. Have this data tied into > tracing results and perhaps client-side metrics (PHOENIX-1819) becomes very > useful. > This might take the form of clients writing data to a new system table, but > other implementation suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
