[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4701: -- Attachment: PHOENIX-4701_addendum2.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_addendum.patch, > PHOENIX-4701_addendum2.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, > PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: (was: PHOENIX-4701_addendum.patch) > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_addendum.patch, > PHOENIX-4701_master.patch, PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, > PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701_addendum.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_addendum.patch, > PHOENIX-4701_master.patch, PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, > PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701_addendum.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_addendum.patch, > PHOENIX-4701_master.patch, PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, > PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Fix Version/s: (was: 4.15.0) 4.14.0 > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, > PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Fix Version/s: 5.0.0 > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, > PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701_v2.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_v2.patch, PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, > PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Issue Type: Improvement (was: Bug) > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Improvement >Reporter: James Taylor >Assignee: Ankit Singhal >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701_master.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_master.patch, > PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701.patch, PHOENIX-4701_wip1.patch, > PHOENIX-4701_wip2.patch, PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal updated PHOENIX-4701: --- Attachment: PHOENIX-4701_wip3.patch > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch, > PHOENIX-4701_wip3.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-4701) Write client-side metrics asynchronously to SYSTEM.LOG
[ https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4701: -- Summary: Write client-side metrics asynchronously to SYSTEM.LOG (was: Store client-side metrics in SYSTEM.LOG) > Write client-side metrics asynchronously to SYSTEM.LOG > -- > > Key: PHOENIX-4701 > URL: https://issues.apache.org/jira/browse/PHOENIX-4701 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: James Taylor >Priority: Major > Fix For: 4.15.0 > > Attachments: PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch > > > Rather than inventing a new, different set of client-side metrics to persist, > we should just persist our [client > metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG. The > metrics captures all the same information as your QueryLogInfo (and much > more), rolls all the information up to a single set of metrics for each > Phoenix statement (aggregating/merging parallel scans, etc), and can emits a > single log line (which could be written in a single upsert statement). At > SFDC, we emit this information into a file system log in a layer above (and > use Splunk to produce nifty dashboard for monitoring), but this could easily > be emitted directly in Phoenix and go through your asynchronous write path > (and then use Phoenix queries to produce the same kind of dashboards). The > only piece would be to add the concept of a log level to each metric to > enable statically controlling which metrics are output. > With this approach, the SYSTEM.LOG table could be declared immutable and use > our dense storage format with a single byte for column encoding and get a > 3-5x perf gain. This would also open the door for users to potentially add > secondary indexes on the table. See schema identified in the wip2 patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
