[
https://issues.apache.org/jira/browse/BEAM-6290?focusedWorklogId=183744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183744
]
ASF GitHub Bot logged work on BEAM-6290:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/19 15:16
Start Date: 10/Jan/19 15:16
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #7458: [BEAM-6290]
Generic schema for metrics
URL: https://github.com/apache/beam/pull/7458
A more future-proof solution for saving metrics for load tests. It will
require cleaning the existing BigQuery tables but better sooner than later. We
do not store valuable data there yet.
@udim @kkucharc could you take a look?
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
| --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 183744)
Time Spent: 10m
Remaining Estimate: 0h
> Make the schema for BQ tables storing metric results more generic (JAVA)
> ------------------------------------------------------------------------
>
> Key: BEAM-6290
> URL: https://issues.apache.org/jira/browse/BEAM-6290
> Project: Beam
> Issue Type: Sub-task
> Components: testing
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, we keep the metrics results in BQ in tables with a schema like
> this:
> timestamp | total_bytes | run_time | (possibly other BQ columns)
> every time we want to add a new column the schema has to be extended. This is
> not convenient given the fact that any load test can have different metrics
> stored. This in turn would cause multiple BQ tables each queried differently.
> We can provide a more generic schema, like so:
> test_id | timestamp | metric | value
> thanks to that, every metric, whatever it's name is, can be saved in the
> table as a separate row. This gives more elasticity in storing metrics and is
> still easy to query and plot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
