Hello,
Metrics are not saved when I put a timestamps with metrics.
I've got through github issue and some of prometheus codes but not yet
getting the root cause.
I would be happy if I can get why I'm getting below errors.
Errors
*Error 1*
I got below error when executing prometheus with prometheus.yml (v1).
Please see blow for the prometheus.yml (v1).
prometheus_1 | level=warn ts=2020-05-25T12:29:17.216Z caller=scrape.go:1240
component="scrape manager" scrape_pool=my-app target=https://<internal
server address>:443/metrics msg="Error on ingesting samples that are too
old or are too far into the future" num_dropped=600
*Error2*
I got massive amount of below log when running prometheus with
prometheus.yml (v2). Please see below for prometheus.yml (v2)
prometheus_1 | level=info ts=2020-05-25T15:57:25.857Z caller=head.go:732
component=tsdb msg="WAL checkpoint complete" first=1164 last=1165
duration=2.1818ms
prometheus_1 | level=info ts=2020-05-25T15:57:25.867Z caller=head.go:662
component=tsdb msg="head GC completed" duration=509.9µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.879Z caller=head.go:662
component=tsdb msg="head GC completed" duration=549.7µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.882Z caller=head.go:732
component=tsdb msg="WAL checkpoint complete" first=1166 last=1167
duration=2.7017ms
prometheus_1 | level=info ts=2020-05-25T15:57:25.892Z caller=head.go:662
component=tsdb msg="head GC completed" duration=564.7µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.903Z caller=head.go:662
component=tsdb msg="head GC completed" duration=651.9µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.906Z caller=head.go:732
component=tsdb msg="WAL checkpoint complete" first=1168 last=1169
duration=2.8861ms
prometheus_1 | level=info ts=2020-05-25T15:57:25.916Z caller=head.go:662
component=tsdb msg="head GC completed" duration=562.8µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.925Z caller=head.go:662
component=tsdb msg="head GC completed" duration=633µs
Background
I'm testing prometheus for custom app.
I've created endpoint which expose metrics in text-based format for the app.
Since metrics are calculated with batch job, I've put a timestamps to
metrics to show when the metrics was calculated.
The batch job is executed in every 5 min, so the timestamp will not be
older than 5 min compared to the current timestamp.
Here is an example of exposed data. There are about 600 lines in real data.
#HELP processed_rows_total the total number of processed rows
#TYPE processed_rows_total counter
processed_rows_total{stream_name="a", result="success"} 93163 1590422899
processed_rows_total{stream_name="a", result="error"} 0 1590422899
processed_rows_total{stream_name="b", result="success"} 45695851 1590422899
processed_rows_total{stream_name="b", result="error"} 0 1590422899
Environment
Running with docker.
docker-compose.yml
prometheus:
image: prom/prometheus:v2.17.2
entrypoint:
- "/bin/prometheus"
- "--log.level=debug"
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "127.0.0.1:9090:9090"
prometheus.yml (v1)
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default
is every 1 minute.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries
scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'my-app'
metrics_path: '/metrics'
scrape_interval: 1m
scheme: 'https'
static_configs:
- targets: ['<internal server address>']
prometheus.yml (v2)
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default
is every 1 minute.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
- job_name: 'my-app'
metrics_path: '/metrics'
scrape_interval: 1m
scheme: 'https'
static_configs:
- targets: ['<internal server address>']
Other notes
With *honor-timestamps: false* configuration in prometheus.yml, metrics are
saved correctly.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/64c8a462-5130-4747-a1a5-49e67b975aa3%40googlegroups.com.
