If you are writing points across a wide time range, change the default shard duration: http://docs.influxdata.com/influxdb/v1.0/concepts/schema_and_data_layout/#shard-group-duration-recommendations
Are you overwriting points? On Tue, Nov 8, 2016 at 12:24 PM, <[email protected]> wrote: > On Tuesday, November 8, 2016 at 9:18:08 PM UTC+2, [email protected] > wrote: > > Hi All > > > > I am inserting a low volume of data (less than 2000 points every 15 > minutes). I do this via a Java program (using a batch). What I have noticed > is inserting the same amount of data takes longer and longer and over time > and the CPU usage constantly increases (almost linearly) over 24 hours (It > is not swapping as far as I can see). > > > > If I restart InfluxDB the CPU usage goes back to normal for a while but > it slowly and steadily climbs with each insert. The only thing that reduces > the CPU load for writes (roughly the same amount of data every time) is if > I restart InfluxDB (I have tried stopping and starting the Java program and > it has no effect (I thought it might be holding onto ports etc). > > > > I am using a very small server (1G / 1 CPU) to test it (there is very > little data) but I was wondering if I am configuring InfluxDB properly. > > > > I am using version 1.0.2 of InfluxDB on Ubuntu Linux (on DigitalOcean) > and the "recommended" Java library. > > > > Here is my conf file > > > > reporting-disabled = false > > bind-address = ":8088" > > > > [meta] > > dir = "/var/lib/influxdb/meta" > > retention-autocreate = true > > logging-enabled = true > > > > [data] > > dir = "/var/lib/influxdb/data" > > engine = "tsm1" > > wal-dir = "/var/lib/influxdb/wal" > > wal-logging-enabled = true > > query-log-enabled = false > > cache-max-memory-size = 524288000 > > cache-snapshot-memory-size = 26214400 > > cache-snapshot-write-cold-duration = "1h0m0s" > > compact-full-write-cold-duration = "24h0m0s" > > max-points-per-block = 0 > > max-series-per-database = 1000000 > > trace-logging-enabled = false > > > > [coordinator] > > write-timeout = "10s" > > max-concurrent-queries = 0 > > query-timeout = "0" > > log-queries-after = "0" > > max-select-point = 0 > > max-select-series = 0 > > max-select-buckets = 0 > > > > [retention] > > enabled = true > > check-interval = "30m0s" > > > > [shard-precreation] > > enabled = true > > check-interval = "10m0s" > > advance-period = "30m0s" > > > > [admin] > > enabled = true > > bind-address = ":8083" > > https-enabled = false > > https-certificate = "/etc/ssl/influxdb.pem" > > > > [monitor] > > store-enabled = true > > store-database = "_internal" > > store-interval = "10s" > > > > [subscriber] > > enabled = true > > http-timeout = "30s" > > write-concurrency = 40 > > write-buffer-size = 1000 > > > > [http] > > enabled = true > > bind-address = ":8086" > > auth-enabled = false > > log-enabled = false > > write-tracing = false > > https-enabled = false > > https-certificate = "/etc/ssl/influxdb.pem" > > https-private-key = "" > > max-row-limit = 10000 > > max-connection-limit = 0 > > shared-secret = "" > > realm = "InfluxDB" > > > > [continuous_queries] > > log-enabled = true > > enabled = true > > run-interval = "1s" > > > > Any pointers will be appreciated. > > > > Regards > > Steve van der Merwe > > My apologies here is a log snippet: > > Nov 8 19:06:35 db influxd[17415]: [tsm1] 2016/11/08 19:06:35 compacted > level 2 group 0 of 2 files into 1 files in 452.912177ms > Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 beginning > level 3 compaction of group 0, 4 TSM files > Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting > level 3 group (0) /var/lib/influxdb/data/db/autogen/50/000000551-000000003.tsm > (#0) > Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting > level 3 group (0) /var/lib/influxdb/data/db/autogen/50/000000555-000000003.tsm > (#1) > Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting > level 3 group (0) /var/lib/influxdb/data/db/autogen/50/000000559-000000003.tsm > (#2) > Nov 8 19:06:36 db influxd[17415]: [tsm1] 2016/11/08 19:06:36 compacting > level 3 group (0) /var/lib/influxdb/data/db/autogen/50/000000563-000000003.tsm > (#3) > Nov 8 19:06:37 db influxd[17415]: [tsm1] 2016/11/08 19:06:37 compacted > level 3 group (0) into > /var/lib/influxdb/data/db/autogen/50/000000563-000000004.tsm.tmp > (#0) > Nov 8 19:06:37 db influxd[17415]: [tsm1] 2016/11/08 19:06:37 compacted > level 3 group 0 of 4 files into 1 files in 798.550193ms > Nov 8 19:07:05 db influxd[17415]: [tsm1] 2016/11/08 19:07:05 Snapshot for > path /var/lib/influxdb/data/db/autogen/50 written in 2.163763999s > Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 Snapshot for > path /var/lib/influxdb/data/db/autogen/50 written in 2.6193556s > Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 beginning > level 1 compaction of group 0, 2 TSM files > Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 compacting > level 1 group (0) /var/lib/influxdb/data/db/autogen/50/000000564-000000001.tsm > (#0) > Nov 8 19:07:37 db influxd[17415]: [tsm1] 2016/11/08 19:07:37 compacting > level 1 group (0) /var/lib/influxdb/data/db/autogen/50/000000565-000000001.tsm > (#1) > Nov 8 19:07:38 db influxd[17415]: [tsm1] 2016/11/08 19:07:38 compacted > level 1 group (0) into > /var/lib/influxdb/data/db/autogen/50/000000565-000000002.tsm.tmp > (#0) > Nov 8 19:07:38 db influxd[17415]: [tsm1] 2016/11/08 19:07:38 compacted > level 1 group 0 of 2 files into 1 files in 490.727946ms > Nov 8 19:12:16 db influxd[17415]: [tsm1] 2016/11/08 19:12:16 Snapshot for > path /var/lib/influxdb/data/db/autogen/50 written in 3.521008625s > Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 Snapshot for > path /var/lib/influxdb/data/db/autogen/50 written in 2.915170587s > Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 beginning > level 1 compaction of group 0, 2 TSM files > Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 compacting > level 1 group (0) /var/lib/influxdb/data/db/autogen/50/000000566-000000001.tsm > (#0) > Nov 8 19:12:51 db influxd[17415]: [tsm1] 2016/11/08 19:12:51 compacting > level 1 group (0) /var/lib/influxdb/data/db/autogen/50/000000567-000000001.tsm > (#1) > Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacted > level 1 group (0) into > /var/lib/influxdb/data/db/autogen/50/000000567-000000002.tsm.tmp > (#0) > Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacted > level 1 group 0 of 2 files into 1 files in 421.739431ms > Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 beginning > level 2 compaction of group 0, 2 TSM files > Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacting > level 2 group (0) /var/lib/influxdb/data/db/autogen/50/000000565-000000002.tsm > (#0) > Nov 8 19:12:52 db influxd[17415]: [tsm1] 2016/11/08 19:12:52 compacting > level 2 group (0) /var/lib/influxdb/data/db/autogen/50/000000567-000000002.tsm > (#1) > Nov 8 19:12:53 db influxd[17415]: [tsm1] 2016/11/08 19:12:53 compacted > level 2 group (0) into > /var/lib/influxdb/data/db/autogen/50/000000567-000000003.tsm.tmp > (#0) > Nov 8 19:12:53 db influxd[17415]: [tsm1] 2016/11/08 19:12:53 compacted > level 2 group 0 of 2 files into 1 files in 573.971276ms > Nov 8 19:13:24 db influxd[17415]: [tsm1] 2016/11/08 19:13:24 Snapshot for > path /var/lib/influxdb/data/db/autogen/50 written in 1.842949453s > > -- > Remember to include the version number! > --- > You received this message because you are subscribed to the Google Groups > "InfluxData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/57d726a8-fcc7-488e-8fce-365ab39e467c%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvN-7miLnqbTV4Vq5f8zJiL_05e4%3DAUuRe%3D%3DdfBTaXSUcg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
