The concurrency is fine. Should be easy for it to handle. Are you using tags?
On Fri, Jan 27, 2017 at 10:39 AM, Jeffery K < jeffery.k...@sightlinesystems.com> wrote: > I checked the schema of one of the measurements (they should all be the > same as we're inserting the same data into multiple measurements). > There are 74 field keys. All float type. the field key names are just > strings, some with spaces, and sometimes characters like % or parenthesis. > (). But we've done the proper escaping to do the insert into influx > correctly. The line feed protocol is correct as far as I know, since it's > accepted, and the fields & data look correct. > We are using a separate thread for each of the 230 (want to get to 1000, > but only got this many) measurements for doing the insert. Is there a > problem with this level of concurrency with influx? to elaborate, for each > measurement, our client is creating a http writer (in java) and feeding in > a batch of writes that have been read and put into line feed protocol in > the last 100ms (we initially had this at 500ms, but to reduce load to > influx, lowered it). After each batch "commit", we re-create the http > connection, for the next batch insert. > > I initially had problems testing on the newer influx 1.2 since it seems > the json of returns has changed to have a statement_id, and my jackson > converter to java object didn't like that, but I've since fixed that and > re-tested this scenario with Influx 1.2, and it behaves the same way. gets > killed very quickly by the OOM killer, using around 22GB of RAM. > > > > On Friday, January 27, 2017 at 10:06:04 AM UTC-5, Paul Dix wrote: >> >> My guess is there's an issue with your schema. What does the data look >> like? >> >> On Thu, Jan 26, 2017 at 6:34 PM, Jeffery K <jeffer...@sightlinesystems. >> com> wrote: >> >>> We are using http to write inserts to the /write endpoint and after a >>> very short duration, influx is being kill by the linux OS for out of >>> memory. >>> We've increased the memory 4 fold from the initial value, and influx now >>> has 20GB, but still it is crashing fairly quickly. We are performing about >>> only 230,000 inserts. 1000 inserts into 230 measurements. We are using >>> batching and flushing every 100 milliseconds. >>> >>> We don't believe the data is more than 1GB, but yet, 20 GB of memory >>> isn't enough to write this?! >>> >>> This was done with 1.1.0 version. >>> >>> -- >>> Remember to include the version number! >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "InfluxData" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to influxdb+u...@googlegroups.com. >>> To post to this group, send email to infl...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/influxdb. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/influxdb/67ec84d0-4320-4044-b881-1b23839b2964%40googlegroups.com >>> <https://groups.google.com/d/msgid/influxdb/67ec84d0-4320-4044-b881-1b23839b2964%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > Remember to include the version number! > --- > You received this message because you are subscribed to the Google Groups > "InfluxData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to influxdb+unsubscr...@googlegroups.com. > To post to this group, send email to influxdb@googlegroups.com. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/25f852c4-2b95-4d38-a250-941c55f54a88%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/25f852c4-2b95-4d38-a250-941c55f54a88%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to influxdb+unsubscr...@googlegroups.com. To post to this group, send email to influxdb@googlegroups.com. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CAHRno_wBGhRcsTe0%3DEuBkHe68mFgisXb1COGmVcrhunbQ%2BtQdA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
