[jira] [Updated] (FLINK-33857) Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic
[ https://issues.apache.org/jira/browse/FLINK-33857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33857: --- Labels: pull-request-available (was: ) > Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic > - > > Key: FLINK-33857 > URL: https://issues.apache.org/jira/browse/FLINK-33857 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: elasticsearch-3.0.1 >Reporter: Peter Schulz >Priority: Major > Labels: pull-request-available > > There's a bug in elasticsearch that may lead to underestimated bulk request > sizes. We are hit by this bug and therefore, I would like to propose a simple > improvement to work around it: > {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' > {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. > Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to > the server when any of the configurable limits is hit (there's another bug: > flushing will happen _after_ the limit has exceeded). > (+) *Proposal:* Expose {{BulkProcessor.flush()}} via > {{RequestIndexer.flush()}}. This way we can easily implement logic to decide > when to flush inside the emitter. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33857) Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic
[ https://issues.apache.org/jira/browse/FLINK-33857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Schulz updated FLINK-33857: - Description: There's a bug in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) was: There's a bug in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush inside the emitter. > Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic > - > > Key: FLINK-33857 > URL: https://issues.apache.org/jira/browse/FLINK-33857 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: elasticsearch-3.0.1 >Reporter: Peter Schulz >Priority: Major > Labels: pull-request-available > > There's a bug in elasticsearch that may lead to underestimated bulk request > sizes. We are hit by this bug and therefore, I would like to propose a simple > improvement to work around it: > {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' > {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. > Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to > the server when any of the configurable limits is hit (there's another bug: > flushing will happen _after_ the limit has exceeded). > (+) *Proposal:* Expose {{BulkProcessor.flush()}} via > {{RequestIndexer.flush()}}. This way we can easily implement logic to decide > when to flush inside the emitter. > I created a [pull > request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. > Feedback is highly welcome. :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33857) Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic
[ https://issues.apache.org/jira/browse/FLINK-33857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Schulz updated FLINK-33857: - Description: There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush, inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) was: There's a bug in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush, inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) > Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic > - > > Key: FLINK-33857 > URL: https://issues.apache.org/jira/browse/FLINK-33857 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: elasticsearch-3.0.1 >Reporter: Peter Schulz >Priority: Major > Labels: pull-request-available > > There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in > elasticsearch that may lead to underestimated bulk request sizes. We are hit > by this bug and therefore, I would like to propose a simple improvement to > work around it: > {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' > {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. > Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to > the server when any of the configurable limits is hit (there's another bug: > flushing will happen _after_ the limit has exceeded). > (+) *Proposal:* Expose {{BulkProcessor.flush()}} via > {{RequestIndexer.flush()}}. This way we can easily implement logic to decide > when to flush, inside the emitter. > I created a [pull > request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. > Feedback is highly welcome. :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33857) Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic
[ https://issues.apache.org/jira/browse/FLINK-33857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Schulz updated FLINK-33857: - Description: There's a bug in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush, inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) was: There's a bug in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) > Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic > - > > Key: FLINK-33857 > URL: https://issues.apache.org/jira/browse/FLINK-33857 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: elasticsearch-3.0.1 >Reporter: Peter Schulz >Priority: Major > Labels: pull-request-available > > There's a bug in elasticsearch that may lead to underestimated bulk request > sizes. We are hit by this bug and therefore, I would like to propose a simple > improvement to work around it: > {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' > {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. > Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to > the server when any of the configurable limits is hit (there's another bug: > flushing will happen _after_ the limit has exceeded). > (+) *Proposal:* Expose {{BulkProcessor.flush()}} via > {{RequestIndexer.flush()}}. This way we can easily implement logic to decide > when to flush, inside the emitter. > I created a [pull > request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. > Feedback is highly welcome. :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33857) Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic
[ https://issues.apache.org/jira/browse/FLINK-33857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Schulz updated FLINK-33857: - Description: There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush, inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) was: There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in elasticsearch that may lead to underestimated bulk request sizes. We are hit by this bug and therefore, I would like to propose a simple improvement to work around it: {{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES' {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the server when any of the configurable limits is hit (there's another bug: flushing will happen _after_ the limit has exceeded). (+) *Proposal:* Expose {{BulkProcessor.flush()}} via {{RequestIndexer.flush()}}. This way we can easily implement logic to decide when to flush, inside the emitter. I created a [pull request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. Feedback is highly welcome. :) > Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic > - > > Key: FLINK-33857 > URL: https://issues.apache.org/jira/browse/FLINK-33857 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: elasticsearch-3.0.1 >Reporter: Peter Schulz >Priority: Major > Labels: pull-request-available > > There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in > elasticsearch that may lead to underestimated bulk request sizes. We are hit > by this bug and therefore, I would like to propose a simple improvement to > work around it: > {{ElasticsearchWriter}} uses {{RequestIndexer}} as a facade for ES' > {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. > Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to > the server when any of the configurable limits is hit (there's another bug: > flushing will happen _after_ the limit has exceeded). > (+) *Proposal:* Expose {{BulkProcessor.flush()}} via > {{RequestIndexer.flush()}}. This way we can easily implement logic to decide > when to flush, inside the emitter. > I created a [pull > request|https://github.com/apache/flink-connector-elasticsearch/pull/85]. > Feedback is highly welcome. :) -- This message was sent by Atlassian Jira (v8.20.10#820010)