Re: Documents not deleted when using DeleteRequest within BulkProcessor
No as soon as you have only one index for this alias, indexing and deleting
should work.
I don’t see anything suspicious here.
Any chance you could share on github your full code?
When you say that nothing happens, do you mean that you never get the debug
LOG « Processing {} … » ?
Or do you mean that the document has not been removed ?
How do you test all that?
--
David Pilato - Developer | Evangelist
elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs
Le 30 avr. 2015 à 19:10, Diana Tuck dtu...@gmail.com a écrit :
Thank you for the reply, David.
We are using an alias to delete. Is that a problem? Indexing with the alias
through the bulk processor works fine.
There are no errors reported, it just seems to disappear into the oblivion.
Here's our code for the BulkProcessor:
public static BulkProcessor getBulkProcessor(Client client, int
esConcurrencyLevel, int esBulkSize, int esFlushInterval) {
return BulkProcessor.builder(client, new BulkProcessor.Listener() {
@Override
public void beforeBulk(long executionId, BulkRequest bulkRequest) {
LOG.debug(Processing {} requests in bulk process {},
bulkRequest.numberOfActions(), executionId);
}
@Override
public void afterBulk(long executionId, BulkRequest bulkRequest,
BulkResponse response) {
if (response.hasFailures()) {
for (BulkItemResponse item : response.getItems()) {
LOG.error(Processing to index \{}\ failed for entity
id {} with message {}, item.getIndex(),
item.getId(), item.getFailureMessage());
}
}
}
@Override
public void afterBulk(long executionId, BulkRequest bulkRequest,
Throwable throwable) {
LOG.error(Failed to process {} requests in bulk request {}: {},
bulkRequest.numberOfActions(),
executionId, throwable.getMessage());
throwable.printStackTrace();
}
})
.setBulkActions(esBulkSize)
.setFlushInterval(TimeValue.timeValueSeconds(esFlushInterval))
.setConcurrentRequests(esConcurrencyLevel)
.build();
}
Code for the delete request:
bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(),
index.getType(), entityId));
where index.getIndexingAlias() is an alias (same alias used for indexing
which is working), type is the document type company and entityId is the
document ID.
What data would be helpful? An example document, the index metadata,
something else?
On Wednesday, April 29, 2015 at 9:53:41 PM UTC-7, David Pilato wrote:
Do you try to delete a doc using an alias?
Any failure or error reported by the bulk processor?
Hard to tell more without seeing the code / data.
David
Le 30 avr. 2015 à 02:03, Diana Tuck dtu...@gmail.com javascript: a écrit :
Trying to index/delete documents within one BulkProcessor object in the Java
API. Indexing documents works great! Deleting, however, does not.
bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(),
index.getType(), entityId));
Nothing happens. Any ideas?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout
https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/01b6ab18-78a8-44d0-b574-c649501ec21a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/01b6ab18-78a8-44d0-b574-c649501ec21a%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout
https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/C7602498-6322-4F06-86ED-01B5147FA1FE
Re: Documents not deleted when using DeleteRequest within BulkProcessor
Thank you for the reply, David.
We are using an alias to delete. Is that a problem? Indexing with the
alias through the bulk processor works fine.
There are no errors reported, it just seems to disappear into the oblivion.
Here's our code for the BulkProcessor:
public static BulkProcessor getBulkProcessor(Client client, int
esConcurrencyLevel, int esBulkSize, int esFlushInterval) {
return BulkProcessor.builder(client, new BulkProcessor.Listener() {
@Override
public void beforeBulk(long executionId, BulkRequest bulkRequest) {
LOG.debug(Processing {} requests in bulk process {},
bulkRequest.numberOfActions(), executionId);
}
@Override
public void afterBulk(long executionId, BulkRequest bulkRequest,
BulkResponse response) {
if (response.hasFailures()) {
for (BulkItemResponse item : response.getItems()) {
LOG.error(Processing to index \{}\ failed for entity id
{} with message {}, item.getIndex(),
item.getId(), item.getFailureMessage());
}
}
}
@Override
public void afterBulk(long executionId, BulkRequest bulkRequest,
Throwable throwable) {
LOG.error(Failed to process {} requests in bulk request {}: {},
bulkRequest.numberOfActions(),
executionId, throwable.getMessage());
throwable.printStackTrace();
}
})
.setBulkActions(esBulkSize)
.setFlushInterval(TimeValue.timeValueSeconds(esFlushInterval))
.setConcurrentRequests(esConcurrencyLevel)
.build();
}
*Code for the delete request:*
bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(),
entityId));
where index.getIndexingAlias() is an alias (same alias used for indexing which
is working), type is the document type company and entityId is the document
ID.
What data would be helpful? An example document, the index metadata, something
else?
On Wednesday, April 29, 2015 at 9:53:41 PM UTC-7, David Pilato wrote:
Do you try to delete a doc using an alias?
Any failure or error reported by the bulk processor?
Hard to tell more without seeing the code / data.
David
Le 30 avr. 2015 à 02:03, Diana Tuck dtu...@gmail.com javascript: a
écrit :
Trying to index/delete documents within one BulkProcessor object in the
Java API. Indexing documents works great! Deleting, however, does not.
bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(),
index.getType(), entityId));
Nothing happens. Any ideas?
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/01b6ab18-78a8-44d0-b574-c649501ec21a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
bulkProcessor never calls afterBulk?
Here's my code example, I get something printed beforeBulk. But never see
anything printed for afterBulk. And yes, a document is getting written.
What am I missing??
BulkProcessor bulkProcessor = ((org.elasticsearch.action.bulk.BulkProcessor)
BulkProcessor).builder(
client,
new BulkProcessor.Listener() {
@Override
public void beforeBulk(long executionId, BulkRequest request) { System.out.
println(BeforeBulk); }
@Override
public void afterBulk(long executionId, BulkRequest request, Throwable
failure) {
System.out.println(Bulk execution failed [+ executionId + ].\n +
failure.toString());
}
@Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse
response) {
System.out.println(Bulk execution completed [+ executionId + ].\n +
Took (ms): + response.getTookInMillis() + \n +
Failures: + response.hasFailures() + \n +
Count: + response.getItems().length);
}
})
.setBulkActions(1)
.setBulkSize(new ByteSizeValue(1, ByteSizeUnit.GB))
.setFlushInterval(TimeValue.timeValueSeconds(5))
.setConcurrentRequests(1)
.build();
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5ed933f3-ffe4-4ef4-ba87-1f3ef522aaa4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Documents not deleted when using DeleteRequest within BulkProcessor
Trying to index/delete documents within one BulkProcessor object in the Java API. Indexing documents works great! Deleting, however, does not. bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId)); Nothing happens. Any ideas? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Documents not deleted when using DeleteRequest within BulkProcessor
Do you try to delete a doc using an alias? Any failure or error reported by the bulk processor? Hard to tell more without seeing the code / data. David Le 30 avr. 2015 à 02:03, Diana Tuck dtu...@gmail.com a écrit : Trying to index/delete documents within one BulkProcessor object in the Java API. Indexing documents works great! Deleting, however, does not. bulkProcessor.add(new DeleteRequest(index.getIndexingAlias(), index.getType(), entityId)); Nothing happens. Any ideas? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e2774458-8542-4634-bd8d-1ccfd9837409%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/512A5411-A00C-4F02-B11E-A0B24A27E7E3%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
Re: bulkProcessor never calls afterBulk?
Try to remove setBulkSize.
What is the other part of your code?
David
Le 30 avr. 2015 à 00:30, Scott Chapman scottedchap...@gmail.com a écrit :
Here's my code example, I get something printed beforeBulk. But never see
anything printed for afterBulk. And yes, a document is getting written.
What am I missing??
BulkProcessor bulkProcessor = ((org.elasticsearch.action.bulk.BulkProcessor)
BulkProcessor).builder(
client,
new BulkProcessor.Listener() {
@Override
public void beforeBulk(long executionId, BulkRequest request) {
System.out.println(BeforeBulk); }
@Override
public void afterBulk(long executionId, BulkRequest request, Throwable
failure) {
System.out.println(Bulk execution failed [+ executionId + ].\n +
failure.toString());
}
@Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse
response) {
System.out.println(Bulk execution completed [+ executionId + ].\n +
Took (ms): + response.getTookInMillis() + \n +
Failures: + response.hasFailures() + \n +
Count: + response.getItems().length);
}
})
.setBulkActions(1)
.setBulkSize(new ByteSizeValue(1, ByteSizeUnit.GB))
.setFlushInterval(TimeValue.timeValueSeconds(5))
.setConcurrentRequests(1)
.build();
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5ed933f3-ffe4-4ef4-ba87-1f3ef522aaa4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/C2258558-7AD5-40FA-A11F-8570A59C5FAF%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
BulkProcessor pest practices
Hey guys,
I'm using the BulkProcessor to index documents in elasticsearch. Its
definitely made my indexing throughput greater than it was before.
Anyway, I was wondering if there were some best practices around exception
handling with the bulk processor. For example it would be good to schedule
retries in certain scenarios.
At the moment all I'm doing is logging. I was wondering if someone could
point me to a resource with an example of handling a
NodeNotConnectedException and doing a retry. I don’t know how to access the
contents of the bulkProcessor from within the afterBulk method in the
Listener.
public void beforeBulk(long executionId, BulkRequest
bulkRequest) {
}
@Override
public void afterBulk(long executionId, BulkRequest
bulkRequest, BulkResponse bulkResponse) {
if (bulkResponse.hasFailures()) {
Log.error(We have failures);
for (BulkItemResponse bulkItemResponse :
bulkResponse.getItems()) {
if (bulkItemResponse.isFailed()) {
Log.error(bulkItemResponse.getId() + failed
with message: + bulkItemResponse.getFailureMessage());
}
}
}
}
@Override
public void afterBulk(long executionId, BulkRequest
bulkRequest, Throwable t) {
Log.error(An exception occurred while indexing, t);
// How do I add this back to the list of requests?
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7e7d476d-6a8d-4a82-bbd0-e331a08d1bb4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: BulkProcessor pest practices
If you make your bulk final, I think this could work:
private final BulkProcessor bulk;
CrmApp() {
Client esClient = new TransportClient(
ImmutableSettings.builder().put(cluster.name, devoxx)
).addTransportAddress(
new InetSocketTransportAddress(127.0.0.1, 9300)
);
bulk = BulkProcessor.builder(esClient, new BulkProcessor.Listener() {
@Override
public void beforeBulk(long executionId, BulkRequest request) {
logger.debug([{}] going to execute {} requests, executionId,
request.numberOfActions());
}
@Override
public void afterBulk(long executionId, BulkRequest request,
BulkResponse response) {
logger.debug([{}] ok, executionId);
}
@Override
public void afterBulk(long executionId, BulkRequest request, Throwable
failure) {
logger.warn(We have a problem, failure);
bulk.add(request);
}
})
.setBulkActions(pageSize)
.setFlushInterval(TimeValue.timeValueSeconds(5))
.build();
}
--
David Pilato - Developer | Evangelist
elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs
Le 24 avr. 2015 à 10:59, mzrth_7810 afrazmam...@gmail.com a écrit :
Hey guys,
I'm using the BulkProcessor to index documents in elasticsearch. Its
definitely made my indexing throughput greater than it was before.
Anyway, I was wondering if there were some best practices around exception
handling with the bulk processor. For example it would be good to schedule
retries in certain scenarios.
At the moment all I'm doing is logging. I was wondering if someone could
point me to a resource with an example of handling a
NodeNotConnectedException and doing a retry. I don’t know how to access the
contents of the bulkProcessor from within the afterBulk method in the
Listener.
public void beforeBulk(long executionId, BulkRequest bulkRequest)
{
}
@Override
public void afterBulk(long executionId, BulkRequest bulkRequest,
BulkResponse bulkResponse) {
if (bulkResponse.hasFailures()) {
Log.error(We have failures);
for (BulkItemResponse bulkItemResponse :
bulkResponse.getItems()) {
if (bulkItemResponse.isFailed()) {
Log.error(bulkItemResponse.getId() + failed
with message: + bulkItemResponse.getFailureMessage());
}
}
}
}
@Override
public void afterBulk(long executionId, BulkRequest bulkRequest,
Throwable t) {
Log.error(An exception occurred while indexing, t);
// How do I add this back to the list of requests?
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com
mailto:elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7e7d476d-6a8d-4a82-bbd0-e331a08d1bb4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7e7d476d-6a8d-4a82-bbd0-e331a08d1bb4%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout
https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4ABE0924-A01E-421F-A5A6-D1A34A0EC236%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.
Re: ActionRequest support for BulkProcessor in the Java API
The BulkProcessor is a helper class for managing write requests where large chunks of documents are combined into a single write request which saves a lot of network acknowledging ping/pong. If you send queries, there is a small write request, after which large response chunks are read, so there is no ping/pong, and no bulk mode is needed. Jörg On Thu, Mar 26, 2015 at 8:13 PM, Andre Encarnacao andre.v.encarna...@gmail.com wrote: Hi, I'm wondering why BulkProcessor (and BulkRequest) only supports IndexRequest, UpdateRequest, and DeleteRequests, and not SearchRequests? I'd like to be able to use Bulk Processor for processing multiple queries instead of having to use the MultiSearch API. Is this not possible? And if so, why not? Thanks! Andre -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7fe75d3-2684-4514-ab3d-c8b361fc6de3%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/e7fe75d3-2684-4514-ab3d-c8b361fc6de3%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE3woWQ8O3MKdE-Vc7quzmJ3So_KQf6Zvmhmqq9Bf4q7w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
ActionRequest support for BulkProcessor in the Java API
Hi, I'm wondering why BulkProcessor (and BulkRequest) only supports IndexRequest, UpdateRequest, and DeleteRequests, and not SearchRequests? I'd like to be able to use Bulk Processor for processing multiple queries instead of having to use the MultiSearch API. Is this not possible? And if so, why not? Thanks! Andre -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7fe75d3-2684-4514-ab3d-c8b361fc6de3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
BulkProcessor issue -- docs appear to be sent by client, but index doesn't change
Can anyone kindly point out why I could be seeing bulk indexing apparently
succeeding on a client but my document count being always zero? I am using
the BulkProcessor API as shown below. If I insert documents individually
via curl, the index is properly updated. I also see all my bulk insert
threads on the server busy and the bulk queue hovering around 150, yet the
filesystem is virtually idle, I see no docs being inserted and no growth on
the index data files. Tried flushing and refreshing manually to no avail.
The code below spins happily reporting no errors. Any help is greatly
appreciated.
class CustomBulkListener extends BulkProcessor.Listener {
override def beforeBulk (executionId:Long, request: BulkRequest)
= {
logger.info(Before bulk: {} actions, request.numberOfActions)
}
override def afterBulk (executionId:Long, request: BulkRequest,
response: BulkResponse) = {
logger.info(After bulk: {} actions, request.numberOfActions)
}
override def afterBulk (executionId:Long, request: BulkRequest,
failure: Throwable) = {
logger.warn(Error executing bulk, failure)
}
}
val bulkProcessor: BulkProcessor = BulkProcessor.
builder(myTransportClient, new CustomBulkListener).
setBulkActions(1000).
setBulkSize(new ByteSizeValue(1, ByteSizeUnit.GB)).
setFlushInterval(TimeValue.timeValueSeconds(5)).
setConcurrentRequests(30).build
var bulkRequest = targetEsClient.prepareBulk
val source = {some:test data}
val totalTuples = 10*1000*1000
logger.info(Starting insert loop for $totalTuples docs)
for (i - 1 to totalTuples) {
val indexRequest = new IndexRequest(Parms.targetIndex,
Parms.docType).source(source)
bulkProcessor.add(indexRequest)
if (i % 1000 == 0) logger.info(sAt $i docs)
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c71ef1f5-294f-4693-b288-0263325e1c37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: BulkProcessor close function
You must flush the BulkProcessor and wait until your code has received all
responses from outstanding requests. The close() method in BulkProcessor
performs a flush - and recently an explicit flush() is also implemented -
https://github.com/elasticsearch/elasticsearch/issues/5570
but BulkProcessor does not wait for all responses. If you close your client
prematurely, you will see no node available or other things.
Since a flush is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of beforeBulk/afterBulk
listener methods so you know when all responses have been received.
I wrote an improved BulkProcessor which provides a waitForResponses
method:
https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java
Jörg
On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabdall...@gmail.com wrote:
Ok, so how do I close the client? I have something like this?
main(){
MyIndexer indexer();
for (doc=0; doc n; doc ++)
indexer.pushDocumentToBulk(doc);
indexer.Shutdown();
}
class MyIndexer(){
//create client
// MyBulkProcesor myBulk;
//PushDocumentToBulk(Doc)
void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here, my
guess bulk is still processing ?
}
}
class MyBulkProcessor(){
BlulkProcessor bulk;
@BeforeBulk ...
@AfterBulk
PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}
On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:
No, BulkProcessor does not close the client.
Jörg
On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.com wrote:
bulkProcessor.close();
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFXCV23bzyz%2BQ3mgS_LM86K4fCJ7WGY1YKL%3DbWVJ_PEyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: BulkProcessor close function
There is an open issue to add a blocking close method to BulkProcessor
https://github.com/elasticsearch/elasticsearch/pull/4180
Matt
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received all
responses from outstanding requests. The close() method in BulkProcessor
performs a flush - and recently an explicit flush() is also implemented -
https://github.com/elasticsearch/elasticsearch/issues/5570
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see no node available or other things.
Since a flush is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of beforeBulk/afterBulk
listener methods so you know when all responses have been received.
I wrote an improved BulkProcessor which provides a waitForResponses
method:
https://github.com/jprante/elasticsearch-support/blob/master/src/main/java/org/xbib/elasticsearch/action/ingest/IngestProcessor.java
Jörg
On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabda...@gmail.comjavascript:
wrote:
Ok, so how do I close the client? I have something like this?
main(){
MyIndexer indexer();
for (doc=0; doc n; doc ++)
indexer.pushDocumentToBulk(doc);
indexer.Shutdown();
}
class MyIndexer(){
//create client
// MyBulkProcesor myBulk;
//PushDocumentToBulk(Doc)
void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here, my
guess bulk is still processing ?
}
}
class MyBulkProcessor(){
BlulkProcessor bulk;
@BeforeBulk ...
@AfterBulk
PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}
On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:
No, BulkProcessor does not close the client.
Jörg
On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:
bulkProcessor.close();
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: BulkProcessor close function
Oh, thanks for the pointer.
Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...
Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.
Jörg
On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston
matthew.pres...@thomsonreuters.com wrote:
There is an open issue to add a blocking close method to BulkProcessor
https://github.com/elasticsearch/elasticsearch/pull/4180
Matt
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -
https://github.com/elasticsearch/elasticsearch/issues/5570
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see no node available or other things.
Since a flush is also executed on a periodical interval, you can either
simply wait in your code for an estimated time so all the bulk requests
should be processed (which assumes the client is up for a long time after
BulkProcessor close), or you can write a BulkProcessor wrapper that is
maintaining a counter on the listener calls of beforeBulk/afterBulk
listener methods so you know when all responses have been received.
I wrote an improved BulkProcessor which provides a waitForResponses
method:
https://github.com/jprante/elasticsearch-support/blob/
master/src/main/java/org/xbib/elasticsearch/action/ingest/
IngestProcessor.java
Jörg
On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabda...@gmail.com wrote:
Ok, so how do I close the client? I have something like this?
main(){
MyIndexer indexer();
for (doc=0; doc n; doc ++)
indexer.pushDocumentToBulk(doc);
indexer.Shutdown();
}
class MyIndexer(){
//create client
// MyBulkProcesor myBulk;
//PushDocumentToBulk(Doc)
void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here,
my guess bulk is still processing ?
}
}
class MyBulkProcessor(){
BlulkProcessor bulk;
@BeforeBulk ...
@AfterBulk
PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}
On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:
No, BulkProcessor does not close the client.
Jörg
On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:
bulkProcessor.close();
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQQjwBRuSU7Ke3iUp6%3D%2BQBX-LznqKrYf3yF59Rm3Dvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: BulkProcessor close function
Hi,
yes, I've tested it with bulk failures - it seems to work well.
Internally, BulkProcessor releases semaphores for bulk failures and other
exceptions that can be thrown by the client, like NoNodeAvailableException,
so it shouldn't ever block forever.
Matt
On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote:
Oh, thanks for the pointer.
Have you tried the blocking close in case of bulk failures? It looks like
it could hang forever...
Personally I would prefer a separate waitForResponses() method on the
semaphore without synchronization, but that is a matter of taste.
Jörg
On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston
matthew...@thomsonreuters.com javascript: wrote:
There is an open issue to add a blocking close method to BulkProcessor
https://github.com/elasticsearch/elasticsearch/pull/4180
Matt
On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote:
You must flush the BulkProcessor and wait until your code has received
all responses from outstanding requests. The close() method in
BulkProcessor performs a flush - and recently an explicit flush() is also
implemented -
https://github.com/elasticsearch/elasticsearch/issues/5570
but BulkProcessor does not wait for all responses. If you close your
client prematurely, you will see no node available or other things.
Since a flush is also executed on a periodical interval, you can
either simply wait in your code for an estimated time so all the bulk
requests should be processed (which assumes the client is up for a long
time after BulkProcessor close), or you can write a BulkProcessor wrapper
that is maintaining a counter on the listener calls of
beforeBulk/afterBulk listener methods so you know when all responses
have been received.
I wrote an improved BulkProcessor which provides a waitForResponses
method:
https://github.com/jprante/elasticsearch-support/blob/
master/src/main/java/org/xbib/elasticsearch/action/ingest/
IngestProcessor.java
Jörg
On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 sabda...@gmail.comwrote:
Ok, so how do I close the client? I have something like this?
main(){
MyIndexer indexer();
for (doc=0; doc n; doc ++)
indexer.pushDocumentToBulk(doc);
indexer.Shutdown();
}
class MyIndexer(){
//create client
// MyBulkProcesor myBulk;
//PushDocumentToBulk(Doc)
void ShutDown(){
myBulk.Close();
//client.close() ; //This creates No node available exception here,
my guess bulk is still processing ?
}
}
class MyBulkProcessor(){
BlulkProcessor bulk;
@BeforeBulk ...
@AfterBulk
PushDocumentToBulk() { ..push doc ..}
void Close(){ bulk.close(); }
}
On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote:
No, BulkProcessor does not close the client.
Jörg
On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 sabda...@gmail.comwrote:
bulkProcessor.close();
Does this also close the transportClient? When I follow up the above
line with client.close(), I get no available exception.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo
glegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=emailutm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr
BulkProcessor
if I set Bulk size number of files at 5000, I feed it 5000, 5000, 5000, what happens if the #of files for instance in the last batch is 2000. How does it know that it needs to process the last 2000 ? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8c322041-783d-45fb-9595-38aa6a50d0eb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: BulkProcessor
There is a TimeOut so every x seconds whatever the number of docs in Bulk. Also when the bulkprocessor is closed, remaining elements are processed. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 7 mars 2014 à 21:51, ZenMaster80 sabdall...@gmail.com a écrit : if I set Bulk size number of files at 5000, I feed it 5000, 5000, 5000, what happens if the #of files for instance in the last batch is 2000. How does it know that it needs to process the last 2000 ? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8c322041-783d-45fb-9595-38aa6a50d0eb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D1B9782C-E0C2-426F-81CC-64B16B491E30%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
