Re: Setting id of document with elasticsearch-hadoop that is not in source document

2014-07-13 Thread James Campbell
Thanks for suggesting this option, I would also definitely like to have an exclude option for fields that I currently have to include only to set the _id, _type, and index, resulting in unnecessary fields in _source. On Fri, Jul 11, 2014 at 4:39 PM, Costin Leau wrote: > Hi, > > I've opened up i

Re: Setting id of document with elasticsearch-hadoop that is not in source document

2014-07-11 Thread Costin Leau
Hi, I've opened up issue #230 to address your use case. Rather than offering a dedicated field for the ID, I opted to introduce an "include", "exclude" option to select (or remove) certain fields from a document before being saved to es. This will basically allow documents to be filtered and thus

Re: Setting id of document with elasticsearch-hadoop that is not in source document

2014-07-11 Thread Brian Thomas
I was just curious if there was a way of doing this without doing this, I can add the field if necessary. For alternatives, what if in addition to es.mapping.id, there is another property available also, like es.mapping.id.exlude that will not include the id field in the source document. In el

Re: Setting id of document with elasticsearch-hadoop that is not in source document

2014-07-11 Thread Brian Thomas
I was just curious if there was a way of doing this without doing this, I can add the field if necessary. For alternatives, what if in addition to es.mapping.id, there is another property available also, like es.mapping.id.include.in.src where you could specify whether the src field actually ge

Re: Setting id of document with elasticsearch-hadoop that is not in source document

2014-07-10 Thread Costin Leau
You need to specify the id of the document you want to update somehow. Since in es-hadoop things are batch focused, each doc needs its own id specified somehow hence the use of 'es.mapping.id' to indicate its value. Is there a reason why this approach does not work for you - any alternatives tha