Re: filter for embedded document works only for the first field

[Thierry Templier]

Hello,

Could you give us the mapping of your document? Do you use nested fields
(see this link
http://www.elastic.co/guide/en/elasticsearch/reference/1.5/mapping-nested-type.html
)?

If so, you should consider to use a nested query (see this link
http://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-nested-query.html).
..

Thierry

2015-04-15 16:46 GMT+02:00 hellboy :

> Document has filter sub-document with fields to be searched
>
> {
>   customField1: customFieldValue1,
>   ...
>   // Fields to be filtered
>   filter : {
>   firstFieldName: firstFieldValue,
>   secondFieldName: secondFieldValue,
>   }
> }
>
>
>
> I need to search for exact values within filter fields with  OR - operator.
>
> As I mentioned for exact-value searches, probably filter is the best
> option instead of a query, as a filter will be cached.
>
> But I am not sure
> This search request filters only firstFieldValue properly
>
> POST /index/type/_search
> {
> "filter": {
> "bool": {
> "must": [
> {
> "term": {
> "filter.firstFieldName": "firstFieldValue"
> }
> },
>{
> "term": {
> "filter.secondFieldName": "secondFieldValue"
> }
> },
> ]
> }
> }
> }
>
>
> And query matching works properly:
>
>
> POST /index/type/_search
> {
>"query": {
>"match": {
>"firstFieldName": "firstFieldValue"
> }
>}
> }
>
> why?
>
>
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c43fe35b-09f3-41b1-811a-5bf4bb1d43e7%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/c43fe35b-09f3-41b1-811a-5bf4bb1d43e7%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Best,
Thierry
--
*Thierry Templier, Lead Architect*
ttempl...@restlet.com


Restlet SAS, 6 rue Rose Dieng-Kuntz • 44300 Nantes • France

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL73BC3h9PKGK2NU2rYOep7R6AtOZAFrsAaM4c-fu-skXK5RZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Update denormalized fields in ElasticSearch

[Thierry Templier]

Hello all,

I wonder what is the best approach I should consider regarding the update
of denormalized data.

Within my application, I can easily identify the denormalized data. So when
one of them needs to be updated, I know to build all necessary updates to
send in a bulk request.

My question is how to efficiently handle errors to guarantee data
consistency.

Thanks very much for your help!
Thierry

-- 
Best,
Thierry
--
*Thierry Templier, Lead Architect*
ttempl...@restlet.com


Restlet SAS, 6 rue Rose Dieng-Kuntz • 44300 Nantes • France

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL73BC2qd%3DWfZToYuK6anVroWyM%2Bx1nnJPnnvWpoD_ztLBPPXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Create mapping for nested json

[Thierry Templier]

Hi,

In fact, you don't need to explicitely specify that a field is a list or
not in the mapping. This is automatically supported by ElasticSearch. See
this link in the documentation for more details:
http://www.elastic.co/guide/en/elasticsearch/reference/1.x/mapping-array-type.html
.

Hope it will help you,
Thierry


2015-04-02 6:46 GMT+02:00 :

> Hi,
>
> Noob at ElasticSearch, I am trying to push some nested json to
> Elasticsearch and have the nested objects parsed out as facets. If I use
> dynamic mapping then elasticsearch does not seem to parse out the internal
> objects. I guess I need to define a mapping for my index?
>
> Example:
>
> {
>   "Date": "2015-03-21T00:09:00",
>   "Activities": [
> {
>   "ActivityName": "SSH",
>   "Fields": [
> {
>   "User": [
> {
>   "joe": 2,
>   "jane": 3,
>   "jack": 5
> }
>   ]
> },
> {
>   "DstIP": [
> {
>   "HostA": 3,
>   "HostB": 5,
>   "HostC": 6
> }
>   ]
> }
>   ]
> }
>   ]
> }
>
> I tried to follow the mapping documentation but failed to come up with a 
> mapping that represents the JSON above. I guess I am not sure how to map 
> lists. If it helps, here's how I create the JSON in Scala using the Jackson 
> library:
>
> scala> nestedMap
> res3: scala.collection.immutable.Map[String,Object] = Map(Date -> 
> 2015-03-21T00:09:00, Activities -> List(Map(ActivityName -> SSH, Fields -> 
> List(Map(User -> List(Map(joe -> 2, jane -> 3, jack -> 5))), Map(DstIP -> 
> List(Map(HostA -> 3, HostB -> 5, HostC -> 6)))
>
> scala> println(Serialization.write(nestedMap))
> {"Date":"2015-03-21T00:09:00","Activities":[{"ActivityName":"SSH","Fields":[{"User":[{"joe":2,"jane":3,"jack":5}]},{"DstIP":[{"HostA":3,"HostB":5,"HostC":6}]}]}]}
>
> Is there a way to get Jackson to spit out the schema that can be directly fed 
> to elasticsearch as a mapping/template?
>
> Thanks.
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/735251b3-3b00-46d3-bf41-f02efaad3b12%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/735251b3-3b00-46d3-bf41-f02efaad3b12%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Best,
Thierry
--
*Thierry Templier, Lead Architect*
ttempl...@restlet.com


Restlet SAS, 6 rue Rose Dieng-Kuntz • 44300 Nantes • France

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL73BC21Rx2PoMYNjksbHNinia5%2BAgrHQKTrqiCod6rjvU9Bww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Questions about REST actions in ElasticSearch

[Thierry Templier]

Hello,

I try to embed an OData engine within an ElasticSearch REST plugin. To do
that, I need to know if it's possible and how to handle the following
things:

* Attach a REST action that handles several paths and HTTP methods using a
pattern like /odata.svc/*:

public class ODataRestAction extends BaseRestHandler {
@Inject
public ODataRestAction(Settings settings, Client client,
   RestController controller) {
super(settings, client);
controller.registerHandler(Method.GET, "/odata.svc/*", this);
}

(...)
}

For information, I described a possible solution here:
https://github.com/elasticsearch/elasticsearch/issues/5191.

* Obtain an output stream of the current response in order to directly
write the whole response content in it.

Thanks very much for your help!
Thierry

-- 
Best,
Thierry
--
*Thierry Templier, Lead Architect*
ttempl...@restlet.com


Restlet SAS, 6 rue Rose Dieng-Kuntz • 44300 Nantes • France

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL73BC3SVeprps7hZSGzrjuCdE%2BWis7RoDckZ4O_v9smDhPUfg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to get mapping meta data with java API

[Thierry Templier]

Hi Sven,

I think the following link can help you:
http://templth.wordpress.com/2014/12/22/managing-elasticsearch-metadata/.
See section "Custom metadata for types".

Hope it will help you...

Thierry

2015-01-06 16:13 GMT+01:00 Sven Jörns :

> Hi,
> How to get the meta data of mappings (
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-meta.html)
> using (transport) client Java API?
>
> Thx
> Sven
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/487d879e-952c-4242-94c2-2dc512209229%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/487d879e-952c-4242-94c2-2dc512209229%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Best,
Thierry
--
*Thierry Templier, Lead Architect*
ttempl...@restlet.com


Restlet SAS, 6 rue Rose Dieng-Kuntz • 44300 Nantes • France

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL73BC3gGwrWDOHNDuDZXxxCR0Sqpd9RVJA7q%3DFLxV%2Bjxcs-LQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Validate data before indexing them

[Thierry Templier]

Hi Jörg,

Following our conversation, I implemented a data validation filter. I 
describe my approach at this address: 
http://templth.wordpress.com/2014/03/07/implementing-data-validation-in-elasticsearch/.


I'd be pleased to have your feedback and comments on this! Thanks very much!

Thierry
I think, if this must be server side, this could be done in a REST 
filter plugin, not in the mappings. Matching values against patterns 
do not relate to field mappings.


Still, the best method to filter out unwanted values is at client 
side, either at JSON construction time in the official clients, or in 
the XContentBuilder calls in Java. Reason is to avoid extra load on 
the data nodes that is not related to index or search.


Jörg

--
You received this message because you are subscribed to the Google 
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF6RcbsLsMewFiFgFSdHT%3Dq%2BP%2BteaHBjRSXDk8Or-JXrQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5319F19D.8070107%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Validate data before indexing them

[Thierry Templier]

Thanks very much, Jörg, for your answer! I see the approach...

I understand well that having integrity checks in a schema-less engine 
like ElasticSearch isn't possible. However, would it be possible to have 
checks at field structure level before triggering indexing in the ES 
engine. Perhaps with specifying something like that in the mapping:


{
  "mappings": {
"mydoc": {
  "properties": {
(...)
"name": {
  "type":"string", "store":"yes", "index":"analyzed",
  "checks": "not_null,not_empty,regexp=^[a-zA-Z]$"
 },
(...)
  }
}
  }
}

Thanks very much for your help!
Thierry

You can validate the data at client side in your model before 
serializing it to JSON, or after a complete bulk index run.


There are reasons why Elasticsarch is schema-less. It is equivalent to 
allow any number of different fields (keys) and any content in fields 
(values) without any logical constraints.


In a distributed system, commits per field, or transactions per field, 
or integrity checking can get very expensive. Because the index is 
inverted, and nodes can come and go, there is a significant penalty if 
you want document transaction safety and document integrity checks.


I validate data in ES with the help o a large scan/scroll over the 
docs after bulk indexing, by searching for IDs if they exist or not. 
This is different from integrity constraint checking techniques like 
rule based methos known from RDBMs.


Jörg



On Sat, Feb 15, 2014 at 10:40 PM, Thierry Templier <mailto:temp...@gmail.com>> wrote:


Hello,

I wonder if there is a built-in way to validate data before
indexing them. I see two kinds of validation:

* Structural validation of fields based on a regular expression
for example. Perhaps something can be configured in the mapping...
* Integrity validation of document. For example preventing from
indexing a document with a field value that already exists.

In the case where there is no built-in support at the moment, is
there a way to extend ElasticSearch to add such processing before
indexing using the standard REST calls?

Thanks very much for your help!

-- 
You received this message because you are subscribed to the Google

Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscr...@googlegroups.com
<mailto:elasticsearch%2bunsubscr...@googlegroups.com>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/52FFDEC9.7020007%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google 
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGXXQxQ%2B5PRwrHw33uj3%2B8WwqLKiAZvnQrZ8bYUMfKYSw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5301B2C8.1060305%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Validate data before indexing them

[Thierry Templier]

Hello,

I wonder if there is a built-in way to validate data before indexing 
them. I see two kinds of validation:


* Structural validation of fields based on a regular expression for 
example. Perhaps something can be configured in the mapping...
* Integrity validation of document. For example preventing from indexing 
a document with a field value that already exists.


In the case where there is no built-in support at the moment, is there a 
way to extend ElasticSearch to add such processing before indexing using 
the standard REST calls?


Thanks very much for your help!

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/52FFDEC9.7020007%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Attach handlers for dynamic URIs in REST plugins

[Thierry Templier]

Hello,

I implement a REST plugin for ElasticSearch that provides a set of URI 
to manage and search indices. I attached the supported URIs using the 
RestController and this works fine.


I have however the issue of dynamic URIs depending on the structure of 
types but I can't find a way to configure them. For example, for a type 
that has a field named field1, I would like to support URI like 
"(...)/mytype/id/field1" and "(...)/mytype/id/field1/fieldA" with a 
pattern like "(...)/{type}/{id}/*". As a matter of fact, I can't know 
the depth of URI elements after "(...)/{type}/{id}/" by advance since it 
depends on the structure of type.


In fact I would need to attach handlers for URIs that starts with a 
specific pattern. Is it something supported?


Thanks for your help.
Thierry

--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/52ED11A1.7080208%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: How to get mappings using the Java API with ElasticSearch 0.90.10 and 1.0.0 RC1

[Thierry Templier]

Thanks very much, Luca and Roy for your answers!
Thierry
True, although this relates to the REST api but not to the Java API, 
for which the GetMappingsRequest stays the same. As I said, we 
introduced the get field mapping api and related GetFieldMappingsRequest.



On Tue, Jan 28, 2014 at 3:13 PM, Roy Russo > wrote:


Just a side note that the API response format changed in 1.0.0RC1.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/_indices_apis.html



--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/52EA255C.6010500%40gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

How to get mappings using the Java API with ElasticSearch 0.90.10 and 1.0.0 RC1

[Thierry Templier]

Hello,

I wonder what is the best way to get the mappings for an index using the 
ElasticSearch Java API with
version 0.90.10. I saw that there is only a GetFieldMappingsRequest class 
for this version whereas
a class GetMappingsRequest is now present in 1.0RC1.

Thanks very much for your answer and your help!
Thierry 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3d0fb56f-52c6-4b02-96b5-d0728612101d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.