Re: Hive write data to elastic search

2014-11-28 Thread Atul Paldhikar
yes that's correct, the issue is resolved. Thanks - Atul On Friday, November 28, 2014 4:09:41 AM UTC-8, Costin Leau wrote: > > Based on the other email that you sent, I assume this issue was fixed as > well, right? > > On 11/28/14 8:44 AM, Atul Paldhikar wrote: > > Hi Costin, > > > > actuall

Re: is it possible to use transform scripts in mappings to alter document _id?

2014-11-28 Thread Nikolas Everett
On Nov 28, 2014 10:20 PM, "Reason" wrote: > > The Elasticsearch documentation is always frustratingly silent on the things I seem to need to accomplish to make life easier. Sorry you feel that way. If you are willing to fix the documentation those pull requests are typically merged quickly and do

is it possible to use transform scripts in mappings to alter document _id?

2014-11-28 Thread Reason
The Elasticsearch documentation is always frustratingly silent on the things I seem to need to accomplish to make life easier. Is it possible to use a transform script in a mapping to alter the document _id? This would be a convenient way to de-dup incoming data I have too little control over i

Re: Can't integrate Elasticsearch with Hive

2014-11-28 Thread Atul Paldhikar
Finally I was able to load the data from Hive to Elasticsearch !!! Yes you are right, in the beginning I started with es-hadoop 1..3.x then replaced by 2.0.2. However somewhere in the Hive classpath it remained and caused all this trouble. Now I do have 2 "ex_address" tables. One in "Default" a

Re: Is Elasticsearch also supported on AIX and HP Itanium 11.31

2014-11-28 Thread joergpra...@gmail.com
All in all, it should work. Sigar is totally optional. You may miss some OS metrics in the API but that is all. More critical is IBM support for Java 7 and soon Java 8 on AIX. For HP UX, HP offers Java 7/8 support. Use at least Elasticsearch 1.3 (better 1.4) to avoid bugs due to sun.misc.Unsafe

Re: ES java api: how to handle connectivity problems?

2014-11-28 Thread joergpra...@gmail.com
This is expected behavior. When there are not enough master nodes, and the cluster nodes wait for a new master, the cluster is blocked and all clients hang or get SERVICE_UNAVAILABLE ClusterBlockException after a timeout. >From client side, you can play with fault detection response timeout in th

Re: After using term aggregation search api will not get the desired result. Tried everything ??Help me out

2014-11-28 Thread Adrien Grand
What is the issue with this aggregation? On Fri, Nov 28, 2014 at 7:41 PM, bharat bisht wrote: > Hey guys, > I am stuck in the problem like currently i'm using the term aggregation > nested condition to search the top IP address count used by each users. > My condition is like that in my data i h

After using term aggregation search api will not get the desired result. Tried everything ??Help me out

2014-11-28 Thread bharat bisht
Hey guys, I am stuck in the problem like currently i'm using the term aggregation nested condition to search the top IP address count used by each users. My condition is like that in my data i have users and each user performing n number of activities using more then one IP's. So, my desired res

Trouble formulating a query with Java API

2014-11-28 Thread Maarten Roosendaal
Hi, I have the following (json) query i use: { "fields": ["ID","ID2"], "query": { "filtered": { "query": { "multi_match": { "query": "", "fields": ["FIELD1.not_analyzed","FIELD2.partial"] }

Re: ES java api: how to handle connectivity problems?

2014-11-28 Thread msbreuer
There is nothing special in code. Initially the node-client (not the transport.client) is created. Then in indexing-thread calls something like: irb = client.prepareBulk(...) or irb = client.prepareIndex(...) And finally irb.execute().actionGet() is invoked. With running cluster this code r

Re: Aggregations don't seem to work with template queries.

2014-11-28 Thread Luke Wilson-Mawer
Hi Colin, Thanks Colin. This does seem like it's the problem. I am building the query with the java query DSL but it seems to just stick the whole thing into the query section. TemplateQueryBuilder qb = new TemplateQueryBuilder(myQueryString, myParams) SearchResponse resp = client.prepareSearc

Re: ES java api: how to handle connectivity problems?

2014-11-28 Thread Georgi Ivanov
That's strange. Can it be a problem in the code ? Something like looping forever ? You can set the timeout to bulk request , but there is default timeout of 1 minute. May be some code will help. On Friday, November 28, 2014 3:09:37 PM UTC+1, msbr...@gmail.com wrote: > > While testing how to

doc_values for non analyzed fields

2014-11-28 Thread Georgi Ivanov
Hi, Will it make any difference in terms of field data memory, if I set the field data format to doc_values for all fields that have mapping "index" : "no" ? Are these (non-analyzed) fields ever loaded in memory on first place ? Example field mapping : "rot": { "index": "no", "type": "integer

ES java api: how to handle connectivity problems?

2014-11-28 Thread msbreuer
While testing how to handle es-cluster connectivity issues I ran into a serious problem. The java api node client is connected and then the ES server is killed. The application hangs in some bulkRequest, but this call never returns. It also does not return, even if the cluster was started. On c

Re: Is Elasticsearch also supported on AIX and HP Itanium 11.31

2014-11-28 Thread Gaurav gupta
Basically, I am integrating the Elasticsearch as a search engine in our product which is supported on AIX, HP itanium as well. So, I need to run the elasticsearch as a standalone java application on AIX (64 bit - 6.1 and 7.1) and other platforms. Could you suggest me how I can achieve it (any work

Re: Hive write data to elastic search

2014-11-28 Thread Costin Leau
Based on the other email that you sent, I assume this issue was fixed as well, right? On 11/28/14 8:44 AM, Atul Paldhikar wrote: Hi Costin, actually I think I figured out the issue, my script had a typo (resources instead of resource) create external table ex_address (name String, st_no INT,

Re: Can't integrate Elasticsearch with Hive

2014-11-28 Thread Costin Leau
ESStorageHandler was the name used in es-hadoop 1.3 Beta1 - in 2.0, the name was changed to EsStorageHandler. In case you are using the wrong class, you would get a ClassNotFound Exception - assuming you don't have 1.3 in your classpath. Regarding the error, yes, for some reason Hive since 0.1

Re: Need Full text elasticsearch

2014-11-28 Thread Vijayakumari B N
Hi David, I am facing some problem while invoking restfull webservice. I am getting "NetworkError: 404 /_search - https://localhost:8443/_search?from=0&size=10&q=joe&f_date=&f_country="; error. If my understanding is not wrong it is because of HTTPS and restfull expects HTTP. I am facing this issu

Re: Aggregations don't seem to work with template queries.

2014-11-28 Thread Colin Goodheart-Smithe
Sorry, this is the link I meant to attach: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-template.html Colin On Friday, 28 November 2014 11:48:10 UTC, Colin Goodheart-Smithe wrote: > > Hi Luke, > > The page you linked to is for templating just the query section of

Re: Aggregations don't seem to work with template queries.

2014-11-28 Thread Colin Goodheart-Smithe
Hi Luke, The page you linked to is for templating just the query section of a search request. To template the entire search request there is a dedicated endpoint described in the following link: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-template.html Let me k

Aggregations don't seem to work with template queries.

2014-11-28 Thread Luke Wilson-Mawer
If I send a perfectly well formed aggregation query as a template query, I don't get the aggregation results. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-template-query.html Is this by design, or is there a workaround? I am using the Java query DSL and versio

Document/field sizing

2014-11-28 Thread Derry O' Sullivan
Hi all, Is there a way to get document/field size statistics for a particular index? I currently use the indices stats API to get some information but it only gives me #documents and store size which is an average. e.g.: curl -XGET 'http://localhost:9291/_cat/indices/log*?bytes=b' -s | sort |

Re: poor performance of "full text" searches

2014-11-28 Thread horst knete
Thx for response, Actually the need to search with wildcards is given because our mapping. In our events are indexed a whole lot of urls which are indexed to many terms by the default analyzer, this will lead to an pretty akward output if you are doing the "top 10" search in Kibana in the table

Re: poor performance of "full text" searches

2014-11-28 Thread David Pilato
Some comments: Searching for `program:apache` actually search in field `program`. Searching with wildcards is something you really should not do! Actually when you do full text search on Google, I guess you don’t use wildcards, right? It’s not really user friendly. So wildcards are extremely sl

poor performance of "full text" searches

2014-11-28 Thread horst knete
Hey guys, we are experience poor performance if we do some "full text" ( searches without specifying an field name). If we search in an 200 GB index which lays on SSDs for something like ' program:apache ' the search takes about 10-15 seconds, if we search for ' *apache* ' the whole search goe

heap size/filter cache

2014-11-28 Thread Audrius Bugas
Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters cache. When clear filter cache from api it clears ~4GB memory from heap (way more then filters cache). Is it normal, or I have some problems with cached filters? We are using warmers, but this should not be a reason... --

white spaces and wild cards

2014-11-28 Thread Pablo Blasco
Hi we have this problems with querys (we use the last ElasticSearch 1.4.1) The query target is this string "MANTENIMIENTOS FERNANDEZ" ] We have this query: POST /GDATADEC-1/invoice/_search { "query": { "query_string" : { "query" : "*mantenimie*fer*"