Re: Problem with the river-jdbc sqlite

You must use a JDBC4 driver (jdbc sqlite 3.8.2-SNAPSHOT at least) The "driver" parameter is ignored by JDBC river. Jörg On Thu, Apr 24, 2014 at 1:25 AM, Franck B wrote: > Hello, > > I use jdbc-river to feed my ES 0.92 with sqlite datasource . Everything > is OK. > > I wanted to upgrade ES 1.1

Re: Best way to store variable number of key:value fields

Hi Dominic, It's not very clear to me what your question is. You want to reject Apple documents that have an OS field? I guess you can either have Apples and Mobilephones in their own type. This will make it clear which fields belong to which type of document. And because ES has dynamic mappings

Re: Spaces in terms in request body make the query return no results

> Your message field is analyzed by default using the Standard Analyzer: > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html > > This means your "test message" will become ["test", "message"]. > > On the other hand, the prefix query isn't anal

Re: Spaces in terms in request body make the query return no results

Hi Alexey, Your message field is analyzed by default using the Standard Analyzer: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html This means your "test message" will become ["test", "message"]. On the other hand, the prefix query isn't analyz

Best way to store variable number of key:value fields

Hi, I want to index some Data, where each Document should have a variable number of fields. An example because I can't express myself very good this morning: Apple Color: Green Flavor: Sour Mobilephone Color: black Os: Android My problem is, that "Apple" should not have the field Os and

Spaces in terms in request body make the query return no results

Given a simple index: curl -XPUT http://localhost:9200/twitter/tweet/1 -d '{"message": "test message"}' A query for "test" returns the tweet: curl -XPOST 'http://localhost:9200/twitter/tweet/_search?pretty' -d '{"query": {"prefix": {"message": "test"}}}' curl -XGET 'http://localhos

Re: Aggregation in Kibana or Elasticsearch

The below query worked for me. LogDate:[2013-08-01 TO 2013-08-10] AND (LogDetail:"Antenna plate 1 temperature:*" AND LogDetail:[70.00 TO 80.00]) On Wed, Apr 23, 2014 at 6:19 PM, Praveen Shilavantar wrote: > Hi, > > I am new to elasticsearch and kibana. I have loaded some log data into > elasti

Re: Connecting by means of Java Client and Proxy

Te si do try and Connect by means of port 80 but thuis is forwarded by my proxy to 9300. You mean it Ignores the port settings in THE transport client? Groet, Pieter > Op 24 apr. 2014 om 04:43 heeft David Pilato het volgende > geschreven: > > Just a note: Java client uses Transport Layer (de

Re: unable to index using elastic-hadoop plugin

Hi, If I understand correctly, you can read data from ES (through es-hadoop) but you cannot write to it - am I correct? Can you confirm that you are using the latest es-hadoop, namely 1.3.0.M3? How big is the JSON file you are trying to index? Do you see any activity in the console? There are

unable to index using elastic-hadoop plugin

I am using elasticsearch-hadoop plugin ( https://github.com/elasticsearch/elasticsearch-hadoop) and trying to index some documents. Iam using elasticsearch version 0.90.2 and Hadoop Hortonworks 2.2.0. The search functionality works fine, but while indexing application hangs https://github.com/

Re: High cpu usage after 0.9.13 to 1.0.3

I thought this might help: -- You received

High cpu usage after 0.9.13 to 1.0.3

Hi everyone, Yesterday I change my source.list repository from 0.9 to 1.0 and upgrade my ES to newer version and chose to not change the config file which it was the default. Ever since I started ES the CPU usage is over the top. I used to see high cpu usage in merging, my mongo-river, refresh

Re: Connecting by means of Java Client and Proxy

Just a note: Java client uses Transport Layer (default to 9300 port) and not HTTP layer (default to 9200 port). I think you are trying to set transport to HTTP layer (80 port). -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 24 avr. 2014 à 00:59, Pieter van der Meer a éc

Re: TooLongFrameException: HTTP header is larger than 8192 bytes.

Hi Honza, In fact, this is my feature work, it supports many attractive features, such as persistent connection. But in this development status, it just need some simple REST APIs in matter of time I chose to use requests. maybe a mistake :P. Although I might solve this problem by using the off

Re: TooLongFrameException: HTTP header is larger than 8192 bytes.

Hi Ivan, can I ask you why do you use the requests library instead of the official client? That should work without such issues. Thanks, Honza On Thu, Apr 24, 2014 at 3:54 AM, Ivan Ji wrote: > Hi Jörg, > > thanks for your replies. About what you said: After the last header line, > two line feed

Re: TooLongFrameException: HTTP header is larger than 8192 bytes.

Hi Jörg, thanks for your replies. About what you said: After the last header line, two line feeds are required, do you mean the headers need at least 3 lines? I do not get it. could you explain more? In fact, I do trace the python requests header generation. And yes, it automatically add some h

Elasticsearch field cache always filled up and prevents other query to run.

Hi all, I set up an index containing 500.000 with million nested documents and I run big queries with a lot of aggregation. As expected this is using a LOT of memory and I easily have an OutOfMemory exception if a user perform a query for too many documents So I configured elasticsearch with th

Increase results from completion suggester

I know this has been asked before... However, is there anyway currently to increase the return results size within the completion suggester? mod to source? Upcoming Beta? Anything? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscri

Aggregation in Kibana or Elasticsearch

Hi, I am new to elasticsearch and kibana. I have loaded some log data into elasticsearch and I have a field called "LogDetail" and the content looks like below *Antenna plate 1 temperature: 40.00 degC* I would like to get the log events/documents for temperature > 70.00 degC. This is how we a

Problem with the river-jdbc sqlite

Hello, I use jdbc-river to feed my ES 0.92 with sqlite datasource . Everything is OK. I wanted to upgrade ES 1.1.1 and there is drama , I have this error: [ERROR][org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource] while opening read connection: jdbc:sqlite:/data/db/maDb.db No

Percolate Existing Documents Seems Broke

Maybe I'm off or I missed something where this was covered, but... curl -XPUT 'localhost:9200/test' curl -XPUT 'localhost:9200/test/.percolator/1' -d ' { "query": { "query_string": { "query": "headline:\"apples\"" } }, "attributes": { "query_name": "

Connecting by means of Java Client and Proxy

All, I am attempting to make a connection to my 1 node cluster by means of the Java Client. but when i do i get the following exception: INFO: [Marvin Flumm] failed to get local cluster state for [#transport#-1][flatback][inet[hostnamel/ipnumber:80]], disconnecting... org.elasticsearch.transport

Re: Pop3 River? Mail River? MailList River

IMAP/POP3 river now available https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/imap/elasticsearch/IJrjLbuE7fQ/D-1MAeS7DbsJ https://github.com/salyh/elasticsearch-river-imap Am Donnerstag, 28. Juni 2012 14:52:37 UTC+2 schrieb tullio0106: > > Is the river ready ? > Would You sha

Re: Checking results of bulk inserts, pre- and post-1.0

Hi Axel, unfortunately there is no code in python to shield you from the incompatibilities. There are, however, two releases of elasticsearch-py - 04.X and 1.0X. Use 0.4.X with elasticsearch 0.90.* and 1.0.X with elasticsearch 1.*. That should get you what you need. Honza On Wed, Apr 23, 2014 a

Re: Default Configurations in elasticSearch

The only way to find the default is to search inside the code. If you do not have the source locally, you can always search the project on Github (which uses Elasticsearch BTW). The properties are organized by module, so you normally can just go to the corresponding module/service class without nee

[hadoop] Extra Documents in Elastic Search

Hi, I'm trying to store a lot of documents into ES using pig. The pig job ends successfully but I end up with more documents in Elasticsearch than the number of rows in my input. My pig script is 3 lines: REGISTER 'local/path/to/m2.jar' data = load 'path/to/hdfs/file.tsv' as (field1: chararray,

Extra documents in Elastic Search

Hi, I'm trying to store a lot of documents into ES using pig. The pig job ends successfully but I end up with more documents in Elasticsearch than the number of rows in my input. My pig script is 3 lines: REGISTER 'local/path/to/m2.jar' data = load 'path/to/hdfs/file.tsv' as (field1: chararray,

Re: Python ES and Carrot2

Thanks a ton. On Tue, Apr 22, 2014 at 5:55 PM, Honza Král wrote: > Hi Pratik, > > if you are using elasticsearch-py you can call any API by using: > es.transport.perform_request (0) directly, that will enable you to use > the carrot2 plugin: > > es.transport.perform_request('POST', > '/test/test

Re: Solr SearchComponent-like functionality?

Thanks for your answer ! I have some followup questions 1. I wrote a custom script to return a constant score of 2.5. It works for the following query { "query" : { "function_score": { "query":{ "match_all":{} }, "script_score": { "lang":"native", "script": "constant-sc

Re: Problem with shingles as an autocomplete solution

Any response on if this is a good way to do autocomplete? > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To vi

Re: Is there a ping function available?

Just issue a HTTP HEAD request against port 9200 of the host you want to connect to and check for status 200 curl -I -XHEAD localhost:9200 HTTP/1.1 200 OK Content-Type: text/plain; charset=UTF-8 Content-Length: 0 Or, if you want version information, use a HTTP GET request curl -XGET localhost:92

Re: Is there a ping function available?

Stefan, If the TransportClient or NodeClient is set up as needed, then you can easily wait for the cluster to be ready to use and let Elasticsearch do the per-server discovery itself. At a high level (assuming this is part of some class that includes private Client client;). Error reporting is

match_phrase_prefix ... fuzzy not working on multi-word terms

Fuzziness is working for misspelled single words, but not if they're part of a phrase. Why would this be? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elas

Using parent aggregation key in subaggregation

I want to know if I can access parent key in subaggregation, as { > "size": 0, > "aggs":{ > "*agg1*":{ > "terms":{ > "field":"memberId" > }, > "aggs":{ > "11":{ > "terms":{ >

Re: Facetting by first letter

You can easily accomplish this with value scripts, but I would still choose the Solr approach of indexing a new field for performance reasons. If any content is static in nature, I would always opt to handle the processing at index time. Not only will you save CPU cycles by not having to run a scri

Re: Checking results of bulk inserts, pre- and post-1.0

Hi Honza, sadly, this doesn't seem to work. Comparing the results from 0.90 vs 1.0.1 from the _bulk api: {"took":345,"items":[{"index":{"_index":"test","_type":"type1","_id":"1","_version":1,"ok":true}},{"delete":{"_index":"test","_type":"type1","_id":"2","_version":1,"ok":true}},{"create":{"_in

Re: JDBC river query results collapsing to JSON issue

I hesitate to add "oversophisticated" code to the JDBC river that collapses without reason. Somehow the definition should set "merge points" to control the zone of JSON object/array growth. Maybe an extension of the bracket notation is all that is needed. At least, I will add extensive logging to

Re: BulkProcessor close function

Matt, I see the open issue, but where is the code for this? On Wednesday, April 23, 2014 10:21:13 AM UTC-4, Matt Preston wrote: > > Hi, > > yes, I've tested it with bulk failures - it seems to work well. > Internally, BulkProcessor releases semaphores for bulk failures and other > exceptions t

Re: Performance tuning ES for in-memory

High sustainable bulk indexing is very stable here. I have 3x HP DL165 G7 32 core machines and can index for hours at same speed with this settings https://gist.github.com/jprante/10666960 Jörg On Wed, Apr 23, 2014 at 5:52 PM, John Smith wrote: > Ok so I decided to skip in-memory for now jus

Re: Facetting by first letter

You can for sure define 26 facets, each using a term script: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html#_term_scripts But I'm pretty sure this can be done using script fields, haven't tried that one before tho http://www.elasticsearch.org/g

Re: Performance tuning ES for in-memory

Ok so I decided to skip in-memory for now just to test bassic functionality. I'm running elastic search with defaults as ./elasticsearch -Xms32g -Xmx32g I also got bigdesk installed. Either I'm not getting something... But why as I write more documents to the index the Indexing requests per

Facetting by first letter

Hi, i would like to facett by the first letter of search results. By Solar i need to make a field with the first letter inside. Is there by elasticseach maybe a cool function for this? I would to make a alphabetical list like : A B C D but only for letters where a results behind. Regards and th

Is there a ping function available?

Hi, does the PHP API Version a PING Function has included? I know the PING Function from Solar, its to check if the server is available and if not you can make a Fallback. Or how is the best way to check if the elasticsearch server is reachable? Regards and thanks Stefan. -- You received this

Re: Reducing Disk Space Requirements/ Deduplication? Zipping?

Since netflow data is not text, you're using elasticsearch like a distributed "SQL" database. You should turn of analysis for all of the netflow fields within your ES template, and drop any fields you don't REALLY really need. Also, consider using pmacct (pmacct.net) as a pre-aggregator and

Re: elasticsearch-hadoop 1.3 M3 proxy support not working

To eliminate any issues/confusion please try the dev snapshots as indicated in the previous email/docs. The latest version in Maven OSS is elasticsearch-hadoop-1.3.0.BUILD-20140423.145248-400.jar This passes the entire test suite just fine and should not give any NPE. If that is not the case,

Re: Performance tuning ES for in-memory

On a 32 core machine? Plus I think 1.7_51 uses G1 I have tested another "indexing" api up to 190GB or so with 30,000,000 objects and my latency was 3ms overall including network and app logic. And I haven't tested that many records with elastic search yet ;) On Wednesday, 23 April 2014 10:20:2

Re: elasticsearch-hadoop 1.3 M3 proxy support not working

Hi Costin, Thanks for the update. I have built a jar file from master. I am getting same NPE in stacktrace even when I try without proxy and authentication. With the new jar i cant push data into ElasticSearch anymore. Please do needful. Thanks. On Wednesday, April 23, 2014 5:34:20 AM UTC-

Re: BulkProcessor close function

Hi, yes, I've tested it with bulk failures - it seems to work well. Internally, BulkProcessor releases semaphores for bulk failures and other exceptions that can be thrown by the client, like NoNodeAvailableException, so it shouldn't ever block forever. Matt On Wednesday, 23 April 2014 13:47

Re: Performance tuning ES for in-memory

The ES "memory" or "ram" store (Lucene RAMDirectory) puts enormous pressure on JVM garbage collection. You can not expect that standard JVM with CMS GC can give the best performance. More info in this great article by Mike McCandless http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-wit

Re: Poor Update Performance Despite Refresh Interval Compromise

On Wed, Apr 23, 2014 at 10:01 AM, Nariman Haghighi wrote: Running a 2-node cluster, we're experiencing less than ideal update times > even after adjusting the refresh interval. > > Settings are: > "number_of_replicas":"1","number_of_shards":"5","refresh_interval":"5s" > > The two VMs are 4 cores,

Poor Update Performance Despite Refresh Interval Compromise

Running a 2-node cluster, we're experiencing less than ideal update times even after adjusting the refresh interval. Settings are: "number_of_replicas":"1","number_of_shards":"5","refresh_interval":"5s" The two VMs are 4 cores, 7 GB of ram, and the following are response times reported (on avg

Re: Filter by minimum value per parent (or term)

With aggregations in the new elasticsearch version I came up with a solution to get the cheapest offer per hotel: https://gist.github.com/manuelbrunner/11215712 However, it returns just the bucket key and the value: "buckets" : [ { "key" : "h1", "doc_count" : 2,

Re: JDBC river query results collapsing to JSON issue

That sounds like good news, looking forward to that The only thing that really bothers me about the issue originally listed is that the _river results will collapse to JSON correctly with multiple 1:N relationships if i _river a smaller dataset for instance , if i include in my _river sql criter

Re: Performance tuning ES for in-memory

1.7_51 but i dont see how their could be a limitation. I used java up to 200GB easily and with no issues either... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email

Re: Unable to Send Stats to Monitoring Cluster

Can you access the monitoring cluster from the command line using telnet when logged in on the production machine? All marvel does is an HTTP POST call.. Cheers, Boaz On Tuesday, April 22, 2014 5:13:29 PM UTC+2, Mario Rodriguez wrote: > > I am having an issue with sending Marvel stats to our mo

Re: Marvel page is empty

Somehow Elasticsearch can't find/access the marvel jars & _site. This is indicated by this log: [2014-04-17 18:57:28,899][INFO ][plugins ] [Richard Parker] loaded [], sites [] Is ES running under the same oracle:dba user? Can you double check that the marvel folder under plug

Re: TooLongFrameException: HTTP header is larger than 8192 bytes.

Check your python code for HTTP header generation. After the last header line, two line feeds are required. Default limit on HTTP request is 100MB. Jörg On Wed, Apr 23, 2014 at 10:26 AM, Ivan Ji wrote: > Hi all, > > I am encountering an exception and this situation cannot be always > reproduc

Re: BulkProcessor close function

Oh, thanks for the pointer. Have you tried the blocking close in case of bulk failures? It looks like it could hang forever... Personally I would prefer a separate waitForResponses() method on the semaphore without synchronization, but that is a matter of taste. Jörg On Wed, Apr 23, 2014 at 12

Re: BulkFile Indexing with base64 command utility on linux server

here i need to execute the curl with different file url in loop for multiple file On Wed, Apr 23, 2014 at 6:06 PM, joergpra...@gmail.com < joergpra...@gmail.com> wrote: > This is not bulk indexing, but anyway... > > If you want to post binary data to Elasticsearch from command line, you > shoul

Re: BulkFile Indexing with base64 command utility on linux server

This is not bulk indexing, but anyway... If you want to post binary data to Elasticsearch from command line, you should use echo '{"content":"'$(base64 /path/to/file)'"}' | curl -XPOST '0:9200/test/docs/1' --data-binary @- Jörg On Wed, Apr 23, 2014 at 11:47 AM, wrote: > for the above iam usi

failed to find mapping for

Hi! Guys! Im novice in elesticsearch.. Now I have this request: {"size":100,"from":0, "query":{ "filtered":{ "filter":{ "and":[ {"exists" : {"field" : "psp_name" }} ] } } }, "facets":{ "txtamountusdtransaction_sum":{ "facet_filter

Changing elasticsearch index's shard-count on the next index-rotation

i have an ELK(Elasticsearch-Kibana) stack wherein the elasticsearch node has the default shard value of 5. Logs are pushed to it in logstash format( logstash-.MM.DD), which- correct me if i am wrong- are indexed date-wise. Since i cannot change the shard count of an existing index without

Re: Capacity Planning with ElasticSearch

It depends - on your data set, your queries, your cluster specs.Having tens to hundreds of thousands (or millions) of indexes will have a performance impact that will only increase with numbers, so the lower you can keep it though planning the better. But to counter that, the bigger your indexes, t

Re: BulkProcessor close function

There is an open issue to add a blocking close method to BulkProcessor https://github.com/elasticsearch/elasticsearch/pull/4180 Matt On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: > > You must flush the BulkProcessor and wait until your code has received all > responses from out

Re: BulkProcessor close function

You must flush the BulkProcessor and wait until your code has received all responses from outstanding requests. The close() method in BulkProcessor performs a flush - and recently an explicit flush() is also implemented - https://github.com/elasticsearch/elasticsearch/issues/5570 but BulkProcesso

Re: BulkFile Indexing with base64 command utility on linux server

for the above iam using following command to get the content from file and convert into base64encode. file_path='/path/to/file' file=$(base64 $file_path | perl -pe 's/\n/\\n/g') curl -XPUT "http://eshost.com:9200/index/type/"; -d '{ "file" : "content" : "'$file'" }' -- You received this m

Re: Capacity Planning with ElasticSearch

To follow up on this... As a general rule is it better to have one horse size index or a hundred duck sized indices. I am thinking about those types of searches where you might frequently search a subset of the data. For example keeping a separate index for every customer because normally the a

BulkFile Indexing with base64 command utility on linux server

Hi, when iam indexing file content by using base64 command utility with CURL on linux server iam getting following exception. Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Failed to decode VALUE_STRING as base64 (MIME-NO-LINEFEEDS): Illegal character ''' (code 0x27) in ba

Re: elasticsearch-hadoop 1.3 M3 proxy support not working

Thanks for the detailed report. The user authentication triggered a bug [1] which has been fixed in master - I've already pushed the nightly builds which you can try out [2]. It would be great if you could try it out and report back if it works for you. If you want to stick with M3 until RC1 is

Re: elasticsearch-hadoop 1.3 M3 proxy support not working

Hi Costin, Thanks for your reply. Please find stackrace and other information that you have mentioned. *1. Stacktrace* java.lang.NullPointerException at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.setupHttpProxy(CommonsHttpTransport.java:169) at org.elasticsearch.hadoop.res

TooLongFrameException: HTTP header is larger than 8192 bytes.

Hi all, I am encountering an exception and this situation cannot be always reproduced, or in other word it's hard to reproduce. I use python requests module to communicate with ES server by the REST APIs. At the ES server side, I got the following exception: > org.elasticsearch.comm

Re: elasticsearch-hadoop 1.3 M3 proxy support not working

Hi, Can you post the stacktrace you are getting, I'm not sure what causes the NPE. Can you also confirm the Hadoop/Pig/OS version used? Additionally, could you check your script without the proxy or potentially without authentication - just curious whether it makes any difference. Cheers, On