Re: Hibernate Search with Elasticsearch

Oh I will be using MySQL in the future, right now I'm adding it upon AppFuse (which is a website that lets you log in, add users, and search for users). So I guess, Yes, right now I am only using Hibernate search. What should I do about the postgresql part until I get to using MySQL? And yes, when

Re: Using filtered query with a wildcard

Thank you David. I will evaluate and look into ngram analyzer to see how it can help on the specific scenario. Appreciate your help! On Thu, Feb 12, 2015 at 2:26 AM, David Pilato wrote: > Not answering to your question, but... > > If you are about performance, don’t use wildcard queries. > But p

[Hadoop][Spark] Exclude metadata fields from _source

Hey, I've recently started using Elasticsearch for Spark (Scala application). I've added elasticsearch-spark_2.10 version 2.1.0.BUILD-SNAPSHOT to my Spark application pom file, and used org.apache.spark.rdd.RDD[String].saveJsonToEs() to send documents to Elasticsearch. When the documents are loa

Re: Hibernate Search with Elasticsearch

postgresql is a database. I guess you are using Hibernate with a database, right? Unless you are using only Hibernate search? What do you mean "run in Spring"? Spring is the framework right? mvn jetty:run should work fine David > Le 12 févr. 2015 à 07:11, William Yang a écrit : > > Hey David

Re: Hibernate Search with Elasticsearch

Hey David, Thanks for responding - I found that link too and tried to follow it but I got confused what postgresql is. If I don't know what it is, where do I find it? Also I think I had some problems with a common dependency, I don't have the exact error code on me right now, but there was somethi

Re: Hibernate Search with Elasticsearch

Have a look here. https://github.com/dadoonet/legacy-search It's an Hibernate demo project which uses Spring as well. If you navigate through branches, it will show you how to add elasticsearch. I won't try to connect Hibernate Search and replace Lucene implementation by elasticsearch. My 2 cen

Re: Google Cloud "click-to-deploy"

Answer I got from Google team: "As of right now they'd have 2 options but neither of them are simple and "point and click" . They could create a new click to deploy ES cluster (in a new project), and move the data over manually, OR manually bring up a new instance and edit the config files with

Topology of tribe nodes in a distributed replicated environment and automatic index + alias creation.

Hey guys, We have a slightly different use case than I'm able to find examples for with the tribe node. Any feedback would be appreciated. What we have now: Single region: We create indexes in our code automatically. They're based on timeuuids, so we never have to worry about them conflict

Elasticsearch 1.4.3 and 1.3.8 Release (including CVE)

Sorry for the cross post, but as this release contains a CVE fix then everyone should be aware of the update. http://www.elasticsearch.org/blog/elasticsearch-1-4-3-and-1-3-8-released/ Today, we have released the *security* and bug fix release of *Elasticsearch 1.4.3*, based on *Lucene 4.10.3*, an

Re: Is there a security mailing list?

We do not have a security list. We do notify our direct customers as well as blogging on our site and tweeting and we also register an CVE, which goes out via those usual channels. Moving forward we'll also post to the lists, apologies for not doing so for this latest CVE. On 12 February 2015 at

Re: Async Replication and Multi DC

Here is what I felt I could do with shard allocation filtering: Use the following properties at the node level in each DC: In DC 1: Node 1: node.group1 = xxx node.group2 = yyy node.group4 = zzz Node 2: node.group2 = yyy node.group3 = zzz node.group5 = bbb Node 3: node.group1 = xxx node.grou

Re: Async Replication and Multi DC

Do not setup cross DC clusters, it's not recommended and ends up providing more headaches than it is worth. You are better off replicating your data using either snapshot+restore or sending it to each individual cluster through your code. On 12 February 2015 at 11:56, Parag Shah wrote: > Hi all

Async Replication and Multi DC

Hi all, I am trying to setup async replication with a multi-dc setup. Here is my scenario: 1. I will have 3 nodes in each DC. For now, assume I have only 2 DCs. 2. I would like to use async replication, such that it writes synchronously to the primary shard, but writes to only one node as

Hibernate Search with Elasticsearch

Hi everyone, I am new to the forum and I am also new to ElasticSearch! However for this project, I am trying to replace HibernateSearch with ElasticSearch. I am using Spring Framework, and I have also taken a look at both Spring-ElasticSearch and Spring-Data-ElasticSearch. However, I am still

Is there a security mailing list?

Is there a security announcement list for ELK somewhere? Something like what Ubuntu sends out when they patch security vulnerabilities. I don't see one listed on http://www.elasticsearch.org/community/ The list would have sent out emails about the issues listed on http://www.elasticsearch.or

[ANN] Elasticsearch Mapper Attachment plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch Mapper Attachment plugin, version 2.4.2. The mapper attachments plugin adds the attachment type to Elasticsearch using Apache Tika.. https://github.com/elasticsearch/elasticsearch-mapper-attachments/ Release Notes - elasticsea

Is Unsecured OPTIONS Method A Vulnerability?

I've been working lately on a project utilizing ElasticSearch and Kibana. To secure the ElasticSearch API I've hidden it behind a reverse proxy. The proxy uses a cookie to authenticate the request and forward it to the ElasticSearch server, but if no cookie is present or if the cookie does not

Re: SparkSQL and ElasticSearch not inferring JSON Schema correctly, possible bugs?

Costin Leua saw this on the Spark User Mailing List, and I have filed this as a bug in github: https://github.com/elasticsearch/elasticsearch-hadoop/issues/377 On Tuesday, February 10, 2015 at 5:18:57 PM UTC-8, Aris V wrote: > > I'm using ElasticSearch with elasticsearch-spark-BUILD-SNAPSHOT a

How to structure posts with comments for a blog?

What's the best way to structure a post/comment system using elasticsearch? I'm using elasticsearch as a secondary database. There would be a post with a multithreaded commenting system, maybe two levels deep. Each post can have up to 500-1000 comments. There will be incremental counters for

Google Cloud "click-to-deploy"

Hi, So we are a small startup and we use Google cloud as our cloud provider. Just last month Google announce how they've made it easy to deploy elasticsearch on their servers with just several clicks. I thought that this will be a nice thing to look at as it can save us the hustle of deploying ma

Searching IP Address Range Using CIDR Mask

Hi, I have an ip type defined in my index mapping. I would like to search the IP addresses using CIDR notation. I see IP range for aggregations using CIDR masks, but nothing else. Is what I want to do currently possible? Thanks. --- Ed Brown Software Architect -- You received this message bec

[ANN] Elasticsearch CouchDB River plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch CouchDB River plugin, version 2.4.2. The CouchDB River plugin allows to hook into couchdb _changes feed and automatically index it into elasticsearch.. https://github.com/elasticsearch/elasticsearch-river-couchdb/ Release Note

[ANN] Elasticsearch Twitter River plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch Twitter River plugin, version 2.4.2. The Twitter river indexes the public twitter stream, aka the hose, and makes it searchable. https://github.com/elasticsearch/elasticsearch-river-twitter/ Release Notes - elasticsearch-river

Re: Using filtered query with a wildcard

Not answering to your question, but... If you are about performance, don’t use wildcard queries. But prefer ngram analyzers with match queries. That will be really more efficient. In particular here, wildcard beginning with a star is the worse scenario performance wise. -- David Pilato | Tec

Re: Elasticon Agenda

Other than the Welcome Reception, no :) On 12 February 2015 at 03:04, Glen Smith wrote: > The currently published agenda: > http://www.elasticon.com/apex/Elastic_ON_Agenda > for Monday, March 9 shows just "Registration Opens" until 5pm. > > If registration will really be the only official activi

[ANN] Elasticsearch Phonetic Analysis plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch Phonetic Analysis plugin, version 2.4.2. The Phonetic Analysis plugin integrates phonetic token filter analysis with elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-phonetic/ Release Notes - elasticsear

[ANN] Elasticsearch Japanese (kuromoji) Analysis plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch Japanese (kuromoji) Analysis plugin, version 2.4.2. The Japanese (kuromoji) Analysis plugin integrates Lucene kuromoji analysis module into elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-kuromoji/ Rele

[ANN] Elasticsearch Stempel (Polish) Analysis plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch Stempel (Polish) Analysis plugin, version 2.4.2. The Stempel (Polish) Analysis plugin integrates Lucene stempel (polish) analysis module into elasticsearch.. https://github.com/elasticsearch/elasticsearch-analysis-stempel/ Rel

Re: is there any data size limit for backup and restore?

Yes, that is not a problem. There is no limit to the size of the backup. On 12 February 2015 at 03:17, Subbarao Kondragunta wrote: > > I have one index which has almost more than 10GB data size. > But entire size of all indices with all documents is more than 50 GB data > size. > Is elastic sear

[ANN] Elasticsearch ICU Analysis plugin 2.4.2 released

Heya, We are pleased to announce the release of the Elasticsearch ICU Analysis plugin, version 2.4.2. The ICU Analysis plugin integrates Lucene ICU module into elasticsearch, adding ICU relates analysis components.. https://github.com/elasticsearch/elasticsearch-analysis-icu/ Release Notes -

Using filtered query with a wildcard

Hello everyone, I came across few articles that says filtered query are generally faster. I need to perform some wildcard search on one of the fields. So my questions is... Can I perform wildcard search on the field along with filtered query? If so, can anyone please provide an example? I tried

Re: ClassCast when sorting on a long field (AtomicFieldData$WithOrdinals$1 cannot be cast to AtomicNumericFieldData)

Also note: I'm using the java TransportClient to index the documents. I get the ClassCast problem when querying using the TransportClient, but I *also* get the error when querying using http://localhost:9200/_plugin/marvel/sense/index.html. fwiw -- You received this message because you are

ClassCast when sorting on a long field (AtomicFieldData$WithOrdinals$1 cannot be cast to AtomicNumericFieldData)

This problem is really strange because my data has two long fields, and my search/filter works for one of them but not the other. Even weirder, I tried to create a curl reproducer for this with the exact same data and that works fine! Details here: https://gist.github.com/EricWittmann/86864fd8

SCAN type search and filtered query

Hi everyone, I would like to use a scan search with a filtered query but it doesn't seem to be possible. With a match_all query I get ( size * nb of shards ) hits. With a filtered query I get ( size ) hits as a total. This is why I think the scan type has been disabled. Here are the queries I'

Re: php api and self signed certs

Glad you got it working! Just a note, I don't monitor the ES mailing list very closely...but if you have any more problems/questions with the PHP client in the future, feel free to just open a tick at the repo and I would be happy to help :) -Z On Wednesday, February 11, 2015 at 7:09:57 AM UT

Re: ES OOMing and not triggering cache circuit breakers, using LocalManualCache

LocalManualCache is a component of Guava's LRU cache , which is used by Elasticsearch for both the filter and field data cache. Based on your nod

Re: Failed Shard Recovery

Bumping up. Does anyone have insight into this?? Mario On Tuesday, February 10, 2015 at 3:02:12 PM UTC-6, Mario Rodriguez wrote: > > > > On Tuesday, February 10, 2015 at 2:55:59 PM UTC-6, Mario Rodriguez wrote: >> >> I have 7 indices in our ES cluster, and one of the indices has an issue >> that

Groovy scripting vulnerability

Hi all Elasticsearch versions 1.3.0-1.3.7 and 1.4.0-1.4.2 have a vulnerability in the Groovy scripting engine. The vulnerability allows an attacker to construct Groovy scripts that escape the sandbox and execute shell commands as the user running the Elasticsearch Java VM. We have released Ela

Re: Tips for failing handshake with shield

There should be another message on the other end of the connection that has more details of the actual failure. That message can be seen on the client side of the connection when the connection was closed by the server side. Also, please see http://www.elasticsearch.org/guide/en/shield/current

Re: ES OOMing and not triggering cache circuit breakers, using LocalManualCache

After examining some other nodes that were using a lot of their heap, I think this is actually field data cache: $ curl "http://localhost:9200/_cluster/stats?human&pretty"; ... "fielddata": { "memory_size": "21.3gb", "memory_size_in_bytes": 22888612852, "evictions": 0 }

Re: Elasticsearch as Logwatch

I believe you can tell logwatch to output its reports as a file, which could then be ingested with logstash. Alternatively, logstash has an imap input that you could use to get emails into Elasticsearch. -- The information transmitted in this email is intended only for the person(s) or entity to

Upgrade to 1.4.2 from 1.1.1 causes spike in Disk IO

Hi there, We just completed this past Monday an ES upgrade from 1.1.1 to 1.4.2 and a Java upgrade from 7 to 8. No other changes. Same boxes, same specs, no new queries or filters, no increased requests. What I see in Marvel since the upgrade is a large spike in Disk IO:

Tips for failing handshake with shield

Hi, I am trying to get SSL/TLS to work on my raspberry pi cluster. To bad I get the following message: [2015-02-11 17:51:35,037][ERROR][shield.transport.netty ] [jc-pi-red] SSL/TLS handshake failed, closing channel: null So I guess something is wrong in my certificate chain. I have created my

ES OOMing and not triggering cache circuit breakers, using LocalManualCache

Hi all I have an ES 1.2.4 cluster which is occasionally running out of heap. I have ES_HEAP_SIZE=31G and according to the heap dump generated, my biggest memory users were: org.elasticsearch.common.cache.LocalCache$LocalManualCache 55% org.elasticsearch.indices.cache.filter.IndicesFilterCache 1

Re: Elasticsearch as Logwatch

Greg, I started researching exactly what you suggested: Pumping the daily logwatch results into elasticsearch. I already have an ELK in place and working. Do you know what is needed to implement this? There is no logwatch or email input. Thanks in advance. -- You received this me

is there any data size limit for backup and restore?

I have one index which has almost more than 10GB data size. But entire size of all indices with all documents is more than 50 GB data size. Is elastic search allows this much data to take backup and restore? -- You received this message because you are subscribed to the Google Groups "elastics

Elasticon Agenda

The currently published agenda: http://www.elasticon.com/apex/Elastic_ON_Agenda for Monday, March 9 shows just "Registration Opens" until 5pm. If registration will really be the only official activity, traveling in on Monday might be most convenient. Are there likely to be any other events added

Re: Delete index after backup

but will the index deletion get reflected in the next snapshot invocation? On Wednesday, February 11, 2015 at 5:11:23 PM UTC+2, David Pilato wrote: > > Removing an index does not remove a snapshot. > > -- > *David Pilato* | *Technical Advocate* | *Elasticsearch.com > *

Re: Delete index after backup

Removing an index does not remove a snapshot. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet | @elasticsearchfr | @scrutmydocs > Le 11 févr. 2015 à 16:07, Yarden Bar a

Re: jdbc river won't start

That was it. I deleted the river using DELETE /_river/my_river/_meta but that was wrong, I should have used: DELETE /_river/my_river/ Thanks! Am Mittwoch, 11. Februar 2015 15:13:47 UTC+1 schrieb Jörg Prante: > > You are sure you executed > > curl -XDELETE 'localhost:9200/_river/my_river/' > > b

Delete index after backup

HI all, Is there any procedure to *archive* index to HDFS (or other repo) and delete it? I've looked into the 'Snapshot/Restore' docs and understood that if I execute "snapshot -> delete index -> snapshot" the index deletion will get reflected in the snapshot process. Thanks, Yarden -- You

Weird problem with routing performance

Hi all, i have a weird (at least in my oppinion) situation on my environment. my hardware: 3 data nodes with 32GB of ram (16GB of Java heap for ES) using data from logstash, stored per day manner with 360 5*shard + 1 replica indexes -* NO ROUTING *-> grouped by alias named* norouting* and same

Elasticsearch search result problem ( Urgently)

Hello, I add data add Elasticsearch this code : $data = array("name"=>HelperMC::strToUtf8(trim($name)), "urlname"=>HelperURL::strToURL(trim($name)), "price"=>number_format(HelperParser::clearTextFromPrice($price), 2, ',', '.'), "price_for_filter"=>HelperPa

Re: Elasticsearch search result problem

i think problem price_for_filter but working this -> http://goo.gl/AuxCrR 11 Şubat 2015 Çarşamba 06:09:13 UTC+2 tarihinde Veli Can TEFON yazdı: > > Hello, > > I add data add Elasticsearch this code : > > $data = array("name"=>HelperMC::strToUtf8(trim($name)), > "urlname"=>HelperUR

Re: Phantom Snapshot Process

Found the answer. The /_cluster/state endpoint has a list of all snapshots in progress. For whatever reason, mine was stuck in INIT for a very long time. I deleted via the /_snapshot/repo/snapshot_name endpoint. On Wednesday, February 11, 2015 at 9:31:10 AM UTC-5, Matt Hughes wrote: > > My ES

Re: Large results sets and paging for Aggregations

> > 1.) Would I assume that as my document count would increase, the time for > aggregation calculation would as well increase? > Yes - documents are processed serially through the tree but it happens really quickly and each shard is doing this at the same time to produce its summary which is

Phantom Snapshot Process

My ES cluster seems to think it is in my middle of creating a snapshot yet I don't see any IN_PROGRESS snapshots in my repository. It's supposed to be an hourly snapshot, but I don't see anything that has started within a few days. Yet every time I try and do anything with the repository, it s

Re: jdbc river won't start

You are sure you executed curl -XDELETE 'localhost:9200/_river/my_river/' before creating the new river with same name? I see there is '_version: 12', this means, you did not stop the old river instance with DELETE. Jörg On Wed, Feb 11, 2015 at 2:38 PM, Abid Hussain wrote: > Hi, > > we are ru

Re: Availability of logstash-forwarder debian packages

Thanks, Magnus. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://gr

jdbc river won't start

Hi, we are running ES on a single node cluster. I'm facing the issue that a river is not being started after being re-created. In detail, we first delete the index the river refers to, then delete the river and then create it again. *GET /_river/my_river/_meta* { "_index": "_river", "_ty

Re: Large results sets and paging for Aggregations

aah..! This seems to be the best explanation of how aggregation works. Thanks a ton Mark for that. :) Few other questions: 1.) Would I assume that as my document count would increase, the time for aggregation calculation would as well increase? Reason: Trying to figure out if bucket creation is

Re: Availability of logstash-forwarder debian packages

On Wednesday, February 11, 2015 at 12:53 CET, Dennis Plöger wrote: > Until recently I used the elasticsearch-package repositories > (packages.elasticsearch.org) to install logstash-forwarder. However, > it now seems as if logstash-forwarder isn't available anymore via "deb > http://packages.

Re: Not able to save kibana dashboard

This worked. we need to run curl -X PUT http://<>:9200/kibana-int/ It started working like charm.. Regards, Bharath Singh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send

Re: php api and self signed certs

That is the hint I needed. I had it w/o quotes at first and the page was failing. I just assumed this was because of the lack of quotes, which once added allowed the page to load. I should have looked at the error console, which was showing the missing include :) All is good and working as exp

Availability of logstash-forwarder debian packages

Hi! Until recently I used the elasticsearch-package repositories (packages.elasticsearch.org) to install logstash-forwarder. However, it now seems as if logstash-forwarder isn't available anymore via "deb http://packages.elasticsearch.org/logstashforwarder/debian stable main". Did the structur

Not able to save kibana dashboard

Hi, Am not able to save the kibana dashboard. When I give save it displays "Save failed. Dashboard could not save to elasticsearch". Also I was able to see the below logs in elasticsearch log [2015-02-11 11:41:12,650][DEBUG][action.admin.indices.create] [Tribe Chief] no known master node, sche

Re: How to recover saved kibana dashboard

okies On Wednesday, February 11, 2015 at 4:43:07 PM UTC+5:30, Mark Walkom wrote: > > You should really create start your own thread for this :) > > On 11 February 2015 at 21:32, bharath singh > wrote: > >> Hi, >> Accidentally I deleted kibana-int. Now am not able to save my dashboard >> and its

Re: How to recover saved kibana dashboard

You should really create start your own thread for this :) On 11 February 2015 at 21:32, bharath singh wrote: > Hi, > Accidentally I deleted kibana-int. Now am not able to save my dashboard > and its showing "Save failed. Dashboard cannot be saved to Elastisearch". > In the logs I found this > >

Re: Can I use Newtonsoft.Json 4.5.0.0 instead of 6.0.0.0 with NEST?

I have tried this and also the commented out part... My project won't compile in both cases. I think I am doing something wrong. On Wednesday, February 11, 2015 at 11:38:19 AM UTC+1, Martin Widmer wrote: > > Can I use Newto

Re: Paid help with ES/ELK?

Hi Steve, We are the authors of elasticray - elasticsearch plugin for Liferay - https://www.liferay.com/marketplace/-/mp/application/41044606 https://github.com/R-Knowsys/elasticray also elasticray.com We have completed the official elasticsearch certification courses too. If you are still look

Re: Can I use Newtonsoft.Json 4.5.0.0 instead of 6.0.0.0 with NEST?

Try using assembly redirects, if that doesn't work it means no... -- Itamar Syn-Hershko http://code972.com | @synhershko Freelance Developer & Consultant Lucene.NET committer and PMC member On Wed, Feb 11, 2015 at 12:38 PM, Martin Widmer wrote: > Can I use Newt

Can I use Newtonsoft.Json 4.5.0.0 instead of 6.0.0.0 with NEST?

Can I use Newtonsoft.Json 4.5.0.0 instead of 6.0.0.0 with NEST? How? Thanks for your advice. Martin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsea

Re: Elastic Search - Sentiment analysis

Hi Jürgen, Thanks for the reply. Can you tell me how can the analysis be done to identify that it was a relevant financial document, say based on terms or phrases ? On Wednesday, February 11, 2015 at 2:03:20 AM UTC-8, Ap wrote: > > How do I identify relevant financial documents based on the >

Re: How to recover saved kibana dashboard

Hi, Accidentally I deleted kibana-int. Now am not able to save my dashboard and its showing "Save failed. Dashboard cannot be saved to Elastisearch". In the logs I found this [2015-02-11 192.168.xx.xx,322][DEBUG][action.admin.indices.create] [Tribe Chief] no known master node, scheduling a retr

Re: Elastic Search - Sentiment analysis

Well, you use a linguistic extraction tool in the processing pipeline to yield the information you need. It will be indexed together with the document and may be queried, aggregated, etc. On 11.02.2015 11:03, Ap wrote: > How do I identify relevant financial documents based on the > terms/phrases/s

Elastic Search - Sentiment analysis

How do I identify relevant financial documents based on the terms/phrases/sentiments present inside the Document ? eg. Relevant document might contain--> Wall Street hits a new High. Irrelevant document might contain --> I was walkng on Wall Street and met an old friend. -- You receiv

Re: Large results sets and paging for Aggregations

5k doesn't sound too scary. Think of the aggs tree like a "Bean Machine" [1] - one of those wooden boards with pins arranged on it like a christmas tree and you drop balls at the top of the board and they rattle down a choice of path to the bottom. In the case of aggs, your buckets are the pins

Transport Error 400 - Failed to derive xcontent

Hi all, I'm trying to run a scrapy crawler to feed elasticsearch but I seem to be getting this error and I can't understand how to begin debugging this. File "/var/www/app/venv/local/lib/python2.7/sitepackages/elasticsearch/connection/base.py", line 97, in _raise_error HTTP_EXCEPTIONS.get(st

finding child documents without parents

We have child documents and we suspect that some of them don't have their parents. Can someone suggest a query I can use to find child documents without parents? Thanks, Ron -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from

Re: ES JAVA API | QueryBuilders.java | FilterBuilders does not have nullable queryBuilder as arguement

Ok. That helps a lot in getting the things but still I do feel, since ES is internally marking the query as MatchAll query, the method definition should have Nullable query as well. Thanks once again for helping out. :) Regards Piyush On Wednesday, 11 February 2015 13:29:32 UTC+5:30, David P