Re: [Kibana 4] Add threshold line to visualization

[Yarden Bar]

Thanks Ruvi, that works for now.

On Sunday, April 26, 2015 at 9:58:16 PM UTC+3, P lva wrote:
>
> Not an exact answer but you could use a filter and stacked lines. Have one 
> color for above threshold and a different color and under threshold. 
>
> Thanks
> Ruvi
> ----------
> From: Yarden Bar 
> Sent: ‎4/‎26/‎2015 2:38 PM
> To: elasti...@googlegroups.com 
> Subject: [Kibana 4] Add threshold line to visualization
>
> Hi all,
> I'm sure I'm not the first one with this idea, but couldn't find anything 
> on the net which resembles a solution.
>
> My goal is to have a horizontal line expressing a threshold level.
> The scenario in questions is a Kibana 4 visualization of NginX request 
> duration average. I want to set a horizontal line on the graph, making it 
> easy to point when the threshold is crossed.
>
> One Idea is to add the threshold level as a field in each document and 
> display the average per same time slice of the graph.
>
> Any other ideas?
>
> Thanks,
> Yarden
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/2286424f-61a6-44f6-9e3f-316a29a0ebbc%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/2286424f-61a6-44f6-9e3f-316a29a0ebbc%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/19c1f849-385f-4c9d-93d9-3a96e565c55f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kibana 4] Add threshold line to visualization

[Yarden Bar]

Hi all,
I'm sure I'm not the first one with this idea, but couldn't find anything 
on the net which resembles a solution.

My goal is to have a horizontal line expressing a threshold level.
The scenario in questions is a Kibana 4 visualization of NginX request 
duration average. I want to set a horizontal line on the graph, making it 
easy to point when the threshold is crossed.

One Idea is to add the threshold level as a field in each document and 
display the average per same time slice of the graph.

Any other ideas?

Thanks,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2286424f-61a6-44f6-9e3f-316a29a0ebbc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Applicative version type mapping

[Yarden Bar]

Hi there,

I was looking for a type mapping for application version ? 
Meaning that the following seria 
["3.9.1","4.0.0",2.5.3.1.alpha","6.3.1.beta"] can be 
queried/searched/sorted ?

Does ES has support for that ?

Thanks,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9aa3a12f-96e1-41b9-ad8d-b4bf32f45d87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasicsearch alerts on Icinga

[Yarden Bar]

Thanks Mark for the Nagios plugin suggestion. just that I'm not aiming at 
monitoring ES itself (that would be later, down the road)

What I'm looking for is an efficient way( a framework maybe) to register 
alerts into Icinga/Nagios that will recieve an Elasticsearch query as a 
parameter and start alerting based on a threshold parameters(which are 
supplied as well).

Currently I'm coding each ES based check manually.

Thanks in advance for any ideas..
Yarden

On Thursday, March 19, 2015 at 6:39:06 PM UTC+2, Mark Walkom wrote:
>
> There are a few nagios scripts people have written, eg 
> https://github.com/anchor/nagios-plugin-elasticsearch
>
> On 19 March 2015 at 05:48, Yarden Bar > 
> wrote:
>
>> Hello good people,
>>
>> We have an ELK setup for our nginx / postfix etc logs. it's great.
>>
>> Now we'd like to be able to alert based on various criteria. icinga is 
>> great, we just installed it to play with. 
>>
>> is there a plugin that we can use to query elasticsearch from within 
>> icinga, to create alerts? 
>>
>> we have a POC but it's a ruby script, hence clunky...
>>
>> Thanks,
>> Yarden
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/4d325d11-8197-44ed-a73d-802f21534d9a%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/4d325d11-8197-44ed-a73d-802f21534d9a%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0b2e9bbf-7d6c-4582-9a4a-709c2fead1ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Navigation path

[Yarden Bar]

HI Lior,
Have your tried aggregation 

 
query?

On Wednesday, March 18, 2015 at 6:23:50 PM UTC+2, Lior Goldemberg wrote:
>
> hi all 
>
> How would you handle the following problem. 
> I save for each back office user the list of pages he visited. 
>
> Now I want to fetch all the users who visited page "a" and then page "b" 
> and then page "e" 
>
> Does elasticsearch have something built in? 
>
> Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dce6c97b-0070-46c0-a781-85514a8811d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasicsearch alerts on Icinga

[Yarden Bar]

Hello good people,

We have an ELK setup for our nginx / postfix etc logs. it's great.

Now we'd like to be able to alert based on various criteria. icinga is 
great, we just installed it to play with. 

is there a plugin that we can use to query elasticsearch from within 
icinga, to create alerts? 

we have a POC but it's a ruby script, hence clunky...

Thanks,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4d325d11-8197-44ed-a73d-802f21534d9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ES-1.4.0] Snapshot Queue

[Yarden Bar]

Hi All,

Is there a way to queue snapshot invocations?

e.g.: snapshot_1 -> snapshot_2 ->  -> snapshot_N


Thanks,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/33727f3a-f969-48fb-9c9b-fb360ccd9e08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ES-1.4.0] Changing linux user ID

[Yarden Bar]

Issue was solved by modifying user/group IDs to mactch on all machines.

Thanks for the ideas :) 

On Wednesday, February 18, 2015 at 10:50:50 AM UTC+2, David Pilato wrote:
>
> Well if you are using a shared FS for snapshot/restore, you need to make 
> sure that each node user can write to this dir. 
> Try from all machines to write a file when running as es_01 and es_02
> May be you could add them to the same group and give write privileges to 
> the group?
>
> David
>
> Le 18 févr. 2015 à 09:09, Yarden Bar  > a écrit :
>
> Hi All,
>
> I encountered a 'Permission Denied' error while using ES Snapshot API to 
> add a 'FS' repository pointing a NFS shared folder.
>
> My cluster structure is:
>
>- NFS_MACHINE - user id is 600
>- ES_01 - user ID that runs ES is 1000
>- ES_02 - user ID that runs ES is 600
>
> The issue seems to be that the difference in user IDs between the machines 
> generated the permission issues.
>
> My question is about ES files across a machine file system, are there any 
> files outside ES installation folder (except for data/logs/configuration)?
>
> Thanks,
> Yarden
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/582c5f18-f40a-42cd-85bf-fcfeb8b18e8c%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/582c5f18-f40a-42cd-85bf-fcfeb8b18e8c%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/01f0238b-cdca-46fc-bd38-bf12fcabeaeb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ES-1.4.0] Changing linux user ID

[Yarden Bar]

Hi All,

I encountered a 'Permission Denied' error while using ES Snapshot API to 
add a 'FS' repository pointing a NFS shared folder.

My cluster structure is:

   - NFS_MACHINE - user id is 600
   - ES_01 - user ID that runs ES is 1000
   - ES_02 - user ID that runs ES is 600

The issue seems to be that the difference in user IDs between the machines 
generated the permission issues.

My question is about ES files across a machine file system, are there any 
files outside ES installation folder (except for data/logs/configuration)?

Thanks,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/582c5f18-f40a-42cd-85bf-fcfeb8b18e8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ES-1.4.2] Snapshot FS Repository directory structure

[Yarden Bar]

If I understand you correctly, the monthly/weekly repository will get 
purged when we decide to, right?

Another solution crossed my mind:
Thinking of HDFS, which I'm not so familiar with, but is there some kind of 
HDFS node which is similar to ES node which is not a master nor a data?
This way all snapshot will go through a HDFS 'gateway' node to outside of 
our prod env.

On Monday, February 16, 2015 at 11:30:14 AM UTC+2, Mark Walkom wrote:
>
> In these sorts of cases it may make sense to have a repo for a week/months 
> worth of snapshots, then just retain however much you need in these rolling 
> repos. Kind of like what LS does for its indices.
>
> Regarding deleting the files, take a look at 
> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/backing-up-your-cluster.html#_deleting_snapshots
>
> On 16 February 2015 at 19:04, Yarden Bar <
> yar...@astoncapitalmanagement.com > wrote:
>
>> Hi All,
>> We're using the snapshot API to backup our ES cluster and as time 
>> advances, the repository storage utilization is growing.
>> Currently I use 'rsync' to copy snapshot repository data outside of our 
>> production environment, but eventually I'd need to delete some snapshots to 
>> free space for new ones.
>>
>> The repository keeps files like 'index', 'snapshot-*' and 'metadata-' 
>> files and I suspect that deleting these files will fail the repository from 
>> being functional.
>>
>> Is there a way that i can 'rsync' the snapshot repository to another 
>> location and delete the existing snapshots?
>> Can I delete that snapshot data itself (the 'indices' directory the 
>> resides in the repository) without damaging the repository itself?
>>
>> Thanks for any idea,
>> Yarden
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3d7428c8-2245-4ba1-9109-46c8877f1913%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ES-1.4.2] Snapshot FS Repository directory structure

[Yarden Bar]

Hi All,
We're using the snapshot API to backup our ES cluster and as time advances, 
the repository storage utilization is growing.
Currently I use 'rsync' to copy snapshot repository data outside of our 
production environment, but eventually I'd need to delete some snapshots to 
free space for new ones.

The repository keeps files like 'index', 'snapshot-*' and 'metadata-' files 
and I suspect that deleting these files will fail the repository from being 
functional.

Is there a way that i can 'rsync' the snapshot repository to another 
location and delete the existing snapshots?
Can I delete that snapshot data itself (the 'indices' directory the resides 
in the repository) without damaging the repository itself?

Thanks for any idea,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0396367d-7a1d-4a1b-a320-91bd25f1e9cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Delete index after backup

[Yarden Bar]

Found the last section here 
<http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/retiring-data.html>
 
'archiving old indices' to answer my question.


On Wednesday, February 11, 2015 at 5:17:04 PM UTC+2, Yarden Bar wrote:
>
> but will the index deletion get reflected in the next snapshot invocation?
>
> On Wednesday, February 11, 2015 at 5:11:23 PM UTC+2, David Pilato wrote:
>>
>> Removing an index does not remove a snapshot.
>>
>> -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com 
>> <http://Elasticsearch.com>*
>> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr 
>> <https://twitter.com/elasticsearchfr> | @scrutmydocs 
>> <https://twitter.com/scrutmydocs>
>>
>>
>>  
>> Le 11 févr. 2015 à 16:07, Yarden Bar  a écrit :
>>
>> HI all,
>>
>> Is there any procedure to *archive* index to HDFS (or other repo) and 
>> delete it?
>>
>> I've looked into the 'Snapshot/Restore' docs and understood that if I 
>> execute "snapshot -> delete index -> snapshot" the index deletion will get 
>> reflected in the snapshot process.
>>
>> Thanks,
>> Yarden
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/2fca7f58-4fc3-475d-92dd-3d06b486b81d%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/2fca7f58-4fc3-475d-92dd-3d06b486b81d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f9fb9951-a147-42a5-9048-8611d4f2b60e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Delete index after backup

[Yarden Bar]

but will the index deletion get reflected in the next snapshot invocation?

On Wednesday, February 11, 2015 at 5:11:23 PM UTC+2, David Pilato wrote:
>
> Removing an index does not remove a snapshot.
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com 
> <http://Elasticsearch.com>*
> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr 
> <https://twitter.com/elasticsearchfr> | @scrutmydocs 
> <https://twitter.com/scrutmydocs>
>
>
>  
> Le 11 févr. 2015 à 16:07, Yarden Bar > 
> a écrit :
>
> HI all,
>
> Is there any procedure to *archive* index to HDFS (or other repo) and 
> delete it?
>
> I've looked into the 'Snapshot/Restore' docs and understood that if I 
> execute "snapshot -> delete index -> snapshot" the index deletion will get 
> reflected in the snapshot process.
>
> Thanks,
> Yarden
>
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/2fca7f58-4fc3-475d-92dd-3d06b486b81d%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/2fca7f58-4fc3-475d-92dd-3d06b486b81d%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/58a39a47-6dc2-4f4e-9515-9895dc38fbcf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Delete index after backup

[Yarden Bar]

HI all,

Is there any procedure to *archive* index to HDFS (or other repo) and 
delete it?

I've looked into the 'Snapshot/Restore' docs and understood that if I 
execute "snapshot -> delete index -> snapshot" the index deletion will get 
reflected in the snapshot process.

Thanks,
Yarden


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2fca7f58-4fc3-475d-92dd-3d06b486b81d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.3.4 scrolling never ends

[Yarden Bar]

Finally the issue was solved.

I forgot to mention that I had a Logstash output connected and it's protocol 
 was set to 
'node', meaning that logstash was part of my cluster.
Once I set the protocol to 'transport',scrolling was perfect!!

Credit to my team-leader for guidance


Thanks everyone for your help
Yarden

On Monday, November 10, 2014 7:06:52 PM UTC+2, Brian wrote:
>
> A while back, I wrote my own post-query response sorting so that I could 
> handle cases that Elasticsearch didn't. One case was sorting a scan query. 
> I used a Java TreeSet class and could also limit it to the top 'N' 
> (configurable) items. It is very, very quick, pretty much adding no 
> overhead to the existing scan logic. And it supports an arbitrarily 
> complex compound sort key, much like an SQL ORDERBY statement; it's very 
> easy to construct.
>
> Probably not useful for a normal user query, but it is very useful for an 
> ad-hoc query in which I wish to scan across an indeterminately large result 
> set but still sort the results. 
>
> One of these days, it might make a good plug-in candidate. But I am not 
> sure how to integrate it with the scan API, so for now it's just part of 
> the Java client layer.
>
> Brian
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/44b5c8db-6c09-4527-b440-09d01bde3588%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.3.4 scrolling never ends

[Yarden Bar]

Hi Jorg,

I cant use scan type because I need the documents sorted ASC on a field, scan 
returns the documents in the order they indexed.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1eb0d4dd-1659-48a2-929b-194ebd531465%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.3.4 scrolling never ends

[Yarden Bar]

One issue I identified is the heap size was too small for the query, I've 
increased the heap memory and the CircuitBreakerException stopped happening.

But the scrolling still returning the SAME result.

An updated code example is below:
import org.elasticsearch.action.search.SearchType
import org.elasticsearch.client.transport.TransportClient
import org.elasticsearch.common.settings.ImmutableSettings
import org.elasticsearch.common.transport.InetSocketTransportAddress
import org.elasticsearch.common.unit.TimeValue
import org.elasticsearch.index.query.{FilterBuilders, QueryBuilders}
import org.elasticsearch.search.Scroll
import org.elasticsearch.search.sort.SortOrder

val es_settings = ImmutableSettings.settingsBuilder().put("transport.sniff", 
true).put("cluster.name", "test_acm_es").build()
var client = new TransportClient(es_settings).addTransportAddress(new 
InetSocketTransportAddress("myServer",9300))
val query = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), 
FilterBuilders.queryFilter(
 QueryBuilders.queryString("((market:2 AND feed:55) OR (market:2 AND 
feed:32))")))
var result = client.prepareSearch("orderbook-2014.11.03").setTypes(List(
"level"):_*).setQuery(query).setSearchType(SearchType.DFS_QUERY_THEN_FETCH).
setSize(1).addSort("updateNo", SortOrder.ASC).setScroll(new Scroll(
TimeValue.timeValueMinutes(5))).get()
var scrollId = ""
var itr = 0
do {
 scrollId = result.getScrollId
 result = client.prepareSearchScroll(scrollId).setScroll(TimeValue.
timeValueMinutes(3)).get()
 println(s"Iteration=$itr, scrollResult=${result.getHits.getHits.length}")
// println("")
// result.getHits.getHits.foreach(h => println(h.getId))
// println("")
 itr+=1
} while (result.getHits.getHits.length != 0)

enabling the print block reveals that the searchHit array is the same for 
each iteration...

Thanks,
Yarden

On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:
>
> Hi all,
>
> I'm encountering a strange behavior when executing a search-scroll on a 
> single node of ES-1.3.4 with Java client.
>
> The scenario is as follows:
>
>1. Start a single node of version 1.3.4
>2. Add snapshot repository pointing to version 1.1.1 snapshots
>3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
>4. Execute search on an index with 
>5. client.prepareSearch("my_index*").setQuery(QueryBuilders.
>filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter
>(
>  QueryBuilders.queryString(s"$terms AND 
>snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") ))   )
> .addFields(OBFields.values.map(_.toString).toList: _*).setSize(
>pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)
>.setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet()
>
>
>6. Execute the following search scroll 
>client.prepareSearchScroll(scrollId).setScroll(TimeValue.
>timeValueMinutes(3)).execute().actionGet()
>
> I have a loop iterating over #6, providing the same scrollId and checking 
> for (result.getHits().getHits().legth == 0) to terminate.
> I keep getting the same result 'page' with the same amount of results.
>
>
> Any Idea??
>
>
> Thanks,
> Yarden
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/66e02775-17dd-4ea0-a8b3-39eb7e2a7aca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.3.4 scrolling never ends

[Yarden Bar]

Update:
Only when I set the SearchType to something else than 
the QUERY_AND_FETCH the scroll success to finish.

Any idea why QUERY_THEN_FETCH(the default) brings me to an endless loop?

The full code is:

val client = ESClientFactory.createByNode(ESNode.Builder,cluster = 
"test_acm_es")

val query = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(), 
FilterBuilders.queryFilter(
  QueryBuilders.queryString("((market:2 AND feed:55) OR (market:2 AND 
feed:32)) AND snapshotNo:[12614 TO 12627]")))

var result: SearchResponse = client.prepareSearch("orderbook-2014.10.21")
  .setQuery(query)
  .addFields(OBFields.values.map(_.toString).toList: _*)
  .setSearchType(SearchType.QUERY_THEN_FETCH)
  .setSize(1000)
  .addSort(OBFields.updateNo.toString, SortOrder.ASC)
  .setScroll(new Scroll(TimeValue.timeValueMinutes(5)))
.execute().actionGet()


println(s"Result total hits:${result.getHits.totalHits()}")
println(s"Result hits:${result.getHits.getHits().length}")
  do {
//result = new 
SearchScrollRequestBuilder(client,result.getScrollId).setScroll(TimeValue.timeValueMinutes(2)).execute().actionGet()
result = 
client.prepareSearchScroll(result.getScrollId).setScroll(TimeValue.timeValueMinutes(2)).execute().actionGet()
println(s"Iteration=$itr, 
scrollResult=${result.getHits.getHits.length}")
itr += 1

  } while (result.getHits.getHits.length != 0)



Thanks for any idea...
Yarden

On Wednesday, November 5, 2014 5:52:25 PM UTC+2, Yarden Bar wrote:
>
> I'll try that and report
>
> Thanks,
> Yarden
>
> On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:
>>
>> Hi all,
>>
>> I'm encountering a strange behavior when executing a search-scroll on a 
>> single node of ES-1.3.4 with Java client.
>>
>> The scenario is as follows:
>>
>>1. Start a single node of version 1.3.4
>>2. Add snapshot repository pointing to version 1.1.1 snapshots
>>3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
>>4. Execute search on an index with 
>>5. client.prepareSearch("my_index*").setQuery(QueryBuilders.
>>filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.
>>queryFilter(
>>  QueryBuilders.queryString(s"$terms AND 
>>snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") ))   )
>> .addFields(OBFields.values.map(_.toString).toList: _*).setSize(
>>pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)
>>.setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet
>>()
>>
>>
>>6. Execute the following search scroll 
>>client.prepareSearchScroll(scrollId).setScroll(TimeValue.
>>timeValueMinutes(3)).execute().actionGet()
>>
>> I have a loop iterating over #6, providing the same scrollId and checking 
>> for (result.getHits().getHits().legth == 0) to terminate.
>> I keep getting the same result 'page' with the same amount of results.
>>
>>
>> Any Idea??
>>
>>
>> Thanks,
>> Yarden
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6f1a1178-d4e5-41e9-a464-68c6e9204779%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES 1.3.4 scrolling never ends

[Yarden Bar]

I'll try that and report

Thanks,
Yarden

On Wednesday, November 5, 2014 2:48:46 PM UTC+2, Yarden Bar wrote:
>
> Hi all,
>
> I'm encountering a strange behavior when executing a search-scroll on a 
> single node of ES-1.3.4 with Java client.
>
> The scenario is as follows:
>
>1. Start a single node of version 1.3.4
>2. Add snapshot repository pointing to version 1.1.1 snapshots
>3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
>4. Execute search on an index with 
>5. client.prepareSearch("my_index*").setQuery(QueryBuilders.
>filteredQuery(QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter
>(
>  QueryBuilders.queryString(s"$terms AND 
>snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") ))   )
> .addFields(OBFields.values.map(_.toString).toList: _*).setSize(
>pageSize).addSort(OBFields.updateNo.toString, SortOrder.ASC)
>.setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet()
>
>
>6. Execute the following search scroll 
>client.prepareSearchScroll(scrollId).setScroll(TimeValue.
>timeValueMinutes(3)).execute().actionGet()
>
> I have a loop iterating over #6, providing the same scrollId and checking 
> for (result.getHits().getHits().legth == 0) to terminate.
> I keep getting the same result 'page' with the same amount of results.
>
>
> Any Idea??
>
>
> Thanks,
> Yarden
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c658566c-f2e4-4020-bd14-08e413c81a9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ES 1.3.4 scrolling never ends

[Yarden Bar]

Hi all,

I'm encountering a strange behavior when executing a search-scroll on a 
single node of ES-1.3.4 with Java client.

The scenario is as follows:

   1. Start a single node of version 1.3.4
   2. Add snapshot repository pointing to version 1.1.1 snapshots
   3. Restore snapshots version 1.1.1 snapshot to 1.3.4 node
   4. Execute search on an index with 
   5. client.prepareSearch("my_index*").setQuery(QueryBuilders.filteredQuery
   (QueryBuilders.matchAllQuery(), FilterBuilders.queryFilter(
 QueryBuilders.queryString(s"$terms AND 
   snapshotNo:[${mdp.fromSnapshot} TO ${mdp.toSnapshot}]") ))   )
.addFields(OBFields.values.map(_.toString).toList: _*).setSize(pageSize
   ).addSort(OBFields.updateNo.toString, SortOrder.ASC)
   .setScroll(TimeValue.timeValueMinutes(3)).execute().actionGet()
   
   
   6. Execute the following search scroll 
   client.prepareSearchScroll(scrollId).setScroll(TimeValue.timeValueMinutes
   (3)).execute().actionGet()
   
I have a loop iterating over #6, providing the same scrollId and checking 
for (result.getHits().getHits().legth == 0) to terminate.
I keep getting the same result 'page' with the same amount of results.


Any Idea??


Thanks,
Yarden

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/be0f385c-9d46-492b-a818-9bb04c92b214%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.