Configure an index to always index documents using nested types

[Matthew Dexter]

How can I configure an index such that all documents added to the index 
which are not flat are always indexed as nested regardless of the depth of 
the hierarchy.

I believe this is possible with dynamic templates using the _default_ index 
but I can't find any definitive answer as to how to do this.

Thanks,

Matthew

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5a1fedfb-85c3-4ce3-8443-f7f5190f65a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Filter index to last 24h (REST)

[Matthew]

Hi all,

Is there any way to only load the last 24 hours of indices? I am trying to 
apply a query to only show the number of documents created over the last 24 
hours (over the REST API), but I have not had too much luck.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/07576351-5bea-4f99-af51-16ff76791914%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Indices are missing. Help!

[Matthew Eash]

As a followup - seems there is a major issue in indices created from a 
mapping template.   I installed a fresh copy of ES 1.4.2 standalone on my 
laptop and replicated the issue I had on my ES 1.4.1 cluster -- 
disappearing indices on cluster restart.

https://github.com/elasticsearch/elasticsearch/issues/9223

Would love some insight from ES devs on if it's possible to get the 
"disappeared" indices back and visible in ES.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3658ec01-7901-4054-942e-52509d77bafe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Indices are missing. Help!

[Matthew Eash]

I have a 3 node ES 1.4.1 cluster that runs on CentOS6, Oracle JDK 1.7.0_67. 
  Heap was set to 20G of the 32G on those boxes, with mlockall set.   
Configuration is currently set more towards bulk loading more than it is 
searching.  Purpose of the ES cluster was for time-series indexing of 
logged metrics.  I originally had one larger index (1.1B docs) tracking a 
high frequency metric over the past several months, but recently changed 
schema design to do an index per-day.  I was loading additional metrics as 
well as reimporting the data in that larger index into per-day.   ES search 
usage is very light at the moment.

Last night, I had finished a multi-day bulk import of several months worth 
of multiple log metrics into per-day indices. The per-day indices were all 
either 12M or 48M records with settings of {shards=4, replication=0, 
refresh_interval=-1} while I bulk loaded.  After a day was fully loaded in 
bulk and no more writes necessary, I was optimizing each to 1 segment 
(taking 30-45s), then ultimately was going to set 
{replication=1,refresh_interval=30s} once all were individually optimized. 
 As of last night, I was about 1/4 of the way through optimizing, and none 
of them (beyond the larger index) were replicated.

After bulk import was done, I was poking around ES API, not really doing 
anything extraordinary (some searches, some optimization/merges of 
individual per-day indexes that I had done even while bulk importing).  At 
that time, some event ultimately spun out 2 of the nodes, making them 
inaccessible.  I'm still trying to diagnose what exactly occurred - this 
not the first occurrence of this mystery spin out of a node, but never had 
2 go at once. I believe the JVM is locking up the kernel some how.  I could 
ping them, but could not access the machines in any way.  Through the night 
- it seems the inaccessible machines occasionally attempted to reestablish 
the cluster only to disappear again.  The remaining node just flailed, 
attempting to establish master most of the time.   

This morning, I had to have the machines physically rebooted at the 
console, as they were still unresponsive.

So - I'm still trying to diagnose what exactly went wrong.  I do recall 
seeing the heap size on all the nodes start growing to about double the 20G 
I had assigned - but am unsure if that caused whatever freeze up occurred. 
  (Would love to know where to start looking.)

However, my more immediate issue  when the cluster came back up after 
reboot, only 1 index is showing, my original 1.1B-doc larger, replicated 
index. *  All of my daily per-day indexes created over the past 2 weeks are 
completely missing in ES.* /_cat/indices yesterday, showed 276 happy 
green indexes, today it shows only 1.  After looking at the raw data 
directories (split across 2 volumes on local spinning disks), it's all 
still there... all index directories exist and within them I see all the 
raw Lucene shard dirs and segment files.

Since the cluster reboot, only this stands out in the logs, from the master 
node:
[2015-01-07 12:22:14,348][INFO ][gateway  ] [node3] 
recovered [1] indices into cluster_state
[2015-01-07 12:22:14,440][INFO ][indices.store] [node3] Failed 
to open / find files while reading metadata snapshot

Subsequent reboots only show 1 indices recovered and don't have the 
metadata failure message.

Is there any way to fix the index metadata to reestablish the indices that 
were all there yesterday, and still exist on the disk?  How do I go about 
cleaning this up?  I am finding nothing in ES documentation talking about 
internal index metadata (where it's stored, how to fix corruption, or 
anything about this error message).

I want to root cause the node failures that occurred - but that is likely a 
deep issue that will take a while to research/diagnose.   My more immediate 
need is getting those indexes back first!  Any attempt to see or deal with 
those indices now gets an IndexMissingException.   

My only clue in why this occurred thus far is that one of the failing nodes 
kept trying to reestablish a 2-node cluster with itself as master through 
the night with the lone working node, then kept failing and dropping the 
other node from cluster.  During that time and after the new master found 
itself alone, this appeared in log for many of the per-day indexes:
[2015-01-07 00:05:20,254][DEBUG][action.admin.indices.stats] [node1] 
[temp-2014-11-14][3], node[fwGNfUZJTmmkAj4hpCobWg], [P], s[STARTED]: failed 
to execute 
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@7b542938]
org.elasticsearch.transport.NodeDisconnectedException: 
[node3][inet[/172.16.0.34:9300]][indices:monitor/stats[s]] disconnected

This occurred again 2 hours later. Would the master then expel the index 
after stat request failures?

Any assistance would be greatly appreciated!  Cluster is behaving fine at 
the moment now that nodes were rebooted, just is missing 2

Re: 1.4.0 data node can't join existing 1.3.4 cluster

[Matthew Barrington]

I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:
>
> We are running a 1.3.4 cluster using the AWS plugin and I noticed the same 
> error when I tried to upgrade a single node.
>
> Since I was trying this on my test cluster first I decided to see what 
> would happen if I upgraded a 2nd node. Would it split into 2 clusters, have 
> the same issue, etc.
>
> What I discovered was that when 2 nodes were upgraded to 1.4 they joined 
> the cluster correctly and everything looks to be working.
>
> SO the problem seems to be for the initial node to join, but when you try 
> with two everything works out.
>
> On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
>>
>> On Fri, Nov 14, 2014 at 3:41 AM,   wrote: 
>> > I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 
>> cluster 
>> > with cloud-aws plugin version 2.4.0. Is there a workaround to use 
>> during 
>> > upgrade, since I assume it's not a problem when they're all upgraded to 
>> > 1.4.0. 
>>
>> I ended up starting a new cluster (ignoring all the warnings logged on 
>> startup), and restoring from a snapshot. Once all the 1.3.4 nodes were 
>> gone, no issues. 
>>
>> -- 
>> Eric Jain 
>> Got data? Get answers at zenobase.com. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fc79c5e7-ddd6-4f52-9641-1bd01df3b866%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 1.4.0 data node can't join existing 1.3.4 cluster

[Matthew Barrington]

We are running a 1.3.4 cluster using the AWS plugin and I noticed the same 
error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what 
would happen if I upgraded a 2nd node. Would it split into 2 clusters, have 
the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined 
the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try 
with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:
>
> On Fri, Nov 14, 2014 at 3:41 AM,  > 
> wrote: 
> > I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 
> cluster 
> > with cloud-aws plugin version 2.4.0. Is there a workaround to use during 
> > upgrade, since I assume it's not a problem when they're all upgraded to 
> > 1.4.0. 
>
> I ended up starting a new cluster (ignoring all the warnings logged on 
> startup), and restoring from a snapshot. Once all the 1.3.4 nodes were 
> gone, no issues. 
>
> -- 
> Eric Jain 
> Got data? Get answers at zenobase.com. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f618a79d-0a4f-4f97-8b89-ab6ccb9d1cbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Relational Data Modelling in Elasticsearch

[Matthew Kaufman]

Just cluster it if you need to have a better performance out of your index 
update bulk upsert runs...


On Thursday, November 13, 2014 2:34:09 PM UTC-5, Philip K. Adetiloye wrote:
>
> I'm indexing  data from a relational database to Elasticsearch which 
> requires 
> several joins across multiple tables.
>
> Now, am thinking if using Nested object to model my relationship would be 
> a good idea.
>
> Basically, I want to create separate batch jobs for each related tables to 
> push data 
> from the database to ES using bulk UPSERT. 
>
> i.e if the document id exist,  partially update the child (nested) 
> document  otherwise, creates a new document, set the id and partially 
> update 
> the child (nested) document.
>
>
> Would there be a big performance hit if I have 5 jobs doing partial 
> updates on each document (x 1+ million documents) on my index?
>
> My data rarely change once it's indexed and so far I've not seen any 
> disadvantage of using nested objects.
>
> Please advise.
>
> Thanks,
> - Phil
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f2afba1c-d14a-4591-b750-f0e1ece7b814%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

[Matthew Hall]

I'm using IntelliJ Ultimate, in a linux environment, jdk 1.70_71, I have
maven and ant but haven't used them much in the past :/.

Anyway thanks for trying to help :)

Matt

On Mon, Nov 3, 2014 at 3:27 PM, joergpra...@gmail.com  wrote:

> It is hard to guess what commands you executed and what your environment
> is.
>
> You have to set up a useable IDE environment. Currently:
>
> - I recommend IntelliJ IDEA Community Edition
> - Java JDK 7u65 or Java JDK 8u25
> - Maven 3.2.3
> - Elasticsearch 1.3 or 1.4.Beta1
>
> With IDE support and Maven, detecting dependencies like JVM and
> Elasticsearch jars is a snap.
>
> For example, you could fire up a simple plugin
>
> https://github.com/jprante/elasticsearch-simple-action-plugin
>
> then change the code to add your native script, and save it as your
> project.
>
> Jörg
>
>
>
> On Mon, Nov 3, 2014 at 3:56 PM, Matthew Hall 
> wrote:
>
>> Thanks Jorg, unfortunately when I tried to install the plugin using the
>> plugin manager I get an error:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org.elasticsearch.plugins.PluginManager
>>at gnu.java.lang.MainThread.run(libgcj.so.7rh)
>> Caused by: java.lang.ClassNotFoundException:
>> org.elasticsearch.plugins.PluginManager not found in
>> gnu.gcj.runtime.SystemClassLoader{urls=[],
>> parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
>>at java.net.URLClassLoader.findClass(libgcj.so.7rh)
>>at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
>>at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
>>at gnu.java.lang.MainThread.run(libgcj.so.7rh)
>>
>> I have also tried installing the plugin by following the instructions
>>  from
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
>>
>> "Registering them can either be done by settings, for example:
>> script.native.my.type set to sample.MyNativeScriptFactory will register
>> a script named my. Another option is in a plugin, access ScriptModule and
>> call registerScript on it."
>>
>> if I register the script using the settings file (elasticsearch.yml)  I
>> get the error (as before)
>>
>>  Initialization Failed ...
>> 1) No implementation for
>> java.util.Set was bound.2)
>> NoClassSettingsException[Failed to load class setting [type] with value
>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>
>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>
>> I also tried the steps here as the problems seem identical
>>
>> *https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
>> <https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J>*
>>
>> but with no luck :(
>>
>> Thanks
>>
>> Matt
>>
>> On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> Yes, I understood native scripts. They should go into a plugin. Quoting
>>> the docs:
>>>
>>> "Plugins are a way to enhance the basic elasticsearch functionality in a
>>> custom manner. They range from adding custom mapping types, custom
>>> analyzers (in a more built in fashion), native scripts, custom discovery
>>> and more."
>>>
>>> Jörg
>>>
>>> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
>>> wrote:
>>>
>>>> Hi thanks for the reply! I actually made a mistake in my post :/. I
>>>> meant native scripts rather than plugins.  I tried adding the class-path to
>>>> Manifest.mf before but this did not work.  Thanks for your suggestions
>>>> anyway and sorry about my typo!
>>>>
>>>> Matt
>>>>
>>>> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
>>>> joergpra...@gmail.com> wrote:
>>>>
>>>>> In Java, you have to add "Class-Path" to MANIFEST.MF or the class
>>>>> loader will get confused about dependencies. It depends on the class path
>>>>> when you start Elastisearch.
>>>>>
>>>>> Also, in ES, simply adding extra jars is not the preferred method for
>>>>> extensions.
>>>>>
>>>>> It is supposed that you put a plugin (a zip archive with jars) into
>>>>> the plugins folder and use es-plugin.properties instead of fiddling with
>>>>> jars in the lib folder. These j

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

[Matthew Hall]

Thanks Jorg, unfortunately when I tried to install the plugin using the
plugin manager I get an error:

Exception in thread "main" java.lang.NoClassDefFoundError:
org.elasticsearch.plugins.PluginManager
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.plugins.PluginManager not found in
gnu.gcj.runtime.SystemClassLoader{urls=[],
parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)

I have also tried installing the plugin by following the instructions
 from
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

"Registering them can either be done by settings, for example:
script.native.my.type set to sample.MyNativeScriptFactory will register a
script named my. Another option is in a plugin, access ScriptModule and
call registerScript on it."

if I register the script using the settings file (elasticsearch.yml)  I get
the error (as before)

 Initialization Failed ...
1) No implementation for
java.util.Set was bound.2)
NoClassSettingsException[Failed to load class setting [type] with value
[com.elasticsearch.locationscript.CustomScriptFactory]]

ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]

I also tried the steps here as the problems seem identical

*https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
<https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J>*

but with no luck :(

Thanks

Matt

On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com  wrote:

> Yes, I understood native scripts. They should go into a plugin. Quoting
> the docs:
>
> "Plugins are a way to enhance the basic elasticsearch functionality in a
> custom manner. They range from adding custom mapping types, custom
> analyzers (in a more built in fashion), native scripts, custom discovery
> and more."
>
> Jörg
>
> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
> wrote:
>
>> Hi thanks for the reply! I actually made a mistake in my post :/. I meant
>> native scripts rather than plugins.  I tried adding the class-path to
>> Manifest.mf before but this did not work.  Thanks for your suggestions
>> anyway and sorry about my typo!
>>
>> Matt
>>
>> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
>>> will get confused about dependencies. It depends on the class path when you
>>> start Elastisearch.
>>>
>>> Also, in ES, simply adding extra jars is not the preferred method for
>>> extensions.
>>>
>>> It is supposed that you put a plugin (a zip archive with jars) into the
>>> plugins folder and use es-plugin.properties instead of fiddling with jars
>>> in the lib folder. These jars are treated specially at start up time, and
>>> do not need "Class-Path" in MANIFEST.MF.
>>>
>>> See also:
>>>
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>>>
>>> Jörg
>>>
>>> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
>>> wrote:
>>>
>>>> Hi sorry but I am new to java and elasticsearch.  I have written a
>>>> simple native script in intellij which compiles into classes fine.  I then
>>>> use the command line jar command to create the jar from the class files.  A
>>>> MANIFEST.MF file is generated but is always empty (apart from version and
>>>> created by).  When I put the jar into the elastic search lib directory and
>>>> add the following lines to the elasticsearch.yml
>>>>
>>>> script.native:
>>>>   MyNativeScript.type:
>>>> com.elasticsearch.locationscript.CustomScriptFactory
>>>>
>>>> Unfortunately I get the following errors
>>>>
>>>>  Initialization Failed ...
>>>> 1) No implementation for
>>>> java.util.Set was bound.2)
>>>> NoClassSettingsException[Failed to load class setting [type] with value
>>>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>>>
>>>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>>>
>>>>
>>>> I have tried to compile a simple helloworld jar and I found this did
>>>> n

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

[Matthew Hall]

Thanks Jorge, unfortunately when I tried to install the plugin using the
plugin manager I get an error:

Exception in thread "main" java.lang.NoClassDefFoundError:
org.elasticsearch.plugins.PluginManager
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.plugins.PluginManager not found in
gnu.gcj.runtime.SystemClassLoader{urls=[],
parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at java.lang.ClassLoader.loadClass(libgcj.so.7rh)
   at gnu.java.lang.MainThread.run(libgcj.so.7rh)

I have also tried installing the plugin by following the instructions
 from
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

"Registering them can either be done by settings, for example:
script.native.my.type set to sample.MyNativeScriptFactory will register a
script named my. Another option is in a plugin, access ScriptModule and
call registerScript on it."

if I register the script using the settings file (elasticsearch.yml)  I get
the error (as before)

 Initialization Failed ...
1) No implementation for
java.util.Set was bound.2)
NoClassSettingsException[Failed to load class setting [type] with value
[com.elasticsearch.locationscript.CustomScriptFactory]]

ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]

I also tried the steps here as the problems seem identical

*https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J
<https://groups.google.com/a/elasticsearch.com/forum/#!msg/users/BcaI0z-LAtU/bfFIjsHP4Q8J>*

but with no luck :(

Thanks

Matt



On Mon, Nov 3, 2014 at 9:18 AM, joergpra...@gmail.com  wrote:

> Yes, I understood native scripts. They should go into a plugin. Quoting
> the docs:
>
> "Plugins are a way to enhance the basic elasticsearch functionality in a
> custom manner. They range from adding custom mapping types, custom
> analyzers (in a more built in fashion), native scripts, custom discovery
> and more."
>
> Jörg
>
> On Mon, Nov 3, 2014 at 10:12 AM, Matthew Hall 
> wrote:
>
>> Hi thanks for the reply! I actually made a mistake in my post :/. I meant
>> native scripts rather than plugins.  I tried adding the class-path to
>> Manifest.mf before but this did not work.  Thanks for your suggestions
>> anyway and sorry about my typo!
>>
>> Matt
>>
>> On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
>>> will get confused about dependencies. It depends on the class path when you
>>> start Elastisearch.
>>>
>>> Also, in ES, simply adding extra jars is not the preferred method for
>>> extensions.
>>>
>>> It is supposed that you put a plugin (a zip archive with jars) into the
>>> plugins folder and use es-plugin.properties instead of fiddling with jars
>>> in the lib folder. These jars are treated specially at start up time, and
>>> do not need "Class-Path" in MANIFEST.MF.
>>>
>>> See also:
>>>
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>>>
>>> Jörg
>>>
>>> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
>>> wrote:
>>>
>>>> Hi sorry but I am new to java and elasticsearch.  I have written a
>>>> simple native script in intellij which compiles into classes fine.  I then
>>>> use the command line jar command to create the jar from the class files.  A
>>>> MANIFEST.MF file is generated but is always empty (apart from version and
>>>> created by).  When I put the jar into the elastic search lib directory and
>>>> add the following lines to the elasticsearch.yml
>>>>
>>>> script.native:
>>>>   MyNativeScript.type:
>>>> com.elasticsearch.locationscript.CustomScriptFactory
>>>>
>>>> Unfortunately I get the following errors
>>>>
>>>>  Initialization Failed ...
>>>> 1) No implementation for
>>>> java.util.Set was bound.2)
>>>> NoClassSettingsException[Failed to load class setting [type] with value
>>>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>>>
>>>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>>>
>>>>
>>>> I have tried to compile a simple helloworld jar and I found this did
>>>> n

Re: generating correct manifest.mf file when creating jar file for native elasticsearch plugin

[Matthew Hall]

Hi thanks for the reply! I actually made a mistake in my post :/. I meant
native scripts rather than plugins.  I tried adding the class-path to
Manifest.mf before but this did not work.  Thanks for your suggestions
anyway and sorry about my typo!

Matt

On Sat, Nov 1, 2014 at 4:54 PM, joergpra...@gmail.com  wrote:

> In Java, you have to add "Class-Path" to MANIFEST.MF or the class loader
> will get confused about dependencies. It depends on the class path when you
> start Elastisearch.
>
> Also, in ES, simply adding extra jars is not the preferred method for
> extensions.
>
> It is supposed that you put a plugin (a zip archive with jars) into the
> plugins folder and use es-plugin.properties instead of fiddling with jars
> in the lib folder. These jars are treated specially at start up time, and
> do not need "Class-Path" in MANIFEST.MF.
>
> See also:
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>
> Jörg
>
> On Fri, Oct 31, 2014 at 5:08 PM, Matthew Hall 
> wrote:
>
>> Hi sorry but I am new to java and elasticsearch.  I have written a simple
>> native script in intellij which compiles into classes fine.  I then use the
>> command line jar command to create the jar from the class files.  A
>> MANIFEST.MF file is generated but is always empty (apart from version and
>> created by).  When I put the jar into the elastic search lib directory and
>> add the following lines to the elasticsearch.yml
>>
>> script.native:
>>   MyNativeScript.type:
>> com.elasticsearch.locationscript.CustomScriptFactory
>>
>> Unfortunately I get the following errors
>>
>>  Initialization Failed ...
>> 1) No implementation for
>> java.util.Set was bound.2)
>> NoClassSettingsException[Failed to load class setting [type] with value
>> [com.elasticsearch.locationscript.CustomScriptFactory]]
>>
>> ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]
>>
>>
>> I have tried to compile a simple helloworld jar and I found this did not
>> work either UNTIL I modified the auto-generated (empty) Manifest file.
>>
>> Any help or a point to a comprehensive tutorial covering how to create
>> the jar correctly would be much appreciated!!
>>
>> ttyl
>>
>> Matt
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com
>> <https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/_nQm8Y4EE9o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP6xYB5gHpN%2BauVRby9DKwAijqR_zATesK%2Bimo5JcH%2Bw%40mail.gmail.com
> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGP6xYB5gHpN%2BauVRby9DKwAijqR_zATesK%2Bimo5JcH%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOAWKWFkzCUqrdoo4iO8YqZuYscV_ex7qNzELr9H-1JruXX7Qg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

generating correct manifest.mf file when creating jar file for native elasticsearch plugin

[Matthew Hall]

Hi sorry but I am new to java and elasticsearch.  I have written a simple 
native script in intellij which compiles into classes fine.  I then use the 
command line jar command to create the jar from the class files.  A 
MANIFEST.MF file is generated but is always empty (apart from version and 
created by).  When I put the jar into the elastic search lib directory and 
add the following lines to the elasticsearch.yml

script.native:
  MyNativeScript.type: com.elasticsearch.locationscript.CustomScriptFactory

Unfortunately I get the following errors

 Initialization Failed ...
1) No implementation for 
java.util.Set was bound.2) 
NoClassSettingsException[Failed to load class setting [type] with value 
[com.elasticsearch.locationscript.CustomScriptFactory]]

ClassNotFoundException[com.elasticsearch.locationscript.CustomScriptFactory]


I have tried to compile a simple helloworld jar and I found this did not 
work either UNTIL I modified the auto-generated (empty) Manifest file.

Any help or a point to a comprehensive tutorial covering how to create the 
jar correctly would be much appreciated!!

ttyl

Matt

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/32f808bb-941f-4639-9a7d-dd57252d959e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: More efficient date sorting

[Matthew Painter]

No. But I would suggest that mapping from a long to a int would be
obviously more performant.
On 25 Sep 2014 15:16, "Bleh"  wrote:

> Did you run any experiments comparing sorting on dates vs the custom
> scoring
> suggestion you made ?
>
>
>
> --
> View this message in context:
> http://elasticsearch-users.115913.n3.nabble.com/More-efficient-date-sorting-tp4044842p4063997.html
> Sent from the ElasticSearch Users mailing list archive at Nabble.com.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/Mb9XwhA34j8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1411632317090-4063997.post%40n3.nabble.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAG_V-y6hCzX%2B8eH%3DFLA%3DVL4LV2-FLOLguZ6J60FW8OYPuUwg9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are there any facets that can be used to co-relate log events ?

[Matthew Morrison]

Hi Aditya,

I'm looking to do something similar, did you have any success with this 
problem?

Thanks
Matt

On Wednesday, January 22, 2014 11:53:36 PM UTC+13, Aditya Pavan Kumar 
Vegesna wrote:
>
>  Hi
>
> I am looking for way to co-relate multiple log events and then calculate 
> the time duration between those events?
>
> e.g: Request log event & response log event - to calculate the difference 
> in timestamps to assess the performance of the application.
>
> Can anyone help me how this can be achieved.
>
> Thanks
>
> Pavan Kumar
>

-- 

See why you should attend BroadSoft Connections 2014 


This email is intended solely for the person or entity to which it is 
addressed and may contain confidential and/or privileged information. If 
you are not the intended recipient and have received this email in error, 
please notify BroadSoft, Inc. immediately by replying to this message, and 
destroy all copies of this message, along with any attachment, prior to 
reading, distributing or copying it.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d20a2379-c915-477c-877d-690895b22773%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] ElasticUI AngularJS Directives - Easily Build an Interface on top of Elasticsearch

[Matthew Morrison]

Great idea. I'll give it a try ASAP.


On Wednesday, July 2, 2014 10:56:48 PM UTC+12, Yousef El-Dardiry wrote:
>
> Hi all,
>
> I just open sourced a set of AngularJS Directives for Elasticsearch. It 
> enables developers to rapidly build a frontend (e.g.: faceted search 
> engine) on top of Elasticsearch.
>
> http://www.elasticui.com (or github 
> )
>
> It makes creating an aggregation and listing the buckets as simple as:
>
> * eui-aggregation="ejs.TermsAggregation('text_agg').field('text').size(10)">*
> *{{bucket}}*
> **
>
> I think this was currently missing in the ecosystem, which is why I 
> decided to build and open source it. I'd love any kind of feedback.
>
> - Yousef
>
> *-*
> Another example; add a checkbox facet based on a field using one of the 
> built-in widgets 
> :
>
> **
>
> Resulting in
> [image: checklist screenshot]
>

-- 

See why you should attend BroadSoft Connections 2014 


This email is intended solely for the person or entity to which it is 
addressed and may contain confidential and/or privileged information. If 
you are not the intended recipient and have received this email in error, 
please notify BroadSoft, Inc. immediately by replying to this message, and 
destroy all copies of this message, along with any attachment, prior to 
reading, distributing or copying it.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a2ae383e-154a-4a2f-a925-20461157a827%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cross data center replication

[Matthew Parrott]

I'm interested in this too.
es-reindex seems like it lacks conflict resolution, and as noted in the 
docs, would be better implemented as a river.

On Wednesday, June 4, 2014 9:03:37 PM UTC-7, Todd Nine wrote:
>
> Hey all,
>  
> Sorry to resurrect a dead thread.  Did you ever find a solution for 
> eventual consistency of documents across EC2 regions?
>
> Thanks,
> todd
>
>
>
> On Wednesday, May 1, 2013 5:50:00 AM UTC-7, Norberto Meijome wrote:
>>
>> +1 on all of the above. es-reindex already in my list of things to 
>> investigate (for a number of issues...)
>>
>> cheers,
>> b 
>>
>>
>> On Wed, May 1, 2013 at 6:58 AM, Paul Hill  wrote:
>>
>>> On 4/23/2013 8:44 AM, Daniel Maher wrote:
>>>
 On 2013-04-23 5:22 PM, Saikat Kanjilal wrote:

> Hello Folks,
> [...] does ES out of the box currently support cross data
> center replication,  []
>

 Hello,

 I'd wager that the question you're really asking about is how to 
 control where shards are placed; if you can make deterministic statements 
 about where shards are, then you can create your own "rack-aware" or "data 
 centre-aware" scenarios.  ES has supported this "out of the box" for well 
 over a year now (possibly longer).

 You'll want to investigate "zones" and "routing allocation", which are 
 the key elements of shard placement.  There is an excellent blog post 
 which 
 describes exactly how to set things up here :
 http://blog.sematext.com/2012/05/29/elasticsearch-shard-
 placement-control/ 

  Is shard allocation really the correct solution if the data centers 
>>> are globally distributed?
>>>
>>> If I have a data center in the US intended to server data from the US, 
>>> but it should also have access to Europe and Asia data, and clusters in 
>>> both Europe and Asia with similar needs, would I really want to use zones 
>>> etc. and have one great global cluster with data center aware 
>>> configurations?
>>>
>>> Assuming that the US would be happy to deal with old documents from Asia 
>>> and Europe, when Asia or Europe is off line or just not caught up, it would 
>>> seem that you would NOT want a "world" cluster, because I can't picture how 
>>> you'd configure a 3-part world cluster for both index into the right 
>>> indices, search the right (possible combination of) shards, but also 
>>> preventing "split brain".
>>>
>>> In the scenerio, I've described, I would think each data center might 
>>> better provide availability and eventual consistency (with less concern for 
>>> the remote data from the other region) by having three clusters and some 
>>> type of syncing from one index to copies at the other two locations.  For 
>>> example, the US datacenter might have a US, copyOfEurope, and copyOfAsia 
>>> index.
>>>
>>> Anyone have any observations about such a world-wide scenerio?
>>> Are there any index to index copy utilities?
>>> Is there a river or other plugin that might be useful for this three 
>>> clusters working together scenerio?
>>> How about the project https://github.com/karussell/elasticsearch-reindex
>>> ?
>>> Comments?
>>>
>>> -Paul
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>>
>> -- 
>> Norberto 'Beto' Meijome
>>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/86f03167-6803-4bdd-9278-21b222e56d7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Notifications from Elasticsearch when documents are added.

[Matthew Parrott]

I found this note:
http://www.elasticsearch.org/blog/1-0-0-beta2-released/

Which mentions:
"Later on we plan on making cross data-center replication possible by 
adding the ability to do incremental restores into a read-only index."

Is that feature still on the roadmap?

Thanks

On Friday, June 27, 2014 10:40:55 AM UTC-7, Matthew Parrott wrote:
>
> Hey!
>
> I have looked at tribes, but didn't look deeply because of this:
>
> "The merged view cannot handle indices with the same name in multiple 
> clusters."
>
> I'd like to have indexes replicated across datacenters. Is there a way to 
> accomplish that with tribes?
>
> Thanks!
>
> On Friday, June 27, 2014 2:29:46 AM UTC-7, Jörg Prante wrote:
>>
>> Have you seen the Tribe Node? This is a kind of a "merged state" 
>> multi-master cluster. 
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html
>>
>> Jörg
>>
>>
>> On Fri, Jun 27, 2014 at 1:39 AM, Matthew Parrott  
>> wrote:
>>
>>> Hi!
>>>
>>> Have there been any further explorations in the area of wan replication?
>>>
>>> I have ES clusters in multiple datacenters connected via high-speed 
>>> private network. I'm wondering if multi-master replication would be 
>>> possible in this environment or if we'd need some type of 'shovel' plugin 
>>> like the one described here to ship data between the DCs.
>>>
>>> Thanks,
>>> Matthew
>>>
>>>
>>> On Tuesday, July 23, 2013 10:06:10 AM UTC-7, Jörg Prante wrote:
>>>
>>>> Yes, I once examined Kafka, and discovered that many components are 
>>>> already there in Elasticsearch. For example, the activity stream is 
>>>> already 
>>>> there as ES translog (if you focus on indexing operations) and the ES 
>>>> gateway is a useful persistency store mechanism. What I didn't like was 
>>>> the 
>>>> single Kafka JVM, and the Zookeeper infrastructure, it is all adding up 
>>>> complexity beside ES.
>>>>
>>>> For cross-cluster replication, I think the best approach is distributed 
>>>> log replication. This is hard, because logged ES operations must be 
>>>> synchronized by an external time source (e.g. vector clocks) to use them 
>>>> like a global event stream. A pubsub mechanism could then work at the 
>>>> primary shards of an index in the ES node as a service, merging the 
>>>> translogs for an external agent who previously subscribed to the 
>>>> replication stream. The vector clock is required for a distributed time 
>>>> machine like behavior (snapshots), assuming the translog is not deleted, 
>>>> but stored for a certain time window.
>>>>
>>>> Jörg
>>>>
>>>>
>>>> On Tue, Jul 23, 2013 at 3:55 PM, Vinicius Carvalho <
>>>> vinicius...@gmail.com> wrote:
>>>>
>>>>> Thanks again Jorg, so that you know I'm actually considering using 
>>>>> kafka for intra cluster replication. We want to push the index operations 
>>>>> to a topic and then other clusters on different DCs would subscribe to 
>>>>> this. Conflict resolution will be last commit will win. And in case of 
>>>>> kafka cluster failure we will append changes to a local index, and then 
>>>>> send them over as the bus is back. In the case ES cluster dies, and when 
>>>>> it 
>>>>> recovers, one nice thing on kafka is that one can request messages based 
>>>>> on 
>>>>> an offset, so we could start consuming messages from the last point the 
>>>>> cluster had consume them.
>>>>>
>>>>> It's all ideas I'm working right now. I'll probably have time to start 
>>>>> coding them soon. Thanks for all the support :)
>>>>>
>>>>> Cheers
>>>>>
>>>>>   -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/37b1c902-a74d-4c35-bc41-5e1d5e76e72d%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/elasticsearch/37b1c902-a74d-4c35-bc41-5e1d5e76e72d%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c0caf8a9-98a3-4e00-aa7b-abec5c98a542%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Notifications from Elasticsearch when documents are added.

[Matthew Parrott]

Hey!

I have looked at tribes, but didn't look deeply because of this:

"The merged view cannot handle indices with the same name in multiple 
clusters."

I'd like to have indexes replicated across datacenters. Is there a way to 
accomplish that with tribes?

Thanks!

On Friday, June 27, 2014 2:29:46 AM UTC-7, Jörg Prante wrote:
>
> Have you seen the Tribe Node? This is a kind of a "merged state" 
> multi-master cluster. 
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-tribe.html
>
> Jörg
>
>
> On Fri, Jun 27, 2014 at 1:39 AM, Matthew Parrott  > wrote:
>
>> Hi!
>>
>> Have there been any further explorations in the area of wan replication?
>>
>> I have ES clusters in multiple datacenters connected via high-speed 
>> private network. I'm wondering if multi-master replication would be 
>> possible in this environment or if we'd need some type of 'shovel' plugin 
>> like the one described here to ship data between the DCs.
>>
>> Thanks,
>> Matthew
>>
>>
>> On Tuesday, July 23, 2013 10:06:10 AM UTC-7, Jörg Prante wrote:
>>
>>> Yes, I once examined Kafka, and discovered that many components are 
>>> already there in Elasticsearch. For example, the activity stream is already 
>>> there as ES translog (if you focus on indexing operations) and the ES 
>>> gateway is a useful persistency store mechanism. What I didn't like was the 
>>> single Kafka JVM, and the Zookeeper infrastructure, it is all adding up 
>>> complexity beside ES.
>>>
>>> For cross-cluster replication, I think the best approach is distributed 
>>> log replication. This is hard, because logged ES operations must be 
>>> synchronized by an external time source (e.g. vector clocks) to use them 
>>> like a global event stream. A pubsub mechanism could then work at the 
>>> primary shards of an index in the ES node as a service, merging the 
>>> translogs for an external agent who previously subscribed to the 
>>> replication stream. The vector clock is required for a distributed time 
>>> machine like behavior (snapshots), assuming the translog is not deleted, 
>>> but stored for a certain time window.
>>>
>>> Jörg
>>>
>>>
>>> On Tue, Jul 23, 2013 at 3:55 PM, Vinicius Carvalho <
>>> vinicius...@gmail.com> wrote:
>>>
>>>> Thanks again Jorg, so that you know I'm actually considering using 
>>>> kafka for intra cluster replication. We want to push the index operations 
>>>> to a topic and then other clusters on different DCs would subscribe to 
>>>> this. Conflict resolution will be last commit will win. And in case of 
>>>> kafka cluster failure we will append changes to a local index, and then 
>>>> send them over as the bus is back. In the case ES cluster dies, and when 
>>>> it 
>>>> recovers, one nice thing on kafka is that one can request messages based 
>>>> on 
>>>> an offset, so we could start consuming messages from the last point the 
>>>> cluster had consume them.
>>>>
>>>> It's all ideas I'm working right now. I'll probably have time to start 
>>>> coding them soon. Thanks for all the support :)
>>>>
>>>> Cheers
>>>>
>>>>   -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/37b1c902-a74d-4c35-bc41-5e1d5e76e72d%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/37b1c902-a74d-4c35-bc41-5e1d5e76e72d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/28808d58-62c2-433e-b932-c93d824f0a97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Notifications from Elasticsearch when documents are added.

[Matthew Parrott]

Hi!

Have there been any further explorations in the area of wan replication?

I have ES clusters in multiple datacenters connected via high-speed private 
network. I'm wondering if multi-master replication would be possible in 
this environment or if we'd need some type of 'shovel' plugin like the one 
described here to ship data between the DCs.

Thanks,
Matthew

On Tuesday, July 23, 2013 10:06:10 AM UTC-7, Jörg Prante wrote:
>
> Yes, I once examined Kafka, and discovered that many components are 
> already there in Elasticsearch. For example, the activity stream is already 
> there as ES translog (if you focus on indexing operations) and the ES 
> gateway is a useful persistency store mechanism. What I didn't like was the 
> single Kafka JVM, and the Zookeeper infrastructure, it is all adding up 
> complexity beside ES.
>
> For cross-cluster replication, I think the best approach is distributed 
> log replication. This is hard, because logged ES operations must be 
> synchronized by an external time source (e.g. vector clocks) to use them 
> like a global event stream. A pubsub mechanism could then work at the 
> primary shards of an index in the ES node as a service, merging the 
> translogs for an external agent who previously subscribed to the 
> replication stream. The vector clock is required for a distributed time 
> machine like behavior (snapshots), assuming the translog is not deleted, 
> but stored for a certain time window.
>
> Jörg
>
> On Tue, Jul 23, 2013 at 3:55 PM, Vinicius Carvalho  > wrote:
>
>> Thanks again Jorg, so that you know I'm actually considering using kafka 
>> for intra cluster replication. We want to push the index operations to a 
>> topic and then other clusters on different DCs would subscribe to this. 
>> Conflict resolution will be last commit will win. And in case of kafka 
>> cluster failure we will append changes to a local index, and then send them 
>> over as the bus is back. In the case ES cluster dies, and when it recovers, 
>> one nice thing on kafka is that one can request messages based on an 
>> offset, so we could start consuming messages from the last point the 
>> cluster had consume them.
>>
>> It's all ideas I'm working right now. I'll probably have time to start 
>> coding them soon. Thanks for all the support :)
>>
>> Cheers
>>
>> 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/37b1c902-a74d-4c35-bc41-5e1d5e76e72d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can you define time-to-live-seconds on a queue in 3.x?

[Matthew Painter]

I know this was possible in 2.x using backing map, but what about in 3.x? 

It seems to have disappeared from the Config :(

Thanks

Matt

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2f1d6af1-4d31-4819-b13c-fcc5dba9fb28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Problems with span_not query

[Matthew Brown]

Having some problems with queries containing span_not. I've simplified the 
query down to a test example however the query returns additional documents 
I don't think you be returned.

https://gist.github.com/m-brown/59b9b5ad6f68a5d12d0a

In short I want to find the documents that contain 'foo' but not 'bar' from:
foo
foo bar
bar foo
foo foo bar
foo bar foo
bar foo foo

The below query returns two docs ('foo' and 'bar foo foo') rather than the 
one I was expecting:
{
  "query": {
"span_not": {
  "include": {
"span_term": {
  "field1": "foo"
}
  },
  "exclude": {
"span_near": {
  "in_order": false,
  "clauses": [
{
  "span_term": {
"field1": "bar"
  }
},
{
  "span_term": {
"field1": "foo"
  }
}
  ],
  "slop": 1000
}
  }
}
  }
}

Why does 'bar foo foo' match the query, and given that it does, why don't 
any of the others given in_order is false?

Tested on elasticsearch 1.0.1 and 1.1.1 on Ububtu 12.04.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/711fe54f-436c-453e-8a9d-59ff73c57c67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

multimaster over wan

[Matthew Parrott]

Hi,

I'm wondering if ES supports a multi-master configuration over WAN. Let's 
say I have node A and node B and they're in different parts of the world. 
Can I write to the instance closest to my location and have the data 
automatically replicated to the other?

Reading through the cluster configuration docs, it wasn't quite clear if 
this was possible.

Thanks,
Matthew

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d62d143e-5405-4f6f-8a3e-722f473832ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

rangeFilter execution in Java API for 1.0

[Matthew Kehrt]

Hi,

I'm investigating upgrading an ES installation to 1.0 and I see that 
numericRangeFilter is deprecated, with the recommendation that rangeFilter 
with "execution": "fielddata" be used instead. However, in the Java API, 
RangeFilterBuilder has no "execution" method, or anything similar, and 
neither do any of its superclasses.  In fact, code containing the 
expression "FilterBuilders.rangeFilter..execution("fielddata")" will not compile for exactly this reason.

I see that TermsFilterBuilder *does* have an "execution" method, at least 
according the docs.  Should there be one in RangeFilterBuilder as well?  Or 
am I missing something?

Thanks,
Matt

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8056961b-0514-4190-ba7a-667b3ff63c26%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Querying unique pageviews in log data

[Matthew Boynes]

Hi Ivan, thanks for taking the time to respond. After reading up on this, I 
believe you're correct: field collapsing would give me exactly what I want. 
I also started reading about aggregations, and *perhaps* that will work as 
well -- it seems like I could create a bucket for each uid and then count 
the number of buckets. Since the docs are pretty scarce on aggregations 
thus far, it's hard to say. When I have some free time, I'll check out the 
1.0 beta and see if I can come up with something. Thanks again!

Matt


On Saturday, January 11, 2014 3:28:53 PM UTC-5, Ivan Brusic wrote:
>
> Sounds like what you are looking for is field collapsing which is not yet 
> supported in elasticsearch. ETA is post 1.0 release. Perhaps there is a way 
> with the new aggregations framework, but I have yet to try it out.
>
> Cheers,
>
> Ivan
>
>
> On Fri, Jan 10, 2014 at 3:13 PM, Matthew Boynes <
> mbo...@alleyinteractive.com > wrote:
>
>> Hey Brian,
>> Thanks for taking the time to respond. I looked at the post you suggested 
>> and I don't think that would give me what I need. I believe that what 
>> you're suggesting would provide me with the most views of a single url by a 
>> unique visitor. In other words, if a url has 1004 entries, where one person 
>> viewed it a thousand times and 4 people each viewed it once, the crazy 
>> person who kept refreshing would be at the top of the facet response. I 
>> need to know that the url had 5 unique visitors. If there was only one url 
>> in the index this would work in a roundabout way, because I could look at 
>> the total number of terms returned in the facet. Unfortunately, that's just 
>> not the case here.
>>
>> Of course, if I'm misunderstanding what your post suggests, or if I've 
>> missed something, please let me know!
>>
>> Thanks,
>> Matt
>>
>>
>>
>> On Friday, January 10, 2014 5:54:46 PM UTC-5, InquiringMind wrote:
>>>
>>> Matthew,
>>>
>>> I don't know if this is simple (though it was easy enough for me in 
>>> Java), or even if it's exactly what you had in mind. But it sounds as if 
>>> you are asking for a hierarchical combination to include the top URLs by 
>>> uid. Is that correct?
>>>
>>> If so, perhaps 
>>> this<https://groups.google.com/d/msg/elasticsearch/_oMbAnpjSGg/II4Tzf6RoSwJ>will
>>>  give you some ideas.
>>>
>>> Hope this helps! Good luck!
>>>
>>> Brian
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/3e7ef115-1b27-4e67-b6dd-185b84f9c76d%40googlegroups.com
>> .
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f6ab78c6-3146-4106-b273-3b92b3ab5293%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Querying unique pageviews in log data

[Matthew Boynes]

Hey Brian,
Thanks for taking the time to respond. I looked at the post you suggested 
and I don't think that would give me what I need. I believe that what 
you're suggesting would provide me with the most views of a single url by a 
unique visitor. In other words, if a url has 1004 entries, where one person 
viewed it a thousand times and 4 people each viewed it once, the crazy 
person who kept refreshing would be at the top of the facet response. I 
need to know that the url had 5 unique visitors. If there was only one url 
in the index this would work in a roundabout way, because I could look at 
the total number of terms returned in the facet. Unfortunately, that's just 
not the case here.

Of course, if I'm misunderstanding what your post suggests, or if I've 
missed something, please let me know!

Thanks,
Matt


On Friday, January 10, 2014 5:54:46 PM UTC-5, InquiringMind wrote:
>
> Matthew,
>
> I don't know if this is simple (though it was easy enough for me in Java), 
> or even if it's exactly what you had in mind. But it sounds as if you are 
> asking for a hierarchical combination to include the top URLs by uid. Is 
> that correct?
>
> If so, perhaps 
> this<https://groups.google.com/d/msg/elasticsearch/_oMbAnpjSGg/II4Tzf6RoSwJ>will
>  give you some ideas.
>
> Hope this helps! Good luck!
>
> Brian
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3e7ef115-1b27-4e67-b6dd-185b84f9c76d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Querying unique pageviews in log data

[Matthew Boynes]

Let's say that I'm using Elasticsearch to host and query log data, and I 
want to do some basic analysis of it like I would in Google Analytics. One 
of the most basic differentiations I'm having trouble with is pageviews vs. 
unique pageviews. I'm sure there's a very simple solution, but for some 
reason it's escaping me.

Here's a sample mapping:

 {
  "log-data": {
"properties": {
  "date": {
"type": "date",
"format": "dateOptionalTime"
  },
  "uid": {
"type": "string"
  },
  "url": {
"type": "string"
  }
}
  }
 }


"date" and "url" are pretty obvious; "uid" is a unique identifier.

If I want to get the total pageviews for the top 10 URLs for the past week, 
I can query it like so:

 {
  "query": {
"range": {
  "date": {
"gte" : "now-1w"
  }
}
  },
  "facets": {
"pageview": {
  "terms": {
"field": "url",
"size": 10
  }
}
  },
  "size": 0
 }


What I'm struggling with is bringing the "uid" into the query to get unique 
pageviews. Any help would be greatly appreciated!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/215897fd-4639-4b90-8308-80318fd2a283%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.