Re: dangling index info messages in logs
The problem here is that the new master was not synchronized with an old master. So he started with an empty cluster state. To do that kind of operation, you have to bring your new master nodes up in a running cluster so they will be synchronized, then stop the cluster, change data nodes settings and restart. You should test on a dev machine what would happen if you set again old node to be master again. I think it won't work and will still try to remove your old data. If you have a backup somewhere of your cluster state, that would probably be useful. My 2 cents. -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 2 mars 2014 à 06:42, hrishikesh prabhune a écrit : > Hello all, > > I have an elasticsearch cluster of 16 nodes , all of which are eligible > master and data nodes. I had to add some load balancing nodes to the cluster > so i added 6 extra nodes to the cluster . These newly added nodes are all > eligible master nodes only(not eligible data nodes). I restarted the whole > cluster by making the original 16 nodes to be eligible data nodes only and > the newly added 6 nodes were kept as eligible master nodes only. > During the first restart after changing the whole configuration of cluster I > started getting these INFO messages in the logs : > [2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] > [%user%index] dangling index, exists on local file system, but not in cluster > metadata, scheduling to delete in [2h], auto import to cluster state [YES] > > I thought the indices were about to be deleted so I immediately shutdown the > cluster and reverted back to the original configuration. > Can anyone shed a light on these info messages? According to my > understanding, only the cluster meta data of theses indices were about to > changed or deleted but not the data in the index. It would be great if anyone > could give a quick explanation about why I am getting these messages. > Thanks in advance!! > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/fa169794-6d3c-48a9-b98c-69d4c60aae58%40googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48B542DA-6856-4D57-8693-FC680225BF17%40pilato.fr. For more options, visit https://groups.google.com/groups/opt_out.
"Illegal character in path"
Just had this error (elasticsearch 1.0.1), don't recall seeing it before: java.net.URISyntaxException: Illegal character in path at index 29: /cache/11_/seg0/index12450141[12450141].ts Should I be concerned? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/34bb0d6a-598a-4ae7-9fa1-200e51da92a2%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Get list of.concret indexes behind aliases in elasticsear 1.X+
Hey maybe a look at RestGetIndicesAliasesAction can help you... --Alex On Fri, Feb 28, 2014 at 3:52 PM, Daniel Myers wrote: > I had this working on 0.90.7 but since upgrading to 1.0.1 I havent been > able to reachieve that functionailty. > > It seems something like > clientgetMetaData().aliases().get(aliasName).? > > Anyone have a suggestion? Been stuck on this for a couple hours... > > Cheers, > Dan > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/823ab197-5c29-4c4f-aaa5-3969eddae81f%40googlegroups.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM88Jkg%2BG1%2BNhCTWbeNK9SFvqV4%2BrA42gXrB3v9tW8p3Pw%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
dangling index info messages in logs
Hello all, I have an elasticsearch cluster of 16 nodes , all of which are eligible master and data nodes. I had to add some load balancing nodes to the cluster so i added 6 extra nodes to the cluster . These newly added nodes are all eligible master nodes only(not eligible data nodes). I restarted the whole cluster by making the original 16 nodes to be eligible data nodes only and the newly added 6 nodes were kept as eligible master nodes only. During the first restart after changing the whole configuration of cluster I started getting these INFO messages in the logs : [2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user% index] dangling index, exists on local file system, but not in cluster metadata, scheduling to delete in [2h], auto import to cluster state [YES] I thought the indices were about to be deleted so I immediately shutdown the cluster and reverted back to the original configuration. Can anyone shed a light on these info messages? According to my understanding, only the cluster meta data of theses indices were about to changed or deleted but not the data in the index. It would be great if anyone could give a quick explanation about why I am getting these messages. Thanks in advance!! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa169794-6d3c-48a9-b98c-69d4c60aae58%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: dangling index info messages in logs
Oh I forgot to mention... I am using Elasticsearch version 1.0.1 Thanks! On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote: > > Hello all, > > I have an elasticsearch cluster of 16 nodes , all of which are eligible > master and data nodes. I had to add some load balancing nodes to the > cluster so i added 6 extra nodes to the cluster . These newly added nodes > are all eligible master nodes only(not eligible data nodes). I restarted > the whole cluster by making the original 16 nodes to be eligible data nodes > only and the newly added 6 nodes were kept as eligible master nodes only. > During the first restart after changing the whole configuration of cluster > I started getting these INFO messages in the logs : > [2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user > %index] dangling index, exists on local file system, but not in cluster > metadata, scheduling to delete in [2h], auto import to cluster state [YES] > > I thought the indices were about to be deleted so I immediately shutdown > the cluster and reverted back to the original configuration. > Can anyone shed a light on these info messages? According to my > understanding, only the cluster meta data of theses indices were about to > changed or deleted but not the data in the index. It would be great if > anyone could give a quick explanation about why I am getting these messages. > Thanks in advance!! > On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote: > > Hello all, > > I have an elasticsearch cluster of 16 nodes , all of which are eligible > master and data nodes. I had to add some load balancing nodes to the > cluster so i added 6 extra nodes to the cluster . These newly added nodes > are all eligible master nodes only(not eligible data nodes). I restarted > the whole cluster by making the original 16 nodes to be eligible data nodes > only and the newly added 6 nodes were kept as eligible master nodes only. > During the first restart after changing the whole configuration of cluster > I started getting these INFO messages in the logs : > [2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user > %index] dangling index, exists on local file system, but not in cluster > metadata, scheduling to delete in [2h], auto import to cluster state [YES] > > I thought the indices were about to be deleted so I immediately shutdown > the cluster and reverted back to the original configuration. > Can anyone shed a light on these info messages? According to my > understanding, only the cluster meta data of theses indices were about to > changed or deleted but not the data in the index. It would be great if > anyone could give a quick explanation about why I am getting these messages. > Thanks in advance!! > On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote: > > Hello all, > > I have an elasticsearch cluster of 16 nodes , all of which are eligible > master and data nodes. I had to add some load balancing nodes to the > cluster so i added 6 extra nodes to the cluster . These newly added nodes > are all eligible master nodes only(not eligible data nodes). I restarted > the whole cluster by making the original 16 nodes to be eligible data nodes > only and the newly added 6 nodes were kept as eligible master nodes only. > During the first restart after changing the whole configuration of cluster > I started getting these INFO messages in the logs : > [2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user > %index] dangling index, exists on local file system, but not in cluster > metadata, scheduling to delete in [2h], auto import to cluster state [YES] > > I thought the indices were about to be deleted so I immediately shutdown > the cluster and reverted back to the original configuration. > Can anyone shed a light on these info messages? According to my > understanding, only the cluster meta data of theses indices were about to > changed or deleted but not the data in the index. It would be great if > anyone could give a quick explanation about why I am getting these messages. > Thanks in advance!! > On Saturday, March 1, 2014 9:42:04 PM UTC-8, hrishikesh prabhune wrote: > > Hello all, > > I have an elasticsearch cluster of 16 nodes , all of which are eligible > master and data nodes. I had to add some load balancing nodes to the > cluster so i added 6 extra nodes to the cluster . These newly added nodes > are all eligible master nodes only(not eligible data nodes). I restarted > the whole cluster by making the original 16 nodes to be eligible data nodes > only and the newly added 6 nodes were kept as eligible master nodes only. > During the first restart after changing the whole configuration of cluster > I started getting these INFO messages in the logs : > [2014-03-02 04:33:57,857][INFO ][gateway.local.state.meta ] [node1] [%user > %index] dangling index, exists on local file system, but not in cl
failed to send join request to master
Hey guys, I'm currently working with an ES (0.90.10) cluster of six nodes, hosted on EC2. It worked fine for most of time, but sometimes nodes are missing from cluster. I restarted the node missing from cluster, and got following log messages, which suggest the node couldn't connect to the master. I tried to telnet the master on port 9300, 9200, and 9500, all worked fine. Any idea what's going on? Or how can I better troubleshoot this problem? Thanks in advance!!! [2014-03-01 17:37:04,377][DEBUG][discovery.ec2] [prod-insights-es-3] using dynamic discovery nodes [[#cloud-i-4e0e9f60-0][inet[/10.238.151.xxx:9300]], [#cloud-i-846afaaa-0][inet[/10.4.159.xxx:9300]], [#cloud-i-f2fa69dc-0][inet[/10.225.21.xxx:9300]], [#cloud-i-db841ff5-0][inet[/10.138.38.xxx:9300]], [#cloud-i-76b82358-0][inet[/10.141.169.xxx:9300]], [#cloud-i-2eb12a00-0][inet[/10.225.31.xxx:9300]]] [2014-03-01 17:37:04,390][DEBUG][discovery.ec2] [prod-insights-es-3] filtered ping responses: (filter_client[true], filter_data[false]) --> target [[prod-insights-es-4][VxPFIFgTRQKjKE6UdV8r5w][inet[/10.141.169.xxx:9300]]], master [[prod-insights-es-5][1L19uMCpRgy6lTLU7l30fg][inet[/10.225.31.xxx:9300]]] --> target [[prod-insights-es-0][MtJ8gh1ASwyXduy9IDzZXg][inet[/10.238.151.xxx:9300]]], master [null] --> target [[prod-insights-es-2][lQu3NC_QTuybRYb6JV0saA][inet[/10.225.21.xxx:9300]]], master [[prod-insights-es-5][1L19uMCpRgy6lTLU7l30fg][inet[/10.225.31.xxx:9300]]] --> target [[prod-insights-es-1][RkEb00GxTY-4pk481I0hNg][inet[/10.4.159.xxx:9300]]], master [[prod-insights-es-5][1L19uMCpRgy6lTLU7l30fg][inet[/10.225.31.xxx:9300]]] --> target [[prod-insights-es-5][1L19uMCpRgy6lTLU7l30fg][inet[/10.225.31.xxx:9300]]], master [[prod-insights-es-5][1L19uMCpRgy6lTLU7l30fg][inet[/10.225.31.xxx:9300]]] [2014-03-01 17:37:19,391][INFO ][discovery.ec2] [prod-insights-es-3] failed to send join request to master [[prod-insights-es-5][1L19uMCpRgy6lTLU7l30fg][inet[/10.225.31.xxx:9300]]], reason [org.elasticsearch.ElasticSearchTimeoutException: Timeout waiting for task.] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d36c6de8-0634-4978-a283-1868b8211f33%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Custom Aggregations
Hi, I'm trying to create a new aggregator plugin. I was able to understand the structure needed to create and register my new aggregator but I don't know who to get the document field values inside collect method. How can I do that? Em quarta-feira, 12 de fevereiro de 2014 11h49min28s UTC-2, Alexander Reelsen escreveu: > > Hey, > > sure, you can write a custom plugin, which implements your custom > aggregations, if you want to. > Any specific functionality you are missing (might be interesting to get it > in the core anyway?). > > > --Alex > > > On Tue, Feb 11, 2014 at 9:41 PM, Justin Uang > > wrote: > >> Is there any way we can define our own aggregation functions beyond the >> provided metric and bucket aggregations? >> >> Thanks! >> >> Justin >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/90accf78-6540-4126-8268-7f997353ad74%40googlegroups.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/15e052d8-41d6-48f8-9f29-5482bc4bf8f0%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [hadoop] Store GeoJson using PIG
Index mapping (defined before running the PIG script):
curl -XPOST localhost:9200/location -d '{
"mappings" : {
"cell" : {
"properties" : {
"mapLocation" : {"type" : "geo_point" },
"name" : {"type" : "string"}
}
}
}
}'
PIG script:
location = load '/user/root/locations/cells_location.txt' USING
PigStorage('|') as (name: chararray, latitude: double, longitude: double);
locationToStore = FOREACH location GENERATE name, TOBAG(longitude,
latitude) as mapLocation;
STORE locationToStore INTO 'location/cell' USING
org.elasticsearch.hadoop.pig.EsStorage();
For the last test I used elasticsearch-hadoop-1.3.0.M2.jar
Thanks!
Dumitru
On Thu, Feb 27, 2014 at 12:07 AM, Costin Leau wrote:
> Can you post your Pig script and your index mapping? Note that for
> non-native types (Pig doesn't support IP or geo) you should define the
> mapping in advanced in ES since es-hadoop will map it as strings as
> otherwise.
>
>
> On 2/5/2014 8:30 PM, Dumitru Pascu wrote:
>
>> Anyone tried this before?
>>
>> Regards,
>> Dumitru
>>
>> On Monday, February 3, 2014 9:49:10 PM UTC+2, Dumitru Pascu wrote:
>>
>> How can I store GeoJson using PIG?
>>
>> I tried to use TOBAG(lon, lat), however I got in the location stored
>> in the following form:
>>
>> location: [
>> {
>> 0: 23.80323889
>> }
>> {
>> 0: 44.31903611
>> }
>> ]
>>
>> Regards,
>> Dumitru
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to
>>
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/0d4c090b-
>> 8ae3-4a00-bf11-8cbf8a4b5da0%40googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
> --
> Costin
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/elasticsearch/3XWKWd-XiPk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/530E65BF.8040503%40gmail.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPUuK1YHLXOh7iP6NiCJQv64m3kva3bitr1S3OTmB1htLpgaUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
elasticsearch is giving error as "failed to start shard"
Hi , I'm trying to move elastic search on a new system, but while trying to start the service its giving below exception. Althrough the process is running. There was old elasticsearch 0.90 copied which is not running now as I checked. Please suggest. [2014-02-28 16:40:13,588][WARN ][transport.netty ] [DJ] Message not fully read (response) for [4] handler future(org.elasticsearch.indices.recovery.RecoveryTarget$4@1b4268b), error [true], resetting [2014-02-28 16:40:13,589][WARN ][transport.netty ] [DJ] Message not fully read (response) for [3] handler future(org.elasticsearch.indices.recovery.RecoveryTarget$4@151d6cb), error [true], resetting [2014-02-28 16:40:13,590][WARN ][indices.cluster ] [DJ] [logstash-2014.02.27][1] failed to start shard org.elasticsearch.indices.recovery.RecoveryFailedException: [logstash-2014.02.27][1]: Recovery failed from [Brutacus][L0THepPDR6Cmx7O2jqVLdg][cloudclient1.aricent.com][inet[/10.203.251.143:9300]] into [DJ][LajLfvlYSAGZdsyFnorwEA][cloudclient.aricent.com][inet[/10.203.251.142:9300]] at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:303) at org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65) at org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:171) Thanks, Subahdip -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1ee34fd2-27f2-45bb-82d0-f11cdf07aaf2%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
