Great, thanks for the help. I did few things and I am not sure which ended up wiping the index. I killed the cluster, I upgraded to 1.2.2, I rebooted the 2 AWS instances, I indexed on each node separately, then I joined the cluster. Things look fine now.
On Tuesday, August 12, 2014 10:46:33 AM UTC-4, Jörg Prante wrote: > > There is one node with zen unicast that can not connect. I do not know how > to find more about that, it seems an EC2 issue. > > Except you are being attacked in vain by the scripting vulnerability, > there are only syntax errors in configuration and queries... you should be > able to fix that. > > Jörg > > > > On Tue, Aug 12, 2014 at 4:25 PM, Sam2014 <sabda...@gmail.com <javascript:> > > wrote: > >> Just wondering, will remove/re-install get rid of the stagnant index >> since I plan on upgrading from 1.2.1 to 1.2.2? >> >> >> On Tuesday, August 12, 2014 9:42:10 AM UTC-4, Sam2014 wrote: >>> >>> The link doesn't work. >>> I am not sure about the difference, I have always deleted the indices >>> the same way, curl -XPUT http... //to create, and curl -XDelete /http:... >>> I did not create any aliases, I am just using Get _aliases to see whats >>> still on the node. >>> Did the log tell you anything, there are some weird errors in there that >>> I don't know how to interpret, a quick look there may tell you something. >>> >>> On Tuesday, August 12, 2014 3:04:39 AM UTC-4, Jörg Prante wrote: >>>> >>>> So you are wrestling with aliases. You can not delete aliases by file >>>> system operations. Have you checked >>>> >>>> http://www.elasticsearch.org/guide/en/elasticsearch/ >>>> reference/current/indices-aliases.html#deleting >>>> >>>> for deleting aliases? >>>> >>>> Jörg >>>> >>>> >>>> On Tue, Aug 12, 2014 at 4:10 AM, Sam2014 <sabda...@gmail.com> wrote: >>>> >>>>> ///Attached is the cluster log that covers the whole sequence from >>>>> shutting the cluster and re-activating it. >>>>> >>>>> //Show me current state of cluster >>>>> { >>>>> "cluster_name": "rexCluster", >>>>> "status": "red", >>>>> "timed_out": false, >>>>> "number_of_nodes": 2, >>>>> "number_of_data_nodes": 2, >>>>> "active_primary_shards": 7, >>>>> "active_shards": 7, >>>>> "relocating_shards": 0, >>>>> "initializing_shards": 2, >>>>> "unassigned_shards": 21 >>>>> } >>>>> >>>>> //Show me indices on the cluster >>>>> >>>>> GET _aliases >>>>> { >>>>> "foo": { >>>>> "aliases": {} >>>>> }, >>>>> "mediregs": { >>>>> "aliases": {} >>>>> }, >>>>> "twitter": { >>>>> "aliases": {} >>>>> } >>>>> } >>>>> >>>>> >>>>> >>>>> //Shut down the cluster >>>>> >>>>> POST /_cluster/nodes/_shutdown >>>>> { >>>>> "cluster_name": "rexCluster", >>>>> "nodes": { >>>>> "_vPEfmc9TruIpugk4NWleg": { >>>>> "name": "node 1" >>>>> }, >>>>> "DB94rWiiSvS9HUTXwI7GTw": { >>>>> "name": "node 2" >>>>> } >>>>> } >>>>> } >>>>> >>>>> //Go into the fs and delete files under /data >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$ >>>>> >>>>> ls >>>>> >>>>> *indices* node.lock *_state* >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0$ >>>>> >>>>> cd indices/ >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$ >>>>> >>>>> ls >>>>> >>>>> *foo* *mediregs* *twitter* >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$ >>>>> >>>>> sudo rm -rf foo >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$ >>>>> >>>>> sudo rm -rf mediregs >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$ >>>>> >>>>> sudo rm -rf twitter >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/rexCluster/nodes/0/indices$ >>>>> >>>>> ls >>>>> >>>>> ubuntu@domU-12-31-39-0A-99-EA:/opt/cores/elasticsearch/data/ >>>>> rexCluster/nodes/0/indices$ >>>>> //Restart nodes 1 &2 - should be in log file >>>>> >>>>> Get _aliases //Again - they came back!! >>>>> { >>>>> "foo": { >>>>> "aliases": {} >>>>> }, >>>>> "mediregs": { >>>>> "aliases": {} >>>>> }, >>>>> "twitter": { >>>>> "aliases": {} >>>>> } >>>>> } >>>>> >>>>> >>>>> //Ok - Try to delete index >>>>> DELETE /mediregs >>>>> >>>>> //takes little while and comes back >>>>> { >>>>> "acknowledged": false >>>>> } >>>>> >>>>> >>>>> >>>>> >>>>> On Monday, August 11, 2014 7:12:22 PM UTC-4, Jörg Prante wrote: >>>>> >>>>>> Can you post your detailed steps and debug logs? >>>>>> >>>>>> Make also sure there is no third node - this looks like ES is >>>>>> shipping shards to a third node from which the two nodes are recovering >>>>>> from. >>>>>> >>>>>> Jörg >>>>>> >>>>>> >>>>>> On Tue, Aug 12, 2014 at 1:06 AM, Sam2014 <sabda...@gmail.com> wrote: >>>>>> >>>>>>> I stopped node 1, then I stopped node 2. I disabled the cluster, >>>>>>> deleted files from nodes, Made sure no Indices on either node. I >>>>>>> re-enabled >>>>>>> cluster in yml, restarted cluster, and the index came back! >>>>>>> >>>>>>> >>>>>>> On Monday, August 11, 2014 6:16:42 PM UTC-4, Jörg Prante wrote: >>>>>>> >>>>>>>> There is no vicious circle. First you must stop the cluster. Then, >>>>>>>> after processes have exited, continue with file operations. After >>>>>>>> that, you >>>>>>>> can start the cluster nodes again. >>>>>>>> >>>>>>>> Jörg >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 11, 2014 at 11:49 PM, Sam2014 <sabda...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Oops, When I restarted the cluster/nodes, I see shards under an >>>>>>>>> "unassigned node", neither of the two nodes of the cluster. Is there >>>>>>>>> a way >>>>>>>>> I can route/assign /delete out of this vicious cycle? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Monday, August 11, 2014 5:46:25 PM UTC-4, Sam2014 wrote: >>>>>>>>>> >>>>>>>>>> I also deleted the directory, restarted the nodes, and it >>>>>>>>>> recovered the index still, which is null. >>>>>>>>>> >>>>>>>>>> On Monday, August 11, 2014 5:42:53 PM UTC-4, Sam2014 wrote: >>>>>>>>>>> >>>>>>>>>>> I wish it was that simple, I deleted the directory of the index >>>>>>>>>>> from /data. Yet, the index is still there and won't be deleted or >>>>>>>>>>> recreated.. I still get acknowledged = false. >>>>>>>>>>> >>>>>>>>>>> On Monday, August 11, 2014 5:31:36 PM UTC-4, Jörg Prante wrote: >>>>>>>>>>>> >>>>>>>>>>>> Your JVMs (at least one of them) are not able to process the >>>>>>>>>>>> command, so you should check out the data directory where the file >>>>>>>>>>>> of the >>>>>>>>>>>> index to be deleted are located. Shut your cluster down, remove >>>>>>>>>>>> the index >>>>>>>>>>>> files of the index to be deleted from the filesystem, and start >>>>>>>>>>>> the cluster >>>>>>>>>>>> up again. >>>>>>>>>>>> >>>>>>>>>>>> To avoid such situations, find out why the JVMs could not >>>>>>>>>>>> react, most probably there are exceptions in the server logs, and >>>>>>>>>>>> fix the >>>>>>>>>>>> trouble. >>>>>>>>>>>> >>>>>>>>>>>> Jörg >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 11, 2014 at 11:24 PM, Sam2014 <sabda...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I got myself in trouble by changing the mapping. >>>>>>>>>>>>> >>>>>>>>>>>>> I am on a cluster of 2 nodes, and trying to delete the index, >>>>>>>>>>>>> I keep getting acknowleged = false with normal delete curl -X >>>>>>>>>>>>> GET "http://localhost:9200/myIndex >>>>>>>>>>>>> >>>>>>>>>>>>> If I look at my index through HEAD plugin, I get >>>>>>>>>>>>> "myIndex" >>>>>>>>>>>>> size: unknown >>>>>>>>>>>>> docs: unknown >>>>>>>>>>>>> >>>>>>>>>>>>> and I see no shards assigned to either node apparently >>>>>>>>>>>>> (Everything in this index indicates Null). I would just like to >>>>>>>>>>>>> be able to >>>>>>>>>>>>> delete it. How can I force a delete? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "elasticsearch" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to elasticsearc...@googlegroups.com. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/c6818593-550 >>>>>>>>>>>>> 0-4762-9f59-46a2e2a3f772%40googlegroups.com >>>>>>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/c6818593-5500-4762-9f59-46a2e2a3f772%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elasticsearch" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/8e616783-23f >>>>>>>>> e-4029-9a53-6e30389abe6c%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/elasticsearch/8e616783-23fe-4029-9a53-6e30389abe6c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to elasticsearc...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/255a872f-02d >>>>>>> 3-49e5-917e-9736ffcf5c3b%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/255a872f-02d3-49e5-917e-9736ffcf5c3b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to elasticsearc...@googlegroups.com. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/980ceb4c-1312-4844-8d70-00f85d39b51e% >>>>> 40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/980ceb4c-1312-4844-8d70-00f85d39b51e%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/03892de6-5a8a-420e-b915-153076b68f73%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/03892de6-5a8a-420e-b915-153076b68f73%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c93f2b1-1b26-4044-91c8-fc7ada122884%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
