Its been true for as long as I've used Elasticsearch which as .9X (can't remember which). Copying everything is pretty common because the index operations are non-deterministic so the files might not be the same at all. I pretty much assume I'm going to have to copy _almost_ everything every time I restart. It turns my rolling restarts into two day long affairs. I know its being worked on.
Nik On Fri, Nov 21, 2014 at 10:51 AM, Yves Dorfsman <y...@zioup.com> wrote: > Thanks Nicolas. > > Is this true on versions 0.9, or only on > 1? > I've had nodes die and restart, and they did copy everything! > > On 2014-11-20 22:02, Nikolas Everett wrote: > > The thing is that this is a disk level operation. It pretty much rsyncs > the > > files from the current master shard to the node when it comes back > online. > > This would be OK if the replica shards matched the master but that is > only > > normally the case if the shard was moved to the node after it was mostly > > complete and then you've had only a few writes. Normally shards don't > match > > each other because the way the index is maintained is nondeterministic. > > > > The translog replay is only used as a catch up after the rsync-like step. > > > > This is something that is being worked on. Its certainly my biggest > complaint > > about elasticsearch but I'm confident that it'll get better. > > > > Nik > > > > On Nov 20, 2014 11:11 PM, "Mark Walkom" <markwal...@gmail.com > > <mailto:markwal...@gmail.com>> wrote: > > > > It will enter recovery where it syncs at the segment level from the > > current primary, then the translog gets shipped over and (re)played, > which > > brings it all up to date. > > > > On 21 November 2014 14:51, Yves Dorfsman <y...@zioup.com > > <mailto:y...@zioup.com>> wrote: > > > > > > If you do disable allocation before you reboot a node and a > client > > writes to a shard that had a replica on that node, does the > entire > > replica gets copied when the node come up? Or does it get just > updated? > > > > On Thursday, 20 November 2014 19:52:26 UTC-7, Mark Walkom wrote: > > > > You should disable allocation before you reboot, that will > save a > > lot of shard shuffling - > > > http://www.elasticsearch.org/__guide/en/elasticsearch/__reference/current/setup-__upgrade.html#rolling-upgrades > > < > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#rolling-upgrades > > > > > > On 21 November 2014 13:48, Konstantin Erman < > kon...@gmail.com> wrote: > > > > I work on an experimental cluster of ES nodes running on > > Windows Server machines. Once in a while we have a need > to > > reboot machines. The initial state - cluster is green > and well > > balanced. One machine is gracefully taken offline and > then > > after necessary service is performed it comes back > online. All > > the hardware and file system content is intact. As soon > as ES > > service starts on that machine, it assumes that there is > no > > usable data locally and recovers as much data as it deems > > necessary for balancing from other nodes. > > > > This behavior puzzles me, because most of the data shards > > stored on that machine file system can be reused as they > are. > > Cluster stores logs, so all indices except those for > > the current day never ever change until they get deleted. > > Can't ES node detect that it has perfect copies of some > > (actually most) of the shards and instead of copying > them over > > just mark them as up to date? > > > > I suspect I don't know about some step to enable this > behavior > > and I'm looking to enable it. Any advice? > > > > Thank you! > > Konstantin > > > > -- > > You received this message because you are subscribed to the > Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from > it, send > > an email to elasticsearch+unsubscr...@googlegroups.com > > <mailto:elasticsearch+unsubscr...@googlegroups.com>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com > > < > https://groups.google.com/d/msgid/elasticsearch/51b3ff69-a126-4f2f-9838-0098bc26694d%40googlegroups.com?utm_medium=email&utm_source=footer > >. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, > send an > > email to elasticsearch+unsubscr...@googlegroups.com > > <mailto:elasticsearch+unsubscr...@googlegroups.com>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com > > < > https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmxZuSjJAJPj_yKT6d8_L-Mx6ceZfDNmJCLkSOXsfeydQ%40mail.gmail.com?utm_medium=email&utm_source=footer > >. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to a topic in the > Google > > Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/elasticsearch/ijQ1N-CAfc8/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > > elasticsearch+unsubscr...@googlegroups.com > > <mailto:elasticsearch+unsubscr...@googlegroups.com>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com > > < > https://groups.google.com/d/msgid/elasticsearch/CAPmjWd09LRJk89wdHybYy48FMpCaYa1wTJ9HX9uX%2BjvNjvYq2g%40mail.gmail.com?utm_medium=email&utm_source=footer > >. > > For more options, visit https://groups.google.com/d/optout. > > > -- > http://yves.zioup.com > gpg: 4096R/32B0F416 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/546F5F72.3070002%40zioup.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0L1_UGUxipKAL9R8BEUNfNGfdMxHXHOmMirZzDtp9xpg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
