Re: knapsack use case

[Matteo Moci]

…just a small update in case anyone was wondering:

I completed an elasticsearch-knapsack export to file from a 0.90.7 (with
the plugin built with 0.90.7 dependencies) that was correctly re-imported
in a 1.4.2 instance with the latest plugin version, including settings and
aliases.

I just checked out the source from github and changed 2-3 lines due to API
changes and assembled the plugin to be installed on a 0.90.7 instance.

Just wanted to say thanks!

Best,
Matteo


On Thu, Jul 31, 2014 at 11:33 AM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Snapshot/restore is always recommended, but is a 1.0 feature. This is a
> standard API of ES and well supported by the ES team. With that, you can
> handle all kinds of index data safely on a binary level, fully and
> incrementally.
>
> Knapsack plugin is for document export/import only. I wrote it to
> transport _source data harvested over a long time period from a  < 1.0
> system to a production system. It works on _source or stored fields only.
> It uses search/query and bulk indexing API without snapshots, so it is up
> to the admin to stop index writes while knapsack runs. There is also a
> lookup of index settings and mappings, this information is also included in
> the export archive file, and re-applied at import time. But, there is no
> check if these settings/mappings can be applied on the target successfully,
> this is left to the admin to prepare plugins, analyzers, etc. Aliases are
> not transported but this is a good idea for improvement.
>
> Currently, knapsack plugin does not work on ES 1.3 but I am progressing to
> implement this. I am adding a Java-level API. Currently it is REST only.
>
> Jörg
>
>
> On Thu, Jul 31, 2014 at 11:05 AM, Matteo Moci  wrote:
>
>> Hi All,
>> I have some questions about the knapsack plugin [1].
>>
>> My idea to use the tool to do a backup to a file, starting from a 0.90.x
>> instance and then restore it on a different 1.2.x or 1.3.x instance. I see
>> it can't be done directly, copying to a local/remote cluster.
>>
>> Would it work doing an intermediate step with a file?
>> Or the backup still has metadata about the es version it was generated
>> from, making it impossible?
>>
>> Is the snapshot and restore feature [2] useful in my use case, or not?
>>
>> Is the knapsack plugin able to backup and restore also aliases and
>> mappings, or do I have to manually migrate them before restoring data?
>>
>> Thanks for the patience and the great work!
>> Matteo
>>
>> [1] https://github.com/jprante/elasticsearch-knapsack
>> [2]
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>>
>> --
>> Matteo Moci
>> http://mox.fm
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGrXthJjCchEf2oyvXKnSZyBp31nvnAeXwAZJaEkvnT5Q%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Matteo Moci
http://mox.fm

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAONgFZ6PvJyeF04ERyeb26LhXoHR%3DMMs5sc5KV2ASLF_UK6b%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: knapsack use case

[joergpra...@gmail.com]

Snapshot/restore is always recommended, but is a 1.0 feature. This is a
standard API of ES and well supported by the ES team. With that, you can
handle all kinds of index data safely on a binary level, fully and
incrementally.

Knapsack plugin is for document export/import only. I wrote it to transport
_source data harvested over a long time period from a  < 1.0 system to a
production system. It works on _source or stored fields only. It uses
search/query and bulk indexing API without snapshots, so it is up to the
admin to stop index writes while knapsack runs. There is also a lookup of
index settings and mappings, this information is also included in the
export archive file, and re-applied at import time. But, there is no check
if these settings/mappings can be applied on the target successfully, this
is left to the admin to prepare plugins, analyzers, etc. Aliases are not
transported but this is a good idea for improvement.

Currently, knapsack plugin does not work on ES 1.3 but I am progressing to
implement this. I am adding a Java-level API. Currently it is REST only.

Jörg


On Thu, Jul 31, 2014 at 11:05 AM, Matteo Moci  wrote:

> Hi All,
> I have some questions about the knapsack plugin [1].
>
> My idea to use the tool to do a backup to a file, starting from a 0.90.x
> instance and then restore it on a different 1.2.x or 1.3.x instance. I see
> it can't be done directly, copying to a local/remote cluster.
>
> Would it work doing an intermediate step with a file?
> Or the backup still has metadata about the es version it was generated
> from, making it impossible?
>
> Is the snapshot and restore feature [2] useful in my use case, or not?
>
> Is the knapsack plugin able to backup and restore also aliases and
> mappings, or do I have to manually migrate them before restoring data?
>
> Thanks for the patience and the great work!
> Matteo
>
> [1] https://github.com/jprante/elasticsearch-knapsack
> [2]
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>
> --
> Matteo Moci
> http://mox.fm
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAONgFZ60jWViqzRVO6_U-rYo6dUzunE3ojv%2BR5U8HX1Lwp4PdA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGrXthJjCchEf2oyvXKnSZyBp31nvnAeXwAZJaEkvnT5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.