Re: PUT gzipped data into elasticsearch
Actually, I am more interested about pushing compressed data into elasticsearch than asking for compressed data. So far, I tried it manually, using Google chrome plugin - postman, or curl, without success: curl -sS -X POST --data-binary txn001.json.gz http://localhost:9200/gzip/new -H Content-Type: application/json -H Content-Encoding: gzip Response: curl: (52) Empty reply from server Marcel On Tue, Mar 24, 2015 at 10:18 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Logstash has both Java and HTTP output, but I assume you want to use HTTP. Set http.compression parameter to true in ES configuration, then you can use gzip-compressed HTTP traffic using Accept-Encoding header. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html Jörg On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: PUT gzipped data into elasticsearch
I would guess that your gzipped data would need to be base64 encoded and placed in a field of the JSON along with an extract of whatever other fields you find important or interesting. Then you'll have _source.gzipped available to your application to decode and unzip when you choose. --Karl On Mar 25, 2015 1:04 AM, Marcel Matus matusmar...@gmail.com wrote: Actually, I am more interested about pushing compressed data into elasticsearch than asking for compressed data. So far, I tried it manually, using Google chrome plugin - postman, or curl, without success: curl -sS -X POST --data-binary txn001.json.gz http://localhost:9200/gzip/new -H Content-Type: application/json -H Content-Encoding: gzip Response: curl: (52) Empty reply from server Marcel On Tue, Mar 24, 2015 at 10:18 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Logstash has both Java and HTTP output, but I assume you want to use HTTP. Set http.compression parameter to true in ES configuration, then you can use gzip-compressed HTTP traffic using Accept-Encoding header. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html Jörg On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe . To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to
Re: PUT gzipped data into elasticsearch
Yes, I mean pushing compressed data. You have several wrong assumptions: use PUT instead of POST, mark file with a preceding '@', set 'Accept-Encoding' header to 'gzip,deflate', and use URL for a document creation 'localhost:9200/test/docs/1' for example. Here is my full demo of a correct curl command https://gist.github.com/jprante/15f570be70504285a678 Jörg On Wed, Mar 25, 2015 at 8:03 AM, Marcel Matus matusmar...@gmail.com wrote: Actually, I am more interested about pushing compressed data into elasticsearch than asking for compressed data. So far, I tried it manually, using Google chrome plugin - postman, or curl, without success: curl -sS -X POST --data-binary txn001.json.gz http://localhost:9200/gzip/new -H Content-Type: application/json -H Content-Encoding: gzip Response: curl: (52) Empty reply from server Marcel On Tue, Mar 24, 2015 at 10:18 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Logstash has both Java and HTTP output, but I assume you want to use HTTP. Set http.compression parameter to true in ES configuration, then you can use gzip-compressed HTTP traffic using Accept-Encoding header. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html Jörg On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe . To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this
Re: PUT gzipped data into elasticsearch
Oh sorry, I forgot the curl parameter --compressed Now added to the gist. Jörg On Wed, Mar 25, 2015 at 9:02 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Jorg, thank you for your help - mainly your practical example is great. The header Accept-Encoding isn't really necessary. I was almost there, but was missing either the id of document or at before the binary file. Thanks. Marcel On Wed, Mar 25, 2015 at 2:15 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Yes, I mean pushing compressed data. You have several wrong assumptions: use PUT instead of POST, mark file with a preceding '@', set 'Accept-Encoding' header to 'gzip,deflate', and use URL for a document creation 'localhost:9200/test/docs/1' for example. Here is my full demo of a correct curl command https://gist.github.com/jprante/15f570be70504285a678 Jörg On Wed, Mar 25, 2015 at 8:03 AM, Marcel Matus matusmar...@gmail.com wrote: Actually, I am more interested about pushing compressed data into elasticsearch than asking for compressed data. So far, I tried it manually, using Google chrome plugin - postman, or curl, without success: curl -sS -X POST --data-binary txn001.json.gz http://localhost:9200/gzip/new -H Content-Type: application/json -H Content-Encoding: gzip Response: curl: (52) Empty reply from server Marcel On Tue, Mar 24, 2015 at 10:18 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Logstash has both Java and HTTP output, but I assume you want to use HTTP. Set http.compression parameter to true in ES configuration, then you can use gzip-compressed HTTP traffic using Accept-Encoding header. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html Jörg On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe . To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe . To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web
Re: PUT gzipped data into elasticsearch
Hi Jorg, thank you for your help - mainly your practical example is great. The header Accept-Encoding isn't really necessary. I was almost there, but was missing either the id of document or at before the binary file. Thanks. Marcel On Wed, Mar 25, 2015 at 2:15 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Yes, I mean pushing compressed data. You have several wrong assumptions: use PUT instead of POST, mark file with a preceding '@', set 'Accept-Encoding' header to 'gzip,deflate', and use URL for a document creation 'localhost:9200/test/docs/1' for example. Here is my full demo of a correct curl command https://gist.github.com/jprante/15f570be70504285a678 Jörg On Wed, Mar 25, 2015 at 8:03 AM, Marcel Matus matusmar...@gmail.com wrote: Actually, I am more interested about pushing compressed data into elasticsearch than asking for compressed data. So far, I tried it manually, using Google chrome plugin - postman, or curl, without success: curl -sS -X POST --data-binary txn001.json.gz http://localhost:9200/gzip/new -H Content-Type: application/json -H Content-Encoding: gzip Response: curl: (52) Empty reply from server Marcel On Tue, Mar 24, 2015 at 10:18 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Logstash has both Java and HTTP output, but I assume you want to use HTTP. Set http.compression parameter to true in ES configuration, then you can use gzip-compressed HTTP traffic using Accept-Encoding header. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html Jörg On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe . To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com
Re: PUT gzipped data into elasticsearch
Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: PUT gzipped data into elasticsearch
Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
PUT gzipped data into elasticsearch
Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: PUT gzipped data into elasticsearch
Logstash has both Java and HTTP output, but I assume you want to use HTTP. Set http.compression parameter to true in ES configuration, then you can use gzip-compressed HTTP traffic using Accept-Encoding header. http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html Jörg On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmar...@gmail.com wrote: Hi Joe, no, we post data into elasticsearch using Logstash. Actually, the complete way is: APP - Logstash - RMQ - Logstash - ES. And when I include F5 VIP's, there are 6 services on the way to process these data... Marcel On Tue, Mar 24, 2015 at 1:50 PM, joergpra...@gmail.com joergpra...@gmail.com wrote: Are you using Java API? Jörg On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmar...@gmail.com wrote: Hi, some of our data are big ones (1 - 10 MB), and if there are milions of those, it causes us trouble in our internal network. We would like to compress these data in generation time, and send over the network to elasticsearch in compressed state. I tried to find some solution, but usually I found only retrieving gzipped data. Is there a way, how to put compressed data into elasticsearch? Thanks, Marcel -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Marcel Matus -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
