Think of es-hadoop as a connector between Hadoop and Elasticsearch. You would use it to index data in Hadoop to ES or run queries in ES directly from Hadoop. Where does ES store the data? That depends on its configuration (completely separate from es-hadoop itself). In general (and the default) is to store it onto the local file-system. If you want to use it on a shared file-system or HDFS you can easily do that by mounting it locally (for example, mount HDFS through NFS as a local disk) and point ES to it. ES is happy to work with it however the performance will be _significantly_ degraded and most of the real-time nature of it will go down the window since HDFS is a distributed file-system (and thus even basic operations like opening a file or closing a file mean at least one call over the network) plus you're giving up the amazing OS file-system cache (since the fs is not local). If the FS is slow, anything that sits on top of it (like ES) will be slow as well.
Hope this helps, P.S. By the way, if you want/need to snapshot/restore data to/from ES from/to HDFS you can use the HDFS repository (more info here: http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/setup.html ) On Thu, Jun 5, 2014 at 8:41 PM, ES USER <es.user.2...@gmail.com> wrote: > Try as I might and I have read all the stuff I can find on ES' website > about this I understand somewhat how the integration works but not the > actual nuts and bolts of it. > > For example: > > Is Hadoop just storing the files that would normally be stored in the > local filesystem for the ES indexes or is it storing the data that would > normally be in those indexes and just accessed through es-hadoop? > > If it is the latter how do you go about determining whatto set for the > number of nodes and shards. > > > If anyone has any information on this or even better yet a place to point > me to that has better references so that I can research this on my own it > would be much appreciated. > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/b78f2fa6-42c9-4ae7-a4ab-aacbc2c53293%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/b78f2fa6-42c9-4ae7-a4ab-aacbc2c53293%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmfSrZ49XHgGfnRcfHQTH%3DSy%2B18RQ_%2BwEqR8MYOuZr%3DjZQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
