ES stores data in its own internal format, which typically resides locally. What you are stating is partially correct - with the connector you would move/copy data between Hadoop and ES since, in order for ES to work with data, it needs to actually index it (that is, to see it). So you would use es-hadoop to index data from Hadoop in ES or/and query ES directly from Hadoop.
On Fri, Jun 6, 2014 at 9:29 PM, ES USER <es.user.2...@gmail.com> wrote: > I guess the problem I having wrapping my head around is exactly where the > data is residing and in what format. > > If I understand the Georgi's email above is it that you can run map reduce > jobs against data stored in local ES through by utilizing es-hadoop and you > can also run ES queries against data in Hadoop utilizing es-hadoop. > > > Is that correct? > > > > > On Friday, June 6, 2014 12:39:44 PM UTC-4, Costin Leau wrote: > >> Adding to what Georgi wrote, es-hadoop does not create the shards for you >> - that's up to you or index templates (which I highly recommend). However >> es-hadoop is aware of the target shards and will use them to parallelize >> the reads/writes (such as one task per shard). >> >> >> On Fri, Jun 6, 2014 at 2:45 PM, Georgi Ivanov <georgi....@gmail.com> >> wrote: >> >>> and i don't think this anyhow related with number of shards and nodes >>> >>> >>> On Thursday, June 5, 2014 7:41:34 PM UTC+2, ES USER wrote: >>> >>>> Try as I might and I have read all the stuff I can find on ES' website >>>> about this I understand somewhat how the integration works but not the >>>> actual nuts and bolts of it. >>>> >>>> For example: >>>> >>>> Is Hadoop just storing the files that would normally be stored in the >>>> local filesystem for the ES indexes or is it storing the data that would >>>> normally be in those indexes and just accessed through es-hadoop? >>>> >>>> If it is the latter how do you go about determining whatto set for the >>>> number of nodes and shards. >>>> >>>> >>>> If anyone has any information on this or even better yet a place to >>>> point me to that has better references so that I can research this on my >>>> own it would be much appreciated. >>>> >>>> Thanks. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/90662a91-1557-4f61-86a2-bd2e620aec6f% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/90662a91-1557-4f61-86a2-bd2e620aec6f%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/ed729795-a7d6-4320-9da2-16b214e653b0%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/ed729795-a7d6-4320-9da2-16b214e653b0%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmeacuVOXNYcwdYHBg69TAotrqvyuzre_JeUK-RfAcFBXA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
