Thanks David, I will certainly look into hashtag. Do you think it is a good idea to separate data analysis and indexing into 2 different machines since both require lots of cpu time. If I use hashtag to send files over to ES, will I be able to use native Java API or http, and is there any preference to the API? I have noticed there are somethings that aren't very easy and may be don't even work in the native API? Thanks again.
On Tuesday, January 28, 2014 1:05:32 PM UTC-5, David Pilato wrote: > > Did you try https://github.com/dadoonet/fsriver? > Never tested it with so many docs but may be it could help you here? > > If you have already generated json files on a server, then I would > recommend trying logstash to send them into elasticsearch. > > My 2 cents > > -- > *David Pilato* | *Technical Advocate* | *Elasticsearch.com* > @dadoonet <https://twitter.com/dadoonet> | > @elasticsearchfr<https://twitter.com/elasticsearchfr> > > > Le 28 janvier 2014 at 16:46:06, ZenMaster80 (sabda...@gmail.com<javascript:>) > a écrit: > > I would like to get your perspective on how to load json to index server > in my scenario. > We have about 15 million documents in html/pdf/... on Server 1 > I would like to process the data and convert to json on server 2 > I would like the indexer to index json n a separate machine/server server 3 > > Ideally I thought on Server 2, as I prepare json and have it ready in > memory, I can feed it to indexer. But since data processing is cpu > intensive, I want indexing to be done on a separate machines/server. > How do you guys deal with this since I can no longer feed in-memory json > to the indexer on separate machine? Do I just grab files from server 2 and > index them then? > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearc...@googlegroups.com <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/05b977ac-00d0-45c0-9e58-8df523e6978c%40googlegroups.com > . > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a02427ec-a3d8-484f-9cfb-2ba7628192b1%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
