Thanks Costin, I am aiming at modifying the existing hadoop cluster and hive installation and also modularizing some common es.* properies in a separate common place. I know the first goal can be achieved with hive cli --auxpath option and hive table's TBLPROPERTERTIES. For the secon goal, I am able to move some es.* settings from TBLPROPERTIES declaration to hive's set statments. For example, I can put
set es.nodes=my.domain.com in the same hql file then skip es.nodes setting in TBLPROPERTIES in the external table delcarations in the SAME hql. But I wish I can move the set statetemnt in a separate file. I now realize this is rather a hive question. Regards, Jack On Sun, Jun 15, 2014 at 2:19 AM, Costin Leau <costin.l...@gmail.com> wrote: > Could you please raise an issue with some type of example? Due to the way > Hadoop (and Hive) works, > things tend to be tricky in terms of configuring a job. > > The configuration needs to be created before a job is submitted which in > practice means "dynamic configurations" > are basically impossible (this also has some security implications which > are simply avoided this way). > Thus either one specifies the configuration manually or loads a known > location file (hive-site.xml, core-site.xml...) > upfront, before the job is submitted. > This means when dealing with Hive, Pig, Cascading, etc... unless one adds > a pre-processor to the job content (script, flow, etc...) > by the time es-hadoop kicks in, the job is already running and thus its > changes discarded. > > Cheers, > > On 6/14/14 1:57 AM, Jinyuan Zhou wrote: > >> Hi, >> I am playing with elasticsearch and hive integration. The documentation >> says >> to set configuration like es.nodes, es.port in TBLPROPERTIES. It works. >> But it can cause many reduntant codes. If I have ten data set to index to >> the same es cluster, >> I would have to repeat this information ten times in TBLPROPERTIES. >> Even if >> I use var substitution I still have to rwrite this subtititiov var for >> each table definition. >> What I am looking for is to put these info in say one file and pass the >> location, in some way, to hive cli >> so hive elasticsearch will get these settings when trying to find es >> server to talk to. >> I am not looking into put these info into files like hive-site.xml. >> >> Thanks, >> >> Jack >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to >> elasticsearch+unsubscr...@googlegroups.com <mailto:elasticsearch+ >> unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/7040c805- >> e845-4b3d-a9fe-5e18d8445f7f%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/7040c805- >> e845-4b3d-a9fe-5e18d8445f7f%40googlegroups.com?utm_medium= >> email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Costin > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/elasticsearch/1WH7kOD3uKs/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elasticsearch/539D6507.3080207%40gmail.com. > For more options, visit https://groups.google.com/d/optout. > -- -- Jinyuan (Jack) Zhou -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANBTPCGjBAg5k5R_uz6P3DAuDKXax7A5qPSsd9Kf2gEqtSZZ2Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
