Thanks for sharing - can you also give an example of the table initialization in init.hive vs myscript.hql?
Cheers! On 6/16/14 11:19 PM, Jinyuan Zhou wrote:
Just share a solution I learned hive side. hive cli has an -i option that takes a file of hive commands to initilize the session. so I can put a list of set comand as well as add jar ... command in one file, say inithive then run the cli as this: hive -i init.hive -f myscript.hql. Note table creation hql inside myscript.hql don't have to set es.* properties as long as it appears in init.hive file This solves my problem. Thanks, Jinyuan (Jack) Zhou On Sun, Jun 15, 2014 at 10:24 AM, Jinyuan Zhou <zhou.jiny...@gmail.com <mailto:zhou.jiny...@gmail.com>> wrote: Thanks Costin, I am aiming at modifying the existing hadoop cluster and hive installation and also modularizing some common es.* properies in a separate common place. I know the first goal can be achieved with hive cli --auxpath option and hive table's TBLPROPERTERTIES. For the secon goal, I am able to move some es.* settings from TBLPROPERTIES declaration to hive's set statments. For example, I can put set es.nodes=my.domain.com <http://my.domain.com> in the same hql file then skip es.nodes setting in TBLPROPERTIES in the external table delcarations in the SAME hql. But I wish I can move the set statetemnt in a separate file. I now realize this is rather a hive question. Regards, Jack On Sun, Jun 15, 2014 at 2:19 AM, Costin Leau <costin.l...@gmail.com <mailto:costin.l...@gmail.com>> wrote: Could you please raise an issue with some type of example? Due to the way Hadoop (and Hive) works, things tend to be tricky in terms of configuring a job. The configuration needs to be created before a job is submitted which in practice means "dynamic configurations" are basically impossible (this also has some security implications which are simply avoided this way). Thus either one specifies the configuration manually or loads a known location file (hive-site.xml, core-site.xml...) upfront, before the job is submitted. This means when dealing with Hive, Pig, Cascading, etc... unless one adds a pre-processor to the job content (script, flow, etc...) by the time es-hadoop kicks in, the job is already running and thus its changes discarded. Cheers, On 6/14/14 1:57 AM, Jinyuan Zhou wrote: Hi, I am playing with elasticsearch and hive integration. The documentation says to set configuration like es.nodes, es.port in TBLPROPERTIES. It works. But it can cause many reduntant codes. If I have ten data set to index to the same es cluster, I would have to repeat this information ten times in TBLPROPERTIES. Even if I use var substitution I still have to rwrite this subtititiov var for each table definition. What I am looking for is to put these info in say one file and pass the location, in some way, to hive cli so hive elasticsearch will get these settings when trying to find es server to talk to. I am not looking into put these info into files like hive-site.xml. Thanks, Jack -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@__googlegroups.com <mailto:elasticsearch%2bunsubscr...@googlegroups.com> <mailto:elasticsearch+__unsubscr...@googlegroups.com <mailto:elasticsearch%2bunsubscr...@googlegroups.com>>. To view this discussion on the web visit https://groups.google.com/d/__msgid/elasticsearch/7040c805-__e845-4b3d-a9fe-5e18d8445f7f%__40googlegroups.com <https://groups.google.com/d/msgid/elasticsearch/7040c805-e845-4b3d-a9fe-5e18d8445f7f%40googlegroups.com> <https://groups.google.com/d/__msgid/elasticsearch/7040c805-__e845-4b3d-a9fe-5e18d8445f7f%__40googlegroups.com?utm_medium=__email&utm_source=footer <https://groups.google.com/d/msgid/elasticsearch/7040c805-e845-4b3d-a9fe-5e18d8445f7f%40googlegroups.com?utm_medium=email&utm_source=footer>>. For more options, visit https://groups.google.com/d/__optout <https://groups.google.com/d/optout>. -- Costin -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/__topic/elasticsearch/__1WH7kOD3uKs/unsubscribe <https://groups.google.com/d/topic/elasticsearch/1WH7kOD3uKs/unsubscribe>. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@__googlegroups.com <mailto:elasticsearch%2bunsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/__msgid/elasticsearch/539D6507.__3080207%40gmail.com <https://groups.google.com/d/msgid/elasticsearch/539D6507.3080207%40gmail.com>. For more options, visit https://groups.google.com/d/__optout <https://groups.google.com/d/optout>. -- -- Jinyuan (Jack) Zhou -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com <mailto:elasticsearch+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANBTPCErh1M5_xNa0SE-ZShpUDuXKTPMCYqrWCB1z36%3D9vjaDQ%40mail.gmail.com <https://groups.google.com/d/msgid/elasticsearch/CANBTPCErh1M5_xNa0SE-ZShpUDuXKTPMCYqrWCB1z36%3D9vjaDQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout.
-- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/539F5C5F.5050408%40gmail.com. For more options, visit https://groups.google.com/d/optout.
