What is the version of Hive and the execution engine (MR, Tez, Spark)? HTH
Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 18 April 2016 at 20:59, Shirish Tatikonda <shirish.tatiko...@gmail.com> wrote: > Hi John, > > 2) The shell script is invoked in the mappers of a Hadoop streaming job. > > 1) The use case is that I have to process multiple entities in parallel. > Each entity is associated with its own data set. The processing involves a > few hive queries to do joins and aggregations, which is followed by some > code in Python. My thought process is to put the hive queries and python > invocation in a shell script, and invoke the shell script on multiple > entities in parallel through a streaming mapreduce job. > > Shirish > > > On Sat, Apr 16, 2016 at 12:10 AM, Jörn Franke <jornfra...@gmail.com> > wrote: > >> Just out of curiosity, what is the use case behind this? >> >> How do you call the shell script? >> >> > On 16 Apr 2016, at 00:24, Shirish Tatikonda < >> shirish.tatiko...@gmail.com> wrote: >> > >> > Hello, >> > >> > I am trying to run multiple hive queries in parallel by submitting them >> through a map-reduce job. >> > More specifically, I have a map-only hadoop streaming job where each >> mapper runs a shell script that does two things -- 1) parses input lines >> obtained via streaming; and 2) submits a very simple hive query (via hive >> -e ...) with parameters computed from step-1. >> > >> > Now, when I run the streaming job, the mappers seem to be stuck and I >> don't know what is going on. When I looked on resource manager web UI, I >> don't see any new MR Jobs (triggered from the hive query). I am trying to >> understand this behavior. >> > >> > This may be a bad idea to begin with, and there may be better ways to >> accomplish the same task. However, I would like to understand the behavior >> of such a MR job. >> > >> > Any thoughts? >> > >> > Thank you, >> > Shirish >> > >> > >
