I don’t see any project especially tuned for Hive doing what you described. I have encountered this problem recently as the number of users and queries grew exponentially in my company and I’m interested.
We are currently collecting Hive internal metrics in order to do certain analysis (don’t know what yet) in order to suggest better settings and/or better querying pattern for our users. Mostly involving really large queries that cause OOM error. Hive also has an existing optimizer called cost-based optimizer (CBO) that can perform query rewrite (mostly joins) to speed up queries based on table/column statistics. Another feature that could be beneficial is to identify common pattern of existing queries to suggest a materialized view to build (also a new feature of Hive 3.0). I think the Hive team is planning on this supporting feature on the road map as well. On Wed, Jul 25, 2018 at 3:27 PM Johannes Alberti <johan...@altiscale.com> wrote: > Did you guys already look at Dr Elephant? > > > https://engineering.linkedin.com/blog/2016/04/dr-elephant-open-source-self-serve-performance-tuning-hadoop-spark > > Not sure if there is anything you might find useful, but I would be > interested in hearing about good and bad about Dr Elephant w/ Hive. > > Sent from my iPhone > > On Jul 25, 2018, at 12:13 PM, Zheng Shao <zsh...@gmail.com> wrote: > > Hi, > > I am interested in working on a project that takes a large number of Hive > queries (as well as their meta data like amount of resources used etc) and > find out common sub queries and expensive query groups etc. > > Are there any existing work in this domain? Happy to collaborate as well > if there are shared I interests. > > Zheng > > -- Thai
