On Fri, Mar 17, 2017 at 2:56 PM, hernan saab <hernan_javier_s...@yahoo.com> wrote:
> I have been in a similar world of pain. Basically, I tried to use an > external Hive to have user access controls with a spark engine. > At the end, I realized that it was a better idea to use apache tez instead > of a spark engine for my particular case. > > But the journey is what I want to share with you. > The big data apache tools and libraries such as Hive, Tez, Spark, Hadoop , > Parquet etc etc are not interchangeable as we would like to think. There > are very limited combinations for very specific versions. This is why tools > like Ambari can be useful. Ambari sets a path of combos of versions known > to work and the dirty work is done under the UI. > > More often than not, when you try a version that few people tried, you > will get error messages that will derailed you and cause you to waste a lot > of time. > > In addition, this group, as well as many other apache big data user > groups, provides extremely poor support for users. The answers you usually > get are not even hints to a solution. Their answers usually translate to > "there is nothing I am willing to do about your problem. If I did, I should > get paid" in many cryptic ways. > > If you ask your question to the Spark group they will take you to the Hive > group and viceversa (I can almost guarantee it based on previous > experiences) > > But in hindsight, people who work on this kinds of things typically make > more money that the average developers. If you make more $$s it makes sense > learning this stuff is supposed to be harder. > > Conclusion, don't try it. Or try using Tez/Hive instead of Spark/Hive if > you are querying large files. > > > > On Friday, March 17, 2017 11:33 AM, Stephen Sprague <sprag...@gmail.com> > wrote: > > > :( gettin' no love on this one. any SME's know if Spark 2.1.0 will work > with Hive 2.1.0 ? That JavaSparkListener class looks like a deal breaker > to me, alas. > > thanks in advance. > > Cheers, > Stephen. > > On Mon, Mar 13, 2017 at 10:32 PM, Stephen Sprague <sprag...@gmail.com> > wrote: > > hi guys, > wondering where we stand with Hive On Spark these days? > > i'm trying to run Spark 2.1.0 with Hive 2.1.0 (purely coincidental > versions) and running up against this class not found: > > java.lang. NoClassDefFoundError: org/apache/spark/ JavaSparkListener > > > searching the Cyber i find this: > 1. http://stackoverflow.com/ questions/41953688/setting- > spark-as-default-execution- engine-for-hive > <http://stackoverflow.com/questions/41953688/setting-spark-as-default-execution-engine-for-hive> > > which pretty much describes my situation too and it references this: > > > 2. https://issues.apache.org/ jira/browse/SPARK-17563 > <https://issues.apache.org/jira/browse/SPARK-17563> > > which indicates a "won't fix" - but does reference this: > > > 3. https://issues.apache.org/ jira/browse/HIVE-14029 > <https://issues.apache.org/jira/browse/HIVE-14029> > > which looks to be fixed in hive 2.2 - which is not released yet. > > > so if i want to use spark 2.1.0 with hive am i out of luck - until hive > 2.2? > > thanks, > Stephen. > > > > > Stephan, I understand some of your frustration. Remember that many in open source are volunteering their time. This is why if you pay a vendor for support of some software you might pay 50K a year or $200.00 an hour. If I was your vendor/consultant I would have started the clock 10 minutes ago just to answer this email :). The only "pay" I ever got from Hive is that I can use it as a resume bullet point, and I wrote a book which pays me royalties. As it relates specifically to your problem, when you see the trends you are seeing it probably means you are in a minority of the user base. Either your doing something no one else is doing, you are too cutting edge, or no one has an easy solution. Hive is making the move from the classic MapReduce, two other execution engines have been made Tez and HiveOnSpark. Because we are open source we allow people to "scratch an itch" that is the Apache way. From time to time in means something that was added stops being viable because of lack of support. I agree with your final assessment which is Tez is the most viable engine for Hive. This is by no means a put down of the HiveOnSpark work and it does not mean it will never the most viable. By the same token if the versions fall out of sync and all that exists is complains the viability speaks for itself. Remember that keeping two fast moving things together is no easy chore. I used to run Hive + cassandra. Seems easy, crap two versions of common CLI, shade one version everything works, crap new hive release has different versions of thrift, shade + patch, crap now one of the other dependencies is incompatible fork + shade + patch. At some point you have to say to yourself if I can not make critical mass of this solution such that I am the only one doing/patching it then I give up and find some other way to do it.
