In my work on METRON-608, I’ve found a lot of small but significant bugs in the existing Mpack (version 0.3.0). These bugs integrate with a lot of the oddball tweaks specified in the draft Install docs I saw. I would like to submit a PR for these bugs, but they must be accompanied by changes in the Install doc.
How do we want to manage this? Should we version the Install doc? Or have multiple sections for the different versions? Or have footnotes or sidebar comments about which version certain paragraphs do and do not apply to? I’m inclined to use sidebars within the document, because the dependencies aren’t just on which versions of HDP, Ambari, and Metron you use. They also depend on Python 2.6 vs 2.7, and Centos 6 vs 7. Thanks, --Matt On 12/13/16, 4:13 PM, "[email protected]" <[email protected]> wrote: Sorry about the delay. To be honest I delayed once I saw the management UI PR because I was going to wait for that to be merged into master before I did my mpack install. My initial thought is that there may need to be some sort of a merger between what you provided and the Hortonworks blog post, as I would prefer a single, comprehensive post over fragmentation. Jon On Tue, Dec 13, 2016, 18:29 Dima Kovalyov <[email protected]> wrote: > Thank you Jon, > > Just wondering if you have finished polishing the document? I will have > some time this week to polish and publish it if there is anything left > to format. > > - Dima > > On 12/01/2016 04:41 PM, [email protected] wrote: > > While I can't delegate access to the wiki, I am working on migrating the > > documentation over. I have been modifying the document for clarity and > > updating up some of the formatting, so please give me some time to do so. > > > > Jon > > > > On Thu, Dec 1, 2016 at 9:33 AM Dima Kovalyov <[email protected]> > > wrote: > > > >> Hello, > >> > >> Who can provide me with an access to add new page in this Installation > >> space below? > >> https://cwiki.apache.org/confluence/display/METRON/Installation > >> I want to add up-to-date article on how-to install Metron using Ambari > >> mpack. > >> Please advise. > >> > >> My username is "dima" registered with electronic mail dimdroll hosted by > >> gmail. > >> > >> Right now I have it posted here: > >> https://goo.gl/HWGwpj > >> But instead of maintaining it there it will be convenient for me to have > >> it in confluence instead of gdocs as I am using confluence in our > internal > >> project as well, so I can basically copy formatted doc between each > other. > >> > >> Thank you. > >> > >> - Dima > >> > >> On 11/30/2016 04:57 PM, [email protected]<mailto:[email protected]> > wrote: > >> Probably the best long term place for this would be here< > >> https://cwiki.apache.org/confluence/display/METRON/Installation>, > >> potentially with a new mpack article (to differentiate from the ambari > >> install which is already there, but dated). I can help with migrating > it > >> to the wiki - I will run through your documentation later today and > request > >> edit permissions if I find anything worth updating. > >> > >> Would you be OK if I put your doc in the wiki at some point? > >> > >> Jon > >> > >> On Wed, Nov 30, 2016 at 9:51 AM Dima Kovalyov <[email protected] > >> <mailto:[email protected]>> wrote: > >> As multiple users asked I have published document for installing Metron > on > >> Centos 6 using Ambari 2.4 with HDP 2.5: > >> https://goo.gl/HWGwpj > >> It may look ugly as this is direct export from internal confluence with > >> cosmetic changes and our internal details removed. Let me know if there > is > >> a better place to publish it. > >> > >> Feel free to request edit permissions, I would like the document to be > >> up-to-date, simplified and corrected if there are any errors. The goal > for > >> this document is to provide installation guidance in a manner that Linux > >> administrator with basic knowledge could perform installation without > >> opening any other links or searching for commands, step by step. > >> Thank you. > >> > >> p.s. feel free to let me know if you will encounter any errors as I've > >> seen them bunch. > >> > >> - Dima > >> > >> > >> > >> On 11/30/2016 04:45 PM, [email protected]<mailto:[email protected]> > wrote: > >> I also have written up a method to retrieve and setup the required > >> packages in my script< > >> > https://github.com/JonZeolla/Development/blob/master/Bash/setupMetron.sh> > >> which could probably be cleaned up and incorporated. > >> > >> Jon > >> > >> On Wed, Nov 30, 2016 at 9:24 AM Nick Allen <[email protected]<mailto: > >> [email protected]>> wrote: > >> In regards to your comment about a "full package version dump" there is > at > >> least a start at `metron-deployment/scripts/platform-info.sh`. If we > can > >> enhance that, let's do it. > >> > >> On Wed, Nov 30, 2016 at 4:04 AM, shoggi <[email protected]<mailto: > >> [email protected]>> wrote: > >> Hi > >> > >> Just to wrap this up. I've tried to deploy it on HDP 2.5 but I got the > >> exact same problem. What I did was removing all components for 2.4.3.0 > and > >> then created a new cluster based on 2.5. I left it at that as the most > >> likely issue could be with my base install. May that be wrong version of > >> something or even a missing dev package. > >> > >> As my work also coincided with the announcement by Casey I went ahead > and > >> installed 0.3.0 as a dev installation instead. That worked great as the > >> dev-full-platform guide is very complete. > >> > >> As a general feedback, in order to get more people interested and not > >> having noobs like me come up with redundant questions - At this stage > also > >> sorry for not providing a meaningful subject, I simply was too excited > :) > >> > >> There is much to win with a small extension of the installation guide. I > >> don't think it needs much more prose around but a full package version > dump > >> with every new release would help greatly to get things going. For the > >> things which need to be compiled in, the versioning information should > be > >> enough. I am saying this, as everyone starts with a different base > system. > >> Some have the chance to start from scratch, some have a system running > and > >> build something on top and then there are the experienced Metron users, > who > >> have many base requirements.. they tend to forget. And I believe at the > end > >> it always fails just because of some very tiny small difference. > >> > >> If there is a space somewhere available, this could even be a spot, > where > >> people can say, "hey, I got this running on XYZ and here is the package > >> version dump". A simple table with 'Metron version', 'OS release', > 'Package > >> list file', 'Version Details of every other needed to-be-compiled > package', > >> is already a big help. This really could supplement the existing guide > >> found here: > >> > https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster > >> . > >> > >> But then, I could be alone and others did not experience the same and > >> simply got it to run, I don't know. Anyway, I would be happy to help if > >> help is needed. > >> > >> Cheers > >> Shoggi > >> > >> On Sun, Nov 27, 2016 at 3:00 PM, shoggi <[email protected]<mailto: > >> [email protected]>> wrote: > >> Hi Simon > >> > >> Thank you for your very quick response. I installed it on HDP 2.4.3.0. > >> > >> I had a 0.2.0 instance running and was very conscious about versioning > the > >> various components. Somehow I had the understanding of not touching 2.5 > as > >> of yet. I give that a try. > >> > >> Thanks heaps ! > >> > >> Shoggi > >> > >> On Sun, Nov 27, 2016 at 2:52 PM, Simon Elliston Ball < > >> [email protected]<mailto:[email protected]>> wrote: > >> Hi Shoggi, > >> > >> Can I ask which version of HDP you used as a base. The article is a > little > >> out of date in that is calls for 2.4, which the 0.3.0 release is based > on > >> 2.5. There was a major version difference in storm between the two, > which > >> seems likely to have caused the problem you're seeing. > >> > >> Simon > >> > >> Sent from my iPad > >> > >> On 27 Nov 2016, at 13:48, shoggi <[email protected]<mailto: > [email protected]>> > >> wrote: > >> > >> Hi all > >> > >> I followed the installation guide to deploy Metron via Ambari > >> > https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html > >> . > >> > >> I ended up with all but the Metron Components (Enrichment, Parsing, > >> Parsers) to start up. The closest to pinpoint the issue is, when I try > to > >> start any new parser topology such as with bro: > >> > >> /usr/metron/0.3.0/bin/start_parser_topology.sh -k x.localdomain:6667 -z > >> x.localdomain:2181 -s bro > >> > >> The error I get is: > >> > >> Exception in thread "main" java.lang.BootstrapMethodError: > >> java.lang.NoClassDefFoundError: org/apache/storm/Config > >> > >> Or in more details: > >> > >> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= > >> -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.3.0-227/storm > >> -Dstorm.log.dir=/var/log/storm > >> > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> -Dstorm.conf.file= -cp > >> > /usr/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/clojure-1.6.0.jar:/usr/hdp/ > >> 2.4.3.<http://2.4.3 > >> > .>0-227/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/hadoop-auth-2.7.1.2.4.3.0-227.jar:/usr/hdp/ > >> 2.4.3.<http://2.4.3 > >> > .>0-227/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/ > >> 2.4.3.<http://2.4.3 > >> > .>0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.3.0-227/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> 2.4.3.<http://2.4.3 > >> > .>0-227/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.3.0-227/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/minlog-1.2.jar:/usr/hdp/ > >> 2.4.3.<http://2.4.3 > .>0-227/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.3.0-227/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.3.0-227/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.3.0-227/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.3.0-227/storm/lib/storm-core-0.10.0.2.4.3.0-227.jar:/usr/hdp/2.4.3.0-227/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.3.0-227/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.3.0-227/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.3.0-227/storm/lib/zookeeper.jar:/usr/metron/0.3.0/lib/metron-parsers-0.3.0-uber.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.4.3.0-227/storm/bin > >> -Dstorm.jar=/usr/metron/0.3.0/lib/metron-parsers-0.3.0-uber.jar > >> org.apache.metron.parsers.topology.ParserTopologyCLI -k > x.localdomain:6667 > >> -z x.localdomain:2181 -s bro > >> Exception in thread "main" java.lang.BootstrapMethodError: > >> java.lang.NoClassDefFoundError: org/apache/storm/Config > >> at > >> > org.apache.metron.parsers.topology.ParserTopologyCLI$ParserOptions.<init>(ParserTopologyCLI.java:199) > >> at > >> > org.apache.metron.parsers.topology.ParserTopologyCLI$ParserOptions.<clinit>(ParserTopologyCLI.java:47) > >> at > >> > org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:266) > >> Caused by: java.lang.NoClassDefFoundError: org/apache/storm/Config > >> ... 3 more > >> Caused by: java.lang.ClassNotFoundException: org.apache.storm.Config > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >> ... 3 more > >> > >> Must be something wrong with my config, possibly I have overlooked > >> something but can't put my finger on it. Any help is appreciated. > >> > >> Many thanks > >> Shoggi > >> > >> > >> > >> > >> > >> -- > >> Nick Allen <[email protected]<mailto:[email protected]>> > >> -- > >> > >> Jon > >> > >> Sent from my mobile device > >> > >> -- > >> > >> Jon > >> > >> Sent from my mobile device > >> > >> -- > > Jon > > > > Sent from my mobile device > > > > -- Jon Sent from my mobile device
