Re: SQOOP_URL
Hmmm..
Brefily looking for "sqoop.server.url" brings very little up .
Is this a bleeding edge sqoop parameter?
Sorry, not a sqoop expert.
On Mon, Jul 1, 2013 at 12:19 PM, Roman Shaposhnik wrote:
> On Mon, Jul 1, 2013 at 8:11 AM, Sean Mackrory
> wrote:
> > On Mon, Jul 1, 2013 at 4:00 AM, Jay Vyas wrote:
> >> Hi : Im finding that the pom.xml for the sqoop smoke tests requires a
> >> "SQOOP_URL":
> >>
> >>
> >> org.apache.maven.plugins
> >> enforce-property
> >> SQOOP_URL
> >> ...
> >>
> >> true
> >>
> >>
> >> What is the role this URL plays? Im not finding it used in the code
> >> anywhere.
>
> Not sure what version of Bigtop you're looking at, but if you
> look at 0.6.0/trunk then you'll see this
>
>
>
> ${SQOOP_URL}
> .
>
> (in bigtop-tests/test-execution/smokes/sqoop/pom.xml)
>
> SQOOP_URL is basically used to set sqoop.server.url sysprop
> for the Sqoop2 tests to be able to find the sqoop server.
>
> Thanks,
> Roman.
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: permanant builds on jenkins
Okay thanks for the feedback. On Sun, Jul 7, 2013 at 5:37 PM, Konstantin Boudnik wrote: > [Moving user@ to Bcc: and cross-posting to dev@ This is a more dev > discussion, > let's keep it there] > > I think there's a confusion of some sort here. Apache normally doesn't > provide > binary releases. A release - in ASF - is a source code. While binaries can > be > provided as a convenience artifacts it is most often isn't a case. > > Now, a Bigtop release is a fixed BOM file that states what versions of what > components were built and tested to work against each other. If you are > looking for particular system packages - it is very easy to get by > make rpm|deb > from a particular release tag. I believe we do publish binary packages for > each release, but you should understand that these are stored on s3 that > costs > money. Right now, BT build infrastructure is running on the funds donated > by > commercial vendors (Cloudera is the only one so far, others are chiming in > as > we speak). > > Same goes for a VM: if you need a VM with particular release bits in it: > you > just check out the tag and run boxgrinder that will create identical VM > each > time you run the build. The VM in question isn't meant for development of > BT. > This is a mere test-bench for people who want to try the latest release of > Hadoop based stack and have no skills of going through the whole process by > themselves. > > I wish we could keep a copy of VMs for all stable builds, but this is a > simple > matter of the costs vs. convenience. > > As for #3 below: this is pretty much how our Jenkins infra works. There's a > bunch of slave VMs that are used for package > building/installation/testing. If > you have some improvements in mind - feel free to open a JIRA tickets and > contribute a patch/design document/etc. Any help is highly welcome. > > Cos > > On Sun, Jul 07, 2013 at 12:03PM, Jay Vyas wrote: > > Hi bruno: > > > > 1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is > not > > valid but "https://builds.apache.org/job/Bigtop-trunk/"; seems to work. > Is > > this an error on the website? > > > > 2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to > build > > direct from HEAD. > > > > Are there actual bigtop releases which aren't on jenkins? Everything > seems > > to be centered around the build server, rather than actual frozen, > > downloadable releases. > > > > 3) Regarding testing the VMs. Maybe to start some simple virt-install > > scripts could help confirm that the KVM builds, at least, are working. > If > > we could get them working with static IPs then we could even clone down > the > > bigtop source code and run bigtop hadoop tests on the VMs after > > construction as a validation step. > > > > > > > > > > On Sat, Jul 6, 2013 at 8:58 PM, Bruno MahИ wrote: > > > > > On 07/06/2013 07:33 AM, Jay Vyas wrote: > > > > > >> Hi bigtop: > > >> > > >> Are there any permanant builds saved on jenkins (for the VM matrix)? > > >> > > >> If not it would be nice to add them for certain known well tested, > > >> working disk images . > > >> > > >> (for context, I'm currently running Mr2 build of the KVM box and it > > >> appears to have some intermittent write issues on the DataNode path, > and > > >> also, my namenode appears to really like being in safe mode. these > > >> could just be due to VM setup though, as im changing some things like > > >> adding static IPs and data node write paths... so nothing to be > alarmed > > >> about.) > > >> > > >> -- > > >> Jay Vyas > > >> http://jayunit100.blogspot.com > > >> > > > > > > Hi Jay, > > > > > > Could you defined "permanent build" ? > > > I am not sure if this fits your requirement, but jenkins has a link to > the > > > latest successful build (ex: http://bigtop01.cloudera.org:** > > > 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=** > > > fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz< > http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz > >) > > > > > > We do not store convenient artifacts of VMs since no one has asked > about > > > it before. > > > So ideally, known well tested working dis
Re: [PROPOSAL] EOL'ed releases 0.3.x
For me at least --- the 1.X stuff is very important in the community... or at least... in my community :) Am i missing something? maybe i have misinterpretted the proposal. On Tue, Aug 6, 2013 at 12:22 PM, Konstantin Boudnik wrote: > The reason I have brought up the branch is that it doesn't have anything of > value except for 0.3.1 specific things. But I have no problem with it > laying > around ;) > > Cos > > On Tue, Aug 06, 2013 at 09:04AM, Roman Shaposhnik wrote: > > On Tue, Aug 6, 2013 at 8:29 AM, Sean Mackrory > wrote: > > > No objection from me - although like Bruno, I don't see why we should > > > delete the branch. Even if we have no intention of focussing further > > > work on that line, it might be nice to keep it around for anyone who > > > may want to build some stuff from it. > > > > Same here. +1 to cleaning up the JIRA and as for the branch -- we can > > just let it linger I suppose. > > > > Thanks, > > Roman. > -- Jay Vyas http://jayunit100.blogspot.com
Re: [PROPOSAL] EOL'ed releases 0.3.x
Thanks roman - its a good point you make. certainly i have more pressing matters to attend also, and yes i guess your right, in the near future 2.x is going to be more important. so... i guess i ll just keep playing on the bigtop patches im already interested in / supposed to be looking at :)
Re: Terasort/Teragen in smokes
Hi bigtop! The 1057 patch is now up to date with teragen/terasort benchmarks. See https://issues.apache.org/jira/browse/BIGTOP-1057 . Let me know if this looks okay. Later we can open another ticket for the YCSB benchmarks which will also be a great, and simple smoke to add for smoke testing HBase. On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik wrote: > On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote: > > hmm ok. now. Thinking about teragen makes me think of benchmarking.. > > > > In the longer term we could add benchmarking jobs to all the submodules > not > > just mapreduce. For example there are hi bench and ycsb workloads which > > might be usable or pulled in as bigtop components ... Iff of course > > benchmarking is in the cards for bigtop? > > It indeed is! > > I think doing tera-gen/sort so it can be parameterized will provide a good > basis for future benchmarking (as a bit of reflection: I have did > simplistic > yet efficient way of benchmarking HDFS and MR a couple years ago, but my > employer back then has never let it go into the open. Go figure...) > > And I have a way of building YCSB against a particular version of Hadoop, > so I > guess I will have it packaged as a benchmarking test pretty soon. > > Cos > > > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik wrote: > > > > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas > wrote: > > >> Hi guys: > > >> > > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts. > > >> > > >> Any interest in these as an update to TestHadoopExamples in the > MapReduce > > >> smokes? > > >> > > >> If so I could patch them in :) > > > > > > Sure! Sounds like a useful addition. > > > > > > Thanks, > > > Roman. > -- Jay Vyas http://jayunit100.blogspot.com
Re: Project name conflict heads-up
Why not call it bigtop-spark .. ? Seems to me like nobody but the root project itself should monopolize the hadoop distro primary names since there are so many builds out there... Right? > On Sep 23, 2013, at 4:11 PM, Konstantin Boudnik wrote: > > I'd say Fedora needs to deal with it on their end, considering that they are > coming into this post-Bigtop. Thoughts? > >> On Mon, Sep 23, 2013 at 11:43AM, Sean Mackrory wrote: >> I've been worried about this for a while, and I think we need to come up >> with a strategy for dealing with it in Bigtop. In this case, it's just a >> coincidence that two separate projects are named Spark. However I recently >> submitted a patch for packaging Avro as a separate component in Bigtop and >> had to deal with the fact that a minority of the systems we support already >> packaged part of Avro in a package named 'avro', so I had to use >> 'avro-libs', 'avro-tools' and 'avro-doc'. Fedora's also going to start >> releasing packages for Hadoop, so this problem of name conflicts is only >> going to get worse. >> >> >> On Mon, Sep 23, 2013 at 10:52 AM, Konstantin Boudnik wrote: >> >>> Let's call it spark-asf, perhaps? >>> On Mon, Sep 23, 2013 at 09:57AM, Roman Shaposhnik wrote: Hi! While testing Spark as part of the Bigtop I've come across the following conflict on Debian-based systems: $ apt-cache show spark Package: spark Priority: optional Section: universe/devel Description-en: SPARK programming language toolset SPARK is a formally-defined computer programming language based on the Ada programming language, intended to be secure and to support the development of high integrity software used in applications and systems where predictable and highly reliable operation is essential either for reasons of safety or for business integrity. . This package contains the tools necessary for checking if programs >>> adhere to the SPARK rules and the tools to show freedom of runtime exceptions >>> in those programs. To compile SPARK programs use any standards-compliant >>> Ada compiler, such as GNAT. Homepage: http://libre.adacore.com/libre/tools/spark-gpl-edition/ This has an obvious ramifications for Bigtop packaging, but it may also have ramifications for PODLINGNAMESEARCH for Apache Spark (incubating). Speaking of which -- has anybody else started PODLINGNAMESEARCH already? Thanks, Roman. >>>
Re: Project name conflict heads-up
definetly +1 to append the bigtop suffix to it. if everyone did this their wouldnt be a namespace conflict to begin with :) On Tue, Sep 24, 2013 at 2:27 PM, Konstantin Boudnik wrote: > Yup, -bigtop is even better! > Cos > > On Tue, Sep 24, 2013 at 12:09AM, Bruno Mahé wrote: > > I agree with Cos that a suffix makes it easier to find. > > But I would lean toward "-bigtop" since "-asf" may be vague in some > > cases (ie. what if Apache Hadoop starts to also name its own packages > > hadoop-asf?). > > Using "-bigtop" also makes the brand/project more recognizable and > > easier to look for in search engines. > > > > > > Thanks, > > Bruno > > > > > > On 09/23/2013 06:16 PM, Konstantin Boudnik wrote: > >> Sorta what I suggested with 'spark-asf'. It'd better than 'apache-' > prefix, > >> because it would be easier to find, sorta... > >> > >> Cos > >> > >> On Mon, Sep 23, 2013 at 04:37PM, Peter Linnell wrote: > >>> Putting on my Linux distro hat... I've long been involved with > >>> openSUSE... > >>> > >>> I'd say apache-spark would be best, but this adds minor baggage to > >>> packaging. > >>> > >>> Thoughts ? > >>> > >>> Thanks, > >>> Peter > >>> > >>> On Mon, 23 Sep 2013 16:01:32 -0700 > >>> Konstantin Boudnik wrote: > >>> > >>>> On Mon, Sep 23, 2013 at 06:24PM, Jay Vyas wrote: > >>>>> Why not call it bigtop-spark .. ? > >>>> > >>>> Well, because it isn't bigtop's spark, to start with ;) > >>>> > >>>>> Seems to me like nobody but the root project itself should > >>>>> monopolize the hadoop distro primary names since there are so many > >>>>> builds out there... Right? > >>>> > >>>> What's is root project, sorry? > >>>> > >>>>>> On Sep 23, 2013, at 4:11 PM, Konstantin Boudnik > >>>>>> wrote: > >>>>>> > >>>>>> I'd say Fedora needs to deal with it on their end, considering > >>>>>> that they are coming into this post-Bigtop. Thoughts? > >>>>>> > >>>>>>> On Mon, Sep 23, 2013 at 11:43AM, Sean Mackrory wrote: > >>>>>>> I've been worried about this for a while, and I think we need to > >>>>>>> come up with a strategy for dealing with it in Bigtop. In this > >>>>>>> case, it's just a coincidence that two separate projects are > >>>>>>> named Spark. However I recently submitted a patch for packaging > >>>>>>> Avro as a separate component in Bigtop and had to deal with the > >>>>>>> fact that a minority of the systems we support already packaged > >>>>>>> part of Avro in a package named 'avro', so I had to use > >>>>>>> 'avro-libs', 'avro-tools' and 'avro-doc'. Fedora's also going to > >>>>>>> start releasing packages for Hadoop, so this problem of name > >>>>>>> conflicts is only going to get worse. > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Sep 23, 2013 at 10:52 AM, Konstantin Boudnik > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Let's call it spark-asf, perhaps? > >>>>>>>> > >>>>>>>>> On Mon, Sep 23, 2013 at 09:57AM, Roman Shaposhnik wrote: > >>>>>>>>> Hi! > >>>>>>>>> > >>>>>>>>> While testing Spark as part of the Bigtop I've come > >>>>>>>>> across the following conflict on Debian-based systems: > >>>>>>>>> > >>>>>>>>> $ apt-cache show spark > >>>>>>>>> Package: spark > >>>>>>>>> Priority: optional > >>>>>>>>> Section: universe/devel > >>>>>>>>> Description-en: SPARK programming language toolset > >>>>>>>>> SPARK is a formally-defined computer programming language > >>>>>>>>> based on the Ada programming language, intended to be secure > >>>>>>>>> and to support the development of high integrity software used > >>>>>>>>> in applications and systems where predictable and highly > >>>>>>>>> reliable operation is essential either for reasons of safety > >>>>>>>>> or for business integrity. . > >>>>>>>>> This package contains the tools necessary for checking if > >>>>>>>>> programs > >>>>>>>> adhere > >>>>>>>>> to the SPARK rules and the tools to show freedom of runtime > >>>>>>>>> exceptions > >>>>>>>> in > >>>>>>>>> those programs. To compile SPARK programs use any > >>>>>>>>> standards-compliant > >>>>>>>> Ada > >>>>>>>>> compiler, such as GNAT. > >>>>>>>>> Homepage: > >>>>>>>>> http://libre.adacore.com/libre/tools/spark-gpl-edition/ > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> This has an obvious ramifications for Bigtop packaging, but > >>>>>>>>> it may also have ramifications for PODLINGNAMESEARCH > >>>>>>>>> for Apache Spark (incubating). > >>>>>>>>> > >>>>>>>>> Speaking of which -- has anybody else started PODLINGNAMESEARCH > >>>>>>>>> already? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Roman. > >>>>>>>> > >>> > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: November hackathon
i'd love to do one on the east coast? OR is everone in CA? On Mon, Nov 11, 2013 at 1:18 PM, Konstantin Boudnik wrote: > Guys, > > I want to host another Bigtop hackaton at WANdisco in San Ramon. I have > 11/23 > or 12/7 in mind. Would you please reply with your preferred date and RSVP? > > -- > Take care, > Cos > > -- Jay Vyas http://jayunit100.blogspot.com
Re: November hackathon
Sounds good :) any other remote attendees?
Re: don't support hadoop1 in bigtop after 0.4
1) regarding smoke tests and backwards compatibility: Not sure what components of bigtop your referring to , but FYI the smoke tests on head, or at least a solid subset of them, work enough to test a cluster against mahout, mapreduce, sqoop on hadoop 1.x . - For pig, hive, we are just running shell scripts in the test-execution for those. - for mahout, we run all tests but BayesNewsgroupClassifier - for mapreduce, we run the terasort/teragen, wordcount, grep, and sleep tests. 2) like sean sais.. I assume that the bigtop head dudes would agree with me that, even though alot of bigtop work is driven by vendor needs, but since its an apache project, those of us in the broader community can continue adding patches to it to support our needs as long as they are synergistic with the overall project goals ; do feel free to ping me or create a jira for 1.x wish lists maybe i might share some of those with you and maybe could even help on a patch On Thu, Nov 14, 2013 at 11:21 AM, Sean Mackrory wrote: > Supporting both hadoop1 and hadoop2 in the same distribution of all the > other components is a rather massive undertaking. As a rule, Bigtop limits > itself to being a downstream distribution point (we do not apply our own > patches to the upstream releases) - this would make it even more difficult > to achieve compatibility across all the projects. There was an effort to > release an updated hadoop1 distribution based on Bigtop 0.3 (0.3.1), > however that was not finished. There's nothing I know if to stop a > developer from completing that effort if they wanted to. > > > On Thu, Nov 14, 2013 at 1:39 AM, pengwenwu2008 >wrote: > > > Hi All, > > > > Anyone can help answer "are there any reason don't support hadoop1 in > > bigtop after 0.4"? > > Thanks a lot for any comment !! > > > > Regards, > > Wenwu,Peng > -- Jay Vyas http://jayunit100.blogspot.com
A couple of JIRAs
Hi bigtop ! Just updated a couple of patches need some love :) anyone want to take a look? 1) Mahout updates (fixed urls, modularized downloads for easier debugging, and integer param for number of factorizer iterations for faster smoke tests). https://issues.apache.org/jira/browse/BIGTOP-1128 2) Alot of README updates (made it a README.md file so it looks pretty on github, also tried to explain the package structure alittle more so that bigtop isnt as intimidating to newcomers). https://issues.apache.org/jira/browse/BIGTOP-1145 -- Jay Vyas http://jayunit100.blogspot.com
Reminder on vagrant patch --- thoughts?
hi bigtop ? I'd like to get some initial review on https://issues.apache.org/jira/browse/BIGTOP-1072 soon :) Would love to get an initial merge of it. After that, I will try to embed the smoke tests into it as well, so that bigtop smokes can be auto-tested in a fresh bigtop VM. -- Jay Vyas http://jayunit100.blogspot.com
Ping: Vagrant and Docker
Hi again bigtop ! ** just a freindly reminder that i have some outstanding work on vagrant which i'd like to put into bigtop :) + a new JIRA for docker if anyone wants to collaborate on dockerizing bigtop environment runs. ** - im pretty excited about this because, next iteration, ill put all the smoke tests in, so that we can run all smoke tests from zero , super useful for developing and testing the existing smokes (maybe as part of the bigtop build). 1) I've put in an initial shell based vagrant bigtop recipe: https://issues.apache.org/jira/browse/BIGTOP-1072 , still needs review ; --- Sorry to keep bothering about this, but i know everyone gets busy and sometimes needs a reminder to review incoming lower priority stuff :) --- 2) Also, created this initial stab at a docker JIRA , see https://issues.apache.org/jira/browse/BIGTOP-1154. Let me know if any thoughts on either of these (particularly (1) :) :) :)) -- Jay Vyas http://jayunit100.blogspot.com
Re: Documentation overhaul
This is a good idea. Maybe The website needs to be modernized and integrated with these sort of docs also with CSS. ? > On Dec 9, 2013, at 12:35 AM, Bruno Mahé wrote: > > Hi, > > As you are all probably aware, documentation is not our forte. > In order to help with that, I have been thinking about introducing books > similar to the Apache HBase reference guide. > But given how wide is Apache Bigtop, having a single reference guide would > not make sense. So I have come up with 3 books: > * User guide - hand hold users into installing and configuring an Apache > Bigtop installation. Including packages directly, puppet recipes and all the > system (users, locations, ec2...) and tooling around such installation > (firewall, monitoring, logging...) > * Developer guide - All the inner details about contributing packages, tests > and customizing VMs and Apache Bigtop (build my own stack) > * Cookbook - Step by step instructions to reach a given goal (run hive > queries with an hbase backend, logstash...) > > For now I will solely focus on the first one. I have prepared BIGTOP-1157 to > build an HTML and PDF version of the user guide as part of our website. > It is for now empty but here is a proposal for its content: > > * Preface > > * Introduction [What is Apache Bigtop and why should I care] > > * What is new in Apache Bigtop [features highlights] > ** What is new in Apache Bigtop 0.9.0 > ** Upgrading from Apache Bigtop 0.8.0 to Apache Bigtop 0.9.0 > ** What is new in Apache Bigtop 0.8.0 > ** Upgrading from Apache Bigtop 0.7.0 to Apache Bigtop 0.8.0 > > * What is an application and the need for packages [an application neeeds > users, files, configuration, permissions, ulimits, ports...] > > * Lifecycle of applications > ** Lifecycle of an application [init scripts, logs] > ** Lifecycle of a distributed application [coordination, zookeeper...] > > * GNU/Linux distributions supported by Apache Bigtop > ** centos[presentation, what makes it distinct, how to to main tasks > (install, upgrade, remove packages, restart services, tweak firewall, where > are logs...), setup bigtop repositories] > ** Fedora [presentation, what makes it distinct, how to to main tasks > (install, upgrade, remove packages, restart services, tweak firewall, where > are logs...), setup bigtop repositories] > ** OpenSUSE [presentation, what makes it distinct, how to to main tasks > (install, upgrade, remove packages, restart services, tweak firewall, where > are logs...), setup bigtop repositories] > ** Debian [presentation, what makes it distinct, how to to main tasks > (install, upgrade, remove packages, restart services, tweak firewall, where > are logs...), setup bigtop repositories] > ** Ubuntu [presentation, what makes it distinct, how to to main tasks > (install, upgrade, remove packages, restart services, tweak firewall, where > are logs...), setup bigtop repositories] > > > for project in [Hadoop, HBase, Zookeeper, Flume...]: > * Apache > ** Introduction > ** Installation > ** Configuration > ** Pseudo distribution > ** Distributed > ** Security > ** Monitoring > ** Integration with Apache Hadoop > ** Integration with other project > ** Upgrade? > ** Life cycle (services) > ** Common issues > > = > > > Thoughts, help and ideas welcome! > > > Thanks, > Bruno
Vagrant patch updated
Hi bigtop ! I've rerolled the vagrant patch and think it now satisfies the original comments a little better: - simpler: (only runs "calculate pi 2 2" as the test). - pure dependencies: doesnt force using the vagrant cachier - upgraded to use hostname instead of host_name (that was a vagrant 2 vs 1 issue), thanks for catching it. https://issues.apache.org/jira/browse/BIGTOP-1072 There are now two patches, the second one is the new one: https://issues.apache.org/jira/secure/attachment/12617596/BIGTOP-1072.2.patch -- Jay Vyas http://jayunit100.blogspot.com
Re: December hackathon
cool ... so what are the hack projects? any proposals yet? or just tackling some jiras.? thanks for the remote setup : us outside the west coast would definetly like to get in on the action!
User synchronization, multiuser apps, puppet, ldap...
Hi bigtop: One important aspect of hadoop is management of users for YARN and HDFS, and multitenant apps. https://issues.apache.org/jira/browse/BIGTOP-1136 The Security/Authentication stack should probably be integrated somewhere into bigtop, but im not sure how or where to start. Maybe some initial comments on the above JIRA would help. I've written up one simple way to unifie UID/GIDs in using the JumpBox EC2 appliance here : http://jayunit100.blogspot.com/2013/12/an-easy-way-to-centralize.html . Any thoughts on wether or not a canned user management service should or should not be a part of bigtop? -- Jay Vyas http://jayunit100.blogspot.com
Re: proper approach to smoke tests development
Hi mikhail ! Regarding the need for a cluster, IMO its imperative to run in some kind of cluster, The reason is that: 1) You want your test to transfer a file into a RDBMS, and you want it to test that the distributed cluster is actually breaking the work up properly. 2) Also we would want to confirm that the cluster is able to distribute exporting (copying from the distributed file system into a rdbms) as well. TL;DR : Definetly validate your sqoop smokes in a distributed cluster for sqoop.It can be couple of VMs ... nothing fancy, but more than one machine. For other tests (like wordcount, for example) I guess testing on a single node is probably a little more acceptable. ** If you dont have a cluster just point me at a patch ! we can test it for you .. ** --- Now regarding sqoop tests: By the way ... Are you working on BIGTOP-1019? If so thanks ! One possible implementation is using HSQL in file mode, and then putting the file as a locally readable file on all machines. I havent figured out a way to do that in hadoop with hdfs yet, but in gluster, we typically FUSE mount, so I have a very simple smoke test that works nicely for sqoop on gluster... you might be interested to check it out if you can find a way to make a single file locally available to all nodes on a hadoop cluster, it will work (and be easy to maintain : no network connection required for the JDBC stuff - just pure JDBC ETL)... Its shell scripted here (but we could port the shell commands easily to Itest commands in groovy)... https://github.com/jayunit100/bigtop/blob/master/bigtop-tests/test-artifacts/sqoop/src/main/resources/test2.sh. - You can see that it runs hsql in "file" mode, and stores the database as a file in a gluster mounted directory. -That directory is available to every node of my cluster .. - So that is actually a very concrete example of how, if i didnt run the test in distributed mode, and there was a bug in my code On Sun, Jan 12, 2014 at 8:37 PM, Mikhail Antonov wrote: > Hello everyone, > > I have one question on the "proper" approach to test development. While > working on (for example) tests for Sqoop (migrating them from mysql to > mocked jdbc driver), what's the "right" approach for the iteration? > > Say I have host machine with Fedora, where I have build environment and > build Bigtop. Shall I have 1 manually-created VM with minimal CentOS, where > I copy built rpms, install them, and run smokes (or generate VM in > automated way)? Or shell I have cluster of VMs to run fullly-distributed > cluster? Or is it more or less fine to just run smokes on host machine? > > -- > Thanks, > Mikhail Antonov > -- Jay Vyas http://jayunit100.blogspot.com
Bigtop based apps: how to get maven deps right?
Hi bigtop. If I want to build an app against bigtop Is there a maven repo I can point to that has all the version stuff figured out? As we all know: it's hard to build hadoop apps and test them in a dev environment that matches the classpath of cluster environment... I think bigtop as a distribution might be able to support something like a maven-archetype for bigtop based app development.?
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
Well, on bigpetstore which I'm writing to target bigtop, I have trouble setting up a maven repo with the right dependencies because there are many ecosystem projects. I want to know that my dev environment matches the cluster deployment environment. To do that id like it if there was a maven archetype I could use to build a stub project. I'm actually planning to make bigpetstore into this archetype app So in order to do that I need a systematic way to define the pulled in maven dependencies. Makes sense? > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik wrote: > > Jay, > > I don't think we've been doing bigtop publishing like ever. All use cases that > we had were clearly covered by simple local installation of the artifacts. > > I think it might be a good time to start doing the publishing of the official > Bigtop artifacts to mavencentral. > > Could you get a bit deeper into this whole idea of the applications? I am > interested to hear more ;)! > > Thanks, > Cos > >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote: >> Hi bigtop. >> >> If I want to build an app against bigtop Is there a maven repo I can >> point to that has all the version stuff figured out? >> >> As we all know: it's hard to build hadoop apps and test them in a dev >> environment that matches the classpath of cluster environment... I think >> bigtop as a distribution might be able to support something like a >> maven-archetype for bigtop based app development.?
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
Thanks for all this attention to my issue I'm trying to follow the debate.
My first question:
1) Can you clarify what is meant by "we drive different dependencies off of our
BOM the artifacts end up being different anyway :-("?
Sorry ... Still learning the bigtop vernacular. :).
2) Now in the interim. My current thought is that pig,hive, etc aren't ever
gauranteed to have non conflicting classpaths. And thus any hadoop app which
attempts to integrate them into a single build will have to use stuff like
submodules or profiles.
> On Jan 27, 2014, at 12:24 AM, Mark Grover wrote:
>
>> On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik wrote:
>>
>>> On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover wrote:
>>> Hey Jay,
>>> Currently we don't patch things in Bigtop. That means when we download
>> and
>>> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
>>> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
>>> version as what upstream hadoop released.
>>
>> It is true that we stay away from patching the source, but since we
>> drive different dependencies off of our BOM the artifacts end
>> up being different anyway :-(
>>
>> IOW, an hbase jar from bigtop is NOT the same as published by
>> upstream hbase. In fact it is kind of incompatible. Same goes for
>> most of the other jars with non-trivial dependency chains.
>
> Ah, good point, Roman. Thanks for correcting me.
>
>
>>
>>> So, now 2 options exist for bigpetstore in my opinion:
>>> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
>>> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
>> doesn't
>>> patch, java artifacts should be exactly the same from Bigtop as compared
>> to
>>> Apache Hadoop.
>>> 2. Pull Bigtop artifacts for maven. For this, we will obviously need
>> Bigtop
>>> to a) start updating pom files with its own versioning scheme b) Upload
>>> them to maven central or equivalent.
>>>
>>> As you can see option #2 is a fairly non-trivial overhead for Bigtop but
>> I
>>> would love to hear if you prefer one of the two options and if so why.
>>
>> I think the only real alternative is for us to bite the bullet and start
>> publishing bigtop artifacts.
>>
>> In the ideal world -- we'd be pushing right to Maven central, provided
>> that we're careful about branding the artifacts with an explicit Bigtop
>> version stamp. E.g.
>> org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0
>
> Yeah, I'd be ok with that. The real question is that would we be changing
> our policy to only patch things like version numbers or are we opening up
> the realm of patching in general? Regardless, I think this would be a
> non-trivial cost and we should consider aiming it for later release (0.9?).
>
>>
>> Thanks,
>> Roman.
>
> And, welcome back, Roman!
> Mark
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
just for some color, this is the type of hacking we are doing (See readme)
https://github.com/nsavageJVM/bigpetstoreprofiles
to try to get pig apps to build integration tests in maven. its all
dependency related i beleieve.
So any thoughts on this would be awesome
On Mon, Jan 27, 2014 at 8:43 AM, Jay Vyas wrote:
> Thanks for all this attention to my issue I'm trying to follow the
> debate.
>
> My first question:
>
> 1) Can you clarify what is meant by "we drive different dependencies off
> of our BOM the artifacts end up being different anyway :-("?
>
> Sorry ... Still learning the bigtop vernacular. :).
>
> 2) Now in the interim. My current thought is that pig,hive, etc aren't
> ever gauranteed to have non conflicting classpaths. And thus any hadoop app
> which attempts to integrate them into a single build will have to use stuff
> like submodules or profiles.
>
> > On Jan 27, 2014, at 12:24 AM, Mark Grover wrote:
> >
> >> On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik
> wrote:
> >>
> >>> On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover wrote:
> >>> Hey Jay,
> >>> Currently we don't patch things in Bigtop. That means when we download
> >> and
> >>> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
> >>> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
> >>> version as what upstream hadoop released.
> >>
> >> It is true that we stay away from patching the source, but since we
> >> drive different dependencies off of our BOM the artifacts end
> >> up being different anyway :-(
> >>
> >> IOW, an hbase jar from bigtop is NOT the same as published by
> >> upstream hbase. In fact it is kind of incompatible. Same goes for
> >> most of the other jars with non-trivial dependency chains.
> >
> > Ah, good point, Roman. Thanks for correcting me.
> >
> >
> >>
> >>> So, now 2 options exist for bigpetstore in my opinion:
> >>> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
> >>> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
> >> doesn't
> >>> patch, java artifacts should be exactly the same from Bigtop as
> compared
> >> to
> >>> Apache Hadoop.
> >>> 2. Pull Bigtop artifacts for maven. For this, we will obviously need
> >> Bigtop
> >>> to a) start updating pom files with its own versioning scheme b) Upload
> >>> them to maven central or equivalent.
> >>>
> >>> As you can see option #2 is a fairly non-trivial overhead for Bigtop
> but
> >> I
> >>> would love to hear if you prefer one of the two options and if so why.
> >>
> >> I think the only real alternative is for us to bite the bullet and start
> >> publishing bigtop artifacts.
> >>
> >> In the ideal world -- we'd be pushing right to Maven central, provided
> >> that we're careful about branding the artifacts with an explicit Bigtop
> >> version stamp. E.g.
> >> org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0
> >
> > Yeah, I'd be ok with that. The real question is that would we be changing
> > our policy to only patch things like version numbers or are we opening up
> > the realm of patching in general? Regardless, I think this would be a
> > non-trivial cost and we should consider aiming it for later release
> (0.9?).
> >
> >>
> >> Thanks,
> >> Roman.
> >
> > And, welcome back, Roman!
> > Mark
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
YE FOR GRADLE . On Tue, Jan 28, 2014 at 12:02 PM, Konstantin Boudnik wrote: > On Fri, Jan 24, 2014 at 03:28PM, Mark Grover wrote: > > Hey Jay, > > Currently we don't patch things in Bigtop. That means when we download > and > > include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop > > (say hadoop-common.jar) would have the version 2.2.0 - exactly the same > > version as what upstream hadoop released. > > In the interest of full disclosure, we do patch certain things - just grep > for > sed in the packages source tree. Builds would be one things; we did patch > source on a couple of occasions in the past to deal with some > idiosyncrasies > of the official releases. > > But yes - we frown upon such occasions ;) > > > So, now 2 options exist for bigpetstore in my opinion: > > 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache > > Hadoop artifacts instead of bigtop artifacts. However, since Bigtop > doesn't > > patch, java artifacts should be exactly the same from Bigtop as compared > to > > Apache Hadoop. > > 2. Pull Bigtop artifacts for maven. For this, we will obviously need > Bigtop > > to a) start updating pom files with its own versioning scheme b) Upload > > them to maven central or equivalent. > > > > As you can see option #2 is a fairly non-trivial overhead for Bigtop but > I > > would love to hear if you prefer one of the two options and if so why. > > I think what would be most helpful from the ease of development in the > stack - > and I have stepped on it more than a few times myself - is to be able to do > maven install for the _whole_ project from the top level pom. As of right > now, > one needs to do a bit of rain dance in order to get all the bits in place. > And > that's quite annoying apparently. I guess that'd be the next thing to me to > look into it. > > Just occurred to me, that if package driving Makefile is replaced with > Gradle > that will give a way better consistency of all the parts of the Bigtop > environment and stack. Perhaps, it is my severely under-slept brain is > talking > now ;) > > Cos > > > On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas wrote: > > > > > Well, on bigpetstore which I'm writing to target bigtop, I have trouble > > > setting up a maven repo with the right dependencies because there are > many > > > ecosystem projects. > > > > > > I want to know that my dev environment matches the cluster deployment > > > environment. > > > > > > To do that id like it if there was a maven archetype I could use to > build > > > a stub project. I'm actually planning to make bigpetstore into this > > > archetype app So in order to do that I need a systematic way to > define > > > the pulled in maven dependencies. > > > > > > Makes sense? > > > > > > > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik > wrote: > > > > > > > > Jay, > > > > > > > > I don't think we've been doing bigtop publishing like ever. All use > > > cases that > > > > we had were clearly covered by simple local installation of the > > > artifacts. > > > > > > > > I think it might be a good time to start doing the publishing of the > > > official > > > > Bigtop artifacts to mavencentral. > > > > > > > > Could you get a bit deeper into this whole idea of the applications? > I am > > > > interested to hear more ;)! > > > > > > > > Thanks, > > > > Cos > > > > > > > >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote: > > > >> Hi bigtop. > > > >> > > > >> If I want to build an app against bigtop Is there a maven repo > I can > > > >> point to that has all the version stuff figured out? > > > >> > > > >> As we all know: it's hard to build hadoop apps and test them in a > dev > > > >> environment that matches the classpath of cluster environment... I > think > > > >> bigtop as a distribution might be able to support something like a > > > >> maven-archetype for bigtop based app development.? > > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Bigtop Hack-a-thon in Feb
If anyone wants, bigpetstore is close to its first commit / patch . let me know if you want to hack on it with us : https://github.com/jayunit100/bigpetstore On Wed, Jan 29, 2014 at 12:49 PM, Sean Mackrory wrote: > >> Is this something I can do at the hackathon? > > I believe the protocol at hackathons is to do pretty much whatever the hell > you want :) > > > Thanks for organizing this, Andre! I'm looking forward to this and will try > to be there as much as I can that day. > > For the few who may care, this may be my last Bigtop hackathon for a while. > I will be moving away from the Bay Area in March, although I will be > continuing in my current day job and I don't anticipate any other change in > my involvement with Bigtop. > > > On Wed, Jan 29, 2014 at 9:22 AM, Ron Gmail > wrote: > > > Can I use this time to work on my chef-bigtop cookbook? > > https://github.com/rbogdanoff/chef-bigtop > > > > I now have a real test harness! > > > > What I need to do now is start making bigtop component recipes. In the > > first version I had a bigtop::install recipe but that was a naive > approach > > I want to create a recipe per component like bigtop::hdfs_namenode > > bigtop::hdfs_datanode bigtop::hive etc. > > > > Also want to create a lwrp (lightweight resource) for the cookbook so > > there will be hdfs commands exposed as chef resources, this will make the > > recipes cleaner. > > > > Is this something I can do at the hackathon? > > > > Thanks! > > Ron > > > > Sent from my iPad > > > > > On Jan 28, 2014, at 5:55 PM, Andre Arcilla wrote: > > > > > > Fellow Bigtop devs, > > > > > > I'd like to invite everyone for a hack-a-thon to be held at A9.com (265 > > > Lytton in Palo Alto). The doors are open 10am till late. The food is > > > pizza/sandwiches (specific vendors TBD - your vote counts). I'll > create a > > > video hangout for the people who want to participate remotely. > > > > > > This hack-a-thon will also feature guests from Amazon.com in Seattle. > > They > > > are interested in learning about and contributing to Bigtop, and are > > eager > > > to drink from a font of expertise that is Bigtop community. > > > > > > Date: Fri Feb 7, 2014 > > > Time: 10am - late > > > Address: 265 Lytton Ave., Palo Alto CA > > > > > > Please voice your preferences regarding the above. > > > > > > Hope to see you all, > > > > > > Andre > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Github integration
Im excited and confused at the same time :) Will this be a way of applying patches? Or is this just a way to help github users prep patches by submitting pull requests which get put into the JIRA? On Thu, Feb 13, 2014 at 6:27 PM, Ananda Bhattacharya wrote: > That sounds great! > I would like to use this on one of my projects that I am working on do you > know where I can get access to the Jenkins Scripts for these? > -A > > > Ce qui embellit le désert, dit le petit prince, c'est qu'il cache un puits > quelque part.. > Le Petit Prince > Antoine de Sain Exupéry > > > > On Feb 13, 2014, at 3:25 PM, Roman Shaposhnik wrote: > > > Hi! > > > > this is pretty cool: > > > https://blogs.apache.org/infra/entry/improved_integration_between_apache_and > > > > Can somebody, please, volunteer to do the > > required thing with INFRA? > > > > Thanks, > > Roman. > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Github integration
+1 to adopt it, .but now I'm still wondering , it's not clear: does this mean we can use pull requests to submit patches? It seems to say the jiras will track pull requests...but doesn't say that the github is a proper medium for allowing pull requests to be commited? > On Feb 14, 2014, at 12:22 AM, Mark Grover wrote: > > I am a +1. Expands our ecosystem to developers who prefers using github:-) > > Unless there's a -1, I create an INFRA jira. > > >> On Thu, Feb 13, 2014 at 9:12 PM, Konstantin Boudnik wrote: >> >> What benefits you expect the gain from it? >> >> I am pretty much -0 on the thing, just trying to educate myself ;) >> Cos >> >>> On Thu, Feb 13, 2014 at 03:25PM, Roman Shaposhnik wrote: >>> Hi! >>> >>> this is pretty cool: >> https://blogs.apache.org/infra/entry/improved_integration_between_apache_and >>> >>> Can somebody, please, volunteer to do the >>> required thing with INFRA? >>> >>> Thanks, >>> Roman. >>
Bigtop: generating fake data
Hi bigtop. Are we interested in maintaining our own infra for generating fake data , rather than relying on and downloading external data sources for smokes? Fake data is great for testing I think... In bigpetstore I'm generating fake data , written a lot of code to do this in the custom input formats but I just found : http://codearte.github.io/jfairy/ Which is a groovy tool for doing the same I wonder wether generating fake data for testing big data should be a first-class part of bigtop ? Would others use a utility or just me ? It might be another useful artifact for the community especially for bigpetstore but also for testing a variety of other machine learning related projects I think it's bad to rely on external websites for our tests, maybe in time we could move over to our in internally curated/generated data sets , and a data generation tool like the above moves us in that direction.
Re: Bigtop: generating fake data
Glad to hear there is some interest. Here is a JIRA to take it further. https://issues.apache.org/jira/browse/BIGTOP-1212 @Cos, we need something flexible enough to do differnt types of data sets,and possibly embed patterns in the data, do you know of any place to start ? is GridMix, for example, or SLive, pluggable in that way? If not we might have to hack our own together. Maybe respond in BIGTOP-1212 above. On Sat, Feb 15, 2014 at 9:47 PM, Konstantin Boudnik wrote: > Neat idea! I think the answer depends on what kinda data we want to > generate. > - I had a good run with gridmix for variery of longevity loads (too bad >Cloudera never released the code to open source). > - for HDFS testing we can use SLive and DFSIO (BIGTOP-1208 and > BIGTOP-1209) >are pretty much ready, it seems > > At any rate, I'd rather prefer to incorporate something readily available > that > has good community behind it, so we won't end up supporting an big chunk of > specialized software. > > So, what do you have in mind? Any details? > Cos > > On Sat, Feb 15, 2014 at 09:19AM, Jay Vyas wrote: > > Hi bigtop. Are we interested in maintaining our own infra for generating > > fake data , rather than relying on and downloading external data sources > for > > smokes? Fake data is great for testing I think... > > > > In bigpetstore I'm generating fake data , written a lot of code to do > this > > in the custom input formats but I just found : > > > > http://codearte.github.io/jfairy/ > > > > Which is a groovy tool for doing the same > > > > I wonder wether generating fake data for testing big data should be a > > first-class part of bigtop ? Would others use a utility or just me ? > > > > It might be another useful artifact for the community especially for > > bigpetstore but also for testing a variety of other machine learning > related > > projects > > > > I think it's bad to rely on external websites for our tests, maybe in > time > > we could move over to our in internally curated/generated data sets , > and a > > data generation tool like the above moves us in that direction. > > -- Jay Vyas http://jayunit100.blogspot.com
Re: [VOTE] Regular Bigtop Hack-a-thons
hola bigtop ! i like attending the meetups remotel can you guys try to show us some east coast love to ? And Later is definetly better for those of us that dont get to hack on bigtop full-time ! On Thu, Feb 20, 2014 at 1:35 AM, Konstantin Boudnik wrote: > +1 > > On Fri, Feb 14, 2014 at 05:08PM, Andre Arcilla wrote: > > Resending as a formal vote (thanks Cos)... > > > > Let's pick a tentative schedule for Bigtop hack-a-thons so that we can > plan > > in advance. Once the date draws closer, we can adjust if desired. > > > > I propose: > > > > - Bi-monthly meetup/hack-a-thon > > - To be held on 1st Fri of every other month (Feb, Apr, Jun...) > > - 10am - 5pm > > - At participating locations > > > > Please vote/provide feedback until EOB Thu 2/20. > > > > Thanks > > > > Andre > -- Jay Vyas http://jayunit100.blogspot.com
Re: [VOTE] Regular Bigtop Hack-a-thons
by "east coast love" i mean... specifically, keep up the hangout efforts even though i know it is a bit distracting, its nice to hear your wonderful voices every few weeks :) On Thu, Feb 20, 2014 at 6:55 AM, Jay Vyas wrote: > hola bigtop ! > > i like attending the meetups remotel can you guys try to show us some east > coast love to ? And Later is definetly better for those of us that dont > get to hack on bigtop full-time ! > > > On Thu, Feb 20, 2014 at 1:35 AM, Konstantin Boudnik wrote: > >> +1 >> >> On Fri, Feb 14, 2014 at 05:08PM, Andre Arcilla wrote: >> > Resending as a formal vote (thanks Cos)... >> > >> > Let's pick a tentative schedule for Bigtop hack-a-thons so that we can >> plan >> > in advance. Once the date draws closer, we can adjust if desired. >> > >> > I propose: >> > >> > - Bi-monthly meetup/hack-a-thon >> > - To be held on 1st Fri of every other month (Feb, Apr, Jun...) >> > - 10am - 5pm >> > - At participating locations >> > >> > Please vote/provide feedback until EOB Thu 2/20. >> > >> > Thanks >> > >> > Andre >> > > > > -- > Jay Vyas > http://jayunit100.blogspot.com > -- Jay Vyas http://jayunit100.blogspot.com
Any reason why ">>" might fail in the ITest Shell.java?
Hi bigtop !
I've been trying to do something like :
sh.exec("echo \"hi\" >> myfile.txt");
In ITest. However, the file "myfile.txt" is coming up empty.
Should ITest accept any inline bash command? I assume "yes" but maybe
something funny happens when using pipes/streams inline ... ?
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Any reason why ">>" might fail in the ITest Shell.java?
Well, nevermind, ive tested it in a more simple scenario and echo/cat seem
to work okay, each line of output goes to an array element returned in the
getOut() call.
On Wed, Feb 26, 2014 at 1:26 PM, Jay Vyas wrote:
> Hi bigtop !
>
> I've been trying to do something like :
>
> sh.exec("echo \"hi\" >> myfile.txt");
>
> In ITest. However, the file "myfile.txt" is coming up empty.
>
> Should ITest accept any inline bash command? I assume "yes" but maybe
> something funny happens when using pipes/streams inline ... ?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Any reason why ">>" might fail in the ITest Shell.java?
Thanks cos, maybe when doing minor rewriting during debugging, that was the issue which i might have fixed inadvertently
Bigtop-1221
Hi bigtop ! https://issues.apache.org/jira/browse/BIGTOP-1221 Hope someone can review this one for me ... its a pretty powerful example of how valuable the bigtop smoke tests are becoming to the broader community. this patch also can be used to reproduce a bug in upstream HDFS, which I've JIRA'd here: https://issues.apache.org/jira/browse/HDFS-6027, which now backlinks to BIGTOP-1221. One small step forward for HCFS, one giant leap for BIGTOP ! -- Jay Vyas http://jayunit100.blogspot.com
Re: Hadoop Summit submissions
Thanks roman ! this is a great idea for a talk, as understanding bigtop is important even for folks using commercial distros, IMO. I've also submitted a bigtop-related talk : BigPetStore, to hadoop summit , which is slated to become part of bigtop once we finally get around to polishing its first iteration off :) https://hadoopsummit.uservoice.com/forums/242808-hadoop-driven-business-track/suggestions/5568553-bigpetstore On Thu, Feb 27, 2014 at 5:18 PM, Roman Shaposhnik wrote: > Hi! > > not sure if anybody from the Bigtop community > submitted any talks to Hadoop Summit, but > here's the one I submitted: > > https://hadoopsummit.uservoice.com/forums/242807-hadoop-deployment-operations-track/suggestions/5568492-apache-bigtop-a-crash-course-in-deploying-a-hadoo > > Feel free to upvote if you feel like Bigtop deserves > to be represented at Hadoop Summit. > > Thanks, > Roman. > -- Jay Vyas http://jayunit100.blogspot.com
Re: [jira] [Commented] (BIGTOP-1221) Expand and updated FUSE tests
Sure will do now On Thu, Feb 27, 2014 at 6:40 PM, Mark Grover (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/BIGTOP-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915216#comment-13915216] > > Mark Grover commented on BIGTOP-1221: > - > > I do have a few nitpicks/comments. Mind uploading on reviewboard?:-) > > > Expand and updated FUSE tests > > - > > > > Key: BIGTOP-1221 > > URL: https://issues.apache.org/jira/browse/BIGTOP-1221 > > Project: Bigtop > > Issue Type: Bug > >Reporter: jay vyas > >Assignee: jay vyas > > Attachments: BIGTOP-1221.patch > > > > > > The BigTop TestFuseDFS need > > - tests for consistency after individual writes : consistency in FUSE > mounts is important since there are often associated multitenant workloads. > Writes should result in immediate recognition of a file. > > - Should use HCFS semantics (That way we can confidently use and > maintain this for alternative file system implementations). > > - should use logging instead of System.out. That way all the logs can > be picked up from the same place in case we change the logging framework. > Logging in trace mode, in any case, will allow us to see everything that > the ITest shell does, and so might as well use it for other logs. > > > > -- > This message was sent by Atlassian JIRA > (v6.1.5#6160) > -- Jay Vyas http://jayunit100.blogspot.com
Review Request 18593: BIGTOP-1221
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/ --- Review request for bigtop. Repository: bigtop Description --- this patch makes bigtop fuse tests awesome :) Diffs - bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy c42fb33 Diff: https://reviews.apache.org/r/18593/diff/ Testing --- tested in a pseudodistributed VM. works. Thanks, jay vyas
Re: Review Request 18593: BIGTOP-1221
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/ --- (Updated March 4, 2014, 6:33 p.m.) Review request for bigtop. Changes --- heres the latest ! Repository: bigtop Description --- this patch makes bigtop fuse tests awesome :) Diffs (updated) - bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/TestFuseHCFS.groovy PRE-CREATION bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy c42fb33 Diff: https://reviews.apache.org/r/18593/diff/ Testing --- tested in a pseudodistributed VM. works. Thanks, jay vyas
Re: Review Request 18593: BIGTOP-1221
> On March 4, 2014, 3:07 a.m., Mark Grover wrote: > > bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy, > > line 77 > > <https://reviews.apache.org/r/18593/diff/1/?file=506544#file506544line77> > > > > Dead code, please take it out. > On March 4, 2014, 3:07 a.m., Mark Grover wrote: > > bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy, > > line 169 > > <https://reviews.apache.org/r/18593/diff/1/?file=506544#file506544line169> > > > > Why take out a test? What if someone else's code somewhere uses this > > test? Is this something that just applies to HDFS and not to other HCFSs? took this out because it was a "bad " test (order dependant). > On March 4, 2014, 3:07 a.m., Mark Grover wrote: > > bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy, > > line 202 > > <https://reviews.apache.org/r/18593/diff/1/?file=506544#file506544line202> > > > > Where is "contents" coming from? Is it guaranteed to be there across > > file systems? contents come from the write statement > On March 4, 2014, 3:07 a.m., Mark Grover wrote: > > bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy, > > line 219 > > <https://reviews.apache.org/r/18593/diff/1/?file=506544#file506544line219> > > > > Is this guaranteed? I haven't read the cp spec but I would think the > > only guaranteed thing would be the return code. A better assert here would > > be to check if the destination exists and diff it with the source. > > > > Checking strings from output is a pretty hard to maintain and arguably > > a very strict assert. yes definetly gauranteed that ls will show files copied in. i think... am i missing something? - jay ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/#review36080 --- On March 4, 2014, 6:33 p.m., jay vyas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18593/ > --- > > (Updated March 4, 2014, 6:33 p.m.) > > > Review request for bigtop. > > > Repository: bigtop > > > Description > --- > > this patch makes bigtop fuse tests awesome :) > > > Diffs > - > > > bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/TestFuseHCFS.groovy > PRE-CREATION > > bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy > c42fb33 > > Diff: https://reviews.apache.org/r/18593/diff/ > > > Testing > --- > > tested in a pseudodistributed VM. works. > > > Thanks, > > jay vyas > >
Re: Review Request 18593: BIGTOP-1221
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/ --- (Updated March 6, 2014, 3:31 p.m.) Review request for bigtop. Changes --- oops , here it is Repository: bigtop Description --- this patch makes bigtop fuse tests awesome :) Diffs (updated) - bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/TestFuseHCFS.groovy PRE-CREATION bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy c42fb33 Diff: https://reviews.apache.org/r/18593/diff/ Testing --- tested in a pseudodistributed VM. works. Thanks, jay vyas
Re: Review Request 18593: BIGTOP-1221
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/ --- (Updated March 6, 2014, 5:30 a.m.) Review request for bigtop. Changes --- heres another iteration, fixed all issues see https://issues.apache.org/jira/browse/BIGTOP-1221 Repository: bigtop Description --- this patch makes bigtop fuse tests awesome :) Diffs (updated) - bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/TestFuseHCFS.groovy PRE-CREATION bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy c42fb33 Diff: https://reviews.apache.org/r/18593/diff/ Testing --- tested in a pseudodistributed VM. works. Thanks, jay vyas
Re: Review Request 18593: BIGTOP-1221
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/ --- (Updated March 6, 2014, 3:27 p.m.) Review request for bigtop. Changes --- (5) heres another iteration, fixed all issues see https://issues.apache.org/jira/browse/BIGTOP-1221 Repository: bigtop Description --- this patch makes bigtop fuse tests awesome :) Diffs (updated) - bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy c42fb33 Diff: https://reviews.apache.org/r/18593/diff/ Testing --- tested in a pseudodistributed VM. works. Thanks, jay vyas
Re: Review Request 18593: BIGTOP-1221
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18593/ --- (Updated March 6, 2014, 3:40 p.m.) Review request for bigtop. Repository: bigtop Description --- this patch makes bigtop fuse tests awesome :) Diffs (updated) - bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hcfs/TestFuseHCFS.groovy PRE-CREATION bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoop/hdfs/TestFuseDFS.groovy c42fb33 Diff: https://reviews.apache.org/r/18593/diff/ Testing --- tested in a pseudodistributed VM. works. Thanks, jay vyas
Re: ASF Board Meeting Summary - March 19, 2014
congrats kos ! On Wed, Mar 19, 2014 at 8:02 PM, Roman Shaposhnik wrote: > Its official now! Cos, yoda man! ;-) > > Thanks, > Roman. > > -- Forwarded message -- > From: Brett Porter > Date: Wed, Mar 19, 2014 at 2:48 PM > Subject: ASF Board Meeting Summary - March 19, 2014 > To: committ...@apache.org > > > The March board meeting took place on the 19th. > > The following directors were present: > > Shane Curcuru > Doug Cutting > Bertrand Delacretaz > Roy T. Fielding > Jim Jagielski > Chris Mattmann > Brett Porter > Sam Ruby > Greg Stein > > The following officers were present: > > Ross Gardler > Rich Bowen > Craig L Russell > > The following guests were present: > > Jake Farrell > Sean Kelly > Daniel Gruno > Marvin Humphrey > Cory Johns > David Nalley > Michael Joyce > Henri Yandell > > The February minutes were approved. > Minutes will be posted to > http://www.apache.org/foundation/records/minutes/ > > All of the received reports to the board were approved. > > The following reports were not received and are expected next month: > > Report from the Apache Click Project [Malcolm Edgar] > Report from the Apache Cordova Project [Brian LeRoux] > Report from the Apache Creadur Project [Robert Burrell Donkin] > Report from the Apache Labs Project [Tim Williams] > > The following resolutions were passed unanimously: > > A. Establish the Apache Tajo Project (Hyunsik Choi, VP) > B. Change the Apache CloudStack Project Chair (Hugo Trippaers, VP) > C. Change the Apache Pig Project Chair (Cheolsoo Park, VP) > D. Change the Apache Bigtop Project Chair (Konstantin Boudnik, VP) > E. Change the Apache Hama Project Chair (Chia-Hung Lin, VP) > F. Change the Apache Lucy Project Chair (Logan Bell, VP) > G. Establish the Apache Olingo Project (Stephan Klevenz, VP) > H. Establish the Apache Allura Project (Dave Brondsema, VP) > > The next board meeting will be on the 16th of April. > -- Jay Vyas http://jayunit100.blogspot.com
Thanks for BIGTOP-1200 and BIGTOP-952
Hey bigtop ! Just wanted to thank everyone that helped me with BIGTOP-1200, BIGTOP-952 - Roman for the original patch i stole - Cos for helping get the format right, tireless immediate reviews,and refining the code to perfection - Mark also for helping refine the code, formatting, and logic line by line. - Anyone else i might have forgotten ! Here is why this is important to us We want all file systems to be first class citizens in the hadoop ecosystem, and we're working hard to this end. Now we can take the work which is in BIGTOP, and use it demonstrate to the broader community how they can feedback into the bigtop community for a independant standard of provisioning file systems. My next HCFS Goals will be bridging the gap between ambari and the bigtop / hcfs community, wherever possible: https://issues.apache.org/jira/browse/AMBARI-5181 So , just saying thanks for helping me to maintain HCFS compliance inside of bigtop. This is going to make hadoop even more universal in the big data space, and ensure that all of us have the ability to provide extremely pluggable, high quality storage solutions to the mapreduce and yarn universe. Now, onwards and upwards with BIGTOP-1235 !!! See you folks in denver. -- Jay Vyas http://jayunit100.blogspot.com
Bigtop meetup ApacheCon
Hi bigtop ! Okay. So we finally have the meetup schedules solidified: April 7th AND 8th @ApacheCon, Denver, in the Westin - Teller Room, Please RSVP if possible in this thread so we can try to estimate how to handle food/beer etc. -- Jay Vyas http://jayunit100.blogspot.com
Re: [DISCUSSION]: Adding GridGain component in Bigtop
> Hope this answers your question. > > > > > > > > > > -Dmitriy > > > > > > > > > > > > > > > > > > > > On Mon, Mar 24, 2014 at 11:11 PM, Roman Shaposhnik > > > > > wrote: > > > > > > > > > > > Hi Dmitriy! > > > > > > > > > > > > Welcome to the Bigtop community! > > > > > > > > > > > > On Mon, Mar 24, 2014 at 10:43 PM, Konstantin Boudnik < > > c...@apache.org > > > > > > > > > > wrote: > > > > > > >> One of the main pieces of our platform is our In-Memory Apache > > > > Hadoop > > > > > > >> Accelerator which aims to accelerate HDFS and Map/Reduce by > > > bringing > > > > > > both, > > > > > > >> data and computations into memory. We do it with our GGFS - > > Hadoop > > > > > > >> compliant in-memory file system. For I/O intensive jobs > GridGain > > > > GGFS > > > > > > >> offers performance close to 100x faster than standard HDFS. > More > > > > > > >> information can be found here: > > > > > > >> http://www.gridgain.org/features/hadoop-acceleration/ > > > > > > >> > > > > > > >> We would like to have an opportunity to integrate our Apache > > > Hadoop > > > > > > >> Accelerator with Apache Bigtop. Please let us know if this is > > > > possible > > > > > > and > > > > > > >> what steps are required of us. > > > > > > > > > > > > I've been actually fascinated by the in-memory analytics > platforms > > > > > lately. > > > > > > Things like Apache Spark seem to be a really good addition to the > > > > > > Hadoop ecosystem. > > > > > > > > > > > > Now, I understand that you've got a piece of technology that can > > > > > > essentially > > > > > > serve as a replacement for HDFS, but could you please elaborate > on > > > > > > what other integration points do you have between GridGain and > the > > > rest > > > > > > of Hadoop ecosystem? > > > > > > > > > > > > That, I think, would be a much wider discussion. > > > > > > > > > > > > Thanks, > > > > > > Roman. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > > > > >- Andy > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein > > > > (via Tom White) > > > > > > > > > > > > > > > -- > > Best regards, > > > >- Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: [DISCUSSION]: Adding GridGain component in Bigtop
Yeah sure we can try a google hangout screencast thingy On Wed, Mar 26, 2014 at 5:31 PM, Konstantin Boudnik wrote: > Dmitriy, > > I think your proposal is great as we are just forming up the agenda for the > meetup. And it seems to be great to have an deeper dive into the platform, > which will let folks here to get more familiar with it. > > Jay, do you think we can tape the meetup talks and publish it later? > Cos > > On Wed, Mar 26, 2014 at 02:36PM, Dmitriy Setrakyan wrote: > > I plan to be at ApacheCon on Monday, April 7th. I hear that Bigtop will > > have a meetup there in the evening. Do you think it will be OK if I could > > spend about 20 minutes there to present GridGain GGFS and overall > approach > > to Hadoop acceleration? I think it would be interesting to go through a > > couple of architectural diagrams and may spur a good discussion. > > > > -Dmitriy > > > > On Wed, Mar 26, 2014 at 8:35 AM, Jay Vyas wrote: > > > > > I love the fact that GridGain is going to be part of bigtop ! This > will > > > give us two new compute paradigms, all packaged and testable under the > > > same umbrella. And now with our vagrant recipes, people will be able > to > > > demo grid gain by simply typing "vagrant up" into the console. > > > > > > And Im pretty sure GridGain and Spark will drive each other forward . > Just > > > the same way Ceph, HDFS, and GlusterFS do. > > > > > > Dmitriy will you be at apachecon? If so why dont you come share your > > > thoughts with us at the two bigtop meetups on the 7th and the 8th ? > > > > > > > > > > > > > > > > > > On Wed, Mar 26, 2014 at 10:26 AM, Dmitriy Setrakyan < > > > dsetrak...@gridgain.com > > > > wrote: > > > > > > > Andrew, > > > > > > > > I agree with you. All I meant to say is that currently users of > Hadoop > > > that > > > > would like to improve performance of their deployments have to > switch to > > > > Spark and code to Spark APIs. GridGain, on the other hand, will > provide > > > an > > > > option to accelerate existing Hadoop deployments without any changes > in > > > > code. > > > > > > > > Regards, > > > > -Dmtiriy > > > > > > > > On Tue, Mar 25, 2014 at 4:16 PM, Andrew Purtell > > > > > wrote: > > > > > > > > > Thank you. > > > > > > > > > > On this part of your response: > > > > > > > > > > > GridGain is working on adding native MapReduce component which > will > > > > > provide > > > > > native complete Hadoop integration without changes in API, like > Spark > > > > > currently forces you to do > > > > > > > > > > I'm not sure those flocking to Spark are doing so by force. Nor > that > > > the > > > > > Spark API should be considered a liability when compared to Hadoop > > > > > MapReduce. For your consideration. > > > > > > > > > > > > > > > > > > > > On Tue, Mar 25, 2014 at 12:08 AM, Dmitriy Setrakyan < > > > > > dsetrak...@gridgain.com > > > > > > wrote: > > > > > > > > > > > I think the feature set is pretty close and GGFS would be a good > > > > contract > > > > > > to Tachyon for performance and reliability features. > > > > > > > > > > > > I am not an expert on Tachyon, but I think the main differences > are: > > > > > > > > > > > > - GGFS allows read-through and write-through to/from underlying > HDFS > > > or > > > > > any > > > > > > other Hadoop compliant file system with zero code change. > Essentially > > > > > GGFS > > > > > > entirely removes ETL step from integration. > > > > > > > > > > > > - GGFS has ability to pick and choose what folders stay in > memory, > > > what > > > > > > folders stay on disc, and what folders get synchronized with > > > underlying > > > > > > (HD)FS either synchronously or asynchronously. > > > > > > > > > > > > - GridGain is working on adding native MapReduce component which > will > > > > > > provide native complete Hadoop integration without
Re: Bigtop meetup ApacheCon
Hi roman! What is vOS? Red hat (m) is sponsoring . I think ill figure out the details at the last minute. Can all attendees please remember to send an RSVP.? I need to estimate how much stuff food beer etc > On Apr 4, 2014, at 7:10 PM, Roman Shaposhnik wrote: > > Hi! > > Thanks for twikifying it! I've updated it with some > of the additional agenda items. > > On that note, I'm super excited to announce that > Don Marti from OSv agreed to come to our meetup > and give us a full technical overview of what OSv > can do for cloud architecture around java apps. > > I hope he can also stick around to hack our first > prototype of Bigtop-deployanble OSv Hadoop > environment. > > Honestly, I haven't been this excited about a piece > of technology since dtrace/zfs. > > Thanks, > Roman. > > P.S. What's our situation with food/drink? Do we have anybody > sponsoring yet? > > >> On Thu, Apr 3, 2014 at 2:22 AM, Konstantin Boudnik wrote: >> The page is here >> https://cwiki.apache.org/confluence/display/BIGTOP/ApacheCon%2714 >> >>> On Thu, Apr 03, 2014 at 02:14AM, Konstantin Boudnik wrote: >>> I've posted the agenda to the wiki page. >>> >>> Please update it there if you want to add/changes anything. >>> >>> Cos >>> >>>> On Wed, Mar 26, 2014 at 01:19AM, Jay Vyas wrote: >>>> Hi bigtop ! Okay. >>>> >>>> So we finally have the meetup schedules solidified: >>>> >>>> April 7th AND 8th >>>> >>>> @ApacheCon, Denver, in the Westin - Teller Room, >>>> Please RSVP if possible in this thread so we can try to estimate how to >>>> handle food/beer etc. >>>> >>>> -- >>>> Jay Vyas >>>> http://jayunit100.blogspot.com
Re: build server account --> *roman*
hi mark ! Okay created one here: http://bigtop01.cloudera.org:8080/user/jayunit100/ On Tue, Apr 8, 2014 at 11:36 PM, Mark Grover wrote: > +dev@ > Moving user@ to bcc > > Hey Jay, > I couldn't see an account for you at > http://bigtop01.cloudera.org:8080/people/? > > Can you create an account there and let us know your username? > > > On Tue, Apr 8, 2014 at 4:08 PM, Jay Vyas wrote: > > > hi roman :) can you give me an account on the bigtop build server > > so that i can create a task to implement BIGTOP-1249 > > > > -- > > Jay Vyas > > http://jayunit100.blogspot.com > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Bigtop meetup ApacheCon
Ditto cos... Summary of what we did... Anything I forgot? - alex Newman is getting involved with bigtop, and ready to hack on some jiras with us. Him and roman worked on osV. Is this going to be a part of bigtop-deploy/ soon? - Gridgain is jumping on board with hcfs and putting an alternate mapred platform into bigtop. - 1089- (bigpetstore) ready for review. - 1021- (gradle based make) ready for review. Maybe I can try building bigtop rpms and then help review? ** extras ** - gonna bug my brother to help possibly a hipper bigtop logo . - roman mentioned hcfs in one of his talks so I owe him a hot dog. Cos mentioned bigpetstore so I also owe him a hot dog. > On Apr 9, 2014, at 8:25 PM, Konstantin Boudnik wrote: > > Jay, > > thanks for putting together the room reservations and other logistic around > the meetups! These were great events where we learned about GridGain and OSV > as well as did some hacking around the latter technology. > > The ApacheCon was a lot of fun and interesting work! Was great to see > everyone! > Cos > >> On Wed, Mar 26, 2014 at 01:19AM, Jay Vyas wrote: >> Hi bigtop ! Okay. >> >> So we finally have the meetup schedules solidified: >> >> April 7th AND 8th >> >> @ApacheCon, Denver, in the Westin - Teller Room, >> Please RSVP if possible in this thread so we can try to estimate how to >> handle food/beer etc. >> >> -- >> Jay Vyas >> http://jayunit100.blogspot.com
Re: Using Ambari with vanilla Apache releases
Ye - And On our end we can definetly help with leveraging of the hcfs side of things (provisioning ambari in a filesystem pluggable manner) or smoke testing of standard ambari components. - One area of interest will be resolving overlap between how bigtop vs ambari does provisioning ? Great to see ambari directly opening up into bigtop.! It's got a lot of potential ! > On Apr 11, 2014, at 8:24 PM, Yusaku Sako wrote: > > Hi Roman, > > This is a great idea! > I'm interested in providing some Ambari expertise. > > Yusaku > > > On Fri, Apr 11, 2014 at 4:57 PM, Roman Shaposhnik wrote: > >> On Fri, Apr 11, 2014 at 4:44 PM, Mahadev Konar >> wrote: >>> This would definitely be of a lot of intersest to others as well in the >>> community. Good suggestion Roman. Want to propose something/someday on >> the >>> mailing list? Unfortunately Ill be out for the next couple of weeks but >> am >>> sure others on the list will be interested in a meetup/hackathon as well. >> >> Hi! >> >> I'd be more than happy to host at Pivotal's Palo Alto office in April. >> I think the >> quorum we need for this type of event is a few (at least one ;-)) >> folks intimately >> familiar with Ambari and a few folks who are Bigtop experts (I can >> personally >> volunteer) plus as many innocent bystanders as possible ;-) >> >> Please reply to this thread if interested! >> >> Thanks, >> Roman. > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
Fwd: [jira] [Created] (HADOOP-10500) TestDoAsEffectiveUser fails on JDK7 due to failure to reset proxy user configuration.
Heads up: IIRC at least some of the tests in bigtop were order dependent... if any still floating around probably best to rewrite them with the trick in BIGTOP-1224 .
bigtop-utils
Hi sean and others: I want to replace the init-hdfs.sh (.//bigtop-packages/src/common/hadoop/init-hdfs.sh) with provision.groovy (.//bigtop-packages/src/common/bigtop-utils/provision.groovy) It seems pretty complex. Any pointers on the lifecycle of bigtop-utils files in terms of where and how they get installed in the rpms? Nothing detailed, just some guidelines on how the utilities are leveraged during provisioning of a cluster, and what needs to be done to integrate a new utility into the cluster i can grep around for the specifics.
Re: bigtop-utils
Hmmm ok... I'll ask a different way to clarify. should provision.groovy be in src/common/hadoop as well? Right now it's in another directory all together. After all it is meant to be a drop in replacement for init-hdfs.sh. So in current bigtop... either provison.groovy OR init-hdfs.sh is in the wrong place, don't you think? > On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik wrote: > > Jay, > > I think the script is packed up into the hadoop package and is relied upon by > Puppet. So, it shouldn't be too hard to replace it. What seems to be the > trouble? > > Cos > >> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote: >> Hi sean and others: >> >> I want to replace the init-hdfs.sh >> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh) >> >> with provision.groovy >> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy) >> >> It seems pretty complex. >> >> Any pointers on the lifecycle of bigtop-utils files in terms of where and >> how they get installed in the rpms? >> >> Nothing detailed, just some guidelines on how the utilities are leveraged >> during provisioning of a cluster, and what needs to be done to integrate a >> new utility into >> the cluster >> >> i can grep around for the specifics.
Re: bigtop-utils
Ok thanks Sean that helps. Two questions (response to your q's at bottom). 1) Do you folks think Maybe we should keep init-hdfs.sh in place and call provision.groovy from it? 2) Also..Sean or others: Any more brain dumping you can share on the strategy of packaging scripts of bigtop-util? .. Good point about init-hdfs being hdfs specific. Yes provision.groovy is meant to be 100% hcfs compatible, loading fs from configuration inject and init-hcfs.json rather than explicitly doing things as hdfs. > On Apr 23, 2014, at 9:02 AM, Sean Mackrory wrote: > > init-hdfs.sh is specific to Hadoop's filesystem, so it belongs under > /usr/lib/hadoop. Is Provision.groovy specific to HDFS, or is it for any > HCFS? And is it even intended to always be specific to filesystems, or will > it also be used to do initialization of other services? I think all that > determines where it should be, and unless it is specific to HDFS it doesn't > belong in the same place as init-hdfs.sh is now. > > >> On Wed, Apr 23, 2014 at 6:24 AM, Jay Vyas wrote: >> >> Hmmm ok... I'll ask a different way to clarify. >> >> should provision.groovy be in src/common/hadoop as well? Right now it's >> in another directory all together. >> >> After all it is meant to be a drop in replacement for init-hdfs.sh. >> >> So in current bigtop... either provison.groovy OR init-hdfs.sh is in the >> wrong place, don't you think? >> >>> On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik wrote: >>> >>> Jay, >>> >>> I think the script is packed up into the hadoop package and is relied >> upon by >>> Puppet. So, it shouldn't be too hard to replace it. What seems to be the >>> trouble? >>> >>> Cos >>> >>>> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote: >>>> Hi sean and others: >>>> >>>> I want to replace the init-hdfs.sh >>>> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh) >>>> >>>> with provision.groovy >>>> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy) >>>> >>>> It seems pretty complex. >>>> >>>> Any pointers on the lifecycle of bigtop-utils files in terms of where >> and >>>> how they get installed in the rpms? >>>> >>>> Nothing detailed, just some guidelines on how the utilities are >> leveraged >>>> during provisioning of a cluster, and what needs to be done to >> integrate a >>>> new utility into >>>> the cluster >>>> >>>> i can grep around for the specifics. >>
Re: bigtop-utils
okay just updated the jira to rename it... the point of bigtop-1235 is to integrate all this stuff and do cleanup where necessary, and then push a new way of provisioning bigtop . feel free if you want to add/revise the new bullets in https://issues.apache.org/jira/browse/BIGTOP-1235 On Wed, Apr 23, 2014 at 11:53 AM, Sean Mackrory wrote: > 1) I'm 50/50 on whether or not we keep init-hdfs.sh as a wrapper to the new > groovy script. On the one hand, I personally have a lot of stuff that uses > init-hdfs.sh, on the other hand, I don't want to keep around legacy stuff > without a very good reason. It should be simple enough to migrate to the > new script now that groovy is packaged as a part of Bigtop > > 2) I'm not sure what you mean by the strategy of lifecycle of bigtop-utils. > It's just a few scripts that are useful for lots of components. When we can > improve them we improve them. > > If provision.groovy is only doing filesystem stuff for the foreseeable > future, I'd prefer we just call it init-hcfs.groovy rather than something > as generic as "provision". Although I only just noticed provision.groovy is > already added to the bigtop-utils package, so maybe I'm a bit late to the > party :) > > > On Wed, Apr 23, 2014 at 9:39 AM, Jay Vyas wrote: > > > Ok thanks Sean that helps. Two questions (response to your q's at > bottom). > > > > 1) Do you folks think Maybe we should keep init-hdfs.sh in place and call > > provision.groovy from it? > > > > 2) Also..Sean or others: Any more brain dumping you can share on the > > strategy of packaging scripts of bigtop-util? > > > > .. > > > > Good point about init-hdfs being hdfs specific. Yes provision.groovy is > > meant to be 100% hcfs compatible, loading fs from configuration inject > and > > init-hcfs.json rather than explicitly doing things as hdfs. > > > > > > > On Apr 23, 2014, at 9:02 AM, Sean Mackrory > wrote: > > > > > > init-hdfs.sh is specific to Hadoop's filesystem, so it belongs under > > > /usr/lib/hadoop. Is Provision.groovy specific to HDFS, or is it for any > > > HCFS? And is it even intended to always be specific to filesystems, or > > will > > > it also be used to do initialization of other services? I think all > that > > > determines where it should be, and unless it is specific to HDFS it > > doesn't > > > belong in the same place as init-hdfs.sh is now. > > > > > > > > >> On Wed, Apr 23, 2014 at 6:24 AM, Jay Vyas > wrote: > > >> > > >> Hmmm ok... I'll ask a different way to clarify. > > >> > > >> should provision.groovy be in src/common/hadoop as well? Right now > it's > > >> in another directory all together. > > >> > > >> After all it is meant to be a drop in replacement for init-hdfs.sh. > > >> > > >> So in current bigtop... either provison.groovy OR init-hdfs.sh is in > the > > >> wrong place, don't you think? > > >> > > >>> On Apr 23, 2014, at 12:42 AM, Konstantin Boudnik > > wrote: > > >>> > > >>> Jay, > > >>> > > >>> I think the script is packed up into the hadoop package and is relied > > >> upon by > > >>> Puppet. So, it shouldn't be too hard to replace it. What seems to be > > the > > >>> trouble? > > >>> > > >>> Cos > > >>> > > >>>> On Wed, Apr 23, 2014 at 12:31AM, Jay Vyas wrote: > > >>>> Hi sean and others: > > >>>> > > >>>> I want to replace the init-hdfs.sh > > >>>> (.//bigtop-packages/src/common/hadoop/init-hdfs.sh) > > >>>> > > >>>> with provision.groovy > > >>>> (.//bigtop-packages/src/common/bigtop-utils/provision.groovy) > > >>>> > > >>>> It seems pretty complex. > > >>>> > > >>>> Any pointers on the lifecycle of bigtop-utils files in terms of > where > > >> and > > >>>> how they get installed in the rpms? > > >>>> > > >>>> Nothing detailed, just some guidelines on how the utilities are > > >> leveraged > > >>>> during provisioning of a cluster, and what needs to be done to > > >> integrate a > > >>>> new utility into > > >>>> the cluster > > >>>> > > >>>> i can grep around for the specifics. > > >> > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Using Ambari with vanilla Apache releases
so, are we swapping out bigtop packaged rpms from the hdp ones as the primary goal? ** ambari has ** - ui - provisioning a Gluster or HDFS filesystem - provisioning yarn + ecosystem - monitoring - some packaging via HDP ** bigtop has ** - provisioning hdfs (HCFS on the way) - provisioning yarn + ecosystem - packaging - smoke tests - bigpetstore :) so lots of other (possibly more difficult) avenues for integration On Wed, Apr 23, 2014 at 6:21 PM, Sumit Mohanty wrote: > I am not going to HBaseCon either. > > I will prefer 7th/8th if we are open for that week. > > -Sumit > > > On Wed, Apr 23, 2014 at 3:13 PM, Yusaku Sako > wrote: > > > All, > > > > I can be pretty flexible next week or the week after (though I'm not > > going to HBaseCon). > > Hortonworks should be able to host as well. > > > > Thanks, > > Yusaku > > > > On Wed, Apr 23, 2014 at 1:37 PM, Roman Shaposhnik > > wrote: > > > On Wed, Apr 23, 2014 at 1:26 PM, Konstantin Boudnik > > wrote: > > >> I am out on Hbasecon on 5th & 6th. And next week is pretty crazy for > me > > with > > >> Bigtop 0.8 > > > > > > Actually, this gives me an idea: would it be totally crazy to have > this > > as > > > a hackathon/meetup at HBaseCON? Who should we talk to to make it > > > happen? I'm sure there'll be quite a few folks there -- we might as > well > > > exploit it. > > > > > > Thanks, > > > Roman. > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Jay Vyas http://jayunit100.blogspot.com
Re: New committer: Jay Vyas
Thanks everyone ! And feel free to ping me if something urgent needs to be done : I'm always available to help. right now im working on improving the wiki and I'd like to improve the site as well Can we freely push commits to the bigtop website or do we have a review process for it? On Tue, Apr 29, 2014 at 2:28 PM, Giridharan Kesavan < gkesa...@hortonworks.com> wrote: > congrats jay! > > -Giri > > > On Tue, Apr 29, 2014 at 11:16 AM, Sean Mackrory wrote: > >> Well deserved! >> >> >> On Tue, Apr 29, 2014 at 12:11 PM, Mark Grover wrote: >> >> > Congrats, Jay! >> > >> > >> > On Tue, Apr 29, 2014 at 10:16 AM, Konstantin Boudnik >> > wrote: >> > >> > > Guys, >> > > >> > > The Project Management Committee (PMC) for Apache Bigtop has asked Jay >> > > Vyas to >> > > become a committer and we are pleased to announce that he has >> accepted! >> > > >> > > His ASF account has been created as: j...@apache.org >> > > >> > > Being a committer enables easier contribution to the project since >> there >> > > is no >> > > need to go via the proxy patch submission process. This should enable >> > > better >> > > productivity. >> > > >> > > Just keep in mind that Apache Bigtop practices RTC >> (review-then-commit) >> > > development process which means that most of the time you still have >> to >> > > get at >> > > least one +1 before the checkin. If in doubt exercise your best >> judgement >> > > and >> > > don't hesitate to ask questions on this very mailing list. >> > > >> > > We're happy to have you on board and looking for many more >> contributions >> > to >> > > come! >> > > >> > > With regards, >> > > Cos >> > > >> > >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. -- Jay Vyas http://jayunit100.blogspot.com
Git vs svn question
Hi bigtop ! Can we commit code via git or svn is required for bigtop? If SVN is required, does anyone have a workflow i can borrow for putting patches in (i know git --signoff is part of the process but not sure wether this is only for people who use a git wrapper to svn or something fancy like that). Just wondering if i have to setup svn etc am reading the apache docs -- Jay Vyas http://jayunit100.blogspot.com
Re: Git vs svn question
gotcha thanks sean. On Tue, Apr 29, 2014 at 2:59 PM, Sean Mackrory wrote: > If your credentials are set up properly, you should be able to push to > git.apache.org/bigtop.git > > > On Tue, Apr 29, 2014 at 1:58 PM, Sean Mackrory > wrote: > > > Bigtop uses Git. > > > > > > On Tue, Apr 29, 2014 at 1:56 PM, Jay Vyas wrote: > > > >> Hi bigtop ! > >> > >> Can we commit code via git or svn is required for bigtop? If SVN is > >> required, does anyone have a workflow i can borrow for putting patches > in > >> (i know git --signoff is part of the process but not sure wether this is > >> only for people who use a git wrapper to svn or something fancy like > >> that). > >> > >> Just wondering if i have to setup svn etc ,,,, am reading the apache > docs > >> > >> > >> > >> -- > >> Jay Vyas > >> http://jayunit100.blogspot.com > >> > > > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Git vs svn question
Okay ! I did my first commit ! :) Now, how do i re-deploy the site? On Tue, Apr 29, 2014 at 3:13 PM, Jay Vyas wrote: > gotcha thanks sean. > > > On Tue, Apr 29, 2014 at 2:59 PM, Sean Mackrory wrote: > >> If your credentials are set up properly, you should be able to push to >> git.apache.org/bigtop.git >> >> >> On Tue, Apr 29, 2014 at 1:58 PM, Sean Mackrory >> wrote: >> >> > Bigtop uses Git. >> > >> > >> > On Tue, Apr 29, 2014 at 1:56 PM, Jay Vyas wrote: >> > >> >> Hi bigtop ! >> >> >> >> Can we commit code via git or svn is required for bigtop? If SVN is >> >> required, does anyone have a workflow i can borrow for putting patches >> in >> >> (i know git --signoff is part of the process but not sure wether this >> is >> >> only for people who use a git wrapper to svn or something fancy like >> >> that). >> >> >> >> Just wondering if i have to setup svn etc am reading the apache >> docs >> >> >> >> >> >> >> >> -- >> >> Jay Vyas >> >> http://jayunit100.blogspot.com >> >> >> > >> > >> > > > > -- > Jay Vyas > http://jayunit100.blogspot.com > -- Jay Vyas http://jayunit100.blogspot.com
Re: Git vs svn question
Okay cos thanks. Yes... you deploy it that way but I don't know how the permissions work. It's easy to test deploy though using "mvn site". I'm going to whip up some improvements to the site sometime soon I guess we can all agree it's a little rustic. Will file jira once I get around to looking deeper. > On Apr 29, 2014, at 5:22 PM, Konstantin Boudnik wrote: > > Ah, good! I think site can be only deployed with > mvn site:deploy > > that requires special permissions and all. I am a new chair, so I need to > learn this rope next - gimme a little bit of time, please ;) > > Cos > >> On Tue, Apr 29, 2014 at 04:41PM, Jay Vyas wrote: >> Okay ! I did my first commit ! :) >> >> Now, how do i re-deploy the site? >> >> >>> On Tue, Apr 29, 2014 at 3:13 PM, Jay Vyas wrote: >>> >>> gotcha thanks sean. >>> >>> >>> On Tue, Apr 29, 2014 at 2:59 PM, Sean Mackrory wrote: >>> >>>> If your credentials are set up properly, you should be able to push to >>>> git.apache.org/bigtop.git >>>> >>>> >>>> On Tue, Apr 29, 2014 at 1:58 PM, Sean Mackrory >>>> wrote: >>>> >>>>> Bigtop uses Git. >>>>> >>>>> >>>>>> On Tue, Apr 29, 2014 at 1:56 PM, Jay Vyas wrote: >>>>>> >>>>>> Hi bigtop ! >>>>>> >>>>>> Can we commit code via git or svn is required for bigtop? If SVN is >>>>>> required, does anyone have a workflow i can borrow for putting patches >>>> in >>>>>> (i know git --signoff is part of the process but not sure wether this >>>> is >>>>>> only for people who use a git wrapper to svn or something fancy like >>>>>> that). >>>>>> >>>>>> Just wondering if i have to setup svn etc am reading the apache >>>> docs >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jay Vyas >>>>>> http://jayunit100.blogspot.com >>> >>> >>> >>> -- >>> Jay Vyas >>> http://jayunit100.blogspot.com >> >> >> >> -- >> Jay Vyas >> http://jayunit100.blogspot.com
Bigtop and Gradle: hope we can push each other forward.
hi bigtop! I've posted this question in the gradle forums, specifically directed at hopefully to bump gradle upstream packaging up as a first class priority, so that it will be easier for all of us to adopt bigtop at a broader level in the downstream. http://forums.gradle.org/gradle/topics/what_is_the_gradle_status_in_fedora_and_how_can_we_make_sure_it_continues_to_get_packaged_in_the If any of you know people in the gradle community, maybe we can try to work with them , or discuss with them, about possibilities for making gradle a first class linux tool. In the end it will increase bigtop adoption if its core toolchain has unambiguous licensing ans is easy to package. Any other thoughts about the elephant in the room (gradle is awesome and fun to use, but also its new to enterprises and they arent comfortable with its packaging yet), and how we can improve the situation?Or any of you folks know anyone in the gradle community we can work with on this?
Re: Git vs svn question
Thanks roman. Should i try to publish? On Wed, Apr 30, 2014 at 7:12 PM, Roman Shaposhnik wrote: > Our website deployment experience is clunky. I was about to write it up > in this email, but instead put it on the wiki: > > https://cwiki.apache.org/confluence/display/BIGTOP/Deploying+Bigtop's+Apache+Website > > NOTE: whoever will go through what I outlined should > really take a careful look at svn diff. It look like we > haven't pushed the website in a while and there's quite > a bit of diff in there. > > Thanks, > Roman. > > On Tue, Apr 29, 2014 at 3:44 PM, Jay Vyas wrote: > > Okay cos thanks. Yes... you deploy it that way but I don't know how the > permissions work. It's easy to test deploy though using "mvn site". > > > > I'm going to whip up some improvements to the site sometime soon I guess > we can all agree it's a little rustic. Will file jira once I get around to > looking deeper. > > > >> On Apr 29, 2014, at 5:22 PM, Konstantin Boudnik wrote: > >> > >> Ah, good! I think site can be only deployed with > >> mvn site:deploy > >> > >> that requires special permissions and all. I am a new chair, so I need > to > >> learn this rope next - gimme a little bit of time, please ;) > >> > >> Cos > >> > >>> On Tue, Apr 29, 2014 at 04:41PM, Jay Vyas wrote: > >>> Okay ! I did my first commit ! :) > >>> > >>> Now, how do i re-deploy the site? > >>> > >>> > >>>> On Tue, Apr 29, 2014 at 3:13 PM, Jay Vyas > wrote: > >>>> > >>>> gotcha thanks sean. > >>>> > >>>> > >>>> On Tue, Apr 29, 2014 at 2:59 PM, Sean Mackrory >wrote: > >>>> > >>>>> If your credentials are set up properly, you should be able to push > to > >>>>> git.apache.org/bigtop.git > >>>>> > >>>>> > >>>>> On Tue, Apr 29, 2014 at 1:58 PM, Sean Mackrory > > >>>>> wrote: > >>>>> > >>>>>> Bigtop uses Git. > >>>>>> > >>>>>> > >>>>>>> On Tue, Apr 29, 2014 at 1:56 PM, Jay Vyas > wrote: > >>>>>>> > >>>>>>> Hi bigtop ! > >>>>>>> > >>>>>>> Can we commit code via git or svn is required for bigtop? If SVN > is > >>>>>>> required, does anyone have a workflow i can borrow for putting > patches > >>>>> in > >>>>>>> (i know git --signoff is part of the process but not sure wether > this > >>>>> is > >>>>>>> only for people who use a git wrapper to svn or something fancy > like > >>>>>>> that). > >>>>>>> > >>>>>>> Just wondering if i have to setup svn etc am reading the > apache > >>>>> docs > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Jay Vyas > >>>>>>> http://jayunit100.blogspot.com > >>>> > >>>> > >>>> > >>>> -- > >>>> Jay Vyas > >>>> http://jayunit100.blogspot.com > >>> > >>> > >>> > >>> -- > >>> Jay Vyas > >>> http://jayunit100.blogspot.com > -- Jay Vyas http://jayunit100.blogspot.com
Re: Bigtop and Gradle: hope we can push each other forward.
Thanks will ! I guess for now its not a huge concern: BigTop will still ship and work just fine. And since most of the bigtop sources are in groovy and java anyways, the core stuff can be used in a variety of different ways. On Wed, Apr 30, 2014 at 9:31 PM, Will Benton wrote: > Hi Jay, > > The comments for this bug have some pointers to reasons why Gradle is no > longer in Fedora: > > https://bugzilla.redhat.com/show_bug.cgi?id=1029534 > > > best, > wb > > - Original Message - > > From: "Jay Vyas" > > To: dev@bigtop.apache.org > > Sent: Wednesday, April 30, 2014 6:03:52 PM > > Subject: Bigtop and Gradle: hope we can push each other forward. > > > > hi bigtop! > > > > I've posted this question in the gradle forums, specifically directed at > > hopefully to bump gradle upstream packaging up as a first class priority, > > so that it will be easier for all of us to adopt bigtop at a broader > level > > in the downstream. > > > > > http://forums.gradle.org/gradle/topics/what_is_the_gradle_status_in_fedora_and_how_can_we_make_sure_it_continues_to_get_packaged_in_the > > > > If any of you know people in the gradle community, maybe we can try to > work > > with them , or discuss with them, about possibilities for making gradle a > > first class linux tool. > > > > In the end it will increase bigtop adoption if its core toolchain has > > unambiguous licensing ans is easy to package. > > > > Any other thoughts about the elephant in the room (gradle is awesome and > > fun to use, but also its new to enterprises and they arent comfortable > with > > its packaging yet), and how we can improve the situation?Or any of you > > folks know anyone in the gradle community we can work with on this? > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: Using Ambari with vanilla Apache releases
Me to super interested in how this will materialize maybe a update on the jira: BIGTOP-709? > On May 2, 2014, at 9:38 AM, "Chuck Allen -T (charlall - SPHERION CORPORATION > at Cisco)" wrote: > > Greetings – > > Any chance you could provide the recorded session, for those of us who were > unable to attend? Thanks > Best, > -chuck > > From: Yusaku Sako [mailto:yus...@hortonworks.com] > Sent: Thursday, May 01, 2014 9:08 PM > To: u...@ambari.apache.org; d...@ambari.apache.org > Cc: dev@bigtop.apache.org; Mahadev Konar; Sean Busbey > Subject: Re: Using Ambari with vanilla Apache releases > > Thanks to everyone who attended the Ambari-BigTop meetup! > It was a very informative session and presented a great opportunity for folks > to learn about both Ambari and BigTop. > Hope to have more of these dev-focused sessions in the future. > > Yusaku > > On Thu, May 1, 2014 at 11:15 AM, Yusaku Sako > mailto:yus...@hortonworks.com>> wrote: > For those who wish to attend remotely, here's the WebEx info: > > ### > > Topic: Ambari-Bigtop Hackathon > Date: Thursday, May 1, 2014 > Time: 6:00 pm, Pacific Daylight Time (San Francisco, GMT-07:00) > Meeting Number: 624 773 329 > Meeting Password: bigdata > > > --- > To join the online meeting (Now from mobile devices!) > --- > 1. Go to > https://hortonworks.webex.com/hortonworks/j.php?MTID=m0b48c3cdec484c562f98b0aa95e3c4f6 > 2. If requested, enter your name and email address. > 3. If a password is required, enter the meeting password: bigdata > 4. Click "Join". > > To view in other time zones or languages, please click the link: > https://hortonworks.webex.com/hortonworks/j.php?MTID=mc94a66bcd3489f1e18214fcb9d97242b > > --- > To join the audio conference only > --- > To receive a call back, provide your phone number when you join the meeting, > or call the number below and enter the access code. > Call-in toll-free number (US/Canada): 1-877-668-4493 > Call-in toll number (US/Canada): 1-650-479-3208 > Global call-in numbers: > https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC&ED=304044357&tollFree=1 > Toll-free dialing restrictions: > http://www.webex.com/pdf/tollfree_restrictions.pdf > > Access code:624 773 329 > > ### > > Thanks, > Yusaku > > On Tue, Apr 29, 2014 at 12:23 PM, Yusaku Sako > mailto:yus...@hortonworks.com>> wrote: > Hi all, > > This is a reminder that the Ambari-BigTop hackathon is happening this > Thursday (May 1) at 6pm! > Hortonworks will host this event. > > Hortonworks is located at: > 3460 West Bayshore Road > Palo Alto, CA 94303 > USA > > If you are planning on attending, please reply to this message by Wednesday > evening so that we know how many to expect for ordering food, etc. > > Thanks, > Yusaku > > > > On Fri, Apr 25, 2014 at 9:48 AM, Sumit Mohanty > mailto:smoha...@hortonworks.com>> wrote: > Great. Yusaku and I will look into the logistics requirement and get back. > > On Fri, Apr 25, 2014 at 9:14 AM, Roman Shaposhnik > mailto:ro...@shaposhnik.org>> wrote: >> On Wed, Apr 23, 2014 at 6:24 PM, Sumit Mohanty >> mailto:smoha...@hortonworks.com>> wrote: >> In that case, lets do it next Thursday - May 1st, 6:00 PM onwards. If this >> time does not work pls. suggest an alternative. >> >> Roman, let us know if it can be hosted at Pivotal's Palo Alto office or >> Yusaku and I should arrange for it to be hosted at Hortonworks' Palo Alto >> office. > That would be perfect. Turns out I need a little bit more lead time to > host it at Pivotal, so if we can get together at HW office next week, > that'll be ideal -- personally I can be as flexible as needed, except > for afternoon on Tue (but I can do Tue before 4pm). > > Thanks, > Roman. > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You. > > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly proh
Re: commons logging : should we get rid of it?
Hi' folks... Okay here is my pitch for why we should move to slf4j. 1) hadoop is moving to use slf4j bindings asap, 2) hive actually is moving also to use slf4j: it's a known "bug" that hive has commons-logging bits floating around. Why so much fuss about logging? Because slf4j is easy to debug and manage, whereas commons logging is very tricky to debug (it is biased to log4j, has magic ways of setting up logging that can be tricky to unify on a large cluster, etc. > On May 11, 2014, at 11:44 PM, Mark Grover wrote: > > There are also other components in the stack that use it. See Hive for > example: > https://github.com/apache/hive/blob/trunk/pom.xml#L299 > > Unless I am missing a good reason to remove it, I am personally happy with > the status-quo w.r.t. commons-logging. > > >> On Sat, May 10, 2014 at 9:32 PM, Konstantin Boudnik wrote: >> >> What we are going to replace it with? >> >>> On Fri, May 09, 2014 at 08:27PM, Jay Vyas wrote: >>> Hi bigtop ! >>> >>> Shall we work to start getting rid of commons logging, as hadoop is doing >>> ? I see its a dependency of itest. >>> >>> +--- org.apache.bigtop.itest:itest-common:0.7.0 >>> |+--- org.codehaus.groovy:groovy-all:1.8.6 >>> |+--- junit:junit:4.11 >>> ||\--- org.hamcrest:hamcrest-core:1.3 >>> |+--- commons-logging:commons-logging:1.1 -> 1.1.1 >>> |+--- org.apache.ant:ant:1.8.2 >>> ||\--- org.apache.ant:ant-launcher:1.8.2 >>> |\--- org.apache.ant:ant-junit:1.8.2 >>> | +--- org.apache.ant:ant:1.8.2 (*) >>> | \--- junit:junit:3.8.2 -> 4.11 (*) >>> >>> I personally dont like the way that it figures out how to log (there is >>> complex hierarchy of logic to it). >>> >>> -- >>> Jay Vyas >>> http://jayunit100.blogspot.com >>
Re: Review Request 21293: Adding Apache Accumulo to Apache Bigtop
Hi shawn: I know your excited about getting this in so... given that I cant review this stuff yet all by myself because it willtake me a very long time to grasp everything, if you have the spare time i can review it together with you in a g+ hangout possibly given how big the patch is. just let me know, im available today On Fri, May 9, 2014 at 7:31 PM, Sean Mackrory wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/21293/ > --- > > Review request for bigtop. > > > Bugs: BIGTOP-1175 > https://issues.apache.org/jira/browse/BIGTOP-1175 > > > Repository: bigtop > > > Description > --- > > See JIRA > > > Diffs > - > > bigtop-packages/src/common/accumulo/accumulo-gc.svc PRE-CREATION > bigtop-packages/src/common/accumulo/accumulo-master.svc PRE-CREATION > bigtop-packages/src/common/accumulo/accumulo-monitor.svc PRE-CREATION > bigtop-packages/src/common/accumulo/accumulo-tracer.svc PRE-CREATION > bigtop-packages/src/common/accumulo/accumulo-tserver.svc PRE-CREATION > bigtop-packages/src/common/accumulo/conf/accumulo-env.sh PRE-CREATION > bigtop-packages/src/common/accumulo/conf/accumulo-metrics.xml > PRE-CREATION > bigtop-packages/src/common/accumulo/conf/accumulo-site.xml PRE-CREATION > bigtop-packages/src/common/accumulo/conf/generic_logger.xml PRE-CREATION > bigtop-packages/src/common/accumulo/conf/log4j.properties PRE-CREATION > bigtop-packages/src/common/accumulo/conf/monitor_logger.xml PRE-CREATION > bigtop-packages/src/common/accumulo/do-component-build PRE-CREATION > bigtop-packages/src/common/accumulo/install_accumulo.sh PRE-CREATION > bigtop-packages/src/common/hadoop/init-hcfs.json > d8825aa33839b88d09ee928e0be83c46e88a2992 > bigtop-packages/src/common/hadoop/init-hdfs.sh > 3a5fe361dd56308f465a52214835a42900a618cd > bigtop-packages/src/deb/accumulo/accumulo-doc.install PRE-CREATION > bigtop-packages/src/deb/accumulo/accumulo.install PRE-CREATION > bigtop-packages/src/deb/accumulo/accumulo.postinst PRE-CREATION > bigtop-packages/src/deb/accumulo/accumulo.preinst PRE-CREATION > bigtop-packages/src/deb/accumulo/accumulo.prerm PRE-CREATION > bigtop-packages/src/deb/accumulo/compat PRE-CREATION > bigtop-packages/src/deb/accumulo/control PRE-CREATION > bigtop-packages/src/deb/accumulo/copyright PRE-CREATION > bigtop-packages/src/deb/accumulo/install_init_scripts.sh PRE-CREATION > bigtop-packages/src/deb/accumulo/rules PRE-CREATION > bigtop-packages/src/deb/accumulo/service-postinst.tpl PRE-CREATION > bigtop-packages/src/deb/accumulo/service-postrm.tpl PRE-CREATION > bigtop-packages/src/deb/accumulo/source/format PRE-CREATION > bigtop-packages/src/rpm/accumulo/RPMS/.gitignore PRE-CREATION > bigtop-packages/src/rpm/accumulo/SOURCES/.gitignore PRE-CREATION > bigtop-packages/src/rpm/accumulo/SPECS/.gitignore PRE-CREATION > bigtop-packages/src/rpm/accumulo/SPECS/accumulo.spec PRE-CREATION > bigtop-packages/src/rpm/accumulo/SRPMS/.gitignore PRE-CREATION > bigtop-packages/src/templates/init.d.tmpl > 57923240c2402c66b758790fcd8522186827f836 > bigtop.mk 672562a5d60b1043c7495183c8a5c4bcc571c960 > > Diff: https://reviews.apache.org/r/21293/diff/ > > > Testing > --- > > See JIRA > > > Thanks, > > Sean Mackrory > > -- Jay Vyas http://jayunit100.blogspot.com
commons logging : should we get rid of it?
Hi bigtop ! Shall we work to start getting rid of commons logging, as hadoop is doing ? I see its a dependency of itest. +--- org.apache.bigtop.itest:itest-common:0.7.0 |+--- org.codehaus.groovy:groovy-all:1.8.6 |+--- junit:junit:4.11 ||\--- org.hamcrest:hamcrest-core:1.3 |+--- commons-logging:commons-logging:1.1 -> 1.1.1 |+--- org.apache.ant:ant:1.8.2 ||\--- org.apache.ant:ant-launcher:1.8.2 |\--- org.apache.ant:ant-junit:1.8.2 | +--- org.apache.ant:ant:1.8.2 (*) | \--- junit:junit:3.8.2 -> 4.11 (*) I personally dont like the way that it figures out how to log (there is complex hierarchy of logic to it). -- Jay Vyas http://jayunit100.blogspot.com
Re: [jira] [Created] (BIGTOP-1319) 'gradle hive-rpm' is broken
Looks like a typo the two Looks the same to me?
> 'HIVE_TARBALL_SRC=${HIVE_TARBALL_DST}
> On May 21, 2014, at 4:32 AM, "YoungWoo Kim (JIRA)" wrote:
>
> YoungWoo Kim created BIGTOP-1319:
>
>
> Summary: 'gradle hive-rpm' is broken
> Key: BIGTOP-1319
> URL: https://issues.apache.org/jira/browse/BIGTOP-1319
> Project: Bigtop
> Issue Type: Bug
> Components: Build
>Reporter: YoungWoo Kim
>Priority: Trivial
> Fix For: 0.8.0
>
>
> {noformat}
> $ gradle hive-rpm
>
> ..
>
> :hive_vardefines
> :hive-download FAILED
>
> FAILURE: Build failed with an exception.
>
> * Where:
> Script '/home/vagrant/bigtop/packages.gradle' line: 88
>
> * What went wrong:
> Execution failed for task ':hive-download'.
>> Could not download file
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output.
>
> BUILD FAILED
>
> {noformat}
>
> In bigtop.mk, 'HIVE_TARBALL_SRC=${HIVE_TARBALL_DST}' should be
> 'HIVE_TARBALL_SRC=$(HIVE_TARBALL_DST)'
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.2#6252)
Re: [jira] [Created] (BIGTOP-1319) 'gradle hive-rpm' is broken
Ah ok! Makes sense. Curly braces are isolating variable names.. and
parenthesis are for lambdas Good catch.
> On May 21, 2014, at 6:39 AM, 김영우 wrote:
>
> I just replace curly braces with round brackets. :-)
>
> - Youngwoo
>
>
>> On Wed, May 21, 2014 at 7:08 PM, Jay Vyas wrote:
>>
>> Looks like a typo the two Looks the same to me?
>>> 'HIVE_TARBALL_SRC=${HIVE_TARBALL_DST}
>>
>>
>>>> On May 21, 2014, at 4:32 AM, "YoungWoo Kim (JIRA)"
>>> wrote:
>>>
>>> YoungWoo Kim created BIGTOP-1319:
>>>
>>>
>>>Summary: 'gradle hive-rpm' is broken
>>>Key: BIGTOP-1319
>>>URL: https://issues.apache.org/jira/browse/BIGTOP-1319
>>>Project: Bigtop
>>> Issue Type: Bug
>>> Components: Build
>>> Reporter: YoungWoo Kim
>>> Priority: Trivial
>>>Fix For: 0.8.0
>>>
>>>
>>> {noformat}
>>> $ gradle hive-rpm
>>>
>>> ..
>>>
>>> :hive_vardefines
>>> :hive-download FAILED
>>>
>>> FAILURE: Build failed with an exception.
>>>
>>> * Where:
>>> Script '/home/vagrant/bigtop/packages.gradle' line: 88
>>>
>>> * What went wrong:
>>> Execution failed for task ':hive-download'.
>>>> Could not download file
>>>
>>> * Try:
>>> Run with --stacktrace option to get the stack trace. Run with --info or
>> --debug option to get more log output.
>>>
>>> BUILD FAILED
>>>
>>> {noformat}
>>>
>>> In bigtop.mk, 'HIVE_TARBALL_SRC=${HIVE_TARBALL_DST}' should be
>> 'HIVE_TARBALL_SRC=$(HIVE_TARBALL_DST)'
>>>
>>>
>>>
>>>
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v6.2#6252)
>>
Re: [DISCUSS] Next Bigtop Hack-a-thon
Aren't we mostly spaced out anyways :)? I'll be available in California for a hackathon until the 10th. Maybe if 6th doesn't work, we can do the following Monday (9th). Red hat could possibly also host I think. > On May 23, 2014, at 12:55 AM, Konstantin Boudnik wrote: > > Thanks Andre! Thanks for making it happen again! Do you think we can perhaps > do a week later? 6th would be a day after Hadoop Summit and I presume people > would be pretty much spaced out ;) > > Thoughts? > Cos > >> On Tue, May 20, 2014 at 06:18PM, Andre Arcilla wrote: >> To keep with our bi-monthly hack-a-thon schedule, I propose a new Bigtop >> hack-a-thon for June 6. It can be hosted @ A9 in Palo Alto (subject to >> large CR availability). >> >> Please express your interest, date/time, topic preferences. >> >> Thanks >> >> -- >> Andre
Re: [DISCUSS] Next Bigtop Hack-a-thon
Equally satisfied - either date works for me: votes from others? 6th or 9th? Meanwhile I'm checking to see if I can arrange a room in mountain view. > On May 23, 2014, at 3:10 PM, Konstantin Boudnik wrote: > > To clarify on my point - I am ok with 6th, just saying that it might not be > very crowded. I am +1 for either date. > > Cos > >> On Fri, May 23, 2014 at 07:23AM, Jay Vyas wrote: >> Aren't we mostly spaced out anyways :)? I'll be available in California for >> a hackathon until the 10th. Maybe if 6th doesn't work, we can do the >> following Monday (9th). >> >> Red hat could possibly also host I think. >> >>> On May 23, 2014, at 12:55 AM, Konstantin Boudnik wrote: >>> >>> Thanks Andre! Thanks for making it happen again! Do you think we can perhaps >>> do a week later? 6th would be a day after Hadoop Summit and I presume people >>> would be pretty much spaced out ;) >>> >>> Thoughts? >>> Cos >>> >>>> On Tue, May 20, 2014 at 06:18PM, Andre Arcilla wrote: >>>> To keep with our bi-monthly hack-a-thon schedule, I propose a new Bigtop >>>> hack-a-thon for June 6. It can be hosted @ A9 in Palo Alto (subject to >>>> large CR availability). >>>> >>>> Please express your interest, date/time, topic preferences. >>>> >>>> Thanks >>>> >>>> -- >>>> Andre
Re: BigTop Hackathon :::: Double Header ::: RSVP please !
thanks anyone else ? update the spreadsheet plz :) the more RSVPs the better a job i can do to make sure everyone leaves feeling fat and happy. Maybe we can even schedule a couple of small presentations. On Fri, May 23, 2014 at 6:25 PM, Konstantin Boudnik wrote: > I am in and will bring 3 or 4 people with me at least on Friday. > > Mine main goal is 0.8.0 and the infra for it: there's a couple of tickets > and > gentle nudges from the community to fix the build slaves, so I will work > on it > and am welcoming everyone who can help. > > Looking forward to see as many of the contributors as possible! > And thanks to Jay (and RedHat for hosting us)! > > Andre, could you please post this on the meetup page as well? > Thanks, > Cos > > On Fri, May 23, 2014 at 06:20PM, Jay Vyas wrote: > > Hi folks. Ok we are all set ! > > > > BigTop's first double-header bigdata hackathon *** > > > > *Fill out your details if you can make it, here: * > > > > > https://docs.google.com/spreadsheets/d/1UviFZ2GbItmRPWqNR7EIhqklGfwSiDPBmc3fJRyrlFo/ > > > > > > *I've booked our training room at red hat for the both the 6th and the > > 9th. * > > 1) How many of us are interested in coming for two days? Plenty of coffee > > soft drinks and pizza will be provided for both ! > > > > 2) Can some of you come and collaborate / work on some JIRAs at red hat > > for the day ? > > > > *Anyways, hope to hear back from at least a few people.* > > > > If enough are interested, we will be able to two two days (friday the 6th > > and the 9th). > > > > Ping me back soon So i can start formalizing the arrangements ! > > > > > > > > -- > > Jay Vyas > > http://jayunit100.blogspot.com > > -- Jay Vyas http://jayunit100.blogspot.com
Re: BigTop Hackathon :::: Double Header ::: RSVP please !
Red Hat offices, mountain view ! 444 Castro St, Mountain View, CA 9404 On Fri, May 23, 2014 at 8:17 PM, Giridharan Kesavan < gkesa...@hortonworks.com> wrote: > venue please? > > -giri > > > On Fri, May 23, 2014 at 4:36 PM, Jay Vyas wrote: > > > thanks anyone else ? update the spreadsheet plz :) the more RSVPs the > > better a job i can do to make sure everyone leaves feeling fat and happy. > > Maybe we can even schedule a couple of small presentations. > > > > > > On Fri, May 23, 2014 at 6:25 PM, Konstantin Boudnik > > wrote: > > > > > I am in and will bring 3 or 4 people with me at least on Friday. > > > > > > Mine main goal is 0.8.0 and the infra for it: there's a couple of > tickets > > > and > > > gentle nudges from the community to fix the build slaves, so I will > work > > > on it > > > and am welcoming everyone who can help. > > > > > > Looking forward to see as many of the contributors as possible! > > > And thanks to Jay (and RedHat for hosting us)! > > > > > > Andre, could you please post this on the meetup page as well? > > > Thanks, > > > Cos > > > > > > On Fri, May 23, 2014 at 06:20PM, Jay Vyas wrote: > > > > Hi folks. Ok we are all set ! > > > > > > > > BigTop's first double-header bigdata hackathon *** > > > > > > > > *Fill out your details if you can make it, here: * > > > > > > > > > > > > > > https://docs.google.com/spreadsheets/d/1UviFZ2GbItmRPWqNR7EIhqklGfwSiDPBmc3fJRyrlFo/ > > > > > > > > > > > > *I've booked our training room at red hat for the both the 6th and > the > > > > 9th. * > > > > 1) How many of us are interested in coming for two days? Plenty of > > coffee > > > > soft drinks and pizza will be provided for both ! > > > > > > > > 2) Can some of you come and collaborate / work on some JIRAs at red > > hat > > > > for the day ? > > > > > > > > *Anyways, hope to hear back from at least a few people.* > > > > > > > > If enough are interested, we will be able to two two days (friday the > > 6th > > > > and the 9th). > > > > > > > > Ping me back soon So i can start formalizing the arrangements ! > > > > > > > > > > > > > > > > -- > > > > Jay Vyas > > > > http://jayunit100.blogspot.com > > > > > > > > > > > > -- > > Jay Vyas > > http://jayunit100.blogspot.com > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Jay Vyas http://jayunit100.blogspot.com
Re: BigTop Hackathon :::: Double Header ::: RSVP please !
Ice scheduled it for all day both days.. I figure some folks will show at around noon, others will come around 4 or so. - Not everyone will be available for the whole 2 days but - hopefully enough people will come and go so that we can get some good work done. I'll assume people won't show up until after 11am, but I'll be there by 10 to get things set up. > On May 28, 2014, at 10:37 AM, Mark Grover wrote: > > Thanks for organizing this! > > Is it at 10 am on both days then? It would be good to know when it ends in > case there are some people who can't make it earlier in the morning. > > >> On Fri, May 23, 2014 at 11:07 PM, Andre Arcilla wrote: >> >> The meetup has been announced: >> http://www.meetup.com/Bay-Area-Bigtop-Meetup/events/184893732/ >> >> >> >>> On Fri, May 23, 2014 at 5:56 PM, Jay Vyas wrote: >>> >>> Red Hat offices, mountain view ! >>> 444 Castro St, Mountain View, CA 9404 >>> >>> >>> On Fri, May 23, 2014 at 8:17 PM, Giridharan Kesavan < >>> gkesa...@hortonworks.com> wrote: >>> >>>> venue please? >>>> >>>> -giri >>>> >>>> >>>> On Fri, May 23, 2014 at 4:36 PM, Jay Vyas >> wrote: >>>> >>>>> thanks anyone else ? update the spreadsheet plz :) the more RSVPs the >>>>> better a job i can do to make sure everyone leaves feeling fat and >>> happy. >>>>> Maybe we can even schedule a couple of small presentations. >>>>> >>>>> >>>>> On Fri, May 23, 2014 at 6:25 PM, Konstantin Boudnik >>>>> wrote: >>>>> >>>>>> I am in and will bring 3 or 4 people with me at least on Friday. >>>>>> >>>>>> Mine main goal is 0.8.0 and the infra for it: there's a couple of >>>> tickets >>>>>> and >>>>>> gentle nudges from the community to fix the build slaves, so I will >>>> work >>>>>> on it >>>>>> and am welcoming everyone who can help. >>>>>> >>>>>> Looking forward to see as many of the contributors as possible! >>>>>> And thanks to Jay (and RedHat for hosting us)! >>>>>> >>>>>> Andre, could you please post this on the meetup page as well? >>>>>> Thanks, >>>>>> Cos >>>>>> >>>>>>> On Fri, May 23, 2014 at 06:20PM, Jay Vyas wrote: >>>>>>> Hi folks. Ok we are all set ! >>>>>>> >>>>>>> BigTop's first double-header bigdata hackathon >>> *** >>>>>>> >>>>>>> *Fill out your details if you can make it, here: * >> https://docs.google.com/spreadsheets/d/1UviFZ2GbItmRPWqNR7EIhqklGfwSiDPBmc3fJRyrlFo/ >>>>>>> >>>>>>> >>>>>>> *I've booked our training room at red hat for the both the 6th >> and >>>> the >>>>>>> 9th. * >>>>>>> 1) How many of us are interested in coming for two days? Plenty >> of >>>>> coffee >>>>>>> soft drinks and pizza will be provided for both ! >>>>>>> >>>>>>> 2) Can some of you come and collaborate / work on some JIRAs at >>> red >>>>> hat >>>>>>> for the day ? >>>>>>> >>>>>>> *Anyways, hope to hear back from at least a few people.* >>>>>>> >>>>>>> If enough are interested, we will be able to two two days (friday >>> the >>>>> 6th >>>>>>> and the 9th). >>>>>>> >>>>>>> Ping me back soon So i can start formalizing the arrangements ! >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jay Vyas >>>>>>> http://jayunit100.blogspot.com >>>>> >>>>> >>>>> -- >>>>> Jay Vyas >>>>> http://jayunit100.blogspot.com >>>> >>>> -- >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >> entity >>> to >>>> which it is addressed and may contain information that is confidential, >>>> privileged and exempt from disclosure under applicable law. If the >> reader >>>> of this message is not the intended recipient, you are hereby notified >>> that >>>> any printing, copying, dissemination, distribution, disclosure or >>>> forwarding of this communication is strictly prohibited. If you have >>>> received this communication in error, please contact the sender >>> immediately >>>> and delete it from your system. Thank You. >>> >>> >>> >>> -- >>> Jay Vyas >>> http://jayunit100.blogspot.com >>
Proposed update to bigtop commit rules for documentation
Hi bigtop ! Can you folks +1/-1 this proposal: ** From now on, commiters can push README/doc updates without review, as long as they mention on the JIRA that they are pushing without review. (code review is still critical to the community process, SO to prevent this from getting out of control, id suggest the following 4 rules to go along with this) ** This doesnt apply to anything that is code - even dead or prototype code ** All commits still require JIRAs. ** Commiter has to mention that they are pushing without review. ** Commiter should wait at least an hour after attaching patch before pushing without review , so that others have some time at least to respond. If agreed I'll formalize this in the wiki. -- Jay Vyas http://jayunit100.blogspot.com
Re: Proposed update to bigtop commit rules for documentation
FYI ! +5/-0 votes came in for this ., Okay ! Its official. See new wiki guidelines for commiters ! https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute Now lets get started on automated doc testing. On Wed, Jun 11, 2014 at 9:42 AM, Sean Mackrory wrote: > +1 > > > On Sun, Jun 8, 2014 at 11:04 PM, Konstantin Boudnik > wrote: > > > +1 > > -- > > Regards, > > Cos > > > > On June 3, 2014 5:02:22 PM PDT, Jay Vyas wrote: > > >Hi bigtop ! > > > > > >Can you folks +1/-1 this proposal: > > > > > >** From now on, commiters can push README/doc updates without review, > > >as > > >long as they mention on the JIRA that they are pushing without review. > > > > > >(code review is still critical to the community process, SO to prevent > > >this > > >from getting out of control, id suggest the following 4 rules to go > > >along > > >with this) > > > > > >** This doesnt apply to anything that is code - even dead or prototype > > >code > > >** All commits still require JIRAs. > > >** Commiter has to mention that they are pushing without review. > > >** Commiter should wait at least an hour after attaching patch before > > >pushing without review , so that others have some time at least to > > >respond. > > > > > >If agreed I'll formalize this in the wiki. > > > > > -- Jay Vyas http://jayunit100.blogspot.com
Re: [ANNOUNCE] Bigtop Jenkins EC2 AMIs outage
whoa…. thanks for catching this On Jun 25, 2014, at 12:58 AM, Konstantin Boudnik wrote: > Super! Thank you very much dude! > > On Tue, Jun 24, 2014 at 09:51PM, Roman Shaposhnik wrote: >> Hi! >> >> there has been an outage on AWS today that >> brought down our build slaves and the Jenkins >> instance itself. I now restored it all, but since some >> of the slaves didn't come back cleanly I had >> to either replace them or bring them down >> completely. >> >> Everything is back to normal now, except: >> 1. centos5 is decommissioned and we will >> no longer support it in Bigtop (as per BOM) >> 2. fedora17 is decommissioned and I don't >> think I can justify time bringing it back. >> Hence I'd like to request an amendment >> to the BOM >> 3. Ubuntu Quantal is decommissioned and >> I replaced it with the latest Ubuntu LTS 14.04 >> (as per BOM). >> >> Please let me know if you still see any issues. >> >> Thanks, >> Roman.
Re: Thoughts on Bigtop 0.8.0
Im not casting a vote :BUT thinking out loud: If bigtop is really the "fedora" of bigdata distros, then moving to Java 7 is the natural thing to do. Fedora integrates the trailblazing linux technology - without fear of consequences and I see BigTop as doing the exact same thing in the hadoop community. For me -- BigTop is a platform to innovate and do cool new stuff on -- so I say keep it that way and keep fearlessly moving forward :) On Thu, Jun 26, 2014 at 6:34 PM, Roman Shaposhnik wrote: > Hi! > > based on Andrew P. suggestion I bumped the > version of JDK6 to the latest update (45) and > it seems to have cured the HBase problem. > > As a side note: for a deprecated, unsupported > platform such as JDK6 there's no reason to use > anything but the latest version anyway. > > But even with that, we still do have quite a few > failures in various components (Hive, Pig, etc.). > > How do we attack it? Thoughts? > > Thanks, > Roman. > -- jay vyas
Okay to get ball rolling for pull requests?
Hi folks. Pull requests would be an awesome development for bigtop - it would make it really easy to test and review patches (etc etc I'm sure you all know the benefits :)) so ... Should I get started down the road of investigating how to get pull requests enabled for apache projects ? I'm not sure what it involves, maybe some kind of two way mirroring, but I'm sure it will be a good thing to have. If agreement I'll file a JIRA and track progress there. -- jay vyas
Re: Okay to get ball rolling for pull requests?
Sure we can add gerrit --- but I believe the spark project which is Apache, for example, they have 100% pull request integration. We've had people in the past submit awesome stuff via PR (https://github.com/apache/bigtop/pull/3), which never got through. Im suggesting first class pull requests . Is that really illegal and if so - are the spark folks just given a pass to expedite things? If so … shouldn't we ask for the same pass ? I think this functionality alone could grow the community around bigtop more than any other single action which we could undertake. On Jun 29, 2014, at 7:07 PM, Roman Shaposhnik wrote: > On Sun, Jun 29, 2014 at 3:48 PM, jay vyas wrote: >> Hi folks. >> >> Pull requests would be an awesome development for bigtop - it would make it >> really easy to test and review patches (etc etc I'm sure you all know the >> benefits :)) >> >> so ... Should I get started down the road of investigating how to get pull >> requests enabled for apache projects ? I'm not sure what it involves, maybe >> some kind of two way mirroring, but I'm sure it will be a good thing to >> have. >> >> If agreement I'll file a JIRA and track progress there. > > PR's are already enabled via GitHub integration. We will still required > the JIRA roundtrip, though as per ASF requirements. > > That said, what would be really awesome is if somebody could look > at pull requests via the https://code.google.com/p/gerrit/ a'la: > https://source.android.com/source/developing.html > > Thanks, > Roman.
Re: Okay to get ball rolling for pull requests?
sorry if i wasn't clear but, I do totally agree gerrit will be better for reviews, especially complex ones from different sources , than github. On Jun 29, 2014, at 7:07 PM, Roman Shaposhnik wrote: > On Sun, Jun 29, 2014 at 3:48 PM, jay vyas wrote: >> Hi folks. >> >> Pull requests would be an awesome development for bigtop - it would make it >> really easy to test and review patches (etc etc I'm sure you all know the >> benefits :)) >> >> so ... Should I get started down the road of investigating how to get pull >> requests enabled for apache projects ? I'm not sure what it involves, maybe >> some kind of two way mirroring, but I'm sure it will be a good thing to >> have. >> >> If agreement I'll file a JIRA and track progress there. > > PR's are already enabled via GitHub integration. We will still required > the JIRA roundtrip, though as per ASF requirements. > > That said, what would be really awesome is if somebody could look > at pull requests via the https://code.google.com/p/gerrit/ a'la: > https://source.android.com/source/developing.html > > Thanks, > Roman.
Re: Okay to get ball rolling for pull requests?
push the merge button…… Almost - but not quite !
Sorry for the lack of clarity --- as i recently learned of this process. I've
gotten up to speed on the details: Here they are:
I'll walk through the way this script
https://github.com/apache/spark/blob/master/dev/merge_spark_pr.py
is used…..
1) Rather than actually submitting a patch, the user submits a pull request
and a branch is created
by the commiter, automatically using the script:
run_cmd("git fetch %s pull/%s/head:%s" % (PR_REMOTE_NAME, pr_num,
pr_branch_name))
2) Then, the script checks via github api if the patch is mergeable.
pr = get_json("%s/pulls/%s" % (GITHUB_API_BASE, pr_num))
3) Then, the pull request is merged and pushed to apache:
merge_hash = merge_pr(pr_num, target_ref)
Re: Okay to get ball rolling for pull requests?
The actual script is done by the commiters. You can read about the process in the spark commiter guidelines: https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage So the steps would be 1) Adopt the script for processing pull requests 2) Add it to our bigtop repo 3) Create a pull request and test the script for comitting it. Some might dispute one issue here : The patch system record of is a little diluted. Is that a big deal? Im not sure . I think its a tough decision - its no simple answer. If losing track of patches builds a significantly more vibrant ecosystem, i think its an okay tradeoff. Any other opinions on this? On Jun 30, 2014, at 6:38 PM, Andrew Bayer wrote: > builds
Re: Thoughts on Bigtop 0.8.0
+1 for jdk 7 ! On Mon, Jun 30, 2014 at 10:34 PM, Konstantin Boudnik wrote: > Yup, +1 > > On Sun, Jun 29, 2014 at 05:19PM, Roman Shaposhnik wrote: > > On Thu, Jun 26, 2014 at 7:03 PM, Andrew Purtell > wrote: > > > With a bit of hacking on the build we could cross-compile to avoid the > type > > > related build failures that are creeping in as projects don't try > compiling > > > with JDK 6 any longer. > > > > > > $ javac -target 1.6 -bootclasspath /path/to/jdk/1.6/lib/rt.jar > > > > > > Nasty to require a JDK 6 and 7 for building, though. > > > > At this point, I'd suggest biting the bullet and declaring 0.8.0 > > JDK7+ only. Even for build. Three of the dozen or so projects > > already refuse to be built by anything but: > >* Hue 3.6 (can be hacked to stay with JDK6, but who will do it?) > >* Solr 4.9.0 (ditto) > >* Giraph 1.1.0 (not sure whether JDK7 specific functionality remains) > > > > So, unless somebody volunteers to solve all three of the above, > > I think 0.8.0 is about to move to JDK7. > > > > Scream within a week or forever...you know. > > > > Thanks, > > Roman. > -- jay vyas
Re: Okay to get ball rolling for pull requests?
Okay. Is it okay if a bot +1s the patch on behalf of a commiter ? That way commiter can directly merge a pull request (from their perspective), by running a script which turns the levers and knobs in the jira? Leave any final thoughts here.Now that we are all in reasonabley same page --- Ill create a jira outlining this task!!! > On Jul 1, 2014, at 2:30 PM, Roman Shaposhnik wrote: > >> On Mon, Jun 30, 2014 at 3:38 PM, Andrew Bayer wrote: >> Screw gerrit, we've already got the Jenkins Enterprise GitHub pull >> request builder plugin enabled on builds.apache.org - it needs some >> validation still and hooks to be set up by Infra on the GitHub repos, >> but once that's done, you can have a job that watches for pull >> requests, builds them, and comments on the pull request with the build >> results. > > This would be awesome path forward, actually. I especially like the > possibility for an automated patch validation. > > Thanks, > Roman.
Re: Okay to get ball rolling for pull requests?
s/+1/Resolves Sorry about the confusion : Im not suggesting bots review patches and +1 them :) On Jul 1, 2014, at 3:37 PM, Jay Vyas wrote: > reasonabley
[VOTE] Removing support for Makefile ?
BIGTOP-1314 highlights the fact that we now have 2 build systems: Makefile and build.gradle. And the fact that the Makefile approach is now deprecated. As per Mark's suggestion in that JIRA, the future of Makefile should be decided on the mailing list before the next JIRA comes out to further distance from Make and embrace gradle. Should we continue to support the Makefile builder? Vote +1 : To eliminate the Makefile Vote -1 : To keep the makefile and support both. It seems very inelegant to have both build systems at the same time, to me But I know there is another side to the debate.
Re: [DISCUSS] Removing support for Makefile? Was: [VOTE]
I agree wiith mark that we need to make it easy and individuals in the community need to be comfortable with the transition. I propose a solution at the end o this email. Heres where we are at: - Realistically, the debate on which is "better" is not going to be very fruitful We all know the high-level pros and cons of both Makefile and build.gradle . - Neither is perfect,but > 50% of community parties agree gradle is a step forward. jay +.1 (im netural with a slight lean to gradle) cos, roman, sean, +3 (all are ready to move forward) mark effectively might say is a -1 * - But we don't want to leave anyone behind! ** But we also need to move fast ! We dont want a schizophrenic or forked community. SO HERE IS MY SUGGESTION: 1) we schedule a meetup, or a screencast - specifically to go through the gradle code - from A to Z - 2) We validate and build all of bigtop, using gradle, during the screencast. NO EXCEPTIONS. That way we are all 100% sure that it works, and we see it in action. 3) After that screen cast, we ensure that we have a unified community which can self-sufficiently administer the gradle based build - as soon as possible. 4) If all parties to (1) agree that they are now ready , we delete the Makefile forever, in a patch which updates the README file with excellent explanation of how the build.gradle works. This is the most rapid way to move the entire community forward in unison and prevent code rot of maintaining duplicate features . Mark, and others, how do you feel about this idea? On Sun, Jul 13, 2014 at 2:16 PM, Mark Grover wrote: > Thanks for starting this thread, Jay and Cos. > > Here are my thoughts: > Gradle may actually be a better choice for Bigtop. However, in my opinion, > Make should be kept around for a little while to make the transition easy > for the community. > > I would go as far as saying that we shouldn't be deprecating Make right > now. In my opinion, it's best to have a transition period where we all > support both Gradle and Make. During this time, contributors and committers > work (albeit, with a little extra pain) with both and develop their own > opinion on which tool is the best for the project. Some of us may already > have developed such experience by using various tools in the past while > others may have not. The idea is to give all members of the community an > opportunity to make such a decision for themselves and share them with the > community. > And, consequently, the official decision to deprecate a tool from the > project shouldn't happen before this transition period, but after. > > Hope that makes sense. > Mark > > > > On Sat, Jul 12, 2014 at 6:34 AM, Sean Mackrory > wrote: > > > I'm not super familiar with Gradle yet, but here are my thoughts: > > > > - It can't be any worse than make. I mean, $$($(1)-deb). I'd much prefer > > debugging a more modern language, and as Cos points out, it's intended > for > > a JVM ecosystem but can do other things when needed. > > - I'd prefer we leave Make around for perhaps one more release just to > > really solidify the Gradle system a bit more. However if we keep it > > deprecated, I see no point in "maintaining" both, meaning that if people > > want to add new features to the build (several JIRAs going on for that > > right now) - there's no need to keep adding that to the Makefile, let's > > just keep the versions and metadata for new projects up to date. I don't > > believe we routinely run into bugs, so I doubt much work will have to be > > put in outside of bigtop.mk. > > > > > > On Fri, Jul 11, 2014 at 11:54 PM, Roman Shaposhnik > > > wrote: > > > > > On Fri, Jul 11, 2014 at 6:01 PM, Konstantin Boudnik > > > wrote: > > > > I am temporarily putting the [VOTE] thread to halt and instead > starting > > > > [DISCUSS]. Per Jay's: > > > > > > > > BIGTOP-1314 highlights the fact that we now have 2 build systems: > > > Makefile > > > > and build.gradle. And the fact that the Makefile approach is now > > > > deprecated. > > > > > > > > As per Mark's suggestion in that JIRA, the future of Makefile should > be > > > > decided on the mailing list before the next JIRA comes out to further > > > > distance from Make and embrace gradle. > > > > > > > > Should we continue to support the Makefile builder? > > > > > > I'd be more than happy to get rid of it right after we release Bigtop > > 0.8.0 > > > > > > Thanks, > > > Roman. > > > > > > -- jay vyas
Re: Starting Hadoop in Distributed Mode
I suggest using puppet as well, way easier than doing it manually. basically i think you could - clone down thebigtop github and checkout branch-0.7.0 - put those puppet recipes on your bare metal nodes, and updeate the config csv file to point to ip of master - run puppet apply on each node thats it. It should all i think just work automagically. right? On Mon, Jul 21, 2014 at 2:44 PM, David Fryer wrote: > Yes, Bigtop 0.7.0 is installed. > > -David Fryer > > > On Mon, Jul 21, 2014 at 2:33 PM, Konstantin Boudnik > wrote: > >> Sorry for being a nag - did you install Bigtop 0.7.0? >> >> Cc'ing dev@ list as well >> Cos >> >> On Mon, Jul 21, 2014 at 01:15PM, David Fryer wrote: >> > I activated the bigtop yum repository, and installed the required hadoop >> > packages via yum. All of the computers in the cluster are running CentOS >> > 6.5. >> > >> > -David Fryer >> > >> > >> > On Mon, Jul 21, 2014 at 1:01 PM, Konstantin Boudnik >> wrote: >> > >> > > I see that your daemon is trying to log to >> > > /usr/lib/hadoop/logs whereas Bigtop logs under /car/log as required by >> > > Linux services good behavior rules. >> > > >> > > The way namenode recognizes DNs isn't via slaves file, but by DNs >> register >> > > with NN via RPC mechanism. >> > > >> > > How did you install the Hadoop? Using Bigtop packages or via a >> different >> > > mechanism? The fact that you are seeing error message about cygwin not >> > > found tells me that you are using a derivative bits, not pure Bigtop. >> Is >> > > this the case? >> > > >> > > Regards >> > > Cos >> > > >> > > On July 21, 2014 9:32:48 AM PDT, David Fryer >> wrote: >> > > >When I tried starting hadoop using the init scripts provided, the >> > > >master >> > > >couldn't find any of the datanodes. It is my understanding that the >> > > >masters >> > > >file is optional, but the slaves file is required. The scripts that >> > > >reference the slaves file are named in plural (instead of >> > > >hadoop-daemon.sh, >> > > >use hadoop-daemons.sh). I tried modifying the init scripts to run >> > > >hadoop-daemons.sh, and the script attempted to spawn processes on the >> > > >slaves referenced in the slaves file, but that produced the error: >> > > >Starting Hadoop namenode: [ OK ] >> > > >slave2: starting namenode, logging to >> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-loki.out >> > > >master: starting namenode, logging to >> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-odin.out >> > > >slave3: starting namenode, logging to >> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-tyr.out >> > > >slave1: starting namenode, logging to >> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-thor.out >> > > >slave2: /usr/lib/hadoop-hdfs/bin/hdfs: line 34: >> > > >/usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or >> > > >directory >> > > >slave2: /usr/lib/hadoop-hdfs/bin/hdfs: line 150: cygpath: command not >> > > >found >> > > >slave2: /usr/lib/hadoop-hdfs/bin/hdfs: line 191: exec: : not found >> > > >slave3: /usr/lib/hadoop-hdfs/bin/hdfs: line 34: >> > > >/usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or >> > > >directory >> > > >slave3: /usr/lib/hadoop-hdfs/bin/hdfs: line 150: cygpath: command not >> > > >found >> > > >slave3: /usr/lib/hadoop-hdfs/bin/hdfs: line 191: exec: : not found >> > > >master: /usr/lib/hadoop-hdfs/bin/hdfs: line 34: >> > > >/usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or >> > > >directory >> > > >master: /usr/lib/hadoop-hdfs/bin/hdfs: line 150: cygpath: command not >> > > >found >> > > >master: /usr/lib/hadoop-hdfs/bin/hdfs: line 191: exec: : not found >> > > >slave1: /usr/lib/hadoop-hdfs/bin/hdfs: line 34: >> > > >/usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or >> > > >directory >> > > >slave1: /usr/lib/hadoop-hdfs/bin/hdfs: line 150: cygpath: command not >> > > >found >> > > >slave1: /usr/lib/hadoop-hdfs/bin/hdfs: line 191: exec: : not found >> > > > >> > > >-David Fryer >> > > > >> > > > >> > > >On Mon, Jul 21, 2014 at 12:18 PM, Konstantin Boudnik > > >> > > >wrote: >> > > > >> > > >> Hi David. >> > > >> >> > > >> Slaves files are really optional if I remember right. In Bigtop we >> > > >are >> > > >> usually >> > > >> deploy Hadoop with provided Puppet recipes which are >> battle-hardened >> > > >over >> > > >> the >> > > >> years :) >> > > >> >> > > >> Cos >> > > >> >> > > >> On Mon, Jul 21, 2014 at 10:53AM, David Fryer wrote: >> > > >> > Hi Bigtop! >> > > >> > >> > > >> > I'm working on trying to get hadoop running in distributed mode, >> > > >but the >> > > >> > init scripts don't seem to be referencing the slaves file in >> > > >> > /etc/hadoop/conf. Has anyone encountered this before? >> > > >> > >> > > >> > Thanks, >> > > >> > David Fryer >> > > >> >> > > >> > > >> > > -- jay vyas
Re: Starting Hadoop in Distributed Mode
FYI Im going to try what i suggested to make sure im not sending you off on a wild goose chase @davide :) got a centos box spun up, will let you know if it works from scratch. then you can copy the recipe. I'll create a wiki page for it. On Mon, Jul 21, 2014 at 6:44 PM, jay vyas wrote: > I suggest using puppet as well, way easier than doing it manually. > > basically i think you could > > - clone down thebigtop github and checkout branch-0.7.0 > - put those puppet recipes on your bare metal nodes, and updeate the > config csv file to point to ip of master > - run puppet apply on each node > > thats it. It should all i think just work automagically. > right? > > > > On Mon, Jul 21, 2014 at 2:44 PM, David Fryer wrote: > >> Yes, Bigtop 0.7.0 is installed. >> >> -David Fryer >> >> >> On Mon, Jul 21, 2014 at 2:33 PM, Konstantin Boudnik >> wrote: >> >>> Sorry for being a nag - did you install Bigtop 0.7.0? >>> >>> Cc'ing dev@ list as well >>> Cos >>> >>> On Mon, Jul 21, 2014 at 01:15PM, David Fryer wrote: >>> > I activated the bigtop yum repository, and installed the required >>> hadoop >>> > packages via yum. All of the computers in the cluster are running >>> CentOS >>> > 6.5. >>> > >>> > -David Fryer >>> > >>> > >>> > On Mon, Jul 21, 2014 at 1:01 PM, Konstantin Boudnik >>> wrote: >>> > >>> > > I see that your daemon is trying to log to >>> > > /usr/lib/hadoop/logs whereas Bigtop logs under /car/log as required >>> by >>> > > Linux services good behavior rules. >>> > > >>> > > The way namenode recognizes DNs isn't via slaves file, but by DNs >>> register >>> > > with NN via RPC mechanism. >>> > > >>> > > How did you install the Hadoop? Using Bigtop packages or via a >>> different >>> > > mechanism? The fact that you are seeing error message about cygwin >>> not >>> > > found tells me that you are using a derivative bits, not pure >>> Bigtop. Is >>> > > this the case? >>> > > >>> > > Regards >>> > > Cos >>> > > >>> > > On July 21, 2014 9:32:48 AM PDT, David Fryer >>> wrote: >>> > > >When I tried starting hadoop using the init scripts provided, the >>> > > >master >>> > > >couldn't find any of the datanodes. It is my understanding that the >>> > > >masters >>> > > >file is optional, but the slaves file is required. The scripts that >>> > > >reference the slaves file are named in plural (instead of >>> > > >hadoop-daemon.sh, >>> > > >use hadoop-daemons.sh). I tried modifying the init scripts to run >>> > > >hadoop-daemons.sh, and the script attempted to spawn processes on >>> the >>> > > >slaves referenced in the slaves file, but that produced the error: >>> > > >Starting Hadoop namenode: [ OK ] >>> > > >slave2: starting namenode, logging to >>> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-loki.out >>> > > >master: starting namenode, logging to >>> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-odin.out >>> > > >slave3: starting namenode, logging to >>> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-tyr.out >>> > > >slave1: starting namenode, logging to >>> > > >/usr/lib/hadoop/logs/hadoop-hadoopuser-namenode-thor.out >>> > > >slave2: /usr/lib/hadoop-hdfs/bin/hdfs: line 34: >>> > > >/usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or >>> > > >directory >>> > > >slave2: /usr/lib/hadoop-hdfs/bin/hdfs: line 150: cygpath: command >>> not >>> > > >found >>> > > >slave2: /usr/lib/hadoop-hdfs/bin/hdfs: line 191: exec: : not found >>> > > >slave3: /usr/lib/hadoop-hdfs/bin/hdfs: line 34: >>> > > >/usr/lib/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or >>> > > >directory >>> > > >slave3: /usr/lib/hadoop-hdfs/bin/hdfs: line 150: cygpath: command >>> not >>> > > >found >>> > > >slave3: /usr/lib/hadoop-hdfs/bin/hdfs: line 191: exec: : not found >&g
Re: Starting Hadoop in Distributed Mode
Okay ! Here you go. https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet I got it working from scratch in very short order. I think you've gone and done some manual stuff unnecessarily. Use puppet, its good for your health :)
Re: Starting Hadoop in Distributed Mode
No prob cos. Now I'm getting curious : What is our take on passwordless SSH for bigtop hadoop distro ? I never set up passwordless ssh in bigtop that i remember (but maybe im forgetting). I think that puppet apply on each slave is sufficient for a cluster, because it starts the slaves for you and connects them to the master - without need of master ssh'ing anyhwere. On Tue, Jul 22, 2014 at 12:20 AM, Konstantin Boudnik wrote: > Well done - thank you very much Jay! > > Cos > > On Mon, Jul 21, 2014 at 10:52PM, jay vyas wrote: > > Okay ! Here you go. > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet > > > > I got it working from scratch in very short order. I think you've gone > and > > done some manual stuff unnecessarily. > > > > Use puppet, its good for your health :) > -- jay vyas
Time to upgrade the site !
hey guys. https://issues.apache.org/jira/secure/attachment/12657862/BIGTOP-1312.patch looks good to me from screen shots and am testing now. assuming all goes well ill be redeploying the site today. anyone else has any objections? -- jay vyas
[VOTE]
Hi bigtopTime to vote on the great gradle vs makefile debate. Vote: +1 to remove makefile support after 0.8.0 release, -1 If you want to keep make file post 0.8.0.
Re: [VOTE]
okay folks: The VOTE to retire the support of the make build system after the 0.8.0 release, has now passed. 5 +1 0 0 0 -1 On Mon, Aug 4, 2014 at 1:36 PM, Roman Shaposhnik wrote: > On Sat, Aug 2, 2014 at 10:38 AM, Sean Mackrory > wrote: > > +1 > > > > I was going to be funny and use the Make syntax for incrementing a > variable > > to cast my vote, but then I remembered why I was voting +1 in the first > > place. > > Best! +1! Ever! ;-) > > And a belated +1 from me as well. > > Thanks, > Roman. > -- jay vyas
Re: Bigtop 0.8.0 progress
Hi roman and others: how do you feel about this: i think it might be ** better for the bigtop community** if we **gated release** on **fully cleaned up CI** even if that means delaying things a few weeks. do you folks think this might be a good idea? after all, maybe the best way to force ourselves to clean it up is to get 0.8.0 release on functioning CI ! and then , with a functioning CI, the time saved will allow to release 0.9.0 way faster, so the time will be made up shortly/// opinions on this +1/-1/0 ? if others think maybe its a good idea , maybe we can schedule a g+ hangout for wednesday, after work, to carve a path forward. On Mon, Aug 11, 2014 at 2:25 AM, Roman Shaposhnik wrote: > With the latest changes that went in over the weekend > I'm happy to report that at least on the minimum set of > package tests we're down to one issue only: > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-packagetest/label=lucid-slave/lastCompletedBuild/testReport/ > > Various hive-* packages seem to fail on install > on Ubuntu 10.04. > > If anybody can take a look -- it'll be greatly appreciated. > > Also, the Jenkins EC2 issues still remains and it > seems to consistently affect only 2 of our dynamic > slaves: fedora18 and opensuse12. The rest are fine. > > Thanks, > Roman. > -- jay vyas
Re: Bigtop 0.8.0 progress
no prob !... actually i beleive i was the one that steered this thread off topic .:). sounds good to keep proceeding w/ release . just wanted to see what others thought about the CI and release status.
