Re: [DISCUSS] Unblocking Juju development

[Pete Vander Giessen]

Hi All,

> I don't think a mega PR will help with the issue much, as the review of 
> mega-anything
is a mega-PITA.

A mega PR sounds like a hassle to me, too. :-) Maybe we're misunderstanding
what you're asking us to do.

> Sometimes these fixes are intertwined with the charms and that's where
the snug is: most of us don't have enough hands-on experience with Juju at
this point to make an informed review and/or test the changes (as has
been discussed
earlier).

Would it help to put a charm and the puppet changes in the same PR? I know
that I generally try to separate them, partially to encourage myself to
only make puppet changes that make sense alone, outside of the context of a
charm. I do tend to validate and test inside a charm, though.

Is testing the main blocker? I typically test by building a charm and then
using bundletester to deploy the charm to AWS and run tests, including the
Bigtop smoke tests. If anybody needs an AWS account for charm testing
purposes, they can go to https://developer.juju.solutions/ and set one up.
I made some improvements to the juju documentation on running tests (
https://jujucharms.com/docs/stable/developer-testing#executing-tests-via-bundletester),
but I'm happy to help people out on IRC if anything is unclear (more
feedback means that I can write better documentation).

Regards,
~ PeteVG

Re: [DISCUSS] Unblocking Juju development

[Konstantin Boudnik]

On Tue, Aug 09, 2016 at 05:58PM, Pete Vander Giessen wrote:
> Hi All,
> 
> Would it make sense to to make a branch that lived in our fork of the
> bigtop project (github.com/juju-solutions/bigtop)? That way, we charmers
> all have commit access, and can grant commit access to those working on a
> charm. We can then open a sort of mega PR against the Apache bigtop repo,
> and you can review and merge that when you're ready.

Having a branch on your own never was prohibited nor it should be :) The way
people choose to collaborate on the GH isn't really a concern here, but the
way we can efficiently get the changes into the project master is. 

I don't think a mega PR will help with the issue much, as the review of
mega-anything is a mega-PITA. This is the reason most of the projects are
trying to make commits narrowly focused and reasonable in size.

> The issue that I see is coordinating upstream merges with downstream
> additions. Are there points where you want us to freeze the branch while
> you prepare to merge it?

The coordination is a part of it. Another part is the addition of the charms
which lead to discovery of various issues in the Puppet and elsewhere.
Sometimes these fixes are intertwined with the charms and that's where the
snug is: most of us don't have enough hands-on experience with Juju at this
point to make an informed review and/or test the changes (as has been
discussed earlier). This results in slowing down of the rate of the acceptance
of the changes. Which isn't good for anybody.

> Also, are you proposing that the branch contain just the charms themselves
> -- the files in in bigtop-packages/src/charms -- or do you want us to
> include puppet fixes and changes in the branch? I think that it might make
> a lot of sense to keep all the directly charm related stuff in a branch,
> but would strongly prefer to keep the puppet related fixes in their own,
> per ticket, branches. That mega branch is going to get rebased frequently,
> and I'd rather cut down the chances of conflicts and messiness to a minimum.

Bigtop source code doesn't live in multiple repositories, so I don't see how
we can fragment the code this way. In my practice, you'd have all the code on
the branch, but people will be working just on the particular areas of the
code as they see fit. Once there's a consensus that the branch is in a good
shape it should be merged back to the master, It is normally done in one swift
motion. No need to freeze anything as the granularity of the branches could
vary - per feature, per fix, per mega-functionality, etc.

Hope it makes sense.
  Cos

Re: [DISCUSS] Unblocking Juju development

[Pete Vander Giessen]

Hi All,

Would it make sense to to make a branch that lived in our fork of the
bigtop project (github.com/juju-solutions/bigtop)? That way, we charmers
all have commit access, and can grant commit access to those working on a
charm. We can then open a sort of mega PR against the Apache bigtop repo,
and you can review and merge that when you're ready.

The issue that I see is coordinating upstream merges with downstream
additions. Are there points where you want us to freeze the branch while
you prepare to merge it?

Also, are you proposing that the branch contain just the charms themselves
-- the files in in bigtop-packages/src/charms -- or do you want us to
include puppet fixes and changes in the branch? I think that it might make
a lot of sense to keep all the directly charm related stuff in a branch,
but would strongly prefer to keep the puppet related fixes in their own,
per ticket, branches. That mega branch is going to get rebased frequently,
and I'd rather cut down the chances of conflicts and messiness to a minimum.

Regards,
~ PeteVG

Re: [DISCUSS] Unblocking Juju development

[Roman Shaposhnik]

Now my time to be late with a replay, but the good news -- I'm back
from vacation! :-)

On Tue, Aug 2, 2016 at 12:09 PM, Konstantin Boudnik  wrote:
> Sorry for the belated joining of the party - crazy travel schedule an all
> that. I like the idea of making a special branch for this development, so ppl
> working on would have the ability to commit the code as they see fit and, once
> it is ready, we can simply merge it into the master.
>
> Is this something along the lines you had in mind, Roman?

Yes. That's exactly what I had in mind. But if we agree that the
branch is the right
way to proceed, we then need to ask how to enable non-committers to be able
to commit it.

Thanks,
Roman.

Re: [DISCUSS] Unblocking Juju development

[Konstantin Boudnik]

Sorry for the belated joining of the party - crazy travel schedule an all
that. I like the idea of making a special branch for this development, so ppl
working on would have the ability to commit the code as they see fit and, once
it is ready, we can simply merge it into the master.

Is this something along the lines you had in mind, Roman?
  Cos

On Tue, Jul 19, 2016 at 03:13PM, Roman Shaposhnik wrote:
> On Tue, Jul 19, 2016 at 3:06 PM, Cory Johns  wrote:
> > Also, we realized that while we did internal review among the four of us of
> > the PRs we created, we didn't document those reviews on the Jira tickets.
> > I'd like to have us go back and re-review those to have it documented on
> > the Jira tickets exactly what we did to test those charms and what the
> > results were, which I think would be very helpful for your review.
> 
> Sure that would be helpful, but like I said -- my biggest hurdle is not being
> able to see forest for the trees. That's why I can't really review
> much as it stands.
> 
> Well, I suppose I can recreate that branch I'm talking about myself,
> but I'd rather
> you guys maintained it ;-)
> 
> Thanks,
> Roman.

Re: [DISCUSS] Unblocking Juju development

[Roman Shaposhnik]

On Tue, Jul 19, 2016 at 3:30 PM, Cory Johns  wrote:
> The individual Jiras that we proposed should be self-contained if we don't
> mark them as dependent on another Jira ticket and ready for review if they
> have a patch attached.

They depend on the patches that the Bigtop base layer throws in. That connection
is not articulated anywhere but in the code of the Bigtop layer.

A slightly more esoteric problem is that the Charms I've looked at so
far are structured
slightly differently. Now, I get the fact that commonality among all
the charms is elusive,
but perhaps we can at least get to certain commonality in the
Bigtop-derived Charms.

Here's an example from my recent memory: I remember looking at one
Charm implementing
tests as a call to smoke-test action and another one just running some
code. If I had a single
branch I could've immediately looked up how the rest of the Bigtop
charms do it. Stuff like
that.

> The issue with the bigtop-base layer having to live outside the Bigtop repo
> is a limitation of the charm-build tooling (
> https://github.com/juju/charm-tools/issues/212) and I don't think it will
> be resolved soon.

Right and this very well may be my inexperience with Juju, but my ideal state
is havign Bigtop layer in the same repo so I can freely move code between
Puppet, Charms and Bigtop layer for refactoring purposes.

> However, the build process is always going to depend on
> some external layers, such as the reactive base layer, and some interface
> layers.  This is just an aspect of how building charms from layers works.
> Do you consider this an issue for the charms in Bigtop?

At least initially Bigtop layer is really tightly coupled. Over time,
I'd agree -- it
will be less and less useful to have it closeby in the same repo.

> We tend to think
> of it like a "compile" step for the charms and the layers pulled from
> http://interfaces.juju.solutions/ to be analogous to using libraries from
> PyPI.

Sure. But running with your analogy -- I agree that while I can link
both glibc and
libthrift just as opaque binary objects I need source code for thrift
a lot more than
I do for glibc.

Thanks,
Roman.

Re: [DISCUSS] Unblocking Juju development

[Cory Johns]

The individual Jiras that we proposed should be self-contained if we don't
mark them as dependent on another Jira ticket and ready for review if they
have a patch attached.

The issue with the bigtop-base layer having to live outside the Bigtop repo
is a limitation of the charm-build tooling (
https://github.com/juju/charm-tools/issues/212) and I don't think it will
be resolved soon.  However, the build process is always going to depend on
some external layers, such as the reactive base layer, and some interface
layers.  This is just an aspect of how building charms from layers works.
Do you consider this an issue for the charms in Bigtop?  We tend to think
of it like a "compile" step for the charms and the layers pulled from
http://interfaces.juju.solutions/ to be analogous to using libraries from
PyPI.

On Tue, Jul 19, 2016 at 6:13 PM, Roman Shaposhnik 
wrote:

> On Tue, Jul 19, 2016 at 3:06 PM, Cory Johns 
> wrote:
> > Also, we realized that while we did internal review among the four of us
> of
> > the PRs we created, we didn't document those reviews on the Jira tickets.
> > I'd like to have us go back and re-review those to have it documented on
> > the Jira tickets exactly what we did to test those charms and what the
> > results were, which I think would be very helpful for your review.
>
> Sure that would be helpful, but like I said -- my biggest hurdle is not
> being
> able to see forest for the trees. That's why I can't really review
> much as it stands.
>
> Well, I suppose I can recreate that branch I'm talking about myself,
> but I'd rather
> you guys maintained it ;-)
>
> Thanks,
> Roman.
>

Re: [DISCUSS] Unblocking Juju development

[Roman Shaposhnik]

On Tue, Jul 19, 2016 at 3:06 PM, Cory Johns  wrote:
> Also, we realized that while we did internal review among the four of us of
> the PRs we created, we didn't document those reviews on the Jira tickets.
> I'd like to have us go back and re-review those to have it documented on
> the Jira tickets exactly what we did to test those charms and what the
> results were, which I think would be very helpful for your review.

Sure that would be helpful, but like I said -- my biggest hurdle is not being
able to see forest for the trees. That's why I can't really review
much as it stands.

Well, I suppose I can recreate that branch I'm talking about myself,
but I'd rather
you guys maintained it ;-)

Thanks,
Roman.

Re: [DISCUSS] Unblocking Juju development

[Roman Shaposhnik]

On Tue, Jul 19, 2016 at 11:11 AM, Cory Johns  wrote:
> Roman,
>
> Can you explain a bit more about the Bigtop branch workflow?  We certainly
> want to make sure we follow the best practices to make reviewing our
> proposals as easy as possible, and we realize that introducing a new tool
> (Juju) adds a barrier as it is.

What I meant is to create a dedicated branch for you where you will put all
of your current code and will keep refactoring it until it is ready for review
and inclusion in Bigtop. That way, I can always check the entire branch
when I do the review instead of guessing the context of various patches.

Here's a silly example that still tripped me up quite a bit: I didn't
realize that
Charms on JIRAs require a Bigtop layer to be self-contained. Stuff like that.

Thanks,
Roman.

Re: [DISCUSS] Unblocking Juju development

[Cory Johns]

Also, we realized that while we did internal review among the four of us of
the PRs we created, we didn't document those reviews on the Jira tickets.
I'd like to have us go back and re-review those to have it documented on
the Jira tickets exactly what we did to test those charms and what the
results were, which I think would be very helpful for your review.

On Tue, Jul 19, 2016 at 2:11 PM, Cory Johns 
wrote:

> Roman,
>
> Can you explain a bit more about the Bigtop branch workflow?  We certainly
> want to make sure we follow the best practices to make reviewing our
> proposals as easy as possible, and we realize that introducing a new tool
> (Juju) adds a barrier as it is.
>
> On Mon, Jul 18, 2016 at 8:38 PM, Roman Shaposhnik  wrote:
>
>> Hi!
>>
>> going over amazing work that Canonical folks are
>> pushing our way:
>> https://github.com/apache/bigtop/pulls
>> made me realize that a piecemeal approach to
>> reviewing and committing those patches probably
>> is not going to work for. At least not for me.
>>
>> The challenge is that there's a certain degree of
>> interdependency that makes it difficult to review
>> standalone patches.
>>
>> This is the kind of work that is meant to be done
>> on a branch. Canonical folks, however, are
>> not Bigtop committers which makes a self-managed
>> branch not an option for them.
>>
>> What would be the best way to facilitate their work?
>> Anybody has any thoughts on that?
>>
>> Thanks,
>> Roman.
>>
>
>

[DISCUSS] Unblocking Juju development

[Roman Shaposhnik]

Hi!

going over amazing work that Canonical folks are
pushing our way:
https://github.com/apache/bigtop/pulls
made me realize that a piecemeal approach to
reviewing and committing those patches probably
is not going to work for. At least not for me.

The challenge is that there's a certain degree of
interdependency that makes it difficult to review
standalone patches.

This is the kind of work that is meant to be done
on a branch. Canonical folks, however, are
not Bigtop committers which makes a self-managed
branch not an option for them.

What would be the best way to facilitate their work?
Anybody has any thoughts on that?

Thanks,
Roman.