Hi,
When I reached out to the Apache Yetus dev mailing list for some
information regarding the addition of precommit check.
I got the reply from them which said that AVRO might want to change the
approach of which they run the tests.
I wanted to know if everyone was okay with what was being suggested before
I go around trying to make it work.
What is the opinion of people ?
---------- Forwarded message ----------
> From: Allen Wittenauer <a...@effectivemachines.com>
> Date: Tue, Sep 13, 2016 at 8:13 AM
> Subject: Re: Avro Pre-commit.
> To: d...@yetus.apache.org
>
>
> I've been thinking about this issue. With a bit of work, I think the
> approach should probably be to:
> a) make a custom build tool that knows which dirs use which other
> tools
> b) add a new file in each dir that can tell precommit where to
> break for modules
>
> For example, let's say we have a dir structure like this:
>
> build.sh
> c/
> Makefile
> java/
> pom.xml
> perl/
> Makefile.PL
>
> Let's add a file called multibuild.sh that precommit can use:
>
> build.sh
> multibuild.sh
> c/
> Makefile
> multibuild.sh
> java/
> pom.xml
> multibuild.sh
> perl/
> Makefile.PL
> multibuild.sh
>
>
> In our "multibuild" build tool plugin, we'd define
>
> function multibuild_buildfile
> {
> echo "multibuild.sh"
> }
>
> Now when precommit gets a patch, it will know that a change in c/
> is a different module than a change in java/ or perl/ because of the
> presence of the multibuild.sh file. This prevents the "build the world"
> problem for a largely diverse project like avro.
>
> The place where this gets hard is our theoretical
> multibuild_executor function because precommit doesn't pass as a param the
> module that's currently being processed. (we should probably fix that) But
> we can still make it work and reality, it might have led us down a false
> path. If we make this function look like this:
>
> function multibuild_executor
> {
> echo "multibuild.sh"
> }
>
> and make sure that BUILDTOOLCWD=module (the default), then precommit will
> run our file that we placed to actually do the work. This file can then
> take whatever parameters we're getting passed and convert it as appropriate
> to the build tool for that directory. Since this is a custom build tool,
> you can dictate what parameters to pass to which modules via
> _modules_workers. For example, an easy route might be:
>
> function multibuild_modules_workers
> {
> modules_workers $2
> }
>
> then multibuild.sh for each language becomes a series of case statements.
> For C, it might look like:
>
> testname=$1
> shift
>
> case ${testname} in
> compile)
> make -Dcustomflag=1
> ;;
> doc)
> make man
> unit)
> make -Danotherflag=1 test
> ;;
> *)
> ;;
> esac
>
> It's pretty hacky, but theoretically it would work until we get a
> chance to make precommit smarter about multiple build tools in a single
> project. (which would be a pretty hard undertaking, but something we should
> probably do)
>
>
> > On Sep 10, 2016, at 6:32 PM, suraj acharya <suraj....@gmail.com> wrote:
> >
> > Yes.
> > The ./build.sh test
> > <https://github.com/apache/avro/blob/master/build.sh#L40> command helps
> run
> > all the tests from the top level.
> > However, I am not sure if we need to run all the tests when a change is
> > made?
> > I was thinking of running language specific tests for changes in the
> > specific directory.
> > For example for changes in python will trigger ant test
> > <https://github.com/apache/avro/blob/master/build.sh#L43>.
>
-Suraj Acharya
On Fri, Aug 19, 2016 at 9:54 AM, Sean Busbey <bus...@cloudera.com> wrote:
> If we have files that fail to include the required ASF licensing
> headers, you're correct that's important to fix.
>
> I think either working on it before or after would be fine. Ideally,
> Yetus Precommit should properly let us know when a patch is fixing
> those kinds of problems, so it would be a good validation that things
> are working.
>
> On Thu, Aug 18, 2016 at 8:18 PM, suraj acharya <suraj....@gmail.com>
> wrote:
> > Okay. I will look at the docker command and the image.
> >
> > One other question I have is that the asflicense is an important aspect
> of
> > yetus. However, many java files have that missing. And whenever a patch
> > touches that file it returns a -1. Do you think I should first fix all
> the
> > licenses and then continue with this?
> >
> > S
> >
> > -Suraj Acharya
> >
> > On Thu, Aug 18, 2016 at 12:37 PM, Sean Busbey <bus...@cloudera.com>
> wrote:
> >
> >> is the 3-5 minutes doing the tests across all of the language
> >> libraries or just the java ones?
> >>
> >> docker will definitely be needed, due to the number of different
> >> system dependencies needed to build the different languages.
> >> I was hoping we could reuse the Docker image that is currently
> >> used for the "./build.sh docker" command.
> >>
> >>
> >> On Wed, Aug 17, 2016 at 6:41 PM, suraj acharya <suraj....@gmail.com>
> >> wrote:
> >> > Hi,
> >> > I am a new guy and I have been working on setting up pre-commit check
> for
> >> > Avro : AVRO-1887 <https://issues.apache.org/jira/browse/AVRO-1887>.
> >> >
> >> > I was looking at the unit tests being run currently. It takes
> somewhere
> >> > around 3-5 minutes depending on the network and the repo's present.
> >> > I was thinking that since the time is so low why not run all the tests
> >> for
> >> > every commit. Also, the tests being lightweight I was thinking that
> there
> >> > is no need for docker to be needed.
> >> >
> >> > I would like to gather your opinions on this topic before going
> further
> >> > down this path.
> >> >
> >> > Thanks
> >> >
> >> > -Suraj Acharya
> >>
> >>
> >>
> >> --
> >> busbey
> >>
>
>
>
> --
> busbey
>
