I have the current builds down to about 30 minutes if that helps as a data
point. I think we can build/test -j3 on the makefile without blowing up
memory. Perhaps we could give this a try while you refactor.
Cheers,
Brian
On Tue, Jun 13, 2017 at 10:18 AM, Greg Landrum <greg.land...@gmail.com>
wrote:
>
>
> On Tue, Jun 13, 2017 at 3:24 PM, Brian Kelley <fustiga...@gmail.com>
> wrote:
>
>> I'm unsure if that staging will help, aren't the builds and tests already
>> done in parallel effectively?
>>
>
> Kind of. There are currently two jobs that run in parallel:
> - job 1: build the core library, the python 2.7 wrappers, and the java
> wrappers and test everything
> - job 2: build the core library, the python 3.5 wrappers, and the java
> wrappers and test everything
>
> Those have no dependencies on each other. I would hope that by pulling the
> core library and test builds out into a separate job and then using the
> artifacts (=libraries) from that for the python and java builds we could
> shorten the individual jobs.
>
> -greg
>
>
>
>
>> On Tue, Jun 13, 2017 at 9:20 AM, Greg Landrum <greg.land...@gmail.com>
>> wrote:
>>
>>> Given that the current variability is almost a factor of two I don't
>>> think smaller changes are going to help.
>>>
>>> What I'm likely going to try is doing staged builds:
>>> - stage 1: build and test core library
>>> - stage 2a: build and test python 2.7 wrappers
>>> - stage 2b: build and test python 3.5 wrappers
>>> - stage 2c: build and test java wrappers
>>>
>>> the three "stage 2" things can run in parallel.
>>>
>>> From the bit of reading I did on the train this morning, it looks like
>>> this is not that horrible to do in jenkins; just need to invest a bit of
>>> time.
>>>
>>> -greg
>>>
>>>
>>> On Tue, Jun 13, 2017 at 3:12 PM, Brian Kelley <fustiga...@gmail.com>
>>> wrote:
>>>
>>>> I tried using ninja for the builds in stead of make, after an evening
>>>> of not building I successfully had two builds:
>>>>
>>>> https://travis-ci.org/bp-kelley/rdkit/builds/242399233
>>>>
>>>> Commit:
>>>>
>>>> https://github.com/bp-kelley/rdkit/blob/baac00bc1eb7f6641301
>>>> 1aed239413fe74da3388/.travis.yml
>>>>
>>>> Note that the default ninja build is #cores + 2 which blew out the
>>>> memory on the VMs so I used -j3 we could try this in the makefile as well.
>>>>
>>>> I also think we could shave off a few minutes by
>>>>
>>>> 1. building in the root conda env
>>>> 2. having only one conda install for *everything*
>>>>
>>>> If you like I can try these fixes.
>>>>
>>>> Cheers,
>>>> Brian
>>>>
>>>> On Tue, Jun 13, 2017 at 3:44 AM, Greg Landrum <greg.land...@gmail.com>
>>>> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> We're getting more and more timeouts with RDKit builds on travis. I'm
>>>>> reasonably sure that this is due to some variability in the performance of
>>>>> the VMs that the builds end up running on. A good demonstration of that is
>>>>> this particular build (on a branch where I'm playing with simplifying the
>>>>> travis builds), which works fine for py35 but times out for py27:
>>>>> https://travis-ci.org/rdkit/rdkit/builds/242301061?utm_sourc
>>>>> e=email&utm_medium=notification
>>>>>
>>>>> If you look into the details you'll see that the main compile "only"
>>>>> takes 1475 seconds for the python 3.5 build but it is killed after 2541
>>>>> seconds in the python2.7 build. Given that it's the same code being built,
>>>>> it seems like this more or less has to be infrastructure variability.
>>>>>
>>>>> I'll continue to explore, particularly breaking the build into stages,
>>>>> but if anyone has suggestions on how to improve this please let me know.
>>>>>
>>>>> -greg
>>>>>
>>>>>
>>>>> ------------------------------------------------------------
>>>>> ------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>> _______________________________________________
>>>>> Rdkit-devel mailing list
>>>>> Rdkit-devel@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/rdkit-devel
>>>>>
>>>>>
>>>>
>>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Rdkit-devel mailing list
Rdkit-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-devel
