I'm pretty sure we don't want to expose a raw ExecutorService. We are
pretty careful about exposing Gradle APIs that we have control over, rather
than APIs that we don't have control over.
One of the benefits of providing a specific API for parallel external
processes is that Gradle could automatically buffer the input/output,
provide progress reporting, etc. I haven't put a lot of thought into what
our Gradle-wide parallel mechanism will look like, but if we start with an
AsyncExecAction (or similar) we can later change AsyncExecAction to wrap
the new API.
If you want to create a completely separate service for this (instead of
extending ExecActionFactory) then that would be fine, too.
Daz
On Sat, Jul 5, 2014 at 1:20 PM, Daniel Lacasse <daniel.lacass...@gmail.com>
wrote:
> I was thinking about this Daz. Would it be more flexible, extensible and
> powerful if we would just inject a singleton whenever someone request it of
> a ExecutorService that is configured according to the parallel-threads
> count?
>
> Creating an AsyncExecAction is more code to maintain and support as
> opposed to letting the developers choose to just wrap an normal ExecAction
> into a Callable and submitting it to the ExecutorService.
>
> The pro is we get closer to option #2 and #3 for a Gradle-wide parallel
> mechanism. The con is do we really want to expose the actual
> ExecutorService?
>
> I realise that we should explore this a bit more. What are your thoughts
> on this?
>
> ---
> Daniel
> ------------------------------
> On Thursday, July 3, 2014 11:09 PM, Daz DeBoer <
> darrell.deb...@gradleware.com> wrote:
>
>
>
>
> On Wed, Jul 2, 2014 at 7:36 PM, Daniel Lacasse <
> daniel.lacass...@gmail.com> wrote:
>
>> Sounds like a better plan as it would add more building blocks to gradle.
>>
>> Correct me if I'm wrong. They would probably be a global ExecutorService
>> to avoid recreating it at ever task and share the threads between everyone
>> who use it.
>
>
> Yes. ExecActionFactory is currently implemented by
> DefaultFileOperations, which is created per-project. So if you just make
> the change there you could have an ExecutorService per project. We could
> further improve this if we like by adding something to BuildScopeServices
> that is shared by all project instances.
>
> For now, I'd keep the implementation of
> ExecActionFactory.createAsyncAction nice and simple, because Adam is sure
> to have some ideas on how this could be better structured, when he's back
> from holidays.
>
> The problem I see - which could be a temporary thing until option #3 is
>> implemented - is parallel-threads will be the number of parallel task
>> executed at the same time as the number of compiler fork. For someone using
>> the parallel flags, it may be non-intuitive. It basically boils down to how
>> do we share the parallel-threads number between parallel task execution and
>> number of compiler fork? In my mind parallel-threads is the maximum number
>> of threads that Gradle should be allowed to create for his parallel
>> business.
>>
>
> Yes, we'd be overloading the 'parallel-threads' setting, so that you'd
> have both N projects being built in parallel, as well as N files being
> compiled in parallel. But the end goal is to use a single setting to
> control all parallelism, and this is a step in that direction.
>
>
>> Thanks again Daz and I will move in the direction you are suggesting.
>>
>> ---
>> Daniel
>> ------------------------------
>> On Wednesday, July 2, 2014 09:34 AM, Daz DeBoer <
>> darrell.deb...@gradleware.com> wrote:
>>
>> On Mon, Jun 30, 2014 at 7:24 PM, Daniel Lacasse <
>> daniel.lacass...@gmail.com> wrote:
>>
>>> I did some research on how to implement option #1 and here is the
>>> implementation:
>>>
>>> - In
>>> org.gradle.nativebinaries.language.c.tasks.AbstractNativeCompilerTask:
>>> - Add option int maximumNumberOfCompilerFork = 1
>>> - 1 is to prevent any side effect. It will mainly behave just
>>> like it was previously.
>>> - I'm open to suggestion on the chosen name.
>>> - No @Input on the new option
>>> - In org.gradle.nativebinaries.toolchain.internal.NativeCompileSpec:
>>> - set/getMaximumNumberOfCompilerFork
>>> - In
>>> org.gradle.nativebinaries.language.internal.AbstractNativeCompilerSpec:
>>> - set/getMaximumNumberOfCompilerFork implementation
>>> - In
>>>
>>> org.gradle.nativebinaries.toolchain.internal.{gcc|msvcpp}.NativeCompilerSpec:
>>> - Add modification to method WorkResult execute(T spec)
>>> - Use a ThreadPoolExecutor with maximumPoolSize set to
>>> maximumNumberOfCompileFork and corePoolSize set to 1
>>> - Submit anonymous Callable<WorkResult> class to the
>>> ThreadPoolExecutor
>>> - Use the get method on each Future returned by Submit to join
>>> all the work together.
>>>
>>> This cover pretty much the option #1 for parallel compilation in the
>>> native extension. The 2 open issues would be:
>>>
>>> - Is maximumNumberOfCompilerFork a good choice for the option name?
>>> If not, what would be the appropriate name?
>>>
>>> Let's avoid adding a configuration option for this at this stage.
>> What we really want is to make this configurable at a much higher level,
>> and auto-detect the optimal setting.
>>
>> For now, let's use the value of 'parallel-threads' for this setting.
>> Even though this is a bit coarse grained, it takes us in the right
>> direction. So instead of passing the value from
>> Task->CompileSpec->NativeCompiler, we would access the value from
>> StartParameter#getParallelThreadCount.
>>
>> I think the easiest way to do this would be to add a method to
>> ExecActionFactory that provided an AsyncExecAction, which would be similar
>> to ExecAction, but something like:
>>
>> public interface AsyncExecAction extends ExecSpec {
>> Future<ExecResult> submit();
>> }
>>
>> You could then modify
>> org.gradle.nativebinaries.toolchain.internal.CommandLineTool to use async
>> actions, and to present some sort of submit/collect API. This would mean
>> that the parallel functionality would be available everywhere that
>> CommandLineTool is used, which means both VisualCpp and Gcc based tool
>> chains.
>>
>> How does this sound? While it may seem a bit more involved, the design
>> will be largely the same. You'll still need a ThreadPoolExecuter and code
>> to submit/collect, but this design takes your suggested solution and moves
>> most of the code out of the NativeCompiler implementations into the
>> CommandLineTool and ExecActionFactory. The nice thing about this is that
>> it makes this feature more generally available, and makes is simpler for us
>> to have a global setting for parallelisation. (The funny thing is that this
>> is looking more like Option #2!)
>>
>> *So native compilation would happen in parallel whenever --parallel
>> and/or --parallel-threads is used*. As Adam suggested, we should really
>> separate these 2 options so that --parallel is a separate option to enable
>> parallel project execution, while parallel-threads is a general purpose
>> setting that defaults to a reasonable value.
>> (Currently StartParameter#getParallelThreadCount defaults to zero if
>> neither --parallel or --parallel-threads is specified).
>>
>>>
>>> - How can we test this new feature? The only way I can think of
>>> right now is to compile 2 file with maximumNumberOfCompileFork set to 1
>>> and
>>> a second time with the option set to 2 and compare the execution time.
>>> This
>>> heavily depend on timing which will probably be non deterministic.
>>>
>>> No, timing-sensitive tests are not the way to go. We could certainly
>> unit test the AsyncExecAction stuff, and also integration test with a few
>> 'sleep 1 second' actions and ensure that they don't take much more than 1
>> second in total. I guess we could integration test parallel native
>> compilation by using a 'sleep exe' in place of the actual compiler... I'll
>> think about this a little more.
>>
>> Thanks for pushing to get this done. You're right that we really need
>> parallel compilation to be a serious C++ build tool.
>>
>> Daz
>>
>>>
>>>
>>>
>>>
>>> On Sun, Jun 29, 2014 at 11:02 PM, Adam Murdoch <
>>> adam.murd...@gradleware.com> wrote:
>>>
>>>>
>>>> On 30 Jun 2014, at 12:21 pm, Daniel Lacasse <
>>>> daniel.lacass...@gmail.com> wrote:
>>>>
>>>> Thanks Adam for your insight. I agree that option #2 would probably
>>>> be a really good start. Unfortunately, for someone who is pretty new to the
>>>> code base, I would rather start with #1 by exposing n on the compile task,
>>>> set it to one by default and have the logic there to fork compiler tag. I
>>>> would stress that it's a temporary fix and it will be stream line later
>>>> with #2 and #3. It would also be much easier for me to contribute #1 in a
>>>> reasonable time, say for 2.1, and take more time to implement a full
>>>> solution. How does that sound?
>>>>
>>>>
>>>> Sounds good to me.
>>>>
>>>>
>>>> If you strongly prefer #2 as a start point, could you point me to a
>>>> couple place where changes should be applied so I start looking into it?
>>>>
>>>>
>>>> On Sun, Jun 29, 2014 at 9:25 PM, Adam Murdoch <
>>>> adam.murd...@gradleware.com> wrote:
>>>>
>>>>>
>>>>> On 30 Jun 2014, at 1:51 am, Daniel Lacasse <
>>>>> daniel.lacass...@gmail.com> wrote:
>>>>>
>>>>> I have been using the native extension for Gradle inside an actual
>>>>> project for a couple months. The biggest limitation I'm seeing is the
>>>>> inflexibility of the compile task when compared to other build tool in the
>>>>> industry. The --parallel switch works great for concurrent execution of
>>>>> tasks. When it comes to the compile task, all files are compiled one after
>>>>> the other. The native compiler are quite slow especially when it comes to
>>>>> compiling C++ templates. As a comparison, my dev box is mostly idle while
>>>>> compiling with Gradle as oppose to a fairly important load when compiling
>>>>> Visual Studio.
>>>>>
>>>>> The main use case of this feature is the speed up of the compilation
>>>>> process for the native extension. To highlight how this feature is
>>>>> important, I will point out that some company where I previously worked at
>>>>> use system like Incredibuild to perform parallel distributed compilation.
>>>>> Even with such system, the compilation was still pretty time consuming. In
>>>>> it's present form, Gradle is not suitable in term of speed for those
>>>>> scenario. I talked to a couple Gradleware engineers during Gradle Summit
>>>>> 2014 and some insane features are planed to address this problem.
>>>>> Unfortunately, a quicker solution is needed in order to speed up the
>>>>> adoption of Gradle as a native build tool. I also want this feature to be
>>>>> in accordance to the long term Gradle road map.
>>>>>
>>>>> I would like to start the discussion for contributing this into
>>>>> Gradle.
>>>>>
>>>>>
>>>>> That would be great.
>>>>>
>>>>> From my limiting knowledge of Gradle here are a couple open issues
>>>>> I have. I hope some brilliant minds from Gradleware can share there wisdom
>>>>> on where to move forward with this feature.
>>>>>
>>>>> Open issues
>>>>> - What is the current road map for such feature.
>>>>>
>>>>>
>>>>> I think there are 3 potential steps we could take:
>>>>>
>>>>> 1. The compilation tasks do something specific, where they fork n
>>>>> concurrent compilations, and n is just some setting on the compilation
>>>>> tasks.
>>>>>
>>>>> 2. Then, we introduce some general service that tasks can use for
>>>>> coordinating concurrent work. This would be integrated with
>>>>> —parallel-threads. The compilation tasks, the test tasks, and the task
>>>>> executor would all use this service to ensure that an appropriate amount
>>>>> of
>>>>> concurrent work happens at any given time. This would be a public service
>>>>> that any task implementation could use.
>>>>>
>>>>> 3. Then later, we add more capabilities to this so that the work can
>>>>> treated more like tasks - with their own up-to-date checks, dependencies,
>>>>> and so on.
>>>>>
>>>>> We could start with #1 and later extract #2, or we could jump
>>>>> straight to #2. In some ways, it might be nice to start with #2.
>>>>>
>>>>>
>>>>> - This could also be used in any language which require
>>>>> compilation such as Java.
>>>>> - Allowing custom implementation of this feature could allow a
>>>>> company to plug Gradle in there current Incredibuild infrastructure or any
>>>>> other distributed framework.
>>>>> - How this feature fits with the --parallel flag?
>>>>>
>>>>>
>>>>> For every build, there should be two settings that you can tweak:
>>>>>
>>>>> 1. The maximum amount of parallel work that can be performed by
>>>>> Gradle.
>>>>> 2. Whether or not tasks should be executed in parallel.
>>>>>
>>>>> That is, Gradle will be able to do stuff in parallel even if the
>>>>> tasks aren’t executed concurrently. It already does this with test
>>>>> execution.
>>>>>
>>>>> I would change —parallel-threads to control the maximum work but to
>>>>> not enable parallel task execution. It would default to some ‘reasonable’
>>>>> value - the number of cores, say, at least to start with.
>>>>>
>>>>>
>>>>> - How the number of parallel compilation unit will be configure
>>>>> aka number of files that can be compiled in parallel?
>>>>>
>>>>>
>>>>> As above.
>>>>>
>>>>> - Should this feature be always on by default or have a toggle
>>>>> flag?
>>>>>
>>>>>
>>>>> Always on, I think. Why would you turn it off?
>>>>>
>>>>>
>>>>> --
>>>>> Adam Murdoch
>>>>> Gradle Co-founder
>>>>> http://www.gradle.org
>>>>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>>>>> http://www.gradleware.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Daniel
>>>>
>>>>
>>>>
>>>> --
>>>> Adam Murdoch
>>>> Gradle Co-founder
>>>> http://www.gradle.org
>>>> CTO Gradleware Inc. - Gradle Training, Support, Consulting
>>>> http://www.gradleware.com
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Daniel
>>>
>>
>>
>>
>> --
>> Darrell (Daz) DeBoer
>> Principal Software Engineer, *Gradleware <http://gradleware.com>*
>>
>> Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA:
>> http://www.gradlesummit.com
>>
>>
>
>
> --
> Darrell (Daz) DeBoer
> Principal Software Engineer, *Gradleware <http://gradleware.com>*
>
> Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA:
> http://www.gradlesummit.com
>
>
--
Darrell (Daz) DeBoer
Principal Software Engineer, *Gradleware <http://gradleware.com>*
Join us for Gradle Summit 2014, June 12th and 13th in Santa Clara, CA:
http://www.gradlesummit.com
