It is a dependency spaghetti. =P It is a fairly deep dependency graph. With lots of libraries depending on other libraries.
I am going to put a tool together real quick to get you the dependency tree. I will just rename all the projects and libraries. I will try to get it to you today some time. Thanks for your continued interest and assistance, it is greatly appreciated. On Thursday, January 16, 2014 3:55:10 PM UTC-6, Xavier Ducrohet wrote: > > (pre)dexing is a known issue. I'm not sure how much faster it's going to > be in 0.8 with parallel dexing since you have a lot of projects and you can > use parallelization there already. > > 18secs for a manifest merge seems like a crazy amount of time. do you have > a lot of manifest data to merge? > > I'm going to build a fake multi-project setup with 100s of project to look > at scaling issues. Can you tell me a bit more about your inter-project > dependencies? Basically do you have 1 apps and 139 flat libs that don't > depend on each other, or do you have a fairly deep dependency graph with > lots of transitive dependencies? > > > On Thu, Jan 16, 2014 at 12:09 PM, Ashton Cummings > <[email protected]<javascript:> > > wrote: > >> *"clean build" with no changes = 3h1m53.88s* >> >> Top 15 tasks (roughly the same for each project) >> :projectA:preDexRelease 1m26.29s >> :projectA:preDexDebug 1m25.14s >> :projectA:processDebugManifest 18.452s >> :projectA:processReleaseManifest 17.705s >> :projectA:lint 10.699s >> :projectA:mergeReleaseResources 8.264s >> :projectA:dexDebug 7.742s >> :projectA:dexRelease 7.449s >> :projectA:mergeDebugResources 4.303s >> :projectA:compileDebugJava 1.672s >> :projectA:processReleaseResources 1.662s >> :projectA:processDebugResources 1.325s >> :projectA:compileReleaseJava 1.219s >> :projectA:packageDebug 1.179s >> :projectA:packageRelease 0.923s >> >> >> *clean assembleDebug with pre-dex turned off = 58m58.29s* >> >> Top 15 tasks (roughly the same for each project) >> :projectA:dexDebug 56.567s >> :projectA:mergeDebugResources 5.135s >> :projectA:processDebugManifest 3.517s >> :projectA:compileDebugJava 1.579s >> :projectA:packageDebug 1.174s >> :projectA:processDebugResources 0.972s >> :projectA:processDebugJavaRes 0.373s UP-TO-DATE >> :projectA:prepareLibrary1UnspecifiedLibrary 0.340s >> :projectA:prepareLibrary2UnspecifiedLibrary 0.218s >> :projectA:compileDebugAidl 0.057s >> :projectA:compileDebugRenderscript 0.046s >> :projectA:prepareLibrary3UnspecifiedLibrary 0.045s >> :projectA:prepareLibrary4UnspecifiedLibrary 0.043s >> :projectA:prepareLibrary5UnspecifiedLibrary 0.029s >> :projectA:prepareLibrary6UnspecifiedLibrary 0.028s >> >> I can not find my data for the parallel run. I will do another one of >> those runs and post the data. Let me know if this is helpful for you. >> >> Thanks >> >> >> On Thursday, January 16, 2014 1:46:45 PM UTC-6, Ashton Cummings wrote: >>> >>> I have been using --profile to figure out the bottlenecks. However, due >>> to the nature of our project, i can not share that information. =/ >>> >>> I can give you times and stuff, just not project names and details. So, >>> unless i modify the data substantially, i can not share the profile info. >>> Sorry. >>> >>> However, i will give you the build time improvements and where the >>> bottlenecks are. If you need any specifics, let me know. I will try and get >>> you details that i can. >>> >>> I will get the info to you as soon as i can. I have so much profile data >>> now, i am going to have to run through it and see if i can find the correct >>> data. Might just do the runs again to make sure to get you the correct data. >>> >>> On Thursday, January 16, 2014 12:21:46 PM UTC-6, Xavier Ducrohet wrote: >>>> >>>> Do you remember the time gain you had when disabling pre-dexing? >>>> >>>> If you wouldn't mind. I'd love to get the output of running gradle with >>>> --profile. It will create a report in the build folder of the main project >>>> which will show me build time for all projects and all tasks in all >>>> projects. >>>> >>>> >>>> On Thu, Jan 16, 2014 at 10:06 AM, Ashton Cummings >>>> <[email protected]>wrote: >>>> >>>>> I hope that we can. Our alternatives that i am testing are Maven and >>>>> ANT. Right now our maven approach is showing times that are comparable to >>>>> our ANT. I personally want to stick with gradle, however, the decision is >>>>> not up to me. I just give the higher ups my results and conclusions. =) >>>>> >>>>> I will keep an eye out for those performance improvements and will >>>>> keep our people informed. >>>>> >>>>> Thank you for your time and assistance. If you figure out anything >>>>> else or have any other suggestions, please continue to contact me. I >>>>> really >>>>> hope to convince them to move to gradle. >>>>> >>>>> >>>>> On Thursday, January 16, 2014 11:13:32 AM UTC-6, Xavier Ducrohet wrote: >>>>> >>>>>> BTW, we are still a ways off from 1.0. I totally understand not >>>>>> wanting to switch to Gradle right now (especially given your build >>>>>> times), >>>>>> but if you decide to not switch right now, please don't plan to stick >>>>>> with >>>>>> Ant for 2+ years (or some other random long duration). >>>>>> The Gradle build system will see a lot of changes in the next 6 >>>>>> months. I would recommend revisiting it mid-year at least :) >>>>>> >>>>>> >>>>>> On Thu, Jan 16, 2014 at 9:10 AM, Xavier Ducrohet <[email protected]>wrote: >>>>>> >>>>>>> http://tools.android.com/tech-docs/new-build-system contains our >>>>>>> changelog. >>>>>>> >>>>>>> We generally don't include Gradle specific changes in it even when >>>>>>> we start requiring newer versions of Gradle, but performance fixes will >>>>>>> probably be indicated in it because we know it's a pain point. >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 16, 2014 at 8:54 AM, Ashton Cummings <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Xavier, >>>>>>>> >>>>>>>> Thanks for the information. We are coming to a point now where the >>>>>>>> decision to switch our build process is coming to an end. We would >>>>>>>> really >>>>>>>> like to move to gradle, hopefully you guys are able to get these >>>>>>>> optimizations in soon. Once we make the decision, we are going to be >>>>>>>> stuck >>>>>>>> with it for a while. >>>>>>>> >>>>>>>> Where can i keep up to date with the exact changes that are going >>>>>>>> into each build of the gradle plugin? I need to know when these >>>>>>>> changes are >>>>>>>> in place as soon as possible. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> On Thursday, January 16, 2014 10:48:52 AM UTC-6, Xavier Ducrohet >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Right now each project will pre-dex its dependencies on its own. >>>>>>>>> This means 2 components depending on the same library will both run >>>>>>>>> pre-dex >>>>>>>>> on that library's classes.jar which is silly. We're looking at fixing >>>>>>>>> this. >>>>>>>>> >>>>>>>>> Another thing we are looking at is parallelization at the task >>>>>>>>> level. Right now Gradle only supports it at the projects level. It'll >>>>>>>>> build >>>>>>>>> 2 sub projects in parallel (if there are no dependencies between the >>>>>>>>> two), >>>>>>>>> but in a single project it won't parallelize its task. We are working >>>>>>>>> with >>>>>>>>> the Gradle developers to change this. >>>>>>>>> >>>>>>>>> Of course the tasks themselves can do their work in parallel, and >>>>>>>>> most of our tasks do already. Pre-dexing doesn't though, so if a >>>>>>>>> project >>>>>>>>> has more than one dependencies it'll run pre-dex one by one. This is >>>>>>>>> fixed >>>>>>>>> in the next Gradle version. >>>>>>>>> >>>>>>>>> We do take performance very seriously, but we also wanted to >>>>>>>>> provide a lot of features and flexibility in our build. This means we >>>>>>>>> do >>>>>>>>> some extra work. For instance our resource merger is extra work that >>>>>>>>> aapt >>>>>>>>> does in a way but differently (more efficiently in some ways, less in >>>>>>>>> others, but with much less flexibility). We are working to do more to >>>>>>>>> optimize each task and, as I mentioned above, provide generic >>>>>>>>> performance >>>>>>>>> improvements as well. >>>>>>>>> >>>>>>>>> I'm a bit surprised your build went from 1:20 to only 0:58 when >>>>>>>>> running assembleDebug vs build. 'build' does both debug and release >>>>>>>>> but >>>>>>>>> also runs lint. It should be a lot more work than just assembleDebug. >>>>>>>>> Also, >>>>>>>>> do you have flavors? >>>>>>>>> >>>>>>>>> One other thing to consider is incremental build. Pre-dexing for >>>>>>>>> instance makes clean build clearly slower but incremental builds >>>>>>>>> faster. >>>>>>>>> The rest of the build is also very good at only doing what needs to >>>>>>>>> be done >>>>>>>>> and nothing extra, which Ant wasn't very good at. >>>>>>>>> >>>>>>>>> Disabling pre-dexing on all projects without touching each >>>>>>>>> project's build.gradle doesn't seem to be easy. The issue is that >>>>>>>>> doing (in >>>>>>>>> the main build.gradle): >>>>>>>>> >>>>>>>>> subprojects { >>>>>>>>> /// some config >>>>>>>>> } >>>>>>>>> >>>>>>>>> this gets applied before all the other projects' own build.gradle >>>>>>>>> are applied. So unless you can apply the 'android' or >>>>>>>>> 'android-library' in >>>>>>>>> there, you won't be able to use our DSL. I need to look into this. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jan 16, 2014 at 6:51 AM, Ashton Cummings < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> My Stack post <http://stackoverflow.com/q/21125302/427763> >>>>>>>>>> >>>>>>>>>> Using gradle 1.9-rc-3 >>>>>>>>>> When building with gradle on a multi-project setup containing >>>>>>>>>> roughly 140 projects/libraries, the build time took 1 hour and 22 >>>>>>>>>> minutes. >>>>>>>>>> And i was using "--parallel". And our ANT build takes less than >>>>>>>>>> 20 minutes without parallel building. >>>>>>>>>> >>>>>>>>>> Here is what i was initially doing: >>>>>>>>>> >>>>>>>>>> ./gradlew clean >>>>>>>>>>> ./gradlew build --parallel >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This took 1 hour and 22 minutes, roughly. I did a little testing >>>>>>>>>> it seems like the dexing is taking the longest amount of time. Is >>>>>>>>>> there a >>>>>>>>>> way to get the gradle process to re-use the stuff it has already >>>>>>>>>> dexed? If >>>>>>>>>> the libraries have already been built, it should re-use the already >>>>>>>>>> dexed >>>>>>>>>> libraries. >>>>>>>>>> >>>>>>>>>> I saw the option --no-rebuild, but when i run with that option >>>>>>>>>> it says the following: >>>>>>>>>> >>>>>>>>>>> File '/path/to/project/build/libs/project.aar' specified for >>>>>>>>>>> property 'bundle' does not exist. >>>>>>>>>>> >>>>>>>>>>> I replaced the path and the project name with generic stuff. >>>>>>>>>> >>>>>>>>>> *** >>>>>>>>>> >>>>>>>>>> After some assistance i added "preDexLibraries = false" to all >>>>>>>>>> of my "build.gradle" files. However, i still would like to know a >>>>>>>>>> centralize place that i can put that entry and it affect all the >>>>>>>>>> other >>>>>>>>>> "build.gradle" files. Because having to add that to all of the >>>>>>>>>> "build.gradle" files is annoying and if i ever have to remove it, it >>>>>>>>>> will >>>>>>>>>> be more annoying. This did not reduce the time to what we are >>>>>>>>>> needing. >>>>>>>>>> >>>>>>>>>> After some more assistance, i ran "assembleDebug" instead of >>>>>>>>>> "build". This way it will only do debug builds instead of both >>>>>>>>>> release and >>>>>>>>>> debug. This still did not bring us down to the time we are getting >>>>>>>>>> with >>>>>>>>>> ANT. With all these changes we are only down to 58 minutes. And our >>>>>>>>>> ANT >>>>>>>>>> process is less than 20 minutes. Our ANT process is what android >>>>>>>>>> provides >>>>>>>>>> only modified to support large multi-project setups and to re-use >>>>>>>>>> dexed >>>>>>>>>> libraries. The dex process in the gradle process is taking 1+ minute >>>>>>>>>> for >>>>>>>>>> each project. It is by far the largest time consumer. >>>>>>>>>> >>>>>>>>>> *** >>>>>>>>>> >>>>>>>>>> So, how can we speed the process up more? Is gradle re-using the >>>>>>>>>> dexed libraries after they have been dexed? What are your >>>>>>>>>> suggestions? For >>>>>>>>>> us to move over to gradle, it has to be able to compete with the ANT >>>>>>>>>> process. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "adt-dev" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> >>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Xavier Ducrohet >>>>>>>>> Android SDK Tech Lead >>>>>>>>> Google Inc. >>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>> >>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "adt-dev" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Xavier Ducrohet >>>>>>> Android SDK Tech Lead >>>>>>> Google Inc. >>>>>>> http://developer.android.com | http://tools.android.com >>>>>>> >>>>>>> Please do not send me questions directly. Thanks! >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Xavier Ducrohet >>>>>> Android SDK Tech Lead >>>>>> Google Inc. >>>>>> http://developer.android.com | http://tools.android.com >>>>>> >>>>>> Please do not send me questions directly. Thanks! >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "adt-dev" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>>> >>>> -- >>>> Xavier Ducrohet >>>> Android SDK Tech Lead >>>> Google Inc. >>>> http://developer.android.com | http://tools.android.com >>>> >>>> Please do not send me questions directly. Thanks! >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "adt-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Xavier Ducrohet > Android SDK Tech Lead > Google Inc. > http://developer.android.com | http://tools.android.com > > Please do not send me questions directly. Thanks! > -- You received this message because you are subscribed to the Google Groups "adt-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
