I'd love to see it write a .dot file of this info as well with good labels, colors, and weights to allow visualizing the graph.
--- Jake Wharton http://about.me/jakewharton On Fri, Jan 17, 2014 at 8:35 AM, Xavier Ducrohet <[email protected]> wrote: > androidDep is kinda crapy (I did it!), and it's kind of obsolete now that > we have composite Configuration object per-variant anyway. It also doesn't > show non Android dependencies I think. > > What's really missing is a global dependency graph output that shows: > - inter-project dependencies > - dependencies of each project (it could combine the different > Configuration, it wouldn't really matter) + local jars (which is missing > from the "dependencies" task). > - all of this in a single graph. > > > On Fri, Jan 17, 2014 at 7:15 AM, Siva Velusamy <[email protected]> wrote: > >> You can try "gradle androidDep" to see the dependency tree. >> On Jan 17, 2014 7:05 AM, "Ashton Cummings" <[email protected]> wrote: >> >>> It is a dependency spaghetti. =P >>> >>> It is a fairly deep dependency graph. With lots of libraries depending >>> on other libraries. >>> >>> I am going to put a tool together real quick to get you the dependency >>> tree. I will just rename all the projects and libraries. I will try to get >>> it to you today some time. >>> >>> Thanks for your continued interest and assistance, it is greatly >>> appreciated. >>> >>> On Thursday, January 16, 2014 3:55:10 PM UTC-6, Xavier Ducrohet wrote: >>>> >>>> (pre)dexing is a known issue. I'm not sure how much faster it's going >>>> to be in 0.8 with parallel dexing since you have a lot of projects and you >>>> can use parallelization there already. >>>> >>>> 18secs for a manifest merge seems like a crazy amount of time. do you >>>> have a lot of manifest data to merge? >>>> >>>> I'm going to build a fake multi-project setup with 100s of project to >>>> look at scaling issues. Can you tell me a bit more about your inter-project >>>> dependencies? Basically do you have 1 apps and 139 flat libs that don't >>>> depend on each other, or do you have a fairly deep dependency graph with >>>> lots of transitive dependencies? >>>> >>>> >>>> On Thu, Jan 16, 2014 at 12:09 PM, Ashton Cummings >>>> <[email protected]>wrote: >>>> >>>>> *"clean build" with no changes = 3h1m53.88s* >>>>> >>>>> Top 15 tasks (roughly the same for each project) >>>>> :projectA:preDexRelease 1m26.29s >>>>> :projectA:preDexDebug 1m25.14s >>>>> :projectA:processDebugManifest 18.452s >>>>> :projectA:processReleaseManifest 17.705s >>>>> :projectA:lint 10.699s >>>>> :projectA:mergeReleaseResources 8.264s >>>>> :projectA:dexDebug 7.742s >>>>> :projectA:dexRelease 7.449s >>>>> :projectA:mergeDebugResources 4.303s >>>>> :projectA:compileDebugJava 1.672s >>>>> :projectA:processReleaseResources 1.662s >>>>> :projectA:processDebugResources 1.325s >>>>> :projectA:compileReleaseJava 1.219s >>>>> :projectA:packageDebug 1.179s >>>>> :projectA:packageRelease 0.923s >>>>> >>>>> >>>>> *clean assembleDebug with pre-dex turned off = 58m58.29s* >>>>> >>>>> Top 15 tasks (roughly the same for each project) >>>>> :projectA:dexDebug 56.567s >>>>> :projectA:mergeDebugResources 5.135s >>>>> :projectA:processDebugManifest 3.517s >>>>> :projectA:compileDebugJava 1.579s >>>>> :projectA:packageDebug 1.174s >>>>> :projectA:processDebugResources 0.972s >>>>> :projectA:processDebugJavaRes 0.373s UP-TO-DATE :projectA: >>>>> prepareLibrary1UnspecifiedLibrary 0.340s >>>>> :projectA:prepareLibrary2UnspecifiedLibrary 0.218s >>>>> :projectA:compileDebugAidl 0.057s >>>>> :projectA:compileDebugRenderscript 0.046s >>>>> :projectA:prepareLibrary3UnspecifiedLibrary 0.045s >>>>> :projectA:prepareLibrary4UnspecifiedLibrary 0.043s >>>>> :projectA:prepareLibrary5UnspecifiedLibrary 0.029s >>>>> :projectA:prepareLibrary6UnspecifiedLibrary 0.028s >>>>> >>>>> I can not find my data for the parallel run. I will do another one of >>>>> those runs and post the data. Let me know if this is helpful for you. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> On Thursday, January 16, 2014 1:46:45 PM UTC-6, Ashton Cummings wrote: >>>>>> >>>>>> I have been using --profile to figure out the bottlenecks. However, >>>>>> due to the nature of our project, i can not share that information. =/ >>>>>> >>>>>> I can give you times and stuff, just not project names and details. >>>>>> So, unless i modify the data substantially, i can not share the profile >>>>>> info. Sorry. >>>>>> >>>>>> However, i will give you the build time improvements and where the >>>>>> bottlenecks are. If you need any specifics, let me know. I will try and >>>>>> get >>>>>> you details that i can. >>>>>> >>>>>> I will get the info to you as soon as i can. I have so much profile >>>>>> data now, i am going to have to run through it and see if i can find the >>>>>> correct data. Might just do the runs again to make sure to get you the >>>>>> correct data. >>>>>> >>>>>> On Thursday, January 16, 2014 12:21:46 PM UTC-6, Xavier Ducrohet >>>>>> wrote: >>>>>>> >>>>>>> Do you remember the time gain you had when disabling pre-dexing? >>>>>>> >>>>>>> If you wouldn't mind. I'd love to get the output of running gradle >>>>>>> with --profile. It will create a report in the build folder of the main >>>>>>> project which will show me build time for all projects and all tasks in >>>>>>> all >>>>>>> projects. >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 16, 2014 at 10:06 AM, Ashton Cummings < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I hope that we can. Our alternatives that i am testing are Maven >>>>>>>> and ANT. Right now our maven approach is showing times that are >>>>>>>> comparable >>>>>>>> to our ANT. I personally want to stick with gradle, however, the >>>>>>>> decision >>>>>>>> is not up to me. I just give the higher ups my results and >>>>>>>> conclusions. =) >>>>>>>> >>>>>>>> I will keep an eye out for those performance improvements and will >>>>>>>> keep our people informed. >>>>>>>> >>>>>>>> Thank you for your time and assistance. If you figure out anything >>>>>>>> else or have any other suggestions, please continue to contact me. I >>>>>>>> really >>>>>>>> hope to convince them to move to gradle. >>>>>>>> >>>>>>>> >>>>>>>> On Thursday, January 16, 2014 11:13:32 AM UTC-6, Xavier Ducrohet >>>>>>>> wrote: >>>>>>>> >>>>>>>>> BTW, we are still a ways off from 1.0. I totally understand not >>>>>>>>> wanting to switch to Gradle right now (especially given your build >>>>>>>>> times), >>>>>>>>> but if you decide to not switch right now, please don't plan to stick >>>>>>>>> with >>>>>>>>> Ant for 2+ years (or some other random long duration). >>>>>>>>> The Gradle build system will see a lot of changes in the next 6 >>>>>>>>> months. I would recommend revisiting it mid-year at least :) >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jan 16, 2014 at 9:10 AM, Xavier Ducrohet <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> http://tools.android.com/tech-docs/new-build-system contains our >>>>>>>>>> changelog. >>>>>>>>>> >>>>>>>>>> We generally don't include Gradle specific changes in it even >>>>>>>>>> when we start requiring newer versions of Gradle, but performance >>>>>>>>>> fixes >>>>>>>>>> will probably be indicated in it because we know it's a pain point. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jan 16, 2014 at 8:54 AM, Ashton Cummings < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Xavier, >>>>>>>>>>> >>>>>>>>>>> Thanks for the information. We are coming to a point now where >>>>>>>>>>> the decision to switch our build process is coming to an end. We >>>>>>>>>>> would >>>>>>>>>>> really like to move to gradle, hopefully you guys are able to get >>>>>>>>>>> these >>>>>>>>>>> optimizations in soon. Once we make the decision, we are going to >>>>>>>>>>> be stuck >>>>>>>>>>> with it for a while. >>>>>>>>>>> >>>>>>>>>>> Where can i keep up to date with the exact changes that are >>>>>>>>>>> going into each build of the gradle plugin? I need to know when >>>>>>>>>>> these >>>>>>>>>>> changes are in place as soon as possible. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thursday, January 16, 2014 10:48:52 AM UTC-6, Xavier Ducrohet >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Right now each project will pre-dex its dependencies on its >>>>>>>>>>>> own. This means 2 components depending on the same library will >>>>>>>>>>>> both run >>>>>>>>>>>> pre-dex on that library's classes.jar which is silly. We're >>>>>>>>>>>> looking at >>>>>>>>>>>> fixing this. >>>>>>>>>>>> >>>>>>>>>>>> Another thing we are looking at is parallelization at the task >>>>>>>>>>>> level. Right now Gradle only supports it at the projects level. >>>>>>>>>>>> It'll build >>>>>>>>>>>> 2 sub projects in parallel (if there are no dependencies between >>>>>>>>>>>> the two), >>>>>>>>>>>> but in a single project it won't parallelize its task. We are >>>>>>>>>>>> working with >>>>>>>>>>>> the Gradle developers to change this. >>>>>>>>>>>> >>>>>>>>>>>> Of course the tasks themselves can do their work in parallel, >>>>>>>>>>>> and most of our tasks do already. Pre-dexing doesn't though, so if >>>>>>>>>>>> a >>>>>>>>>>>> project has more than one dependencies it'll run pre-dex one by >>>>>>>>>>>> one. This >>>>>>>>>>>> is fixed in the next Gradle version. >>>>>>>>>>>> >>>>>>>>>>>> We do take performance very seriously, but we also wanted to >>>>>>>>>>>> provide a lot of features and flexibility in our build. This means >>>>>>>>>>>> we do >>>>>>>>>>>> some extra work. For instance our resource merger is extra work >>>>>>>>>>>> that aapt >>>>>>>>>>>> does in a way but differently (more efficiently in some ways, less >>>>>>>>>>>> in >>>>>>>>>>>> others, but with much less flexibility). We are working to do more >>>>>>>>>>>> to >>>>>>>>>>>> optimize each task and, as I mentioned above, provide generic >>>>>>>>>>>> performance >>>>>>>>>>>> improvements as well. >>>>>>>>>>>> >>>>>>>>>>>> I'm a bit surprised your build went from 1:20 to only 0:58 when >>>>>>>>>>>> running assembleDebug vs build. 'build' does both debug and >>>>>>>>>>>> release but >>>>>>>>>>>> also runs lint. It should be a lot more work than just >>>>>>>>>>>> assembleDebug. Also, >>>>>>>>>>>> do you have flavors? >>>>>>>>>>>> >>>>>>>>>>>> One other thing to consider is incremental build. Pre-dexing >>>>>>>>>>>> for instance makes clean build clearly slower but incremental >>>>>>>>>>>> builds >>>>>>>>>>>> faster. The rest of the build is also very good at only doing what >>>>>>>>>>>> needs to >>>>>>>>>>>> be done and nothing extra, which Ant wasn't very good at. >>>>>>>>>>>> >>>>>>>>>>>> Disabling pre-dexing on all projects without touching each >>>>>>>>>>>> project's build.gradle doesn't seem to be easy. The issue is that >>>>>>>>>>>> doing (in >>>>>>>>>>>> the main build.gradle): >>>>>>>>>>>> >>>>>>>>>>>> subprojects { >>>>>>>>>>>> /// some config >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> this gets applied before all the other projects' own >>>>>>>>>>>> build.gradle are applied. So unless you can apply the 'android' or >>>>>>>>>>>> 'android-library' in there, you won't be able to use our DSL. I >>>>>>>>>>>> need to >>>>>>>>>>>> look into this. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jan 16, 2014 at 6:51 AM, Ashton Cummings < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> My Stack post <http://stackoverflow.com/q/21125302/427763> >>>>>>>>>>>>> >>>>>>>>>>>>> Using gradle 1.9-rc-3 >>>>>>>>>>>>> When building with gradle on a multi-project setup containing >>>>>>>>>>>>> roughly 140 projects/libraries, the build time took 1 hour and 22 >>>>>>>>>>>>> minutes. >>>>>>>>>>>>> And i was using "--parallel". And our ANT build takes less >>>>>>>>>>>>> than 20 minutes without parallel building. >>>>>>>>>>>>> >>>>>>>>>>>>> Here is what i was initially doing: >>>>>>>>>>>>> >>>>>>>>>>>>> ./gradlew clean >>>>>>>>>>>>>> ./gradlew build --parallel >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> This took 1 hour and 22 minutes, roughly. I did a little >>>>>>>>>>>>> testing it seems like the dexing is taking the longest amount of >>>>>>>>>>>>> time. Is >>>>>>>>>>>>> there a way to get the gradle process to re-use the stuff it has >>>>>>>>>>>>> already >>>>>>>>>>>>> dexed? If the libraries have already been built, it should re-use >>>>>>>>>>>>> the >>>>>>>>>>>>> already dexed libraries. >>>>>>>>>>>>> >>>>>>>>>>>>> I saw the option --no-rebuild, but when i run with that >>>>>>>>>>>>> option it says the following: >>>>>>>>>>>>> >>>>>>>>>>>>>> File '/path/to/project/build/libs/project.aar' specified for >>>>>>>>>>>>>> property 'bundle' does not exist. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I replaced the path and the project name with generic stuff. >>>>>>>>>>>>> >>>>>>>>>>>>> *** >>>>>>>>>>>>> >>>>>>>>>>>>> After some assistance i added "preDexLibraries = false" to >>>>>>>>>>>>> all of my "build.gradle" files. However, i still would like to >>>>>>>>>>>>> know a >>>>>>>>>>>>> centralize place that i can put that entry and it affect all the >>>>>>>>>>>>> other >>>>>>>>>>>>> "build.gradle" files. Because having to add that to all of the >>>>>>>>>>>>> "build.gradle" files is annoying and if i ever have to remove it, >>>>>>>>>>>>> it will >>>>>>>>>>>>> be more annoying. This did not reduce the time to what we are >>>>>>>>>>>>> needing. >>>>>>>>>>>>> >>>>>>>>>>>>> After some more assistance, i ran "assembleDebug" instead of >>>>>>>>>>>>> "build". This way it will only do debug builds instead of both >>>>>>>>>>>>> release and >>>>>>>>>>>>> debug. This still did not bring us down to the time we are >>>>>>>>>>>>> getting with >>>>>>>>>>>>> ANT. With all these changes we are only down to 58 minutes. And >>>>>>>>>>>>> our ANT >>>>>>>>>>>>> process is less than 20 minutes. Our ANT process is what android >>>>>>>>>>>>> provides >>>>>>>>>>>>> only modified to support large multi-project setups and to re-use >>>>>>>>>>>>> dexed >>>>>>>>>>>>> libraries. The dex process in the gradle process is taking 1+ >>>>>>>>>>>>> minute for >>>>>>>>>>>>> each project. It is by far the largest time consumer. >>>>>>>>>>>>> >>>>>>>>>>>>> *** >>>>>>>>>>>>> >>>>>>>>>>>>> So, how can we speed the process up more? Is gradle re-using >>>>>>>>>>>>> the dexed libraries after they have been dexed? What are your >>>>>>>>>>>>> suggestions? >>>>>>>>>>>>> For us to move over to gradle, it has to be able to compete with >>>>>>>>>>>>> the ANT >>>>>>>>>>>>> process. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "adt-dev" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> >>>>>>>>>>>>> For more options, visit https://groups.google.com/grou >>>>>>>>>>>>> ps/opt_out. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Xavier Ducrohet >>>>>>>>>>>> Android SDK Tech Lead >>>>>>>>>>>> Google Inc. >>>>>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>>>>> >>>>>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "adt-dev" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Xavier Ducrohet >>>>>>>>>> Android SDK Tech Lead >>>>>>>>>> Google Inc. >>>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>>> >>>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Xavier Ducrohet >>>>>>>>> Android SDK Tech Lead >>>>>>>>> Google Inc. >>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>> >>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "adt-dev" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Xavier Ducrohet >>>>>>> Android SDK Tech Lead >>>>>>> Google Inc. >>>>>>> http://developer.android.com | http://tools.android.com >>>>>>> >>>>>>> Please do not send me questions directly. Thanks! >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "adt-dev" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>>> >>>> -- >>>> Xavier Ducrohet >>>> Android SDK Tech Lead >>>> Google Inc. >>>> http://developer.android.com | http://tools.android.com >>>> >>>> Please do not send me questions directly. Thanks! >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "adt-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "adt-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Xavier Ducrohet > Android SDK Tech Lead > Google Inc. > http://developer.android.com | http://tools.android.com > > Please do not send me questions directly. Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "adt-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "adt-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
