indeed.
On Fri, Jan 17, 2014 at 10:41 AM, Jake Wharton <jakewhar...@gmail.com>wrote: > I'd love to see it write a .dot file of this info as well with good > labels, colors, and weights to allow visualizing the graph. > > > --- > Jake Wharton > http://about.me/jakewharton > > > On Fri, Jan 17, 2014 at 8:35 AM, Xavier Ducrohet <x...@android.com> wrote: > >> androidDep is kinda crapy (I did it!), and it's kind of obsolete now that >> we have composite Configuration object per-variant anyway. It also doesn't >> show non Android dependencies I think. >> >> What's really missing is a global dependency graph output that shows: >> - inter-project dependencies >> - dependencies of each project (it could combine the different >> Configuration, it wouldn't really matter) + local jars (which is missing >> from the "dependencies" task). >> - all of this in a single graph. >> >> >> On Fri, Jan 17, 2014 at 7:15 AM, Siva Velusamy <vs...@google.com> wrote: >> >>> You can try "gradle androidDep" to see the dependency tree. >>> On Jan 17, 2014 7:05 AM, "Ashton Cummings" <prolink...@gmail.com> wrote: >>> >>>> It is a dependency spaghetti. =P >>>> >>>> It is a fairly deep dependency graph. With lots of libraries depending >>>> on other libraries. >>>> >>>> I am going to put a tool together real quick to get you the dependency >>>> tree. I will just rename all the projects and libraries. I will try to get >>>> it to you today some time. >>>> >>>> Thanks for your continued interest and assistance, it is greatly >>>> appreciated. >>>> >>>> On Thursday, January 16, 2014 3:55:10 PM UTC-6, Xavier Ducrohet wrote: >>>>> >>>>> (pre)dexing is a known issue. I'm not sure how much faster it's going >>>>> to be in 0.8 with parallel dexing since you have a lot of projects and you >>>>> can use parallelization there already. >>>>> >>>>> 18secs for a manifest merge seems like a crazy amount of time. do you >>>>> have a lot of manifest data to merge? >>>>> >>>>> I'm going to build a fake multi-project setup with 100s of project to >>>>> look at scaling issues. Can you tell me a bit more about your >>>>> inter-project >>>>> dependencies? Basically do you have 1 apps and 139 flat libs that don't >>>>> depend on each other, or do you have a fairly deep dependency graph with >>>>> lots of transitive dependencies? >>>>> >>>>> >>>>> On Thu, Jan 16, 2014 at 12:09 PM, Ashton Cummings >>>>> <proli...@gmail.com>wrote: >>>>> >>>>>> *"clean build" with no changes = 3h1m53.88s* >>>>>> >>>>>> Top 15 tasks (roughly the same for each project) >>>>>> :projectA:preDexRelease 1m26.29s >>>>>> :projectA:preDexDebug 1m25.14s >>>>>> :projectA:processDebugManifest 18.452s >>>>>> :projectA:processReleaseManifest 17.705s >>>>>> :projectA:lint 10.699s >>>>>> :projectA:mergeReleaseResources 8.264s >>>>>> :projectA:dexDebug 7.742s >>>>>> :projectA:dexRelease 7.449s >>>>>> :projectA:mergeDebugResources 4.303s >>>>>> :projectA:compileDebugJava 1.672s >>>>>> :projectA:processReleaseResources 1.662s >>>>>> :projectA:processDebugResources 1.325s >>>>>> :projectA:compileReleaseJava 1.219s >>>>>> :projectA:packageDebug 1.179s >>>>>> :projectA:packageRelease 0.923s >>>>>> >>>>>> >>>>>> *clean assembleDebug with pre-dex turned off = 58m58.29s* >>>>>> >>>>>> Top 15 tasks (roughly the same for each project) >>>>>> :projectA:dexDebug 56.567s >>>>>> :projectA:mergeDebugResources 5.135s >>>>>> :projectA:processDebugManifest 3.517s >>>>>> :projectA:compileDebugJava 1.579s >>>>>> :projectA:packageDebug 1.174s >>>>>> :projectA:processDebugResources 0.972s >>>>>> :projectA:processDebugJavaRes 0.373s UP-TO-DATE :projectA: >>>>>> prepareLibrary1UnspecifiedLibrary 0.340s >>>>>> :projectA:prepareLibrary2UnspecifiedLibrary 0.218s >>>>>> :projectA:compileDebugAidl 0.057s >>>>>> :projectA:compileDebugRenderscript 0.046s >>>>>> :projectA:prepareLibrary3UnspecifiedLibrary 0.045s >>>>>> :projectA:prepareLibrary4UnspecifiedLibrary 0.043s >>>>>> :projectA:prepareLibrary5UnspecifiedLibrary 0.029s >>>>>> :projectA:prepareLibrary6UnspecifiedLibrary 0.028s >>>>>> >>>>>> I can not find my data for the parallel run. I will do another one of >>>>>> those runs and post the data. Let me know if this is helpful for you. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> On Thursday, January 16, 2014 1:46:45 PM UTC-6, Ashton Cummings wrote: >>>>>>> >>>>>>> I have been using --profile to figure out the bottlenecks. However, >>>>>>> due to the nature of our project, i can not share that information. =/ >>>>>>> >>>>>>> I can give you times and stuff, just not project names and details. >>>>>>> So, unless i modify the data substantially, i can not share the profile >>>>>>> info. Sorry. >>>>>>> >>>>>>> However, i will give you the build time improvements and where the >>>>>>> bottlenecks are. If you need any specifics, let me know. I will try and >>>>>>> get >>>>>>> you details that i can. >>>>>>> >>>>>>> I will get the info to you as soon as i can. I have so much profile >>>>>>> data now, i am going to have to run through it and see if i can find the >>>>>>> correct data. Might just do the runs again to make sure to get you the >>>>>>> correct data. >>>>>>> >>>>>>> On Thursday, January 16, 2014 12:21:46 PM UTC-6, Xavier Ducrohet >>>>>>> wrote: >>>>>>>> >>>>>>>> Do you remember the time gain you had when disabling pre-dexing? >>>>>>>> >>>>>>>> If you wouldn't mind. I'd love to get the output of running gradle >>>>>>>> with --profile. It will create a report in the build folder of the main >>>>>>>> project which will show me build time for all projects and all tasks >>>>>>>> in all >>>>>>>> projects. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 16, 2014 at 10:06 AM, Ashton Cummings < >>>>>>>> proli...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I hope that we can. Our alternatives that i am testing are Maven >>>>>>>>> and ANT. Right now our maven approach is showing times that are >>>>>>>>> comparable >>>>>>>>> to our ANT. I personally want to stick with gradle, however, the >>>>>>>>> decision >>>>>>>>> is not up to me. I just give the higher ups my results and >>>>>>>>> conclusions. =) >>>>>>>>> >>>>>>>>> I will keep an eye out for those performance improvements and will >>>>>>>>> keep our people informed. >>>>>>>>> >>>>>>>>> Thank you for your time and assistance. If you figure out anything >>>>>>>>> else or have any other suggestions, please continue to contact me. I >>>>>>>>> really >>>>>>>>> hope to convince them to move to gradle. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thursday, January 16, 2014 11:13:32 AM UTC-6, Xavier Ducrohet >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> BTW, we are still a ways off from 1.0. I totally understand not >>>>>>>>>> wanting to switch to Gradle right now (especially given your build >>>>>>>>>> times), >>>>>>>>>> but if you decide to not switch right now, please don't plan to >>>>>>>>>> stick with >>>>>>>>>> Ant for 2+ years (or some other random long duration). >>>>>>>>>> The Gradle build system will see a lot of changes in the next 6 >>>>>>>>>> months. I would recommend revisiting it mid-year at least :) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jan 16, 2014 at 9:10 AM, Xavier Ducrohet < >>>>>>>>>> x...@android.com> wrote: >>>>>>>>>> >>>>>>>>>>> http://tools.android.com/tech-docs/new-build-system contains >>>>>>>>>>> our changelog. >>>>>>>>>>> >>>>>>>>>>> We generally don't include Gradle specific changes in it even >>>>>>>>>>> when we start requiring newer versions of Gradle, but performance >>>>>>>>>>> fixes >>>>>>>>>>> will probably be indicated in it because we know it's a pain point. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jan 16, 2014 at 8:54 AM, Ashton Cummings < >>>>>>>>>>> proli...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Xavier, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the information. We are coming to a point now where >>>>>>>>>>>> the decision to switch our build process is coming to an end. We >>>>>>>>>>>> would >>>>>>>>>>>> really like to move to gradle, hopefully you guys are able to get >>>>>>>>>>>> these >>>>>>>>>>>> optimizations in soon. Once we make the decision, we are going to >>>>>>>>>>>> be stuck >>>>>>>>>>>> with it for a while. >>>>>>>>>>>> >>>>>>>>>>>> Where can i keep up to date with the exact changes that are >>>>>>>>>>>> going into each build of the gradle plugin? I need to know when >>>>>>>>>>>> these >>>>>>>>>>>> changes are in place as soon as possible. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thursday, January 16, 2014 10:48:52 AM UTC-6, Xavier >>>>>>>>>>>> Ducrohet wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Right now each project will pre-dex its dependencies on its >>>>>>>>>>>>> own. This means 2 components depending on the same library will >>>>>>>>>>>>> both run >>>>>>>>>>>>> pre-dex on that library's classes.jar which is silly. We're >>>>>>>>>>>>> looking at >>>>>>>>>>>>> fixing this. >>>>>>>>>>>>> >>>>>>>>>>>>> Another thing we are looking at is parallelization at the task >>>>>>>>>>>>> level. Right now Gradle only supports it at the projects level. >>>>>>>>>>>>> It'll build >>>>>>>>>>>>> 2 sub projects in parallel (if there are no dependencies between >>>>>>>>>>>>> the two), >>>>>>>>>>>>> but in a single project it won't parallelize its task. We are >>>>>>>>>>>>> working with >>>>>>>>>>>>> the Gradle developers to change this. >>>>>>>>>>>>> >>>>>>>>>>>>> Of course the tasks themselves can do their work in parallel, >>>>>>>>>>>>> and most of our tasks do already. Pre-dexing doesn't though, so >>>>>>>>>>>>> if a >>>>>>>>>>>>> project has more than one dependencies it'll run pre-dex one by >>>>>>>>>>>>> one. This >>>>>>>>>>>>> is fixed in the next Gradle version. >>>>>>>>>>>>> >>>>>>>>>>>>> We do take performance very seriously, but we also wanted to >>>>>>>>>>>>> provide a lot of features and flexibility in our build. This >>>>>>>>>>>>> means we do >>>>>>>>>>>>> some extra work. For instance our resource merger is extra work >>>>>>>>>>>>> that aapt >>>>>>>>>>>>> does in a way but differently (more efficiently in some ways, >>>>>>>>>>>>> less in >>>>>>>>>>>>> others, but with much less flexibility). We are working to do >>>>>>>>>>>>> more to >>>>>>>>>>>>> optimize each task and, as I mentioned above, provide generic >>>>>>>>>>>>> performance >>>>>>>>>>>>> improvements as well. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm a bit surprised your build went from 1:20 to only 0:58 >>>>>>>>>>>>> when running assembleDebug vs build. 'build' does both debug and >>>>>>>>>>>>> release >>>>>>>>>>>>> but also runs lint. It should be a lot more work than just >>>>>>>>>>>>> assembleDebug. >>>>>>>>>>>>> Also, do you have flavors? >>>>>>>>>>>>> >>>>>>>>>>>>> One other thing to consider is incremental build. Pre-dexing >>>>>>>>>>>>> for instance makes clean build clearly slower but incremental >>>>>>>>>>>>> builds >>>>>>>>>>>>> faster. The rest of the build is also very good at only doing >>>>>>>>>>>>> what needs to >>>>>>>>>>>>> be done and nothing extra, which Ant wasn't very good at. >>>>>>>>>>>>> >>>>>>>>>>>>> Disabling pre-dexing on all projects without touching each >>>>>>>>>>>>> project's build.gradle doesn't seem to be easy. The issue is that >>>>>>>>>>>>> doing (in >>>>>>>>>>>>> the main build.gradle): >>>>>>>>>>>>> >>>>>>>>>>>>> subprojects { >>>>>>>>>>>>> /// some config >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> this gets applied before all the other projects' own >>>>>>>>>>>>> build.gradle are applied. So unless you can apply the 'android' or >>>>>>>>>>>>> 'android-library' in there, you won't be able to use our DSL. I >>>>>>>>>>>>> need to >>>>>>>>>>>>> look into this. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jan 16, 2014 at 6:51 AM, Ashton Cummings < >>>>>>>>>>>>> proli...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> My Stack post <http://stackoverflow.com/q/21125302/427763> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using gradle 1.9-rc-3 >>>>>>>>>>>>>> When building with gradle on a multi-project setup containing >>>>>>>>>>>>>> roughly 140 projects/libraries, the build time took 1 hour and >>>>>>>>>>>>>> 22 minutes. >>>>>>>>>>>>>> And i was using "--parallel". And our ANT build takes less >>>>>>>>>>>>>> than 20 minutes without parallel building. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is what i was initially doing: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ./gradlew clean >>>>>>>>>>>>>>> ./gradlew build --parallel >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> This took 1 hour and 22 minutes, roughly. I did a little >>>>>>>>>>>>>> testing it seems like the dexing is taking the longest amount of >>>>>>>>>>>>>> time. Is >>>>>>>>>>>>>> there a way to get the gradle process to re-use the stuff it has >>>>>>>>>>>>>> already >>>>>>>>>>>>>> dexed? If the libraries have already been built, it should >>>>>>>>>>>>>> re-use the >>>>>>>>>>>>>> already dexed libraries. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I saw the option --no-rebuild, but when i run with that >>>>>>>>>>>>>> option it says the following: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> File '/path/to/project/build/libs/project.aar' specified for >>>>>>>>>>>>>>> property 'bundle' does not exist. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I replaced the path and the project name with generic stuff. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *** >>>>>>>>>>>>>> >>>>>>>>>>>>>> After some assistance i added "preDexLibraries = false" to >>>>>>>>>>>>>> all of my "build.gradle" files. However, i still would like to >>>>>>>>>>>>>> know a >>>>>>>>>>>>>> centralize place that i can put that entry and it affect all the >>>>>>>>>>>>>> other >>>>>>>>>>>>>> "build.gradle" files. Because having to add that to all of the >>>>>>>>>>>>>> "build.gradle" files is annoying and if i ever have to remove >>>>>>>>>>>>>> it, it will >>>>>>>>>>>>>> be more annoying. This did not reduce the time to what we are >>>>>>>>>>>>>> needing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> After some more assistance, i ran "assembleDebug" instead of >>>>>>>>>>>>>> "build". This way it will only do debug builds instead of both >>>>>>>>>>>>>> release and >>>>>>>>>>>>>> debug. This still did not bring us down to the time we are >>>>>>>>>>>>>> getting with >>>>>>>>>>>>>> ANT. With all these changes we are only down to 58 minutes. And >>>>>>>>>>>>>> our ANT >>>>>>>>>>>>>> process is less than 20 minutes. Our ANT process is what android >>>>>>>>>>>>>> provides >>>>>>>>>>>>>> only modified to support large multi-project setups and to >>>>>>>>>>>>>> re-use dexed >>>>>>>>>>>>>> libraries. The dex process in the gradle process is taking 1+ >>>>>>>>>>>>>> minute for >>>>>>>>>>>>>> each project. It is by far the largest time consumer. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *** >>>>>>>>>>>>>> >>>>>>>>>>>>>> So, how can we speed the process up more? Is gradle re-using >>>>>>>>>>>>>> the dexed libraries after they have been dexed? What are your >>>>>>>>>>>>>> suggestions? >>>>>>>>>>>>>> For us to move over to gradle, it has to be able to compete with >>>>>>>>>>>>>> the ANT >>>>>>>>>>>>>> process. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "adt-dev" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to adt-dev+u...@googlegroups.com. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For more options, visit https://groups.google.com/grou >>>>>>>>>>>>>> ps/opt_out. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Xavier Ducrohet >>>>>>>>>>>>> Android SDK Tech Lead >>>>>>>>>>>>> Google Inc. >>>>>>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>>>>>> >>>>>>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "adt-dev" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to adt-dev+u...@googlegroups.com. >>>>>>>>>>>> For more options, visit https://groups.google.com/grou >>>>>>>>>>>> ps/opt_out. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Xavier Ducrohet >>>>>>>>>>> Android SDK Tech Lead >>>>>>>>>>> Google Inc. >>>>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>>>> >>>>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Xavier Ducrohet >>>>>>>>>> Android SDK Tech Lead >>>>>>>>>> Google Inc. >>>>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>>>> >>>>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "adt-dev" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to adt-dev+u...@googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Xavier Ducrohet >>>>>>>> Android SDK Tech Lead >>>>>>>> Google Inc. >>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>> >>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "adt-dev" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to adt-dev+u...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Xavier Ducrohet >>>>> Android SDK Tech Lead >>>>> Google Inc. >>>>> http://developer.android.com | http://tools.android.com >>>>> >>>>> Please do not send me questions directly. Thanks! >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "adt-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to adt-dev+unsubscr...@googlegroups.com. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "adt-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to adt-dev+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Xavier Ducrohet >> Android SDK Tech Lead >> Google Inc. >> http://developer.android.com | http://tools.android.com >> >> Please do not send me questions directly. Thanks! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "adt-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to adt-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "adt-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to adt-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- Xavier Ducrohet Android SDK Tech Lead Google Inc. http://developer.android.com | http://tools.android.com Please do not send me questions directly. Thanks! -- You received this message because you are subscribed to the Google Groups "adt-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to adt-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
