OK, I think there is a solution that will make everyone happy, while maintaining compatibility with existing after_prepare hooks: An incremental prepare can copy both files that are newer AND FILES THAT ARE OLDER than the target files in the platform's www directory, while skipping any files which have an identical last-modified time. That way, any after_prepare hook that modifies a prepared file will still see the file refreshed on every prepare, as it does today, while all the unchanged files will be skipped, greatly speeding up the prepare.
With this approach, I think there is no need to put the incremental prepare behind a --incremental option; it can just be the default behavior. (I am still adding prepared files to cordova clean also.) And since this shouldn't break anything, it won't require a major version bump. But I suppose it should be a minor version bump, since it's new functionality. Jason -----Original Message----- From: Jason Ginchereau [mailto:jason...@microsoft.com] Sent: Tuesday, April 26, 2016 9:39 AM To: dev@cordova.apache.org Subject: RE: [DISCUSS] Faster incremental builds I don't understand how adding a file list to the hook parameters will help solve this problem. Any hook that currently fails after an incremental prepare would need to be updated either way. It would be simpler to just update it to correctly handle the case when a file wasn't updated by prepare. We still don't have any real examples of hooks where this is a problem, do we? But it still seems safest to save this change for a major version update, unfortunately. Jason -----Original Message----- From: Carlos Santana [mailto:csantan...@gmail.com] Sent: Friday, April 22, 2016 5:03 PM To: dev@cordova.apache.org Subject: Re: [DISCUSS] Faster incremental builds Why your saying is more complicated? Complicated for hook authors? for the hooks that don't need they don't have to do anything just ignore the argument in context. "cordova prepare --clean" == "cordova clean && cordova prepare" - Carlos @csantanapr > On Apr 22, 2016, at 7:45 PM, Jason Ginchereau <jason...@microsoft.com> wrote: > > I don't think making the after_prepare hook API more complicated is a good > design. Instead can we just document that after_prepare hooks should be > designed to handle the files being copied only sometimes? Most of them > probably are anyway, because Cordova never made an explicit guarantee about > the files being fresh every time the hook is called. Anyway I guess it could > still be considered a breaking change so maybe it should trigger a > major-version bump? > > I don't understand what you are suggesting with "cordova prepare --clean". > > -----Original Message----- > From: Carlos Santana [mailto:csantan...@gmail.com] > Sent: Thursday, April 21, 2016 17:16 > To: dev@cordova.apache.org > Subject: Re: [DISCUSS] Faster incremental builds > > I guess it's fine to have default to be faster, we just need to pass the > files to be copied to feed the hook to be aware. It's kind of a break API on > CLI so maybe CLI 7.x but not sure. > > Also for folks that want the conversation approach with one command we can > also have a flag --clean "cordova prepare --clean" > > > - Carlos > @csantanapr > >> On Apr 21, 2016, at 7:42 PM, Jason Ginchereau <jason...@microsoft.com> wrote: >> >> Thanks Carlos, that's a good point. >> >> The Ionic add_platform_class hook >> (https://github.com/driftyco/ionic-package-hooks) actually would be fine; it >> doesn't do anything if the CSS class to be added is already there. >> >> But I guess it's possible some other after_prepare hooks might not be as >> smart. I'm not sure what to do about that. >> >> >> -----Original Message----- >> From: Carlos Santana [mailto:csantan...@gmail.com] >> Sent: Thursday, April 21, 2016 2:25 PM >> To: dev@cordova.apache.org >> Subject: Re: [DISCUSS] Faster incremental builds >> >> The only thing to watch out is folks modified files during after_prepare >> event, then they will be broken if the file have not changed and they >> modified/add content to the file a second time. >> >> To make this work the hook event will need to inform batch/node and plugins >> hooks on before_prepare and after_prepare the list of files that are going >> to be copy, so the hook owner can know not to modified the file if their >> file is not I the list. >> >> One real use case is Ionic and I think MobileFirst. >> For MobileForst I think we don't modified I think we just recreate >> new file ok every prepare >> >> For Ionic they edit the index.html to a class to to the body element >> to for platform identification >> >> - Carlos >> @csantanapr >> >>> On Apr 21, 2016, at 5:12 PM, Jason Ginchereau <jason...@microsoft.com> >>> wrote: >>> >>> If "cordova clean" would also take care of deleting the files copied by >>> prepare, then I'd be confident in making prepare incremental by default. >>> Then if anyone was ever concerned about whether the incremental prepare was >>> doing the right thing, "cordova clean" would be a fairly obvious way to >>> reset the build. I still can't think of a realistic developer scenario >>> where that would be necessary, but it seems like the right thing for a >>> "clean" command to do regardless. >>> >>> So I will work on enhancing the "clean" function for each platform at the >>> same time I enable the incremental prepare. >>> >>> Jason >>> >>> -----Original Message----- >>> From: Jason Ginchereau [mailto:jason...@microsoft.com] >>> Sent: Wednesday, April 20, 2016 10:00 AM >>> To: dev@cordova.apache.org >>> Subject: RE: [DISCUSS] Faster incremental builds >>> >>> My concern with making this the default behavior is that Cordova >>> prepare doesn't have a corresponding "clean" operation. While there >>> is a "cordova clean" command, it only does a clean for the >>> native-code project (e.g. gradle clean), which doesn't affect the >>> files copied during prepare. I guess this would only be a problem if >>> you did something to give your source files older last-modified >>> times... then a cordova prepare/build/run that was incremental would >>> not pick up the older files, and "cordova clean" wouldn't fix it. I >>> was thinking a git checkout could do that, but upon investigation it >>> actually doesn't reset files' last-modified times. (I guess any >>> source control tools shouldn't, as it would mess up any kind of >>> incremental build system.) >>> >>> Maybe "cordova clean" should also delete the files that are copied by >>> prepare? Or maybe there could be something like a "cordova prepare --force" >>> flag for the rare case when a non-incremental prepare is needed? Between >>> those two I'd prefer making "cordova clean" more thorough; I didn't think >>> of that until now. >>> >>> Jason >>> >>> -----Original Message----- >>> From: Steven Gill [mailto:stevengil...@gmail.com] >>> Sent: Wednesday, April 20, 2016 9:18 AM >>> To: dev@cordova.apache.org >>> Subject: Re: [DISCUSS] Faster incremental builds >>> >>> Thanks for sharing the stats! Looks like a great improvement! >>> >>> I'm fine with this going in sans flag if we test it bit more first. >>> >>> Have your tried it with the --browserify flag? >>>> On Apr 20, 2016 7:18 AM, "Victor Sosa" <sosah.vic...@gmail.com> wrote: >>>> >>>> This is an amazing improvement in Cordova performance. Thanks Jason >>>> +1!! >>>> >>>>> On Wed, Apr 20, 2016 at 9:14 AM Shazron <shaz...@gmail.com> wrote: >>>>> >>>>> +1 what Jesse said >>>>> >>>>>> On Tuesday, April 19, 2016, Jesse <purplecabb...@gmail.com> wrote: >>>>>> >>>>>> This sounds great! >>>>>> Personally, I don't think this belongs behind a flag, why would >>>>>> anyone >>>>> not >>>>>> want to build faster? >>>>>> I think we just need to test more to make sure it functions as >>>>>> expected everywhere, and then it should just make it's way in >>>>>> directly, without >>>>> the >>>>>> overhead of addition flag code, or documentation ... >>>>>> >>>>>> >>>>>> @purplecabbage >>>>>> risingj.com >>>>>> >>>>>> On Tue, Apr 19, 2016 at 5:58 PM, Jason Ginchereau < >>>>> jason...@microsoft.com >>>>>> <javascript:;>> >>>>>> wrote: >>>>>> >>>>>>> I collected data for 4 projects with varying numbers and sizes >>>>>>> of >>>>> source >>>>>>> files (mostly .js and .png): >>>>>>> Tiny : 17 files, 0.24 MB (A new Cordova project created from >>>>> the >>>>>>> VS template) >>>>>>> Small : 55 files, 7.7 MB >>>>>>> Medium: 242 files, 44 MB >>>>>>> Large : 12,923 files, 134 MB >>>>>>> >>>>>>> Following are full -> incremental times for 'cordova prepare android' >>>>>>> commands, without any source file changes in between, averaged >>>>>>> over 5 >>>>>> warm >>>>>>> runs on my medium-spec Windows desktop machine with SSD: >>>>>>> >>>>>>> Tiny : 1.98 -> 1.84 s >>>>>>> Small : 2.40 -> 1.94 s >>>>>>> Medium: 4.22 -> 2.02 s >>>>>>> Large : 50.24 -> 5.81 s >>>>>>> >>>>>>> As you can see, larger projects see greatly improved prepare >>>>>>> times >>>> when >>>>>>> using the incremental option. >>>>>>> >>>>>>> Also, I confirmed that without using the --incremental option >>>>>>> with >>>> this >>>>>>> change, times are not measurably different compared to the >>>>>>> released >>>>>> version >>>>>>> 6.1.1 of Cordova. So there doesn't seem to be any regression >>>>>>> compared >>>>> to >>>>>>> the previous use of shelljs.cp('-R') to blindly copy files when >>>>>> preparing. >>>>>>> >>>>>>> Jason >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Steven Gill [mailto:stevengil...@gmail.com <javascript:;>] >>>>>>> Sent: Tuesday, April 19, 2016 3:14 PM >>>>>>> To: dev@cordova.apache.org <javascript:;> >>>>>>> Subject: Re: [DISCUSS] Faster incremental builds >>>>>>> >>>>>>> Sounds like a worthy cause. Do you have any stats on how much >>>>>>> time is saved? >>>>>>> >>>>>>> Definitely put it behind the --incremental flag to start. >>>>>>> >>>>>>> On Tue, Apr 19, 2016 at 2:43 PM, Jason Ginchereau < >>>>>> jason...@microsoft.com <javascript:;>> >>>>>>> wrote: >>>>>>> >>>>>>>> We've had a few customers complain that the dev inner loop for >>>>> Cordova >>>>>>>> apps is slow compared to native app development. So recently >>>>>>>> I've >>>>> been >>>>>>>> looking at ways to optimize it. The two largest pieces of a >>>>>>>> Cordova build are "prepare" and "compile" phases. While there's >>>>>>>> not much we can realistically do to speed up the native-code >>>>>>>> compilation (in >>>>> which >>>>>>>> gradle/xcodebuild/etc is invoked), it's clear that the prepare >>>> phase >>>>>>>> is not nearly as efficient as it could be. So I opened a JIRA >>>>>>>> for that, and I have a solution in the works that I want to get >>>> feedback >>>>>> on. >>>>>>>> >>>>>>>> CB-11117: Preparing platforms should skip copying files which >>>> haven't >>>>>>>> changed< >>>>> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f >>>>>>>> %2fissues.apache.org >>>>> %2fjira%2fbrowse%2fCB-11117&data=01%7c01%7cjasongi >>>>>>>> n%40microsoft.com >>>>> %7c297f714a60a94c3f258108d3689ff306%7c72f988bf86f141a >>>>> f91ab2d7cd011db47%7c1&sdata=E2Ww9LusEmVkUgjXPPGnxcGz%2fMH8fWruqGWU >>>>> r >>>>> L >>>>> Xq >>>>>>>> KRk%3d> Many cordova CLI commands include a "prepare" >>>>>>>> operation, including 'cordova build', 'cordova run', 'cordova >>>>>>>> plugin add', and more. Every time each of those commands runs, >>>>>>>> the target platform >>>> is >>>>>>>> "prepared", which involves copying all files from >>>>>>>> [<project>/www, <project>/platforms/<platform>/platform_www, >>>>>>>> <project>/merges/<platform>] to the platform's target www >>>>>>>> folder, >>>> as >>>>>>>> well as copying a bunch of icons and splash screens to >>>>>> platform-specific >>>>>>> locations. >>>>>>>> >>>>>>>> For the very first prepare of a platform, all that file copying >>>>>>>> is necessary. But most of the time after that most of the files >>>>>>>> being copied have not changed and therefore don't really need >>>>>>>> to be >>>> copied >>>>>>>> again. So the typical developer inner loop (edit a few source >>>> files, >>>>>>>> build and run the app, repeat) is a lot slower than it could be >>>> for a >>>>>>>> Cordova project, especially one that includes a significant >>>>>>>> number >>>> of >>>>>>>> source files or resources. >>>>>>>> >>>>>>>> Instead, Cordova should be smart enough to skip copying of >>>>>>>> files >>>> that >>>>>>>> haven't changed, based on their last-modified timestamp. But >>>>>>>> there should still be a way to force a >>>>>>>> clean/full/non-incremental copy if desired. To preserve >>>>>>>> compatibility with all possible existing workflows, I'm leaning >>>>>>>> toward keeping the full copy as the default behavior, and >>>>>>>> enabling the optimization only with a new >>>> --incremental >>>>>>>> flag, at least initially. If we find the incremental option is >>>>> working >>>>>>>> well for everyone, we can promote that behavior to default in a >>>>> future >>>>>>> major version update. >>>>>>>> >>>>>>>> See PRs here for implementation of this idea... >>>>>>>> https://github.com/apache/cordova-lib/pull/429: Add FileUpdater >>>>> module >>>>>>>> to cordova-common >>>>>>>> https://github.com/apache/cordova-android/pull/295: Use >>>> FileUpdater >>>>> to >>>>>>>> optimize prepare for android platform I'll submit PRs for >>>> cordova-ios >>>>>>>> and cordova-windows platforms soon, as well as a cordova-cli >>>>>>>> change >>>>> to >>>>>>>> enable the optimization via a new --incremental flag (assuming >>>>>>>> we don't want to make it the default behavior). >>>>>>>> >>>>>>>> Jason >>> >>> -------------------------------------------------------------------- >>> - To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org >>> For additional commands, e-mail: dev-h...@cordova.apache.org >>> ÃÂâÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂà>>> ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂà>>> à >>> ÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂà >>> ÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂà >>>  >>> à>>> ÃÂÃÂÃÂ¥Fò >>> Vç7V'67&ÃÂ&RÃÂRÃÂÃÂÃÂÃÂâFWbÃÂVç7V'67&ÃÂ& >>> T6÷&F÷fæ6ÃÂRæà >>> ·&pÃÂäf÷"FFÃÂFÃÂöæ >>> ÃÂ6öÃÂÃÂæG2ÃÂRÃÂÃÂÃÂÃÂâFWbÃÂÃÂVà>>> 6÷&F÷fæ6ÃÂRæ÷&pà>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org >> For additional commands, e-mail: dev-h...@cordova.apache.org >> >> BÂKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK >> KK >> KCBÂà [ÂÃXÂÃÃÂXÂKK[XZ[ > Â]Â][ÂÃXÂÃÃÂXÂPÃÃÂÃÂKÂ\XÃKÂÃÂÃBÂÂÃÂY >> ][ÃÂ[ÃÃ[X[ ÃK[XZ[ > Â]ÂZ[ ÃÃÂÃÂKÂ\XÃKÂÃÂÃB > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org > For additional commands, e-mail: dev-h...@cordova.apache.org > > BKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK > KCBÈ [ÝXØÜXKK[XZ[ ]][ÝXØÜXPÛÜÝK\XÚKÜÃBÜY > ][Û[ÛÛ[X[ ËK[XZ[ ]Z[ ÛÜÝK\XÚKÜÃB --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org For additional commands, e-mail: dev-h...@cordova.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cordova.apache.org For additional commands, e-mail: dev-h...@cordova.apache.org
