Hi Rom, So if I understand correctly, with vm_cache.vdi marked immutable, VBox makes a differencing VDI. To persist this, we would need to pick this file up and copy it to projects/, then on the next run load *it* up instead of vm_cache? If so this seems unnecessarily complicated, why not just have vm_cache.vdi be mutable and simply attach it to the VM straight from the projects/ folder? In practice, it would be marked NOT <copy_file/> so we get a softlink to it in slots, vboxwrapper follows this to projects/ and attaches the VM, so it would seem in line with the logic you'd expect from an app dealing with a softlinked file.
Thanks, Marius On Mon, Jun 29, 2015 at 12:11 PM, Rom Walton <[email protected]> wrote: > Vboxwrapper.cpp:673 would be the place you would want to start with any > local changes. > > > > I think we wanted to go one step further when removing the <copy_file/> > flag and also mark the VDI as immutable. That would cause VirtualBox to > create a differencing disk in the local slot directory and avoid the disk > image copy operation to the slot directory. > > > > Doing so will probably lead to some bigger changes though. > > > > ----- Rom > > > > *From:* Marius Millea [mailto:[email protected]] > *Sent:* Monday, June 29, 2015 1:30 AM > *To:* Rom Walton <[email protected]>; [email protected] > > *Subject:* Re: [boinc_dev] is there a way to persist files on the host > > > > Hi Rom, > > > > I saw your commits regarding the scratch directory. Thanks for the > amazingly quick response! > > > > Unfortunately, I now realize its not going to work out as I initially > imaged it (see this discussion > <https://forums.docker.com/t/how-does-boot2docker-persistence-work/2077> > with one of the boot2docker guys). Basically, the docker directory can't be > on a VBox shared folder due to insufficiencies of vboxfs. > > > > I should have realized that the correct way to do this is via > boot2docker's normal way of persistence, which just uses a VDI disk. > Essentially a persistent VDI would live in the BOINC project/ directory, > and vboxwrapper would attach that same one each time to the VM (meaning you > can only have 1 of these kinds of tasks running at a time, granted that was > always the case without properly installing the docker client on each > host). This is in fact *almost* exactly what the wrapper currently does. > The only change needed would be to allow vm_cache.vdi to not have > <copy_file/> and to have vboxwrapper correctly follow this link (I don't > think it currently does). I'm wondering what you think about possibly > adding an option to do that? I feel bad having you take the time for this > other scratch directory feature which was not really needed by me (I think > it might still be useful to some anyway), plus I acknowledge I don't know > what other road blocks I might hit. If you wanted to just point me to where > I might make this change myself, I'm happy to do it. > > > > In any case, I had been hoping I could avoid changes to vboxwrapper so > that I could have the convenience of using the precompiled versions you > guys put up. Since that doesn't look like its going to be possible, is > there any help you could give me on cross compiling? What does your setup > look like? (I've never done this before) > > > > Thanks, > > > > Marius > > > > > > On Thu, Jun 25, 2015 at 4:40 PM, Marius Millea <[email protected]> > wrote: > > Thanks for the interest. Thus far just Linux, although a Windows machine > is waiting once any of this kind of works. The way I currently envisage it > (which could well change), there would be no forking vboxwrapper required. > Basically I modified the boot2docker ISO to act as "multi purpose app" (in > the language of http://boinc.berkeley.edu/trac/wiki/VboxApps) so on boot > it mounts shared/ and from there runs some boinc_app script. That script > then calls "docker run..." etc.. so its all happening inside the VM, the > host system never needs to interact directly with the docker daemon. It > could certainly be done other ways too though. > > > > And agreed, the boot image size is very enticing. There is also the size > of the docker image that gets run, but there are decent to good base images > in the 5-100Mb range, and with this persistence thing working it would be a > one time download (and thank's to Docker, if you're running different > images all based on the same base image, you only download the changes). > > > > Marius > > > > On Thu, Jun 25, 2015 at 4:18 PM, Rom Walton <[email protected]> wrote: > > Marius, > > What host platform are you testing against? > > ----- Rom > > -----Original Message----- > From: boinc_dev [mailto:[email protected]] On Behalf Of > Rom Walton > Sent: Thursday, June 25, 2015 5:43 PM > To: Marius Millea <[email protected]> > Cc: [email protected] > Subject: Re: [boinc_dev] is there a way to persist files on the host > > Well, that is an interesting idea. > > Ultimately I think there will need to be a fork of vboxwrapper that knows > how to handle the docker daemon as well as VirtualBox. > > A 27MB boot image is pretty enticing. > > ----- Rom > > From: Marius Millea [mailto:[email protected]] > Sent: Thursday, June 25, 2015 4:43 PM > To: Rom Walton <[email protected]> > Cc: [email protected] > Subject: Re: [boinc_dev] is there a way to persist files on the host > > Yea sorry ambiguous what I meant by host there. The idea is that BOINC > clients are running VBoxwrapper which loads up boot2docker inside of which > I run my Docker apps. > > Marius > > On Thu, Jun 25, 2015 at 1:38 PM, Rom Walton <[email protected]<mailto: > [email protected]>> wrote: > Hosts or guests? > > Using VirtualBox implies that your docker application would be running > within a Linux guest. > > ----- Rom > > From: Marius Millea [mailto:[email protected]<mailto:[email protected] > >] > Sent: Thursday, June 25, 2015 4:10 PM > To: Rom Walton <[email protected]<mailto:[email protected]>> > Cc: [email protected]<mailto:[email protected]> > > Subject: Re: [boinc_dev] is there a way to persist files on the host > > Hi Rom, > > Yea, I think that's exactly what I'd need. I had started looking at the > source for vboxwrapper, but if that's something you could add officially to > BOINC then that'd be amazing, thanks! > > My end goal btw is to have hosts running Docker, and this would allow > persisting images between tasks. We'll see if there's any other road > blocks... > > Marius > > > > > > On Thu, Jun 25, 2015 at 12:25 PM, Rom Walton <[email protected]<mailto: > [email protected]>> wrote: > I can add another shared directory that points to a scratch area in the > project's directory. > > Would that work for you? > > ----- Rom > > -----Original Message----- > From: boinc_dev [mailto:[email protected]<mailto: > [email protected]>] On Behalf Of Marius Millea > Sent: Thursday, June 25, 2015 2:27 PM > > To: [email protected]<mailto:[email protected]> > Subject: Re: [boinc_dev] is there a way to persist files on the host > > (sorry if this screws up the threading, I can't figure out how to reply to > a specific message given I receive only the digest?) > > > I see, that makes a lot of sense, thanks. I suppose a followup problem is > that this is a VBox app, which AFAIK only gets access to the mounted > shared/ folder. Is there any way around this? > > Marius > > > The easiest way is to not tell BOINC about the file - > > just create it in the project directory, and look for it there in > > subsequent jobs. > > > > The app can get the project dir from the APP_INIT_DATA: > > http://boinc.berkeley.edu/trac/wiki/StatusApi > > > > -- David > > > > > > On 25-Jun-2015 2:10 AM, Marius Millea wrote: > > > Suppose the result of a computation yields a large file which I > > > would > > like > > > > > > 1) to remain on host > > > 2) to be used in subsequent computations > > > 3) to not be uploaded > > > > > > I think I know how to do 1), if I have the file in the output > > > template I can add the <copy_file> flag. Is there actually any way > > > to do 2 or 3 though? > > > > > > Thanks, > > > > > > Marius > _______________________________________________ > boinc_dev mailing list > > [email protected]<mailto:[email protected]> > > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev > To unsubscribe, visit the above URL and > (near bottom of page) enter your email address. > > > _______________________________________________ > boinc_dev mailing list > [email protected] > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev > To unsubscribe, visit the above URL and > (near bottom of page) enter your email address. > > > > > _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
