I am have created FAT 16 disk with bytes per sector as 4096 but when reading the data off the disk I am not able to read the all bytes because bytes per sector is harded coded to be 512 in the underlying implementation. Is there a way to get around this ? I tried to pin the source code of fat-filesystem but couldn't get the changes to be reflected. Any pointers would be helpful.
On Tue, Feb 16, 2016 at 3:47 PM, Vanshdeep Singh <[email protected]> wrote: > Working on the tracing part but here's an interesting observation, If I > don't increment the file_offset > after writing each page_buffer then the all of the data is uploaded, only > when I increment the > file_offset after writing each page_buffer it gets stuck after writing > 3-4Mb of data. > Further, I tried creating a FAT 16 disk with logical sector size 4096 i.e. > mkfs.fat -F 16 -S 4096 -C disk.img 1024000 > using this disk I was able to write almost 30MB of data after which it > started to hang again. > I am speculating that it hangs in write_to_location > <https://github.com/mirage/ocaml-fat/blob/master/lib/fs.ml#L286> while > allocating new sectors. Any ideas on this ? > > > On Sun, Feb 14, 2016 at 3:38 AM, Thomas Leonard <[email protected]> wrote: > >> On 13 February 2016 at 07:38, Vanshdeep Singh <[email protected]> wrote: >> > >> > >> > On Fri, Feb 12, 2016 at 1:14 PM, Thomas Leonard <[email protected]> >> wrote: >> >> >> >> On 12 February 2016 at 05:56, Vanshdeep Singh <[email protected]> >> wrote: >> >> > Hi, >> >> > I tried running the my implementation directly on xen and performance >> >> > was >> >> > much better (no idea why). >> >> > But I have run into new issues, >> >> > - Tried creating a disk of size 1000Mb using "fat create disk.img >> >> > 102400KiB" >> >> > and it returned >> >> > "fat: unimplemented" even though the disk was created. >> >> >> >> Boot_sector.format_of_clusters will choose FAT32 if it needs 65527 or >> >> more clusters. However, it appears that only FAT16 is implemented. I'm >> >> not sure what changes are required for FAT32. >> >> >> >> For testing, you could format it with mkfs (mkfs -t fat disk.img), but >> >> I guess you'll have the same problem using it. >> > >> > I was able to successfully create and run the a 1GB fat disk using >> mkfs.fat >> >> >> >> >> >> > Then I tried running >> >> > it on the xen and got an >> >> > error after I ran the image on xen, >> >> > Fatal error: exception Fs.Make(B)(M).Fs_error(_) >> >> > Raised at file "src/core/lwt.ml", line 789, characters 22-23 >> >> >> >> Mirage error reporting really needs sorting out. For now, you could >> >> use Printexc.register_printer in fs.ml to tell it how to display the >> >> error as something other than "_". >> >> >> >> > - I also tried uploading a file with size around 30MiB onto a >> disk.img >> >> > of >> >> > size 100MiB. The hanged >> >> > after writing 4Mb of data. >> >> >> >> > Any suggestion on how to deal with the above situations ? >> >> >> >> Was it spinning (high CPU load shown in "xl list") or waiting for >> >> something (idle)? >> >> >> >> If spinning, you can grab a stack trace to find out where: >> >> >> >> >> >> >> http://www.brendangregg.com/blog/2016-01-27/unikernel-profiling-from-dom0.html >> >> >> >> If it's waiting for something, annotate your main thread with >> >> MProf.Trace.should_resolve and compile with tracing on. When you view >> >> the trace, your thread (which never finishes) will be shown in red and >> >> you can follow the yellow arrows to discover what it was waiting for. >> >> See: >> >> >> >> https://mirage.io/wiki/profiling >> >> >> >> Both of these techniques may be useful for finding performance problems >> >> too. >> > >> > I have tried to narrow down the problem and it turn turns out that the >> code >> > gets >> > stuck at Fs.write because on commenting Fs.write all the data is >> > successfully >> > received and iterated using Lwt_stream.iter_s . But when I try to write >> > using Fs.write >> > first 3 to 4 page buffers are successfully written and then it hangs. I >> > tried to profile >> > the vm using mirage-trace-view but there was not much I could >> understand. I >> > am >> > attaching the results in case you can see and suggest something. >> >> The first two traces seem to be mostly networking stuff. It might be >> worth simplifying the test case so the unikernel just writes test data >> directly (or reads a small request and writes it many times). >> >> The third doesn't have many labels, so it might be mirage-block-xen >> stuff. I see I started adding trace events, but never got around to >> submitting a PR: >> >> https://github.com/talex5/mirage-block-xen/tree/tracing >> >> (trace labels have no cost when compiling without tracing, so it would >> be good to have more!) >> >> The last two traces show the unikernel constantly waking up and them >> immediately sleeping again without doing anything. Very odd. Might be >> worth adding some trace labels around here: >> >> >> https://github.com/mirage/mirage-platform/blob/dfd00d518570c074b4e9b36a59472f5e7354df5f/xen/lib/main.ml#L62 >> >> > Note: I was trying to upload a 30Mb file which could copy into the disk >> > using "fat add" >> > command but when I tried uploading and writing to the disk the Fs.write >> call >> > won't >> > return after writing a few page buffers. >> > >> > about the files: trace.ctf 1-5 show the incremental trace of the vm >> when I >> > upload the >> > 30Mb file >> > >> > >> >> >> >> >> >> > Regards, >> >> > Vansh >> >> > >> >> > >> >> > On Thu, Feb 11, 2016 at 8:11 PM, Vanshdeep Singh <[email protected]> >> >> > wrote: >> >> >> >> >> >> Hi Thomas, >> >> >> I am chosen to implement the disk in FAT format. Drawing inspiration >> >> >> from >> >> >> your code I >> >> >> have tried to do disk writing operations but instead of >> V1_LWT.BLOCK I >> >> >> have chosen to >> >> >> go wo with V1_LWT.FS because for the api but the write performance I >> >> >> get >> >> >> is very poor. >> >> >> I takes more than 11 sec to upload a 67Kb file. The file is uploaded >> >> >> quickly but the time >> >> >> taken to write to disk is long hence they delay. >> >> >> >> >> >> Much of my implementation is similar to this code >> >> >> >> >> >> >> >> >> >> https://github.com/0install/0repo-queue/blob/master/upload_queue.ml#L159-L172 >> >> >> the difference comes in the flush_page_buffer . Since I am using >> >> >> V1_LWT.FS >> >> >> I use >> >> >> FS.write call to write the data to the disk i.e. >> >> >> >> >> >>> >> >> >>> buffered_data = Cstruct.sub page_buffer 0 !page_buffer_offset >> >> >>> >> >> >>> Fs.write fs path !file_offset buffered_data >> >> >> >> >> >> >> >> >> >> >> >> How can I improve the performance ? >> >> >> >> >> >> Note: I am testing this using --unix >> >> >> >> >> >> >> >> >> Regards, >> >> >> Vansh >> >> >> >> >> >> On Sun, Feb 7, 2016 at 11:28 PM, Thomas Leonard <[email protected]> >> >> >> wrote: >> >> >>> >> >> >>> On 6 February 2016 at 20:48, Vanshdeep Singh <[email protected]> >> >> >>> wrote: >> >> >>> > Hi, >> >> >>> > I am trying to build a sample file storage web app and I am need >> >> >>> > some >> >> >>> > directions >> >> >>> > on how to approach it, particularly I am trying to figure out >> how to >> >> >>> > do >> >> >>> > storage. >> >> >>> > Currently, I am drawing my insight from here and here (irmin). >> Any >> >> >>> > kind >> >> >>> > of >> >> >>> > suggestion >> >> >>> > would be really helpful. >> >> >>> > >> >> >>> > NOTE: files of any size could be uploaded so I am aiming at >> >> >>> > streaming >> >> >>> > uploads/downloads. >> >> >>> >> >> >>> Hi Vansh, >> >> >>> >> >> >>> Currently, FAT is the only supported file-system on Mirage/Xen: >> >> >>> >> >> >>> https://github.com/mirage/ocaml-fat >> >> >>> >> >> >>> If your needs are simpler then you could also implement your own >> >> >>> scheme. The file queue example you linked just stores the files >> >> >>> sequentially on the disk, which is fine for a queue. >> >> >>> >> >> >>> If you want to help build something better (e.g. to support Irmin), >> >> >>> the ocaml-btree project is under development: >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> http://lists.xenproject.org/archives/html/mirageos-devel/2016-01/msg00059.html >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Dr Thomas Leonard http://roscidus.com/blog/ >> >> >>> GPG: DA98 25AE CAD0 8975 7CDA BD8E 0713 3F96 CA74 D8BA >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> >> Dr Thomas Leonard http://roscidus.com/blog/ >> >> GPG: DA98 25AE CAD0 8975 7CDA BD8E 0713 3F96 CA74 D8BA >> > >> > >> >> >> >> -- >> Dr Thomas Leonard http://roscidus.com/blog/ >> GPG: DA98 25AE CAD0 8975 7CDA BD8E 0713 3F96 CA74 D8BA >> > >
_______________________________________________ MirageOS-devel mailing list [email protected] http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel
