Re: NTFS GSoC Project Idea
2012/3/27, Efstratios Karatzas gpf.k...@gmail.com: On Tue, Mar 27, 2012 at 11:06 PM, Gleb Kurtsou gleb.kurt...@gmail.comwrote: On (26/03/2012 21:13), Efstratios Karatzas wrote: Greetings, I am a FreeBSD GSoC 2010 student, looking to participate in this years GSoC. The project that I wish to work on is the FreeBSD NTFS driver. I 've already discussed my project idea with Attilio@ who suggested that I forward my proposal to the hackers mailing list in order to get more feedback and see if there's anyone interested in mentoring a NTFS project. The current draft of the technical part of my proposal(pdf plain text) can be found here: http://cgi.di.uoa.gr/~std06101/ntfs/ntfs_proposal.tar The project idea focuses on mp-safing the NTFS driver as well as adding extra write support. I've tried to merge the two conflicting NTFS project ideas in the ideas wiki page, into one. One of them suggesting that work should be done on the current driver (mp-safe it etc) and the other one suggesting that we port Apple's NTFS driver from scratch. The concept is that we keep most of our vnode/vfs code (i.e. ntfs_vfsops.c, ntfs_vnops.c) and rework existing infrastructure as needed as well as implement new features using Apple's driver as a guide. I'm not sure I follow your idea, but I'd suggest sticking to a single project: either improve FreeBSD NTFS or do the port. FreeBSD and Darwin ntfs implementations are completely unrelated thus porting features from Darwin won't be easy. This way, we avoid the major changes in Apple's VFS (is there any documentation to be found?) and port code that won't break current functionality. I bet major changes in Apple's VFS are easier to deal with than merging two unrelated file systems. XNU VFS is based on FreeBSD 5 VFS and they still share a lot in common, e.g. vnode operations themselves are nearly the same, e.g. extended attributes, locking, buffer cache are different. Take a look at HFS+ port. It's unmaintained and outdated but page contains link to CVS repository snapshot. http://people.freebsd.org/~yar/hfs/ I tried to keep this e-mail brief so If you wish to know more, please refer to the proposal. Thank you -- Efstratios GPF Karatzas Since the FreeBSD wiki has two conflicting NTFS ideas, I thought of submitting two proposals: one to work on the current driver and one to port Apple's driver and let the FreeBSD team choose, should they wish to pick one of them. Attilio suggested that perhaps a merge of those two ideas was better than two separate proposals and this was my attempt to merge those ideas; I hear what you say though. What I specifically had in mind is: - make NTFS mpsafe - bugbusting - surf in Apple implementation/history, check for bugfixes there and import in the FreeBSD implementation Unfortunately our idea page is not as updated as you want it to be, thus it happens to find duplicate ideas list, not well merged, etc. Attilio -- Peace can only be achieved by understanding - A. Einstein ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NTFS GSoC Project Idea
On Wed, Mar 28, 2012 at 2:45 PM, Attilio Rao atti...@freebsd.org wrote: 2012/3/27, Efstratios Karatzas gpf.k...@gmail.com: On Tue, Mar 27, 2012 at 11:06 PM, Gleb Kurtsou gleb.kurt...@gmail.comwrote: On (26/03/2012 21:13), Efstratios Karatzas wrote: Greetings, I am a FreeBSD GSoC 2010 student, looking to participate in this years GSoC. The project that I wish to work on is the FreeBSD NTFS driver. I 've already discussed my project idea with Attilio@ who suggested that I forward my proposal to the hackers mailing list in order to get more feedback and see if there's anyone interested in mentoring a NTFS project. The current draft of the technical part of my proposal(pdf plain text) can be found here: http://cgi.di.uoa.gr/~std06101/ntfs/ntfs_proposal.tar The project idea focuses on mp-safing the NTFS driver as well as adding extra write support. I've tried to merge the two conflicting NTFS project ideas in the ideas wiki page, into one. One of them suggesting that work should be done on the current driver (mp-safe it etc) and the other one suggesting that we port Apple's NTFS driver from scratch. The concept is that we keep most of our vnode/vfs code (i.e. ntfs_vfsops.c, ntfs_vnops.c) and rework existing infrastructure as needed as well as implement new features using Apple's driver as a guide. I'm not sure I follow your idea, but I'd suggest sticking to a single project: either improve FreeBSD NTFS or do the port. FreeBSD and Darwin ntfs implementations are completely unrelated thus porting features from Darwin won't be easy. This way, we avoid the major changes in Apple's VFS (is there any documentation to be found?) and port code that won't break current functionality. I bet major changes in Apple's VFS are easier to deal with than merging two unrelated file systems. XNU VFS is based on FreeBSD 5 VFS and they still share a lot in common, e.g. vnode operations themselves are nearly the same, e.g. extended attributes, locking, buffer cache are different. Take a look at HFS+ port. It's unmaintained and outdated but page contains link to CVS repository snapshot. http://people.freebsd.org/~yar/hfs/ I tried to keep this e-mail brief so If you wish to know more, please refer to the proposal. Thank you -- Efstratios GPF Karatzas Since the FreeBSD wiki has two conflicting NTFS ideas, I thought of submitting two proposals: one to work on the current driver and one to port Apple's driver and let the FreeBSD team choose, should they wish to pick one of them. Attilio suggested that perhaps a merge of those two ideas was better than two separate proposals and this was my attempt to merge those ideas; I hear what you say though. What I specifically had in mind is: - make NTFS mpsafe - bugbusting - surf in Apple implementation/history, check for bugfixes there and import in the FreeBSD implementation Unfortunately our idea page is not as updated as you want it to be, thus it happens to find duplicate ideas list, not well merged, etc. Attilio -- Peace can only be achieved by understanding - A. Einstein Here's a draft for a NTFS proposal that focuses on improving our current driver(mp-safe + extra write support). I believe it's well defined and shouldn't cause any confusion. Also a couple of fixes due to Pedro's e-mail (thanks!). The draft can be found here: http://cgi.di.uoa.gr/~std06101/ntfs/ntfs_mpsafe_proposal.tar Let me know what you think. -- Efstratios GPF Karatzas ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Reverse engineering; How to...
Greetings, Over the past year, in an effort to convert my server farm to wireless, I've purchased some half a dozen USB wireless dongles, at a total cost of ~150.00. Unfortunately, none of them are (yet) supported — I know, I know, I've already had this debate with both dev's, users. On the up-side, I've devised a resource that will greatly assist would-be adopters in selecting, and researching these, and other adapters _currently supported_ under under FreeBSD. That said; the adapter I most recently purchased, is quite nice (Cisco(Linksys) AE2500 Wireless-N). Boasts 2.5/5GHz @300Mbps. I figured (wrongly) because Linksys is so well supported on FreeBSD, that the likelihood of this being supported would be good. At any rate, given it's not, and because I _do_ have the Window$ drivers on the install CD. What are the possibilities I can reverse-engineer the drivers into a FreeBSD loadable module? I can unpack the setup file to extract the .sys files. While I _could_ utilize the ndisulator to load them, that's not my goal. Should I unpack the .sys file, and attempt to decompile/disassemble it? Or attempt to load it, and dump it from memory? — hacker/cracker advice _strongly_ desired — ## #usbconfig -d ugen1.2 dump_device_desc ugen1.2: Linksys AE2500 Cisco at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x00ff bDeviceSubClass = 0x bDeviceProtocol = 0x bMaxPacketSize0 = 0x0040 idVendor = 0x13b1 idProduct = 0x003a bcdDevice = 0x0001 iManufacturer = 0x0001 Cisco iProduct = 0x0002 Linksys AE2500 iSerialNumber = 0x0003 0001 bNumConfigurations = 0x0001 ## P.S. This message was sent from my smart phone. Apologies for any (mis)formatting. :-( --Chris.H -- FreeBSD 8.2-STABLE /AMD64 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Reverse engineering; How to...
Reverse engineering a whole driver could take a very long time, even with the proper tools. If it's possible, return the adapter, and buy a new one and verify that the chipset is supported before you buy it. Last time I bought a wireless card I sat in the store looking at the Wireless support list for BSD before buying. http://www.freebsd.org/relnotes/CURRENT/hardware/support.html#WLAN I very strongly suggest that you get a card with an Atheros chipset, as those are by far the best supported on BSD. -Brandon On 3/28/2012 4:22 PM, Chris.H wrote: Greetings, Over the past year, in an effort to convert my server farm to wireless, I've purchased some half a dozen USB wireless dongles, at a total cost of ~150.00. Unfortunately, none of them are (yet) supported — I know, I know, I've already had this debate with both dev's, users. On the up-side, I've devised a resource that will greatly assist would-be adopters in selecting, and researching these, and other adapters _currently supported_ under under FreeBSD. That said; the adapter I most recently purchased, is quite nice (Cisco(Linksys) AE2500 Wireless-N). Boasts 2.5/5GHz @300Mbps. I figured (wrongly) because Linksys is so well supported on FreeBSD, that the likelihood of this being supported would be good. At any rate, given it's not, and because I _do_ have the Window$ drivers on the install CD. What are the possibilities I can reverse-engineer the drivers into a FreeBSD loadable module? I can unpack the setup file to extract the .sys files. While I _could_ utilize the ndisulator to load them, that's not my goal. Should I unpack the .sys file, and attempt to decompile/disassemble it? Or attempt to load it, and dump it from memory? — hacker/cracker advice _strongly_ desired — ## #usbconfig -d ugen1.2 dump_device_desc ugen1.2: Linksys AE2500 Cisco at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x00ff bDeviceSubClass = 0x bDeviceProtocol = 0x bMaxPacketSize0 = 0x0040 idVendor = 0x13b1 idProduct = 0x003a bcdDevice = 0x0001 iManufacturer = 0x0001 Cisco iProduct = 0x0002 Linksys AE2500 iSerialNumber = 0x0003 0001 bNumConfigurations = 0x0001 ## P.S. This message was sent from my smart phone. Apologies for any (mis)formatting. :-( --Chris.H ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Alright guys, I'm at the end of my rope here. For those that haven't seen my previous emails here's the (not so) quick breakdown: Overview: FreeBSD ?? - 7.4 never crash FreeBSD 8.0 - 8.2 crashes FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our production at this time, and we were hoping we could base some stuff on 8.3 for long term stability...) ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others. History: Over the course of the last 2 years we've been banging our heads on the wall. VMWare is done debugging this. They claim it's not a VMWare issue. They can't identify what the heck happens. We had a glimmer of hope with ESXi 5.0 fixing it because we never saw any crashes in the handful of deployments, but our dreams were crushed today -- two days before an outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server and FreeBSD 8.2 guest crashed. Crash Details: The keyboard/mouse usually stops responding for input on the console; normally we can't type in a username or password. However, we can switch VTs. If there's a shell on the console and we can type, we can only run things in memory. Any time we try to access the disk it will hang indefinitely. The server still has network access. We can ping it without issue. SSH of course kicks you out because it can't do any I/O. If we were to serve a lightweight http server off a memory backed filesystem I'm confident it would run just fine as long as it wasn't logging or anything. On ESXi you see that there is a CPU spike of 100% that goes on indefinitely. No idea what the FreeBSD OS itself thinks it is doing because we can't run top during the crash. This crash can affect a server and happen multiple times a week. It can also not show up for 180 days or more. But it does happen. The server can be 100% idle and crash. We have servers that do more I/O than the ones that crash could ever attempt to do and these don't crash at all. Completely inexplicable. Things we've looked into: Nothing about the installed software matters. We've tried cross referencing the crashed servers by the programs they run but the base OS is the only common denominator due to the wide variety of servers it has affected. Storage doesn't matter. We've tried different iSCSI SANs, we've tried different switches, we've tried local datastores on the ESXi servers themselves. HP servers, Dell servers -- doesn't seem to matter either. (All with latest firmwares, BIOSes, etc) VMWare gave us a ton of debugging tasks, and we've given them gigabytes of debugging info and data; they can't find anything. VMWare tools -- with, without, using open-vm-tools makes no difference. I think we've done a fair job ruling out VMWare. I think we've finally found enough data that this is definitely something in the FreeBSD world. I'm going to begin prepping some of the known crashy servers with more debugging. Any suggestions on what I should build the kernel with? They never do a proper panic, but I definitely want to at least *try* to get into the debugger the next time it crashes. And when it crashes, what the heck should I be running? I've never played with the KDB before... Thank you for any suggestions and help you can give me ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Reverse engineering; How to...
Hi, If there's a linux/netbsd/openbsd driver then you could work with us to port them. We in the wireless stack hacker group are sorely lacking developers working on the chipsets that are out there. OpenBSD tends to have a bunch of wireless drivers that Damien has either ported or reimplemented from various (Linux) upstreams, so you could use that as a basis for future work. If someone has the following hardware AND can actually actively work on helping me port support from OpenBSD/Linux to FreeBSD, please drop me a line ASAP: (These are all USB) * AR9170 series NICs (Linux - carl9170, OpenBSD: ar9170) * AR7010 CPU + (AR9280, AR9285, AR9287) NICs (Linux, ath9k_htc, OpenBSD: athn) * AR9271 CPU+Wifi NIC (ath9k_htc/athn) I've got documentation and reference hardware here, so I can assist in helping debug and add 802.11n support. I just have no time to do all the initial USB bring up of these things. If in doubt, jump on freebsd-wirel...@freebsd.org and start talking. :) Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Please help me diagnose this crazy VMWare/FreeBSD 8.x crash
Hi, * have you filed a PR? * is the crash easily reproducable? * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create a ramdisk image using nanobsd?) and do some stress testing inside that? It sounds like you've established it's a storage issue, or at least interrupt handling for storage issue. So I'd definitely try the ramdisk-only boot and thrash it using lighttpd/httperf or something. If that survives fine, I'd look at trying to establish whether there's something wrong in the disk driver(s) freebsd is using. I'm not that cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x and 8.0 which has caused this to surface. 2c, Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NTFS GSoC Project Idea
Hello Efstratios ; In general, I agree with Gleb that you should start from the Apple Darwin port instead of spending time on the current FreeBSD driver. Please note that last year someone attempted to bring in smbfs from Darwin with your same strategy and failed: http://lists.freebsd.org/pipermail/soc-status/2011-June/000340.html Making it MPsafe may look relatively easy but it will take valuable time. In the case of ext2fs, for example, I am convinced it was only done in time because the ext2fs code is based on UFS1. Also the lack of write support has made the current NTFS driver undesirable and for a while, even before the MP-unsafe axing was defined, the driver was being considered for deprecation. Quite honestly I think we want the Darwin driver. If it serves as further encouragement, when I asked Yar about his HFS port he said everything he took from Darwin basically compiled. cheers, Pedro. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NTFS GSoC Project Idea
On (28/03/2012 18:43), Pedro Giffuni wrote: Hello Efstratios ; In general, I agree with Gleb that you should start from the Apple Darwin port instead of spending time on the current FreeBSD driver. I'd suggest submitting two proposals -- too much depends on a mentor availability for a particular project. Please note that last year someone attempted to bring in smbfs from Darwin with your same strategy and failed: http://lists.freebsd.org/pipermail/soc-status/2011-June/000340.html Making it MPsafe may look relatively easy but it will take valuable time. In the case of ext2fs, for example, I am convinced it was only done in time because the ext2fs code is based on UFS1. Also the lack of write support has made the current NTFS driver undesirable and for a while, even before the MP-unsafe axing was defined, the driver was being considered for deprecation. Quite honestly I think we want the Darwin driver. If it serves as further encouragement, when I asked Yar about his HFS port he said everything he took from Darwin basically compiled. My experience with Darwin NTFS was similar. I had most of it compiling in several evenings. At least extended attributes and utf-8 routines had to be disabled. Sorry, I can't recall the details, had no time to get back to it afterwards. Thanks, Gleb. cheers, Pedro. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
