Write from mmapped buffer to O_DIRECT output file
Hello,
I have a devices which writes to a video buffer. This buffer is allocated
in system memory using CMA and I want to implement streaming write from
this buffer to a block device. My application opens video buffer with mmap
and I would like to use O_DIRECT write to avoid page cache related
overhead. Basically, the pseudo-code of the application looks like this:
f_in = open("/dev/videobuf", O_RDONLY);
f_mmap = mmap(0, BUFFER_SIZE, PROT_READ, MAP_SHARED, f_in, 0);
f_out = open("/dev/sda", O_WRONLY | O_DIRECT);
write(f_out, f_mmap, BLOCK_SIZE);
where BLOCK_SIZE is sector aligned value. f_out is opened without any
errors, but write results in EFAULT. I tried to track down this issue and
it turned out that mmap implementation in video buffer's driver uses
remap_pfn_range(), which sets VM_IO and VM_PFNMAP flags for VMA. The
O_DIRECT path in block device drivers checks these flags and returns
EFAULT. As far as I understand (from reading comments to VMA flags),
O_DIRECT writes need to pin the memory pages, but VMA flags indicate the
absence of "struct page" for underlying memory which causes an error. Am I
right here?
And the main question is how to correctly implement O_DIRECT write from
mmapped buffer? I have video buffer driver sources and can modify it
appropriately.
Mikhail
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Patch Question
On 19.04.2017 18:46, Perry Hooker wrote: > > Understood. I'm still open to the possibility that I've made a mistake > - I don't want to re-submit the patch if my analysis is incorrect. > > Maybe I didn't make it clear (my apologies if so) - what I'm really > looking for here is help confirming or refuting my work. If > re-submitting the patch is the best way to do this, then I can > certainly go that route. > Before you do this you should double check that the raised objections are indeed not justified. I have not looked too deep into the code but in function WILC_WFI_p2p_rx() the buffer is conditionally passed to cfg80211_rx_mgmt() which handles the passed data as being little endian. To me this is a strong indication that the data in the buffer is also little endian (which is what Dan pointed out and why the change you propose is not correct). Regards, Lino ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: What is the fastest way to build and boot a kernel
On Wed, Apr 19, 2017 at 3:32 AM, Tobin C. Harding wrote: > On Tue, Apr 18, 2017 at 08:59:36AM -0700, Code Soldier1 wrote: > [snip] > > Why the moniker? Why not ? unlike most people today I value my privacy. -- CS1 ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: What is the fastest way to build and boot a kernel
On Tue, Apr 18, 2017 at 08:59:36AM -0700, Code Soldier1 wrote: > Can someone suggest what is the fastest way to build a minimal kernel > and boot. For example if I am working on TCP and would like to make a > small change and reboot, what is the fastest way, I rather not build > the whole kernel. To add to the already great list of advice (some of which I could have used earlier in my life), I have found that virtual machines can be very useful. In particular, there are tools [1] that allow you to run a linux kernel within a virtual machine that shares your filesystem. You can use a quite minimal kernel configuration, and the resulting build times are around 5 seconds for incremental builds, and booting your new kernel in a virtual machine happens in mere seconds. Stephen [1]: https://github.com/g2p/vido ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: How to make per process firewall ?
> I would like to constrain process (by name) or group of process to specific > network interface and to specific port. This sounds like an excellent use-case for network namespaces [1]. They create an entire virtualized network stack within the kernel. This includes everything from network devices all the way up to firewall rules. You may create and administer namespaces using ip-netns(8). Alternatively, you can simply create a new one when you clone(2), by providing CLONE_NEWNET argument. You can run commands that affect namespaces created by ip-netns(8) using `ip netns exec`. If you didn't create a namespace with ip-netns, you can still run commands within any process's namespace via the nsenter(1) command, provided by util-linux. If you don't have that command (due to outdated util-linux), you can implement your own in less than 20 lines of C using the setns(2) system call. The manual page even provides a full implementation. In summary, the easiest way, with ip-netns(8), would be: ip netns add blue ip netns exec blue iptables -nvL # an empty firewall ip netns exec blue ip link # just a loopback # You'll likely want to create a veth pair, add one end to the "blue" netns, # and then set up routes. You'll have a separate IP address within the # netns, but I don't believe there's any way around that. ip netns exec blue iptables -A # your rule here ip netns exec blue YOUR-PROGRAMS Note that this is how Linux containers (e.g. Docker, LXC) work anyway, however, they virtualize other components of the kernel too (filesystem, process IDs, and much more). If all you want is to virtualize network resources, network namespaces are a more direct way to do this than containers, which will virtualize the rest as well. ALTERNATIVE [2]: You can apparently create iptables rules which match based on PID (not a great idea) or by UID/GID (a much better idea). If the overhead of network namespaces (veth pairs, new IPs, creating routes) is too much, you could create a user and run your processes as this user. Then create iptables rules that match based on the user. You do this with the "owner" module, and you can check whether it exists on your system by running: iptables -m owner [1]: https://lwn.net/Articles/580893/ [1]: also `man 7 namespaces` [2]: http://stackoverflow.com/questions/4314163/create-iptables-rule-per-process-service ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Patch Question
Thank you for your reply. > It's _your_ responsibility to convince me that your patch is correct and > should be applied. If no one respondes after a week or so, resend it, > with your additional information in the changelog so that the same > conversation doesn't happen again. Understood. I'm still open to the possibility that I've made a mistake - I don't want to re-submit the patch if my analysis is incorrect. Maybe I didn't make it clear (my apologies if so) - what I'm really looking for here is help confirming or refuting my work. If re-submitting the patch is the best way to do this, then I can certainly go that route. Regards, Perry ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Patch Question
On Wed, Apr 19, 2017 at 09:54:33AM -0600, Perry Hooker wrote: > At this point, no one has commented on the accuracy of my analysis. > Such comments are exactly what I'm hoping for (and what I requested - > not "demanded" - in my replies). As the one responsible for actually applying this specific patch, I find this pretty "interesting"... It's _your_ responsibility to convince me that your patch is correct and should be applied. If no one respondes after a week or so, resend it, with your additional information in the changelog so that the same conversation doesn't happen again. Patches get dropped all the time, remember, I get _hundreds_ of them every day. If I see a disagreement on a patch, then it goes to the end of the line, or usually just deleted from my queue. I rely on the fact that it's up to the submitter to resend and handle the discussion. Yes, we waste engineering time and effort by doing this, and that's fine, we have thousands of developers, but very few maintainers. It's their time that is valuable and the process is optimized for. good luck! greg k-h ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Patch Question
> Dan Carpenter is very good at what he dose. I would be hesitant to > ever call him or anyone as experienced 'flat-out incorrect'. Mr. Carpenter's first assertion that the patch "introduces bugs" was incorrect by his own admission: https://lkml.org/lkml/2017/3/22/120 Additionally, his assertion that the "buff likely is just a regular ieee80211_hdr struct" was also incorrect. > In your comments you in no way display that you understand exactly > what the code is doing and why it should by changed. Your initial > patch does not have an appropriate changelog message, please read > Documentation/process/submitting-patches.rst (section 2 Describing > your changes). I disagree. I've written both where the buffer is allocated and where it's filled, and why the contents need to be converted to little-endian byte order. Can you comment on whether my analysis is correct? Also, here are several commits that fix similar issues with similar changelog messages: https://github.com/torvalds/linux/commit/a15505e69cd2f8d0ebf566cd5c5838bd5c2d56e3 https://github.com/torvalds/linux/commit/47910a49db876397150b9754bc66f0c179448854 https://github.com/torvalds/linux/commit/af27e01cfcfcdf7f45488e023b474eb6de5f732e Can you comment on what's considered an "appropriate changelog message?" > A more subtle point - you may have more success if you do not put > demands onto people (eg can you explain this..) but rather write out > your understanding of the code explaining why your hold the views you > do. Others can then comment or this, agreeing or disagreeing as the > case may be. People like to help by giving their knowledge, no one > likes doing chores. As I mentioned (and is contained in the thread), I've written out my understanding of where the buffer is allocated, where it's filled, and why the conversion needs to happen. At this point, no one has commented on the accuracy of my analysis. Such comments are exactly what I'm hoping for (and what I requested - not "demanded" - in my replies). So, if you can provide some useful analysis, I'd be happy to listen. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Patch Question
> As far as I understood Dan Carpenters (last) post in that thread, the content > in the buffer > is already in little endian order. In this case the code is correct as it is > and there is no need > for the change you propose. Yes, I believe Mr. Carpenter is mistaken - I think the data is in host-endian order. Please see my analysis of where that buffer's contents are filled. Can you comment on whether my analysis is correct or not? Regards, Perry ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Patch Question
Hi, On 18.04.2017 01:28, Perry Hooker wrote: > Hi everyone, > > I recently submitted a patch to the kernel mailing list: > https://lkml.org/lkml/2017/3/21/712 > > I received some feedback on the patch. After a bit of polite > back-and-forth, the respondent stopped replying when I asked for more > information, and I haven't heard anything from the maintainers. > > Based on my analysis (contained in the thread), I still think the > patch is correct & appropriate. As far as I understood Dan Carpenters (last) post in that thread, the content in the buffer is already in little endian order. In this case the code is correct as it is and there is no need for the change you propose. Regards, Lino ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: What is the fastest way to build and boot a kernel
>> On Wed, Apr 19, 2017 at 6:49 AM, wrote: >> On Wed, 19 Apr 2017 20:32:31 +1000, "Tobin C. Harding" said: >> On Tue, Apr 18, 2017 at 08:59:36AM -0700, Code Soldier1 wrote: >> [snip] >> >> Why the moniker? >>Could be worse. "Code Cannon Fodder", for example And this is what happens when you slap the code with a flying trout: https://paste.debian.net/928354/ Pay attention to the build times. Yes this is a pre-compiled kernel but that last build was 0m3.720s which I think we can safely say is fast enough for most of us :) ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: What is the fastest way to build and boot a kernel
>> On Tue, Apr 18, 2017 at 11:59 AM, Code Soldier1 wrote: >> Hi, >> Can someone suggest what is the fastest way to build a minimal kernel >> and boot. Two ways to build a minimal kernel. Try make localmodconfig or make tinyconfig and see what fits your needs best. If this is still not what you need then you have to tweak things using make menuconfig ( can be a real pain in the butt... :) >> For example if I am working on TCP and would like to make a >> small change and reboot, what is the fastest way, I rather not build >> the whole kernel. Stay away from make menuconfig after the initial compile. The first time you use make menuconfig it will create .config for you. As long as you do not mess with the .config you can simply run make and should be able to build just the file(s) you changed. Keep in mind somethings are statically linked in to the kernel, meaning 'built-in' and if you touch those may trigger a full compile. To get a faster compile/build use make parallel build with -j option. Where -j is the number of processors available to you. For eg: make -j4 Also compile for the target architecture only: make ARCH= -jN UML ( User mode linux ) I have found is faster for reasons yet to be comprehended by me :) Hope this helps and monikers are good, whats wrong with slap that code with a flying trout eh ? Aruna > Thanks > > -- > CS1 > > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: What is the fastest way to build and boot a kernel
On Wed, 19 Apr 2017 20:32:31 +1000, "Tobin C. Harding" said: > On Tue, Apr 18, 2017 at 08:59:36AM -0700, Code Soldier1 wrote: > [snip] > > Why the moniker? Could be worse. "Code Cannon Fodder", for example pgpv0BFLq0gT5.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: What is the fastest way to build and boot a kernel
On Tue, Apr 18, 2017 at 08:59:36AM -0700, Code Soldier1 wrote: [snip] Why the moniker? ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
