Re: Wikibook Linux kernel
On Thu, May 06, 2021 at 01:58:35PM +0300, Constantine Shulyupin wrote: > Dear Linux kernel documentation writers and readers: > > Writing Linux documentation is a huge complex collaborative process. > To make it better I invite you to contribute to > https://en.wikibooks.org/wiki/The_Linux_Kernel There are some wiki's that are available at *.wiki.kernel.org. For example, ext4.wiki.kernel.org. We've largely abandoned it, in favor of using Documentation in the kernel sources, because if you leave it "updated by anyone", unless you have people constantly watching for spam or trash updates which have to be reverted, it quickly becomes a mess. Or you can keep tight control over who you give accounts to, but then it doesn't get updated all that often. Keeping the documentation in sync with the kernel sources means it's much more likely for the documentation to be updated when the kernel is updated, and so for example we've migrated: https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout to: https://www.kernel.org/doc/html/latest/filesystems/ext4/index.html with the sources available at: https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/tree/Documentation/filesystems/ext4 ... and been much happier with the result. Cheers, - Ted ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: How to verify linux-next
On Sun, Oct 01, 2017 at 09:58:37PM +0530, Pintu Kumar wrote: > Ok thank you so much for your reply. > Now I am able to boot with v4.14-rc2. But now I am facing another problem. > Now, I am not able to connect to internet from virtual box. > When I switch back to the default 4.10 the internet works normally. > I think the dlclient stopped working. > I am getting continuous logs related to apparmor like this: > apparmor="DENIED" comm=dhclient > apparmor="DENIED" comm=cups-browsed How about compiling the kernel without apparmor? Here are some very minimal kernel configs that are designed to work with qemu and Google Compute Engine (if you use the 64-bit configs): https://github.com/tytso/xfstests-bld/tree/master/kernel-configs Grab the latest 4.9 kernel configs (e.g., [1]); it will work with 4.14-rc2, copy it into your kernel build tree as .config, and then run "make olddefconfig". That's why I use with qemu (32-bit and 64-bit) and Google Compute Engine using a minimal debian chroot, and it works Just Fine. [1] https://github.com/tytso/xfstests-bld/blob/master/kernel-configs/x86_64-config-4.9 If you want an example of how to build a automatically generated test appliance image (which is repeatably built, and can be used as part of a discplined release engineering process), please see: https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/gen-image Cheers, - Ted P.S. And here is an example of how a completely automated kernel testing report might look like. I ran a single command, and a few hours later, the following lands in my e-mail's inbox. Note the lack of using putty to manually configure a single test, and not a single terminal was opened during the entire test run. :-) TESTRUNID: ltm-20171001045435 KERNEL:kernel 4.14.0-rc2-ext4-5-gcdfa281a30f5 #513 SMP Sun Oct 1 00:52:09 EDT 2017 x86_64 CMDLINE: --kernel gs://gce-xfstests/bzImage full CPUS: 2 MEM: 7680 ext4/4k 337 tests, 23 skipped, 6 errors, 10569 seconds generic/233 generic/361 generic/388 generic/451 generic/456 generic/459 ext4/1k 337 tests, 35 skipped, 8 errors, 12440 seconds generic/018 generic/232 generic/273 generic/361 generic/388 generic/451 generic/456 generic/459 ext4/ext3 336 tests, 73 skipped, 6 errors, 10639 seconds generic/232 generic/361 generic/382 generic/388 generic/451 generic/459 ext4/encrypt 327 tests, 97 skipped, 8 errors, 6785 seconds ext4/022 ext4/028 generic/081 generic/361 generic/382 generic/388 generic/459 shared/298 ext4/nojournal 334 tests, 60 skipped, 6 errors, 6107 seconds ext4/301 generic/113 generic/232 generic/441 generic/451 generic/459 ext4/ext3conv 336 tests, 23 skipped, 4 errors, 10039 seconds generic/361 generic/388 generic/451 generic/459 ext4/adv 335 tests, 28 skipped, 8 errors, 8732 seconds generic/232 generic/233 generic/361 generic/388 generic/399 generic/451 generic/456 generic/459 ext4/dioread_nolock 336 tests, 23 skipped, 5 errors, 7434 seconds generic/081 generic/233 generic/388 generic/451 generic/459 ext4/data_journal 336 tests, 33 skipped, 5 errors, 13027 seconds generic/361 generic/371 generic/388 generic/441 generic/459 ext4/bigalloc_1k 317 tests, 44 skipped, 10 errors, 8506 seconds generic/204 generic/235 generic/269 generic/273 generic/361 generic/388 generic/422 generic/451 generic/456 generic/459 FSTESTIMG: gce-xfstests/xfstests-201709302211 FSTESTPRJ: gce-xfstests FSTESTVER: e2fsprogs v1.43.6-85-g7595699d0 (Wed, 6 Sep 2017 22:04:14 -0400) FSTESTVER: fio fio-2.21 (Thu, 15 Jun 2017 12:25:03 -0600) FSTESTVER: quota 16f31b1 (Tue, 5 Sep 2017 16:53:11 +0200) FSTESTVER: stress-ng 977ae35 (Wed, 6 Sep 2017 23:45:03 -0400) FSTESTVER: xfsprogs v4.12.0 (Thu, 20 Jul 2017 10:57:14 -0500) FSTESTVER: xfstests-bld 0d85f98 (Sat, 30 Sep 2017 21:42:59 -0400) FSTESTVER: xfstests linux-v3.8-1693-ga71d59bc (Fri, 29 Sep 2017 23:56:42 -0400) FSTESTSET: -g auto FSTESTOPT: aex ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: How to verify linux-next
On Sat, Sep 30, 2017 at 09:28:09AM +0530, Pintu Kumar wrote: > I need to submit a patch to mainline which should be verified against > linux-next tree with latest API. If you want to verify a patch that you intend to submit upstream, my suggestion is to *not* use linux-next, but rather use the latest tagged -rc from Linus's tree. So for example, you might want to use v4.14-rc2 as your base, and then apply your patch on top of v4.14-rc2. And then test v4.14-rc2. That way you don't need to worry about debugging problems that might be caused by code in other people's development trees. If you know which subsystem tree your commit is going to be sent to, you might use as your base the current development branch of that subsystem tree. But in general, it's fine to use something like v4.14-rc2; if the subsystem maintainer you plan to be submitting your patch has other preference, he or she will let you know, or take care of rebasing your patch onto his subsystme tree. > My patch is related to some test utility based on client/server model. > So, I need 2 terminal, one for server and one for client. That implies you're running the commands to run the test by hand. In the ideal world, tests should be automated, even those that are using client/server so that tests can be run unattended, over and over again. For example, here's an example of test involving a client and a server in xfstests: https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/tests/generic/131 See? No terminal required, and certainly not two terminals! Remember, it's important not just to run one test, because the risk is that fixing one bug might cause a test regression somewhere else. So when I "validate" a kernel, I'm running thousands of tests, just to test the ext4 file system. For each bug that we fix, we try to add a new automated test, so we can be sure that some future change doesn't cause a bug to reappear. And if you're running hundreds or thousands of tests, you certainly aren't going to be wanting to manually set up each test by using putty to login to the VM using ssh! > 1) How to resolve linux-next build error with ubuntu virtual box 5.1.28 Virtual box is not relevant. What is relevant is the kernel config file you are using, and what compiler version / distro are you using to build the kernel. And as I said, you're better off using something like v4.14-rc2 instead of linux-next. - Ted ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: How to verify linux-next
On Fri, Sep 29, 2017 at 07:56:41PM +0530, Pintu Kumar wrote: > BTW, I am more interested in my another query about QEMU arm. > This will be much quicker and easy for me. > But the problem is I wanted to use multiple ssh shell on qemu. > Also I needed a pre-built rootfs image for qemu-arm, cortex-a9 > versatilepb machine. If you want to get more useful help, it might be interesting if you were to specify exactly what kind of "verification" you are interested in doing. What sort of kernel testing are you interested in doing? What part of the kernel are you interested in testing? The fact that you are trying to test both a Ubuntu x86 box as will as a virtual ARM box makes it unclear what part of the kernel you are most intested in testing. In particular, why do you care about accessing the VM via ssh / networking? What sort of testing do you plan to do after manage to get the kernel running? And do you care what distribution you use? I have a huge amount of test automation built for testing kernel file systems. This includes building root_fs images for x86 for use with kvm[1], and arm chroots for use in testing Android systems[2]. There is also a turn-key images for running tests using the Google Cloud Platform[3], and even a Dockerfile[4] so people can run kernel tests using a private Kubernetes cluster. [1] https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-xfstests.md [2] https://thunk.org/android-xfstests [3] https://thunk.org/gce-xfstests [4] https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/Dockerfile If you don't have a file-system centric view of the world, and want to do more generalized kernel testing, the above might not be as interesting to you, although some of the utilities in the xfstests-bld git tree for setting up and building in build chroots, using debootstrap to create root_fs.img files, scripts for manipulating xUnit test result files (the XML format used by Jenkins), using 9p to communicate between the host system running qemu/kvm and the test VM, etc. The point is that if you really want to get serious about kernel testing, you should really think hard about test automation. And in that world, using networking often makes things harder, not easier. For kvm-xfstests we just do our communications using the serial port, which made it easy for us to adapt things for android-xfstests, where we comunicate test runner script via "adb shell". For gce-xfstests things _do_ get a bit more complicated, where the test summary gets e-mail'ed back to the developer, while the full set of test artifacts are archived on Google Cloud Storage. But one of the most powerful things about my setup is vast majority of the test automation code stays the same regardless of whether the kernel being tested is being run in KVM, on a physical Android hardware, or in the Cloud using GCE. > 2) Let, please point me to a pre-built qemu-arm busy box image with > full features. Define "full features". Busy box images are generally _not_ full featured. There is a reason why I use a minimal Debian system; a lot of the tests I run require bash, and modern shell utilities, and Python so I can have scripts which manipulate xUnit XML files. Nevertheless, the complete x86 test VM is still only 87 megs, which is still small enough that it doesn't cause me any problems. On the other hand, since I find networking in the test VM to be completely superfluous (and in fact, gets in the way, since a VM which is on the corporate network can be a security problem, and may run afoul of corporate I/T security policies --- and if you don't have those kinds of security policies, you really should). So my root_fs's general have no networking support whatsoever. It keeps $WORK's secops team *much* happier. :-) - Ted ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Work (really slow directory access on ext4)
I don't subscribe to kernelnewbies, but I came across this thread in the mail archive while researching an unrelated issue. Valdis' observations are on the mark here. It's almost certain that you are getting overwhelmed with other disk traffic, because your directory isn't *that* big. That being said, there are certainly issues with really really big directories, and solving this is certainly not going to be a newbie project (if it was easy to solve, it would have been addressed a long time ago). See: http://en.it-usenet.org/thread/11916/10367/ for the background. It's a little bit dated, in that we do use a 64-bit hash on 64-bit systems, but the fundamental issues are still there. If you sort the readdir files by inode order, this can help significantly. Some userspace programs, such as mutt, do this. Unfortunately ls does not. (That might be a good newbie project, since it's a userspace-only project. However, I'm pretty sure the shellutils maintainers will also react negatively if they are sent patches which don't compile. :-) A proof of concept of how this can be a win can be found here: http://git.kernel.org/cgit/fs/ext2/e2fsprogs.git/tree/contrib/spd_readdir.c LD_PRELOAD aren't guaranteed to work on all programs, so this is much more of a hack than something I'd recommend for extended production use. But it shows that if you have a readdir+stat workload, sorting by inode makes a huge difference. As far as getting traces to better understand problems, I strongly suggest that you try things like vmstat, iostat, and blktrace; system call traces like strace aren't going to get you very far. (See http://brooker.co.za/blog/2013/07/14/io-performance.html for a nice introduction to blktrace). Use the scientific method; collect baseline statistics using vmstat, iostat, sar, before you run your test workload, so you know how much I/O is going on before you start your test. If you can run your test on a quiscient system, that's a really good idea. Then collect statistics as your run your workload, and then only tweak one variable at a time, and record everything in a systematic way. Finally, if you have more problems of a technical nature with respect to the ext4, there is the ext3-us...@redhat.com list, or the developer's list at linux-e...@vger.kernel.org. It would be nice if you tried the ext3-users or the kernel-newbies or tried googling to see if anyone else has come across the problem and figured out the solution already, but if you can't figure things out any other way, do feel free to ask the linux-ext4 list. We won't bite. :-) Cheers, - Ted P.S. If you have a large number of directories which are much larger than you expect, and you don't want to do the mkdir foo.new; mv foo/* foo.new ; rmdir foo; mv foo.new foo trick on a large number of directories, you can also schedule downtime and while the file system is unmounted, use e2fsck -fD. See the man page for more details. It won't solve all of your problems, and it might not solve any of your problem, but it will probably make the performance of large directories somewhat better. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: How to automate checkpatch get_maintainers git send-email of commits range?
On Fri, Jul 18, 2014 at 05:38:30PM +0300, Andrey Utkin wrote: Is there script for automated checkpatch.pl get_maintainers.pl git send-email for range of commits? I see none. Would it be welcome to submit such one to kernel tree? Too much automation can be a really bad thing. You **really** want to sanity check the output of checkpatch.pl and get_maintainers.pl. Their output is not always correct, and some amount of human common sense is required. (Granted Nick Krause hasn't shown much in the way of common sense, but some human intervention --- assuming he is human and not an badly written, automated troll program --- is better than none.) The other thing is that I strongly recommend that people use git format-patch first, to make sure that what you will be blasting out to thousands of peoples' mail boxes is in fact sane. And then think very hard about which patches people need to see in order to be able to evaluate a patch. For example, if you have patch 1 out of a series which adds a new function, and then patches 2 through 1000 modify a thousand different drivers to use that new function, if you use an automated get_maintainers.pl script to send each patch to just the maintainer, then the device driver maintainer might not see patch #1 which is critical context to understanding the patch that you want to make to his driver. And then you will have several hundred very angry and annoyed developers wondering why you sent them patch 345/1000, with no other context, and wondering what the heck they are supposed to do with the email that you have just launched into their inbox. There's a reason why many developers cordially hate these scripts; it's too easy to misuse them, and unfortunately, there are too many Nick Krause like idiots out there who mindlessly use such scripts and cause pain to maintainers. The critical step is to force such idiots to stop and ***think*** and unfortunately, automation seems to encourage the opposite behaviour. Regards, - Ted ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
