RE: Questions for Storage Engineers
>This is a favor for the btrfs developers, one of my aunt's work's at >IBM and is able to ask questions >to a few storage engineers about features they would like to see in a >file system and it's tools. In Really ? I'm speechless. Its so fantastical.. Abo ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Help with btrfs project
On Wed, Aug 20, 2014 at 8:03 AM, Nick Krause wrote: > Hey Guys, > This commit seems to be needed to fixed up based on what the btrfs > developers, are stating on their wiki. > Other then the TODO list, which we can discuss here, is there any > other parts of this patch that need to > be rewritten, it's merged as of now but I can recreate new patches if > needed from Linus's tree. The project > idea is Clear unallocated space on the btrfs wiki. I AM NOT asking you > to write the code just want to known > is how you want this cleaned up. > Cheers Nick > Hey Nick, It seems like you are an expert on btrfs . Can you please let me know a few questions i have : 1) is btrfs better than ext4 ? if yes , how ? 2) what is the main difference in the implementation of ext4 and btrfs ? 3) I know very little about file systems , but i know ext4 uses journaling file system. will you please care to explain what is journaling file system? 4) btrfs supports online defragmentation ... how that is implemented ? Thanks Sudip PS : a simple google search will give the answers , but i will want the answers from Nick . ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Kernel Compilation Error
i TRIED TO SOLVE THIS AND I AM UNABLE TO DO IT. Totally confused. On Tue, Aug 19, 2014 at 2:54 PM, Paul Bolle wrote: > [Send this to the correct list again. What a mess...] > > On Tue, 2014-08-19 at 11:23 +0200, Paul Bolle wrote: > > [Have you've been lectured on top posting already?] > > > > On Tue, 2014-08-19 at 14:30 +0530, Niamathullah sharief wrote: > > > I didnt find the place to enable CONFIG_MODULE_SIG_ALL > > > > > > Its like > > > > > > "ifdef CONFIG_MODULE_SIG_ALL > > > MODSECKEY = ./signing_key.priv > > > MODPUBKEY = ./signing_key.x509 > > > export MODPUBKEY > > > mod_sign_cmd = perl $(srctree)/scripts/sign-file > > > $(CONFIG_MODULE_SIG_HASH) $(MODSECKEY) $(MODPUBKEY) > > > else > > > mod_sign_cmd = true > > > endif > > > export mod_sign_cmd > > > " > > > > Personally I do > > $ git grep -nw MODULE_SIG_ALL > > init/Kconfig:1857:config MODULE_SIG_ALL > > init/Kconfig:1866: depends on MODULE_SIG_FORCE && > !MODULE_SIG_ALL > > > > But you can search for CONFIG_MODULE_SIG_ALL in the interface of "make > > menuconfig" too. > > > > (I noticed I have both signing_key.priv and signing_key.x509 in the root > > of my kernel tree. They appear to be, well, faux: grep for > > slartibartfast@magrathea.h2g2 to see what I mean. These keys are months > > old, and survived numerous invocations of "make clean". I don't remember > > which command generated them, sorry.) > > > > > > Paul Bolle > > > ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
a question on linux header extraction
Hello, I'm using linux3.3 and recently while building busybox for a new command, found the BB uses linux kernel headers. So I looked up the internet and did 'make headers_install ARCH=.. CROSS_COMPILE=.. INSTALL_HDR_PATH=..'. Then I used the new header files instead of files under sparc-snake-linux/sys-include. But I had to copy over some missing files from the sys-include to the new header directories and had to copy some missiong definitions from the sys-include files to the corresponing file in the new header files. Is this what I am supposed to do? (why are there some missing files? Doesn't it work well for versions later than 2.6?) And do I have to build the glibc again with this new header files? (I'm using glibc for busybox) any help would be deeply appreciated. Thanks Chan ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Help with btrfs project
Hey Guys,
This commit seems to be needed to fixed up based on what the btrfs
developers, are stating on their wiki.
Other then the TODO list, which we can discuss here, is there any
other parts of this patch that need to
be rewritten, it's merged as of now but I can recreate new patches if
needed from Linus's tree. The project
idea is Clear unallocated space on the btrfs wiki. I AM NOT asking you
to write the code just want to known
is how you want this cleaned up.
Cheers Nick
>From 55ec5c00022be8f2bbed06b99b5f4be5832a5451 Mon Sep 17 00:00:00 2001
From: David Sterba
Date: Thu, 19 Apr 2012 15:09:09 +0200
Subject: [PATCH 1/1] btrfs: ioctl to clear unused space
abuse discard infrastructure and pass down whether we want to trim or
just zero out. works, does not corrupt data.
TODO:
- more options how to clean (ie. a known signature of freed block, 2
types of them, or random garbage, or all zeros)
- what to do for SSD mixed with HDD ?
Signed-off-by: David Sterba
---
fs/btrfs/ctree.h|5 ++-
fs/btrfs/disk-io.c |2 +-
fs/btrfs/extent-tree.c | 86 +--
fs/btrfs/free-space-cache.c | 24 +++
fs/btrfs/free-space-cache.h |3 +-
fs/btrfs/ioctl.c| 29 ++
fs/btrfs/ioctl.h|9
7 files changed, 134 insertions(+), 24 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index fa5c45b..e7a87cb 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2655,7 +2655,10 @@ u64
btrfs_account_ro_block_groups_free_space(struct btrfs_space_info
*sinfo);
int btrfs_error_unpin_extent_range(struct btrfs_root *root,
u64 start, u64 end);
int btrfs_error_discard_extent(struct btrfs_root *root, u64 bytenr,
- u64 num_bytes, u64 *actual_bytes);
+ u64 num_bytes, u64 *actual_bytes,
+ int do_trim);
+int btrfs_clear_free_space(struct btrfs_root *root,
+ struct btrfs_ioctl_clear_free_args *range);
int btrfs_force_chunk_alloc(struct btrfs_trans_handle *trans,
struct btrfs_root *root, u64 type);
int btrfs_trim_fs(struct btrfs_root *root, struct fstrim_range *range);
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 2936ca4..77a704d 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3580,7 +3580,7 @@ again:
if (btrfs_test_opt(root, DISCARD))
ret = btrfs_error_discard_extent(root, start,
end + 1 - start,
- NULL);
+ NULL, 1);
clear_extent_dirty(unpin, start, end, GFP_NOFS);
btrfs_error_unpin_extent_range(root, start, end);
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 6e1d367..26514b9 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -1818,35 +1818,50 @@ static int remove_extent_backref(struct
btrfs_trans_handle *trans,
}
static int btrfs_issue_discard(struct block_device *bdev,
- u64 start, u64 len)
+ u64 start, u64 len, int do_trim)
{
- return blkdev_issue_discard(bdev, start >> 9, len >> 9, GFP_NOFS, 0);
+ printk(KERN_DEBUG "ISSUE ZERO start %llu len %llu\n",
+ (unsigned long long)start, (unsigned long long)len);
+ if (do_trim) {
+ return blkdev_issue_discard(bdev, start >> 9, len >> 9,
+ GFP_NOFS, 0);
+ } else {
+ return blkdev_issue_zeroout(bdev, start >> 9, len >> 9,
+ GFP_NOFS);
+ }
}
static int btrfs_discard_extent(struct btrfs_root *root, u64 bytenr,
- u64 num_bytes, u64 *actual_bytes)
+ u64 num_bytes, u64 *actual_bytes, int do_trim)
{
int ret;
u64 discarded_bytes = 0;
struct btrfs_bio *bbio = NULL;
+ int rw;
+
+ printk(KERN_DEBUG "discard extent %d: bytenr %llu len %llu\n", do_trim,
+ (unsigned long long)bytenr,
+ (unsigned long long)num_bytes);
+ rw = do_trim ? REQ_DISCARD : REQ_WRITE;
+ rw = REQ_DISCARD;
/* Tell the block device(s) that the sectors can be discarded */
- ret = btrfs_map_block(&root->fs_info->mapping_tree, REQ_DISCARD,
+ ret = btrfs_map_block(&root->fs_info->mapping_tree, rw,
bytenr, &num_bytes, &bbio, 0);
/* Error condition is -ENOMEM */
if (!ret) {
struct btrfs_bio_stripe *stripe = bbio->stripes;
int i;
-
for (i = 0; i < bbio->num_stripes; i++, stripe++) {
- if (!stripe->dev->can_discard)
+ if (do_trim && !stripe->dev->can_discard)
continue;
ret = btrfs_issue_discard(stripe->dev->bdev,
stripe->physical,
- stripe->length);
+ stripe->length,
+ do_trim);
if (!ret)
discarded_bytes += stripe->length;
else if (ret != -EOPNOTSUPP)
@@ -4924,7 +4939,7 @@ int btrfs_finish_extent_commit(struct
btrfs_trans_handle *trans,
if (btrfs_test_opt(root, DISCARD))
ret = btrfs_discard_extent(root, start,
- end + 1 - start, NULL);
+ end + 1 - start, NULL, 1);
clear_extent_dirty(unpin, start, end, GFP_NOFS);
unpin_extent_range(root, start, end);
@@ -5905,7 +5920,7 @@ static int __btrfs_free_reserved_extent(struct
btrfs_root *root,
}
if (btrfs_test_opt(root, DISCARD))
- ret = btrfs_discard_extent(root, start, len, NULL);
+ ret = btrfs_discard_extent(root, start, len, NULL, 1);
if (pin)
pin_down_extent(root, cache, start, le
Re: Questions for Storage Engineers
On Tue, Aug 19, 2014 at 3:32 PM, wrote: > On Tue, 19 Aug 2014 14:05:25 -0400, Nick Krause said: >> I can get the information :). I been banned from the list so I can't >> get them involved directly and it's better > > You *do* realize that the btrfs maintainers haven't been banned from > anyplace, and have enough name recognition that if they want info, they > can go talk to people themselves, right? > > In other words, what's the value-added that you're providing here? > > Oh, and if IBM wants a feature in Linux, they in general don't make a mention > of what feature they want - they *submit a patch*, properly tested, with > benchmarks, documentation, and matching userspace tool patches. > > > > > Very well then I guess this is closed then. Nick ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions for Storage Engineers
On Tue, 19 Aug 2014 14:05:25 -0400, Nick Krause said: > I can get the information :). I been banned from the list so I can't > get them involved directly and it's better You *do* realize that the btrfs maintainers haven't been banned from anyplace, and have enough name recognition that if they want info, they can go talk to people themselves, right? In other words, what's the value-added that you're providing here? Oh, and if IBM wants a feature in Linux, they in general don't make a mention of what feature they want - they *submit a patch*, properly tested, with benchmarks, documentation, and matching userspace tool patches. pgpeLeSs0ZnTO.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions for Storage Engineers
I agree with Valdis. This is how it works. On Wed, Aug 20, 2014 at 12:52 AM, wrote: > On Tue, 19 Aug 2014 23:49:48 +0530, Saket Sinha said: >> Each filesystem has its own use-case. Like XFS is for big data, btrfs >> with its COW and other features meets some specific use-cases like >> facebook where they have a scenario(refer >> http://lwn.net/Articles/591780/ ) where btrfs suites then perfectly. > > Right. The point I was making is that although btrfs has some *really* > nice use-cases, they are by and large places where IBM is *not* going > to engage one of their storage engineers. A "use btrfs" recommendation > is going to come from a different engineering group, and a "here's > how to tune btrfs" is probably going to come from somebody in their > Professional Services (at $200/hour and up). ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions for Storage Engineers
On Tue, 19 Aug 2014 23:49:48 +0530, Saket Sinha said: > Each filesystem has its own use-case. Like XFS is for big data, btrfs > with its COW and other features meets some specific use-cases like > facebook where they have a scenario(refer > http://lwn.net/Articles/591780/ ) where btrfs suites then perfectly. Right. The point I was making is that although btrfs has some *really* nice use-cases, they are by and large places where IBM is *not* going to engage one of their storage engineers. A "use btrfs" recommendation is going to come from a different engineering group, and a "here's how to tune btrfs" is probably going to come from somebody in their Professional Services (at $200/hour and up). pgp3Ytdud7WlL.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions for Storage Engineers
Each filesystem has its own use-case. Like XFS is for big data, btrfs with its COW and other features meets some specific use-cases like facebook where they have a scenario(refer http://lwn.net/Articles/591780/ ) where btrfs suites then perfectly. So first research, google and then post suchh questions directly on the mailing list. Regards, Saket Sinha On Tue, Aug 19, 2014 at 11:35 PM, Nick Krause wrote: > On Tue, Aug 19, 2014 at 1:58 PM, wrote: >> On Tue, 19 Aug 2014 13:33:22 -0400, Nick Krause said: >>> This is a favor for the btrfs developers, one of my aunt's work's at >>> IBM and is able to ask questions >>> to a few storage engineers about features they would like to see in a >>> file system and it's tools. In >>> addition if you would like to ask them about how to improve btrfs too, >>> I can send another email >>> to her with your questions. >> >> The two most obvious questions are: >> >> 1) Why do you think that IBM storage engineers have any interesting comments >> about btrfs? Hint: if IBM is including a storage engineer in the discussion >> of a project, it's probably not a good usage case for btrfs. Storage >> engineers >> get involved when the right answer is Lustre, or IBM's GPFS, or SGI's CXFS, >> or large-scale offerings from EMC or NetApp, or similar over-the-top >> solutions. >> >> (Full disclosure: I work with those sorts of storage systems for a living, >> and >> everything mentioned in the previous paragraph is on the floor across the >> hall) >> >> 2) Why aren't the btrfs developers doing the talking themselves? > That's fair Valdis, the interesting thing through is that btrfs is > their chosen file system internally at least and > they seem to be moving their clients off ext4 to btrfs. The client's > who are using btrfs will be more useful if > I can get the information :). I been banned from the list so I can't > get them involved directly and it's better > I try to get the information from her them you guys as I have known > her for longer. > Cheers Nick > > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions for Storage Engineers
On Tue, Aug 19, 2014 at 1:58 PM, wrote: > On Tue, 19 Aug 2014 13:33:22 -0400, Nick Krause said: >> This is a favor for the btrfs developers, one of my aunt's work's at >> IBM and is able to ask questions >> to a few storage engineers about features they would like to see in a >> file system and it's tools. In >> addition if you would like to ask them about how to improve btrfs too, >> I can send another email >> to her with your questions. > > The two most obvious questions are: > > 1) Why do you think that IBM storage engineers have any interesting comments > about btrfs? Hint: if IBM is including a storage engineer in the discussion > of a project, it's probably not a good usage case for btrfs. Storage > engineers > get involved when the right answer is Lustre, or IBM's GPFS, or SGI's CXFS, > or large-scale offerings from EMC or NetApp, or similar over-the-top > solutions. > > (Full disclosure: I work with those sorts of storage systems for a living, and > everything mentioned in the previous paragraph is on the floor across the > hall) > > 2) Why aren't the btrfs developers doing the talking themselves? That's fair Valdis, the interesting thing through is that btrfs is their chosen file system internally at least and they seem to be moving their clients off ext4 to btrfs. The client's who are using btrfs will be more useful if I can get the information :). I been banned from the list so I can't get them involved directly and it's better I try to get the information from her them you guys as I have known her for longer. Cheers Nick ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Questions for Storage Engineers
On Tue, 19 Aug 2014 13:33:22 -0400, Nick Krause said: > This is a favor for the btrfs developers, one of my aunt's work's at > IBM and is able to ask questions > to a few storage engineers about features they would like to see in a > file system and it's tools. In > addition if you would like to ask them about how to improve btrfs too, > I can send another email > to her with your questions. The two most obvious questions are: 1) Why do you think that IBM storage engineers have any interesting comments about btrfs? Hint: if IBM is including a storage engineer in the discussion of a project, it's probably not a good usage case for btrfs. Storage engineers get involved when the right answer is Lustre, or IBM's GPFS, or SGI's CXFS, or large-scale offerings from EMC or NetApp, or similar over-the-top solutions. (Full disclosure: I work with those sorts of storage systems for a living, and everything mentioned in the previous paragraph is on the floor across the hall) 2) Why aren't the btrfs developers doing the talking themselves? pgp7lgCmtCUcn.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Questions for Storage Engineers
Hey Guys, This is a favor for the btrfs developers, one of my aunt's work's at IBM and is able to ask questions to a few storage engineers about features they would like to see in a file system and it's tools. In addition if you would like to ask them about how to improve btrfs too, I can send another email to her with your questions. Regards Nick ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: How to mmap kernel memory area to userspace ?
On Tue, 19 Aug 2014 17:54:14 +0800, lxgeek said: >I want to mmap a kernel memory area which sk_buf->data pointer > into userspace. I want to do this , because this way can reduce a copy > from kernel to userspace. >How to fix it? Or, which book or project can help me ? Look at how other drivers do zero-copy. Hint: mmap is probably not the best way to approach it, because that would require userspace to do something different to use it. pgpVYdeBOFUh4.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
RE: How to mmap kernel memory area to userspace ?
> -Original Message- > From: kernelnewbies-boun...@kernelnewbies.org > [mailto:kernelnewbies-boun...@kernelnewbies.org] On Behalf Of lxgeek > Sent: Tuesday, August 19, 2014 2:54 AM > To: kernelnewbies > Subject: How to mmap kernel memory area to userspace ? > > hi all: >I want to mmap a kernel memory area which sk_buf->data pointer > into userspace. I want to do this , because this way can reduce a copy > from kernel to userspace. >How to fix it? Or, which book or project can help me ? > > Thank you. One approach that I had some limited success with recently as part of an investigative exercise was to mmap /dev/kmem in the context of the process you want to access the skb data in. You'll need to figure out the block of kernel virtual addresses to map based on your target architecture, but basically it needs to cover the kmem cache that the TCP/IP stack allocates skbs from. Then you need to get the virtual address of the skbs' data up to user space, somehow (for instance create a device that registers hooks with netfilter, steals and queues the skbs that get passed to it and then make the virtual addresses in the queued skbs available to be read from the device). Then it's just a matter of some pointer arithmetic to determine the corresponding address of the skb payload in the pointer you get back from the call to mmap(). I am aware of a couple of potential obstacles to contend with here depending on your kernel and architecture: CONFIG_STRICT_DEVMEM - If this kernel config is enabled in your kernel, you won't be able to do this. It's a security thing. You'd need to turn this off to proceed down this path. CONFIG_X86_PAT - If your target is X86_64, then this kernel config will prevent you from mmap()'ing /dev/kmem, too. It's not clear to me what the effect of turning this config off is, I'm no expert in X86_64 memory management. When I tried it, the resultant kernel booted ok and seemed to function on our X86_64 target systems, but just doing read access of the data from the mmap()'ed region did occasionally generate error messages from the kernel and eventual soft lockups. For me, it worked well enough to complete the investigation but we didn't try to productize this. Good luck, what you are trying to do is not easy to get right in my experience, Jeff Haran ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: one thing at a time
On Tue, Aug 19, 2014 at 8:13 AM, Greg Freemyer wrote: > > > On August 18, 2014 10:59:05 PM EDT, Nick Krause wrote: >>On Mon, Aug 18, 2014 at 10:55 PM, Nick Krause >>wrote: >>> On Mon, Aug 18, 2014 at 10:06 PM, Rik van Riel >>wrote: On 08/18/2014 01:13 PM, Nick Krause wrote: > Valdis, > I was interested in both at the same time, just asked about >>Beagle-boards first. > I aren't having any difficulty with it , I just wanted to known >>more about this > area as the docs out there are terrible and not worth reading on >>this part of > the networking stack. Nick, the Linux kernel requires a lot of focus. I have no doubt that you are smart enough to learn things about the Linux kernel, but nobody is smart enough to learn everything at once. Nobody. If you are serious about learning Linux kernel internals, it would be good to focus on one thing at a time. One thing. Learn everything about it, before moving on to the next topic. Kernel hacking is much more about attitude than about aptitude. Focus is the key. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies >>> Absolutely, I am trying to learn btrfs and networking for now. >>> Cheers Nick >>My issue now is that due to my issues I can't mail the right list for >>being >>banned for my questions. If someone wants to talk to the other >>developers >>about removing the ban that would be great and very helpful as I can my >>questions to the right maintainers/developers. >>Cheers Nick > > I checked my watch, then my sundial, and finally my calendar. All said check > back after beard has grown 6 additional inches. > > I have no idea if you will ever be un-banned, but it certainly not going to > happen in 2014. I doubt seriously it will happen in 2015. > > You do realize that most of the kernel devs do it both for a living and out > of sheer love of the work. Your participation the last few months impacted > both in a negative way. I've never heard of anyone being banned before, so > you are novel. None of us how long the blackball will stay in place, maybe > forever. > > Greg > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. That's understandable and I hope to get back on, I did screw up and yes I understand your points. Cheers Nick ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: fd type from number
Sure, here's some more context. I'm one of the developers of sysdig (www.sysdig.org), a tool that captures system calls and uses them to offer advanced system monitoring. One of the features that our diver offers is the tcpdump-derived concept of "snaplen": when a system call with a buffer is captured, it's possible to choose how many bytes of that buffer are copied to the driver capture buffer. This makes it possible to tune buffer utilization and CPU usage vs completeness of data. Since this feature is important and heavily used, I'd like to extend it so that the user has per-fd-type snaplen control. A typical use case is: "I want 1000 bytes of each socket buffer, because I'm interested in looking at protocol activity, but I don't care about files and so I'm ok with just 20 bytes from them". In order for this feature to be useful, it needs to be very fast: we use tracepoints to capture system calls, so we slow down the original process if we take too long. And since I'm here, let me expand my question. Another useful thing to do would be per-filename snaplen. Use case: "I want the whole content of reads and writes to files that are in /etc, but I want only 20 bytes from any other system call". This would I guess involve unpacking the file structure and retrieving the full file name. Is there any way to do it safely and efficiently? Thanks, Loris On 8/19/2014 9:02 AM, valdis.kletni...@vt.edu wrote: > On Tue, 19 Aug 2014 08:38:24 -0700, Loris Degioanni said: > >> I'm looking for an efficient way to determine the type of an fd (file, >> socket...) given its number, from a kernel module. > What problem are you trying to solve here? There may be a better API for > your problem. So step back - what are you trying to accomplish? ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: fd type from number
On 8/19/2014 9:10 AM, Greg KH wrote: > On Tue, Aug 19, 2014 at 08:38:24AM -0700, Loris Degioanni wrote: >> (resending making sure this is not part of another thread) >> >> I'm looking for an efficient way to determine the type of an fd (file, >> socket...) given its number, from a kernel module. > You don't have a "number" from within the kernel, you have a pointer to > the full file descriptor structure, right? If so, what is missing from > that structure that you can not find? I do have the FD number, because it's coming from intercepting a system call. > >> The closest thing I found by looking at the kernel sources is >> sockfd_lookup(), which works but is limited to telling me if the fd is a >> socket or not. >> >> Is there something else I can look at? > What exactly are you trying to do with this? And what does it matter > what "type" of thing a file descriptor is? What can you do with that > information? I just sent another email with some explanation. Loris > > thanks, > > greg k-h ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: fd type from number
On Tue, Aug 19, 2014 at 08:38:24AM -0700, Loris Degioanni wrote: > (resending making sure this is not part of another thread) > > I'm looking for an efficient way to determine the type of an fd (file, > socket...) given its number, from a kernel module. You don't have a "number" from within the kernel, you have a pointer to the full file descriptor structure, right? If so, what is missing from that structure that you can not find? > The closest thing I found by looking at the kernel sources is > sockfd_lookup(), which works but is limited to telling me if the fd is a > socket or not. > > Is there something else I can look at? What exactly are you trying to do with this? And what does it matter what "type" of thing a file descriptor is? What can you do with that information? thanks, greg k-h ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: fd type from number
On Tue, 19 Aug 2014 08:38:24 -0700, Loris Degioanni said: > I'm looking for an efficient way to determine the type of an fd (file, > socket...) given its number, from a kernel module. What problem are you trying to solve here? There may be a better API for your problem. So step back - what are you trying to accomplish? pgp6vTYA0dAtj.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: fd type from number
On Tue, Aug 19, 2014 at 6:38 PM, Loris Degioanni wrote: > (resending making sure this is not part of another thread) > > I'm looking for an efficient way to determine the type of an fd (file, > socket...) given its number, from a kernel module. > The closest thing I found by looking at the kernel sources is > sockfd_lookup(), which works but is limited to telling me if the fd is a > socket or not. > > Is there something else I can look at? https://www.kernel.org/doc/Documentation/filesystems/files.txt See point 4. Daniel. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
fd type from number
(resending making sure this is not part of another thread) I'm looking for an efficient way to determine the type of an fd (file, socket...) given its number, from a kernel module. The closest thing I found by looking at the kernel sources is sockfd_lookup(), which works but is limited to telling me if the fd is a socket or not. Is there something else I can look at? Thank you in advance, Loris ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Linked List versus Hashed Linked iIst
On Tue, 19 Aug 2014 07:45:48 -0400, Greg Freemyer said: > You can take that excellent code into your user space app and test it to your > heart's content. Not only can you do that, for something like a linked list > evaluation, you should do that. You have implied "tested" code is code that > compiles. If a developer wanted to replace the hashed linked link > implementation it would be expected that they had done significant testing of > the new code in user space with highly varied loads to show what they work > well > on and when the new code performs less well. And in today's data structures lesson: 1) How is the kernel's linked-list implementation different than what they usually teach in Data Structures 101? 2) Why do we do this? (Consider it a Eudyptula-type challenge - figure it out *for yourself* rather than spamming the list. ;) pgpIZDG2MHbRf.pgp Description: PGP signature ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: one thing at a time
On August 18, 2014 10:59:05 PM EDT, Nick Krause wrote: >On Mon, Aug 18, 2014 at 10:55 PM, Nick Krause >wrote: >> On Mon, Aug 18, 2014 at 10:06 PM, Rik van Riel >wrote: >>> On 08/18/2014 01:13 PM, Nick Krause wrote: Valdis, I was interested in both at the same time, just asked about >Beagle-boards first. I aren't having any difficulty with it , I just wanted to known >more about this area as the docs out there are terrible and not worth reading on >this part of the networking stack. >>> Nick, >>> >>> the Linux kernel requires a lot of focus. >>> >>> I have no doubt that you are smart enough to learn >>> things about the Linux kernel, but nobody is smart >>> enough to learn everything at once. Nobody. >>> >>> If you are serious about learning Linux kernel internals, >>> it would be good to focus on one thing at a time. >>> >>> One thing. Learn everything about it, before moving on >>> to the next topic. >>> >>> Kernel hacking is much more about attitude than about >>> aptitude. Focus is the key. >>> >>> ___ >>> Kernelnewbies mailing list >>> Kernelnewbies@kernelnewbies.org >>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies >> Absolutely, I am trying to learn btrfs and networking for now. >> Cheers Nick >My issue now is that due to my issues I can't mail the right list for >being >banned for my questions. If someone wants to talk to the other >developers >about removing the ban that would be great and very helpful as I can my >questions to the right maintainers/developers. >Cheers Nick I checked my watch, then my sundial, and finally my calendar. All said check back after beard has grown 6 additional inches. I have no idea if you will ever be un-banned, but it certainly not going to happen in 2014. I doubt seriously it will happen in 2015. You do realize that most of the kernel devs do it both for a living and out of sheer love of the work. Your participation the last few months impacted both in a negative way. I've never heard of anyone being banned before, so you are novel. None of us how long the blackball will stay in place, maybe forever. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: fd type from number
On August 18, 2014 8:46:16 PM EDT, Loris Degioanni wrote: >I'm looking for an efficient way to determine the type of an fd (file, >socket...) given its number, from a kernel module. >The closest thing I found by looking at the kernel sources is >sockfd_lookup(), which works but is limited to telling me if the fd is >a >socket or not. > >Is there something else I can look at? > >Thank you in advance. Loris, Apparently you replied to a different email then changed the subject. Email clients that thread conversations based on "in-reply-to" will lump your supposedly new thread in with the one you replied to. That guarantees a reduced audience of readers, especially considering the email you replied to. You should repost by composing a brand new email. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Linked List versus Hashed Linked iIst
On August 18, 2014 10:53:12 PM EDT, Nick Krause wrote: >On Mon, Aug 18, 2014 at 7:01 PM, Greg Freemyer > wrote: >> On Mon, Aug 18, 2014 at 4:20 PM, Nick Krause >wrote: >>> What are the advantages of the hashed linked list version over the >>> standard one and does it >>> increase the memory usage and overhead of the linked list more if I >>> use a hashed version? >> >> Seriously? Do you know what a hash is? >> >> A hash is a well-defined many to one algorithm. >> >> If I have a universe of a million items that hash down to 100 unique >> hashes, then I can group those million items by hash and have 100 >> groups of roughly 10,000 items each. >> >> The better the hashing algorithm versus my original universe of 1 >> million items, the more even the distribution. >> >> Now that I have 100 segregated groups I can build an array of 100 >> linked lists all maintained separately. >> >> Thus: >> >> hash_index = my_hash(item) >> >> add_item(linked_list[hash_item], item) is how I add my item to the >> hashed linked list. >> >> is_in_list(linked_list[hash_item], item) is how I check to see if my >> item is already in the list. >> >> So in my example I have to have 100 linked lists, but each list is on >> average 100x smaller than a simple linked list would be. >> >> Is adding an item to the hashed linked list faster? >> >> Absolutely not, I have to hash the item first then do a normal linked >> list insertion. That will always be slower. >> >> Is finding the item faster? >> >> That is the whole point of the exercise. The theory is you ONLY use >a >> hashed linked list if the overhead of hashing the item is less than >> the amount of time saved by traversing shorter lists when you search. >> >> It is the job of the programmer to make the determination if a hashed >> list is a better choice or not on a case by case basis. It depends >on >> the length of the list without breaking it into pieces and how well >> the hash algorithm can do at generating roughly similar segregated >> groups. >> >> For the size question, write yourself a userspace app and test it. >> Obviously that is more work than asking here, but it is ASSUMED you >> are doing research on your OWN before you post questions here. >> >> fyi: this question has little to do with the linux kernel. It is >part >> of what people mean when they say you need to go learn c before you >> start on the kernel. Using linked lists and hashed linked lists is >> stuff you can fully explore in userspace. >> >> Greg >No I known what the advantages are for user space was wondering if >there were any issues that differ in >kernel space. >Nick 1) your original question needs either a highly generic answer like I gave, or a highly specific one that depends on the exact nature of the data, the number of the items tracked, the ratio of searches vs. adds, and how smooth the hash grouping is. Since you didn't provide the exact use case, only the generic answer is possible. In fact your question implies that the answer is relatively straight forward. A much better question would have been "for a specific use case, how is the choice of a normal linked list vs a hashed linked list performed?" Note the answer to that has nothing to do with user space vs kernel space. 2) The kernel is not a magic place. Sure there are issues like locking and interrupts that make the kernel more complex than user space, but for data algorithms it is just that the quality of the code is pretty universally excellent. It is excellent because it has been open for 20+ years and some great developers have worked on it during that time. Poorly written code in any of the core areas was eradicated long ago. You can take that excellent code into your user space app and test it to your heart's content. Not only can you do that, for something like a linked list evaluation, you should do that. You have implied "tested" code is code that compiles. If a developer wanted to replace the hashed linked link implementation it would be expected that they had done significant testing of the new code in user space with highly varied loads to show what they work well on and when the new code performs less well. Then do an analysis of the existing kernel data structures which use hashed linked lists and prove that the new method is an improvement for the actual kernel use cases. It would be months of work, but that is what it takes if you actually want to improve the kernel in a meaningful way. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
How to mmap kernel memory area to userspace ?
hi all: I want to mmap a kernel memory area which sk_buf->data pointer into userspace. I want to do this , because this way can reduce a copy from kernel to userspace. How to fix it? Or, which book or project can help me ? Thank you. ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Kernel Compilation Error
[Send this to the correct list again. What a mess...] On Tue, 2014-08-19 at 11:23 +0200, Paul Bolle wrote: > [Have you've been lectured on top posting already?] > > On Tue, 2014-08-19 at 14:30 +0530, Niamathullah sharief wrote: > > I didnt find the place to enable CONFIG_MODULE_SIG_ALL > > > > Its like > > > > "ifdef CONFIG_MODULE_SIG_ALL > > MODSECKEY = ./signing_key.priv > > MODPUBKEY = ./signing_key.x509 > > export MODPUBKEY > > mod_sign_cmd = perl $(srctree)/scripts/sign-file > > $(CONFIG_MODULE_SIG_HASH) $(MODSECKEY) $(MODPUBKEY) > > else > > mod_sign_cmd = true > > endif > > export mod_sign_cmd > > " > > Personally I do > $ git grep -nw MODULE_SIG_ALL > init/Kconfig:1857:config MODULE_SIG_ALL > init/Kconfig:1866: depends on MODULE_SIG_FORCE && !MODULE_SIG_ALL > > But you can search for CONFIG_MODULE_SIG_ALL in the interface of "make > menuconfig" too. > > (I noticed I have both signing_key.priv and signing_key.x509 in the root > of my kernel tree. They appear to be, well, faux: grep for > slartibartfast@magrathea.h2g2 to see what I mean. These keys are months > old, and survived numerous invocations of "make clean". I don't remember > which command generated them, sorry.) > > > Paul Bolle ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Kernel Compilation Error
[Send this to the correct list. Does kernelnewb...@nl.linux.org even exist?] On Tue, 2014-08-19 at 09:48 +0200, Paul Bolle wrote: > [fixed top posting.] > > On Tue, 2014-08-19 at 12:51 +0530, Pramod Gurav wrote: > > On Tue, Aug 19, 2014 at 12:03 PM, Niamathullah sharief > > wrote: > > > when i give "make modules_install" command i am getting the following > > > error > > > > > > [root@localhost linux-3.15.1]# make modules_install > > > INSTALL arch/x86/crypto/crc32-pclmul.ko > > > Can't read private key > > > make[1]: *** [arch/x86/crypto/crc32-pclmul.ko] Error 2 > > > make: *** [_modinst_] Error 2 > > > > > It would always be better if you can provide the details of things you > > are doing and steps you have carried out until it failed. > > But if I can guess it right you are trying to upgrade kernel on local > > machine which is x86 based. The failure seems while trying to install > > newly compiled modules from new kernel(3.15). > > > > I am not sure if your compilation has gone well but I think you should > > be root to install kernel and modules. > > Try to install the kernel and moudules as root as it installs them in > > path which is not accessible to user(under /). > > In Niamathullah Sharief's message I see > [root@localhost linux-3.15.1]# make modules_install > > So it appears make modules_install was run as root (sudo is not > installed on that box?). > > By the way, I've never ran into this error. So I did > $ git grep -n "Can't read private key" > scripts/sign-file:38:die "Can't read private key\n" if (!$signature_file > && !-r $private_key); > > If I'd ran into this error myself I'd start with looking at > scripts/sign-file and the place(s) where that script is called. > > Hope this helps. > > > Paul Bolle ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Re: Kernel Compilation Error
Hi Sharief, It would always be better if you can provide the details of things you are doing and steps you have carried out until it failed. But if I can guess it right you are trying to upgrade kernel on local machine which is x86 based. The failure seems while trying to install newly compiled modules from new kernel(3.15). I am not sure if your compilation has gone well but I think you should be root to install kernel and modules. Try to install the kernel and moudules as root as it installs them in path which is not accessible to user(under /). On Tue, Aug 19, 2014 at 12:03 PM, Niamathullah sharief wrote: > Dear Friends > > Please advice me what to do for the below error which i am getting while > install modules > > when i give "make modules_install" command i am getting the following error > > > [root@localhost linux-3.15.1]# make modules_install > INSTALL arch/x86/crypto/crc32-pclmul.ko > Can't read private key > make[1]: *** [arch/x86/crypto/crc32-pclmul.ko] Error 2 > make: *** [_modinst_] Error 2 > > > Please help me > > Regards > Sharief > Kernel Newbie > > ___ > Kernelnewbies mailing list > Kernelnewbies@kernelnewbies.org > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies > -- Thanks and Regards Pramod ___ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
