Re: [SLUG] linked folders show different files
Ben Donohue donoh...@icafe.com.au writes: I really don't understand what is going on here... I have a folder structure that is replicated several times by linking the base folder A hard-link, or a soft-link? I am guessing the later, since you can't hard-link a directory, but it isn't entirely clear. If you mean cloned by hard-linking all the files then the answer is rather different to what I am about to give you.[1] Some folders down within the master folder I have two files called 1.jpg (a picture of a computer mouse), and 2.jpg (a picture of a computer) When I look at the linked folders via konqueror and with thumbnails on, I see the files correctly. When I delete 1.jpg and then rename 2.jpg to 1.jpg, some folders show a picture of the mouse, others show the picture of the computer. What's going on here? It could be that you are seeing a cache effect, where the system has a copy of the file in either an HTTP cache (if you are accessing this via the web) or a local icon cache. Everything should be a picture of a computer. Why are the linked folders/files not updating with the changes of the file name? Assuming they are soft-linked directories, so the change actually happens to the one single real directory, cache effects. (browsing to the linux box via a windows computer show the correct computer image in all the linked structure) I've turned off caching in konqueror. Maybe it needs more of a kick to do the right thing? The reason I'm asking is that I'm importing the images into a joomla site and the images are now incorrect. I've traced it so far to the linking I think... Given your description, it sounds suspiciously like HTTP caching, but it is hard to be sure. Does the access log for Joomla show that the image in being downloaded by the client, not just a not modified response? Does Joomla maintain an internal cache of, say, rescaled images that might need to be flushed? Daniel Footnotes: [1] Specifically, when you delete a file with hard links, you only remove that one name associated with it. The other links are unaffected. -- ✣ Daniel Pittman✉ dan...@rimspace.net☎ +61 401 155 707 ♽ made with 100 percent post-consumer electrons -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] linked folders show different files
Thanks Daniel, I'm embarassed to say that I don't know whether the files/folders are hard or soft linked! How to tell? I'm using a bit of software that does all that structure setup for me and I have not delved in to see how it sets these folders up. I'm remote desktop (VNC) to the linux box and viewing the images via konqueror. So there should not be any http caching happening. I've turned off caching in Konqueror also. I'm trying to simplify what I'm doing to try to find out why this is happening. But it seems that Joomla is picking up the cached copy when an image is uploaded rather than the changed file. If I start in a different area with a new file it works correctly. It's only when an existing image is overwritten with the same name that the old image shows up in Joomla but ALSO in Konqueror. Which makes me think it's not Joomla but something in Linux underneath somewhere. Really bizarre! Anyway still trying to home in on where things are going nuts. Ben On 1/07/2010 4:04 PM, Daniel Pittman wrote: Ben Donohuedonoh...@icafe.com.au writes: I really don't understand what is going on here... I have a folder structure that is replicated several times by linking the base folder A hard-link, or a soft-link? I am guessing the later, since you can't hard-link a directory, but it isn't entirely clear. If you mean cloned by hard-linking all the files then the answer is rather different to what I am about to give you.[1] Some folders down within the master folder I have two files called 1.jpg (a picture of a computer mouse), and 2.jpg (a picture of a computer) When I look at the linked folders via konqueror and with thumbnails on, I see the files correctly. When I delete 1.jpg and then rename 2.jpg to 1.jpg, some folders show a picture of the mouse, others show the picture of the computer. What's going on here? It could be that you are seeing a cache effect, where the system has a copy of the file in either an HTTP cache (if you are accessing this via the web) or a local icon cache. Everything should be a picture of a computer. Why are the linked folders/files not updating with the changes of the file name? Assuming they are soft-linked directories, so the change actually happens to the one single real directory, cache effects. (browsing to the linux box via a windows computer show the correct computer image in all the linked structure) I've turned off caching in konqueror. Maybe it needs more of a kick to do the right thing? The reason I'm asking is that I'm importing the images into a joomla site and the images are now incorrect. I've traced it so far to the linking I think... Given your description, it sounds suspiciously like HTTP caching, but it is hard to be sure. Does the access log for Joomla show that the image in being downloaded by the client, not just a not modified response? Does Joomla maintain an internal cache of, say, rescaled images that might need to be flushed? Daniel Footnotes: [1] Specifically, when you delete a file with hard links, you only remove that one name associated with it. The other links are unaffected. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
RE: [SLUG] linked folders show different files
You can hard link a directory (historically done as root using /etc/link or /sbin/link rather than /usr/bin/ln or /bin/ln). Doing so traditionally requires (1) root; and (2) a direct call to the link system call (which is what the old /etc/link and /sbin/link did). However if this is indeed what has been done, then I can speculate as to what might have happened. I seem to recall that the Linux kernel caches the file system hierarchy, in which case there is might be a chance that the kernel cached hierarchy does not reflect what is on disk. Hard linking directories is a sufficiently unusual thing to do that it is possible such a cache was never tested against it. Regards, Troy Rollo Solicitor Parry Carroll Commercial Lawyers Direct: (02) 8257 3177 Fax: (02) 9221 1375 Switch: (02) 9221 3899 E-mail: t...@parrycarroll.com.au Web: www.parrycarroll.com.au http://www.parrycarroll.com.au Liability limited by a scheme approved under Professional Standards Legislation This message and any attachments are confidential to Parry Carroll. If you have received it my mistake, please let us know by reply and then delete it from your system. You must not copy the message, alter it or disclose its contents to anyone. Thank you. -Original Message- From: slug-boun...@slug.org.au [mailto:slug-boun...@slug.org.au] On Behalf Of Daniel Pittman Sent: Thursday, 1 July 2010 4:05 PM To: slug@slug.org.au Subject: Re: [SLUG] linked folders show different files Ben Donohue donoh...@icafe.com.au writes: I really don't understand what is going on here... I have a folder structure that is replicated several times by linking the base folder A hard-link, or a soft-link? I am guessing the later, since you can't hard-link a directory, but it isn't entirely clear. If you mean cloned by hard-linking all the files then the answer is rather different to what I am about to give you.[1] Some folders down within the master folder I have two files called 1.jpg (a picture of a computer mouse), and 2.jpg (a picture of a computer) When I look at the linked folders via konqueror and with thumbnails on, I see the files correctly. When I delete 1.jpg and then rename 2.jpg to 1.jpg, some folders show a picture of the mouse, others show the picture of the computer. What's going on here? It could be that you are seeing a cache effect, where the system has a copy of the file in either an HTTP cache (if you are accessing this via the web) or a local icon cache. Everything should be a picture of a computer. Why are the linked folders/files not updating with the changes of the file name? Assuming they are soft-linked directories, so the change actually happens to the one single real directory, cache effects. (browsing to the linux box via a windows computer show the correct computer image in all the linked structure) I've turned off caching in konqueror. Maybe it needs more of a kick to do the right thing? The reason I'm asking is that I'm importing the images into a joomla site and the images are now incorrect. I've traced it so far to the linking I think... Given your description, it sounds suspiciously like HTTP caching, but it is hard to be sure. Does the access log for Joomla show that the image in being downloaded by the client, not just a not modified response? Does Joomla maintain an internal cache of, say, rescaled images that might need to be flushed? Daniel Footnotes: [1] Specifically, when you delete a file with hard links, you only remove that one name associated with it. The other links are unaffected. -- ✣ Daniel Pittman✉ dan...@rimspace.net☎ +61 401 155 707 ♽ made with 100 percent post-consumer electrons -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html smime.p7s Description: S/MIME cryptographic signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] linked folders show different files
Ben Donohue donoh...@icafe.com.au writes: I'm embarassed to say that I don't know whether the files/folders are hard or soft linked! How to tell? OK. I am going to start from the very basics here, so please don't be annoyed if I am telling you what you already know — I figure better cover those quickly in case you don't than make you ask again. ;) In Unix there are three parts to a file: one is the raw data itself, which is uninteresting. The second is the inode, which contains things like the permissions, and a list of where the data is found, and all. The inode does *NOT* have a name. The third is a directory entry, which is a record in a directory somewhere that points to the inode, and which has a name. A file actually has zero or more directory entries pointed to the inode, which are used to keep track of it. When there are zero it can still hang around while the file is open, though, because the file handle references the *inode*, not the directory entry. Anyway, a hard-link is when you have more than one name pointing to the same inode. You can see this in the link count field, which is the second field in 'ls -l', or in 'stat filename', or whatever. A soft-link, or symlink, is more like a pointer: it is an inode that contains a path string. When you try to open it the kernel will read that path string, then refer to that object instead. So, the easy way to tell if your image is hard-linked or soft-linked is to look at it with ls: ] ls -l example[12].jpg -rw-rw-r-- 1 daniel daniel 0 Dec 7 2007 example1.jpg -rw-rw-r-- 2 daniel daniel 0 Dec 7 2007 example2.jpg In this case example1 has only one link, but example2 has two. :) If you look at a normal file (eg: added by the software, not manually deleted or changed) you should find out quickly if it has another hard-link or not. :) [...] I'm trying to simplify what I'm doing to try to find out why this is happening. But it seems that Joomla is picking up the cached copy when an image is uploaded rather than the changed file. FWIW, my guess about hard-links would be that you had something that looked like this: /foo/bar/baz.jpg = inode 12 /host/whatever/baz.jpg = inode 12 If you then replace the /foo/bar/baz.jpg *name* with another file, by deleting it and renaming something to the same name then /host/whatever/baz.jpg is not at all changed by this. Which it doesn't really sound like is happening. If I start in a different area with a new file it works correctly. It's only when an existing image is overwritten with the same name that the old image shows up in Joomla but ALSO in Konqueror. Which makes me think it's not Joomla but something in Linux underneath somewhere. Really bizarre! *nod* That is very odd. If you were actually overwriting it you *should* see that content change, at least in the current directory. :/ Does it reflect changed content if you open it in some image display tool /other/ than Konqueror there? How about if you get the md5sum of the image before and after? Daniel -- ✣ Daniel Pittman✉ dan...@rimspace.net☎ +61 401 155 707 ♽ made with 100 percent post-consumer electrons -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] linked folders show different files
Troy Rollo t...@parrycarroll.com.au writes:
You can hard link a directory (historically done as root using /etc/link or
/sbin/link rather than /usr/bin/ln or /bin/ln).
On which file system, without causing it serious errors? I can't say
categorically that nothing allows it, but it is considered to be a serious
violation of the semantics of the system by ... well, most things.
Certainly all the common file system drivers in Linux, as well as by the core
VFS; see http://lxr.linux.no/linux+v2.6.34/fs/namei.c#L2423 for vfs_link:
2445if (S_ISDIR(inode-i_mode))
2446return -EPERM;
So, you could, but only if you go back far enough that unlink('/') worked,
along with those other nasty ways to break the file system and all. :)
Doing so traditionally requires (1) root; and (2) a direct call to the
link system call (which is what the old /etc/link and /sbin/link did).
...while you used to be able to break things as root, on some platforms, it
wasn't ever considered a healthy thing to do — at least, to the best of my
knowledge on the topic.
However if this is indeed what has been done, then I can speculate as to
what might have happened. I seem to recall that the Linux kernel caches the
file system hierarchy
Nope. That isn't even close to accurate; the way data is cached varies wildly
between different systems, but it isn't especially hierarchical; file data is
usually in the page cache indexed by inode.
in which case there is might be a chance that the kernel cached hierarchy
does not reflect what is on disk.
Well, if the file system was fundamentally broken it would certainly break
caching, but usually in vastly more nasty ways than this.
Hard linking directories is a sufficiently unusual thing to do that it is
possible such a cache was never tested against it.
Actually, it is impossible in anything resembling a recent Linux kernel as
noted, so this categorically *cannot* be the cause of the issue.
Daniel
--
✣ Daniel Pittman✉ dan...@rimspace.net☎ +61 401 155 707
♽ made with 100 percent post-consumer electrons
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] linked folders show different files
Ben == Ben Donohue donoh...@icafe.com.au writes: Ben Hi all, I really don't understand what is going on here... Ben I have a folder structure that is replicated several times by Ben linking the base folder Konqueror caches its thumbnails by a hash of the pathname to the file. As the symlinks (this has to be a tree of symlinks for this to happen) haven't changed (they still have the same timestamp), but what they point to has changed, the cached thumbnail for KDE/Konqueror is now wrong. I don't know enough KDE fu to be able to know how to force Konquerer to update its cache. Peter C -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] linked folders show different files
peter == peter pe...@chubb.wattle.id.au writes: Ben == Ben Donohue donoh...@icafe.com.au writes: Ben Hi all, I really don't understand what is going on here... Ben I have a folder structure that is replicated several times by Ben linking the base folder peter I don't know enough KDE fu to be able to know how to force peter Konquerer to update its cache. You could try deleting $HOME/.thumbnails --- that'd probably work. Peter C -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] SyPy 1st July: Lightning talks
** NOTE: CHANGE OF VENUE ** RSVP: http://anyvite.com/tnrhu7by8i 6:15: Arrive and enjoy beer provided by Anchor Systems 6:30: Five minute lightning talks (open to all) - Cool things you got from pycon-au or pycon-apac, or anything else pythonic. Bring slides or just your spiel. 7:30: PJ Obriens : pycon-au debrief debrief involving beer. Further Details: We will also have slots for 5 minute lightning talks. If you think its interesting and you're a pythonista then chances are we will to. Put together a few slides and bring it along. *RSVP: Please RSVP on so we have an idea of numbers. No registration required* http://anyvite.com/tnrhu7by8i Getting There: It's a 10min walk from Town Hall station (directions http://bit.ly/dBUBb1) Anchor Systems Pty Ltd - Hosting Heavyweights Level 4, 81 York Street, Sydney NSW 2000 Go to level 4. Free free to join us at the pub instead. PJ Obriens (outside)http://bit.ly/aabuJY . There is food available If have any problems call me. --- Dylan Jay Plone Solutions Manager. www.pretaweb.com P +612 80819071 M +61421477460 skype - dylan_jaytwitter - djay75 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
