Spurious write permission error
Running GiP 2.94 on Windows. I had two downloads running simultaneously, both writing to the same directory. One download completed without any problem indicated, the other put out the following on the console at the end of the download: INFO: MP4 tagging MP4 file Started writing to temp file. Progress: ===> 99% | Finished writing to temp file. Unable to write to a directory lacking write permission.INFO: Command exit code 255 (raw code = 65280) WARNING: Failed to tag MP4 file When I have a look in the output directory, there is an output MP4 file for the download that gave the error with "-temp-22448" appended to the end of the filename. I can play the MP4 in VLC without any error. I've also noticed on some occasions that after the download has finished, both the permanent download file and the file with "-temp-x" appended are present in the output directory. Nick ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
RE: Spurious write permission error
Please see below ... www.macfh.co.uk/CEMH.html > -Original Message- > From: get_iplayer [mailto:get_iplayer-boun...@lists.infradead.org]On > Behalf Of Nick Payne > Sent: 02 May 2016 13:41 > To: get_iplayer@lists.infradead.org > Subject: Spurious write permission error > > Running GiP 2.94 on Windows. I had two downloads running > simultaneously, > both writing to the same directory. I don't suppose much, if any, thought has been given to running multiple instances of GiP simultaneously - so, for example, the download_history is in a non-standard text format, rather than an open or de facto standard multi-user database format such as SQLite - and consequently it seems very likely that both downloads were attempting to write to the same file, which may not necessarily be the one that you might guess from examining the files remaining afterwards. If you wish to run two GiP instances simultaneously, I'd suggest that they should work from different directories, but, of course, the problem with this idea is keeping the download_histories and caches in the two directories in sync, see below. Thus in general, I'd suggest only ever running one instance of GiP at a time, and always from the same machine, so that the information available to it is always up-to-date. That said, I do meet circumstances, such as Wimbledon and the Proms, where it can be useful to have two or more instances running together. Usually I do this on two seperate NASs - my version of RAID is two have two identical NASs which are synced in the early hours of the morning. Thus one NAS takes care of the usual nightly downloads, while the other grinds full time trying to keep up with, say, Wimbledon. In this situation, it's not so necessary that each knows what the other has already downloaded, but, nevertheless, I have written a Bash script for my Linux machines such as these NASs which when run on one causes it to sync various support files with the another. The script is tolerably well-documented internally, but nevertheless is probably not for the programmatically or technologically faint-hearted! However, if there is sufficient interest, I could see about making it publicly available on an unsupported, users' risk only basis. Regards ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
Until recently I have routinely run 2/3/4 instances of gip from one Linux Terminal box with no problems - open multiple tabs in one session and run separate downloads in each, sometimes all radio or TV and other times a mix. I have not encountered a conflict with the download history getting corrupt. I have always refreshed the cache in one before running multiple. I know this does not help much but it does share an experience. However, I have recently started getting download failures when running parallel downloads; one of the pair (and not always the first or second to start) will fail part way through with a time-out/not responding error. I do not think this is a download cache problem as at the point of failure neither has completed. I delete the partial and start again, it restarts without a --force so I guess the download history has not been written. Clive ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
hello I am running GiP under MacOSX and Debian Linux . I run multiple different GiP fetches from the same directory nearly everyday on both MacOSX and Linux. My multiple I mean 3 to 4 different fetches going at once from the same directory. Never have an issue. The times I have had issues is when the disk suddenly was full, which is no surprise. A few times I mistakenly attempted to fetch the same file at the same time with multiple GiP which all failed as they should. These were generally caused by being to quick on the command recall on Linux -- terry l. ridder ><> ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
RE: Spurious write permission error
Please see below ... www.macfh.co.uk/CEMH.html > -Original Message- > From: get_iplayer [mailto:get_iplayer-boun...@lists.infradead.org]On > Behalf Of Clive > Sent: 02 May 2016 17:16 > To: get_iplayer@lists.infradead.org > Subject: Re: Spurious write permission error > > > Until recently I have routinely run 2/3/4 instances of gip from one > Linux Terminal box with no problems - open multiple tabs in > one session > and run separate downloads in each, sometimes all radio or TV > and other > times a mix. I have not encountered a conflict with the > download history > getting corrupt. I have always refreshed the cache in one > before running > multiple. I know this does not help much but it does share an > experience. ... and ... > -Original Message- > From: artisticforge . [mailto:artisticfo...@gmail.com] > Sent: 02 May 2016 18:36 > To: c.e.macfarl...@macfh.co.uk > Cc: Nick Payne; get_iplayer > Subject: Re: Spurious write permission error > > I am running GiP under MacOSX and Debian Linux . > > I run multiple different GiP fetches from the same directory nearly > everyday on both MacOSX and Linux. > My multiple I mean 3 to 4 different fetches going at once from the > same directory. > Never have an issue. There is no doubt that if you run multiple instances, there must be at least a theoretical possibility that two or more instances will require access to a file at the same time, and I am certain that I have such contention troubles in the past. As I have suggested, assuming that only one instance is downloading any one particular programme, the most likely culprits are the shared download history and caches. However, there are various things that might alter how much of a problem this might be in practice, for example here are two: 1) The OS - the OP was working with Windows, while all none of his respondents are. It might be that Windows is less forgiving of attempts by one process to access a file that is currently being updated by another process. 2) The speed of the hardware. It is likely that filesystem timeouts are fairly accurately timed and constant across different hardware, while the time taken to actually update a file will be determined, or rather inversely determined, by the speed of the hardware. Therefore, the faster the hardware, the less the likelihood of a contention causing a timeout. I have fairly old hardware. Regards ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
hello Race conditions as you suggest would be the fault of the operating system and/or filesystem design. Nearly every operating system has drivers which queue requests so that there is no chaos in the allocation of resources. A design flaw that glaring would relegate the operating system to nothing but a toy and the operating system would never be used in a commercial environment. Having read and studied the GiP Perl code over the years, no where in the code are any cache files nor the download_history file opened and kept open. They are looked at at start up , at which point they are "sucked into" memory to be used. The download_history file is opened for append writes after a file has been downloaded . Does not mean that the file has been downloaded correctly just that it has been downloaded. I would chalk the OP errors to computer "hiccups" , cause unknown. A glitch in time & space. A blink of the Twilight Zone or is it Night Gallery. The bits which go bump in the night. On Mon, May 2, 2016 at 4:40 PM, C E Macfarlane wrote: > Please see below ... > > However, there are various things that might alter how much of a problem this > might be in practice, for example here are two: > > 1) The OS - the OP was working with Windows, while all none of his > respondents are. It might be that Windows is less forgiving of attempts by > one process to access a file that is currently being updated by another > process. > > 2) The speed of the hardware. It is likely that filesystem timeouts are > fairly accurately timed and constant across different hardware, while the > time taken to actually update a file will be determined, or rather inversely > determined, by the speed of the hardware. Therefore, the faster the > hardware, the less the likelihood of a contention causing a timeout. I have > fairly old hardware. > > Regards > -- terry l. ridder ><> ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
RE: Spurious write permission error
Please see below ... www.macfh.co.uk/CEMH.html > -Original Message- > From: artisticforge . [mailto:artisticfo...@gmail.com] > Sent: 03 May 2016 12:27 > To: c.e.macfarl...@macfh.co.uk > Cc: Nick Payne; get_iplayer > Subject: Re: Spurious write permission error > > Race conditions as you suggest I never mentioned race conditions. If race conditions were encountered, at very least Task Manager would have to be invoked to end each process involved in the race, perhaps even the computer rebooted if the race hogged so much CPU that TM could not be invoked. > would be the fault of the operating > system and/or filesystem design. > Nearly every operating system has drivers which queue requests so that > there is no chaos in the allocation of resources. Yes. > A design flaw that glaring would relegate the operating system to > nothing but a toy and the operating system would never > be used in a commercial environment. Here's a test for you ... Open a file to play in Window Media Player, but then press the stop button so that nothing is actually being played. If you like, another media file and choose to add it to WMP's current playlist. IME, even though WMP is doing nothing with either file, commonly both are now locked, and cannot be deleted, renamed, or written to, or, if they are appear to be successfully deleted or renamed, examination of the directory shows that they have not been, and won't be until WMP is either exited or the files are deleted from its current playlist. Sometimes, if a file gets updated when within WMP's playlist - say if you use the back button to go to the previous playlist, then update a file in the current playlist by, say, re-extracting it from a longer file with slightly different start and stop time parameters, then use the forward button to return to the current playlist, WMP gets hopelessly confused, and tries to play the new file as if it were the old, and aborts with an er ror message, rather than simply realising that the file has been updated, and refetching its vital statistics from the new version of the file and carrying on from there. ... so your nice well-behaved picture of how an OS and the software running on it should behave just doesn't always happen in practice. > Having read and studied the GiP Perl code over the years, no where in > the code are any cache files nor the download_history file opened and > kept open. I never said that they were, merely that different processes might attempt to write to a given file at the same time. > They are looked at at start up , at which point they are > "sucked into" memory > to be used. The download_history file is opened for append writes > after a file has been downloaded . Does not mean that the file has > been downloaded correctly just that it has been downloaded. Yes, so, as I suggested, if two instances happen simultaneously to finish downloading their respective programmes, contention could occur when they try simultaneously to update the download history accordingly. > I would chalk the OP errors to computer "hiccups" , cause unknown. A > glitch in time & space. A blink of the Twilight Zone or is it Night > Gallery. The bits which go bump in the night. That is great deal less rational than my explanation! I fear we'll be discussing Ley Lines next :-( ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
On Mon, May 02, 2016 at 10:40:01PM +0100, C E Macfarlane wrote: > 1)The OS - the OP was working with Windows, while all none of his > respondents are. It might be that Windows is less forgiving of attempts by > one process to access a file that is currently being updated by another > process. File locking semantics are completely different on Windows compared to Unix, and get_iplayer is written in perl, which has very much a Unixy background. I've not looked at the code, but unless there's extra special Windows sauce in there it might not work. -- David Cantrell | semi-evolved ape-thing I'm in retox ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
hello First a request. can you refrain from inserting your web page with every post? it is annoying second, I do not do "Windows". That said, any operating system should prevent the deletion of a file that is in use. MacOSX & Linux will not let me delete a directory if that directory is the "current working directory" for a process. The test of WMP would only prove that the programmers were lazy and/or just plain sloppy. I use VLC or iTunes. 99.99% of the time it is iTunes. I have 1 Toshiba laptop with Windows 10 installed. It sits in the corner turned off until the rare event where i need to use windows for something. Which are becoming more rare with each passing day. I do not think it has been on this year. The operating system is not going to allow N different processes access the same resource at the exactly the same time. Not going to happen. So the chance of N GiP all going for a write append to the download_history file at the exact same instant in time is ZERO. The operating system is not going to allow it to occur. The requests will all go into the queue and processed in order. What you have been describing all along is a "Race condition". Concerning "Computer Hiccups", events of unknown causes, I have been around enough years to say that "Stuff Happens" which can never be explained. A Flipped bit, a random byte, etc. Nothing is prefect and "Stuff Happens". Those random bit that go bump in the night. I have farm fields that need tending and firewood that needs to be cut. the tractors do not drive themselves, at least not yet. So enough of this diversion. On Tue, May 3, 2016 at 7:26 AM, C E Macfarlane wrote: > Please see below ... > >> -Original Message- >> From: artisticforge . [mailto:artisticfo...@gmail.com] >> Sent: 03 May 2016 12:27 >> To: c.e.macfarl...@macfh.co.uk >> Cc: Nick Payne; get_iplayer >> Subject: Re: Spurious write permission error >> >> Race conditions as you suggest > > I never mentioned race conditions. If race conditions were encountered, at > very least Task Manager would have to be invoked to end each process involved > in the race, perhaps even the computer rebooted if the race hogged so much > CPU that TM could not be invoked. > >> would be the fault of the operating >> system and/or filesystem design. >> Nearly every operating system has drivers which queue requests so that >> there is no chaos in the allocation of resources. > > Yes. > >> A design flaw that glaring would relegate the operating system to >> nothing but a toy and the operating system would never >> be used in a commercial environment. > > Here's a test for you ... > > Open a file to play in Window Media Player, but then press the stop button so > that nothing is actually being played. If you like, another media > file and choose to add it to WMP's current playlist. IME, even though WMP is > doing nothing with either file, commonly both are now locked, and cannot be > deleted, renamed, or written to, or, if they are appear to be successfully > deleted or renamed, examination of the directory shows that they have not > been, and won't be until WMP is either exited or the files are deleted from > its current playlist. Sometimes, if a file gets updated when within WMP's > playlist - say if you use the back button to go to the previous playlist, > then update a file in the current playlist by, say, re-extracting it from a > longer file with slightly different start and stop time parameters, then use > the forward button to return to the current playlist, WMP gets hopelessly > confused, and tries to play the new file as if it were the old, and aborts > with an error message, rather than simply realising that the file has been updated, and refetching its vital statistics from the new version of the file and carrying on from there. >> They are looked at at start up , at which point they are >> "sucked into" memory >> to be used. The download_history file is opened for append writes >> after a file has been downloaded . Does not mean that the file has >> been downloaded correctly just that it has been downloaded. > > Yes, so, as I suggested, if two instances happen simultaneously to finish > downloading their respective programmes, contention could occur when they try > simultaneously to update the download history accordingly. > -- terry l. ridder ><> ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
RE: Spurious write permission error
Please see below ... www.macfh.co.uk/CEMH.html > -Original Message- > From: artisticforge . [mailto:artisticfo...@gmail.com] > Sent: 03 May 2016 14:49 > To: c.e.macfarl...@macfh.co.uk > Cc: Nick Payne; get_iplayer > Subject: Re: Spurious write permission error > > hello > > First a request. can you refrain from inserting your web page with > every post? it is annoying Tough shit. It's my standard email sig, and I'm not changing it just for you. > second, I do not do "Windows". That said, any operating system should > prevent the deletion of a file that is in use. > MacOSX & Linux will not let me delete a directory if that directory is > the "current working directory" for a process. > The test of WMP would only prove that the programmers were lazy and/or > just plain sloppy. You appear not to have noticed that these programmers work for the same firm that writes the OS! No more need be said! [snip irrelevancies] > The operating system is not going to allow N different processes > access the same resource at the exactly the same time. Not going to > happen. Indeed. > So the chance of N GiP all going for a write append to the > download_history file at the exact same instant in time is ZERO. In theory, it rather depends on how you define 'instant in time', in practice, it's just plain incorrect. It is sufficient for contention to occur that a first process wants access to a file, as it is the first to request this access will be granted, and it is still updating the file when a second reaches a state where it requests access to the same file, and the second request times out and is denied, generating an error message. > The operating system is not going to allow it to occur. Exactly, so an error is produced. > The requests will > all go into the queue and processed in order. NOT TRUE! This is the root of your misunderstanding. > What you have been describing all along is a "Race condition". No, a race condition is when one process has locked a resource and for some reason cannot or will not unlock it, while a second wants access to that resource, but doesn't abort gracefully when it can't gain access to it, and keeps endlessly retrying. That is not what is happening here, if it was one or more of the GiP instances would just hang indefinitely, whereas in fact they are correctly aborting with an error message. > Concerning "Computer Hiccups", events of unknown causes, I have been > around enough years to say that "Stuff Happens" > which can never be explained. A Flipped bit, a random byte, etc. > Nothing is prefect and "Stuff Happens". Those random bit that go bump > in the night. By Occam's razor, the simplest explanation is the one that I've given, so we don't need to invent any paranormal ghosts in the machine. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
On Tue, May 03, 2016 at 08:48:47AM -0500, artisticforge . wrote: > second, I do not do "Windows". That said, any operating system should > prevent the deletion of a file that is in use. Indeed. However, Unix-a-likes do let you delete directory entries for files that are in use - the file is only deleted when there are no directory entries for it and no open file handles left. That's not quite the same, of course, but in unless one is being unnaturally precise in one's writing it is rare for people to discriminate between them. On Windows, on the other hand, my understanding is that you can't delete a dirent for a file that is in use. > MacOSX & Linux will not let me delete a directory if that directory is > the "current working directory" for a process. This is technically true - if you 'rmdir' a directory that is some process's cwd then the dirent will go away but the inode will remain until it is no longer in use. However, you won't be able to do anything meaningful with it. -- David Cantrell | Enforcer, South London Linguistic Massive I think the most difficult moment that anyone could face is seeing their domestic servants, whether maid or drivers, run away -- Abdul Rahman Al-Sheikh, writing on 25 Jan 2004 at http://www.arabnews.com/node/243486 ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
On Tue, May 03, 2016 at 03:55:48PM +0100, C E Macfarlane wrote: > From: artisticforge . [mailto:artisticfo...@gmail.com] > > First a request. can you refrain from inserting your web page with > > every post? it is annoying > Tough shit. It's my standard email sig, and I'm not changing it just for you. It is customary to put signatures at the end of messages, not the beginning. I'm sure that if you put it at the end no-one would object. -- David Cantrell | http://www.cantrell.org.uk/david Please stop rolling your Jargon Dice and explain the problem you are having to me in plain English, using small words. -- John Hardin, in the Monastery ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
RE: Spurious write permission error
Please see below ... www.macfh.co.uk/CEMH.html > -Original Message- > From: get_iplayer [mailto:get_iplayer-boun...@lists.infradead.org]On > Behalf Of David Cantrell > Sent: 03 May 2016 17:12 > To: get_iplayer@lists.infradead.org > Subject: Re: Spurious write permission error > > > On Tue, May 03, 2016 at 03:55:48PM +0100, C E Macfarlane wrote: > > From: artisticforge . [mailto:artisticfo...@gmail.com] > > > First a request. can you refrain from inserting your web page with > > > every post? it is annoying > > Tough shit. It's my standard email sig, and I'm not > changing it just for you. > > It is customary to put signatures at the end of messages, not the > beginning. I'm sure that if you put it at the end no-one would object. ... and ... > -Original Message- > From: Roger Bell_West [mailto:ro...@firedrake.org] > Sent: 03 May 2016 16:04 > To: C E Macfarlane > Subject: Re: Spurious write permission error > > On Tue, May 03, 2016 at 03:55:48PM +0100, C E Macfarlane wrote: > >Tough shit. It's my standard email sig, and I'm not > changing it just for you. > > No, a sig is something that goes at the _end_ of a message The sig is put in automatically by Outlook. > and is > separated from it by "\n-- \n". You're thinking of newsgroups, this is email. > No call for swearing just because someone calls you on your choice to > do something non-standard. I consider it rude for someone to pick on such trivia as posting styles, sigs, etc, just because that person is losing an argument, hence my down-to-earth response. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
Re: Spurious write permission error
> Sent: Monday, May 02, 2016 at 1:41 PM > From: "Nick Payne" > [snip] > Unable to write to a directory lacking write permission.INFO: Command > exit code 255 (raw code = 65280) > WARNING: Failed to tag MP4 file That error message comes from AtomicParsley. It is not related to GiP itself or to accessing the download_history file. That file is written before AtomicParsley runs. As you implied with the use of "spurious", it isn't a problem with access permissions - that is just assumed by AtomicParsley. The message really just means that AtomicParsley failed to rename the temporary (tagged) file back to the original file name for some reason. I used to occasionally see this happen on a USB disk attached to a router. I would also see ffmpeg glitches with the same setup, so I just chalked it up to disk contention and stopped using a network disk to host my output directory. Of course, that may not apply to your case. ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
RE: Spurious write permission error
Please see below ... www.macfh.co.uk/CEMH.html > -Original Message- > From: get_iplayer [mailto:get_iplayer-boun...@lists.infradead.org]On > Behalf Of iz > Sent: 03 May 2016 19:59 > To: Nick Payne > Cc: get_iplayer@lists.infradead.org > Subject: Re: Spurious write permission error > > > Sent: Monday, May 02, 2016 at 1:41 PM > > From: "Nick Payne" > > [snip] > > Unable to write to a directory lacking write > permission.INFO: Command > > exit code 255 (raw code = 65280) > > WARNING: Failed to tag MP4 file > > That error message comes from AtomicParsley. Well spotted - searching for ... Unable to write to a directory lacking write permission ... in the GiP PERL source draws a blank, but it's at 0x44D80 in AtomicParsley.exe AP has problems tagging large files, but I've just checked back to my original post on that, and it generates a different error message, so I don't know what to suggest further. > It is not > related to GiP itself or to accessing the download_history > file. That file is written before AtomicParsley runs. As you > implied with the use of "spurious", it isn't a problem with > access permissions - that is just assumed by AtomicParsley. > The message really just means that AtomicParsley failed to > rename the temporary (tagged) file back to the original file > name for some reason. I used to occasionally see this happen > on a USB disk attached to a router. I would also see ffmpeg > glitches with the same setup, so I just chalked it up to disk > contention and stopped using a network disk to host my output > directory. Of course, that may not apply to your case. Perhaps a timing and/or write cache issue - if the new contents of the file have not been entirely written to disk by the time the rename instruction comes, then the file will still be locked. If a write cache is in use, flushing it may solve that, which usually occurs automatically on file closure, so is GiP/AP closing the file properly? Or perhaps putting a sleep command in between file closure and renaming may remove the problem. But note that file contentions CAN occur with multiple instances of GiP running as I have suggested - I know, because I've had them in the dim and distant past! ___ get_iplayer mailing list get_iplayer@lists.infradead.org http://lists.infradead.org/mailman/listinfo/get_iplayer
