Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Hi, You might want to try to look into the Distiller side of things. That's what I always did. I am a DTP guy. 1) I believe you are using Rundirex.txt file to convert all the .ps's into one .pdf. This page from Adobe confirms that it will take the files in directory order under Windows: http://kb.adobe.com/selfservice/viewContent.do?externalId=318674 -- Acrobat Distiller for Windows will process the files in the order in which you put them into the folder and create the PDF pages in the order in which it processes the files. -- Acrobat Distiller for Mac OS will process the files in alphabetical order. (one solution would be getting a mac, hehehe). Strange that you never hit the wrong order problem before, since according to that page, you should... Regardless of what that paper says, Distiller has ALWAYS processed the files in alphabetical order under Windows. I have been doing so since 2000 and Acrobat Distiller 4. We are now at 9. I refer, of course, to the use of rundirx. 2) That page also talks about Runfilex.ps file, which is basically the same, only you have to list each .ps file in the order you want them to be included. I already addressed that on my first post. I tried runfilex.ps but then Distiller takes 30 to 40 minutes to do the same job that it now does in 3 to 4 minutes, which really is not an option for a newspaper at closing time. I will do some more experiences, from the Distiller side and the Linux side, and I will report here. Thank you for your answers. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Miguel Medalha wrote:
Regardless of what that paper says, Distiller has ALWAYS processed the
files in alphabetical order under Windows. I have been doing so since
2000 and Acrobat Distiller 4. We are now at 9. I refer, of course, to
the use of rundirx.
again, Windows NTFS directories are inherently stored in sorted order
because they are B-Tree indexes on the filename.
if this distiller process is being run from a DOS batch job in
Windows, you could perhaps use something like...
for /f %%F in ('dir /b /on *.ps') DO @\path\to\distiller %%F
to run it on all *.ps files in the current working directory in
alphabetic order.
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
again, Windows NTFS directories are inherently stored in sorted order
because they are B-Tree indexes on the filename.
if this distiller process is being run from a DOS batch job in
Windows, you could perhaps use something like...
for /f %%F in ('dir /b /on *.ps') DO @\path\to\distiller %%F
to run it on all *.ps files in the current working directory in
alphabetic order.
Please note that what Distiller is doing is not run on all *.ps files
in alphabetic order. If only that were the case, I wouldn't be here
bothering people...
Instructed by a special PS file, Distiller is running a set of complex
operations on a group of files in alphabetic order.
I can modify that special PS file to make Distiller process the files in
any order I want.
The problem is that when the order is not provided by the filesystem
itself, the process takes forever.
That's why I was looking for a solution at the filesystem level. I was
trying to understand the inner workings of EXT3 and looking for a
workaround.
Thank you for your tip, though. Maybe some day I will need it.
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
On Sat, Jan 24, 2009 at 12:43 PM, Miguel Medalha miguelmeda...@sapo.pt wrote:
again, Windows NTFS directories are inherently stored in sorted order
because they are B-Tree indexes on the filename.
if this distiller process is being run from a DOS batch job in
Windows, you could perhaps use something like...
for /f %%F in ('dir /b /on *.ps') DO @\path\to\distiller %%F
to run it on all *.ps files in the current working directory in
alphabetic order.
Please note that what Distiller is doing is not run on all *.ps files
in alphabetic order. If only that were the case, I wouldn't be here
bothering people...
Instructed by a special PS file, Distiller is running a set of complex
operations on a group of files in alphabetic order.
I can modify that special PS file to make Distiller process the files in
any order I want.
The problem is that when the order is not provided by the filesystem
itself, the process takes forever.
That's why I was looking for a solution at the filesystem level. I was
trying to understand the inner workings of EXT3 and looking for a
workaround.
Thank you for your tip, though. Maybe some day I will need it.
Have you tried what the different codepages do to sort order in Samba?
Check out these options:
dos charset
unix charset
display charset
-Ross
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
http://code.google.com/p/samba-dirsort-vfs/ Did you try that? I think someone recommended it to you. Well, I did try to compile it but make fails on all the Linux computers I have access to. They all run CentOS 5.2. It would be nice to have a .rpm... I am a sysadmin, not a programmer, I am not able to solve most compile errors. (...) think your real problem lies in your processing software in the file ordering. I would have a really good look at the software doing it. The problem lies in EXT3. I discovered that if I mv the files to another directory the files will then appear on the samba shares in alphanumerical order and will be processed by Acrobat Distiller accordingly. The move can even be done by Windows Explorer working on the Samba share. This seems a bit strange to me. Why doesn't EXT3 present the files in alphanumerical order after they are first created one by one but then presents them alphanumerically after a bulk move to another directory? Also, I connected a FAT32 formated USB flash drive to the server and directed Distiller to there. The files are correctly processed at the first trial. I suppose I will install a smallish FAT32 formated IDE disk on the server just for this purpose. Thank you to all who answered my questions. We form a great community indeed! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
On Fri, 23 Jan 2009, Miguel Medalha wrote: This seems a bit strange to me. Why doesn't EXT3 present the files in alphanumerical order after they are first created one by one but then presents them alphanumerically after a bulk move to another directory? This sounds to me like the dir_index option was applied to a file system that didn't originally have it and an fsck -Df wasn't run at the time. Steve ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
This sounds to me like the dir_index option was applied to a file system that didn't originally have it and an fsck -Df wasn't run at the time. That may well be the most relevant information given here! I will *certainly* give it a try. Thank you! ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
On Fri, 2009-01-23 at 19:43 +, Miguel Medalha wrote: snip (...) think your real problem lies in your processing software in the file ordering. I would have a really good look at the software doing it. The problem lies in EXT3. I discovered that if I mv the files to another directory the files will then appear on the samba shares in alphanumerical order and will be processed by Acrobat Distiller accordingly. The move can even be done by Windows Explorer working on the Samba share. This seems a bit strange to me. Why doesn't EXT3 present the files in alphanumerical order after they are first created one by one but then presents them alphanumerically after a bulk move to another directory? In addition to the other reply about the dir_index/fsck reply, keep in mind that a typical move (mv dir/* newdir/) will present the list of files in alphanumeric order to the mv/cp command. So regardless of the underlying order in the original directory, the order in the target directory should be alphanumeric. In that case, I would expect your software, which apparently processes the directory itself, would see the stuff in the new directory in the desired order, as seems to be indicated by your results above. Also, I connected a FAT32 formated USB flash drive to the server and directed Distiller to there. The files are correctly processed at the first trial. I suppose I will install a smallish FAT32 formated IDE disk on the server just for this purpose. There has to be a better solution. Maybe the mv as a predecessor to the application processing would be acceptable, presuming the dir_index facility is really not working as hoped? Thank you to all who answered my questions. We form a great community indeed! snip sig stuff I still think the dir_index _ought_ to do what you need it to do. But I've never had to depend on it for that purpose so it is just wishful supposition on my part. -- Bill ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Miguel Medalha wrote: (...) think your real problem lies in your processing software in the file ordering. I would have a really good look at the software doing it. The problem lies in EXT3. I discovered that if I mv the files to another directory the files will then appear on the samba shares in alphanumerical order and will be processed by Acrobat Distiller accordingly. The move can even be done by Windows Explorer working on the Samba share. This seems a bit strange to me. Why doesn't EXT3 present the files in alphanumerical order after they are first created one by one but then presents them alphanumerically after a bulk move to another directory? Directories grow as they are filled the first time. If you use a shell script with a wildcard to do the move, the shell will sort the list on the command line as it expands it, so the names are linked into the new directory in sorted order. However if you repeat this in the same directory instead of creating new ones each time it may not continue to work as existing empty slots may be reused in a different order. Also, I connected a FAT32 formated USB flash drive to the server and directed Distiller to there. The files are correctly processed at the first trial. I suppose I will install a smallish FAT32 formated IDE disk on the server just for this purpose. Did you consider sharing a directory from the machine running distiller and cifs-mounting it on the linux side to get ntfs behavior? Also, I'm curious about the timing of the runs. It doesn't sound like the file operations are grouped atomically. How do you ensure that the whole set is present when distiller starts, or that only one set is present? If I were doing it, I'd probably create a new tmp directory for each set of files (which should fix the ordering as a side effect) and rename it to the expected name after all files are present so you see all of them or none. Or, I might put cygwin sshd on the windows box and use scp or rsync to copy the files over in a batch, then start the Distiller run (if you can start it from the command line). -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
I still think the dir_index _ought_ to do what you need it to do. But I've never had to depend on it for that purpose so it is just wishful supposition on my part. I am now almost certain that dir_index will solve the problem. I already remotely did fsck -fD to that filesystem. Now I will have to wait for monday to do the Distiller stuff. Thank you. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Miguel Medalha wrote: I still think the dir_index _ought_ to do what you need it to do. But I've never had to depend on it for that purpose so it is just wishful supposition on my part. I am now almost certain that dir_index will solve the problem. I already remotely did fsck -fD to that filesystem. Now I will have to wait for monday to do the Distiller stuff. I thought dir_index worked with a hash of the filename. Without knowing the hash technique I wouldn't assume that the hash sort order would match the unhashed sort order - but it might. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Did you consider sharing a directory from the machine running distiller and cifs-mounting it on the linux side to get ntfs behavior? That is out of question. The Windows machines are graphic workstations which are not all connected all the time and the Distiller service is essential to the network. Also, I'm curious about the timing of the runs. It doesn't sound like the file operations are grouped atomically. How do you ensure that the whole set is present when distiller starts, or that only one set is present? This is a very peculiar implementation. As I said om my first post, we are a newspaper and, as all newspapers, we don't have a fixed time to close the edition. It closes when it is ready, that's all. The PDFs for print are automatically produced one by one from PostScript files. The PS files fall on a folder watched by Acrobat Distiller and after being stable for more than 10 seconds the conversion begins. Each one contains only one page, which will then be joined to others to form a plan for a platesetter. When all the pages have been produced, one of the graphics people places a special text file on a folder watched by Distiller and it begins to bulk process all the individual PS files: downsampling images, converting the color space to sRGB, consolidating font subsets, creating bookmarks and indexes, etc. The result is a multipage PDF for electronic distribution, containing the whole newspaper in the sRGB color space. This always worked flawlessly until some days ago I replaced the win2k server with a new CentOS/Samba one. Everything worked better and faster except... the pages on this last PDF were in what seemed like an aleatory order. Ordering them by hand is a time consuming and error prone process, specially when everybody is now tired... Producing a newspaper is a pretty tense work, you know. The difficulty with the scripted solutions proposed here is that we cannot know in advance at what time this process will take place and what the number of pages involved will be. At the end of each issue every minute counts. A watching process would have to poll the status of the workflow for several hours with very small intervals, which would be a waste of processor cicles. And not a very elegant thing to do, I feel. I am (for now...) convinced that the tip given to me here about dir_index and the use of fsck -fD will solve this problem. Monday I will know. It will be a lng wait for me. Thank you again. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Miguel Medalha wrote: Did you consider sharing a directory from the machine running distiller and cifs-mounting it on the linux side to get ntfs behavior? That is out of question. The Windows machines are graphic workstations which are not all connected all the time and the Distiller service is essential to the network. I was under the impression that the Distiller app was running under Windows. If it isn't, it doesn't make much sense for it to expect NTFS filesystem semantics. When all the pages have been produced, one of the graphics people places a special text file on a folder watched by Distiller and it begins to bulk process all the individual PS files: [...] The difficulty with the scripted solutions proposed here is that we cannot know in advance at what time this process will take place and what the number of pages involved will be. Can't the trigger operation of placing the special text file be replaced by that person starting the script instead (perhaps click a button on a web page or something similar)? At the end of each issue every minute counts. A watching process would have to poll the status of the workflow for several hours with very small intervals, which would be a waste of processor cicles. And not a very elegant thing to do, I feel. While I wouldn't call it elegant, filesystem caching makes such things efficient enough that you'll never notice them running. If you need a script that looks for a file to appear or expands a wildcard in a directory, go ahead and use one as long as you can sleep for at least a few seconds in the loop. It's cheaper than having a person rearrange something. -- Les Mikesell lesmikes...@gmail.com I am (for now...) convinced that the tip given to me here about dir_index and the use of fsck -fD will solve this problem. Monday I will know. It will be a lng wait for me. Thank you again. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
I was under the impression that the Distiller app was running under Windows. If it isn't, it doesn't make much sense for it to expect NTFS filesystem semantics. Yes, Distiller is running under Windows. When pages start to get ready, one of the graphic operators opens Distiller on his/her workstation which then starts watching a folder *on the server*. Can't the trigger operation of placing the special text file be replaced by that person starting the script instead (perhaps click a button on a web page or something similar)? Yes, that would be a possibility. But those people have strong rooted habits and they are not in the least technically minded. As such, I would prefer to keep a workflow that has been functioning very well. (By the way, that special text file is a snippet of PostScript code that instructs Distiller on where to find the files and how to process them. It would be needed anyway.) Perhaps this obstacle will be removed by applying the correct parameters to the EXT3 file system, as suggested by William Maltby and Steve Thompson above in this thread: mount option dir_index followed by a fsck -Df. I will try this Monday. Thank you for answering. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
On Fri, Jan 23, 2009 at 2:43 PM, Miguel Medalha miguelmeda...@sapo.pt wrote: http://code.google.com/p/samba-dirsort-vfs/ Did you try that? I think someone recommended it to you. Well, I did try to compile it but make fails on all the Linux computers I have access to. They all run CentOS 5.2. It would be nice to have a .rpm... I am a sysadmin, not a programmer, I am not able to solve most compile errors. I will have a hack at compiling it later on because I am very interested in it. If I manage to get it rolling I will send out a mail to you and update the thread here on the list. I have had great success with the clamav vfs module. JohnStanley ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Hi, On Fri, Jan 23, 2009 at 15:29, Miguel Medalha miguelmeda...@sapo.pt wrote: I am now almost certain that dir_index will solve the problem. I already remotely did fsck -fD to that filesystem. I don't really think so... I believe dir_index is the default, your filesystem was probably already created with the dir_index option, and yet your files are out of order. Looking at the man page, it's sorted by the hash of the filename. The purpose is not to present you the files in order, but to make it quicker to open a file in a directory with a huge number of files. Now I will have to wait for monday to do the Distiller stuff. You don't necessarily have to wait to see what the Distiller would do. ls -U shows the files unsorted, in the directory order, that is probably the order in which the Distiller is using them. HTH, Filipe ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Hi, You might want to try to look into the Distiller side of things. 1) I believe you are using Rundirex.txt file to convert all the .ps's into one .pdf. This page from Adobe confirms that it will take the files in directory order under Windows: http://kb.adobe.com/selfservice/viewContent.do?externalId=318674 -- Acrobat Distiller for Windows will process the files in the order in which you put them into the folder and create the PDF pages in the order in which it processes the files. -- Acrobat Distiller for Mac OS will process the files in alphabetical order. (one solution would be getting a mac, hehehe). Strange that you never hit the wrong order problem before, since according to that page, you should... 2) That page also talks about Runfilex.ps file, which is basically the same, only you have to list each .ps file in the order you want them to be included. Any chance you could use this one instead of Rundirex? Is the list of included files fixed? Could the Runfilex.ps file be somehow generated on the server based on the list of files that are there (maybe by a CGI in a web interface) instead of copied by the guy? 3) From what I see, Rundirex.txt (even with a .txt extension) is a Postscript file. AFAIK, Postscript is a full programming language, I've even seen webservers written in Postscript. I'm sure there is a way to sort the list of files from inside Postscript. However, I don't know the language and wouldn't know how to do that, or even how to start looking for it. I searched on the web for someone that did implement this on Rundirex.txt specifically, but with no luck. Maybe someone else on the list will know Postscript, or you could try to look for it in a Postscript list, I'm sure the solution will exist there. Good luck! And let us know how you fixed it! Filipe ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] OT? File order on CentOS/Samba server -- SOLVED (kind of...)
Hi,
On Fri, Jan 23, 2009 at 20:45, Filipe Brandenburger
filbran...@gmail.com wrote:
3) Rundirex.txt (even with a .txt extension) is a Postscript file. [...]
[...] way to sort the list of files from inside Postscript.
I think I did it.
Inside your Rundirex.txt, you should have this snippet:
/RunDir { % Uses PathName variable on the operand stack
{ /mysave save def % Performs a save before running the PS file
dup = flush % Shows name of PS file being run
RunFile % Calls built in Distiller procedure
clear cleardictstack % Cleans up after PS file
mysave restore% Restores save level
}
255 string
filenameforall
} def
Right? If so, then add the definition of a bubble sort routine before
that (which I got from Wikipedia), and then modify /RunDir into the
snippet below. Ghostscript has a .sort built-in that does exaclty
that, but I'm including it here as I don't know if Distiller will too.
% Bubble sort from Wikibooks page on PostScript
/mybubblesort
{ 1 index length 1 sub -1 1
{ 2 index exch 2 copy get 3 copy % arr proc arr i arr[i] arr i arr[i]
0 1 3 index 1 sub
{ 3 index 1 index get % arr proc arr i arr[i] arr
imax amax j arr[j]
2 index 1 index 10 index exec
{ % ... amax arr[j]
4 2 roll
}
if pop pop
}
for % arr proc arr i
arr[i] arr imax amax
4 -1 roll exch 4 1 roll put put
}
for
pop
} bind def
/RunDir { % Uses PathName variable on the operand stack
/nf 0 def % Reset counter for number of files
{ 255 string copy % Copy to a separate string (otherwise
would be overwritten)
/nf nf 1 add def % Increment counter of number of files
}
255 string
filenameforall
nf array astore % Put all filenames in an array
{ lt } mybubblesort % And sort it
{ /mysave save def % Performs a save before running the PS file
dup = flush % Shows name of PS file being run
RunFile % Calls built in Distiller procedure
clear cleardictstack % Cleans up after PS file
mysave restore% Restores save level
}
forall % Execute original procedure, but
using sorted array
} def
Of course I did not test it with Distiller which I don't have... I did
test the part of sorting the list of files with Ghostscript and it
works.
Maybe word wrapping in the e-mail will ruin the snippet, if that's the
case please let me know and I'll send it attached to you.
Let us know if that works!
Filipe
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
