Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
On 07 Nov 2014, at 12:59 , Sven Van Caekenberghe s...@stfx.eu wrote: Yes, weird code ;-) Seems to me open:forWrite: would be better off using retryWithGC:until:, which would trigger GC regardless of whether there are registered descriptors for the currently requested handle. That would (hopefully) help against the case where too many handles have been opened (but some are waiting to be finalized), not just the case where you try opening a file in write mode that's already been opened in write mode and discarded. (which is only a restriction on Windows, IIRC) retryWithGC:until:forFileNamed: seems more intended for operations where you are not allowed *any* open handles on the file, such as renames/deletes/moves, etc. Cheers, Henry
[Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow writing,
otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable:
writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name exists |
blockValue foundIt | blockValue := execBlock value. (testBlock value:
blockValue) ifTrue:[^blockValue]. See if we have a file with the given
name foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName]. foundIt
ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue :=
execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect. ^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is
there or not and fail fast, not have all of this thing (debugging kind of
nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs
back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry.
I guess the GC is there to trigger weak reference cleanup.
On the other hand, there is always a limited on the number of open files at the
OS level.
The main thing is, if you use tons of (open) files, to always close them as
fast as possible, and to not let that happen automatically or lose them in the
case of exceptions.
On 07 Nov 2014, at 12:42, p...@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow
writing, otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable:
writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name exists
| blockValue foundIt |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
See if we have a file with the given name
foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName].
foundIt ifFalse:[^blockValue].
Smalltalk garbageCollectMost.
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect.
^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is
there or not and fail fast, not have all of this thing (debugging kind of
nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs
back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
Well, I am not keeping things open... (some local vars to get some easier
stepping)
measurementForTarget: aTargetId onSection: aSection at: aRegistryKey
| item fileRef |
item := 'ACQ_', aTargetId, '-', aSection, '_', aRegistryKey,'.txt'.
fileRef := self monitoringDirectory / item.
^ [ fileRef readStreamDo: [ :s | s next: Config maxFileCharsToRead ] ]
on: Error do: [ :ex |
self warn: ('Cannot measure for target {1} on section {2} and registry key
{3}' format: { aTargetId asString. aSection asString. aRegistryKey asString
}).
ex return: ''
]
It works for a couple files but at one point, it just doesn't open things.
Even if files are closed.
When reusing the same files over and over, it works (like readStreamDo: for
1 times, I get the contents back).
I've simplified the call to retryWithGC: ... to avoid the registry
completely:
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name exists
| blockValue |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
^blockValue
But the symptom is the same.
Time to go into the filePlugin?
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe s...@stfx.eu wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry.
I guess the GC is there to trigger weak reference cleanup.
On the other hand, there is always a limited on the number of open files
at the OS level.
The main thing is, if you use tons of (open) files, to always close them
as fast as possible, and to not let that happen automatically or lose them
in the case of exceptions.
On 07 Nov 2014, at 12:42, p...@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow
writing, otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f
writable: writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name
exists
| blockValue foundIt |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
See if we have a file with the given name
foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName].
foundIt ifFalse:[^blockValue].
Smalltalk garbageCollectMost.
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect.
^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file
is there or not and fail fast, not have all of this thing (debugging kind
of nightmarish...)
I have tested with 100 files and of course it all works since I get the
IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe s...@stfx.eu wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry.
I guess the GC is there to trigger weak reference cleanup.
On the other hand, there is always a limited on the number of open files
at the OS level.
The main thing is, if you use tons of (open) files, to always close them
as fast as possible, and to not let that happen automatically or lose them
in the case of exceptions.
On 07 Nov 2014, at 12:42, p...@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow
writing, otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f
writable: writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name
exists
| blockValue foundIt |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
See if we have a file with the given name
foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName].
foundIt ifFalse:[^blockValue].
Smalltalk garbageCollectMost.
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect.
^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file
is there or not and fail fast, not have all of this thing (debugging kind
of nightmarish...)
I have tested with 100 files and of course it all works since I get the
IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
On 07 Nov 2014, at 13:16, p...@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently.
Anyway, I just wrote this as a test:
'/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base |
0 to: 99 do: [ :each |
base / each asString writeStreamDo: [ :out |
out print: each; crlf; each asWords; crlf ] ] ].
10 timesRepeat: [
| file |
file := 99 atRandom asString.
'/tmp/pharo-files-test/' asFileReference / file readStreamDo: [ :in |
self assert: (in upToEnd beginsWith: file) ] ]
And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe s...@stfx.eu wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry.
I guess the GC is there to trigger weak reference cleanup.
On the other hand, there is always a limited on the number of open files at
the OS level.
The main thing is, if you use tons of (open) files, to always close them as
fast as possible, and to not let that happen automatically or lose them in
the case of exceptions.
On 07 Nov 2014, at 12:42, p...@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow
writing, otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable:
writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name exists
| blockValue foundIt |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
See if we have a file with the given name
foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName].
foundIt ifFalse:[^blockValue].
Smalltalk garbageCollectMost.
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect.
^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is
there or not and fail fast, not have all of this thing (debugging kind of
nightmarish...)
I have tested with 100 files and of course it all works since I get the IDs
back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
I did tests like that and they work ok.
I am still puzzled on why this happens in my code as only do these
readStreamDo: [] and nothing is in parallel.
I've been putting more caching in my own code but still, there is something
hiding.
Phil
On Fri, Nov 7, 2014 at 1:27 PM, Sven Van Caekenberghe s...@stfx.eu wrote:
On 07 Nov 2014, at 13:16, p...@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently.
Anyway, I just wrote this as a test:
'/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base
|
0 to: 99 do: [ :each |
base / each asString writeStreamDo: [ :out |
out print: each; crlf; each asWords; crlf ] ] ].
10 timesRepeat: [
| file |
file := 99 atRandom asString.
'/tmp/pharo-files-test/' asFileReference / file readStreamDo: [
:in |
self assert: (in upToEnd beginsWith: file) ] ]
And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe s...@stfx.eu
wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry.
I guess the GC is there to trigger weak reference cleanup.
On the other hand, there is always a limited on the number of open files
at the OS level.
The main thing is, if you use tons of (open) files, to always close them
as fast as possible, and to not let that happen automatically or lose them
in the case of exceptions.
On 07 Nov 2014, at 12:42, p...@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow
writing, otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f
writable: writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name
exists
| blockValue foundIt |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
See if we have a file with the given name
foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName].
foundIt ifFalse:[^blockValue].
Smalltalk garbageCollectMost.
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect.
^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file
is there or not and fail fast, not have all of this thing (debugging kind
of nightmarish...)
I have tested with 100 files and of course it all works since I get
the IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
On 07 Nov 2014, at 13:42, p...@highoctane.be wrote:
I did tests like that and they work ok.
I am still puzzled on why this happens in my code as only do these
readStreamDo: [] and nothing is in parallel.
I've been putting more caching in my own code but still, there is something
hiding.
Yes, I think so, happy hunting ;-)
You could try to hunt for instances and references when the error occurs.
Phil
On Fri, Nov 7, 2014 at 1:27 PM, Sven Van Caekenberghe s...@stfx.eu wrote:
On 07 Nov 2014, at 13:16, p...@highoctane.be wrote:
I've now set ulimit -n 3000 and things are working.
Sorry for the noise, but this GC thing is still weird.
That would mean that you keep 1000s of files open concurrently.
Anyway, I just wrote this as a test:
'/tmp/pharo-files-test/' asFileReference ensureCreateDirectory in: [ :base |
0 to: 99 do: [ :each |
base / each asString writeStreamDo: [ :out |
out print: each; crlf; each asWords; crlf ] ] ].
10 timesRepeat: [
| file |
file := 99 atRandom asString.
'/tmp/pharo-files-test/' asFileReference / file readStreamDo: [ :in |
self assert: (in upToEnd beginsWith: file) ] ]
And it worked fine.
Phil
On Fri, Nov 7, 2014 at 12:59 PM, Sven Van Caekenberghe s...@stfx.eu wrote:
Yes, weird code ;-)
The registry seems unlimited, it is just a WeakRegistry.
I guess the GC is there to trigger weak reference cleanup.
On the other hand, there is always a limited on the number of open files at
the OS level.
The main thing is, if you use tons of (open) files, to always close them as
fast as possible, and to not let that happen automatically or lose them in
the case of exceptions.
On 07 Nov 2014, at 12:42, p...@highoctane.be wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow
writing, otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f
writable: writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name
exists
| blockValue foundIt |
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
See if we have a file with the given name
foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName].
foundIt ifFalse:[^blockValue].
Smalltalk garbageCollectMost.
blockValue := execBlock value.
(testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect.
^execBlock value.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is
there or not and fail fast, not have all of this thing (debugging kind of
nightmarish...)
I have tested with 100 files and of course it all works since I get the
IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
Re: [Pharo-dev] StandardFileStreamopen:forWrite: serious isse...
Hi Phil,
On Fri, Nov 7, 2014 at 3:42 AM, p...@highoctane.be p...@highoctane.be
wrote:
Code (Transcript calls of mine)
open: fileName forWrite: writeMode
Open the file with the given name. If writeMode is true, allow writing,
otherwise open the file in read-only mode.
| f |
f := fileName asVmPathName.
Transcript log: 'File:', f.
fileID := StandardFileStream retryWithGC:[self primOpen: f writable:
writeMode]
until:[:id| id notNil]
forFileNamed: fileName.
fileID ifNil: [
' Cannot get fileID' logCr.
^ nil]. allows sender to detect failure
(' Got fileID: ', fileID printString) logCr.
name := fileName.
self register.
rwmode := writeMode.
buffer1 := String new: 1.
self enableReadBuffering
I have a ton of files to open. And some were slow to return a problem.
After digging, I got to the method above.
Now at one moment, I get nil fileIDs.
What is that retryWithGC: until: thing???
retryWithGC: execBlock until: testBlock forFileNamed: fullName
Re-implemented to only force GC if a file with the given name exists |
blockValue foundIt | blockValue := execBlock value. (testBlock value:
blockValue) ifTrue:[^blockValue]. See if we have a file with the given
name foundIt := self registry keys hold on strongly for now
anySatisfy:[:file| file name sameAs: fullName]. foundIt
ifFalse:[^blockValue]. Smalltalk garbageCollectMost. blockValue :=
execBlock value. (testBlock value: blockValue) ifTrue:[^blockValue].
Smalltalk garbageCollect. ^execBlock value.
I think its horrible personally but it /should/ work. The idea is that
files shouldn't need to be closed explicitly; which is fine; we want to be
able to rely on finalization to close them eventually. Hence retryWithGC:
is called when an open fails, presumably because it failed because the
process is out of file handles, presumably because several
as-yet-unfinalized files have yet to be finalized. So running the GC
/should/ run finalization. One issue could be that there are bugs meaning
that finalization does /not/ happen. For example, Igor could take a look
and find out quickly whether that's the case. Another issue could be that
the finalization loop is at too low a priority, or has been terminated for
some reason.
But finalization can not be relied upon to be timely. A better solution is
to explicity close files where possible using a form such
as forceNewFileNamed:do: et al. Another solution is possible in Spur where
we can simplify the finalization by using ephemerons so there are no
proxies for files. But that's not relevant immediately.
Huh, yeah, some lookup going on in a registry...
and GCs? For file opens?
I am doing these opens in a webapp, what I want is to know if the file is
there or not and fail fast, not have all of this thing (debugging kind of
nightmarish...)
I have tested with 100 files and of course it all works since I get the
IDs back from the registry.
But at one point, it all fails. Registry full?
What to do here?
TIA
Phil
--
best,
Eliot
