On 2019-Sep-11, Amit Khandekar wrote: > I reproduced the error "exceeded maxAllocatedDescs (492) while trying > to open file ...", which was also discussed about in the thread [1]. > This issue is similar but not exactly the same as [1]. In [1], the > file for which this error used to show up was > "pg_logical/mappings/map...." , while here it's the .spill file. And > here the issue , in short, seems to be : The .spill file does not get > closed there and then, unlike in [1] where there was a file descriptor > leak.
Uh-oh :-( Thanks for the reproducer -- I confirm it breaks things. > Looking at the code, what might be happening is, > ReorderBufferIterTXNInit()=>ReorderBufferRestoreChanges() opens the > files, but leaves them open if end of file is not reached. Eventually > if end of file is reached, it gets closed. The function returns back > without closing the file descriptor if reorder buffer changes being > restored are more than max_changes_in_memory. Probably later on, the > rest of the changes get restored in another > ReorderBufferRestoreChanges() call. But meanwhile, if there are a lot > of such files opened, we can run out of the max files that PG decides > to keep open (it has some logic that takes into account system files > allowed to be open, and already opened). Makes sense. > Offhand, what I am thinking is, we need to close the file descriptor > before returning from ReorderBufferRestoreChanges(), and keep track of > the file offset and file path, so that next time we can resume reading > from there. I think doing this all the time would make restore very slow -- there's a reason we keep the files open, after all. It would be better if we can keep the descriptors open as much as possible, and only close them if there's trouble. I was under the impression that using OpenTransientFile was already taking care of that, but that's evidently not the case. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
