Re: [PD] Large File Support on Linux
On 03/16/2013 23:42, Miller Puckette wrote: > It's in the source - but Pd has to be compiled with '_LARGEFILE64_SOURCE' > defined for this to work. Through 0.43 the configure script I was using > was checking for this somehow (too complicted for me to understand how). > > Anyhow, this is a bug all right - thanks for reporting it! > > ... and can you tell me what OS you're on so I can check if I'm actually > fixing it? it's in the source, the configure script already defines _LARGEFILE64_SOURCE (iirc) but something still fails (yea, definitely a bug) i confirmed that behaviour a few months ago on linux. fgamr IOhannes fbgmasdr IOhannes ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
So a quick (and perhaps stupid) question - can I just use #idfef __linux__ to alias open to open64, etc? It works on my x86_64 machine and on my Raspberry Pi so I'm guessing all modern linuxes define an open64(). My main fear is that android might define __linux__ but not open64() - anyone know about that? thanks Miller On Sun, Mar 17, 2013 at 10:27:41AM +0100, IOhannes m zmölnig wrote: > On 03/16/2013 23:42, Miller Puckette wrote: > > It's in the source - but Pd has to be compiled with '_LARGEFILE64_SOURCE' > > defined for this to work. Through 0.43 the configure script I was using > > was checking for this somehow (too complicted for me to understand how). > > > > Anyhow, this is a bug all right - thanks for reporting it! > > > > ... and can you tell me what OS you're on so I can check if I'm actually > > fixing it? > > it's in the source, the configure script already defines > _LARGEFILE64_SOURCE (iirc) but something still fails (yea, definitely a bug) > > i confirmed that behaviour a few months ago on linux. > > fgamr > IOhannes > > > fbgmasdr > IOhannes > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Hi, thanks for taking that up. On my machine (archlinux, Linux 3.7.10-1-ARCH i686 GNU/Linux, glibc 2.17), changing the LARGEFILE define to __linux__ (or __gnu_linux__ yields: git pull ./autogen.sh ./configure --prefix=/opt/pd-vanilla/git --enable-jack make prefix=/opt/pd-vanilla-git [...] gcc -DPACKAGE_NAME=\"pd\" -DPACKAGE_TARNAME=\"pd\" -DPACKAGE_VERSION=\"0.44.0\" -DPACKAGE_STRING=\"pd\ 0.44.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"pd\" -DVERSION=\"0.44.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_LIBDL=1 -DSTDC_HEADERS=1 -DHAVE_ALLOCA_H=1 -DHAVE_ALLOCA=1 -DHAVE_FCNTL_H=1 -DHAVE_LIMITS_H=1 -DHAVE_MALLOC_H=1 -DHAVE_NETDB_H=1 -DHAVE_NETINET_IN_H=1 -DHAVE_STDDEF_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_IOCTL_H=1 -DHAVE_SYS_PARAM_H=1 -DHAVE_SYS_SOCKET_H=1 -DHAVE_SYS_SOUNDCARD_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_SYS_TIMEB_H=1 -DHAVE_UNISTD_H=1 -DHAVE_FORK=1 -DHAVE_VFORK=1 -DHAVE_WORKING_VFORK=1 -DHAVE_WORKING_FORK=1 -DHAVE_STDLIB_H=1 -DHAVE_MALLOC=1 -DHAVE_STDLIB_H=1 -DHAVE_REALLOC=1 -DRETSIGTYPE=void -DHAVE_DUP2=1 -DHAVE_GETCWD=1 -DHAVE_GETHOSTBYNAME=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMMOVE=1 -DHAVE_MEMSET=1 -DHAVE_REGCOMP=1 -DHAVE_SELECT=1 -DHAVE_SOCKET=1 -DHAVE_STRCHR=1 -DHAVE_STRERROR=1 -DHAVE_STRRCHR=1 -DHAVE_STRSTR=1 -DHAVE_STRTOL=1 -I.-DPD -DINSTALL_PREFIX=\"/opt/pd-vanilla-git\" -DUSEAPI_ALSA -DUSEAPI_JACK -DJACK_XRUN -DUSEAPI_OSS -DUSEAPI_PORTAUDIO -I../portaudio/include -O6 -funroll-loops -fomit-frame-pointer -MT pd-s_path.o -MD -MP -MF .deps/pd-s_path.Tpo -c -o pd-s_path.o `test -f 's_path.c' || echo './'`s_path.c s_path.c: In function ‘sys_trytoopenone’: s_path.c:290:21: error: storage size of ‘statbuf’ isn’t known [...] And that can make sense since stat is now stat64, the structure name: struct stat statbuf is struct stat64 statbuf; So I tried with these defines instead : #ifdef __gnu_linux__ # define open(x, ...) open64(x ,__VA_ARGS__) # define lseek(x, ...) lseek64(x ,__VA_ARGS__) # define fstat(x, ...) fstat64(x ,__VA_ARGS__) # define stat(x, ...) stat64(x ,__VA_ARGS__) #endif but opening a large file still fails. Miller Puckette wrote: > So a quick (and perhaps stupid) question - can I just use > #idfef __linux__ to alias open to open64, etc? It works on my > x86_64 machine and on my Raspberry Pi so I'm guessing all modern > linuxes define an open64(). My main fear is that android might define > __linux__ but not open64() - anyone know about that? ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-03-18 11:18, Charles Goyard wrote: > Hi, > > thanks for taking that up. > > On my machine (archlinux, Linux 3.7.10-1-ARCH i686 GNU/Linux, > glibc 2.17), changing the LARGEFILE define to __linux__ (or > __gnu_linux__ yields: nonono. you MUST have "_LARGEFILE64_SOURCE" defined before including unistd.h, else you won't have LFS support. there are configure checks that test whether doing defining _LARGEFILE64_SOURCE makes sense on your system. having said that, i just noticed that these checks seem to have gone from configure(.ac) during the 0.43 rewrite of the buildsystem. so most likely this is the source of the bug. fgasdrm IOhannes -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlFHBBYACgkQkX2Xpv6ydvSerACaAgihDxsR9kHRcW7CwXNxMDCE f60AnA4PIofPX8M0ClZiu7Rf/RRGz6tK =RFmt -END PGP SIGNATURE- ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-03-18 13:10, IOhannes m zmoelnig wrote: > On 2013-03-18 11:18, Charles Goyard wrote: >> Hi, > >> thanks for taking that up. > >> On my machine (archlinux, Linux 3.7.10-1-ARCH i686 GNU/Linux, >> glibc 2.17), changing the LARGEFILE define to __linux__ (or >> __gnu_linux__ yields: > > > nonono. you MUST have "_LARGEFILE64_SOURCE" defined before > including unistd.h, else you won't have LFS support. there are > configure checks that test whether doing defining > _LARGEFILE64_SOURCE makes sense on your system. > > having said that, i just noticed that these checks seem to have > gone from configure(.ac) during the 0.43 rewrite of the > buildsystem. so most likely this is the source of the bug. > so i rechecked and things are a bit more complicated than i originally thought when it comes to large file support (LFS). it is (well, should be) easy enough to open/read/write large (>2GB) files within Pd. simply using "_LARGEFILE64_SOURCE" or "_FILE_OFFSET_BITS=64" is the recommended way. those defines have to be set whenever unistd is included for open/close/lseek/... now Pd uses some magic in d_soundfile.c (e.g. redefining "open()" to "open64()"), but that doesn't really help at all, since the files are not opened in d_soundfile.c but instead are opened by open_via_path() in s_path.c. open_via_path() currently does not handle LFS. the easy way would obviously be to move LFS into s_path.c. the problem with that is, that s_path implements an API available for externals. if open_via_path() returns a filehandle for an LFS file, and the external has been compiled without LFS, the filehande will be incompatible. see [1] for a discussion. i guess the only clean way to solve that, is to provide a complete wrapper around the file API, so an external has functions guaranteed to work with the filehandle returned by Pd. currently there are sys_open()/sys_close() and its f*-variants. but we would need at least sys_(f)seek and sys_(f)stat, but preferrably a complete wrapper around LFS-compatible file functions [2]. all this functionality (including the handilng of UTF filenames on W32) is not really Pd-related, and could well be factored out into a separate library. fgasd, IOhannes [1] http://ohse.de/uwe/articles/lfs.html [2] http://opengroup.org/platform/lfs.html -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlFHDpoACgkQkX2Xpv6ydvQ+wACeMLeUFh9Bgfor7SV4YvcZokwL MhwAn23vz/Zl6xpp+/EWQPvWrW8gyEm8 =AE4m -END PGP SIGNATURE- ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
> the problem with that is, that s_path implements an API available for > externals. > if open_via_path() returns a filehandle for an LFS file, and the > external has been compiled without LFS, the filehande will be > incompatible. see [1] for a discussion. > > i guess the only clean way to solve that, is to provide a complete > wrapper around the file API, so an external has functions guaranteed > to work with the filehandle returned by Pd. > currently there are sys_open()/sys_close() and its f*-variants. > but we would need at least sys_(f)seek and sys_(f)stat, but > preferrably a complete wrapper around LFS-compatible file functions [2]. > > all this functionality (including the handilng of UTF filenames on > W32) is not really Pd-related, and could well be factored out into a > separate library. Thanks for the clear explanation of the matter, IOhannes. Why not simply recompile externals after fixing s_path? Pd-extended already comes with most externals recompiled for every new release and adding legacy stuff just creates more confusion in maintaining the source down the road. In other words, FWIW and IMHO I think there is way too much effort the community is trying to pour into binary compatibility for a system that clearly begs for further enhancements in the core API. Best wishes, Ico ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
To answer Ico's question, the trouble I forsee is musician A gives musician B a patch, containing compiled externs - and then musician B runs it and gets a crash instead of music. Sub-optimal in my opinion :) I now think that it should be OK to use open64() only in the file d_soundfile.c and only for linux - so putting at the head of the file, #ifdef __linux__ #define _LARGEFILE64_SOURCE #endif ... then, because opening soundfiles currently uses open_via_path, simply closing the file and re-opening it via open64(). There's a similar hack to open "binbufs" via paths in the function binbuf_read_via_canvas(). There are two other festering problems in open_via_path() that all should probably be fixed in one go by defining an updated function call: 1. externs further down the path are chosen in front of abstractions that should be taken instead; 2. open_via_path isn't thread safe - d_soundfile.c could call it at the same moment someone in the "Pd" thread is changing the path. This should almost never hapen but should be fixed. This is a big enough change that I think it should wait for 0.45, whereas the hack described above could go in right now as a local bug-fix. BTW I've got a couple of other bug-fixes underway; wil push to git now. cheers Miller Oe Mon, Mar 18, 2013 at 12:47:47PM -0400, Ivica Ico Bukvic wrote: > > the problem with that is, that s_path implements an API available for > > externals. > > if open_via_path() returns a filehandle for an LFS file, and the > > external has been compiled without LFS, the filehande will be > > incompatible. see [1] for a discussion. > > > > i guess the only clean way to solve that, is to provide a complete > > wrapper around the file API, so an external has functions guaranteed > > to work with the filehandle returned by Pd. > > currently there are sys_open()/sys_close() and its f*-variants. > > but we would need at least sys_(f)seek and sys_(f)stat, but > > preferrably a complete wrapper around LFS-compatible file functions [2]. > > > > all this functionality (including the handilng of UTF filenames on > > W32) is not really Pd-related, and could well be factored out into a > > separate library. > > Thanks for the clear explanation of the matter, IOhannes. > > Why not simply recompile externals after fixing s_path? Pd-extended already > comes with most externals recompiled for every new release and adding legacy > stuff just creates more confusion in maintaining the source down the road. > In other words, FWIW and IMHO I think there is way too much effort the > community is trying to pour into binary compatibility for a system that > clearly begs for further enhancements in the core API. > > Best wishes, > > Ico > > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
I completely agree with your example, Miller, as far as pd vanilla is concerned. OTOH pd-extended ships with all its externals compiled with each version, so this would be for the most part a non-issue unless: 1) users A or B are using different versions of pd (which could be elaborated upon in the documentation when sharing patches which is rather common in other software environments), or 2) if A is using custom externals that are not found in pd-extended (in which case B would not want to use those anyways unless both A and B are using the same platform, which is the only scenario where it would make sense to keep this as a workaround) Either way FWIW I still feel this is too much of a workaround for little gain and potentially a lot of headache down the road. Best wishes, Ico > -Original Message- > From: Miller Puckette [mailto:m...@ucsd.edu] > Sent: Monday, March 18, 2013 2:47 PM > To: Ivica Ico Bukvic > Cc: 'IOhannes m zmoelnig'; pd-list@iem.at > Subject: Re: [PD] Large File Support on Linux > > To answer Ico's question, the trouble I forsee is musician A gives > musician B a patch, containing compiled externs - and then musician B > runs it and gets a crash instead of music. Sub-optimal in my opinion :) > > I now think that it should be OK to use open64() only in the file d_soundfile.c > and only for linux - so putting at the head of the file, > > #ifdef __linux__ > #define _LARGEFILE64_SOURCE > #endif > > ... then, because opening soundfiles currently uses open_via_path, simply > closing the file and re-opening it via open64(). There's a similar hack > to open "binbufs" via paths in the function binbuf_read_via_canvas(). > > There are two other festering problems in open_via_path() that all should > probably be fixed in one go by defining an updated function call: > > 1. externs further down the path are chosen in front of abstractions that > should be taken instead; > > 2. open_via_path isn't thread safe - d_soundfile.c could call it at the same > moment someone in the "Pd" thread is changing the path. This should > almost > never hapen but should be fixed. > > This is a big enough change that I think it should wait for 0.45, whereas the > hack described above could go in right now as a local bug-fix. > > BTW I've got a couple of other bug-fixes underway; wil push to git now. > > cheers > Miller > > Oe Mon, Mar 18, 2013 at 12:47:47PM -0400, Ivica Ico Bukvic wrote: > > > the problem with that is, that s_path implements an API available for > > > externals. > > > if open_via_path() returns a filehandle for an LFS file, and the > > > external has been compiled without LFS, the filehande will be > > > incompatible. see [1] for a discussion. > > > > > > i guess the only clean way to solve that, is to provide a complete > > > wrapper around the file API, so an external has functions guaranteed > > > to work with the filehandle returned by Pd. > > > currently there are sys_open()/sys_close() and its f*-variants. > > > but we would need at least sys_(f)seek and sys_(f)stat, but > > > preferrably a complete wrapper around LFS-compatible file functions [2]. > > > > > > all this functionality (including the handilng of UTF filenames on > > > W32) is not really Pd-related, and could well be factored out into a > > > separate library. > > > > Thanks for the clear explanation of the matter, IOhannes. > > > > Why not simply recompile externals after fixing s_path? Pd-extended > already > > comes with most externals recompiled for every new release and adding > legacy > > stuff just creates more confusion in maintaining the source down the road. > > In other words, FWIW and IMHO I think there is way too much effort the > > community is trying to pour into binary compatibility for a system that > > clearly begs for further enhancements in the core API. > > > > Best wishes, > > > > Ico > > > > > > ___ > > Pd-list@iem.at mailing list > > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Hi, I am in favour of breaking binary compatibility and keeping the code simple. People that compile their externals themselves can understand and cope with it. Other people mostly won't notice. My intuition is that if the LFS project was unable to provide a transparent API in the first place, there's no reason we will be able to do anything clean. Just communicate enough about the breakage and enjoy :). Miller Puckette wrote: > To answer Ico's question, the trouble I forsee is musician A gives > musician B a patch, containing compiled externs - and then musician B > runs it and gets a crash instead of music. Sub-optimal in my opinion :) ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
On further thought, I don't think it's even possible to change open_via_path() to use open64() and maintain even source compatibility for externs - if one of them calls lseek or fstat we're sunk - and I don't see any robust way of aliasing those calls to lseek64() etc. I'm now realizing too that I don't know what approaches the Mac and Microsoft pltforms are taking to large file support - I think any solution will have to somehow address all of the platforms. For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do that internally to d_soundfile.c so as to cause the least possible disruption in a 'bugfix' pd release (probably 0.44-3). cheers Miller On Tue, Mar 19, 2013 at 08:55:07PM +0100, Charles Goyard wrote: > Hi, > > I am in favour of breaking binary compatibility and keeping the code > simple. People that compile their externals themselves can understand > and cope with it. Other people mostly won't notice. > > My intuition is that if the LFS project was unable to provide a transparent > API in the first place, there's no reason we will be able to do anything > clean. > > Just communicate enough about the breakage and enjoy :). > > Miller Puckette wrote: > > To answer Ico's question, the trouble I forsee is musician A gives > > musician B a patch, containing compiled externs - and then musician B > > runs it and gets a crash instead of music. Sub-optimal in my opinion :) > > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
On 03/20/2013 18:02, Miller Puckette wrote: > On further thought, I don't think it's even possible to change open_via_path() > to use open64() and maintain even source compatibility for externs - if > one of them calls lseek or fstat we're sunk - and I don't see any robust > way of aliasing those calls to lseek64() etc. yep. that's why i said that the only real solution would be to provide a more-or-less complete set of fileIO-functions: sys_lseek, sys_stat (including f-variants). > > I'm now realizing too that I don't know what approaches the Mac and Microsoft > pltforms are taking to large file support - I think any solution will have to > somehow address all of the platforms. which would be possible if we provided the full set. of file accessors. (but to repeat, this is not really a Pd-problem, and could (and maybe should) be solved in an auxiliary lib). > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do > that internally to d_soundfile.c so as to cause the least possible disruption > in a 'bugfix' pd release (probably 0.44-3). which i think is a good enough approach. but of course there's a catch: sys_open() will gracefully handle UTF-8 filenames, whereas on some platforms open() will not. so with your solution, soundfiles with non-ascii chars will fail to open on W32. since open_via_path() is so often used to determine the full path of a file somewhere in the searchpath, it might be a good idea to wrap that functionality into a find_via_path() function that returns the canonicalized filename. it would be great if that filename would be mangled in a way so UTF8 doesn't make problems any more, but i think this is simply not possible with the way w32 handles opening such filenames. fgmadsr IOhannes ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do the "right" thing. I guess on Windows this would have to be somehow both UTF8 and 64-bit safe - all the more reason to do as you suggest and hide the whole mes behind sys_this_and_that() variants. I thought exactly the same thing about a find_via_path routine. It seems to me that open_via_path, which tries not just to verify that the file exists but also that it can actually be opened, is perhaps working too hard; if a file that proves impossible to open is in an earlier spot on the path, the best thing might be to try and fail to open it instead of going out to another, perhaps incorrect, version of the file. Also, it could be fixed to take the path as an argument so that d_soundfile.c can call it from other threads safely (using separate copies of the path). cheers Miller On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote: > On 03/20/2013 18:02, Miller Puckette wrote: > > On further thought, I don't think it's even possible to change > > open_via_path() > > to use open64() and maintain even source compatibility for externs - if > > one of them calls lseek or fstat we're sunk - and I don't see any robust > > way of aliasing those calls to lseek64() etc. > > yep. > that's why i said that the only real solution would be to provide a > more-or-less complete set of fileIO-functions: sys_lseek, sys_stat > (including f-variants). > > > > > I'm now realizing too that I don't know what approaches the Mac and > > Microsoft > > pltforms are taking to large file support - I think any solution will have > > to > > somehow address all of the platforms. > > which would be possible if we provided the full set. of file accessors. > (but to repeat, this is not really a Pd-problem, and could (and maybe > should) be solved in an auxiliary lib). > > > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do > > that internally to d_soundfile.c so as to cause the least possible > > disruption > > in a 'bugfix' pd release (probably 0.44-3). > > which i think is a good enough approach. > but of course there's a catch: sys_open() will gracefully handle UTF-8 > filenames, whereas on some platforms open() will not. > so with your solution, soundfiles with non-ascii chars will fail to open > on W32. > > since open_via_path() is so often used to determine the full path of a > file somewhere in the searchpath, it might be a good idea to wrap that > functionality into a find_via_path() function that returns the > canonicalized filename. it would be great if that filename would be > mangled in a way so UTF8 doesn't make problems any more, but i think > this is simply not possible with the way w32 handles opening such filenames. > > > fgmadsr > IOhannes > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Miller, Does this mean if one were not so concerned with backwards compatibility and included define you listed below in the s_path.c that this would fix the LFS issue albeit at the expense of backwards/cross-platform compatibility? Also, in Linux is open64 safe for both 32-bit and 64-bit OS variants? > -Original Message- > From: pd-list-boun...@iem.at [mailto:pd-list-boun...@iem.at] On Behalf Of > Miller Puckette > Sent: Wednesday, March 20, 2013 6:46 PM > To: IOhannes m zmölnig > Cc: pd-list@iem.at > Subject: Re: [PD] Large File Support on Linux > > OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do > the "right" thing. I guess on Windows this would have to be somehow both > UTF8 and 64-bit safe - all the more reason to do as you suggest and hide the > whole mes behind sys_this_and_that() variants. > > I thought exactly the same thing about a find_via_path routine. It seems to > me that open_via_path, which tries not just to verify that the file exists but > also that it can actually be opened, is perhaps working too hard; if a file > that proves impossible to open is in an earlier spot on the path, the best > thing might be to try and fail to open it instead of going out to another, > perhaps incorrect, version of the file. > > Also, it could be fixed to take the path as an argument so that d_soundfile.c > can call it from other threads safely (using separate copies of the path). > > cheers > Miller > > On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote: > > On 03/20/2013 18:02, Miller Puckette wrote: > > > On further thought, I don't think it's even possible to change > open_via_path() > > > to use open64() and maintain even source compatibility for externs - if > > > one of them calls lseek or fstat we're sunk - and I don't see any robust > > > way of aliasing those calls to lseek64() etc. > > > > yep. > > that's why i said that the only real solution would be to provide a > > more-or-less complete set of fileIO-functions: sys_lseek, sys_stat > > (including f-variants). > > > > > > > > I'm now realizing too that I don't know what approaches the Mac and > Microsoft > > > pltforms are taking to large file support - I think any solution will have to > > > somehow address all of the platforms. > > > > which would be possible if we provided the full set. of file accessors. > > (but to repeat, this is not really a Pd-problem, and could (and maybe > > should) be solved in an auxiliary lib). > > > > > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I can do > > > that internally to d_soundfile.c so as to cause the least possible disruption > > > in a 'bugfix' pd release (probably 0.44-3). > > > > which i think is a good enough approach. > > but of course there's a catch: sys_open() will gracefully handle UTF-8 > > filenames, whereas on some platforms open() will not. > > so with your solution, soundfiles with non-ascii chars will fail to open > > on W32. > > > > since open_via_path() is so often used to determine the full path of a > > file somewhere in the searchpath, it might be a good idea to wrap that > > functionality into a find_via_path() function that returns the > > canonicalized filename. it would be great if that filename would be > > mangled in a way so UTF8 doesn't make problems any more, but i think > > this is simply not possible with the way w32 handles opening such > filenames. > > > > > > fgmadsr > > IOhannes > > > > ___ > > Pd-list@iem.at mailing list > > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
I believe not - every extern that used open_bia_path0 would have to be fixed at the source level to use lseek64(), fstat64() in place of lseek() and fstat(). I can't see any way to "fix" this source-compatibly. I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I think it's save to say open64 exists on all modern linuxes, both 64 and 32 bit. cheers Miller On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote: > Miller, > > Does this mean if one were not so concerned with backwards compatibility and > included define you listed below in the s_path.c that this would fix the LFS > issue albeit at the expense of backwards/cross-platform compatibility? Also, > in Linux is open64 safe for both 32-bit and 64-bit OS variants? > > > -Original Message- > > From: pd-list-boun...@iem.at [mailto:pd-list-boun...@iem.at] On Behalf Of > > Miller Puckette > > Sent: Wednesday, March 20, 2013 6:46 PM > > To: IOhannes m zmölnig > > Cc: pd-list@iem.at > > Subject: Re: [PD] Large File Support on Linux > > > > OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do > > the "right" thing. I guess on Windows this would have to be somehow both > > UTF8 and 64-bit safe - all the more reason to do as you suggest and hide > the > > whole mes behind sys_this_and_that() variants. > > > > I thought exactly the same thing about a find_via_path routine. It seems > to > > me that open_via_path, which tries not just to verify that the file exists > but > > also that it can actually be opened, is perhaps working too hard; if a > file > > that proves impossible to open is in an earlier spot on the path, the best > > thing might be to try and fail to open it instead of going out to another, > > perhaps incorrect, version of the file. > > > > Also, it could be fixed to take the path as an argument so that > d_soundfile.c > > can call it from other threads safely (using separate copies of the path). > > > > cheers > > Miller > > > > On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote: > > > On 03/20/2013 18:02, Miller Puckette wrote: > > > > On further thought, I don't think it's even possible to change > > open_via_path() > > > > to use open64() and maintain even source compatibility for externs - > if > > > > one of them calls lseek or fstat we're sunk - and I don't see any > robust > > > > way of aliasing those calls to lseek64() etc. > > > > > > yep. > > > that's why i said that the only real solution would be to provide a > > > more-or-less complete set of fileIO-functions: sys_lseek, sys_stat > > > (including f-variants). > > > > > > > > > > > I'm now realizing too that I don't know what approaches the Mac and > > Microsoft > > > > pltforms are taking to large file support - I think any solution will > have to > > > > somehow address all of the platforms. > > > > > > which would be possible if we provided the full set. of file accessors. > > > (but to repeat, this is not really a Pd-problem, and could (and maybe > > > should) be solved in an auxiliary lib). > > > > > > > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I > can do > > > > that internally to d_soundfile.c so as to cause the least possible > disruption > > > > in a 'bugfix' pd release (probably 0.44-3). > > > > > > which i think is a good enough approach. > > > but of course there's a catch: sys_open() will gracefully handle UTF-8 > > > filenames, whereas on some platforms open() will not. > > > so with your solution, soundfiles with non-ascii chars will fail to open > > > on W32. > > > > > > since open_via_path() is so often used to determine the full path of a > > > file somewhere in the searchpath, it might be a good idea to wrap that > > > functionality into a find_via_path() function that returns the > > > canonicalized filename. it would be great if that filename would be > > > mangled in a way so UTF8 doesn't make problems any more, but i think > > > this is simply not possible with the way w32 handles opening such > > filenames. > > > > > > > > > fgmadsr > > > IOhannes > > > > > > ___ > > > Pd-list@iem.at mailing list > > > UNSUBSCRIBE and account-management -> > > http://lists.puredata.info/listinfo/pd-list > > > > ___ > > Pd-list@iem.at mailing list > > UNSUBSCRIBE and account-management -> > > http://lists.puredata.info/listinfo/pd-list > ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Could one simply redefine lseek and fstat to use lseek64 and fstat64 instead and be done with it (again, assuming one is not concerned with backwards and/or x-platform compatibility)? Best wishes, Ico > -Original Message- > From: Miller Puckette [mailto:m...@ucsd.edu] > Sent: Wednesday, March 20, 2013 9:21 PM > To: Ivica Ico Bukvic > Cc: 'IOhannes m zmölnig'; pd-list@iem.at > Subject: Re: [PD] Large File Support on Linux > > I believe not - every extern that used open_bia_path0 would have to be > fixed at the source level to use lseek64(), fstat64() in place of > lseek() and fstat(). I can't see any way to "fix" this source-compatibly. > > I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I think > it's save to say open64 exists on all modern linuxes, both 64 and 32 bit. > > cheers > Miller > On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote: > > Miller, > > > > Does this mean if one were not so concerned with backwards compatibility > and > > included define you listed below in the s_path.c that this would fix the LFS > > issue albeit at the expense of backwards/cross-platform compatibility? > Also, > > in Linux is open64 safe for both 32-bit and 64-bit OS variants? > > > > > -Original Message- > > > From: pd-list-boun...@iem.at [mailto:pd-list-boun...@iem.at] On > Behalf Of > > > Miller Puckette > > > Sent: Wednesday, March 20, 2013 6:46 PM > > > To: IOhannes m zmölnig > > > Cc: pd-list@iem.at > > > Subject: Re: [PD] Large File Support on Linux > > > > > > OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do > > > the "right" thing. I guess on Windows this would have to be somehow > both > > > UTF8 and 64-bit safe - all the more reason to do as you suggest and hide > > the > > > whole mes behind sys_this_and_that() variants. > > > > > > I thought exactly the same thing about a find_via_path routine. It seems > > to > > > me that open_via_path, which tries not just to verify that the file exists > > but > > > also that it can actually be opened, is perhaps working too hard; if a > > file > > > that proves impossible to open is in an earlier spot on the path, the best > > > thing might be to try and fail to open it instead of going out to another, > > > perhaps incorrect, version of the file. > > > > > > Also, it could be fixed to take the path as an argument so that > > d_soundfile.c > > > can call it from other threads safely (using separate copies of the path). > > > > > > cheers > > > Miller > > > > > > On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote: > > > > On 03/20/2013 18:02, Miller Puckette wrote: > > > > > On further thought, I don't think it's even possible to change > > > open_via_path() > > > > > to use open64() and maintain even source compatibility for externs - > > if > > > > > one of them calls lseek or fstat we're sunk - and I don't see any > > robust > > > > > way of aliasing those calls to lseek64() etc. > > > > > > > > yep. > > > > that's why i said that the only real solution would be to provide a > > > > more-or-less complete set of fileIO-functions: sys_lseek, sys_stat > > > > (including f-variants). > > > > > > > > > > > > > > I'm now realizing too that I don't know what approaches the Mac and > > > Microsoft > > > > > pltforms are taking to large file support - I think any solution will > > have to > > > > > somehow address all of the platforms. > > > > > > > > which would be possible if we provided the full set. of file accessors. > > > > (but to repeat, this is not really a Pd-problem, and could (and maybe > > > > should) be solved in an auxiliary lib). > > > > > > > > > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I > > can do > > > > > that internally to d_soundfile.c so as to cause the least possible > > disruption > > > > > in a 'bugfix' pd release (probably 0.44-3). > > > > > > > > which i think is a good enough approach. > > > > but of course there's a catch: sys_open() will gracefully handle UTF-8 > > > > filenames, whereas on some platforms open() will not. > > > >
Re: [PD] Large File Support on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-03-21 04:22, Ivica Ico Bukvic wrote: > Could one simply redefine lseek and fstat to use lseek64 and > fstat64 instead and be done with it (again, assuming one is not > concerned with backwards and/or x-platform compatibility)? if you are talking about redefining (as in "#define open open64") in m_pd.h, than please: no NO! that will break compilation of a number of externals immediately. one that comes to my mind is any C++-external (including Gem) that uses "open" methods in classes. e.g. foo::open(s); will expand to foo::open64(s); which will break binary compatibility when linking to any library providing foo::open() simply by including m_pd.h gfmasdr IOhannes -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlFKw88ACgkQkX2Xpv6ydvTaoQCePKd61yXxx88XAUP/n1KGDTVQ v0MAnjHVvlDkaewjStLr5vov8xyZNF/6 =wWiI -END PGP SIGNATURE- ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Hi, Here's a small summary of what has to be done, from http://users.suse.com/~aj/linux_lfs.html : " In a nutshell for using LFS you can choose either of the following: Compile your programs with "gcc -D_FILE_OFFSET_BITS=64". This forces all file access calls to use the 64 bit variants. Several types change also, e.g. off_t becomes off64_t. It's therefore important to always use the correct types and to not use e.g. int instead of off_t. For portability with other platforms you should use getconf LFS_CFLAGS which will return -D_FILE_OFFSET_BITS=64 on Linux platforms but might return something else on e.g. Solaris. For linking, you should use the link flags that are reported via getconf LFS_LDFLAGS. On Linux systems, you do not need special link flags. Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With these defines you can use the LFS functions like open64 directly. Use the O_LARGEFILE flag with open to operate on large files. A complete documentation of the feature test macros like _FILE_OFFSET_BITS and _LARGEFILE_SOURCE is in the glibc manual (run e.g. "info libc 'Feature Test Macros'"). The LFS API is also documented in the LFS standard which is available at http://ftp.sas.com/standards/large.file/x_open.20Mar96.html. " Note than open_via_file returns int and not off_t, so the automagic stuff will fail. I guess Miller is right and it should be targeted to 0.45 or 0.46, that's not a small change. Cheers, Charles Ivica Ico Bukvic wrote: > Could one simply redefine lseek and fstat to use lseek64 and fstat64 instead > and be done with it (again, assuming one is not concerned with backwards > and/or x-platform compatibility)? ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Ivica Ico Bukvic wrote: > Also, in Linux is open64 safe for both 32-bit and 64-bit OS variants? Yes, you don't have to switch back to open() on 64-bits systems. So basically switching everything to the 64-bits variants works on most Linux systems installed since about 2002/2003. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2013-03-21 09:37, Charles Goyard wrote: > Note than open_via_file returns int and not off_t, so the > automagic stuff will fail. that's wrong. the declaration of open64 is: > int open64(const char *pathname, int oflag,...); so it returns an int, not an off_t, just like open_via_file(). iiuc, the problem is, that the file descriptor cannot be used by an ordinary lseek(), but only with lseek64(). that's what i was referring to when i said that one would need to provide sys_lseek(). if you go the way of simply defining _FILE_OFFSET_BITS=64 for compiling Pd, you must compile all externals using _FILE_OFFSET_BITS=64 as well, else they won't be able to the file-handle internally. i think this is **very** bad style, as it forces the use of some obscure and potentially harmful compiler flags in order get externals to work. gamsdr IOhannes -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlFK/qsACgkQkX2Xpv6ydvTAbgCgxWfPNZniRrsQ5vYLjcHtzQ3t 2g0An0I6ELjXXHJNTWcMu/mYLCz6kYqQ =43Zn -END PGP SIGNATURE- ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
IOhannes m zmoelnig wrote: > On 2013-03-21 09:37, Charles Goyard wrote: > > Note than open_via_file returns int and not off_t, so the > > automagic stuff will fail. > > that's wrong. > the declaration of open64 is: > > int open64(const char *pathname, int oflag,...); > so it returns an int, not an off_t, just like open_via_file(). Yes, I got mixed up somewhere between filehandles and file positions. However, that's true of lseek(), that returns an off_t, when d_soundfile.c uses long. Sorry for the confusion. -- Charles ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
OK... now I'm trying to fix the bug but I can't reproduce it. I made a 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) and opened and am reading it using readsf~. This is on a 64 bit linux box. Question for Charles Goyard: were you on a 32-bit linux machine? Or did you do something else besides just read the file from its beginning? thanks Miller On Wed, Mar 20, 2013 at 06:20:59PM -0700, Miller Puckette wrote: > I believe not - every extern that used open_bia_path0 would have to be > fixed at the source level to use lseek64(), fstat64() in place of > lseek() and fstat(). I can't see any way to "fix" this source-compatibly. > > I checked open64() on 32-bit Raspbian (ARM) and it worked fine - so I think > it's save to say open64 exists on all modern linuxes, both 64 and 32 bit. > > cheers > Miller > On Wed, Mar 20, 2013 at 09:14:42PM -0400, Ivica Ico Bukvic wrote: > > Miller, > > > > Does this mean if one were not so concerned with backwards compatibility and > > included define you listed below in the s_path.c that this would fix the LFS > > issue albeit at the expense of backwards/cross-platform compatibility? Also, > > in Linux is open64 safe for both 32-bit and 64-bit OS variants? > > > > > -Original Message- > > > From: pd-list-boun...@iem.at [mailto:pd-list-boun...@iem.at] On Behalf Of > > > Miller Puckette > > > Sent: Wednesday, March 20, 2013 6:46 PM > > > To: IOhannes m zmölnig > > > Cc: pd-list@iem.at > > > Subject: Re: [PD] Large File Support on Linux > > > > > > OK.. I think I'm sold (for 0.45 :) on trying to get sys_open, etc., to do > > > the "right" thing. I guess on Windows this would have to be somehow both > > > UTF8 and 64-bit safe - all the more reason to do as you suggest and hide > > the > > > whole mes behind sys_this_and_that() variants. > > > > > > I thought exactly the same thing about a find_via_path routine. It seems > > to > > > me that open_via_path, which tries not just to verify that the file exists > > but > > > also that it can actually be opened, is perhaps working too hard; if a > > file > > > that proves impossible to open is in an earlier spot on the path, the best > > > thing might be to try and fail to open it instead of going out to another, > > > perhaps incorrect, version of the file. > > > > > > Also, it could be fixed to take the path as an argument so that > > d_soundfile.c > > > can call it from other threads safely (using separate copies of the path). > > > > > > cheers > > > Miller > > > > > > On Wed, Mar 20, 2013 at 09:42:42PM +0100, IOhannes m zmölnig wrote: > > > > On 03/20/2013 18:02, Miller Puckette wrote: > > > > > On further thought, I don't think it's even possible to change > > > open_via_path() > > > > > to use open64() and maintain even source compatibility for externs - > > if > > > > > one of them calls lseek or fstat we're sunk - and I don't see any > > robust > > > > > way of aliasing those calls to lseek64() etc. > > > > > > > > yep. > > > > that's why i said that the only real solution would be to provide a > > > > more-or-less complete set of fileIO-functions: sys_lseek, sys_stat > > > > (including f-variants). > > > > > > > > > > > > > > I'm now realizing too that I don't know what approaches the Mac and > > > Microsoft > > > > > pltforms are taking to large file support - I think any solution will > > have to > > > > > somehow address all of the platforms. > > > > > > > > which would be possible if we provided the full set. of file accessors. > > > > (but to repeat, this is not really a Pd-problem, and could (and maybe > > > > should) be solved in an auxiliary lib). > > > > > > > > > For right now I'd like a way to fix 0.44's sfread~ - I'm thinking I > > can do > > > > > that internally to d_soundfile.c so as to cause the least possible > > disruption > > > > > in a 'bugfix' pd release (probably 0.44-3). > > > > > > > > which i think is a good enough approach. > > > > but of course there's a catch: sys_open() will gracefully handle UTF-8 > > > > filenames, whereas on some platforms open() will not. > > > > so with your solution, s
Re: [PD] Large File Support on Linux
Never mind - it stopped after 74 seconds (instead of 450) - presumably 4GB early. But I don't believe this has anything at all to do with my using open() instead of open64() - apparently writesf~ put the wrong number in the soundfile header. cheers Miller On Sat, Mar 23, 2013 at 08:15:47PM -0700, Miller Puckette wrote: > OK... now I'm trying to fix the bug but I can't reproduce it. I made a > 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) > and opened and am reading it using readsf~. This is on a 64 bit linux > box. > > Question for Charles Goyard: were you on a 32-bit linux machine? Or did > you do something else besides just read the file from its beginning? > > thanks > Miller > ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
Hi Miller, Yes I am on 32bit linux. For what I understand on 64 bits Linux LFS is out of the box. The problem stands only for 32 bits hosts it seems. I can follow a branch on git and test for you if you wish. Alternatively I recall there's debian package to host a 32 bits linux in a 64 bits linux. Thanks Charles Miller Puckette wrote: > OK... now I'm trying to fix the bug but I can't reproduce it. I made a > 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) > and opened and am reading it using readsf~. This is on a 64 bit linux > box. > > Question for Charles Goyard: were you on a 32-bit linux machine? Or did > you do something else besides just read the file from its beginning? ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux
No worries I'll just go crank up an old 32 bit linux box myself :) M On Sun, Mar 24, 2013 at 04:07:12PM +0100, Charles Goyard wrote: > Hi Miller, > > Yes I am on 32bit linux. For what I understand on 64 bits Linux LFS is > out of the box. The problem stands only for 32 bits hosts it seems. > > I can follow a branch on git and test for you if you wish. > > Alternatively I recall there's debian package to host a 32 bits linux in > a 64 bits linux. > > Thanks > Charles > > > Miller Puckette wrote: > > OK... now I'm trying to fix the bug but I can't reproduce it. I made a > > 5 GB soundfile (wave format, 60 channels, 4-bit floats, 450 seconds long) > > and opened and am reading it using readsf~. This is on a 64 bit linux > > box. > > > > Question for Charles Goyard: were you on a 32-bit linux machine? Or did > > you do something else besides just read the file from its beginning? > > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux (was: readsf fails to read 32 bits float wav)
Hi, after some research, it seems more related to the size of the file after all. I thought the problem was 32 bits files because it worked when I converted them to 16 bits. But it was just the file size dropped under 2gb. To sum up: a file over about 2Gb fails to open. This is related to how Linux handles large files on 32 bits systems. On my test patch I get the "done" bang as soon as I send the "start" message. Reading the code, it looks like the s_path.c/sys_open() oflags is missing O_LARGEFILE support. pd-extended and pd-l2ork suffer from the same. Am I right there's no Large File Support for pd on 32 bits Linux ? If so that's a major shortcoming, and something a bit heavy to fix. Does it work out of the box on 64 bits linux ? Thanks, -- Charles Hans-Christoph Steiner wrote: > > Sounds like it, especially if you can reproduce it everytime. File a > bug report and include as simple a patch as possible to reproduce the > issue, and the soundfile. > > > today I noticed a readsf~ (on vanilla) opening a 5 wave file containing > > 32 bits float audio fails silently. The doc says 4 bytes is unavailable > > for AIFF, but I use WAVE. > > > > The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz. > > > > Converting the audio to 16 bits PCM works. > > > > Is that a bug ? On vanilla 0.44. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux (was: readsf fails to read 32 bits float wav)
Darn, I just found there's a feature request for just that from IOhannes pending since 2007. https://sourceforge.net/tracker/index.php?func=detail&aid=1638701&group_id=55736&atid=478073 Cheers, -- Charles Charles Goyard wrote: > Hi, > > after some research, it seems more related to the size of the file after > all. I thought the problem was 32 bits files because it worked when I > converted them to 16 bits. But it was just the file size dropped under > 2gb. > > To sum up: a file over about 2Gb fails to open. This is related to how > Linux handles large files on 32 bits systems. > > On my test patch I get the "done" bang as soon as I send the "start" > message. > > Reading the code, it looks like the s_path.c/sys_open() oflags is > missing O_LARGEFILE support. pd-extended and pd-l2ork suffer from the > same. > > Am I right there's no Large File Support for pd on 32 bits Linux ? If so > that's a major shortcoming, and something a bit heavy to fix. Does it > work out of the box on 64 bits linux ? > > Thanks, > > -- > Charles > > Hans-Christoph Steiner wrote: > > > > Sounds like it, especially if you can reproduce it everytime. File a > > bug report and include as simple a patch as possible to reproduce the > > issue, and the soundfile. > > > > > today I noticed a readsf~ (on vanilla) opening a 5 wave file containing > > > 32 bits float audio fails silently. The doc says 4 bytes is unavailable > > > for AIFF, but I use WAVE. > > > > > > The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz. > > > > > > Converting the audio to 16 bits PCM works. > > > > > > Is that a bug ? On vanilla 0.44. ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Re: [PD] Large File Support on Linux (was: readsf fails to read 32 bits float wav)
It's in the source - but Pd has to be compiled with '_LARGEFILE64_SOURCE' defined for this to work. Through 0.43 the configure script I was using was checking for this somehow (too complicted for me to understand how). Anyhow, this is a bug all right - thanks for reporting it! ... and can you tell me what OS you're on so I can check if I'm actually fixing it? thanks Miller On Sat, Mar 16, 2013 at 10:57:46PM +0100, Charles Goyard wrote: > Darn, I just found there's a feature request for just that from IOhannes > pending since 2007. > > https://sourceforge.net/tracker/index.php?func=detail&aid=1638701&group_id=55736&atid=478073 > > Cheers, > -- > Charles > > > Charles Goyard wrote: > > Hi, > > > > after some research, it seems more related to the size of the file after > > all. I thought the problem was 32 bits files because it worked when I > > converted them to 16 bits. But it was just the file size dropped under > > 2gb. > > > > To sum up: a file over about 2Gb fails to open. This is related to how > > Linux handles large files on 32 bits systems. > > > > On my test patch I get the "done" bang as soon as I send the "start" > > message. > > > > Reading the code, it looks like the s_path.c/sys_open() oflags is > > missing O_LARGEFILE support. pd-extended and pd-l2ork suffer from the > > same. > > > > Am I right there's no Large File Support for pd on 32 bits Linux ? If so > > that's a major shortcoming, and something a bit heavy to fix. Does it > > work out of the box on 64 bits linux ? > > > > Thanks, > > > > -- > > Charles > > > > Hans-Christoph Steiner wrote: > > > > > > Sounds like it, especially if you can reproduce it everytime. File a > > > bug report and include as simple a patch as possible to reproduce the > > > issue, and the soundfile. > > > > > > > today I noticed a readsf~ (on vanilla) opening a 5 wave file containing > > > > 32 bits float audio fails silently. The doc says 4 bytes is unavailable > > > > for AIFF, but I use WAVE. > > > > > > > > The file is a 5 channels WAVE file with 5 tracks 32bits float at 48kHz. > > > > > > > > Converting the audio to 16 bits PCM works. > > > > > > > > Is that a bug ? On vanilla 0.44. > > ___ > Pd-list@iem.at mailing list > UNSUBSCRIBE and account-management -> > http://lists.puredata.info/listinfo/pd-list ___ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
