[Wicket-user] Fighting "Too many open files" problem related to wicket resource files
We have found a pretty weird situation with "too many open files" error on our alpha-testing site. Further analysis showed that on each page refresh the following resources get repeatedly obtained from the wicket's .jar and add to the number of open files: 'wicket/ajax/wicket-ajax.js' 'wicket/extensions/ajax/markup/html/modal/res/modal.css' 'wicket/extensions/ajax/markup/html/modal/res/modal.js' 'wicket/extensions/markup/html/tree/res/tree.css' 'wicket/extensions/markup/html/tree/res/tree.js' Eventually (due to garbage collection?) the number of open files goes down again. But we wanted to know why those files stayed open in the first place and were not closed upon retrieving a resource. It looks like switching to the "deployment" mode from "development" one would significantly reduce the peak numbers of the open files/streams to wicket .jar-s, and extracting resources from the .jar would reduce it even better. We were told that the original reason for files staying open is a Java bug (the fact that URLConnection doesnt have a .close), which causes those nasty results when combined with development mode trying to monitor [resource] files for changes and reloading them. We were also told that there is a workaround for that problem in SVN somewhere, but it's probably not backported to 1.2.5 . I have 2 questions in that regard: (1) Where can we find those workarounds in the code? (2) Is it too much work to backport them to 1.2.5 so when it's released it doesn't contain the problem? Thanks, Bob. -- View this message in context: http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 Sent from the Wicket - User mailing list archive at Nabble.com. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Wicket-user mailing list Wicket-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wicket-user
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
can't you run in deployment mode instead of development mode? then that shouldn't happen. I can see if we can backport it. johan On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: We have found a pretty weird situation with "too many open files" error on our alpha-testing site. Further analysis showed that on each page refresh the following resources get repeatedly obtained from the wicket's .jar and add to the number of open files: 'wicket/ajax/wicket-ajax.js' 'wicket/extensions/ajax/markup/html/modal/res/modal.css' 'wicket/extensions/ajax/markup/html/modal/res/modal.js' 'wicket/extensions/markup/html/tree/res/tree.css' 'wicket/extensions/markup/html/tree/res/tree.js' Eventually (due to garbage collection?) the number of open files goes down again. But we wanted to know why those files stayed open in the first place and were not closed upon retrieving a resource. It looks like switching to the "deployment" mode from "development" one would significantly reduce the peak numbers of the open files/streams to wicket .jar-s, and extracting resources from the .jar would reduce it even better. We were told that the original reason for files staying open is a Java bug (the fact that URLConnection doesnt have a .close), which causes those nasty results when combined with development mode trying to monitor [resource] files for changes and reloading them. We were also told that there is a workaround for that problem in SVN somewhere, but it's probably not backported to 1.2.5 . I have 2 questions in that regard: (1) Where can we find those workarounds in the code? (2) Is it too much work to backport them to 1.2.5 so when it's released it doesn't contain the problem? Thanks, Bob. -- View this message in context: http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 Sent from the Wicket - User mailing list archive at Nabble.com. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Wicket-user mailing list Wicket-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wicket-user - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___ Wicket-user mailing list Wicket-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wicket-user
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
We will, when we are on production. Now that we are are still in alpha we prefer "development" (hey, we wrote our first line of wicket code 5-6 weeks ago!) Also, even in deployment mode 'lsof' still shows us a lot of open files (one per resource) if we don't unpack resources from the .jar . It may be smaller number than in development mode, but still... I imagine your "workaround" would close those unnecessarily open files. If you can't backport it, please, tell me where it is in SVN. We'll "hack" it in oursleves for now... Bob Johan Compagner wrote: > > can't you run in deployment mode instead of development mode? > then that shouldn't happen. > > I can see if we can backport it. > > johan > > > On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: >> >> >> We have found a pretty weird situation with "too many open files" error >> on >> our alpha-testing site. Further analysis showed that on each page refresh >> the following resources get repeatedly obtained from the wicket's .jar >> and >> add to the number of open files: >>'wicket/ajax/wicket-ajax.js' >>'wicket/extensions/ajax/markup/html/modal/res/modal.css' >>'wicket/extensions/ajax/markup/html/modal/res/modal.js' >>'wicket/extensions/markup/html/tree/res/tree.css' >>'wicket/extensions/markup/html/tree/res/tree.js' >> >> Eventually (due to garbage collection?) the number of open files goes >> down >> again. But we wanted to know why those files stayed open in the first >> place >> and were not closed upon retrieving a resource. >> >> It looks like switching to the "deployment" mode from "development" one >> would significantly reduce the peak numbers of the open files/streams to >> wicket .jar-s, and extracting resources from the .jar would reduce it >> even >> better. We were told that the original reason for files staying open is a >> Java bug (the fact that URLConnection doesnt have a .close), which causes >> those nasty results when combined with development mode trying to monitor >> [resource] files for changes and reloading them. >> >> We were also told that there is a workaround for that problem in SVN >> somewhere, but it's probably not backported to 1.2.5 . I have 2 questions >> in >> that regard: >> (1) Where can we find those workarounds in the code? >> (2) Is it too much work to backport them to 1.2.5 so when it's released >> it >> doesn't contain the problem? >> >> Thanks, >> Bob. >> -- >> View this message in context: >> http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 >> Sent from the Wicket - User mailing list archive at Nabble.com. >> >> >> - >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job >> easier. >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> ___ >> Wicket-user mailing list >> Wicket-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > - > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier. > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ___ > Wicket-user mailing list > Wicket-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/wicket-user > > -- View this message in context: http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8751579 Sent from the Wicket - User mailing list archive at Nabble.com. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Wicket-user mailing list Wicket-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wicket-user
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
one per resource will i think not really change. johan On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: We will, when we are on production. Now that we are are still in alpha we prefer "development" (hey, we wrote our first line of wicket code 5-6 weeks ago!) Also, even in deployment mode 'lsof' still shows us a lot of open files (one per resource) if we don't unpack resources from the .jar . It may be smaller number than in development mode, but still... I imagine your "workaround" would close those unnecessarily open files. If you can't backport it, please, tell me where it is in SVN. We'll "hack" it in oursleves for now... Bob Johan Compagner wrote: > > can't you run in deployment mode instead of development mode? > then that shouldn't happen. > > I can see if we can backport it. > > johan > > > On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: >> >> >> We have found a pretty weird situation with "too many open files" error >> on >> our alpha-testing site. Further analysis showed that on each page refresh >> the following resources get repeatedly obtained from the wicket's .jar >> and >> add to the number of open files: >>'wicket/ajax/wicket-ajax.js' >>'wicket/extensions/ajax/markup/html/modal/res/modal.css' >>'wicket/extensions/ajax/markup/html/modal/res/modal.js' >>'wicket/extensions/markup/html/tree/res/tree.css' >>'wicket/extensions/markup/html/tree/res/tree.js' >> >> Eventually (due to garbage collection?) the number of open files goes >> down >> again. But we wanted to know why those files stayed open in the first >> place >> and were not closed upon retrieving a resource. >> >> It looks like switching to the "deployment" mode from "development" one >> would significantly reduce the peak numbers of the open files/streams to >> wicket .jar-s, and extracting resources from the .jar would reduce it >> even >> better. We were told that the original reason for files staying open is a >> Java bug (the fact that URLConnection doesnt have a .close), which causes >> those nasty results when combined with development mode trying to monitor >> [resource] files for changes and reloading them. >> >> We were also told that there is a workaround for that problem in SVN >> somewhere, but it's probably not backported to 1.2.5 . I have 2 questions >> in >> that regard: >> (1) Where can we find those workarounds in the code? >> (2) Is it too much work to backport them to 1.2.5 so when it's released >> it >> doesn't contain the problem? >> >> Thanks, >> Bob. >> -- >> View this message in context: >> http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 >> Sent from the Wicket - User mailing list archive at Nabble.com. >> >> >> - >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job >> easier. >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> ___ >> Wicket-user mailing list >> Wicket-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > - > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier. > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ___ > Wicket-user mailing list > Wicket-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/wicket-user > > -- View this message in context: http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8751579 Sent from the Wicket - User mailing list archive at Nabble.com. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Wicket-user mailing list Wicket-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wicket-user - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&k
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
Wasn't it optimized now so that it only hits jars once and only hits per resource where normal files are involved? Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > one per resource will i think not really change. > > > johan > > > On 2/1/07, beboris <[EMAIL PROTECTED] > wrote: > > > > We will, when we are on production. Now that we are are still in alpha we > > prefer "development" (hey, we wrote our first line of wicket code 5-6 > weeks > > ago!) > > > > Also, even in deployment mode 'lsof' still shows us a lot of open files > (one > > per resource) if we don't unpack resources from the .jar . It may be > smaller > > number than in development mode, but still... I imagine your "workaround" > > would close those unnecessarily open files. > > > > If you can't backport it, please, tell me where it is in SVN. We'll "hack" > > it in oursleves for now... > > > > Bob > > > > > > Johan Compagner wrote: > > > > > > can't you run in deployment mode instead of development mode? > > > then that shouldn't happen. > > > > > > I can see if we can backport it. > > > > > > johan > > > > > > > > > On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: > > >> > > >> > > >> We have found a pretty weird situation with "too many open files" error > > >> on > > >> our alpha-testing site. Further analysis showed that on each page > refresh > > >> the following resources get repeatedly obtained from the wicket's .jar > > >> and > > >> add to the number of open files: > > >>'wicket/ajax/wicket-ajax.js' > > >> > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > >> > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > >>'wicket/extensions/markup/html/tree/res/tree.css' > > >>'wicket/extensions/markup/html/tree/res/tree.js' > > >> > > >> Eventually (due to garbage collection?) the number of open files goes > > >> down > > >> again. But we wanted to know why those files stayed open in the first > > >> place > > >> and were not closed upon retrieving a resource. > > >> > > >> It looks like switching to the "deployment" mode from "development" one > > >> would significantly reduce the peak numbers of the open files/streams > to > > >> wicket .jar-s, and extracting resources from the .jar would reduce it > > >> even > > >> better. We were told that the original reason for files staying open is > a > > >> Java bug (the fact that URLConnection doesnt have a .close), which > causes > > >> those nasty results when combined with development mode trying to > monitor > > >> [resource] files for changes and reloading them. > > >> > > >> We were also told that there is a workaround for that problem in SVN > > >> somewhere, but it's probably not backported to 1.2.5 . I have 2 > questions > > >> in > > >> that regard: > > >> (1) Where can we find those workarounds in the code? > > >> (2) Is it too much work to backport them to 1.2.5 so when it's released > > >> it > > >> doesn't contain the problem? > > >> > > >> Thanks, > > >> Bob. > > >> -- > > >> View this message in context: > > >> > http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 > > >> Sent from the Wicket - User mailing list archive at Nabble.com . > > >> > > >> > > >> > - > > >> Using Tomcat but need to do more? Need to support web services, > security? > > >> Get stuff done quickly with pre-integrated technology to make your job > > >> easier. > > >> Download IBM WebSphere Application Server v.1.0.1 based on Apache > > >> Geronimo > > >> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > >> ___ > > >> Wicket-user mailing list > > >> Wicket-user@lists.sourceforge.net > > >> > https://lists.sourceforge.net/lists/listinfo/wicket-user > > >> > > > > > > > - > > > Using Tomcat but need to do more? Need to support web services, > security? > > > Get stuff done quickly with pre-integrated technology to make your job > > > easier. > > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > ___ > > > Wicket-user mailing list > > > Wicket-user@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/wicket-user > > > > > > > > > > -- > > View this message in context: > http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8751579 > > Sent from the Wicket - User mailing list archive at Nabble.com . > > > > > > > - > > Using Tomcat but need to do more? Need to support web services, security? > > Get stuff done quickly with pre-integrated technology to make your job > easier. > > Dow
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
yes the modification checker. But we do need to really load the resource out of the jar file once. So that file handle will be used. johan On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: Wasn't it optimized now so that it only hits jars once and only hits per resource where normal files are involved? Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > one per resource will i think not really change. > > > johan > > > On 2/1/07, beboris <[EMAIL PROTECTED] > wrote: > > > > We will, when we are on production. Now that we are are still in alpha we > > prefer "development" (hey, we wrote our first line of wicket code 5-6 > weeks > > ago!) > > > > Also, even in deployment mode 'lsof' still shows us a lot of open files > (one > > per resource) if we don't unpack resources from the .jar . It may be > smaller > > number than in development mode, but still... I imagine your "workaround" > > would close those unnecessarily open files. > > > > If you can't backport it, please, tell me where it is in SVN. We'll "hack" > > it in oursleves for now... > > > > Bob > > > > > > Johan Compagner wrote: > > > > > > can't you run in deployment mode instead of development mode? > > > then that shouldn't happen. > > > > > > I can see if we can backport it. > > > > > > johan > > > > > > > > > On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: > > >> > > >> > > >> We have found a pretty weird situation with "too many open files" error > > >> on > > >> our alpha-testing site. Further analysis showed that on each page > refresh > > >> the following resources get repeatedly obtained from the wicket's .jar > > >> and > > >> add to the number of open files: > > >>'wicket/ajax/wicket-ajax.js' > > >> > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > >> > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > >>'wicket/extensions/markup/html/tree/res/tree.css' > > >>'wicket/extensions/markup/html/tree/res/tree.js' > > >> > > >> Eventually (due to garbage collection?) the number of open files goes > > >> down > > >> again. But we wanted to know why those files stayed open in the first > > >> place > > >> and were not closed upon retrieving a resource. > > >> > > >> It looks like switching to the "deployment" mode from "development" one > > >> would significantly reduce the peak numbers of the open files/streams > to > > >> wicket .jar-s, and extracting resources from the .jar would reduce it > > >> even > > >> better. We were told that the original reason for files staying open is > a > > >> Java bug (the fact that URLConnection doesnt have a .close), which > causes > > >> those nasty results when combined with development mode trying to > monitor > > >> [resource] files for changes and reloading them. > > >> > > >> We were also told that there is a workaround for that problem in SVN > > >> somewhere, but it's probably not backported to 1.2.5 . I have 2 > questions > > >> in > > >> that regard: > > >> (1) Where can we find those workarounds in the code? > > >> (2) Is it too much work to backport them to 1.2.5 so when it's released > > >> it > > >> doesn't contain the problem? > > >> > > >> Thanks, > > >> Bob. > > >> -- > > >> View this message in context: > > >> > http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 > > >> Sent from the Wicket - User mailing list archive at Nabble.com . > > >> > > >> > > >> > - > > >> Using Tomcat but need to do more? Need to support web services, > security? > > >> Get stuff done quickly with pre-integrated technology to make your job > > >> easier. > > >> Download IBM WebSphere Application Server v.1.0.1 based on Apache > > >> Geronimo > > >> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > >> ___ > > >> Wicket-user mailing list > > >> Wicket-user@lists.sourceforge.net > > >> > https://lists.sourceforge.net/lists/listinfo/wicket-user > > >> > > > > > > > - > > > Using Tomcat but need to do more? Need to support web services, > security? > > > Get stuff done quickly with pre-integrated technology to make your job > > > easier. > > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > ___ > > > Wicket-user mailing list > > > Wicket-user@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/wicket-user > > > > > > > > > > -- > > View this message in context: > http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8751579 > > Sent from the Wicket - User mailing list archive at Nabble.com . > > > > > > > -
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
Yeah, but that would be always one fd for a jar, no matter how many files in it that have to be read, right? Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > yes the modification checker. > But we do need to really load the resource out of the jar file once. So that > file handle will be used. > > johan > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: > > > > Wasn't it optimized now so that it only hits jars once and only hits > > per resource where normal files are involved? > > > > Eelco > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: > > > one per resource will i think not really change. > > > > > > > > > johan > > > > > > > > > On 2/1/07, beboris <[EMAIL PROTECTED] > wrote: > > > > > > > > We will, when we are on production. Now that we are are still in alpha > we > > > > prefer "development" (hey, we wrote our first line of wicket code 5-6 > > > weeks > > > > ago!) > > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot of open > files > > > (one > > > > per resource) if we don't unpack resources from the .jar . It may be > > > smaller > > > > number than in development mode, but still... I imagine your > "workaround" > > > > would close those unnecessarily open files. > > > > > > > > If you can't backport it, please, tell me where it is in SVN. We'll > "hack" > > > > it in oursleves for now... > > > > > > > > Bob > > > > > > > > > > > > Johan Compagner wrote: > > > > > > > > > > can't you run in deployment mode instead of development mode? > > > > > then that shouldn't happen. > > > > > > > > > > I can see if we can backport it. > > > > > > > > > > johan > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: > > > > >> > > > > >> > > > > >> We have found a pretty weird situation with "too many open files" > error > > > > >> on > > > > >> our alpha-testing site. Further analysis showed that on each page > > > refresh > > > > >> the following resources get repeatedly obtained from the wicket's > .jar > > > > >> and > > > > >> add to the number of open files: > > > > >>'wicket/ajax/wicket-ajax.js' > > > > >> > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > > > >> > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > > > >> > 'wicket/extensions/markup/html/tree/res/tree.css' > > > > >> > 'wicket/extensions/markup/html/tree/res/tree.js' > > > > >> > > > > >> Eventually (due to garbage collection?) the number of open files > goes > > > > >> down > > > > >> again. But we wanted to know why those files stayed open in the > first > > > > >> place > > > > >> and were not closed upon retrieving a resource. > > > > >> > > > > >> It looks like switching to the "deployment" mode from "development" > one > > > > >> would significantly reduce the peak numbers of the open > files/streams > > > to > > > > >> wicket .jar-s, and extracting resources from the .jar would reduce > it > > > > >> even > > > > >> better. We were told that the original reason for files staying > open is > > > a > > > > >> Java bug (the fact that URLConnection doesnt have a .close), which > > > causes > > > > >> those nasty results when combined with development mode trying to > > > monitor > > > > >> [resource] files for changes and reloading them. > > > > >> > > > > >> We were also told that there is a workaround for that problem in > SVN > > > > >> somewhere, but it's probably not backported to 1.2.5 . I have 2 > > > questions > > > > >> in > > > > >> that regard: > > > > >> (1) Where can we find those workarounds in the code? > > > > >> (2) Is it too much work to backport them to 1.2.5 so when it's > released > > > > >> it > > > > >> doesn't contain the problem? > > > > >> > > > > >> Thanks, > > > > >> Bob. > > > > >> -- > > > > >> View this message in context: > > > > >> > > > > http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 > > > > >> Sent from the Wicket - User mailing list archive at Nabble.com . > > > > >> > > > > >> > > > > >> > > > > - > > > > >> Using Tomcat but need to do more? Need to support web services, > > > security? > > > > >> Get stuff done quickly with pre-integrated technology to make your > job > > > > >> easier. > > > > >> Download IBM WebSphere Application Server v.1.0.1 based on Apache > > > > >> Geronimo > > > > >> > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > > >> ___ > > > > >> Wicket-user mailing list > > > > >> Wicket-user@lists.sourceforge.net > > > > >> > > > > https://lists.sourceforge.net/lists/listinfo/wicket-user > > > > >> > > > > > > > > > > > > > > - > > > > > Using Tomcat but need to do more? Need to support web services, > > > security? > > > > > Get stuff done quickly with pre-integrat
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
that is what you would think... But why generates a modification check one file handle for every check in the file? because UrlConnection.connect() has again a JarUrlConnection internally that makes a new connection to that jar file and UrlConnection does have a connect() but not a disconnect() so you can't clear it. johan On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: Yeah, but that would be always one fd for a jar, no matter how many files in it that have to be read, right? Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > yes the modification checker. > But we do need to really load the resource out of the jar file once. So that > file handle will be used. > > johan > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: > > > > Wasn't it optimized now so that it only hits jars once and only hits > > per resource where normal files are involved? > > > > Eelco > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: > > > one per resource will i think not really change. > > > > > > > > > johan > > > > > > > > > On 2/1/07, beboris <[EMAIL PROTECTED] > wrote: > > > > > > > > We will, when we are on production. Now that we are are still in alpha > we > > > > prefer "development" (hey, we wrote our first line of wicket code 5-6 > > > weeks > > > > ago!) > > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot of open > files > > > (one > > > > per resource) if we don't unpack resources from the .jar . It may be > > > smaller > > > > number than in development mode, but still... I imagine your > "workaround" > > > > would close those unnecessarily open files. > > > > > > > > If you can't backport it, please, tell me where it is in SVN. We'll > "hack" > > > > it in oursleves for now... > > > > > > > > Bob > > > > > > > > > > > > Johan Compagner wrote: > > > > > > > > > > can't you run in deployment mode instead of development mode? > > > > > then that shouldn't happen. > > > > > > > > > > I can see if we can backport it. > > > > > > > > > > johan > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: > > > > >> > > > > >> > > > > >> We have found a pretty weird situation with "too many open files" > error > > > > >> on > > > > >> our alpha-testing site. Further analysis showed that on each page > > > refresh > > > > >> the following resources get repeatedly obtained from the wicket's > .jar > > > > >> and > > > > >> add to the number of open files: > > > > >>'wicket/ajax/wicket-ajax.js' > > > > >> > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > > > >> > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > > > >> > 'wicket/extensions/markup/html/tree/res/tree.css' > > > > >> > 'wicket/extensions/markup/html/tree/res/tree.js' > > > > >> > > > > >> Eventually (due to garbage collection?) the number of open files > goes > > > > >> down > > > > >> again. But we wanted to know why those files stayed open in the > first > > > > >> place > > > > >> and were not closed upon retrieving a resource. > > > > >> > > > > >> It looks like switching to the "deployment" mode from "development" > one > > > > >> would significantly reduce the peak numbers of the open > files/streams > > > to > > > > >> wicket .jar-s, and extracting resources from the .jar would reduce > it > > > > >> even > > > > >> better. We were told that the original reason for files staying > open is > > > a > > > > >> Java bug (the fact that URLConnection doesnt have a .close), which > > > causes > > > > >> those nasty results when combined with development mode trying to > > > monitor > > > > >> [resource] files for changes and reloading them. > > > > >> > > > > >> We were also told that there is a workaround for that problem in > SVN > > > > >> somewhere, but it's probably not backported to 1.2.5 . I have 2 > > > questions > > > > >> in > > > > >> that regard: > > > > >> (1) Where can we find those workarounds in the code? > > > > >> (2) Is it too much work to backport them to 1.2.5 so when it's > released > > > > >> it > > > > >> doesn't contain the problem? > > > > >> > > > > >> Thanks, > > > > >> Bob. > > > > >> -- > > > > >> View this message in context: > > > > >> > > > > http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 > > > > >> Sent from the Wicket - User mailing list archive at Nabble.com. > > > > >> > > > > >> > > > > >> > > > > - > > > > >> Using Tomcat but need to do more? Need to support web services, > > > security? > > > > >> Get stuff done quickly with pre-integrated technology to make your > job > > > > >> easier. > > > > >> Download IBM WebSphere Application Server v.1.0.1 based on Apache > > > > >> Geronimo > > > > >> > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > > >> ___ > > > > >> Wicket-user ma
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
Would it be possible and useful to cache the URL connection? Does it have a time out and/ or does it use an exclusive lock? Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > that is what you would think... But why generates a modification check one > file handle for every check in the file? > > because UrlConnection.connect() has again a JarUrlConnection internally that > makes a new connection to that jar file > and UrlConnection does have a connect() but not a disconnect() so you can't > clear it. > > johan > > > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: > > Yeah, but that would be always one fd for a jar, no matter how many > > files in it that have to be read, right? > > > > Eelco > > > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > > > yes the modification checker. > > > But we do need to really load the resource out of the jar file once. So > that > > > file handle will be used. > > > > > > johan > > > > > > > > > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: > > > > > > > > Wasn't it optimized now so that it only hits jars once and only hits > > > > per resource where normal files are involved? > > > > > > > > Eelco > > > > > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: > > > > > one per resource will i think not really change. > > > > > > > > > > > > > > > johan > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED] > wrote: > > > > > > > > > > > > We will, when we are on production. Now that we are are still in > alpha > > > we > > > > > > prefer "development" (hey, we wrote our first line of wicket code > 5-6 > > > > > weeks > > > > > > ago!) > > > > > > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot of open > > > files > > > > > (one > > > > > > per resource) if we don't unpack resources from the .jar . It may > be > > > > > smaller > > > > > > number than in development mode, but still... I imagine your > > > "workaround" > > > > > > would close those unnecessarily open files. > > > > > > > > > > > > If you can't backport it, please, tell me where it is in SVN. > We'll > > > "hack" > > > > > > it in oursleves for now... > > > > > > > > > > > > Bob > > > > > > > > > > > > > > > > > > Johan Compagner wrote: > > > > > > > > > > > > > > can't you run in deployment mode instead of development mode? > > > > > > > then that shouldn't happen. > > > > > > > > > > > > > > I can see if we can backport it. > > > > > > > > > > > > > > johan > > > > > > > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: > > > > > > >> > > > > > > >> > > > > > > >> We have found a pretty weird situation with "too many open > files" > > > error > > > > > > >> on > > > > > > >> our alpha-testing site. Further analysis showed that on each > page > > > > > refresh > > > > > > >> the following resources get repeatedly obtained from the > wicket's > > > .jar > > > > > > >> and > > > > > > >> add to the number of open files: > > > > > > >>'wicket/ajax/wicket-ajax.js' > > > > > > >> > > > > > > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > > > > > >> > > > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > > > > > >> > > > 'wicket/extensions/markup/html/tree/res/tree.css' > > > > > > >> > > > 'wicket/extensions/markup/html/tree/res/tree.js' > > > > > > >> > > > > > > >> Eventually (due to garbage collection?) the number of open > files > > > goes > > > > > > >> down > > > > > > >> again. But we wanted to know why those files stayed open in the > > > first > > > > > > >> place > > > > > > >> and were not closed upon retrieving a resource. > > > > > > >> > > > > > > >> It looks like switching to the "deployment" mode from > "development" > > > one > > > > > > >> would significantly reduce the peak numbers of the open > > > files/streams > > > > > to > > > > > > >> wicket .jar-s, and extracting resources from the .jar would > reduce > > > it > > > > > > >> even > > > > > > >> better. We were told that the original reason for files staying > > > open is > > > > > a > > > > > > >> Java bug (the fact that URLConnection doesnt have a .close), > which > > > > > causes > > > > > > >> those nasty results when combined with development mode trying > to > > > > > monitor > > > > > > >> [resource] files for changes and reloading them. > > > > > > >> > > > > > > >> We were also told that there is a workaround for that problem > in > > > SVN > > > > > > >> somewhere, but it's probably not backported to 1.2.5 . I have 2 > > > > > questions > > > > > > >> in > > > > > > >> that regard: > > > > > > >> (1) Where can we find those workarounds in the code? > > > > > > >> (2) Is it too much work to backport them to 1.2.5 so when it's > > > released > > > > > > >> it > > > > > > >> doesn't contain the problem? > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Bob. > > > > > > >> -- > > > > > > >> View this message in context: > > > > > > >> > > > > > > > > > http://www.nabble.com
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
why would you cache? and which one? the url connection to an entry in a jar file (thats the JarUrlConnection) or (i guess) the FileUrlConnection (to the jar file itself) both don't make much sense to cache the first one we don't need to cache we only need to use it once by really loading the resource and i guess when it is finalized it is cleaned up. We already don't use it anymore for the last modified. Because there we use only the second one So the fileUrlConnection to the jarFile itself thats is inside the JarUrlConnection object. on that one we call last modified everytime, But that will not cause the file to open. (because it doesn't have to read the file itself) And we can't construct JarUrlConnections (for reading the jar entries) with the same file url connection because there is no way to initialize the jar url connection directly with the file url connection so they all would use the same. johan On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: Would it be possible and useful to cache the URL connection? Does it have a time out and/ or does it use an exclusive lock? Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > that is what you would think... But why generates a modification check one > file handle for every check in the file? > > because UrlConnection.connect() has again a JarUrlConnection internally that > makes a new connection to that jar file > and UrlConnection does have a connect() but not a disconnect() so you can't > clear it. > > johan > > > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: > > Yeah, but that would be always one fd for a jar, no matter how many > > files in it that have to be read, right? > > > > Eelco > > > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > > > yes the modification checker. > > > But we do need to really load the resource out of the jar file once. So > that > > > file handle will be used. > > > > > > johan > > > > > > > > > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: > > > > > > > > Wasn't it optimized now so that it only hits jars once and only hits > > > > per resource where normal files are involved? > > > > > > > > Eelco > > > > > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: > > > > > one per resource will i think not really change. > > > > > > > > > > > > > > > johan > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED] > wrote: > > > > > > > > > > > > We will, when we are on production. Now that we are are still in > alpha > > > we > > > > > > prefer "development" (hey, we wrote our first line of wicket code > 5-6 > > > > > weeks > > > > > > ago!) > > > > > > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot of open > > > files > > > > > (one > > > > > > per resource) if we don't unpack resources from the .jar . It may > be > > > > > smaller > > > > > > number than in development mode, but still... I imagine your > > > "workaround" > > > > > > would close those unnecessarily open files. > > > > > > > > > > > > If you can't backport it, please, tell me where it is in SVN. > We'll > > > "hack" > > > > > > it in oursleves for now... > > > > > > > > > > > > Bob > > > > > > > > > > > > > > > > > > Johan Compagner wrote: > > > > > > > > > > > > > > can't you run in deployment mode instead of development mode? > > > > > > > then that shouldn't happen. > > > > > > > > > > > > > > I can see if we can backport it. > > > > > > > > > > > > > > johan > > > > > > > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: > > > > > > >> > > > > > > >> > > > > > > >> We have found a pretty weird situation with "too many open > files" > > > error > > > > > > >> on > > > > > > >> our alpha-testing site. Further analysis showed that on each > page > > > > > refresh > > > > > > >> the following resources get repeatedly obtained from the > wicket's > > > .jar > > > > > > >> and > > > > > > >> add to the number of open files: > > > > > > >>'wicket/ajax/wicket-ajax.js' > > > > > > >> > > > > > > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > > > > > >> > > > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > > > > > >> > > > 'wicket/extensions/markup/html/tree/res/tree.css' > > > > > > >> > > > 'wicket/extensions/markup/html/tree/res/tree.js' > > > > > > >> > > > > > > >> Eventually (due to garbage collection?) the number of open > files > > > goes > > > > > > >> down > > > > > > >> again. But we wanted to know why those files stayed open in the > > > first > > > > > > >> place > > > > > > >> and were not closed upon retrieving a resource. > > > > > > >> > > > > > > >> It looks like switching to the "deployment" mode from > "development" > > > one > > > > > > >> would significantly reduce the peak numbers of the open > > > files/streams > > > > > to > > > > > > >> wicket .jar-s, and extracting resources from the .jar would > reduce > > > it > > > > > > >> even > > > > > > >> better. We were told tha
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
Too bad, Eelco On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > why would you cache? and which one? the url connection to an entry in a jar > file (thats the JarUrlConnection) > or (i guess) the FileUrlConnection (to the jar file itself) > > both don't make much sense to cache > the first one we don't need to cache we only need to use it once by really > loading the resource > and i guess when it is finalized it is cleaned up. > We already don't use it anymore for the last modified. Because there we use > only the second one > So the fileUrlConnection to the jarFile itself thats is inside the > JarUrlConnection object. > on that one we call last modified everytime, But that will not cause the > file to open. (because it doesn't have to read the file itself) > > And we can't construct JarUrlConnections (for reading the jar entries) with > the same file url connection because there is > no way to initialize the jar url connection directly with the file url > connection so they all would use the same. > > johan > > > > On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: > > > > Would it be possible and useful to cache the URL connection? Does it > > have a time out and/ or does it use an exclusive lock? > > > > Eelco > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: > > > that is what you would think... But why generates a modification check > one > > > file handle for every check in the file? > > > > > > because UrlConnection.connect() has again a JarUrlConnection internally > that > > > makes a new connection to that jar file > > > and UrlConnection does have a connect() but not a disconnect() so you > can't > > > clear it. > > > > > > johan > > > > > > > > > > > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: > > > > Yeah, but that would be always one fd for a jar, no matter how many > > > > files in it that have to be read, right? > > > > > > > > Eelco > > > > > > > > > > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: > > > > > yes the modification checker. > > > > > But we do need to really load the resource out of the jar file once. > So > > > that > > > > > file handle will be used. > > > > > > > > > > johan > > > > > > > > > > > > > > > > > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > Wasn't it optimized now so that it only hits jars once and only > hits > > > > > > per resource where normal files are involved? > > > > > > > > > > > > Eelco > > > > > > > > > > > > On 2/1/07, Johan Compagner < [EMAIL PROTECTED] > wrote: > > > > > > > one per resource will i think not really change. > > > > > > > > > > > > > > > > > > > > > johan > > > > > > > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED] > wrote: > > > > > > > > > > > > > > > > We will, when we are on production. Now that we are are still > in > > > alpha > > > > > we > > > > > > > > prefer "development" (hey, we wrote our first line of wicket > code > > > 5-6 > > > > > > > weeks > > > > > > > > ago!) > > > > > > > > > > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot of > open > > > > > files > > > > > > > (one > > > > > > > > per resource) if we don't unpack resources from the .jar . It > may > > > be > > > > > > > smaller > > > > > > > > number than in development mode, but still... I imagine your > > > > > "workaround" > > > > > > > > would close those unnecessarily open files. > > > > > > > > > > > > > > > > If you can't backport it, please, tell me where it is in SVN. > > > We'll > > > > > "hack" > > > > > > > > it in oursleves for now... > > > > > > > > > > > > > > > > Bob > > > > > > > > > > > > > > > > > > > > > > > > Johan Compagner wrote: > > > > > > > > > > > > > > > > > > can't you run in deployment mode instead of development > mode? > > > > > > > > > then that shouldn't happen. > > > > > > > > > > > > > > > > > > I can see if we can backport it. > > > > > > > > > > > > > > > > > > johan > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> We have found a pretty weird situation with "too many open > > > files" > > > > > error > > > > > > > > >> on > > > > > > > > >> our alpha-testing site. Further analysis showed that on > each > > > page > > > > > > > refresh > > > > > > > > >> the following resources get repeatedly obtained from the > > > wicket's > > > > > .jar > > > > > > > > >> and > > > > > > > > >> add to the number of open files: > > > > > > > > >>'wicket/ajax/wicket-ajax.js' > > > > > > > > >> > > > > > > > > > > > > > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.css' > > > > > > > > >> > > > > > > > > > > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' > > > > > > > > >> > > > > > 'wicket/extensions/markup/html/tree/res/tree.css' > > > > > > > > >> > > > > > 'wicket/extensions/markup/html/tree/res/tree.js' > > > > > > > > >> > > > > > > > > >> Eventually (due to garbage colle
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
It's hard for me to say, what the right way to fix this situation is. The number of open files (that look to the OS as wicket .jar files) will go up significantly and then go down (down - probably due to the garbage collection). It is during those spikes caused by many users hitting wicket pages around the same time that we got "too many open files" error. It almost looks like the only way to really eliminate those unnecessary open files to numerous Wicket resources (and to guarantee "too many open files" will never hit us) is to unpack those resources from the wicket .jar-s and put them into WEB-INF/classes directory instead. Once we did it, we didn't see all those dozens (and even hundreds) open files ever again... Bob. Eelco Hillenius wrote: > > Too bad, > > Eelco > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: >> why would you cache? and which one? the url connection to an entry in a >> jar >> file (thats the JarUrlConnection) >> or (i guess) the FileUrlConnection (to the jar file itself) >> >> both don't make much sense to cache >> the first one we don't need to cache we only need to use it once by >> really >> loading the resource >> and i guess when it is finalized it is cleaned up. >> We already don't use it anymore for the last modified. Because there we >> use >> only the second one >> So the fileUrlConnection to the jarFile itself thats is inside the >> JarUrlConnection object. >> on that one we call last modified everytime, But that will not cause the >> file to open. (because it doesn't have to read the file itself) >> >> And we can't construct JarUrlConnections (for reading the jar entries) >> with >> the same file url connection because there is >> no way to initialize the jar url connection directly with the file url >> connection so they all would use the same. >> >> johan >> >> >> >> On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > >> > Would it be possible and useful to cache the URL connection? Does it >> > have a time out and/ or does it use an exclusive lock? >> > >> > Eelco >> > >> > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: >> > > that is what you would think... But why generates a modification >> check >> one >> > > file handle for every check in the file? >> > > >> > > because UrlConnection.connect() has again a JarUrlConnection >> internally >> that >> > > makes a new connection to that jar file >> > > and UrlConnection does have a connect() but not a disconnect() so you >> can't >> > > clear it. >> > > >> > > johan >> > > >> > > >> > > >> > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: >> > > > Yeah, but that would be always one fd for a jar, no matter how many >> > > > files in it that have to be read, right? >> > > > >> > > > Eelco >> > > > >> > > > >> > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: >> > > > > yes the modification checker. >> > > > > But we do need to really load the resource out of the jar file >> once. >> So >> > > that >> > > > > file handle will be used. >> > > > > >> > > > > johan >> > > > > >> > > > > >> > > > > >> > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > > > > > >> > > > > > Wasn't it optimized now so that it only hits jars once and only >> hits >> > > > > > per resource where normal files are involved? >> > > > > > >> > > > > > Eelco >> > > > > > >> > > > > > On 2/1/07, Johan Compagner < [EMAIL PROTECTED] > wrote: >> > > > > > > one per resource will i think not really change. >> > > > > > > >> > > > > > > >> > > > > > > johan >> > > > > > > >> > > > > > > >> > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED] > wrote: >> > > > > > > > >> > > > > > > > We will, when we are on production. Now that we are are >> still >> in >> > > alpha >> > > > > we >> > > > > > > > prefer "development" (hey, we wrote our first line of >> wicket >> code >> > > 5-6 >> > > > > > > weeks >> > > > > > > > ago!) >> > > > > > > > >> > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot >> of >> open >> > > > > files >> > > > > > > (one >> > > > > > > > per resource) if we don't unpack resources from the .jar . >> It >> may >> > > be >> > > > > > > smaller >> > > > > > > > number than in development mode, but still... I imagine >> your >> > > > > "workaround" >> > > > > > > > would close those unnecessarily open files. >> > > > > > > > >> > > > > > > > If you can't backport it, please, tell me where it is in >> SVN. >> > > We'll >> > > > > "hack" >> > > > > > > > it in oursleves for now... >> > > > > > > > >> > > > > > > > Bob >> > > > > > > > >> > > > > > > > >> > > > > > > > Johan Compagner wrote: >> > > > > > > > > >> > > > > > > > > can't you run in deployment mode instead of development >> mode? >> > > > > > > > > then that shouldn't happen. >> > > > > > > > > >> > > > > > > > > I can see if we can backport it. >> > > > > > > > > >> > > > > > > > > johan >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: >> > >
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
the os should be able to handle at least a few thousand open file handles. furthermore you can configure it to have more if you need to. only one handle should be open per resource (no way to avoid that) in a jar, so a spike of user activity shouldnt affect this at all as resources cache globally. but yes, unpacking should eliminate these problems. i wish there was an option in tomcat to unpack the jars on deployment, it can already do this for the war. -igor On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: It's hard for me to say, what the right way to fix this situation is. The number of open files (that look to the OS as wicket .jar files) will go up significantly and then go down (down - probably due to the garbage collection). It is during those spikes caused by many users hitting wicket pages around the same time that we got "too many open files" error. It almost looks like the only way to really eliminate those unnecessary open files to numerous Wicket resources (and to guarantee "too many open files" will never hit us) is to unpack those resources from the wicket .jar-s and put them into WEB-INF/classes directory instead. Once we did it, we didn't see all those dozens (and even hundreds) open files ever again... Bob. Eelco Hillenius wrote: > > Too bad, > > Eelco > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: >> why would you cache? and which one? the url connection to an entry in a >> jar >> file (thats the JarUrlConnection) >> or (i guess) the FileUrlConnection (to the jar file itself) >> >> both don't make much sense to cache >> the first one we don't need to cache we only need to use it once by >> really >> loading the resource >> and i guess when it is finalized it is cleaned up. >> We already don't use it anymore for the last modified. Because there we >> use >> only the second one >> So the fileUrlConnection to the jarFile itself thats is inside the >> JarUrlConnection object. >> on that one we call last modified everytime, But that will not cause the >> file to open. (because it doesn't have to read the file itself) >> >> And we can't construct JarUrlConnections (for reading the jar entries) >> with >> the same file url connection because there is >> no way to initialize the jar url connection directly with the file url >> connection so they all would use the same. >> >> johan >> >> >> >> On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > >> > Would it be possible and useful to cache the URL connection? Does it >> > have a time out and/ or does it use an exclusive lock? >> > >> > Eelco >> > >> > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: >> > > that is what you would think... But why generates a modification >> check >> one >> > > file handle for every check in the file? >> > > >> > > because UrlConnection.connect() has again a JarUrlConnection >> internally >> that >> > > makes a new connection to that jar file >> > > and UrlConnection does have a connect() but not a disconnect() so you >> can't >> > > clear it. >> > > >> > > johan >> > > >> > > >> > > >> > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: >> > > > Yeah, but that would be always one fd for a jar, no matter how many >> > > > files in it that have to be read, right? >> > > > >> > > > Eelco >> > > > >> > > > >> > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: >> > > > > yes the modification checker. >> > > > > But we do need to really load the resource out of the jar file >> once. >> So >> > > that >> > > > > file handle will be used. >> > > > > >> > > > > johan >> > > > > >> > > > > >> > > > > >> > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > > > > > >> > > > > > Wasn't it optimized now so that it only hits jars once and only >> hits >> > > > > > per resource where normal files are involved? >> > > > > > >> > > > > > Eelco >> > > > > > >> > > > > > On 2/1/07, Johan Compagner < [EMAIL PROTECTED] > wrote: >> > > > > > > one per resource will i think not really change. >> > > > > > > >> > > > > > > >> > > > > > > johan >> > > > > > > >> > > > > > > >> > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED] > wrote: >> > > > > > > > >> > > > > > > > We will, when we are on production. Now that we are are >> still >> in >> > > alpha >> > > > > we >> > > > > > > > prefer "development" (hey, we wrote our first line of >> wicket >> code >> > > 5-6 >> > > > > > > weeks >> > > > > > > > ago!) >> > > > > > > > >> > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot >> of >> open >> > > > > files >> > > > > > > (one >> > > > > > > > per resource) if we don't unpack resources from the .jar . >> It >> may >> > > be >> > > > > > > smaller >> > > > > > > > number than in development mode, but still... I imagine >> your >> > > > > "workaround" >> > > > > > > > would close those unnecessarily open files. >> > > > > > > > >> > > > > > > > If you can't backport it, please, tell me where it is in >> SVN. >> > > We'll >> > > > > "hack" >> > > > > > > > it
Re: [Wicket-user] Fighting "Too many open files" problem related to wicket resource files
Thanks for the hints. Between those and what we've already known, we should be fine... Bob. Johan Compagner wrote: > > Tomcat can do that: > > http://tomcat.apache.org/tomcat-5.5-doc/config/context.html > > see anti resource and anti jar locking. > > I think the end result will be that tomcat extracts everything. > (dont know which one it really is) > > But as igor said, a high load of users shouldn't affect the number of open > files to the wicket jar > if you are in deployment mode. If you are in development mode then > 1.2.4still have the problem > that the modification watcher does a check every second or so. And that > will > increase open file > handles a lot. > > johan > > > On 2/2/07, Igor Vaynberg <[EMAIL PROTECTED]> wrote: >> >> the os should be able to handle at least a few thousand open file >> handles. >> furthermore you can configure it to have more if you need to. only one >> handle should be open per resource (no way to avoid that) in a jar, so a >> spike of user activity shouldnt affect this at all as resources cache >> globally. >> >> but yes, unpacking should eliminate these problems. i wish there was an >> option in tomcat to unpack the jars on deployment, it can already do this >> for the war. >> >> -igor >> >> On 2/1/07, beboris <[EMAIL PROTECTED]> wrote: >> > >> > >> > It's hard for me to say, what the right way to fix this situation is. >> > The >> > number of open files (that look to the OS as wicket .jar files) will go >> > up >> > significantly and then go down (down - probably due to the garbage >> > collection). It is during those spikes caused by many users hitting >> > wicket >> > pages around the same time that we got "too many open files" error. It >> > almost looks like the only way to really eliminate those unnecessary >> > open >> > files to numerous Wicket resources (and to guarantee "too many open >> > files" >> > will never hit us) is to unpack those resources from the wicket .jar-s >> > and >> > put them into WEB-INF/classes directory instead. Once we did it, we >> > didn't >> > see all those dozens (and even hundreds) open files ever again... >> > >> > Bob. >> > >> > >> > Eelco Hillenius wrote: >> > > >> > > Too bad, >> > > >> > > Eelco >> > > >> > > >> > > On 2/1/07, Johan Compagner < [EMAIL PROTECTED]> wrote: >> > >> why would you cache? and which one? the url connection to an entry >> in >> > a >> > >> jar >> > >> file (thats the JarUrlConnection) >> > >> or (i guess) the FileUrlConnection (to the jar file itself) >> > >> >> > >> both don't make much sense to cache >> > >> the first one we don't need to cache we only need to use it once by >> > >> really >> > >> loading the resource >> > >> and i guess when it is finalized it is cleaned up. >> > >> We already don't use it anymore for the last modified. Because there >> > we >> > >> use >> > >> only the second one >> > >> So the fileUrlConnection to the jarFile itself thats is inside the >> > >> JarUrlConnection object. >> > >> on that one we call last modified everytime, But that will not cause >> > the >> > >> file to open. (because it doesn't have to read the file itself) >> > >> >> > >> And we can't construct JarUrlConnections (for reading the jar >> > entries) >> > >> with >> > >> the same file url connection because there is >> > >> no way to initialize the jar url connection directly with the file >> > url >> > >> connection so they all would use the same. >> > >> >> > >> johan >> > >> >> > >> >> > >> >> > >> On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > >> > >> > >> > Would it be possible and useful to cache the URL connection? Does >> > it >> > >> > have a time out and/ or does it use an exclusive lock? >> > >> > >> > >> > Eelco >> > >> > >> > >> > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: >> > >> > > that is what you would think... But why generates a modification >> > >> check >> > >> one >> > >> > > file handle for every check in the file? >> > >> > > >> > >> > > because UrlConnection.connect() has again a JarUrlConnection >> > >> internally >> > >> that >> > >> > > makes a new connection to that jar file >> > >> > > and UrlConnection does have a connect() but not a disconnect() >> so >> > you >> > >> can't >> > >> > > clear it. >> > >> > > >> > >> > > johan >> > >> > > >> > >> > > >> > >> > > >> > >> > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: >> > >> > > > Yeah, but that would be always one fd for a jar, no matter how >> > many >> > >> > > > files in it that have to be read, right? >> > >> > > > >> > >> > > > Eelco >> > >> > > > >> > >> > > > >> > >> > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: >> > >> > > > > yes the modification checker. >> > >> > > > > But we do need to really load the resource out of the jar >> > file >> > >> once. >> > >> So >> > >> > > that >> > >> > > > > file handle will be used. >> > >> > > > > >> > >> > > > > johan >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > On 2/1/07, Eelco Hillenius
