Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Ramus, thanks for your detailed response. NFS is so common for sharing files that ... This is simply not true. I do have a fair bit of experience in this field, and I don't know of any major sites that do this and I have worked with a good chunk of the largest sites out there. Eh??? Fortune 500 enterprises and governmental departments are pretty conservative. NAS and SAN based iSCSI and FCoE based elastic block storage give great performance for server-specific file-systems, but Brendon is right: for distributed file systems, NFS and CIFS still dominate. By major I meant traffic-wise, not Fortune-500, although there are some of those on the list too. I mostly work with medium-to-large scale Internet companies. Think Yahoo, Facebook, Flickr, Digg, Etsy, WePay, Room77. These types of companies would never consider serving all their Web traffic from NFS. Yes, Yahoo had a ton of Netapp filers as well, but this was for shared data storage, they would never consider putting their application logic on them. Now I agree with you: for this sector of Internet B2C companies, their business is centred around a small number of apps that dominate their revenue streams, so of course they are free to design their infrastructure architecture to optimize the performance of these apps. I also accept that this sector was and is directly or indirectly the major funder of PHP development effort. However, my counter point is that this is no longer the only infrastructure usecase for PHP. Now mature, it has entered other sectors and Brendon and Daniel posts highlight two of them: * Enterprise use as Brendon raises. Enterprises have moved to use internet based technologies to automate internal business processes. These apps work on the company intranet, not on the internet. So when you book a car or go to your bank or order a part from a manufacturer, the assistant may well be sitting in front of a PHP app that never sees the internet but is still core to that business. Thanks partly to the flexibility of cloud resources, CIOs and CTOs are increasingly looking at open technologies such PHP to replace MS ones. Incidentally IMO, its this sort of business stream that will provide hard funding to value-add companies such as Zend. * The hosting service providers as Daniel raises. In terms of sheer numbers this is that largest community of PHP users who buy their +/- $100pa service from a hosting provider. They still care about performance. The providers care about the efficiency of their infrastructure. They (initially) using PHP because Wordpress, Mediawiki, ... are written in it. But this is also a major entry vehicle for a new generation of PHP developers to get an initial internet presence. If PHP runs 3x slower than language X, and X is just as flexible then we are putting up unnecessary barriers to their entry and turning away that new cadre. This is also something that has been like this for 10+ years and nobody has stepped up to fix it so far. It shouldn't be news to anyone that stats and opens over NFS are slow. I am not sure why it should suddenly be an urgent problem for us at this point. But like I said, we may get to it. It's not suddenly urgent but perhaps this is more a question of maybe hitting a tipping point where it might now be wise to address this issue. If the integrated opcode cache happens it becomes easier to manage the flow between the compiler, the cache and the executor and we can probably optimize some things there. +1 And as I mentioned in another thread, let's see some RFCs proposing how to fix some of these things rather than simply posting I wish the PHP devs would do this.. type of messages. These go over really badly with most of the longtime contributors here and they even tend to have the opposite of the desired effect. As I have posted separately, I forked and then rewrote APC to address this sweet spot. OK my LPC is very a much bug-ridden alpha code that fails 10% of the PHP test suite largely due to extension interoperability issues, and I've had other things to do this last month -- including deciding whether to switch to a proper O+ delta. However, my aim was for me to use this as an evaluation test bed, not a serious production contender. However, now that I've written an opcode cache which runs Mediawiki under php-cgi (with ~ 5% of the NFS getattrs, BTW), rolling some key tweeks into the Zend compiler, execution environment and APC -- which I understand well and should be straight forward -- or O+ -- which I don't as yet. My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the corresponding beta APC version which at current rates of adoption might have begin to have an impact in the community sometime in the next 5 years, or (ii) work on a performance patch to the stable APC version which is typically installed with PHP 5.3 which these guys could apply within a few months.
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the corresponding beta APC version which at current rates of adoption might have begin to have an impact in the community sometime in the next 5 years, or (ii) work on a performance patch to the stable APC version which is typically installed with PHP 5.3 which these guys could apply within a few months. or contribute those patches back and integrate them into the vanilla apc? for me it seems that Rasmus opinion was that having improvements in this area would be welcome, but personally it was never a high priority for the current authors and there is a chance that it won't change, so if nobody steps up and create a patch, there is no guarantee that this will be addressed soon. I'll draft the RFCs if the webmaster gives me the karma to do so. sure, just drop a mail to the php-webmaster list (as far as I can see you have yet to do that). -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 22/02/13 11:20, Ferenc Kovacs wrote: My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the corresponding beta APC version which at current rates of adoption might have begin to have an impact in the community sometime in the next 5 years, or (ii) work on a performance patch to the stable APC version which is typically installed with PHP 5.3 which these guys could apply within a few months. or contribute those patches back and integrate them into the vanilla apc? Humm. I think that we are sort of saying the same thing, but at cross purposes. Of course I should offer any up patches for mainstream APC and at best these will go into 3.1.14 or 3.1.15 and may then get adopted sometime for production systems whenever -- that's only if the release of a core O+ doesn't drop APC into legacy status. However Ubuntu 12.04-LTS is a good example of a stable production stack and this uses PHP 5.3.10 and APC 3.1.7. Debian Squeeze is even further behind and it runs PHP 5.3.3 and APC 3.1.3. A performance patch could also be made available based on the last stable version of APC, say 3.1.9 -- that is before the attempts to support the new PHP 5.4 features destabilised it. With this patch, then at least individual system admins would have the option to download a stable version from PECL + patch it to use with their production stacks within the next 3-6 months. Regards Terry
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Fri, Feb 22, 2013 at 1:52 PM, Terry Ellison te...@ellisons.org.ukwrote: On 22/02/13 11:20, Ferenc Kovacs wrote: My challenge is deciding (i) do I work on PHP 5.6 / 5.7 and the corresponding beta APC version which at current rates of adoption might have begin to have an impact in the community sometime in the next 5 years, or (ii) work on a performance patch to the stable APC version which is typically installed with PHP 5.3 which these guys could apply within a few months. or contribute those patches back and integrate them into the vanilla apc? Humm. I think that we are sort of saying the same thing, but at cross purposes. Of course I should offer any up patches for mainstream APC and at best these will go into 3.1.14 or 3.1.15 and may then get adopted sometime for production systems whenever -- that's only if the release of a core O+ doesn't drop APC into legacy status. no comment on that, but I'm expecting/hoping that we will keep apc alive at least until we have supported versions where O+ isn't in the core (5.3 and 5.4). However Ubuntu 12.04-LTS is a good example of a stable production stack and this uses PHP 5.3.10 and APC 3.1.7. Debian Squeeze is even further behind and it runs PHP 5.3.3 and APC 3.1.3. A performance patch could also be made available based on the last stable version of APC, say 3.1.9 -- that is before the attempts to support the new PHP 5.4 features destabilised it. With this patch, then at least individual system admins would have the option to download a stable version from PECL + patch it to use with their production stacks within the next 3-6 months. I still think that it would be better kept as a release, and that we should try to have a release which is stable with both 5.3 and 5.4, but it's easier said than done. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 19/02/13 01:30, Kevin Yung wrote: In our environment, we use NFS for shared storage, we are using APC as well with stat=0. In our setting, we also experiencing high number of stat() calls on our file system. My initial finding of this problem is we enabled the open_basedir setting. And there is already a bug report for this, https://bugs.php.net/bug.php?id=52312 We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them experiencing same issue. Kevin, I've just walked through this in 5.3 and 54 and updated this bugrep. In short there is some silly coding here which should be addressed. Even if we accept that PHP should comply with http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-5178 if open_basedir is set, then the cache should only be ignored on the actual open itself, as this is the only one that is exploitable, but let's have this debate on the bugrep. Let me think about the security and other NFRs and propose a patch.
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Mon, Feb 18, 2013 at 3:35 PM, Rasmus Lerdorf ras...@lerdorf.com wrote: Wow, people are still serving web files over NFS? Sounds painful. On Mon, Feb 18, 2013 at 9:32 PM, Rasmus Lerdorf ras...@lerdorf.com wrote: So they are serving up all their PHP over NFS for some reason. I've been thinking about this and wanted to make a few final points. I think this is potentially a pretty major issue and not something to be dismissed. We've invested a great deal of money in a high performance shared storage system, so of course we want to use this system for absolutely everything we can within reason, including sharing 1500+ PHP files to our many web servers. I don't think most systems administrators would find this unreasonable. Why would we want to split our architecture up so just our PHP files are being synced from local disk and everything else is shared via NFS? Doing this seems far more unreasonable to me and just doesn't make any sense. NFS is so common for sharing files that saying Wow, people are still serving web files over NFS? is like saying Wow, people are still using the ls command to list directory contents on Linux? I think NFS is still very widely used, even for sharing web files. This being said, you stated in bug #59372 that PHP by default does open+fstat for performance reasons and while doing an open+fstat might result in better performance on local file systems, doing so is not optimal when using NFS. NFS close-to-open cache consistency is designed such that opening a file stored on an NFS server causes an automatic getattr or access request. This could also be true for other shared file systems as others have mentioned. All of these open+fstats are definitely causing a measurable impact to our systems. Right now, for example, 50% of the 6-8K+ ops/s being processed by our filer are getattr operations, and a majority of these getattr calls are coming from our Apache/PHP servers. Doing open+fstat bypasses the NFS attribute cache similar to disabling it altogether, where a simple stat would likely utilize the attribute cache. Disabling the attribute cache is generally not recommended due to the performance impact this has on the filer (and the application). I don't think the appropriate answer is don't use NFS because this is ridiculous as a long term solution (NFS is common, and people are going to use it or something similar). I think the appropriate answer is to update PHP to use stat vs. open+fstat or doing something similar that would be optimized for both local AND shared file systems (I would be writing a patch instead of this email if I could). Until then, I think it's important that people know that PHP is NOT optimized for use with NFS (and possilby other shared file systems) even when using an opcode cache like APC. I think this is a very important piece of information that people need to know. Up until this week, I wouldn't have thought twice about sharing PHP files over NFS, and I've been a Unix admin for over twelve years (only new to this environment). Brendon -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 02/21/2013 03:15 PM, Brendon Colby wrote: NFS is so common for sharing files that saying Wow, people are still serving web files over NFS? is like saying Wow, people are still using the ls command to list directory contents on Linux? I think NFS is still very widely used, even for sharing web files. This is simply not true. I do have a fair bit of experience in this field, and I don't know of any major sites that do this and I have worked with a good chunk of the largest sites out there. I don't think the appropriate answer is don't use NFS because this is ridiculous as a long term solution (NFS is common, and people are going to use it or something similar). I think the appropriate answer is to update PHP to use stat vs. open+fstat or doing something similar that would be optimized for both local AND shared file systems (I would be writing a patch instead of this email if I could). If it is of such importance to you and you are not able to do it yourself, then hire someone to do it. We may or may not get around to it, but like most things in PHP, we work on what we need ourselves and I don't think anybody here would even consider putting all their PHP files on an NFS share when performance was important. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Quoting Brendon Colby bren...@newgrounds.com: We've invested a great deal of money in a high performance shared storage system, so of course we want to use this system for absolutely everything we can within reason, including sharing 1500+ PHP files to our many web servers. I don't think most systems administrators would find this unreasonable. Great points Brendon! We have a similar architectural to your setup running NetApp filers, Apache/PHP web servers performing shared hosting for over 40,000+ websites. We pull over 35k IOPS and 80% are from PHP stats and getattr's. Someone earlier mentioned that people now use GlusterFS as an alternative. GlusterFS native client runs like absolute crap with Apache/PHP applications in order to get reasonable performance you must use the GlusterFS NFS implementation and mount the GlusterFS using NFS. You end up in the same performance issues using GlusterFS and PHP. More and more clustered PHP applications are using GlusterFS via NFS. Due to the getattr and stat issues in PHP it doesn't scale well in these sorts of deployments. In order to recover some of the performance on our NetApp's we have invested in 1TB of Flash Cache which caches the majority of getattr and stat requests on the filer however there is still a performance hit as the requests still go via the network and not cached on the local client. It would be great to see the PHP devs address this issue. Regards, Dan Zoltak -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Quoting dan...@zoltak.com: Great points Brendon! We have a similar architectural to your setup running NetApp filers, Apache/PHP web servers performing shared hosting for over 40,000+ websites. We pull over 35k IOPS and 80% are from PHP stats and getattr's. Someone earlier mentioned that people now use GlusterFS as an alternative. GlusterFS native client runs like absolute crap with Apache/PHP applications in order to get reasonable performance you must use the GlusterFS NFS implementation and mount the GlusterFS using NFS. You end up in the same performance issues using GlusterFS and PHP. More and more clustered PHP applications are using GlusterFS via NFS. Due to the getattr and stat issues in PHP it doesn't scale well in these sorts of deployments. I would also like to add that newer Cloud concentric application are developed so that they do not persistent data to local storage. This then negates the use of NFS i.e. we can deploy code via git to the nodes running on local storage. All dynamically generated user data is persisted to object storage and sessions are persisted using the memcache session handler. However in order to support legacy applications that are not Cloud centric then a network file system is the only viable option in shared hosting environments. Regards, Dan Zoltak -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 21/02/13 23:38, Rasmus Lerdorf wrote: On 02/21/2013 03:15 PM, Brendon Colby wrote: NFS is so common for sharing files that saying Wow, people are still serving web files over NFS? is like saying Wow, people are still using the ls command to list directory contents on Linux? I think NFS is still very widely used, even for sharing web files. This is simply not true. I do have a fair bit of experience in this field, and I don't know of any major sites that do this and I have worked with a good chunk of the largest sites out there. Eh??? Fortune 500 enterprises and governmental departments are pretty conservative. NAS and SAN based iSCSI and FCoE based elastic block storage give great performance for server-specific file-systems, but Brendon is right: for distributed file systems, NFS and CIFS still dominate. I don't think the appropriate answer is don't use NFS because this is ridiculous as a long term solution (NFS is common, and people are going to use it or something similar). I think the appropriate answer is to update PHP to use stat vs. open+fstat or doing something similar that would be optimized for both local AND shared file systems (I would be writing a patch instead of this email if I could). If it is of such importance to you and you are not able to do it yourself, then hire someone to do it. We may or may not get around to it, but like most things in PHP, we work on what we need ourselves and I don't think anybody here would even consider putting all their PHP files on an NFS share when performance was important. Again wrong. Apps developers don't do this because that want to; they do it because the IT services group that runs the production infrastructure has mandated standard templates for live deployment to keep the through-life cost of providing the application infrastructure manageable, with all sorts of bureaucratic exception processes if you think that your app is a special case. If you are lucky then they offer a range of EC2-like standard VM templates so that you can deploy an EBS-based approach, but most are still in catch-up mode compared to Amazons offerings. If you are a GM or a American Airlines, or the NIH for example then you will have 1,000s of applications customer facing and internal and you've got to adopt this approach for 90+% of these applications. What Brendon is asking for is reasonable, sensible and in relative terms easy to implement. However, I agree that it may be more sensible to use community effort to achieve this.
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 02/21/2013 04:41 PM, Terry Ellison wrote: On 21/02/13 23:38, Rasmus Lerdorf wrote: On 02/21/2013 03:15 PM, Brendon Colby wrote: NFS is so common for sharing files that saying Wow, people are still serving web files over NFS? is like saying Wow, people are still using the ls command to list directory contents on Linux? I think NFS is still very widely used, even for sharing web files. This is simply not true. I do have a fair bit of experience in this field, and I don't know of any major sites that do this and I have worked with a good chunk of the largest sites out there. Eh??? Fortune 500 enterprises and governmental departments are pretty conservative. NAS and SAN based iSCSI and FCoE based elastic block storage give great performance for server-specific file-systems, but Brendon is right: for distributed file systems, NFS and CIFS still dominate. By major I meant traffic-wise, not Fortune-500, although there are some of those on the list too. I mostly work with medium-to-large scale Internet companies. Think Yahoo, Facebook, Flickr, Digg, Etsy, WePay, Room77. These types of companies would never consider serving all their Web traffic from NFS. Yes, Yahoo had a ton of Netapp filers as well, but this was for shared data storage, they would never consider putting their application logic on them. This is also something that has been like this for 10+ years and nobody has stepped up to fix it so far. It shouldn't be news to anyone that stats and opens over NFS are slow. I am not sure why it should suddenly be an urgent problem for us at this point. But like I said, we may get to it. If the integrated opcode cache happens it becomes easier to manage the flow between the compiler, the cache and the executor and we can probably optimize some things there. And as I mentioned in another thread, let's see some RFCs proposing how to fix some of these things rather than simply posting I wish the PHP devs would do this.. type of messages. These go over really badly with most of the longtime contributors here and they even tend to have the opposite of the desired effect. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Hi! Yes, this is an overhead, but it is small beer compared to doing a getattr() RPC across the server room fabric to a NAS server backend. That depends of what your error handlers do. Some may write to log files, etc. if not configured properly (since error_reporting setting doesn't have to be considered in it). IIRC, for most of the cases O+ should be able to resolve all includes/requires on cached files without syscalls - but file_exists is a different matter since caching it in generic case can be very dangerous. I guess that I should bite the bullet and switch to 5.5. I've been working on an evaluatorfork of APC optimized for CLI/GCI which tackles a lot of these issues head on and performs reasonable well, but I realise that this is a dead-end and will never get deployed, but I am currently considering regressing some of this technology into 5.5 and O+. Are you interested in a version of O+ which supports all SAPIs? I think right now O+ can support CLI (provided enable_cli is set) but for most cases it's kind of useless since scripts are rarely re-included in common CLI scenarios. So if you know how to improve common CLI scenarios it may be interesting, though I imagine it's not the most common use case. But if it adds there without problems for anything else, why not. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 20/02/13 08:26, Stas Malyshev wrote: That depends of what your error handlers do. Some may write to log files, etc. if not configured properly (since error_reporting setting doesn't have to be considered in it). IIRC, for most of the cases O+ should be able to resolve all includes/requires on cached files without syscalls - but file_exists is a different matter since caching it in generic case can be very dangerous. I am not suggesting caching file_exists but rather encouraging coding patterns which avoid its use /if/ the application is intended to give good cached performance -- e.g. apps like mwiki, wordpress and drupal. I guess that I should bite the bullet and switch to 5.5. I've been working on an evaluatorfork of APC optimized for CLI/GCI which tackles a lot of these issues head on and performs reasonable well, but I realise that this is a dead-end and will never get deployed, but I am currently considering regressing some of this technology into 5.5 and O+. Are you interested in a version of O+ which supports all SAPIs? I think right now O+ can support CLI (provided enable_cli is set) but for most cases it's kind of useless since scripts are rarely re-included in common CLI scenarios. So if you know how to improve common CLI scenarios it may be interesting, though I imagine it's not the most common use case. But if it adds there without problems for anything else, why not. Let me get a dev stack for 5.5 and O+ up and then I can comment on this. As far as APC's support for CLI goes -- the cache doesn't connect to the apache2 SMA even if you run the cli process in the same UID, and since the SMA gets released when the process count goes to zero, caching doesn't really do anything as the cache discarded between cli executions, so if practice you always run uncached. As to the frequency of the use case, maybe not in the case of cli, but cgi is still tremendously common, as shared hosting providers still need to implement UID-based separation of scripting environments for different user accounts. There are still significant scaling issues with php-fpm in this usecase, so only a few SHPs offer FastCGI support and then only as a premium option. BTW, I saw that you recommended in one of your blog posts that you to use a bytecode cache if you care about performance should do. However, a lot of app maintainers who use a shared hosting service do care about performance but don't have this as an option :-( For example with my cli-based LPC opcode cache, I can do a series of make test runs (1) with the opcache disabled; (2) with it enabled but the cache empty and (3) with it enabled and the cache(s) primed. Ideally all three (*) should give the same test results, and this gives you confidence that the cache is working properly. I would like to do this with APC or O+, but in practice AFAIK, I can't do (3) so how do I or the developers know what PHP and extension features aren't being supported / working properly when the code is cached? (*) With the tests as currently written, some fail with case (3) because the runs are missing compiler warnings which are only generated if the code isn't cached (and in case of my cache at its current build quite a few fail because it still doesn't play nicely with other extensions like phar.)
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Mon, Feb 18, 2013 at 7:10 PM, Terry Ellison te...@ellisons.org.uk wrote: Brendon, are your scripts doing a log of include_once / require_once calls? Yep, but we also changed our autoload script (auto_prepend_file) to use require instead of require_once, and we still saw the same behavior (open calls for the PHP files required by the auto_prepend_file same as before). In you look at ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER() in Zend/zend_vm_execute.h then you will see that this Zend handler does zend_resolve_path() for any xxx_ONCE instructions and zend_stream_open() on the first request of the same. Yes APC rehooks this handler with a wrapper but this is only for lazy loading. When it honors the xxx_once instructions, it will still open the streams even if the code itself is fully cached in APC and the I/O is entirely nugatory. I suspect that this could generate the NFS traffic that you are discussing. Yes if the streams are opened whether cached or not, then this will result in an automatic getattr request to the NFS server which is exactly what we're seeing. Brendon -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Tue, Feb 19, 2013 at 11:06 AM, Terry Ellison te...@ellisons.org.uk wrote:
Brendon,
Just to follow up with a bit more detail, apart from the obvious NFS tuning
with things like the actimeo mount parameters, you can get a better idea of
what is going on if you use a local copy of one of your apps running under a
local linux apache server. This is an example from my Ubuntu laptop, but it
works just as well on a test VM if you still use WinXX on your PC.
trace_wget () {
sleep 1
coproc sudo strace -p $(ps -C apache2 --no-headers -o pid) -tt -o
/tmp/strace$1.log
sleep 1; ST_PID=$(ps -C strace --no-headers -o pid)
wget http://localhost/$2; -O /dev/null -o /dev/null
sleep 2; sudo kill $ST_PID
}
# start Apache in debug mode
sudo service apache2 stop
sudo bash -c . /etc/apache2/envvars; coproc apache2 -X
# trace three gets
trace_wget 0 phpinfo.php
trace_wget 1 mwiki/index.php?title=Main_Page
trace_wget 2 mwiki/index.php?title=Main_Page
#restart normally
sudo service apache2 stop
sudo service apache2 start
sudo chmod 777 /tmp/strace?.log
grep -c open( /tmp/strace?.log
grep -c stat( /tmp/strace?.log
The first get is just to load and prime the mod_php5 thread (Ubuntu has a
crazy localtime implementation), the second loads and compiles the PHP
MediaWiki modules needed to render a page, the third repeats this now fully
cached in ACP. In this case,
we have:
open()fstat()/lstat()
/tmp/strace1.log 108 650 (Priming the APC cache)
/tmp/strace2.log27 209 (Using the APC cache)
So APC *does* materially reduce the I/O calls, and (if you look at the
traces) it removes most of the mmaps and compilation. The MWiki autoloader
uses require() to load classes but in the code as a whole the most common
method of loading modules is require_once() (414 uses as opposed to 44 in
total of the other include or require functions) and as I said in my
previous post, this I/O is avoidable by recoding the ZEND_INCLUDE_OR_EVAL
hander to work cooperatively with the opcode cache when present.
Note that even when the cache does this, it still can't optimize coding
patterns like:
if ( file_exists( $IP/AdminSettings.php ) ) {
require( $IP/AdminSettings.php );
}
and IIRC mwiki does 4 of these on a page render. Here a pattern based on
(@include($file) == 1) is more cache-friendly.
Now this isn't a material performance issue when the code tree is mounted on
local storage (as is invariably the case for a developer test configuration)
as the stat / open overhead of a pre-(file) cached file is microseconds, but
it is material on scalable production infrastructure which uses network
attached NFS or NTFS file systems.
So in summary, this is largely avoidable but unfortunately I don't think
that we can persuade the dev team to address this issue, even if I gave them
the solution as a patch :-(
Regards
Terry Ellison
Terry,
Thanks for your detailed input. This is essentially what I did in my test setup.
APC is definitely critical to the performance of our site. I'm just
very curious to see how much impact the high number of getattr
operations coming from the Apache/PHP servers is having on the filer.
It looks like we might be able to move our PHP files to local storage
to at least measure the impact this is having.
Brendon
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 20/02/13 23:52, Brendon Colby wrote: Terry, Thanks for your detailed input. This is essentially what I did in my test setup. APC is definitely critical to the performance of our site. I'm just very curious to see how much impact the high number of getattr operations coming from the Apache/PHP servers is having on the filer. It looks like we might be able to move our PHP files to local storage to at least measure the impact this is having. Brendon Brendon, I think that PHP is a great platform; it's just that PHP's sweet spot is just a little off the optimum for typical scalable enterprise infrastructures. IMO, the main issue that you should consider and discuss with your infrastructure support team is how to mitigate the impacts for such high-throughput business critical applications. As you (and Rasmus previously, IIRC) mention one key issue is what is on local and what is on network attached storage. On one system that I got hauled in to troubleshoot, it turned out that the main problem was that the PHP session data was being written to a directory in shared storage, causing write-though overload that ended up hitting half a dozen key apps. It was a trivial configuration change to move it to local storage transforming performance. (I apologise if I am telling how to suck eggs but this is really for others tracking this thread) the main issue here is that the file hierarchies that must have some integrity over the application tier need to be on the NAS, everything else is a matter of convenience vs performance, so for example having a cron job to rsync the apps PHP hierarchy onto local storage might well transform your app performance and give an indirect boost to cohosted apps. Happy hunting :)
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
The point that this thread highlights is that apps developers / administrators at both ends of the scale -- the enterprise and the shared service user -- normally have little say over the infrastructure architecture on which their application runs. In both these cases the infrastructure will be hosting 100s if not 1000s of apps, and the environment cannot be tailored to any one app. So issues like storage architecture and security architecture (e.g. UID-based enforcement of app separation) are a given as NFRs (non-functional requirements). Use of NFS with server / storage separation is still a standard implementation design pattern, when enterprises require simple-to-manage scalability on these tiers. We aren't going to change this, and neither will the O/P. Surely PHP will achieve better penetration and the greater acceptance if it can offer more robust performance in this sort of environment? Sure, but then you can go with something like Redis. But, again, if you go back to the original question, this has nothing to do with often-changing data in a couple of PHP include files: We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving mostly PHP over NFS (we have separate servers for static content). So they are serving up all their PHP over NFS for some reason.
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 19/02/13 09:36, Terry Ellison wrote:
The point that this thread highlights is that apps developers /
administrators at both ends of the scale -- the enterprise and the
shared service user -- normally have little say over the
infrastructure architecture on which their application runs. In both
these cases the infrastructure will be hosting 100s if not 1000s of
apps, and the environment cannot be tailored to any one app. So
issues like storage architecture and security architecture (e.g.
UID-based enforcement of app separation) are a given as NFRs
(non-functional requirements). Use of NFS with server / storage
separation is still a standard implementation design pattern, when
enterprises require simple-to-manage scalability on these tiers. We
aren't going to change this, and neither will the O/P.
Surely PHP will achieve better penetration and the greater acceptance
if it can offer more robust performance in this sort of environment?
Sure, but then you can go with something like Redis.
But, again, if you go back to the original question, this has nothing to
do with often-changing data in a couple of PHP include files:
We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving
mostly PHP over NFS (we have separate servers for static content).
So they are serving up all their PHP over NFS for some reason.
Brendon,
Just to follow up with a bit more detail, apart from the obvious NFS
tuning with things like the actimeo mount parameters, you can get a
better idea of what is going on if you use a local copy of one of your
apps running under a local linux apache server. This is an example from
my Ubuntu laptop, but it works just as well on a test VM if you still
use WinXX on your PC.
trace_wget () {
sleep 1
coproc sudo strace -p $(ps -C apache2 --no-headers -o pid) -tt -o
/tmp/strace$1.log
sleep 1; ST_PID=$(ps -C strace --no-headers -o pid)
wget http://localhost/$2; -O /dev/null -o /dev/null
sleep 2; sudo kill $ST_PID
}
# start Apache in debug mode
sudo service apache2 stop
sudo bash -c . /etc/apache2/envvars; coproc apache2 -X
# trace three gets
trace_wget 0 phpinfo.php
trace_wget 1 mwiki/index.php?title=Main_Page
trace_wget 2 mwiki/index.php?title=Main_Page
#restart normally
sudo service apache2 stop
sudo service apache2 start
sudo chmod 777 /tmp/strace?.log
grep -c open( /tmp/strace?.log
grep -c stat( /tmp/strace?.log
The first get is just to load and prime the mod_php5 thread (Ubuntu has
a crazy localtime implementation), the second loads and compiles the PHP
MediaWiki modules needed to render a page, the third repeats this now
fully cached in ACP. In this case,
we have:
open()fstat()/lstat()
/tmp/strace1.log 108 650 (Priming the APC cache)
/tmp/strace2.log27 209 (Using the APC cache)
So APC *does* materially reduce the I/O calls, and (if you look at the
traces) it removes most of the mmaps and compilation. The MWiki
autoloader uses require() to load classes but in the code as a whole the
most common method of loading modules is require_once() (414 uses as
opposed to 44 in total of the other include or require functions) and as
I said in my previous post, this I/O is avoidable by recoding the
ZEND_INCLUDE_OR_EVAL hander to work cooperatively with the opcode cache
when present.
Note that even when the cache does this, it still can't optimize coding
patterns like:
if ( file_exists( $IP/AdminSettings.php ) ) {
require( $IP/AdminSettings.php );
}
and IIRC mwiki does 4 of these on a page render. Here a pattern based
on (@include($file) == 1) is more cache-friendly.
Now this isn't a material performance issue when the code tree is
mounted on local storage (as is invariably the case for a developer test
configuration) as the stat / open overhead of a pre-(file) cached file
is microseconds, but it is material on scalable production
infrastructure which uses network attached NFS or NTFS file systems.
So in summary, this is largely avoidable but unfortunately I don't think
that we can persuade the dev team to address this issue, even if I gave
them the solution as a patch :-(
Regards
Terry Ellison
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Hi! and IIRC mwiki does 4 of these on a page render. Here a pattern based on (@include($file) == 1) is more cache-friendly. This goes back to the fact that @ does not really disable errors - it only disables reporting of the errors, but the whole message is generated and goes all the cycle up to the actual error reporting before being suppressed. I've tried to address this problem a number of times but looks like there's really no way to do it without sacrificing some parts of current error reporting such as track_errors. So in summary, this is largely avoidable but unfortunately I don't think that we can persuade the dev team to address this issue, even if I gave them the solution as a patch :-( Could you explain what you're talking about here? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 19/02/13 20:32, Stas Malyshev wrote: Hi! and IIRC mwiki does 4 of these on a page render. Here a pattern based on (@include($file) == 1) is more cache-friendly. This goes back to the fact that @ does not really disable errors - it only disables reporting of the errors, but the whole message is generated and goes all the cycle up to the actual error reporting before being suppressed. I've tried to address this problem a number of times but looks like there's really no way to do it without sacrificing some parts of current error reporting such as track_errors. Yup, in this scenario we're trading off two things. What the pattern is saying is if the source file exists then compile it. However, if the file exists, in say 90+% of the cases, and the code is often cached then the file_exists check generates I/O that isn't needed if the file is already APC (or O+ / xcache) cached. Yup if the file is missing then the error is generated and trapped in the reporting cycle, but the final result is that the result is false and the false==1 fails. Other than the wasted cycles in running up this stack, there aren't any other side-effects that I am aware of. Are there any? Yes, this is an overhead, but it is small beer compared to doing a getattr() RPC across the server room fabric to a NAS server backend. So in summary, this is largely avoidable but unfortunately I don't think that we can persuade the dev team to address this issue, even if I gave them the solution as a patch :-( Could you explain what you're talking about here? The fact that you are engaging in this dialogue is great and I really appreciate it. What I am still trying to work out is how to interact with you guys in a way that is mutually productive. I did make the mistake of choosing the mainstay stable version iN live prod systems use -- 5.3 -- to take this whole issue apart and I guess that the dev team regards this as history. I guess that I should bite the bullet and switch to 5.5. I've been working on an evaluatorfork of APC optimized for CLI/GCI which tackles a lot of these issues head on and performs reasonable well, but I realise that this is a dead-end and will never get deployed, but I am currently considering regressing some of this technology into 5.5 and O+. Are you interested in a version of O+ which supports all SAPIs? In architectural terms I feel that having a universal cache option is important. It changes the mindset from something which is only used in niche performance use cases to a standard option. It also means that the cache can be stress tested by the entire php test suite. However, to do some of this you don't start with a patch, but with an RFC informed by evidence, and that's my real reason for doing this demonstrator. //Terry
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 02/19/2013 03:08 PM, Terry Ellison wrote: I guess that I should bite the bullet and switch to 5.5. Yes. I've been working on an evaluatorfork of APC optimized for CLI/GCI which tackles a lot of these issues head on and performs reasonable well, but I realise that this is a dead-end and will never get deployed, but I am currently considering regressing some of this technology into 5.5 and O+. Are you interested in a version of O+ which supports all SAPIs? Yes, but there are some SAPIs that are more common, so these should be prioritized if trade-offs are required. My particular desire is for embed SAPI support. We use this in our Tuxedo application server, but it is a relatively niche use case. In architectural terms I feel that having a universal cache option is important. It changes the mindset from something which is only used in niche performance use cases to a standard option. It also means that the cache can be stress tested by the entire php test suite. However, to do some of this you don't start with a patch, but with an RFC informed by evidence, and that's my real reason for doing this demonstrator. I'm looking forward to seeing your suggestions and patches. Chris -- christopher.jo...@oracle.com http://twitter.com/ghrd Newly updated, free PHP Oracle book: http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual-098250.html -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] PHP causing high number of NFS getattr operations?
Greetings, This seemed like the appropriate list to discuss this - let me know if it isn't. We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving mostly PHP over NFS (we have separate servers for static content). I have been trying to figure out why we're seeing so many getattr requests from our Apache/PHP servers. I think I've narrowed it down to how PHP/APC gets information about a file to see if it has been changed since it was last cached: https://bugs.php.net/bug.php?id=59372 Rasmus said: Just because you see an open() syscall doesn't mean the cache isn't used. The open() is there because PHP by default does open+fstat for performance reasons. If you look carefully you will see there is just an open() and no read() calls. So it opens the file, uses the stat struct from the open fd to get the inode and fetches the op_array from the opcode cache. This can be made more efficient, but this report is incorrect in assuming the cache isn't being used. This is exactly what I'm seeing when I strace an httpd process - mostly open() and fstat() calls w/o any read() calls. This makes sense to me, however, the problem with this is as far as I understand it, that an open() on an NFS file system causes an automatic gettattr() call to the server. Doing this completely bypasses the NFS attribute cache, which is why we're seeing so many getattr requests to our NFS server despite having an attribute cache timeout of 60 seconds. For those familiar with NFS, we could just as well disable our attribute cache at this point, which is almost never recommended because of performance reasons. I found a good explaination of why this happens in this Redhat document: http://www.redhat.com/rhecm/rest-rhecm/jcr/repository/collaboration/jcr:system/jcr:versionStorage/5e75391d7f0110a97b91452f0cee/1/jcr:frozenNode/rh:pdfFile.pdf Avoid unnecessarily opening and closing of files. If you access a file frequently, opening and closing that file triggers the opentoclose cache consistency mechanism in the NFS file system. Triggering this mechanism causes NFS getattr requests to be generated, which can slow performance. By keeping the file open, traffic is kept to a minimum. I've tried doing apc.stat=0 and about every other combination of options, but PHP still does an open()+3 fstat() commands on every single page load no matter what I do. We have autoload set, so that equates to several automatic open() (and subsequent NFS getattr()) calls with every single PHP request. To give you an idea what kind of load this is generating for us: -Out of 7,000 total NFS operations per second right now: -About 45% are getattr calls -Our Apache/PHP servers are generating 52% of all NFS calls -An average of 82% of those calls are getattr requests -As a comparison, our far busier content servers (many times the req/s of the Apache/PHP servers) are generating half the number of NFS requests our PHP servers are, and 98% or so of those requests are read requests with just 1% getattr I am pretty proficient with patching/compiling/profiling, but sadly I am no C coder. I've tried digging into this myself but I'm pretty out of my element here. I'm curious if anyone on this list could give me any pointers on what I can do to change this behavior and reduce the load on our NFS server. If we can't change PHP behavior, we may have to abandon NFS for our PHP files due to the load PHP appears to be generating on our NFS server. I'd hate to do this, so please let me know if any of you have any ideas as to what I can do to change this behavior. Brendon -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 02/18/2013 12:26 PM, Brendon Colby wrote: Rasmus said: Just because you see an open() syscall doesn't mean the cache isn't used. The open() is there because PHP by default does open+fstat for performance reasons. If you look carefully you will see there is just an open() and no read() calls. So it opens the file, uses the stat struct from the open fd to get the inode and fetches the op_array from the opcode cache. This can be made more efficient, but this report is incorrect in assuming the cache isn't being used. This is exactly what I'm seeing when I strace an httpd process - mostly open() and fstat() calls w/o any read() calls. This makes sense to me, however, the problem with this is as far as I understand it, that an open() on an NFS file system causes an automatic gettattr() call to the server. Doing this completely bypasses the NFS attribute cache, which is why we're seeing so many getattr requests to our NFS server despite having an attribute cache timeout of 60 seconds. For those familiar with NFS, we could just as well disable our attribute cache at this point, which is almost never recommended because of performance reasons. Wow, people are still serving web files over NFS? Sounds painful. But, this is what APC's apc.include_once_override setting tries to address. Try turning that on. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Mon, Feb 18, 2013 at 3:35 PM, Rasmus Lerdorf ras...@lerdorf.com wrote:
On 02/18/2013 12:26 PM, Brendon Colby wrote:
Rasmus said:
Wow, people are still serving web files over NFS? Sounds painful.
But, this is what APC's apc.include_once_override setting tries to
address. Try turning that on.
Yep - I think it would be a lot more painful if we didn't use a very
expensive NetApp Filer to serve NFS (the solution before the Filer
would lock up every few weeks). I'm really surprised more people
haven't complained about this. I've seen a few complaints about people
seeing poor PHP app performance on NFS, but they usually move to
gluster or something else before ever figuring it out. Maybe very few
people use NFS to serve web files - who knows.
I tried apc.include_once_override - no change. I set up Apache to one
process and straced it. I clicked around on the site a few times and
see the typical pattern:
open([PATH]/CacheFile.class.php, O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
Out of 486 open() calls, 263 were for PHP files (54%). Here is my
current APC config:
extension=/usr/local/lib/php/extensions/apc.so
apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=256M
apc.ttl=1200
apc.user_ttl=1200
apc.num_files_hint=300
apc.mmap_file_mask=/apc.shm.XX
apc.slam_defense=0
apc.write_lock=1
apc.include_once_override=1
apc.stat=0 doesn't help. Every PHP request causes an open() and
subsequent getattr() to the Filer no matter what options I set.
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Mon, Feb 18, 2013 at 10:03 PM, Brendon Colby bren...@newgrounds.com wrote:
I tried apc.include_once_override - no change. I set up Apache to one
process and straced it. I clicked around on the site a few times and
see the typical pattern:
open([PATH]/CacheFile.class.php, O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
fstat(10, {st_mode=S_IFREG|0664, st_size=2097, ...}) = 0
Assuming that those are relative includes, can you try with:
apc.canonicalize=0
apc.stat=0
Damien
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On Mon, Feb 18, 2013 at 4:32 PM, Damien Tournoud d...@damz.org wrote: Assuming that those are relative includes, can you try with: apc.canonicalize=0 apc.stat=0 Paths are absolute. stat=0 (and canonicalize=0 just to try it) produced the same result. Brendon -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 18/02/13 21:47, Brendon Colby wrote: On Mon, Feb 18, 2013 at 4:32 PM, Damien Tournoud d...@damz.org wrote: Assuming that those are relative includes, can you try with: apc.canonicalize=0 apc.stat=0 Paths are absolute. stat=0 (and canonicalize=0 just to try it) produced the same result. Brendon Brendon, are your scripts doing a log of include_once / require_once calls? In you look at ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER() in Zend/zend_vm_execute.h then you will see that this Zend handler does zend_resolve_path() for any xxx_ONCE instructions and zend_stream_open() on the first request of the same. Yes APC rehooks this handler with a wrapper but this is only for lazy loading. When it honors the xxx_once instructions, it will still open the streams even if the code itself is fully cached in APC and the I/O is entirely nugatory. I suspect that this could generate the NFS traffic that you are discussing. This would be easy to avoid, but this would require replacing this handler entirely or doing dynamic code patching, nether of which APC currently does, I believe. Incidentally because this is Zend feature and nothing directly to with APC, O+ will also have this runtime characteristic. //Terry
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Brendon, in our environment, we use NFS for shared storage, we are using APC as well with stat=0. In our setting, we are experiencing high number of stat() calls on the NFS. My initial finding of this problem is we enabled the open_basedir setting. And there is already a bug report for this, https://bugs.php.net/bug.php?id=52312 We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them experiencing same issue. I m not sure if this is a same problem as yours. But I do hope the community can find out a good solutions to handle the balance of better security and good performance. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Hi Brendon, In our environment, we use NFS for shared storage, we are using APC as well with stat=0. In our setting, we also experiencing high number of stat() calls on our file system. My initial finding of this problem is we enabled the open_basedir setting. And there is already a bug report for this, https://bugs.php.net/bug.php?id=52312 We tested the issue in 5.2.x, 5.3.x and 5.4.x, all of them experiencing same issue. I m not sure if this is a same problem as yours. But I do hope the community can find out a good solutions to handle the balance of better security and good performance. Cheers On Tue, Feb 19, 2013 at 7:26 AM, Brendon Colby bren...@newgrounds.comwrote: Greetings, This seemed like the appropriate list to discuss this - let me know if it isn't. We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving mostly PHP over NFS (we have separate servers for static content). I have been trying to figure out why we're seeing so many getattr requests from our Apache/PHP servers. I think I've narrowed it down to how PHP/APC gets information about a file to see if it has been changed since it was last cached: https://bugs.php.net/bug.php?id=59372 Rasmus said: Just because you see an open() syscall doesn't mean the cache isn't used. The open() is there because PHP by default does open+fstat for performance reasons. If you look carefully you will see there is just an open() and no read() calls. So it opens the file, uses the stat struct from the open fd to get the inode and fetches the op_array from the opcode cache. This can be made more efficient, but this report is incorrect in assuming the cache isn't being used. This is exactly what I'm seeing when I strace an httpd process - mostly open() and fstat() calls w/o any read() calls. This makes sense to me, however, the problem with this is as far as I understand it, that an open() on an NFS file system causes an automatic gettattr() call to the server. Doing this completely bypasses the NFS attribute cache, which is why we're seeing so many getattr requests to our NFS server despite having an attribute cache timeout of 60 seconds. For those familiar with NFS, we could just as well disable our attribute cache at this point, which is almost never recommended because of performance reasons. I found a good explaination of why this happens in this Redhat document: http://www.redhat.com/rhecm/rest-rhecm/jcr/repository/collaboration/jcr:system/jcr:versionStorage/5e75391d7f0110a97b91452f0cee/1/jcr:frozenNode/rh:pdfFile.pdf Avoid unnecessarily opening and closing of files. If you access a file frequently, opening and closing that file triggers the opentoclose cache consistency mechanism in the NFS file system. Triggering this mechanism causes NFS getattr requests to be generated, which can slow performance. By keeping the file open, traffic is kept to a minimum. I've tried doing apc.stat=0 and about every other combination of options, but PHP still does an open()+3 fstat() commands on every single page load no matter what I do. We have autoload set, so that equates to several automatic open() (and subsequent NFS getattr()) calls with every single PHP request. To give you an idea what kind of load this is generating for us: -Out of 7,000 total NFS operations per second right now: -About 45% are getattr calls -Our Apache/PHP servers are generating 52% of all NFS calls -An average of 82% of those calls are getattr requests -As a comparison, our far busier content servers (many times the req/s of the Apache/PHP servers) are generating half the number of NFS requests our PHP servers are, and 98% or so of those requests are read requests with just 1% getattr I am pretty proficient with patching/compiling/profiling, but sadly I am no C coder. I've tried digging into this myself but I'm pretty out of my element here. I'm curious if anyone on this list could give me any pointers on what I can do to change this behavior and reduce the load on our NFS server. If we can't change PHP behavior, we may have to abandon NFS for our PHP files due to the load PHP appears to be generating on our NFS server. I'd hate to do this, so please let me know if any of you have any ideas as to what I can do to change this behavior. Brendon -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Hi! Wow, people are still serving web files over NFS? Sounds painful. Sometimes, there's not much (better) choices if you need to keep writeable files in sync over a number of machines. There are other shared FSes but they would lead to pretty much the same issues. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 02/18/2013 05:50 PM, Stas Malyshev wrote: Hi! Wow, people are still serving web files over NFS? Sounds painful. Sometimes, there's not much (better) choices if you need to keep writeable files in sync over a number of machines. There are other shared FSes but they would lead to pretty much the same issues. Yeah, but NFS, especially without the realpath cache, which you lose if you turn on open_basedir, is deathly slow because of all the stats. Typically PHP scripts are not actually writable files and the way to keep them in synch across multiple machines is to use a deploy mechanism/rsynch to deploy them. You may very well have some writable files that need to be shared at the app-level, but then we wouldn't be talking about PHP stat calls and the realpath cache. And most people have moved to something like memcache for sharing writable data across machines. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
Hi! Yeah, but NFS, especially without the realpath cache, which you lose if you turn on open_basedir, is deathly slow because of all the stats. Typically PHP scripts are not actually writable files and the way to keep them in synch across multiple machines is to use a deploy mechanism/rsynch to deploy them. You may very well have some writable rsync may be way too slow if you need the change appear more or less immediately on all servers. Unless you run rsync each second if which case you're probably better off with NFS :) files that need to be shared at the app-level, but then we wouldn't be talking about PHP stat calls and the realpath cache. And most people You would still if the data is stored as php file. Which last time I tested - given APC or something like - was the fastest way to get a bunch of data into PHP (it was faster than loading and unserializing, even with APC/memcache). have moved to something like memcache for sharing writable data across machines. memcache is non-persistent. It's a cache, not a storage solution, so you can not use it for anything that actually needs to be stored permanently. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] PHP causing high number of NFS getattr operations?
On 02/18/2013 06:14 PM, Stas Malyshev wrote: Hi! Yeah, but NFS, especially without the realpath cache, which you lose if you turn on open_basedir, is deathly slow because of all the stats. Typically PHP scripts are not actually writable files and the way to keep them in synch across multiple machines is to use a deploy mechanism/rsynch to deploy them. You may very well have some writable rsync may be way too slow if you need the change appear more or less immediately on all servers. Unless you run rsync each second if which case you're probably better off with NFS :) files that need to be shared at the app-level, but then we wouldn't be talking about PHP stat calls and the realpath cache. And most people You would still if the data is stored as php file. Which last time I tested - given APC or something like - was the fastest way to get a bunch of data into PHP (it was faster than loading and unserializing, even with APC/memcache). have moved to something like memcache for sharing writable data across machines. memcache is non-persistent. It's a cache, not a storage solution, so you can not use it for anything that actually needs to be stored permanently. Sure, but then you can go with something like Redis. But, again, if you go back to the original question, this has nothing to do with often-changing data in a couple of PHP include files: We have several Apache 2.2 / PHP 5.4 / APC 3.1.13 servers all serving mostly PHP over NFS (we have separate servers for static content). So they are serving up all their PHP over NFS for some reason. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
