php-general Digest 10 Feb 2011 07:07:14 -0000 Issue 7174
Topics (messages 311226 through 311238):
Re: Paging and permissions
311226 by: Ashley Sheridan
311227 by: Arno Kuhl
311228 by: Ashley Sheridan
Re: First PHP site - thanks - euca_phpmysql function library
311229 by: Al
311230 by: Bob McConnell
311231 by: Peter Lind
311232 by: Bob McConnell
311233 by: Peter Lind
improve speed of PHP answers
311234 by: Alain Roger
311235 by: Alain Roger
311236 by: Alain Roger
311238 by: Ashley Sheridan
Re: curl_exec won't return (any data)
311237 by: Tolas Anon
Administrivia:
To subscribe to the digest, e-mail:
[email protected]
To unsubscribe from the digest, e-mail:
[email protected]
To post to the list, e-mail:
[email protected]
----------------------------------------------------------------------
--- Begin Message ---
On Wed, 2011-02-09 at 13:03 +0200, Arno Kuhl wrote:
> On Tue, 2011-02-08 at 14:36 +0200, Arno Kuhl wrote:
>
> I'm hoping some clever php gurus have been here before and are
> willing to
> share some ideas.
>
> I have a site where articles are assigned to categories in
> containers. An
> article can be assigned to only one category per container, but one
> or more
> containers. Access permissions can be set per article, per category
> and/or
> per container, for one or more users and/or user groups. If an
> article is
> assigned to 10 categories and only one of those has a permission
> denying
> access, then the article can't be accessed even if browsing through
> one of
> the other 9 categories. Currently everything works fine, with
> article titles
> showing when browsing through category or search result lists, and a
> message
> is displayed when the article is clicked if it cannot be viewed
> because of a
> permission.
>
> Now there's a requirement to not display the article title in
> category lists
> and search results if it cannot be viewed. I'm stuck with how to
> determine
> the number of results for paging at the start of the list or search.
> The
> site is quite large (20,000+ articles and growing) so reading the
> entire
> result set and sifting through it with permission rules for each
> request is
> not an option. But it might be an option if done once at the start
> of each
> search or list request, and then use that temporary modified result
> set for
> subsequent requests on the same set. I thought of saving the set to
> a
> temporary db table or file (not sure about overhead of
> serializing/unserializing large arrays). A sizing exercise based on
> the
> recordset returned for searches and lists shows a max of about 150MB
> for
> 20,000 articles and 380MB for 50,000 articles that needs to be saved
> temporarily per search or list request - in the vast majority of
> cases the
> set will be *much* smaller but it needs to cope with the worst case,
> and
> still do so a year down the line.
>
> All this extra work because I can't simply get an accurate number of
> results
> for paging, because of permissions!
>
> So my questions are:
> 1. Which is better (performance) for this situation: file or db?
> 2. How do I prepare a potentially very large data set for file or
> fast
> writing to a new table (ie I obviously don't want to write it record
> by
> record)
> 3. Are there any other alternatives worth looking at?
>
> TIA
>
> Cheers
> Arno
>
>
>
> How are you determining (logically, not in code) when an article is allowed
> to be read?
>
> Assume an article on "user permissions in mysql" is in a container called
> 'databases' and in a second one called 'security' and both containers are in
> a category called 'computers'
>
> Now get a user called John who is in a group called 'db admins' and that
> group gives him permissions to view all articles in the 'databases'
> container and any articles in any container in the 'computers' category. Now
> assume John also has explicit user permissions revoking that right to view
> the article in any container.
>
> What I'm getting at is what's the order of privilege for rights? Do group
> rights for categories win out over those for containers, or do individual
> user rights trump all of them overall?
>
> I think once that's figured out, a lot can be done inside the query itself
> to minimise the impact on the script getting the results.
>
>
> Thanks,
> Ash
> http://www.ashleysheridan.co.uk
>
> -----------
>
> The simple structure is articles in categories, categories in containers,
> only one article per container/category, in one or more containers. If an
> article permission explicitly allows or denies access then the permission
> applies, otherwise the container/s and category/s permissions are checked.
> The permission checks user access first then group. A user can belong to
> multiple groups.
>
> There's no query to handle this that can return a neat recordset for paging.
> Currently the complete checks are only done for an article request. The
> category list only checks access to the category and the container it
> belongs to, so the list is either displayed in its entirety (including
> titles of articles that can't be viewed) or not at all, and obviously the
> paging works perfectly because the total number of titles is known up front
> and remains constant for subsequent requests.
>
> If I use read-ahead to make allowance for permissions and remove paging
> (just keep prev/next) the problem goes away. Or I could use "best-guess"
> paging, which could range from 100% accurate to 99% wrong. At first glance
> that's not really acceptable, but I noticed recently Google does the same
> thing with their search results.
>
> First prize is to work out a proper solution that is fast and accurate and
> works on fairly large results, and I'm still hoping for some suggestions.
> But as a last resort I'll go the "best-guess" route. If Google can do it...
>
> Cheers
> Arno
>
>
>
Well, without seeing your DB structure I can't say for definite, but
that should be OK in a single SQL statement with some IF clauses, etc.
However, if you go with the multiple queries route and use PHP to sort,
just grab a list of article ID's a user (the current logged in one) has
access to read given a set of criteria, so you should end up with a list
of article IDs for explicit user permissions, user category permissions
and user container permissions.
You can then iterate each list to get the final permissions and totals.
This is very simplified, and needs fleshing out, but it should give you
the general idea.
Thanks,
Ash
http://www.ashleysheridan.co.uk
--- End Message ---
--- Begin Message ---
Instead of serializing the articles, you only need their IDs. Using
$sql .= ' where id in (' . implode(',', $ids) . ')';
you can load the data for a page of results in a single query. Storing the
IDs is much cheaper than the articles.
If the permissions are fairly static (i.e. access for user X to article Y
doesn't change every two minutes) you could create a calculated permission
table as a many-to-many between user and article. Here's the logic flow for
a query:
1. Run the query to find matching article IDs
2. Load permissions from table for all IDs
3. For each article without a calculated permission, calculate it and insert
a row (do a batch insert to save time)
If you flag the query in the middle tier as having been processed as above,
you can join to the calculated permissions each time you need another page.
The downside is that the code that runs the queries has to operate in two
modes: raw and joined to the permissions. If most users end up querying for
all articles, the table could grow. Plus you need to purge rows any time the
permissions for an article/user changes which could get fairly complicated.
David
-----------
Storing only the IDs is way cheaper than storing the entire resultset, and
I'd been thinking along the same lines. Getting a complete list of valid IDs
in the first place is turning out to be a different matter. The permissions
for article/user aren't that straight-forward, and in fact the most common
permissions are group/category and group/container, where an article can be
assigned to one or more category/containers. Using a temporary permission
table could be the solution. Thanks.
Cheers
Arno
--- End Message ---
--- Begin Message ---
On Wed, 2011-02-09 at 13:27 +0200, Arno Kuhl wrote:
> Instead of serializing the articles, you only need their IDs. Using
>
> $sql .= ' where id in (' . implode(',', $ids) . ')';
>
> you can load the data for a page of results in a single query. Storing the
> IDs is much cheaper than the articles.
>
> If the permissions are fairly static (i.e. access for user X to article Y
> doesn't change every two minutes) you could create a calculated permission
> table as a many-to-many between user and article. Here's the logic flow for
> a query:
>
> 1. Run the query to find matching article IDs
> 2. Load permissions from table for all IDs
> 3. For each article without a calculated permission, calculate it and insert
> a row (do a batch insert to save time)
>
> If you flag the query in the middle tier as having been processed as above,
> you can join to the calculated permissions each time you need another page.
> The downside is that the code that runs the queries has to operate in two
> modes: raw and joined to the permissions. If most users end up querying for
> all articles, the table could grow. Plus you need to purge rows any time the
> permissions for an article/user changes which could get fairly complicated.
>
> David
>
> -----------
>
> Storing only the IDs is way cheaper than storing the entire resultset, and
> I'd been thinking along the same lines. Getting a complete list of valid IDs
> in the first place is turning out to be a different matter. The permissions
> for article/user aren't that straight-forward, and in fact the most common
> permissions are group/category and group/container, where an article can be
> assigned to one or more category/containers. Using a temporary permission
> table could be the solution. Thanks.
>
> Cheers
> Arno
>
>
>
You can get the same set of results with a join betwixt the tables, and
it should be slightly faster than creating a temporary table if you've
got your indexes right.
Thanks,
Ash
http://www.ashleysheridan.co.uk
--- End Message ---
--- Begin Message ---
On 2/8/2011 4:58 PM, Donovan Brooke wrote:
Hello,
Just wanted to say thanks to those that helped me get through my first PHP
project (over the last month).
As is with much of the work we server-side language people do, the back-end
(non-public) side of this site is perhaps the more interesting.
However, here is the link to the site:
http://www.impactseven.org/
They have full control over the content in the admin pages, and much
of this content will soon change as I simply copy/pasted some of their old
site's content to the database fields.
btw, I7 is a great source for working capitol if you are in the need, and if you
are in Wisconsin, USA. ;-)
Also, for good karma ;-), here is a link to a small function library containing
just a few (mostly MySQL) functions that I created for this site:
http://www.euca.us/downloads/euca_phpmysql.zip (4KB)
(if used, please keep the 'www.euca.us' credit in place)
It has 4 functions:
dbconnect
global_id
list_formvars
list_vars
You can read all about them in the file, but here is the basic rundown.
dbconnect - basic connection/error reporting for MySQL
global_id - If you've ever run into data relations changing between
related tables, you may want to look into this one. ;-)
list_formvars - list all request vars (for testing) with the option to
display only certain matched vars.
list_vars - list all set vars (for testing) with option to display only
certain matched vars.
The later two I usually post either at the end of the page, or at the end of
page within <!-- --> for testing/development purposes.
Lastly, I'm sure I will add to this library as time goes by, but if
you find that you've used it and made changes, drop me the file so I
can learn as well.
Thanks again!,
Donovan
Suggestion: Design for XHTML 1.1. It really doesn't require any significant
additional effort and you'll already be current when it becomes the W3C
standard. I like it because it forces me to create better, cleaner html code.
--- End Message ---
--- Begin Message ---
From: Al
> On 2/8/2011 4:58 PM, Donovan Brooke wrote:
>> Hello,
>>
>> Just wanted to say thanks to those that helped me get through my
first PHP
>> project (over the last month).
>>
>> As is with much of the work we server-side language people do, the
back-end
>> (non-public) side of this site is perhaps the more interesting.
>>
>
> Suggestion: Design for XHTML 1.1. It really doesn't require any
significant
> additional effort and you'll already be current when it becomes the
W3C
> standard. I like it because it forces me to create better, cleaner
html code.
You should also use the HTML Validator plug-in for Firefox to make sure
you are producing valid XHTML. That makes it so much easier to find
those invisible problems. I can't count how many times it has pointed
right at a logic flaw in my code.
Bob McConnell
--- End Message ---
--- Begin Message ---
On 9 February 2011 14:57, Bob McConnell <[email protected]> wrote:
> From: Al
>
>> On 2/8/2011 4:58 PM, Donovan Brooke wrote:
>>> Hello,
>>>
>>> Just wanted to say thanks to those that helped me get through my
> first PHP
>>> project (over the last month).
>>>
>>> As is with much of the work we server-side language people do, the
> back-end
>>> (non-public) side of this site is perhaps the more interesting.
>>>
>>
>> Suggestion: Design for XHTML 1.1. It really doesn't require any
> significant
>> additional effort and you'll already be current when it becomes the
> W3C
>> standard. I like it because it forces me to create better, cleaner
> html code.
>
> You should also use the HTML Validator plug-in for Firefox to make sure
> you are producing valid XHTML. That makes it so much easier to find
> those invisible problems. I can't count how many times it has pointed
> right at a logic flaw in my code.
>
> Bob McConnell
Or go with the more likely candidate for a future html standard: html
5. Has the added benefit of easing you in to the new tags that will be
used as standard in a few years but won't be in xhtml.
Regards
Peter
--
<hype>
WWW: plphp.dk / plind.dk
LinkedIn: plind
BeWelcome/Couchsurfing: Fake51
Twitter: kafe15
</hype>
--- End Message ---
--- Begin Message ---
From: Peter Lind
> On 9 February 2011 14:57, Bob McConnell <[email protected]> wrote:
>> From: Al
>>
>>> On 2/8/2011 4:58 PM, Donovan Brooke wrote:
>>>> Hello,
>>>>
>>>> Just wanted to say thanks to those that helped me get through my
>> first PHP
>>>> project (over the last month).
>>>>
>>>> As is with much of the work we server-side language people do, the
>> back-end
>>>> (non-public) side of this site is perhaps the more interesting.
>>>>
>>>
>>> Suggestion: Design for XHTML 1.1. It really doesn't require any
>> significant
>>> additional effort and you'll already be current when it becomes the
>> W3C
>>> standard. I like it because it forces me to create better, cleaner
>> html code.
>>
>> You should also use the HTML Validator plug-in for Firefox to make sure
>> you are producing valid XHTML. That makes it so much easier to find
>> those invisible problems. I can't count how many times it has pointed
>> right at a logic flaw in my code.
>
> Or go with the more likely candidate for a future html standard: html
> 5. Has the added benefit of easing you in to the new tags that will be
> used as standard in a few years but won't be in xhtml.
I don't believe HTML 5 will ever be completed. Microsoft is working hard behind
the scenes to block it unless it only allows their codec's behind the video and
canvas tags. (Their efforts are very reminiscent of their sabotage of ISO with
the OOXML specification.) From a recent announcement(*), it appears that even
the committee has given up ever having a usable consensus, but will accept
whatever the browser developers want to implement even if they are incompatible
with other browsers. That's not a standard.
Bob McConnell
(*) <http://blog.whatwg.org/html-is-the-new-html5>
--- End Message ---
--- Begin Message ---
On 9 February 2011 17:22, Bob McConnell <[email protected]> wrote:
> From: Peter Lind
>
>> On 9 February 2011 14:57, Bob McConnell <[email protected]> wrote:
>>> From: Al
>>>
>>>> On 2/8/2011 4:58 PM, Donovan Brooke wrote:
>>>>> Hello,
>>>>>
>>>>> Just wanted to say thanks to those that helped me get through my
>>> first PHP
>>>>> project (over the last month).
>>>>>
>>>>> As is with much of the work we server-side language people do, the
>>> back-end
>>>>> (non-public) side of this site is perhaps the more interesting.
>>>>>
>>>>
>>>> Suggestion: Design for XHTML 1.1. It really doesn't require any
>>> significant
>>>> additional effort and you'll already be current when it becomes the
>>> W3C
>>>> standard. I like it because it forces me to create better, cleaner
>>> html code.
>>>
>>> You should also use the HTML Validator plug-in for Firefox to make sure
>>> you are producing valid XHTML. That makes it so much easier to find
>>> those invisible problems. I can't count how many times it has pointed
>>> right at a logic flaw in my code.
>>
>> Or go with the more likely candidate for a future html standard: html
>> 5. Has the added benefit of easing you in to the new tags that will be
>> used as standard in a few years but won't be in xhtml.
>
> I don't believe HTML 5 will ever be completed. Microsoft is working hard
> behind the scenes to block it unless it only allows their codec's behind the
> video and canvas tags. (Their efforts are very reminiscent of their sabotage
> of ISO with the OOXML specification.) From a recent announcement(*), it
> appears that even the committee has given up ever having a usable consensus,
> but will accept whatever the browser developers want to implement even if
> they are incompatible with other browsers. That's not a standard.
>
Html 5 has a fair chance of not being completed (that's going by what
Ian Hickson has stated as well as reading blogs and such on the matter
- see http://en.wikipedia.org/wiki/HTML5 for a pointer or two). That's
besides the point, though: html 5 *is* where the web is headed, as
opposed to xhtml. In part this is happening by browsers implementing
features piece by piece as they become stable - hence, you can already
use some html 5 features and be rather certain they'll work as
expected later on. MS may hate things not working how they like it,
but big and stupid as they are they have in fact figured out that the
web will happily move on without them - so they'll get round to
implementing html 5 features as well, at some point, even if it
doesn't use their proprietary code.
Regards
Peter
--
<hype>
WWW: plphp.dk / plind.dk
LinkedIn: plind
BeWelcome/Couchsurfing: Fake51
Twitter: kafe15
</hype>
--- End Message ---
--- Begin Message ---
Hi,
i have apache/PHP server installed on Windows 7.
my computer is a quad-core CPU with 6 GB RAM and i would like to speed up
PHP answer to requests.
how can i do that ?
which parameter should i tune ?
thx.
--
Alain
-----------------------------------------------------------
Windows 7 x64 / Fedora 14 x64
PostgreSQL 8.3.5 / MySQL 5
Apache 2.2.16
PHP 5.3.1
C# 2005-2008
--- End Message ---
--- Begin Message ---
---------- Forwarded message ----------
From: Alain Roger <[email protected]>
Date: Wed, Feb 9, 2011 at 6:45 PM
Subject: Re: [PHP] improve speed of PHP answers
To: [email protected]
yes i understand however even Linux has bugs and issues...
i'm currently installing Fedora 14 as web server and joomla has big issues
on fedora 14 with right permissions and writable configuration.php file...
everybody raises issue about that and till now none of their solution
worked... :-(
basically web page needs between 1 to 2 s to load, but as it is a testing
computer it is not a huge problem... main problem is that component for
joomla as akeeba backup fails to backup web site while under Windows XP it
worked perfectly...
A.
On Wed, Feb 9, 2011 at 6:41 PM, Alexis Antonakis <[email protected]>wrote:
> The Operating System :)
>
> It could be a million and one things...how long does it currently take to
> load a page and what is that page trying to do?
>
> Alexis
>
>
>
>
> On 09/02/11 10:36, Alain Roger wrote:
>
>> Hi,
>>
>> i have apache/PHP server installed on Windows 7.
>> my computer is a quad-core CPU with 6 GB RAM and i would like to speed up
>> PHP answer to requests.
>> how can i do that ?
>> which parameter should i tune ?
>> thx.
>>
>>
--
Alain
-----------------------------------------------------------
Windows 7 x64 / Fedora 14 x64
PostgreSQL 8.3.5 / MySQL 5
Apache 2.2.16
PHP 5.3.1
C# 2005-2008
--
Alain
-----------------------------------------------------------
Windows 7 x64 / Fedora 14 x64
PostgreSQL 8.3.5 / MySQL 5
Apache 2.2.16
PHP 5.3.1
C# 2005-2008
--- End Message ---
--- Begin Message ---
it's a php component for joomla so it is written in PHP.
it access to DB as also the whole site root structure to backup database
structure , data as also web site folders and files.
due to slow freeing ressources (php pointers, memory, andso on...) the
backup process fails each time...
A.
On Wed, Feb 9, 2011 at 6:47 PM, Alexis Antonakis <[email protected]>wrote:
> But what is the webpage trying to do?
> Is it straight html? Are you accessing data from a database? What exactly?
>
>
> On 09/02/11 10:45, Alain Roger wrote:
>
>> yes i understand however even Linux has bugs and issues...
>> i'm currently installing Fedora 14 as web server and joomla has big issues
>> on fedora 14 with right permissions and writable configuration.php file...
>> everybody raises issue about that and till now none of their solution
>> worked... :-(
>>
>> basically web page needs between 1 to 2 s to load, but as it is a testing
>> computer it is not a huge problem... main problem is that component for
>> joomla as akeeba backup fails to backup web site while under Windows XP it
>> worked perfectly...
>>
>> A.
>>
>> On Wed, Feb 9, 2011 at 6:41 PM, Alexis Antonakis<[email protected]
>> >wrote:
>>
>> The Operating System :)
>>>
>>> It could be a million and one things...how long does it currently take to
>>> load a page and what is that page trying to do?
>>>
>>> Alexis
>>>
>>>
>>>
>>>
>>> On 09/02/11 10:36, Alain Roger wrote:
>>>
>>> Hi,
>>>>
>>>> i have apache/PHP server installed on Windows 7.
>>>> my computer is a quad-core CPU with 6 GB RAM and i would like to speed
>>>> up
>>>> PHP answer to requests.
>>>> how can i do that ?
>>>> which parameter should i tune ?
>>>> thx.
>>>>
>>>>
>>>>
>>
>>
--
Alain
-----------------------------------------------------------
Windows 7 x64 / Fedora 14 x64
PostgreSQL 8.3.5 / MySQL 5
Apache 2.2.16
PHP 5.3.1
C# 2005-2008
--- End Message ---
--- Begin Message ---
On Wed, 2011-02-09 at 18:57 +0100, Alain Roger wrote:
> it's a php component for joomla so it is written in PHP.
> it access to DB as also the whole site root structure to backup database
> structure , data as also web site folders and files.
> due to slow freeing ressources (php pointers, memory, andso on...) the
> backup process fails each time...
>
> A.
>
> On Wed, Feb 9, 2011 at 6:47 PM, Alexis Antonakis <[email protected]>wrote:
>
> > But what is the webpage trying to do?
> > Is it straight html? Are you accessing data from a database? What exactly?
> >
> >
> > On 09/02/11 10:45, Alain Roger wrote:
> >
> >> yes i understand however even Linux has bugs and issues...
> >> i'm currently installing Fedora 14 as web server and joomla has big issues
> >> on fedora 14 with right permissions and writable configuration.php file...
> >> everybody raises issue about that and till now none of their solution
> >> worked... :-(
> >>
> >> basically web page needs between 1 to 2 s to load, but as it is a testing
> >> computer it is not a huge problem... main problem is that component for
> >> joomla as akeeba backup fails to backup web site while under Windows XP it
> >> worked perfectly...
> >>
> >> A.
> >>
> >> On Wed, Feb 9, 2011 at 6:41 PM, Alexis Antonakis<[email protected]
> >> >wrote:
> >>
> >> The Operating System :)
> >>>
> >>> It could be a million and one things...how long does it currently take to
> >>> load a page and what is that page trying to do?
> >>>
> >>> Alexis
> >>>
> >>>
> >>>
> >>>
> >>> On 09/02/11 10:36, Alain Roger wrote:
> >>>
> >>> Hi,
> >>>>
> >>>> i have apache/PHP server installed on Windows 7.
> >>>> my computer is a quad-core CPU with 6 GB RAM and i would like to speed
> >>>> up
> >>>> PHP answer to requests.
> >>>> how can i do that ?
> >>>> which parameter should i tune ?
> >>>> thx.
> >>>>
> >>>>
> >>>>
> >>
> >>
>
>
Surely there are already plenty of backup modules for Joomla?
If you're writing it as a module for that CMS, don't forget that you're
inheriting the whole memory footprint of Joomla at the same time. I
recently ran into such a problem on a project written on the CodeIgniter
framework. Now CodeIgniter isn't that heavy on resources, but even it
struggled with the default memory available to it.
If you're trying to speed up any PHP script, generally you need to step
through it and see what code you can remove entirely, or alter to have
less of an impact. For example, PHP code that grabs a set of results
from the DB and then filters that down is far slower and uses more
memory than if you left the filtering in MySQL.
Also, look at freeing up resources when you don't need them. Avoid large
global variables (if they are global the garbage collection can't work
until the script has ended).
Perhaps split the task into several smaller ones that are more
manageable at once for the server.
Finally, you can increase the memory that can be allocated to a PHP
script, but address this last, as the other factors are more important
in my opinion.
Thanks,
Ash
http://www.ashleysheridan.co.uk
--- End Message ---
--- Begin Message ---
exec-sum update :
details continued at http://curl.haxx.se/mail/lib-2011-02/0101.html
it turns out using the internetdomainname of the apache server is the
source of the problem.
the adsl router could be dropping the connection, or something else.
calling curl_exec() to a localhost address will work fine for the time
that it takes.
(no traffic back for 1hr, then a short status msg only)
however my media import routines are useless if i only can use 'm on localhost.
so the idea i will try tommorow is outputting keepalive weird-bytes on
the text/html level
from the php_script import-script, and then stripping them in php_daemon_script.
i would like to know if i can use feof() to check that ffmpeg launched
via popen() has terminated..
i think so, but the docs aren't perfectly clear.
--- End Message ---
