Re: [fw-general] Strange problem while batch indexing documents into Zend Lucene
Thanks for the reply. I will take a look at the API and see if I can find something of that sort. You are probably right about me going about the indexing all wrong. In fact that would make the most sense to me. I would love to see examples of how other people have successfully indexed hundreds of thousands of documents at once, or alternative suggestions of how I should go about doing it. Is it generally a bad idea to run such scripts from a browser? Should I be running them from the command line instead? -jay On 2/8/08, Michael B Allen <[EMAIL PROTECTED]> wrote: > > I don't know spit about Zend_Lucene but I'm going to make some bold > statements and wild guesses. Here goes ... > > Just from looking at the point of failure in your stack trace it's > clear that it's waiting for an index to be generated. Which is to > suggest that perhaps Zend_Lucene is spawning something that generates > the index and then polls for the result. I would look at the API and > see if there's an alternative MO where you invoke the index generator > but the calling script completes immediately. Then independently > you're supposed to query the server periodically to determine if the > index generation has completed. > > Again, I don't know anything about Zend_Lucene but in my experience > when something takes a long time (e.g. more than 30 seconds) you > really don't want to have the browser just sitting there waiting for > the result. So, perhaps the Zend_Lucene people have considered this > scenario and you need to refactor how your using it overall? > > Mike >
Re: [fw-general] Strange problem while batch indexing documents into Zend Lucene
I don't know spit about Zend_Lucene but I'm going to make some bold statements and wild guesses. Here goes ... Just from looking at the point of failure in your stack trace it's clear that it's waiting for an index to be generated. Which is to suggest that perhaps Zend_Lucene is spawning something that generates the index and then polls for the result. I would look at the API and see if there's an alternative MO where you invoke the index generator but the calling script completes immediately. Then independently you're supposed to query the server periodically to determine if the index generation has completed. Again, I don't know anything about Zend_Lucene but in my experience when something takes a long time (e.g. more than 30 seconds) you really don't want to have the browser just sitting there waiting for the result. So, perhaps the Zend_Lucene people have considered this scenario and you need to refactor how your using it overall? Mike
[fw-general] Strange problem while batch indexing documents into Zend Lucene
I have a mysql table consisting of about 500K rows which is about 300MB in
size, that I've been trying to index into Zend Lucene. It worked perfectly
fine when adding small batches at a time, but if I tried to index more than
2000 at a time, the script would do strange things after indexing around
2500 documents. The script would end prematurely, and I would end up with
some documents being indexed twice.
I stripped the script down to the most basic form I could where it still
produced the problem, and realized what was happening. First of all, here is
the code:
require_once('Zend/Search/Lucene.php');
$index = Zend_Search_Lucene::open('lucene/videos');
set_time_limit(0);
for ($x=0;$x<20;$x++)
{
$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::Keyword('field',
'value'));
$index->addDocument($doc);
}
As you can see, I'm not retrieving anything from the DB anymore, i'm just
trying to see how many test documents I can insert before the script fails.
The script ran fine with $x<10. When I upped it to 20, the problem
occurred.
I realized after some testing that it wasn't the number of documents that
was the issue, it was an issue of time. I added a line at the beginning of
the script to insert a timestamped row into a test table every time the
script started running, and discovered that for some reason, if the script
executed for more than a certain amount of time, it would start over again
from the beginning. Actually, it wouldn't just start over, but it would
clone another instance of itself, which would run alongside of the original
instance of the script. Then sometime after that, it would clone another,
and another, and then all the scripts would fizzle out for unknown reasons,
sometimes giving an error.
So I would run the script, and check the test table, and see anywhere from 2
to 5 rows, indicating that the script had started up again that many times.
While the number of documents it would index before starting over would
vary, the TIME would remain the same. The first restart would always come
EXACTLY 5 minutes and 4 seconds after the initial running of the script. I
know this by comparing the timestamps of the two table rows. Then any
successive rows would be almost exactly 3 minutes after the previous one,
but sometimes 2 minutes.
While observing the actual PHP processes in linux TOP, I'd see the lsphp5
process running at high cpu power as the script executes, and when the
"restarts" occur, I would see multiple instances of lsphp5 appear, all
running at high CPU%. (I am using Litespeed web server, and have tried with
Apache and the same thing happened with the httpd processes.)
It is absolutely blowing my mind as to why this is happening. Does anyone
have any clue at all what could be going on, or anything else I could try?
Here are the 2 errors I have managed to catch. These are the errors I
sometimes would see when the script (or one of them at least) died after
executing it from my web browser. Sometimes I'd see this error but other
instances of the script would continue to run on (as I watched from TOP).
Fatal error: Uncaught exception 'Zend_Search_Lucene_Exception' with
message 'Index is under processing now' in
/home/public_html/boombada.com/private/ZendFramework-1.0.3/library/Zend/Search/Lucene.php:240
Stack trace: #0
/home/public_html/boombada.com/private/ZendFramework-1.0.3/library/Zend/Search/Lucene.php(416):
Zend_Search_Lucene::getActualGeneration(Object(Zend_Search_Lucene_Storage_Directory_Filesystem))
#1
/home/public_html/boombada.com/private/ZendFramework-1.0.3/library/Zend/Search/Lucene.php(175):
Zend_Search_Lucene->__construct('lucene/videos', true) #2
/home/public_html/boombada.com/private/system/application/controllers/test.php(184):
Zend_Search_Lucene::open('lucene/videos') #3 [internal function]:
Test->luceneC() #4
/home/public_html/private/system/codeigniter/CodeIgniter.php(216):
call_user_func_array(Array, Array) #5
/home/public_html/public/index.php(132):
require_once('/home/public_ht...') #6 {main} thrown in
/home/public_html/private/ZendFramework-1.0.3/library/Zend/Search/Lucene.php
on line 240
Fatal error: Ignoring exception from
Zend_Search_Lucene_Proxy::__destruct() while an exception is already
active (Uncaught Zend_Search_Lucene_Exception in
/home/public_html/private/ZendFramework-1.0.3/library/Zend/Search/Lucene/Storage/File/Filesystem.php
on line 59) in /home/public_html/private/system/application/controllers/test.php
on line 228
Here are a few things I think I've ruled out:
1) Operating system
I've tried running the script on both Ubuntu Gutsy and Windows XP, and the
same thing happens on both. The times between the restarts is longer on XP,
but they still occur.
2) Web server
I've tried both Litespeed and Apache, the problem occurs on both.
3) Framework
I'm running this script from the CodeIgniter framework, but have tried
Re: [fw-general] Zend_Form - Zend_Form_Element_Multiselect using ini file...
-- Peter Atkins <[EMAIL PROTECTED]> wrote (on Friday, 08 February 2008, 01:22 PM -0800): > How do I set my options for my select in my ini file? In the second example > it makes sense but I could not find any examples for ini files. > > My usage example: > > ccard.form.elements.card_xmonth.type = "multiselect" > ccard.form.elements.card_xmonth.options.label = "Expiration Month: " > ccard.form.elements.card_xmonth.options.required = true > ccard.form.elements.card_xmonth.options.select = ccard.form.elements.card_xmonth.options.multiOptions. = The various elements that extend Zend_Form_Element_Multi all have accessors for 'multiOptions', which contains the key/value pairs used to create the options list. -- Matthew Weier O'Phinney PHP Developer| [EMAIL PROTECTED] Zend - The PHP Company | http://www.zend.com/
Re: [fw-general] Zend_Form - Zend_Form_Element_Multiselect using ini file...
Okay never mind I figured it out.
ccard.form.elements.card_xmonth.options.multiOptions.ca = "CA"
ccard.form.elements.card_xmonth.options.multiOptions.ct = "CT"
On Feb 8, 2008, at 1:22 PM, Peter Atkins wrote:
All,
How do I set my options for my select in my ini file? In the second
example it makes sense but I could not find any examples for ini
files.
My usage example:
ccard.form.elements.card_xmonth.type = "multiselect"
ccard.form.elements.card_xmonth.options.label = "Expiration Month: "
ccard.form.elements.card_xmonth.options.required = true
ccard.form.elements.card_xmonth.options.select =
General Example:
$element = new Zend_Form_Element_Multiselect('test');
$element->setLabel('My Tester');
$options = array(
'foo' => 'bar',
'derp' => 'zot'
);
$element->setMultiOptions($options);
$element->addValidator('InArray', true, array(array_keys($options)));
Cheers,
Peter
[fw-general] Zend_Form - Zend_Form_Element_Multiselect using ini file...
All,
How do I set my options for my select in my ini file? In the second
example it makes sense but I could not find any examples for ini files.
My usage example:
ccard.form.elements.card_xmonth.type = "multiselect"
ccard.form.elements.card_xmonth.options.label = "Expiration Month: "
ccard.form.elements.card_xmonth.options.required = true
ccard.form.elements.card_xmonth.options.select =
General Example:
$element = new Zend_Form_Element_Multiselect('test');
$element->setLabel('My Tester');
$options = array(
'foo' => 'bar',
'derp' => 'zot'
);
$element->setMultiOptions($options);
$element->addValidator('InArray', true, array(array_keys($options)));
Cheers,
Peter
Re: [fw-general] Proporsal under review?
Hy Aldemar, Your two proposals are happily coupled with 27 others which are waiting to get reviewed... they are not lonly :D But my two have only one cousine and are waiting to get approved. :o Myself was also already affected by others because Zend_File_Transfer has not moved on. But as Metthew mentioned there is much work until 1.5 has been finished and we, inclusive me, want to have 1.5 finished very fast. So instead of writing mails to force the finishing of the approvement I fix some issues. We all will have to wait for some days / weeks... that's life... And I think it will take some time to get all actual 30 proposals approved by the dev team. :-) Much much work... Greetings Thomas Weidner, I18N Team Leader http://www.thomasweidner.com - Original Message - From: "Aldemar Bernal" <[EMAIL PROTECTED]> To: "Matthew Weier O'Phinney" <[EMAIL PROTECTED]>; Sent: Friday, February 08, 2008 6:50 PM Subject: Re: [fw-general] Proporsal under review? Good to hear that, my two proporsals: a.. Zend_Acl & MVC Integration - Aldemar Bernal a.. Zend_Acl dynamic loading - Aldemar Bernal were feeling quite lonely =P Aldemar PS: I said nothing has move to 'Proporsals under review' in months, not nothing has happened in months =P , I know you've been working a lot =) - Original Message - From: "Matthew Weier O'Phinney" <[EMAIL PROTECTED]> To: Sent: Friday, February 08, 2008 12:40 PM Subject: Re: [fw-general] Proporsal under review? -- Aldemar Bernal <[EMAIL PROTECTED]> wrote (on Friday, 08 February 2008, 12:04 PM -0500): anyonw knows what's going on with zf proporsals? nothing has move to -Proporsal Under Review- in months. We *are* aware of the lack of movement. :-) However, your statement that nothing has happened in "months" is not entirely true; we've reviewed and approved a number of proposals in the past month, and a number of new proposals have been marked as ready for review or under review. However, I'll be the first to agree that the momentum is sluggish. That said, the internal Zend team has been focussed for the past 6-8 weeks on completing requirements for the 1.5.0 release, and once we finalize that release, we'll be addressing the proposal process, and hopefully making it both easier and faster. Stay tuned! -- Matthew Weier O'Phinney PHP Developer| [EMAIL PROTECTED] Zend - The PHP Company | http://www.zend.com/
[fw-general] Zend_Form creation, reuse and layout
All,
We have now started working with Zend_Forms and have a few questions.
1) How could we have multiple forms added on one page but each live in
a separate config file?
We reuse the billing form through the site depending on users actions.
So we want one form config that we can reuse but when we build the
form object it puts in the form tags and has limited layout options.
See code/configs below:
$config = new Zend_Config_Ini('../application/config/billingForm.ini',
'development');
$bill_form = new Zend_Form($config->bill->form);
See code/configs below:
$shipconfig = new Zend_Config_Ini('../application/config/
shippingForm.ini', 'development');
$ship_form = new Zend_Form($shipconfig->ship->form);
return $form;
2) We also want content between the billing form and shipping form:
Content One
billing form display
Content Two
shipping form display
More Content
3) The last thing we need to do is add some AJAX calls for zip
validation, etc.
Any help would be great this is something we just started and the docs
are good but we need more a little more hand holding.
Code Example:
BillingForm.ini
ShippingForm.ini
CreditCardForm.ini
BillingForm.ini --
[development]
; billing form --
; general form metainformation
bill.form.action = "/cart/process"
bill.form.method = "post"
; First Name element
bill.form.elements.bill_firstname.type = "text"
bill.form.elements.bill_firstname.options.label = "First Name: "
bill.form.elements.bill_firstname.options.validators.alnum.validator =
"alnum"
bill.form.elements.bill_firstname.options.validators.regex.validator =
"regex"
bill
.form.elements.bill_firstname.options.validators.regex.options.pattern
= "/^[a-z]/i"
bill.form.elements.bill_firstname.options.validators.strlen.validator
= "StringLength"
bill
.form.elements.bill_firstname.options.validators.strlen.options.min =
"2"
;bill
.form.elements.bill_firstname.options.validators.strlen.options.max =
"50"
bill.form.elements.bill_firstname.options.required = true
; Last Name element
bill.form.elements.bill_lastname.type = "text"
bill.form.elements.bill_lastname.options.label = "Last Name: "
bill.form.elements.bill_lastname.options.validators.alnum.validator =
"alnum"
bill.form.elements.bill_lastname.options.validators.regex.validator =
"regex"
bill
.form.elements.bill_lastname.options.validators.regex.options.pattern
= "/^[a-z]/i"
bill.form.elements.bill_lastname.options.validators.strlen.validator =
"StringLength"
bill.form.elements.bill_lastname.options.validators.strlen.options.min
= "2"
;bill
.form.elements.bill_lastname.options.validators.strlen.options.max =
"50"
bill.form.elements.bill_lastname.options.required = true
etc...
ShippingForm.ini --
[development]
; shipping form --
; general form metainformation
ship.form.action = "/cart/process"
ship.form.method = "post"
; First Name element
ship.form.elements.ship_firstname.type = "text"
ship.form.elements.ship_firstname.options.label = "First Name: "
ship.form.elements.ship_firstname.options.validators.alnum.validator =
"alnum"
ship.form.elements.ship_firstname.options.validators.regex.validator =
"regex"
ship
.form.elements.ship_firstname.options.validators.regex.options.pattern
= "/^[a-z]/i"
ship.form.elements.ship_firstname.options.validators.strlen.validator
= "StringLength"
ship
.form.elements.ship_firstname.options.validators.strlen.options.min =
"2"
;ship
.form.elements.ship_firstname.options.validators.strlen.options.max =
"50"
ship.form.elements.ship_firstname.options.required = true
; Last Name element
ship.form.elements.ship_lastname.type = "text"
ship.form.elements.ship_lastname.options.label = "Last Name: "
ship.form.elements.ship_lastname.options.validators.alnum.validator =
"alnum"
ship.form.elements.ship_lastname.options.validators.regex.validator =
"regex"
ship
.form.elements.ship_lastname.options.validators.regex.options.pattern
= "/^[a-z]/i"
ship.form.elements.ship_lastname.options.validators.strlen.validator =
"StringLength"
ship.form.elements.ship_lastname.options.validators.strlen.options.min
= "2"
;ship
.form.elements.ship_lastname.options.validators.strlen.options.max =
"50"
ship.form.elements.ship_lastname.options.required = true
etc...
Cheers,
Peter
Re: [fw-general] Proporsal under review?
Good to hear that, my two proporsals: a.. Zend_Acl & MVC Integration - Aldemar Bernal a.. Zend_Acl dynamic loading - Aldemar Bernal were feeling quite lonely =P Aldemar PS: I said nothing has move to 'Proporsals under review' in months, not nothing has happened in months =P , I know you've been working a lot =) - Original Message - From: "Matthew Weier O'Phinney" <[EMAIL PROTECTED]> To: Sent: Friday, February 08, 2008 12:40 PM Subject: Re: [fw-general] Proporsal under review? > -- Aldemar Bernal <[EMAIL PROTECTED]> wrote > (on Friday, 08 February 2008, 12:04 PM -0500): >> anyonw knows what's going on with zf proporsals? nothing has move to >> -Proporsal Under Review- in months. > > We *are* aware of the lack of movement. :-) However, your statement that > nothing has happened in "months" is not entirely true; we've reviewed > and approved a number of proposals in the past month, and a number of > new proposals have been marked as ready for review or under review. > However, I'll be the first to agree that the momentum is sluggish. > > That said, the internal Zend team has been focussed for the past 6-8 > weeks on completing requirements for the 1.5.0 release, and once we > finalize that release, we'll be addressing the proposal process, and > hopefully making it both easier and faster. Stay tuned! > > -- > Matthew Weier O'Phinney > PHP Developer| [EMAIL PROTECTED] > Zend - The PHP Company | http://www.zend.com/ >
Re: [fw-general] Proporsal under review?
-- Aldemar Bernal <[EMAIL PROTECTED]> wrote (on Friday, 08 February 2008, 12:04 PM -0500): > anyonw knows what's going on with zf proporsals? nothing has move to > -Proporsal Under Review- in months. We *are* aware of the lack of movement. :-) However, your statement that nothing has happened in "months" is not entirely true; we've reviewed and approved a number of proposals in the past month, and a number of new proposals have been marked as ready for review or under review. However, I'll be the first to agree that the momentum is sluggish. That said, the internal Zend team has been focussed for the past 6-8 weeks on completing requirements for the 1.5.0 release, and once we finalize that release, we'll be addressing the proposal process, and hopefully making it both easier and faster. Stay tuned! -- Matthew Weier O'Phinney PHP Developer| [EMAIL PROTECTED] Zend - The PHP Company | http://www.zend.com/
[fw-general] Proporsal under review?
Hi, anyonw knows what's going on with zf proporsals? nothing has move to -Proporsal Under Review- in months. Aldemar
[fw-general] Announcement: New core component
Dear Community, I am pleased to announce that the translation view helper Zend_View_Helper_Translate has approved for core. So we now have an official way for translation within the View and Form components. See the documentation for proper usage and have fun with it... Greetings Thomas Weidner, I18N Team Leader http://www.thomasweidner.com
Re: [fw-general] Filtering inputs
Hi Łukasz, You're not doing anything wrong that I can see. :) This behavior of Zend_Filter_Input is a side effect of Zend_Loader no longer throwing an exception when Z_F_I had been expecting it to do so. Simon Mundy has a nice solution for this that lives in the incubator at the moment: http://framework.zend.com/issues/browse/ZF-2550 I am inclined to say that we should be able to promote the improvements to the trunk very soon, though I'm a bit unsure about backward-compatibility. Perhaps you and others can comment on the issue? Thanks! Best regards, Darby Łukasz Wojciechowski wrote: > I'm doing such thing: http://pastebin.com/m11af8f6f > > and it generates load of warnings, here are examples (there are 16 > warning in total): > > Warning: Zend_Loader::include_once(Zend\Filter\Between.php) > [function.Zend-Loader-include-once]: failed to open stream: No such > file or directory in > C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 > > Warning: Zend_Loader::include_once() [function.include]: Failed > opening 'Zend\Filter\Between.php' for inclusion > (include_path='.;\xampplite\php\pear\;C:\xampplite\htdocs\imprezowicz/lib/ZendIncubator;C:\xampplite\htdocs\imprezowicz/lib;C:\xampplite\htdocs\imprezowicz/app') > in C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 > > Warning: Zend_Loader::include_once(Zend\Filter\EmailAddress.php) > [function.Zend-Loader-include-once]: failed to open stream: No such > file or directory in > C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 > > Warning: Zend_Loader::include_once(Zend\Filter\GreaterThan.php) > [function.Zend-Loader-include-once]: failed to open stream: No such > file or directory in > C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 > > etc... > > Why are validators looked for in Filters library? or what am I doing wrong? >
[fw-general] Filtering inputs
I'm doing such thing: http://pastebin.com/m11af8f6f and it generates load of warnings, here are examples (there are 16 warning in total): Warning: Zend_Loader::include_once(Zend\Filter\Between.php) [function.Zend-Loader-include-once]: failed to open stream: No such file or directory in C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 Warning: Zend_Loader::include_once() [function.include]: Failed opening 'Zend\Filter\Between.php' for inclusion (include_path='.;\xampplite\php\pear\;C:\xampplite\htdocs\imprezowicz/lib/ZendIncubator;C:\xampplite\htdocs\imprezowicz/lib;C:\xampplite\htdocs\imprezowicz/app') in C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 Warning: Zend_Loader::include_once(Zend\Filter\EmailAddress.php) [function.Zend-Loader-include-once]: failed to open stream: No such file or directory in C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 Warning: Zend_Loader::include_once(Zend\Filter\GreaterThan.php) [function.Zend-Loader-include-once]: failed to open stream: No such file or directory in C:\xampplite\htdocs\imprezowicz\lib\Zend\Loader.php on line 83 etc... Why are validators looked for in Filters library? or what am I doing wrong? -- Łukasz Wojciechowski
[fw-general] Zend_CheckDigit ready for community review
Hi all, This component will include Luhn (MOD10) and Verhoeff to begin with, and possibly some more. http://framework.zend.com/wiki/display/ZFPROP/Zend_CheckDigit+Proposal+-+Ste ven+Brown I imagine this could be implemented pretty quickly, so let me know if you have any questions/comments/suggestions. Cheers, Steven
Re: [fw-general] Using include_once
Thanks Philip,
I'll follow your advice, meanwhile I had to put the include files into the
bootstrap file. If I place them in the controller files, they fail.
For now, my problem is solved.
Philip G wrote:
>
> On Feb 7, 2008 11:51 AM, till <[EMAIL PROTECTED]> wrote:
>>
>> If you have no access to your php.ini, check various logs (syslog,
>> etc.) for PHP errors.
>>
>> Cheers,
>> Till
>>
>
> Slight correction / addition. If you do not have access to php.ini use:
>
> error_reporting(E_ALL);
> ini_set('display_errors', true);
>
> Place that above ALL your code, before you even do one include.
>
> --
> Philip
> [EMAIL PROTECTED]
> http://www.gpcentre.net/
>
>
--
View this message in context:
http://www.nabble.com/Using-include_once-tp15333284s16154p15351265.html
Sent from the Zend Framework mailing list archive at Nabble.com.
