Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Matthew Weier O'Phinney]

-- Mark Maynereid <[EMAIL PROTECTED]> wrote
(on Sunday, 09 March 2008, 09:06 PM +):
> Sorry the question provoked repetition. 

No worries. It helps to clarify these things.

> I get it now and look forward to post-filtering (if it goes ahead).

It will, but not until after the 1.5.0 final release. If you want to
watch progress, follow this issue:

http://framework.zend.com/issues/browse/ZF-2834


> On Sunday 09 March 2008 16:43, Matthew Weier O'Phinney wrote:
> > -- Mark Maynereid <[EMAIL PROTECTED]> wrote
> >
> > (on Saturday, 08 March 2008, 06:31 PM +):
> > > On Saturday 08 March 2008 13:53, Matthew Weier O'Phinney wrote:
> > > > I'll consider a postfilter post-1.5.0 GA release when I start looking
> > > > at associating forms with models.
> > >
> > > I sense I would welcome this as my efforts on CRUD seem to be focusing me
> > > towards these issues from a generic standpoint. Perhaps like some others,
> > > my table classes provide all meta info to enable automated form building.
> > >
> > > But wouldn't this post-filter addition also begin to make Zend_Form
> > > completely overlap ZFI? I did ask about ZFI's role some time ago back
> > > when optional fields in Zend_Form were not yet supported and never quite
> > > understood why ZFI was not leveraged wholesale by Zend_Form (perhaps with
> > > BC adaption to ZFI)?
> > >
> > > With a postfilter addition being considered now, it looks to me like
> > > convergence, and so I can't help wondering again if ZFI should be serving
> > > this role for Zend_Form?
> > >
> > > Sorry if I don't understand it well enough but it just seems to me like
> > > fairly significant duplication. Which is better? Added BC support for
> > > Zend_Form's needs within ZFI,  or, what looks to me like a trend towards
> > > duplication with ZFI (perhaps for sounds reasons)?
> >
> > As I've mentioned several times on the list and in issue reports, ZFI's
> > design does not fit Zend_Form. I tried, but the problem is summed up
> > with the following:
> >
> >   * Zend_Form requires that filter and validation chains be per-element:
> > * to allow re-use of an element with multiple forms
> > * to allow validating single elements
> >   * to allow Zend_Form the ability to do partial validation
> >   * Zend_Filter_Input operates on *sets* of data
> > * The data set is either valid or not, depending on whether or not
> >   all items in the set validate
> >
> > ZFI does not work for Zend_Form's design because the form itself has no
> > awareness of the filter and validation chains; it delegates all of this
> > to the elements. This delegation is necessary to, as noted, allow re-use
> > of elements, as well as easy customization of existing elements on a
> > per-form or per-element basis.
> >
> > Basically, I see ZFI's role as being for those who have a need for a
> > lightweight solution -- they may have very simple form needs, or they
> > may be tying validation into their models directly, and have no need to
> > have the models render their own forms for any number of reasons: they
> > may be using the model with web services such as SOAP, REST, or XML-RPC;
> > they may have CLI tools that interface with the model; etc.
> >
> > Zend_Form is for those who need to be able to *render* the various input
> > fields as forms, and who may need to be able to validate individual
> > elements or subsets of elements.  Right now, that rendering is
> > restricted to XHTML, but you could easily add XForms, FPDF, GTK, or
> > ncurses into the mix to have other types of forms. Regarding validation,
> > ZFI requires you to validate a *set* of data; you can then query to see
> > if individual elements were valid, but it is not designed to validate a
> > single element or a subset of elements. These two things significantly
> > differentiate the purposes of the two classes.
> >
> > Yes, there is some duplication and overlap. They do very similar things,
> > but they do so with slightly different goals and with different
> > responsibilities. You could certainly engineer Zend_Form to act simply
> > as a filter/validator chain ala ZFI (particularly as you can now disable
> > loading decorators), but it's a heavier-weight solution than ZFI.
> 

-- 
Matthew Weier O'Phinney
PHP Developer| [EMAIL PROTECTED]
Zend - The PHP Company   | http://www.zend.com/

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Mark Maynereid]

Sorry the question provoked repetition. I get it now and look forward to 
post-filtering (if it goes ahead).


On Sunday 09 March 2008 16:43, Matthew Weier O'Phinney wrote:
> -- Mark Maynereid <[EMAIL PROTECTED]> wrote
>
> (on Saturday, 08 March 2008, 06:31 PM +):
> > On Saturday 08 March 2008 13:53, Matthew Weier O'Phinney wrote:
> > > I'll consider a postfilter post-1.5.0 GA release when I start looking
> > > at associating forms with models.
> >
> > I sense I would welcome this as my efforts on CRUD seem to be focusing me
> > towards these issues from a generic standpoint. Perhaps like some others,
> > my table classes provide all meta info to enable automated form building.
> >
> > But wouldn't this post-filter addition also begin to make Zend_Form
> > completely overlap ZFI? I did ask about ZFI's role some time ago back
> > when optional fields in Zend_Form were not yet supported and never quite
> > understood why ZFI was not leveraged wholesale by Zend_Form (perhaps with
> > BC adaption to ZFI)?
> >
> > With a postfilter addition being considered now, it looks to me like
> > convergence, and so I can't help wondering again if ZFI should be serving
> > this role for Zend_Form?
> >
> > Sorry if I don't understand it well enough but it just seems to me like
> > fairly significant duplication. Which is better? Added BC support for
> > Zend_Form's needs within ZFI,  or, what looks to me like a trend towards
> > duplication with ZFI (perhaps for sounds reasons)?
>
> As I've mentioned several times on the list and in issue reports, ZFI's
> design does not fit Zend_Form. I tried, but the problem is summed up
> with the following:
>
>   * Zend_Form requires that filter and validation chains be per-element:
> * to allow re-use of an element with multiple forms
> * to allow validating single elements
>   * to allow Zend_Form the ability to do partial validation
>   * Zend_Filter_Input operates on *sets* of data
> * The data set is either valid or not, depending on whether or not
>   all items in the set validate
>
> ZFI does not work for Zend_Form's design because the form itself has no
> awareness of the filter and validation chains; it delegates all of this
> to the elements. This delegation is necessary to, as noted, allow re-use
> of elements, as well as easy customization of existing elements on a
> per-form or per-element basis.
>
> Basically, I see ZFI's role as being for those who have a need for a
> lightweight solution -- they may have very simple form needs, or they
> may be tying validation into their models directly, and have no need to
> have the models render their own forms for any number of reasons: they
> may be using the model with web services such as SOAP, REST, or XML-RPC;
> they may have CLI tools that interface with the model; etc.
>
> Zend_Form is for those who need to be able to *render* the various input
> fields as forms, and who may need to be able to validate individual
> elements or subsets of elements.  Right now, that rendering is
> restricted to XHTML, but you could easily add XForms, FPDF, GTK, or
> ncurses into the mix to have other types of forms. Regarding validation,
> ZFI requires you to validate a *set* of data; you can then query to see
> if individual elements were valid, but it is not designed to validate a
> single element or a subset of elements. These two things significantly
> differentiate the purposes of the two classes.
>
> Yes, there is some duplication and overlap. They do very similar things,
> but they do so with slightly different goals and with different
> responsibilities. You could certainly engineer Zend_Form to act simply
> as a filter/validator chain ala ZFI (particularly as you can now disable
> loading decorators), but it's a heavier-weight solution than ZFI.

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Matthew Weier O'Phinney]

-- Mark Maynereid <[EMAIL PROTECTED]> wrote
(on Saturday, 08 March 2008, 06:31 PM +):
> On Saturday 08 March 2008 13:53, Matthew Weier O'Phinney wrote:
> > I'll consider a postfilter post-1.5.0 GA release when I start looking at
> > associating forms with models.
> 
> I sense I would welcome this as my efforts on CRUD seem to be focusing me 
> towards these issues from a generic standpoint. Perhaps like some others, my 
> table classes provide all meta info to enable automated form building.
> 
> But wouldn't this post-filter addition also begin to make Zend_Form 
> completely 
> overlap ZFI? I did ask about ZFI's role some time ago back when optional 
> fields in Zend_Form were not yet supported and never quite understood why ZFI 
> was not leveraged wholesale by Zend_Form (perhaps with BC adaption to ZFI)?
>
> With a postfilter addition being considered now, it looks to me like 
> convergence, and so I can't help wondering again if ZFI should be serving 
> this role for Zend_Form?
>
> Sorry if I don't understand it well enough but it just seems to me like 
> fairly 
> significant duplication. Which is better? Added BC support for Zend_Form's 
> needs within ZFI,  or, what looks to me like a trend towards duplication with 
> ZFI (perhaps for sounds reasons)?

As I've mentioned several times on the list and in issue reports, ZFI's
design does not fit Zend_Form. I tried, but the problem is summed up
with the following:

  * Zend_Form requires that filter and validation chains be per-element:
* to allow re-use of an element with multiple forms
* to allow validating single elements
  * to allow Zend_Form the ability to do partial validation
  * Zend_Filter_Input operates on *sets* of data
* The data set is either valid or not, depending on whether or not
  all items in the set validate

ZFI does not work for Zend_Form's design because the form itself has no
awareness of the filter and validation chains; it delegates all of this
to the elements. This delegation is necessary to, as noted, allow re-use
of elements, as well as easy customization of existing elements on a
per-form or per-element basis.

Basically, I see ZFI's role as being for those who have a need for a
lightweight solution -- they may have very simple form needs, or they
may be tying validation into their models directly, and have no need to
have the models render their own forms for any number of reasons: they
may be using the model with web services such as SOAP, REST, or XML-RPC;
they may have CLI tools that interface with the model; etc. 

Zend_Form is for those who need to be able to *render* the various input
fields as forms, and who may need to be able to validate individual
elements or subsets of elements.  Right now, that rendering is
restricted to XHTML, but you could easily add XForms, FPDF, GTK, or
ncurses into the mix to have other types of forms. Regarding validation,
ZFI requires you to validate a *set* of data; you can then query to see
if individual elements were valid, but it is not designed to validate a
single element or a subset of elements. These two things significantly
differentiate the purposes of the two classes.

Yes, there is some duplication and overlap. They do very similar things,
but they do so with slightly different goals and with different
responsibilities. You could certainly engineer Zend_Form to act simply
as a filter/validator chain ala ZFI (particularly as you can now disable
loading decorators), but it's a heavier-weight solution than ZFI.

-- 
Matthew Weier O'Phinney
PHP Developer| [EMAIL PROTECTED]
Zend - The PHP Company   | http://www.zend.com/

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Mark Maynereid]

On Saturday 08 March 2008 13:53, Matthew Weier O'Phinney wrote:
> I'll consider a postfilter post-1.5.0 GA release when I start looking at
> associating forms with models.

Hi Matthew,

I sense I would welcome this as my efforts on CRUD seem to be focusing me 
towards these issues from a generic standpoint. Perhaps like some others, my 
table classes provide all meta info to enable automated form building.

But wouldn't this post-filter addition also begin to make Zend_Form completely 
overlap ZFI? I did ask about ZFI's role some time ago back when optional 
fields in Zend_Form were not yet supported and never quite understood why ZFI 
was not leveraged wholesale by Zend_Form (perhaps with BC adaption to ZFI)?

With a postfilter addition being considered now, it looks to me like 
convergence, and so I can't help wondering again if ZFI should be serving 
this role for Zend_Form?

Sorry if I don't understand it well enough but it just seems to me like fairly 
significant duplication. Which is better? Added BC support for Zend_Form's 
needs within ZFI,  or, what looks to me like a trend towards duplication with 
ZFI (perhaps for sounds reasons)?

Regards,
Mark

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Matthew Weier O'Phinney]

-- Laurent Melmoux <[EMAIL PROTECTED]> wrote
(on Saturday, 08 March 2008, 12:16 PM +0100):
> I’ve read the discussion and I really agree with Matthew first answer 
> (more below ) :

Some more information here:

ZFI and Zend_Form do pre-filtering in order to ensure that you can
perform validations. For example, spurious spaces at the beginning or
end of input should not be counted against the string length; when
validating a phone number, there's no need to validate anything other
than the digits (this prevents the need for complex regexen to check for
all potential phone number formats); content that should not contain
HTML may still be valid... so long as you strip the tags first.

Escaping is another operation entirely. ZFI does escaping, but ZFI's
primary use case is with models -- pulling in data and then escaping it
for your model. Zend_Form does escaping already... when rendering.

I can see the value of a postfilter for when you associate a Zend_Form
with a model. Right now, this is possible in a couple of ways:

* Load a new set of filters into your elements after validation, and
  then pull your values.
* Create a filter chain in your model (using Zend_Filter), and pass
  the data through that. This is probably the better place to do so,
  as you then know that the filtering is specific to your model.

I'll consider a postfilter post-1.5.0 GA release when I start looking at
associating forms with models.


> Matthew Weier O'Phinney-3 wrote:
>> Validation merely makes sure that the submitted data passes certain
>> rules. Filtering is for output -- to a database, to the screen,
>> whatever. You validate unfiltered data, and then filter it for your
>> output stream.
>>
>> Another use for filtering is normalization -- taking input and modifying
>> it to fit a 'normal' specification. An example of this is taking a phone
>> number -- (800) 555-1212 -- and normalizing it for your database --
>> 8885551212. You will often take a normalized value and filter it again
>> for display -- 800.555.1212.
>>
>> Let's look at this:
>>
>> If I have a filter, StripTags, and then a validator, NoTags, and run the
>> data through the filter prior to validating what's the point of the
>> validation then? you've already ensured that it will pass no matter
>> what.. but now when you return the data to the user, it looks unfamiliar
>> to them.
>>
>> Better is to check for tags in the first place, and let the user know
>> they aren't allowed... or, allow them, but strip them out prior to
>> storing or displaying the information (to prevent XSS attacks, for
>> example).
>
> Then Matthew changed it to pre-filter base on what I've been done in ZFI
>
>> For those who are interested:
>>
>> The idea is to allow some normalization of data prior to sending it to
>> validation. The example given me was one of a username:
>>
>> * Allow only 3-16 characters
>>
>> and you get ' mweierophinneyzend ' as input... it would fail
>> validation, but likely shouldn't have. If the input were run through
>> StringTrim first, it would pass.
>>
>> Basically, such normalization ensures that the data is *ready* for
>> validation. 
>
> It looks to me that only StringTrim fit the use case of pre-filtering data. 
> And I’m style convinced that post filtering is a better approaches. :)
>
>
> --
> Laurent Melmoux
> Annecy, France
>
> Amr Mostafa a écrit :
>> Hi Laurent,
>>
>> While I don't have the answer to your question specifically, I know that 
>> Zend_Form used to validate before filtering like you are suggesting. But 
>> that was modified per this discussion:
>>
>> http://www.nabble.com/Zend_Form_Element-validation-bug-tt15589810s16154.html
>>
>> As a side note, I've done something very similar to the use case you 
>> describe, and didn't run into this problem. The only difference I had was 
>> that the filter converts the given date to a Zend_Date instance. Which 
>> then runs through Zend_Validate_Date and it's has been working fine. Think 
>> I got lucky there and I've even noticed it :)
>>
>> Back to the discussion linked above, in one post I can tell that Matthew 
>> acknowledged the important of a pre/postFilter.
>>
>> Best Regards,
>> - Amr
>>
>> On Sat, Mar 8, 2008 at 11:26 AM, Laurent Melmoux <[EMAIL PROTECTED] 
>> > wrote:
>>
>> Hi Matthew and every body,
>>
>> First a use case:
>>
>> On an i18n application the user can chose is preferred date format for
>> display and input.
>> Let's say 'dd/MM/'.
>> I n the backend I will need a validator to validate the date
>> format and
>> a filter to convert the date to a Mysql format '-MM-dd'.
>>
>> But here the validation against 'dd/MM/' won't work because
>> the date
>> will be filtered before validation. Well, an easy work around will
>> be to
>> validate against the '-MM-dd' format but it sound not very
>> clean and
>> I could not use an error message saying the right f

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Laurent Melmoux]

I’ve read the discussion and I really agree with Matthew first answer 
(more below ) :



Matthew Weier O'Phinney-3 wrote:

Validation merely makes sure that the submitted data passes certain
rules. Filtering is for output -- to a database, to the screen,
whatever. You validate unfiltered data, and then filter it for your
output stream.

Another use for filtering is normalization -- taking input and modifying
it to fit a 'normal' specification. An example of this is taking a phone
number -- (800) 555-1212 -- and normalizing it for your database --
8885551212. You will often take a normalized value and filter it again
for display -- 800.555.1212.

Let's look at this:

If I have a filter, StripTags, and then a validator, NoTags, and run the
data through the filter prior to validating what's the point of the
validation then? you've already ensured that it will pass no matter
what.. but now when you return the data to the user, it looks unfamiliar
to them.

Better is to check for tags in the first place, and let the user know
they aren't allowed... or, allow them, but strip them out prior to
storing or displaying the information (to prevent XSS attacks, for
example).


Then Matthew changed it to pre-filter base on what I've been done in ZFI


For those who are interested:

The idea is to allow some normalization of data prior to sending it to
validation. The example given me was one of a username:

* Allow only 3-16 characters

and you get ' mweierophinneyzend ' as input... it would fail
validation, but likely shouldn't have. If the input were run through
StringTrim first, it would pass.

Basically, such normalization ensures that the data is *ready* for
validation. 


It looks to me that only StringTrim fit the use case of pre-filtering 
data. And I’m style convinced that post filtering is a better approaches. :)



--
Laurent Melmoux
Annecy, France

Amr Mostafa a écrit :

Hi Laurent,

While I don't have the answer to your question specifically, I know 
that Zend_Form used to validate before filtering like you are 
suggesting. But that was modified per this discussion:


http://www.nabble.com/Zend_Form_Element-validation-bug-tt15589810s16154.html

As a side note, I've done something very similar to the use case you 
describe, and didn't run into this problem. The only difference I had 
was that the filter converts the given date to a Zend_Date instance. 
Which then runs through Zend_Validate_Date and it's has been working 
fine. Think I got lucky there and I've even noticed it :)


Back to the discussion linked above, in one post I can tell that 
Matthew acknowledged the important of a pre/postFilter.


Best Regards,
- Amr

On Sat, Mar 8, 2008 at 11:26 AM, Laurent Melmoux <[EMAIL PROTECTED] 
> wrote:


Hi Matthew and every body,

First a use case:

On an i18n application the user can chose is preferred date format for
display and input.
Let's say 'dd/MM/'.
I n the backend I will need a validator to validate the date
format and
a filter to convert the date to a Mysql format '-MM-dd'.

But here the validation against 'dd/MM/' won't work because
the date
will be filtered before validation. Well, an easy work around will
be to
validate against the '-MM-dd' format but it sound not very
clean and
I could not use an error message saying the right format to use.

So I'm wondering if data should be filtered prior to validation. Is
there some security concern?
While I see some advantage to filter data, like removing white space,
before it is validated. It looks better to me to run it after the
validation and let know the user that is input is wrong. Then filter
data for formatting and security.

Best Regards,

--
Laurent Melmoux
Annecy, France

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Laurent Melmoux]

Thanks Amr,

I should have search the list :)

--
Laurent Melmoux
Annecy, France


Amr Mostafa a écrit :

Hi Laurent,

While I don't have the answer to your question specifically, I know 
that Zend_Form used to  validate before filtering like you are 
suggesting. But that was modified per this discussion:


http://www.nabble.com/Zend_Form_Element-validation-bug-tt15589810s16154.html

As a side note, I've done something very similar to the use case you 
describe, and didn't run into this problem. The only difference I had 
was that the filter converts the given date to a Zend_Date instance. 
Which then runs through Zend_Validate_Date and it's has been working 
fine. Think I got lucky there and I've even noticed it :)


Back to the discussion linked above, in one post I can tell that 
Matthew acknowledged the important of a pre/postFilter.


Best Regards,
- Amr

On Sat, Mar 8, 2008 at 11:26 AM, Laurent Melmoux <[EMAIL PROTECTED] 
> wrote:


Hi Matthew and every body,

First a use case:

On an i18n application the user can chose is preferred date format for
display and input.
Let's say 'dd/MM/'.
I n the backend I will need a validator to validate the date
format and
a filter to convert the date to a Mysql format '-MM-dd'.

But here the validation against 'dd/MM/' won't work because
the date
will be filtered before validation. Well, an easy work around will
be to
validate against the '-MM-dd' format but it sound not very
clean and
I could not use an error message saying the right format to use.

So I'm wondering if data should be filtered prior to validation. Is
there some security concern?
While I see some advantage to filter data, like removing white space,
before it is validated. It looks better to me to run it after the
validation and let know the user that is input is wrong. Then filter
data for formatting and security.

Best Regards,

--
Laurent Melmoux
Annecy, France

Re: [fw-general] Zend_Form - should data validation occur on filtered value?

[Amr Mostafa]

Hi Laurent,

While I don't have the answer to your question specifically, I know that
Zend_Form used to  validate before filtering like you are suggesting. But
that was modified per this discussion:

http://www.nabble.com/Zend_Form_Element-validation-bug-tt15589810s16154.html

As a side note, I've done something very similar to the use case you
describe, and didn't run into this problem. The only difference I had was
that the filter converts the given date to a Zend_Date instance. Which then
runs through Zend_Validate_Date and it's has been working fine. Think I got
lucky there and I've even noticed it :)

Back to the discussion linked above, in one post I can tell that Matthew
acknowledged the important of a pre/postFilter.

Best Regards,
- Amr

On Sat, Mar 8, 2008 at 11:26 AM, Laurent Melmoux <[EMAIL PROTECTED]> wrote:

> Hi Matthew and every body,
>
> First a use case:
>
> On an i18n application the user can chose is preferred date format for
> display and input.
> Let's say 'dd/MM/'.
> I n the backend I will need a validator to validate the date format and
> a filter to convert the date to a Mysql format '-MM-dd'.
>
> But here the validation against 'dd/MM/' won't work because the date
> will be filtered before validation. Well, an easy work around will be to
> validate against the '-MM-dd' format but it sound not very clean and
> I could not use an error message saying the right format to use.
>
> So I'm wondering if data should be filtered prior to validation. Is
> there some security concern?
> While I see some advantage to filter data, like removing white space,
> before it is validated. It looks better to me to run it after the
> validation and let know the user that is input is wrong. Then filter
> data for formatting and security.
>
> Best Regards,
>
> --
> Laurent Melmoux
> Annecy, France
>
>
>

[fw-general] Zend_Form - should data validation occur on filtered value?

[Laurent Melmoux]

Hi Matthew and every body,

First a use case:

On an i18n application the user can chose is preferred date format for 
display and input.

Let’s say 'dd/MM/'.
I n the backend I will need a validator to validate the date format and 
a filter to convert the date to a Mysql format '-MM-dd'.


But here the validation against 'dd/MM/' won’t work because the date 
will be filtered before validation. Well, an easy work around will be to 
validate against the '-MM-dd' format but it sound not very clean and 
I could not use an error message saying the right format to use.


So I’m wondering if data should be filtered prior to validation. Is 
there some security concern?
While I see some advantage to filter data, like removing white space, 
before it is validated. It looks better to me to run it after the 
validation and let know the user that is input is wrong. Then filter 
data for formatting and security.


Best Regards,

--
Laurent Melmoux
Annecy, France