<#part sign=pgpmime>
On Tue, 20 Mar 2012 16:54:52 -0700, "Pierre-Loup A. Griffais"
<pgriff...@nvidia.com> wrote:
> Wasted a good chunk of time on that one.
> From f1a65814f119cb83d16e15093e81946f3adcde61 Mon Sep 17 00:00:00 2001
> From: Pierre-Loup A. Griffais <pgriff...@nvidia.com>
> Date: Tue, 20 Mar 2012 16:46:22 -0700
> Subject: [PATCH] xrandr: move transform limit checking after scaling
>
> This would trigger for legit scaled matrices, resulting in the wrong
> extents getting computed.
>
> Signed-off-by: Pierre-Loup A. Griffais <pgriff...@nvidia.com>
> ---
> xrandr.c | 7 ++++---
> 1 files changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/xrandr.c b/xrandr.c
> index 35dff3e..622f6a9 100644
> --- a/xrandr.c
> +++ b/xrandr.c
> @@ -432,14 +432,15 @@ transform_point (XTransform *transform, double *xp,
> double *yp)
> v = 0;
> for (i = 0; i < 3; i++)
> v += (XFixedToDouble (transform->matrix[j][i]) * vector[i]);
> - if (v > 32767 || v < -32767)
> - return False;
> result[j] = v;
> }
> if (!result[2])
> return False;
> - for (j = 0; j < 2; j++)
> + for (j = 0; j < 2; j++) {
> vector[j] = result[j] / result[2];
> + if (vector[j] > 32767 || vector[j] < -32767)
> + return False;
> + }
Yeah, that looks right - transform from homogeneous back to pixel
coordinates before doing the range check.
Reviewed-by: Keith Packard <kei...@keithp.com>
(sorry I didn't see this on Tuesday, thanks for the ping!)
--
keith.pack...@intel.com
_______________________________________________
xorg-devel@lists.x.org: X.Org development
Archives: http://lists.x.org/archives/xorg-devel
Info: http://lists.x.org/mailman/listinfo/xorg-devel
