Hi,
This patch fixes sample clipping in the ConvolveOp and RescaleOp; it
should now work for packed sample models as well.
Regards,
Francis
2006-09-13 Francis Kung <[EMAIL PROTECTED]>
* java/awt/image/ConvolveOp.java (filter(Raster, WritableRaster)):
Removed hard-coded max sample value.
* java/awt/image/RescaleOp.java (filter(Raster, WritableRaster)):
Fixed finding of max sample value.
Index: java/awt/image/ConvolveOp.java
===================================================================
RCS file: /cvsroot/classpath/classpath/java/awt/image/ConvolveOp.java,v
retrieving revision 1.10
diff -u -r1.10 ConvolveOp.java
--- java/awt/image/ConvolveOp.java 1 Sep 2006 20:25:51 -0000 1.10
+++ java/awt/image/ConvolveOp.java 13 Sep 2006 15:32:04 -0000
@@ -249,6 +249,11 @@
int top = kernel.getYOrigin();
int bottom = Math.max(kHeight - top - 1, 0);
+ // Calculate max sample values for clipping
+ int[] maxValue = src.getSampleModel().getSampleSize();
+ for (int i = 0; i < maxValue.length; i++)
+ maxValue[i] = (int)Math.pow(2, maxValue[i]) - 1;
+
// process the region that is reachable...
int regionW = src.width - left - right;
int regionH = src.height - top - bottom;
@@ -270,12 +275,10 @@
// the samples array to make the tests pass. I haven't worked
// out why this is necessary.
- // This clipping is pretty strange, and seems to be hard-coded
- // at 255 (regardless of the raster's datatype or transfertype).
- // But it's what the reference does.
- if (v > 255)
- v = 255;
- if (v < 0)
+ // This clipping is is undocumented, but determined by testing.
+ if (v > maxValue[b])
+ v = maxValue[b];
+ else if (v < 0)
v = 0;
dest.setSample(x + kernel.getXOrigin(), y + kernel.getYOrigin(),
Index: java/awt/image/RescaleOp.java
===================================================================
RCS file: /cvsroot/classpath/classpath/java/awt/image/RescaleOp.java,v
retrieving revision 1.7
diff -u -r1.7 RescaleOp.java
--- java/awt/image/RescaleOp.java 6 Sep 2006 16:12:50 -0000 1.7
+++ java/awt/image/RescaleOp.java 13 Sep 2006 15:32:05 -0000
@@ -287,10 +287,9 @@
float scaleFactor, offset;
// Find max sample value, to be used for clipping later
- int maxValue = (int)Math.pow(2,
- DataBuffer.getDataTypeSize(src.getDataBuffer()
- .getDataType()))
- - 1;
+ int[] maxValue = src.getSampleModel().getSampleSize();
+ for (int i = 0; i < maxValue.length; i++)
+ maxValue[i] = (int)Math.pow(2, maxValue[i]) - 1;
// TODO: can this be optimized further?
// Filter all samples of all requested bands
@@ -318,8 +317,8 @@
// Clip if needed
if (values[i] < 0)
values[i] = 0;
- if (values[i] > maxValue)
- values[i] = maxValue;
+ if (values[i] > maxValue[band])
+ values[i] = maxValue[band];
}
dest.setSamples(dest.getMinX(), dest.getMinY(), dest.getWidth(),
