Re: [PATCH v4 3/8] media: adv7180: add support for NEWAVMODE

[Steve Longerbeam]

On 10/16/2016 05:18 AM, Laurent Pinchart wrote:

Hi Steve,

Thank you for the patch.

On Wednesday 03 Aug 2016 11:03:45 Steve Longerbeam wrote:

Parse the optional v4l2 endpoint DT node. If the bus type is
V4L2_MBUS_BT656 and the endpoint node specifies "newavmode",
configure the BT.656 bus in NEWAVMODE.

Signed-off-by: Steve Longerbeam 

---

v4: no changes
v3:
- the newavmode endpoint property is now private to adv7180.
---
  .../devicetree/bindings/media/i2c/adv7180.txt  |  4 ++
  drivers/media/i2c/adv7180.c| 46 +--
  2 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/Documentation/devicetree/bindings/media/i2c/adv7180.txt
b/Documentation/devicetree/bindings/media/i2c/adv7180.txt index
0d50115..6c175d2 100644
--- a/Documentation/devicetree/bindings/media/i2c/adv7180.txt
+++ b/Documentation/devicetree/bindings/media/i2c/adv7180.txt
@@ -15,6 +15,10 @@ Required Properties :
"adi,adv7282"
"adi,adv7282-m"

+Optional Endpoint Properties :
+- newavmode: a boolean property to indicate the BT.656 bus is operating
+  in Analog Device's NEWAVMODE. Valid for BT.656 busses only.

This is a vendor-specific property, it should be prefixed with "adi,".


Ok, I'll do that in next version.


  Could
you also explain how this mode works ? I'd like to make sure it qualifies for
a DT property.


The blurb in the ADV718x manual is terse:

"When NEWAVMODE is 0 (enabled), EAV/SAV codes are generated to
suit Analog Devices encoders. No adjustments are possible."

"Setting NEWAVMODE to 1 (default) enables the manual position
of the VSYNC, FIELD, and AV codes using Register 0x32 to
Register 0x33 and Register 0xE5 to Register 0xEA. Default register
settings are CCIR656 compliant;"

So it's not clear to me how the generated EAV and SAV codes are
different from standard CCIR656, but apparently they are.

Steve





+
  Example:

i2c0@1c22000 {
diff --git a/drivers/media/i2c/adv7180.c b/drivers/media/i2c/adv7180.c
index 6e093c22..467953e 100644
--- a/drivers/media/i2c/adv7180.c
+++ b/drivers/media/i2c/adv7180.c
@@ -31,6 +31,7 @@
  #include 
  #include 
  #include 
+#include 
  #include 
  #include 

@@ -106,6 +107,7 @@
  #define ADV7180_REG_SHAP_FILTER_CTL_1 0x0017
  #define ADV7180_REG_CTRL_20x001d
  #define ADV7180_REG_VSYNC_FIELD_CTL_1 0x0031
+#define ADV7180_VSYNC_FIELD_CTL_1_NEWAVMODE 0x02
  #define ADV7180_REG_MANUAL_WIN_CTL_1  0x003d
  #define ADV7180_REG_MANUAL_WIN_CTL_2  0x003e
  #define ADV7180_REG_MANUAL_WIN_CTL_3  0x003f
@@ -214,6 +216,7 @@ struct adv7180_state {
struct mutexmutex; /* mutual excl. when accessing chip */
int irq;
v4l2_std_id curr_norm;
+   boolnewavmode;
boolpowered;
boolstreaming;
u8  input;
@@ -864,9 +867,15 @@ static int adv7180_init(struct adv7180_state *state)
if (ret < 0)
return ret;

-   /* Manually set V bit end position in NTSC mode */
-   return adv7180_write(state, ADV7180_REG_NTSC_V_BIT_END,
-   ADV7180_NTSC_V_BIT_END_MANUAL_NVEND);
+   if (!state->newavmode) {
+   /* Manually set V bit end position in NTSC mode */
+   ret = adv7180_write(state, ADV7180_REG_NTSC_V_BIT_END,
+   ADV7180_NTSC_V_BIT_END_MANUAL_NVEND);
+   if (ret < 0)
+   return ret;
+   }
+
+   return 0;
  }

  static int adv7180_set_std(struct adv7180_state *state, unsigned int std)
@@ -1217,6 +1226,13 @@ static int init_device(struct adv7180_state *state)
if (ret)
goto out_unlock;

+   if (state->newavmode) {
+   ret = adv7180_write(state, ADV7180_REG_VSYNC_FIELD_CTL_1,
+   ADV7180_VSYNC_FIELD_CTL_1_NEWAVMODE);
+   if (ret < 0)
+   goto out_unlock;
+   }
+
ret = adv7180_program_std(state);
if (ret)
goto out_unlock;
@@ -1257,6 +1273,28 @@ out_unlock:
return ret;
  }

+static void adv7180_of_parse(struct adv7180_state *state)
+{
+   struct i2c_client *client = state->client;
+   struct device_node *np = client->dev.of_node;
+   struct device_node *endpoint;
+   struct v4l2_of_endpoint ep;
+
+   endpoint = of_graph_get_next_endpoint(np, NULL);
+   if (!endpoint) {
+   v4l_warn(client, "endpoint node not found\n");
+   return;
+   }
+
+   v4l2_of_parse_endpoint(endpoint, &ep);
+   if (ep.bus_type == V4L2_MBUS_BT656) {
+   if (of_property_read_bool(endpoint, "newavmode"))
+   state->newavmode = true;
+   }
+

Re: [PATCH] media: imx: allow to build with COMPILE_TEST

[Steve Longerbeam]

On 12/19/2017 03:42 AM, Philipp Zabel wrote:

Allow building this driver for other platforms under COMPILE_TEST.

Suggested-by: Mauro Carvalho Chehab 
Signed-off-by: Philipp Zabel 


Acked-by: Steve Longerbeam 


---
  drivers/staging/media/imx/Kconfig | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/staging/media/imx/Kconfig 
b/drivers/staging/media/imx/Kconfig
index 2be921cd0d55a..59b380cc6d223 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -1,6 +1,7 @@
  config VIDEO_IMX_MEDIA
tristate "i.MX5/6 V4L2 media core driver"
-   depends on MEDIA_CONTROLLER && VIDEO_V4L2 && ARCH_MXC && IMX_IPUV3_CORE
+   depends on ARCH_MXC || COMPILE_TEST
+   depends on MEDIA_CONTROLLER && VIDEO_V4L2 && IMX_IPUV3_CORE
depends on VIDEO_V4L2_SUBDEV_API
select VIDEOBUF2_DMA_CONTIG
select V4L2_FWNODE

Re: IMX6 interlaced capture

[Steve Longerbeam]

Hi Tim,

Happy New Year! And pardon the delay.

On 12/28/2017 01:56 PM, Tim Harvey wrote:

Steve/Hans,

I'm trying to get interlaced capture working with the TDA1997x driver
I've been working on which is connected to an IMX6 CSI.

The particular board I'm currently testing on is an IMX6Q which has
both a TDA19971 HDMI receiver as well as an ADV7180 analog video
decoder. The media-ctl topology for this board can be found at
http://dev.gateworks.com/docs/linux/media/imx6q-gw54xx-media.png.

For adv7180 everything appears to be working as expected:
- media-ctl --get-v4l2 '"adv7180 2-0020":0' shows:
[fmt:UYVY8_2X8/720x480 field:interlaced colorspace:smpte170m]
- he following captures/streams from the adv7180 using the vdic to de-interlace:
media-ctl --link "adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]
media-ctl --link "ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]
media-ctl --link "ipu2_csi1":1 -> "ipu2_vdic":0[1]
media-ctl --link "ipu2_vdic":2 -> "ipu2_ic_prp":0[1]
media-ctl --link "ipu2_ic_prp":2 -> "ipu2_ic_prpvf":0[1]
media-ctl --link "ipu2_ic_prpvf":1 -> "ipu2_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'adv7180 2-0020':0 [fmt:UYVY2X8/720x480]
media-ctl --set-v4l2 'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x480 field:interlaced]
media-ctl --set-v4l2 'ipu2_csi1':1 [fmt:UYVY2X8/720x480 field:interlaced]
media-ctl --set-v4l2 'ipu2_vdic':2 [fmt:UYVY2X8/720x480 field:interlaced]
media-ctl --set-v4l2 'ipu2_ic_prp':2 [fmt:UYVY2X8/720x480 field:none]
media-ctl --set-v4l2 'ipu2_ic_prpvf':1 [fmt:UYVY2X8/720x480 field:none]
v4l2-ctl -d /dev/video3 --set-fmt-video=width=720,height=480,pixelformat=UYVY
# capture 1 frame
v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
# stream jpeg/rtp
gst-launch-1.0 v4l2src device=/dev/video3 !
   "video/x-raw,width=720,height=480,format=UYVY" !
jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=5000"

For the tda1997x I'm trying to do something similar:
- media-ctl --get-v4l2 '"tda19971 2-0048":0' shows:
[fmt:UYVY8_1X16/1920x1080 field:alternate colorspace:srgb]
 still not sure V4L2_FIELD_ALTERNATE/SRGB returned from tda1997x
get_fmt is correct
- I setup the pipeline with:
media-ctl --link "tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]
media-ctl --link "ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]
media-ctl --link "ipu1_csi0":1 -> "ipu1_vdic":0[1]
media-ctl --link "ipu1_vdic":2 -> "ipu1_ic_prp":0[1]
media-ctl --link "ipu1_ic_prp":2 -> "ipu1_ic_prpvf":0[1]
media-ctl --link "ipu1_ic_prpvf":1 -> "ipu1_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080 field:alternate]
media-ctl --set-v4l2 'ipu1_csi0':1[fmt:UYVY8_1X16/1920x1080 field:alternate]
media-ctl --set-v4l2 'ipu1_vdic':2[fmt:UYVY8_1X16/1920x1080 field:alternate]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:UYVY8_1X16/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080 field:none]
v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080,pixelformat=UYVY
v4l2-ctl -d /dev/v4l-subdev1 --set-dv-bt-timings=query
v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
ipu1_csi0: bayer/16-bit parallel buses must go to IDMAC pad
ipu1_ic_prpvf: pipeline start failed with -22
VIDIOC_STREAMON: failed: Invalid argument


Right, according to the i.MX6 reference manual, if the CSI muxes
are receiving from the parallel bus input with width >= 16 bits,
that data can't be passed to the IC. It never really made much sense
to me, and I can't remember if I ever tried it, maybe not, because
I don't have such hardware.

Try this hack as an experiment: modify is_parallel_16bit_bus() in
imx-media-csi.c to simply return false, and see if the above pipeline
works.


- if I try to use the idmac for deinterlace I configure the pipeline with:
media-ctl --link "tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]
media-ctl --link "ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]
media-ctl --link "ipu1_csi0":1 -> "ipu1_ic_prp":0[1]
media-ctl --link "ipu1_ic_prp":2 -> "ipu1_ic_prpvf":0[1]
media-ctl --link "ipu1_ic_prpvf":1 -> "ipu1_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080 field:alternate]
media-ctl --set-v4l2 'ipu1_csi0':1[fmt:UYVY8_1X16/1920x1080 field:alternate]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:UYVY8_1X16/1920x1080 field:alternate]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080 field:none]
v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080,pixelformat=UYVY
v4l2-ctl -d /dev/v4l-subdev1 --set-dv-bt-timings=query
v4l2-ctl -d /dev/video1 --stream-mmap --stream-to=/tmp/x.raw --stream-count=1
ipu1_csi0: bayer/16-bit parallel buses must go to IDMAC pad
ipu1_ic_prpvf: pipeline start failed with -22
VIDIOC_STREAMON: failed: Invalid argument


For idmac de-interlace (interweaving w/o motion compensation), you
don't need to use the Image Converter pat

Re: IMX6 interlaced capture

[Steve Longerbeam]

On 01/04/2018 09:57 AM, Tim Harvey wrote:

On Tue, Jan 2, 2018 at 6:00 PM, Steve Longerbeam  wrote:

Hi Tim,

Happy New Year! And pardon the delay.


On 12/28/2017 01:56 PM, Tim Harvey wrote:

Steve/Hans,

I'm trying to get interlaced capture working with the TDA1997x driver
I've been working on which is connected to an IMX6 CSI.

The particular board I'm currently testing on is an IMX6Q which has
both a TDA19971 HDMI receiver as well as an ADV7180 analog video
decoder. The media-ctl topology for this board can be found at
http://dev.gateworks.com/docs/linux/media/imx6q-gw54xx-media.png.

For adv7180 everything appears to be working as expected:
- media-ctl --get-v4l2 '"adv7180 2-0020":0' shows:
[fmt:UYVY8_2X8/720x480 field:interlaced colorspace:smpte170m]
- he following captures/streams from the adv7180 using the vdic to
de-interlace:
media-ctl --link "adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]
media-ctl --link "ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]
media-ctl --link "ipu2_csi1":1 -> "ipu2_vdic":0[1]
media-ctl --link "ipu2_vdic":2 -> "ipu2_ic_prp":0[1]
media-ctl --link "ipu2_ic_prp":2 -> "ipu2_ic_prpvf":0[1]
media-ctl --link "ipu2_ic_prpvf":1 -> "ipu2_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'adv7180 2-0020':0 [fmt:UYVY2X8/720x480]
media-ctl --set-v4l2 'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x480
field:interlaced]
media-ctl --set-v4l2 'ipu2_csi1':1 [fmt:UYVY2X8/720x480 field:interlaced]
media-ctl --set-v4l2 'ipu2_vdic':2 [fmt:UYVY2X8/720x480 field:interlaced]
media-ctl --set-v4l2 'ipu2_ic_prp':2 [fmt:UYVY2X8/720x480 field:none]
media-ctl --set-v4l2 'ipu2_ic_prpvf':1 [fmt:UYVY2X8/720x480 field:none]
v4l2-ctl -d /dev/video3
--set-fmt-video=width=720,height=480,pixelformat=UYVY
# capture 1 frame
v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
# stream jpeg/rtp
gst-launch-1.0 v4l2src device=/dev/video3 !
"video/x-raw,width=720,height=480,format=UYVY" !
 jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=5000"

For the tda1997x I'm trying to do something similar:
- media-ctl --get-v4l2 '"tda19971 2-0048":0' shows:
[fmt:UYVY8_1X16/1920x1080 field:alternate colorspace:srgb]
 still not sure V4L2_FIELD_ALTERNATE/SRGB returned from tda1997x
get_fmt is correct
- I setup the pipeline with:
media-ctl --link "tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]
media-ctl --link "ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]
media-ctl --link "ipu1_csi0":1 -> "ipu1_vdic":0[1]
media-ctl --link "ipu1_vdic":2 -> "ipu1_ic_prp":0[1]
media-ctl --link "ipu1_ic_prp":2 -> "ipu1_ic_prpvf":0[1]
media-ctl --link "ipu1_ic_prpvf":1 -> "ipu1_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080
field:alternate]
media-ctl --set-v4l2 'ipu1_csi0':1[fmt:UYVY8_1X16/1920x1080
field:alternate]
media-ctl --set-v4l2 'ipu1_vdic':2[fmt:UYVY8_1X16/1920x1080
field:alternate]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:UYVY8_1X16/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080
field:none]
v4l2-ctl -d /dev/video1
--set-fmt-video=width=1920,height=1080,pixelformat=UYVY
v4l2-ctl -d /dev/v4l-subdev1 --set-dv-bt-timings=query
v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
ipu1_csi0: bayer/16-bit parallel buses must go to IDMAC pad
ipu1_ic_prpvf: pipeline start failed with -22
VIDIOC_STREAMON: failed: Invalid argument

Steve,

Thanks for the help.


Right, according to the i.MX6 reference manual, if the CSI muxes
are receiving from the parallel bus input with width >= 16 bits,
that data can't be passed to the IC. It never really made much sense
to me, and I can't remember if I ever tried it, maybe not, because
I don't have such hardware.

hmmm... that's not good. I may have to dig into what's being done in
my 3.14 kernel with the Freescale capture driver where I can capture
1080p60 fine with my tda1997x driver there.


Try this hack as an experiment: modify is_parallel_16bit_bus() in
imx-media-csi.c to simply return false, and see if the above pipeline
works.

I'm currently on 4.15-rc1 which doesn't have a
'is_parallel_16bit_bus()' but if I comment out the check we are
talking about in csi_link_validate as such:

--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -999,6 +999,7 @@ static int csi_link_validate(struct v4l2_subdev *sd,
 is_csi2 = (sensor_ep->bus_type == V4L2

Re: IMX6 interlaced capture

[Steve Longerbeam]

On 01/04/2018 10:51 AM, Steve Longerbeam wrote:



On 01/04/2018 09:57 AM, Tim Harvey wrote:




Try this hack as an experiment: modify is_parallel_16bit_bus() in
imx-media-csi.c to simply return false, and see if the above pipeline
works.

I'm currently on 4.15-rc1 which doesn't have a
'is_parallel_16bit_bus()' but if I comment out the check we are
talking about in csi_link_validate as such:

--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -999,6 +999,7 @@ static int csi_link_validate(struct v4l2_subdev *sd,
 is_csi2 = (sensor_ep->bus_type == V4L2_MBUS_CSI2);
 incc = priv->cc[CSI_SINK_PAD];

+/*
 if (priv->dest != IPU_CSI_DEST_IDMAC &&
 (incc->bayer || (!is_csi2 &&
sensor_ep->bus.parallel.bus_width >= 16))) {
@@ -1007,6 +1008,7 @@ static int csi_link_validate(struct v4l2_subdev 
*sd,

 ret = -EINVAL;
 goto out;
 }
+*/

 if (is_csi2) {
 int vc_num = 0;

I get a pipeline start failure for ipu1_ic_prpvf:

root@ventana:~# v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
[  909.993353] tda1997x 2-0048: tda1997x_get_pad_format
[  909.998342] tda1997x 2-0048: tda1997x_fill_format
 my tda1997x driver debug messages
[  910.004483] ipu1_ic_prpvf: pipeline start failed with -32
VIDIOC_STREAMON: failed: Broken pipe


The driver doesn't really support V4L2_FIELD_ALTERNATE, the CSI subdev
attempts to translate this to sequential-top-bottom or 
sequential-bottom-top
at the CSI output pads, since alternate holds no info about field 
order. I doubt
alternate is correct for the TDA19971 anyway, so that should be fixed 
in the
tda19971 driver. Until then, set to "seq-bt" downstream from the CSI, 
as in:


media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080 
field:alternate]
media-ctl --set-v4l2 'ipu1_csi0':1[fmt:UYVY8_1X16/1920x1080 
field:alternate]

media-ctl --set-v4l2 'ipu1_vdic':2[fmt:UYVY8_1X16/1920x1080 field:seq-bt]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:UYVY8_1X16/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080 
field:none]


Actually just noticed another problem with the original pipeline, the 
ipu1_vdic output
pad is always progressive/non-interlaced. The output pad enforces 
progressive, but

to be technically correct this should be:

media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080 
field:alternate]
media-ctl --set-v4l2 'ipu1_csi0':1[fmt:UYVY8_1X16/1920x1080 
field:alternate]

media-ctl --set-v4l2 'ipu1_vdic':2[fmt:UYVY8_1X16/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:UYVY8_1X16/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080 field:none]


But as said in my last email, if the tda19971 supports one of the 
sequential field
types that explicitly specifies field order, it would be better to start 
with that right

up front:

media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080 
field:seq-tb]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080 
field:seq-tb]

media-ctl --set-v4l2 'ipu1_csi0':1[fmt:AYUV32/1920x1080 field:seq-tb]
media-ctl --set-v4l2 'ipu1_vdic':2[fmt:AYUV32/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:AYUV32/1920x1080 field:none]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080 field:none]

Note also the IPU-internal pixel formats should be AYUV32.

Steve

Re: IMX6 interlaced capture

[Steve Longerbeam]

On 01/04/2018 01:11 PM, Tim Harvey wrote:

On Thu, Jan 4, 2018 at 10:51 AM, Steve Longerbeam  wrote:


On 01/04/2018 09:57 AM, Tim Harvey wrote:

On Tue, Jan 2, 2018 at 6:00 PM, Steve Longerbeam 
wrote:

Hi Tim,

Happy New Year! And pardon the delay.


On 12/28/2017 01:56 PM, Tim Harvey wrote:

Steve/Hans,

I'm trying to get interlaced capture working with the TDA1997x driver
I've been working on which is connected to an IMX6 CSI.

The particular board I'm currently testing on is an IMX6Q which has
both a TDA19971 HDMI receiver as well as an ADV7180 analog video
decoder. The media-ctl topology for this board can be found at
http://dev.gateworks.com/docs/linux/media/imx6q-gw54xx-media.png.

For adv7180 everything appears to be working as expected:
- media-ctl --get-v4l2 '"adv7180 2-0020":0' shows:
[fmt:UYVY8_2X8/720x480 field:interlaced colorspace:smpte170m]
- he following captures/streams from the adv7180 using the vdic to
de-interlace:
media-ctl --link "adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]
media-ctl --link "ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]
media-ctl --link "ipu2_csi1":1 -> "ipu2_vdic":0[1]
media-ctl --link "ipu2_vdic":2 -> "ipu2_ic_prp":0[1]
media-ctl --link "ipu2_ic_prp":2 -> "ipu2_ic_prpvf":0[1]
media-ctl --link "ipu2_ic_prpvf":1 -> "ipu2_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'adv7180 2-0020':0 [fmt:UYVY2X8/720x480]
media-ctl --set-v4l2 'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x480
field:interlaced]
media-ctl --set-v4l2 'ipu2_csi1':1 [fmt:UYVY2X8/720x480
field:interlaced]
media-ctl --set-v4l2 'ipu2_vdic':2 [fmt:UYVY2X8/720x480
field:interlaced]
media-ctl --set-v4l2 'ipu2_ic_prp':2 [fmt:UYVY2X8/720x480 field:none]
media-ctl --set-v4l2 'ipu2_ic_prpvf':1 [fmt:UYVY2X8/720x480 field:none]
v4l2-ctl -d /dev/video3
--set-fmt-video=width=720,height=480,pixelformat=UYVY
# capture 1 frame
v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
# stream jpeg/rtp
gst-launch-1.0 v4l2src device=/dev/video3 !
 "video/x-raw,width=720,height=480,format=UYVY" !
  jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=5000"

For the tda1997x I'm trying to do something similar:
- media-ctl --get-v4l2 '"tda19971 2-0048":0' shows:
[fmt:UYVY8_1X16/1920x1080 field:alternate colorspace:srgb]
 still not sure V4L2_FIELD_ALTERNATE/SRGB returned from tda1997x
get_fmt is correct
- I setup the pipeline with:
media-ctl --link "tda19971 2-0048":0 -> "ipu1_csi0_mux":1[1]
media-ctl --link "ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]
media-ctl --link "ipu1_csi0":1 -> "ipu1_vdic":0[1]
media-ctl --link "ipu1_vdic":2 -> "ipu1_ic_prp":0[1]
media-ctl --link "ipu1_ic_prp":2 -> "ipu1_ic_prpvf":0[1]
media-ctl --link "ipu1_ic_prpvf":1 -> "ipu1_ic_prpvf capture":0[1]
media-ctl --set-v4l2 'tda19971 2-0048':0[fmt:UYVY8_1X16/1920x1080]
media-ctl --set-v4l2 'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080
field:alternate]
media-ctl --set-v4l2 'ipu1_csi0':1[fmt:UYVY8_1X16/1920x1080
field:alternate]
media-ctl --set-v4l2 'ipu1_vdic':2[fmt:UYVY8_1X16/1920x1080
field:alternate]
media-ctl --set-v4l2 'ipu1_ic_prp':2[fmt:UYVY8_1X16/1920x1080
field:none]
media-ctl --set-v4l2 'ipu1_ic_prpvf':1[fmt:UYVY8_1X16/1920x1080
field:none]
v4l2-ctl -d /dev/video1
--set-fmt-video=width=1920,height=1080,pixelformat=UYVY
v4l2-ctl -d /dev/v4l-subdev1 --set-dv-bt-timings=query
v4l2-ctl -d /dev/video1 --stream-mmap --stream-skip=1
--stream-to=/tmp/x.raw --stream-count=1
ipu1_csi0: bayer/16-bit parallel buses must go to IDMAC pad
ipu1_ic_prpvf: pipeline start failed with -22
VIDIOC_STREAMON: failed: Invalid argument

Steve,

Thanks for the help.


Right, according to the i.MX6 reference manual, if the CSI muxes
are receiving from the parallel bus input with width >= 16 bits,
that data can't be passed to the IC. It never really made much sense
to me, and I can't remember if I ever tried it, maybe not, because
I don't have such hardware.

hmmm... that's not good. I may have to dig into what's being done in
my 3.14 kernel with the Freescale capture driver where I can capture
1080p60 fine with my tda1997x driver there.


Try this hack as an experiment: modify is_parallel_16bit_bus() in
imx-media-csi.c to simply return false, and see if the above pipeline
works.

I'm currently on 4.15-rc1 which doesn't have a
'is_parallel_16bit_bus()' but if I comment out the check we are
talking about in csi_link_validate as such:

--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -

Re: [PATCH] media: staging/imx: Checking the right variable in vdic_get_ipu_resources()

[Steve Longerbeam]

Acked-by: Steve Longerbeam 


On 01/15/2018 12:11 AM, Dan Carpenter wrote:

We recently changed this error handling around but missed this error
pointer check.  We're testing "priv->vdi_in_ch_n" instead of "ch" so the
error handling can't be triggered.

Fixes: 0b2e9e7947e7 ("media: staging/imx: remove confusing IS_ERR_OR_NULL 
usage")
Signed-off-by: Dan Carpenter 

diff --git a/drivers/staging/media/imx/imx-media-vdic.c 
b/drivers/staging/media/imx/imx-media-vdic.c
index 433474d58e3e..ed356844cdf6 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -177,7 +177,7 @@ static int vdic_get_ipu_resources(struct vdic_priv *priv)
priv->vdi_in_ch = ch;
  
  		ch = ipu_idmac_get(priv->ipu, IPUV3_CHANNEL_MEM_VDI_NEXT);

-   if (IS_ERR(priv->vdi_in_ch_n)) {
+   if (IS_ERR(ch)) {
err_chan = IPUV3_CHANNEL_MEM_VDI_NEXT;
ret = PTR_ERR(ch);
goto out_err_chan;

Re: [PATCH] staging: imx-media-vdic: fix inconsistent IS_ERR and PTR_ERR

[Steve Longerbeam]

Acked-by: Steve Longerbeam 


On 01/23/2018 04:43 PM, Gustavo A. R. Silva wrote:

Fix inconsistent IS_ERR and PTR_ERR in vdic_get_ipu_resources.
The proper pointer to be passed as argument is ch.

This issue was detected with the help of Coccinelle.

Fixes: 0b2e9e7947e7 ("media: staging/imx: remove confusing IS_ERR_OR_NULL 
usage")
Signed-off-by: Gustavo A. R. Silva 
---
  drivers/staging/media/imx/imx-media-vdic.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/media/imx/imx-media-vdic.c 
b/drivers/staging/media/imx/imx-media-vdic.c
index 433474d..ed35684 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -177,7 +177,7 @@ static int vdic_get_ipu_resources(struct vdic_priv *priv)
priv->vdi_in_ch = ch;
  
  		ch = ipu_idmac_get(priv->ipu, IPUV3_CHANNEL_MEM_VDI_NEXT);

-   if (IS_ERR(priv->vdi_in_ch_n)) {
+   if (IS_ERR(ch)) {
err_chan = IPUV3_CHANNEL_MEM_VDI_NEXT;
ret = PTR_ERR(ch);
goto out_err_chan;

Re: [PATCH 16/16] media: imx: add mem2mem device

[Steve Longerbeam]

On 07/16/2018 07:12 AM, Philipp Zabel wrote:

Hi Steve,

On Thu, 2018-07-05 at 15:09 -0700, Steve Longerbeam wrote:
[...]
[...]

+   halign = 0;
+   break;
+   }
+   if (f->type == V4L2_BUF_TYPE_VIDEO_OUTPUT) {
+   /*
+* The IC burst reads 8 pixels at a time. Reading beyond the
+* end of the line is usually acceptable. Those pixels are
+* ignored, unless the IC has to write the scaled line in
+* reverse.
+*/
+   if (!ipu_rot_mode_is_irt(ctx->rot_mode) &&
+   ctx->rot_mode && IPU_ROT_BIT_HFLIP)
+   walign = 3;

This looks wrong. Do you mean:

if (ipu_rot_mode_is_irt(ctx->rot_mode) || (ctx->rot_mode & IPU_ROT_BIT_HFLIP))
  walign = 3;
else
  walign = 1;

The input DMA burst width alignment is only necessary if the lines are
scanned from right to left (that is, if HF is enabled) in the scaling
step.


Ok, thanks for the explanation, that makes sense.


If the rotator is used, the flipping is done in the rotation step
instead,


Ah, I missed or forgot about that detail in the ref manual,
I reviewed it again and you are right...


  so the alignment restriction would be on the width of the
intermediate tile (and thus on the output height). This is already
covered by the rotator 8x8 pixel block alignment.


so this makes sense too.




That is, require 8 byte width alignment for IRT or if HFLIP is enabled.

No, I specifically meant (!IRT && HFLIP).


Right, but there is still a typo:

if (!ipu_rot_mode_is_irt(ctx->rot_mode) && ctx->rot_mode && IPU_ROT_BIT_HFLIP)

should be:

if (!ipu_rot_mode_is_irt(ctx->rot_mode) && (ctx->rot_mode & IPU_ROT_BIT_HFLIP))




The rotator itself doesn't cause any input alignment restrictions, we
just have to make sure that the intermediate tiles after scaling are 8x8
aligned.


Also, why not simply call ipu_image_convert_adjust() in
mem2mem_try_fmt()? If there is something missing in the former
function, then it should be added there, instead of adding the
missing checks in mem2mem_try_fmt().

ipu_image_convert_adjust tries to adjust both input and output image at
the same time, here we just have the format of either input or output
image. Do you suggest to split this function into an input and an output
version?


See b4362162c0 ("media: imx: mem2mem: Use ipu_image_convert_adjust
in try format")

in my mediatree fork at g...@github.com:slongerbeam/mediatree.git.

Let's discuss this further in the v2 patches.

Steve

Re: [PATCH v2 00/16] i.MX media mem2mem scaler

[Steve Longerbeam]

Hi Philipp,


On 07/19/2018 08:30 AM, Philipp Zabel wrote:

Hi,

this is the second version of the i.MX mem2mem scaler series.
Patches 8 and 16 have been modified.

Changes since v1:
  - Fix inverted allow_overshoot logic
  - Correctly switch horizontal / vertical tile alignment when
determining seam positions with the 90° rotator active.


Yes, this fixes the specific rotation test that was broken
(720x480, UYVY --> 1280x768, UYVY, rotate 90).

But running more tests on this v2 reveals more issues. I chose a
somewhat random upscaling-only example as a first try:

640x480, YV12 --> full HD 2560x1600, YV12 (no rotation or flip).

This produces division by zero backtraces and the conversion hangs:


[  131.079978] Division by zero in kernel.
[  131.083853] CPU: 0 PID: 683 Comm: mx6-m2m Tainted: G W 
4.18.0-rc2-13448-g678218d #7

[  131.092830] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  131.099372] Backtrace:
[  131.101858] [] (dump_backtrace) from [] 
(show_stack+0x18/0x1c)

[  131.109450]  r7: r6:600f0013 r5: r4:c107db3c
[  131.115135] [] (show_stack) from [] 
(dump_stack+0xb4/0xe8)
[  131.122380] [] (dump_stack) from [] 
(__div0+0x18/0x20)
[  131.129274]  r9:ec37d800 r8:0003 r7: r6: 
r5: r4:ec37dae8

[  131.137036] [] (__div0) from [] (Ldiv0+0x8/0x10)
[  131.143425] [] (ipu_image_convert_prepare) from 
[] (mem2mem_start_streaming+0xe0/0x1c0)
[  131.153186]  r10:c0b9f640 r9:c071958c r8:0280 r7:01e0 
r6:32315659 r5:c1008908

[  131.161030]  r4:ecf9c800
[  131.163588] [] (mem2mem_start_streaming) from [] 
(vb2_start_streaming+0x64/0x160)

[  131.172826]  r8:c1008908 r7:0001 r6:ed01b808 r5:ed01b934 r4:ed01b810
[  131.179547] [] (vb2_start_streaming) from [] 
(vb2_core_streamon+0x10c/0x164)
[  131.188351]  r9:c071958c r8:c1008908 r7:0001 r6:ec1038f8 
r5: r4:ed01b808
[  131.196114] [] (vb2_core_streamon) from [] 
(vb2_streamon+0x34/0x58)

[  131.204133]  r5:40045612 r4:ed01b800
[  131.207733] [] (vb2_streamon) from [] 
(v4l2_m2m_streamon+0x24/0x3c)
[  131.215758] [] (v4l2_m2m_streamon) from [] 
(v4l2_m2m_ioctl_streamon+0x18/0x1c)

[  131.224732]  r5:40045612 r4:c072f354
[  131.228330] [] (v4l2_m2m_ioctl_streamon) from [] 
(v4l_streamon+0x24/0x28)
[  131.236878] [] (v4l_streamon) from [] 
(__video_do_ioctl+0x284/0x4f8)

[  131.244984]  r5:40045612 r4:ecc50800
[  131.248583] [] (__video_do_ioctl) from [] 
(video_usercopy+0x260/0x55c)
[  131.256866]  r10:0004 r9: r8:c1008908 r7:ed747dfc 
r6: r5:0004

[  131.264709]  r4:40045612
[  131.267265] [] (video_usercopy) from [] 
(video_ioctl2+0x14/0x1c)
[  131.275026]  r10:0036 r9:0003 r8:ed480068 r7:c0254840 
r6:ecc76000 r5:bea6ab38

[  131.282869]  r4:c071fcd8
[  131.285424] [] (video_ioctl2) from [] 
(v4l2_ioctl+0x44/0x5c)
[  131.292845] [] (v4l2_ioctl) from [] 
(do_vfs_ioctl+0xa8/0xa4c)

[  131.300343]  r5:bea6ab38 r4:c1008908
[  131.303940] [] (do_vfs_ioctl) from [] 
(ksys_ioctl+0x3c/0x60)
[  131.311355]  r10:0036 r9:ed746000 r8:bea6ab38 r7:40045612 
r6:0003 r5:ecc76000

[  131.319198]  r4:ecc76000
[  131.321752] [] (ksys_ioctl) from [] 
(sys_ioctl+0x10/0x14)
[  131.328907]  r9:ed746000 r8:c01011e4 r7:0036 r6:00010960 
r5: r4:00012620
[  131.336672] [] (sys_ioctl) from [] 
(ret_fast_syscall+0x0/0x28)

[  131.344256] Exception stack(0xed747fa8 to 0xed747ff0)
[  131.349327] 7fa0:   00012620  0003 
40045612 bea6ab38 0003
[  131.357524] 7fc0: 00012620  00010960 0036  
 45d8 bea6abac

[  131.365717] 7fe0: 0002312c bea6aaa4 00012308 45e58d5c


To aid in debugging this I created branch 'imx-mem2mem.stevel' in my
mediatree fork on github. I moved the mem2mem driver to the beginning
and added a few patches:

d317a7771c ("gpu: ipu-cpmem: add WARN_ON_ONCE() for unaligned dma buffers")
b4362162c0 ("media: imx: mem2mem: Use ipu_image_convert_adjust in try 
format")

4758be0cf8 ("gpu: ipu-v3: image-convert: Fix width/height alignment")
d069163c7f ("gpu: ipu-v3: image-convert: Fix input bytesperline clamp in 
adjust")


(feel free to squash some of those if you agree with them for v3).

By moving the mem2mem driver before the seam avoidance patches, and making
it independent of the image converter implementation, the driver can be 
tested with

and without the seam avoidance changes.

If you run a git rebase and build/run the kernel when stopped at 
b4362162c0 (e.g.
without the seam avoidance patches), you will find that the above 
640x480 -->

2560x1600 conversion succeeds, albeit with the expected visible seams at the
tile boundaries.

Also, I'm trying to parse the functions find_best_seam() and 
find_seams(). Can

you provide some more background on the behavior of those functions?

Steve


  - Fix SPDX-License-Identifier and remove superfluous license
text.
  - Fix uninitialized walign in try_fmt

Previous cover letter:

we have image conversion code for scaling 

Re: [PATCH v2 00/16] i.MX media mem2mem scaler

[Steve Longerbeam]

On 07/22/2018 11:30 AM, Steve Longerbeam wrote:

Hi Philipp,


On 07/19/2018 08:30 AM, Philipp Zabel wrote:

Hi,

this is the second version of the i.MX mem2mem scaler series.
Patches 8 and 16 have been modified.

Changes since v1:
  - Fix inverted allow_overshoot logic
  - Correctly switch horizontal / vertical tile alignment when
    determining seam positions with the 90° rotator active.


Yes, this fixes the specific rotation test that was broken
(720x480, UYVY --> 1280x768, UYVY, rotate 90).

But running more tests on this v2 reveals more issues. I chose a
somewhat random upscaling-only example as a first try:

640x480, YV12 --> full HD 2560x1600, YV12 (no rotation or flip).

This produces division by zero backtraces and the conversion hangs:



The hang is apparently because the conversion is re-attempted over and
over again, with an endless WARN() from
drivers/media/common/videobuf2/videobuf2-core.c:900.

I fixed the hang with an additional patch:

50026cbe08 ("media: imx: mem2mem: Remove buffers on device_run failures")

With this the conversion completes, but the below div-by-zero errors
persist, and the resultant image is blank.

Steve



[  131.079978] Division by zero in kernel.
[  131.083853] CPU: 0 PID: 683 Comm: mx6-m2m Tainted: G W 
4.18.0-rc2-13448-g678218d #7

[  131.092830] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  131.099372] Backtrace:
[  131.101858] [] (dump_backtrace) from [] 
(show_stack+0x18/0x1c)

[  131.109450]  r7: r6:600f0013 r5: r4:c107db3c
[  131.115135] [] (show_stack) from [] 
(dump_stack+0xb4/0xe8)
[  131.122380] [] (dump_stack) from [] 
(__div0+0x18/0x20)
[  131.129274]  r9:ec37d800 r8:0003 r7: r6: 
r5: r4:ec37dae8

[  131.137036] [] (__div0) from [] (Ldiv0+0x8/0x10)
[  131.143425] [] (ipu_image_convert_prepare) from 
[] (mem2mem_start_streaming+0xe0/0x1c0)
[  131.153186]  r10:c0b9f640 r9:c071958c r8:0280 r7:01e0 
r6:32315659 r5:c1008908

[  131.161030]  r4:ecf9c800
[  131.163588] [] (mem2mem_start_streaming) from 
[] (vb2_start_streaming+0x64/0x160)
[  131.172826]  r8:c1008908 r7:0001 r6:ed01b808 r5:ed01b934 
r4:ed01b810
[  131.179547] [] (vb2_start_streaming) from [] 
(vb2_core_streamon+0x10c/0x164)
[  131.188351]  r9:c071958c r8:c1008908 r7:0001 r6:ec1038f8 
r5: r4:ed01b808
[  131.196114] [] (vb2_core_streamon) from [] 
(vb2_streamon+0x34/0x58)

[  131.204133]  r5:40045612 r4:ed01b800
[  131.207733] [] (vb2_streamon) from [] 
(v4l2_m2m_streamon+0x24/0x3c)
[  131.215758] [] (v4l2_m2m_streamon) from [] 
(v4l2_m2m_ioctl_streamon+0x18/0x1c)

[  131.224732]  r5:40045612 r4:c072f354
[  131.228330] [] (v4l2_m2m_ioctl_streamon) from 
[] (v4l_streamon+0x24/0x28)
[  131.236878] [] (v4l_streamon) from [] 
(__video_do_ioctl+0x284/0x4f8)

[  131.244984]  r5:40045612 r4:ecc50800
[  131.248583] [] (__video_do_ioctl) from [] 
(video_usercopy+0x260/0x55c)
[  131.256866]  r10:0004 r9: r8:c1008908 r7:ed747dfc 
r6: r5:0004

[  131.264709]  r4:40045612
[  131.267265] [] (video_usercopy) from [] 
(video_ioctl2+0x14/0x1c)
[  131.275026]  r10:0036 r9:0003 r8:ed480068 r7:c0254840 
r6:ecc76000 r5:bea6ab38

[  131.282869]  r4:c071fcd8
[  131.285424] [] (video_ioctl2) from [] 
(v4l2_ioctl+0x44/0x5c)
[  131.292845] [] (v4l2_ioctl) from [] 
(do_vfs_ioctl+0xa8/0xa4c)

[  131.300343]  r5:bea6ab38 r4:c1008908
[  131.303940] [] (do_vfs_ioctl) from [] 
(ksys_ioctl+0x3c/0x60)
[  131.311355]  r10:0036 r9:ed746000 r8:bea6ab38 r7:40045612 
r6:0003 r5:ecc76000

[  131.319198]  r4:ecc76000
[  131.321752] [] (ksys_ioctl) from [] 
(sys_ioctl+0x10/0x14)
[  131.328907]  r9:ed746000 r8:c01011e4 r7:0036 r6:00010960 
r5: r4:00012620
[  131.336672] [] (sys_ioctl) from [] 
(ret_fast_syscall+0x0/0x28)

[  131.344256] Exception stack(0xed747fa8 to 0xed747ff0)
[  131.349327] 7fa0:   00012620  0003 
40045612 bea6ab38 0003
[  131.357524] 7fc0: 00012620  00010960 0036  
 45d8 bea6abac

[  131.365717] 7fe0: 0002312c bea6aaa4 00012308 45e58d5c


To aid in debugging this I created branch 'imx-mem2mem.stevel' in my
mediatree fork on github. I moved the mem2mem driver to the beginning
and added a few patches:

d317a7771c ("gpu: ipu-cpmem: add WARN_ON_ONCE() for unaligned dma 
buffers")
b4362162c0 ("media: imx: mem2mem: Use ipu_image_convert_adjust in try 
format")

4758be0cf8 ("gpu: ipu-v3: image-convert: Fix width/height alignment")
d069163c7f ("gpu: ipu-v3: image-convert: Fix input bytesperline clamp 
in adjust")


(feel free to squash some of those if you agree with them for v3).

By moving the mem2mem driver before the seam avoidance patches, and 
making
it independent of the image converter implementation, the driver can 
be tested with

and without the seam avoidance changes.

If you run a git rebase and build/run the kernel when stopped at 
b4362162c0

Re: [PATCH 16/16] media: imx: add mem2mem device

[Steve Longerbeam]

On 07/23/2018 12:31 AM, Philipp Zabel wrote:


ipu_image_convert_adjust tries to adjust both input and output image at
the same time, here we just have the format of either input or output
image. Do you suggest to split this function into an input and an output
version?

See b4362162c0 ("media: imx: mem2mem: Use ipu_image_convert_adjust
in try format")

Alright, this looks fine to me. I was worried about inter-format
limitations, but the only one seems to be the output size lower bound to
1/4 of the input size. Should S_FMT(OUT) also update the capture format
if adjustments were made to keep a consistent state?


That's a good question, I don't know if the mem2mem API allows for
that, but if it does we should do that for consistent state as you said.

In b4362162c0, the current capture format is used to adjust output
format during S_FMT(OUT) but any capture format changes are
dropped, and vice-versa.

Steve

[PATCH RESEND v6 15/17] media: platform: Switch to v4l2_async_notifier_add_subdev

[Steve Longerbeam]

Switch all media platform drivers to call v4l2_async_notifier_add_subdev()
to add asd's to a notifier, in place of referencing the notifier->subdevs[]
array. These drivers also must now call v4l2_async_notifier_init() before
adding asd's to their notifiers.

There may still be cases where a platform driver maintains a list of
asd's that is a duplicate of the notifier asd_list, in which case its
possible the platform driver list can be removed, and can reference the
notifier asd_list instead. One example of where a duplicate list has
been removed in this patch is xilinx-vipp.c. If there are such cases
remaining, those drivers should be optimized to remove the duplicate
platform driver asd lists.

None of the changes to the platform drivers in this patch have been
tested. Verify that the async subdevices needed by the platform are
bound at load time, and that the driver unloads and reloads correctly
with no memory leaking of asd objects.

Suggested-by: Sakari Ailus 
Signed-off-by: Steve Longerbeam 
---
Changes since v5:
- remove reference to notifier.num_subdevs, and call
  v4l2_async_notifier_init(). Suggested by Sakari Ailus.
- removed "OF" qualifier when passing fwnode pointers to printk in
  xilinx-vipp.c. Reported by Dan Carpenter.
- fixed double node put in vpif_capture.c. Reported by Sakari.
---
 drivers/media/pci/intel/ipu3/ipu3-cio2.c   |   2 +-
 drivers/media/platform/am437x/am437x-vpfe.c|  82 ++--
 drivers/media/platform/atmel/atmel-isc.c   |  15 ++-
 drivers/media/platform/atmel/atmel-isi.c   |  17 +--
 drivers/media/platform/cadence/cdns-csi2rx.c   |  28 ++--
 drivers/media/platform/davinci/vpif_capture.c  |  71 +-
 drivers/media/platform/davinci/vpif_display.c  |  25 ++--
 drivers/media/platform/exynos4-is/media-dev.c  |  32 +++--
 drivers/media/platform/exynos4-is/media-dev.h  |   1 -
 drivers/media/platform/pxa_camera.c|  25 ++--
 drivers/media/platform/qcom/camss-8x16/camss.c |  86 ++--
 drivers/media/platform/qcom/camss-8x16/camss.h |   2 +-
 drivers/media/platform/rcar-vin/rcar-core.c|   2 +-
 drivers/media/platform/rcar-vin/rcar-csi2.c|  22 ++--
 drivers/media/platform/rcar_drif.c |  18 ++-
 drivers/media/platform/renesas-ceu.c   |  53 
 drivers/media/platform/soc_camera/soc_camera.c |  35 +++--
 drivers/media/platform/stm32/stm32-dcmi.c  |  24 ++--
 drivers/media/platform/ti-vpe/cal.c|  48 +--
 drivers/media/platform/xilinx/xilinx-vipp.c| 173 -
 drivers/media/platform/xilinx/xilinx-vipp.h|   4 -
 21 files changed, 423 insertions(+), 342 deletions(-)

diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2.c 
b/drivers/media/pci/intel/ipu3/ipu3-cio2.c
index 4a5f7c3..ca3ad11 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2.c
@@ -1508,7 +1508,7 @@ static int cio2_notifier_init(struct cio2_device *cio2)
if (ret < 0)
return ret;
 
-   if (!cio2->notifier.num_subdevs)
+   if (list_empty(&cio2->notifier.asd_list))
return -ENODEV; /* no endpoint */
 
cio2->notifier.ops = &cio2_async_ops;
diff --git a/drivers/media/platform/am437x/am437x-vpfe.c 
b/drivers/media/platform/am437x/am437x-vpfe.c
index b05738a..b19e0ab 100644
--- a/drivers/media/platform/am437x/am437x-vpfe.c
+++ b/drivers/media/platform/am437x/am437x-vpfe.c
@@ -2423,30 +2423,32 @@ static const struct v4l2_async_notifier_operations 
vpfe_async_ops = {
 };
 
 static struct vpfe_config *
-vpfe_get_pdata(struct platform_device *pdev)
+vpfe_get_pdata(struct vpfe_device *vpfe)
 {
struct device_node *endpoint = NULL;
struct v4l2_fwnode_endpoint bus_cfg;
+   struct device *dev = vpfe->pdev;
struct vpfe_subdev_info *sdinfo;
struct vpfe_config *pdata;
unsigned int flags;
unsigned int i;
int err;
 
-   dev_dbg(&pdev->dev, "vpfe_get_pdata\n");
+   dev_dbg(dev, "vpfe_get_pdata\n");
 
-   if (!IS_ENABLED(CONFIG_OF) || !pdev->dev.of_node)
-   return pdev->dev.platform_data;
+   v4l2_async_notifier_init(&vpfe->notifier);
 
-   pdata = devm_kzalloc(&pdev->dev, sizeof(*pdata), GFP_KERNEL);
+   if (!IS_ENABLED(CONFIG_OF) || !dev->of_node)
+   return dev->platform_data;
+
+   pdata = devm_kzalloc(dev, sizeof(*pdata), GFP_KERNEL);
if (!pdata)
return NULL;
 
for (i = 0; ; i++) {
struct device_node *rem;
 
-   endpoint = of_graph_get_next_endpoint(pdev->dev.of_node,
- endpoint);
+   endpoint = of_graph_get_next_endpoint(dev->of_node, endpoint);
if (!endpoint)
break;
 
@@ -2473,16 +2475,16 @@ vpfe_get_

[PATCH v3 00/14] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

History:
v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
  ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
  Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
  CSI (and PRPENCVF) at the sink and source pads. In v3, this
  has been moved one hop downstream: interweave is now determined
  using field type at source pad, and field type selected at
  capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
  type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
  and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
  Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Philipp Zabel (1):
  gpu: ipu-v3: Allow negative offsets for interlaced scanning

Steve Longerbeam (13):
  media: videodev2.h: Add more field helper macros
  gpu: ipu-csi: Check for field type alternate
  gpu: ipu-csi: Swap fields according to input/output field types
  gpu: ipu-v3: Fix U/V offset macros for planar 4:2:0
  gpu: ipu-v3: Add planar support to interlaced scan
  media: imx: Fix field negotiation
  media: imx-csi: Double crop height for alternate fields at sink
  media: imx: interweave and odd-chroma-row skip are incompatible
  media: imx-csi: Allow skipping odd chroma rows for YVU420
  media: imx: vdic: rely on VDIC for correct field order
  media: imx-csi: Move crop/compose reset after filling default mbus
fields
  media: imx: Allow interweave with top/bottom lines swapped
  media: imx.rst: Update doc to reflect fixes to interlaced capture

 Documentation/media/v4l-drivers/imx.rst   |  93 ++-
 drivers/gpu/ipu-v3/ipu-cpmem.c|  45 ++-
 drivers/gpu/ipu-v3/ipu-csi.c  | 136 ++---
 drivers/staging/media/imx/imx-ic-prpencvf.c   |  48 ++--
 drivers/staging/media/imx/imx-media-capture.c |  14 +++
 drivers/staging/media/imx/imx-media-csi.c | 166 ++
 drivers/staging/media/imx/imx-media-vdic.c|  12 +-
 include/uapi/linux/videodev2.h|   7 ++
 include/video/imx-ipu-v3.h|   6 +-
 9 files changed, 377 insertions(+), 150 deletions(-)

-- 
2.7.4

Re: [PATCH v2 0/2] media: i2c: ov5640: Re-work MIPI startup sequence

[Steve Longerbeam]

Hi Jacopo,


On 08/14/2018 08:35 AM, jacopo mondi wrote:

Hi Steve,
sorry for resurecting this.



I'm sorry I'm not sur I'm following. Does this mean that with that bug
you are referring to up here fixed by my last patch you have capture
working?

No, capture still not working for me on SabreSD, even after fixing
the bug in 476dec0 "media: ov5640: Add horizontal and vertical totals",
by either using your patchset, or by running version 476dec0 of ov5640.c
with the call to ov5640_set_timings() moved to the correct places as
described below.


I've been reported a bug on exposure handling that makes the first
captured frames all black. Both me and Hugues have tried to fix the
issue (him with a more complete series, but that's another topic).
See [1] and [2]

It might be possible that you're getting blank frames with this series
applied? I never seen them as I'm skipping the first frames when
capturing, but I've now tested and without the exposure fixes (either
[1] or [2]) I actually have blank frames.

If that's the case for you too (which I hope so much) would you be
available to test again this series with exposure fixes on top?
On my platform that actually makes all frames correct.

Thanks
j

[1] [PATCH 0/2] media: ov5640: Fix set_timings and auto-exposure
[2] [PATCH v2 0/5] Fix OV5640 exposure & gain



It's not clear to me which patch sets you would like me to test.
Just [1] and [2], or [1], [2], and "media: i2c: ov5640: Re-work MIPI 
startup sequence"?


Steve

Re: [PATCH v2 0/2] media: i2c: ov5640: Re-work MIPI startup sequence

[Steve Longerbeam]

On 08/14/2018 10:38 AM, jacopo mondi wrote:

Hi Steve,

On Tue, Aug 14, 2018 at 09:51:04AM -0700, Steve Longerbeam wrote:

Hi Jacopo,


On 08/14/2018 08:35 AM, jacopo mondi wrote:

Hi Steve,
sorry for resurecting this.



I'm sorry I'm not sur I'm following. Does this mean that with that bug
you are referring to up here fixed by my last patch you have capture
working?

No, capture still not working for me on SabreSD, even after fixing
the bug in 476dec0 "media: ov5640: Add horizontal and vertical totals",
by either using your patchset, or by running version 476dec0 of ov5640.c
with the call to ov5640_set_timings() moved to the correct places as
described below.


I've been reported a bug on exposure handling that makes the first
captured frames all black. Both me and Hugues have tried to fix the
issue (him with a more complete series, but that's another topic).
See [1] and [2]

It might be possible that you're getting blank frames with this series
applied? I never seen them as I'm skipping the first frames when
capturing, but I've now tested and without the exposure fixes (either
[1] or [2]) I actually have blank frames.

If that's the case for you too (which I hope so much) would you be
available to test again this series with exposure fixes on top?
On my platform that actually makes all frames correct.

Thanks
j

[1] [PATCH 0/2] media: ov5640: Fix set_timings and auto-exposure
[2] [PATCH v2 0/5] Fix OV5640 exposure & gain


It's not clear to me which patch sets you would like me to test.
Just [1] and [2], or [1], [2], and "media: i2c: ov5640: Re-work MIPI startup
sequence"?


I have tested on my board the following:
v4.18-rc2 + MIPI Fix + Timings + Hugues' exposure fix

Without Hugues' patches I get blank frames (the first ones at least)
Without MIPI startup reowkr and timings I get the LP-11 error on the
CSI-2 bus.

As Hugues' series has to be rebased on mine, I have prepared a branch
here for you if you feel like testing it:
git://jmondi.org/linux ov5640/timings_exposure


Hi Jacopo, that branch works on SabreSD!

Feel free to add

Tested-by: Steve Longerbeam 
on i.MX6q SabreSD with MIPI CSI-2 OV5640 module

to whichever ov5640 patches are appropriate.

Steve

Re: [PATCH v2 00/23] V4L2 fwnode rework; support for default configuration

[Steve Longerbeam]

Hi Sakari,


On 08/27/2018 02:29 AM, Sakari Ailus wrote:

Hello everyone,

I've long thought the V4L2 fwnode framework requires some work (it's buggy
and it does not adequately serve common needs). This set should address in
particular these matters:

- Most devices support a particular media bus type but the V4L2 fwnode
   framework was not able to use such information, but instead tried to
   guess the bus type with varying levels of success while drivers
   generally ignored the results. This patchset makes that possible ---
   setting a bus type enables parsing configuration for only that bus.
   Failing that check results in returning -ENXIO to be returned.

- Support specifying default configuration. If the endpoint has no
   configuration, the defaults set by the driver (as documented in DT
   bindings) will prevail. Any available configuration will still be read
   from the endpoint as one could expect. A common use case for this is
   e.g. the number of CSI-2 lanes. Few devices support lane mapping, and
   default 1:1 mapping is provided in absence of a valid default or
   configuration read OF.

- Debugging information is greatly improved.

- Recognition of the differences between CSI-2 D-PHY and C-PHY. All
   currently supported hardware (or at least drivers) is D-PHY only, so
   this change is still easy.

The smiapp driver is converted to use the new functionality. This patchset
does not address remaining issues such as supporting setting defaults for
e.g. bridge drivers with multiple ports, but with Steve Longerbeam's
patchset we're much closer to that goal. I've rebased this set on top of
Steve's. Albeit the two deal with the same files, there were only a few
trivial conflicts.

Note that I've only tested parsing endpoints for the CSI-2 bus (no
parallel IF hardware). Jacopo has tested an earlier version of the set
with a few changes to the parallel bus handling compared to this one.

Comments are welcome.


I got around to testing this. The following diff needs to be added
to initialize bus_type before calling v4l2_fwnode_endpoint_parse()
in imx-media driver, this should probably be squashed with
"v4l: fwnode: Initialise the V4L2 fwnode endpoints to zero":

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c

index 539159d..ac9d718 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1050,7 +1050,7 @@ static int csi_link_validate(struct v4l2_subdev *sd,
 struct v4l2_subdev_format *sink_fmt)
 {
    struct csi_priv *priv = v4l2_get_subdevdata(sd);
-   struct v4l2_fwnode_endpoint upstream_ep = {};
+   struct v4l2_fwnode_endpoint upstream_ep = { .bus_type = 0 };
    bool is_csi2;
    int ret;

@@ -1164,7 +1164,7 @@ static int csi_enum_mbus_code(struct v4l2_subdev *sd,
  struct v4l2_subdev_mbus_code_enum *code)
 {
    struct csi_priv *priv = v4l2_get_subdevdata(sd);
-   struct v4l2_fwnode_endpoint upstream_ep;
+   struct v4l2_fwnode_endpoint upstream_ep = { .bus_type = 0 };
    const struct imx_media_pixfmt *incc;
    struct v4l2_mbus_framefmt *infmt;
    int ret = 0;
@@ -1403,7 +1403,7 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 {
    struct csi_priv *priv = v4l2_get_subdevdata(sd);
    struct imx_media_video_dev *vdev = priv->vdev;
-   struct v4l2_fwnode_endpoint upstream_ep;
+   struct v4l2_fwnode_endpoint upstream_ep = { .bus_type = 0 };
    const struct imx_media_pixfmt *cc;
    struct v4l2_pix_format vdev_fmt;
    struct v4l2_mbus_framefmt *fmt;
@@ -1542,7 +1542,7 @@ static int csi_set_selection(struct v4l2_subdev *sd,
 struct v4l2_subdev_selection *sel)
 {
    struct csi_priv *priv = v4l2_get_subdevdata(sd);
-   struct v4l2_fwnode_endpoint upstream_ep;
+   struct v4l2_fwnode_endpoint upstream_ep = { .bus_type = 0 };
    struct v4l2_mbus_framefmt *infmt;
    struct v4l2_rect *crop, *compose;
    int pad, ret;


After making that change, capture from CSI-2 OV5640 and parallel
OV5642 on the imx6q Sabrelite is working fine. Feel free to add my
Tested-by on that platform.



I've pushed the patches (including Steve's) here:

https://git.linuxtv.org/sailus/media_tree.git/log/?h=v4l2-fwnode-next>

since v1:

- Rebase it all on current media tree master --- there was a conflict in
   drivers/media/platform/qcom/camss/camss.c in Steve's patch "media:
   platform: Switch to v4l2_async_notifier_add_subdev"; I hope the
   resolution was fine.


I checked your resolution to camss.c and it was the same resolution I
made as well.

Thanks,
Steve



- Default to Bt.656 bus in guessing the bus type if no properties
   suggesting otherwise are set. In v1 and error was returned, which would
   have been troublesome for the existing drivers.

- Set the bus_type field to zero (i.e. guess) for existing callers of
   v4l2_fwnode

Re: [PATCH v4 2/2] media: ov5640: Fix timings setup code

[Steve Longerbeam]

On 09/14/2018 08:58 AM, Jacopo Mondi wrote:

From: Jacopo Mondi 

As of:
commit 476dec012f4c ("media: ov5640: Add horizontal and vertical totals")
the timings parameters gets programmed separately from the static register
values array.

When changing capture mode, the vertical and horizontal totals gets inspected
by the set_mode_exposure_calc() functions, and only later programmed with the
new values. This means exposure, light banding filter and shutter gain are
calculated using the previous timings, and are thus not correct.

Fix this by programming timings right after the static register value table
has been sent to the sensor in the ov5640_load_regs() function.

Fixes: 476dec012f4c ("media: ov5640: Add horizontal and vertical totals")
Tested-by: Steve Longerbeam 
on i.MX6q SabreSD with MIPI CSI-2 OV5640 module


Acked-by: Steve Longerbeam 


Tested-by: Loic Poulain 
on Dragonboard-410c with MIPI CSI-2 OV5640 module
Signed-off-by: Samuel Bobrowicz 
Signed-off-by: Maxime Ripard 
Signed-off-by: Jacopo Mondi 
---
  drivers/media/i2c/ov5640.c | 50 +++---
  1 file changed, 21 insertions(+), 29 deletions(-)

diff --git a/drivers/media/i2c/ov5640.c b/drivers/media/i2c/ov5640.c
index 7ade416..2b9e84f 100644
--- a/drivers/media/i2c/ov5640.c
+++ b/drivers/media/i2c/ov5640.c
@@ -908,6 +908,26 @@ static int ov5640_mod_reg(struct ov5640_dev *sensor, u16 
reg,
  }

  /* download ov5640 settings to sensor through i2c */
+static int ov5640_set_timings(struct ov5640_dev *sensor,
+ const struct ov5640_mode_info *mode)
+{
+   int ret;
+
+   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_DVPHO, mode->hact);
+   if (ret < 0)
+   return ret;
+
+   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_DVPVO, mode->vact);
+   if (ret < 0)
+   return ret;
+
+   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_HTS, mode->htot);
+   if (ret < 0)
+   return ret;
+
+   return ov5640_write_reg16(sensor, OV5640_REG_TIMING_VTS, mode->vtot);
+}
+
  static int ov5640_load_regs(struct ov5640_dev *sensor,
const struct ov5640_mode_info *mode)
  {
@@ -935,7 +955,7 @@ static int ov5640_load_regs(struct ov5640_dev *sensor,
usleep_range(1000 * delay_ms, 1000 * delay_ms + 100);
}

-   return ret;
+   return ov5640_set_timings(sensor, mode);
  }

  /* read exposure, in number of line periods */
@@ -1398,30 +1418,6 @@ static int ov5640_set_virtual_channel(struct ov5640_dev 
*sensor)
return ov5640_write_reg(sensor, OV5640_REG_DEBUG_MODE, temp);
  }

-static int ov5640_set_timings(struct ov5640_dev *sensor,
- const struct ov5640_mode_info *mode)
-{
-   int ret;
-
-   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_DVPHO, mode->hact);
-   if (ret < 0)
-   return ret;
-
-   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_DVPVO, mode->vact);
-   if (ret < 0)
-   return ret;
-
-   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_HTS, mode->htot);
-   if (ret < 0)
-   return ret;
-
-   ret = ov5640_write_reg16(sensor, OV5640_REG_TIMING_VTS, mode->vtot);
-   if (ret < 0)
-   return ret;
-
-   return 0;
-}
-
  static const struct ov5640_mode_info *
  ov5640_find_mode(struct ov5640_dev *sensor, enum ov5640_frame_rate fr,
 int width, int height, bool nearest)
@@ -1665,10 +1661,6 @@ static int ov5640_set_mode(struct ov5640_dev *sensor,
if (ret < 0)
return ret;

-   ret = ov5640_set_timings(sensor, mode);
-   if (ret < 0)
-   return ret;
-
ret = ov5640_set_binning(sensor, dn_mode != SCALING);
if (ret < 0)
return ret;
--
2.7.4

Re: [PATCH v4 1/2] media: ov5640: Re-work MIPI startup sequence

[Steve Longerbeam]

On 09/14/2018 08:58 AM, Jacopo Mondi wrote:

From: Jacopo Mondi 

Rework the MIPI interface startup sequence with the following changes:

- Remove MIPI bus initialization from the initial settings blob
- At set_power(1) time power up MIPI Tx/Rx and set data and clock lanes in
   LP11 during 'sleep' and 'idle' with MIPI clock in non-continuous mode.
- At s_stream time enable/disable the MIPI interface output.
- Restore default settings at set_power(0) time.

Before this commit the sensor MIPI interface was initialized with settings
that require a start/stop sequence at power-up time in order to force lanes
into LP11 state, as they were initialized in LP00 when in 'sleep mode',
which is assumed to be the sensor manual definition for the D-PHY defined
stop mode.

The stream start/stop was performed by enabling disabling clock gating,
and had the side effect to change the lanes sleep mode configuration when
stream was stopped.

Clock gating/ungating:
-   ret = ov5640_mod_reg(sensor, OV5640_REG_MIPI_CTRL00, BIT(5),
-on ? 0 : BIT(5));
-   if (ret)

Set lanes in LP11 when in 'sleep mode':
-   ret = ov5640_write_reg(sensor, OV5640_REG_PAD_OUTPUT00,
-  on ? 0x00 : 0x70);

This commit fixes an issue reported by Jagan Teki on i.MX6 platforms that
prevents the host interface from powering up correctly:
https://lkml.org/lkml/2018/6/1/38

It also improves MIPI capture operations stability on my testing platform
where MIPI capture often failed and returned all-purple frames.

fixes: f22996db44e2 ("media: ov5640: add support of DVP parallel interface")
Tested-by: Steve Longerbeam 
on i.MX6q SabreSD with MIPI CSI-2 OV5640 module


Acked-by: Steve Longerbeam 


Tested-by: Loic Poulain 
on Dragonboard-410c with MIPI CSI-2 OV5640 module
Reported-by: Jagan Teki 
Signed-off-by: Jacopo Mondi 
---
  drivers/media/i2c/ov5640.c | 99 --
  1 file changed, 79 insertions(+), 20 deletions(-)

diff --git a/drivers/media/i2c/ov5640.c b/drivers/media/i2c/ov5640.c
index 071f4bc..7ade416 100644
--- a/drivers/media/i2c/ov5640.c
+++ b/drivers/media/i2c/ov5640.c
@@ -286,10 +286,10 @@ static const struct reg_value 
ov5640_init_setting_30fps_VGA[] = {
{0x3a0d, 0x04, 0, 0}, {0x3a14, 0x03, 0, 0}, {0x3a15, 0xd8, 0, 0},
{0x4001, 0x02, 0, 0}, {0x4004, 0x02, 0, 0}, {0x3000, 0x00, 0, 0},
{0x3002, 0x1c, 0, 0}, {0x3004, 0xff, 0, 0}, {0x3006, 0xc3, 0, 0},
-   {0x300e, 0x45, 0, 0}, {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
+   {0x302e, 0x08, 0, 0}, {0x4300, 0x3f, 0, 0},
{0x501f, 0x00, 0, 0}, {0x4713, 0x03, 0, 0}, {0x4407, 0x04, 0, 0},
{0x440e, 0x00, 0, 0}, {0x460b, 0x35, 0, 0}, {0x460c, 0x22, 0, 0},
-   {0x4837, 0x0a, 0, 0}, {0x4800, 0x04, 0, 0}, {0x3824, 0x02, 0, 0},
+   {0x4837, 0x0a, 0, 0}, {0x3824, 0x02, 0, 0},
{0x5000, 0xa7, 0, 0}, {0x5001, 0xa3, 0, 0}, {0x5180, 0xff, 0, 0},
{0x5181, 0xf2, 0, 0}, {0x5182, 0x00, 0, 0}, {0x5183, 0x14, 0, 0},
{0x5184, 0x25, 0, 0}, {0x5185, 0x24, 0, 0}, {0x5186, 0x09, 0, 0},
@@ -1102,12 +1102,25 @@ static int ov5640_set_stream_mipi(struct ov5640_dev 
*sensor, bool on)
  {
int ret;

-   ret = ov5640_mod_reg(sensor, OV5640_REG_MIPI_CTRL00, BIT(5),
-on ? 0 : BIT(5));
-   if (ret)
-   return ret;
-   ret = ov5640_write_reg(sensor, OV5640_REG_PAD_OUTPUT00,
-  on ? 0x00 : 0x70);
+   /*
+* Enable/disable the MIPI interface
+*
+* 0x300e = on ? 0x45 : 0x40
+*
+* FIXME: the sensor manual (version 2.03) reports
+* [7:5] = 000  : 1 data lane mode
+* [7:5] = 001  : 2 data lanes mode
+* But this settings do not work, while the following ones
+* have been validated for 2 data lanes mode.
+*
+* [7:5] = 010  : 2 data lanes mode
+* [4] = 0  : Power up MIPI HS Tx
+* [3] = 0  : Power up MIPI LS Rx
+* [2] = 1/0: MIPI interface enable/disable
+* [1:0] = 01/00: FIXME: 'debug'
+*/
+   ret = ov5640_write_reg(sensor, OV5640_REG_IO_MIPI_CTRL00,
+  on ? 0x45 : 0x40);
if (ret)
return ret;

@@ -1786,23 +1799,69 @@ static int ov5640_set_power(struct ov5640_dev *sensor, 
bool on)
if (ret)
goto power_off;

+   /* We're done here for DVP bus, while CSI-2 needs setup. */
+   if (sensor->ep.bus_type != V4L2_MBUS_CSI2)
+   return 0;
+
+   /*
+* Power up MIPI HS Tx and LS Rx; 2 data lanes mode
+*
+* 0x300e = 0x40
+* [7:5] = 010  : 2 data lanes mode (see FIXME note in
+*"ov5640_set_stream_mipi()")
+*

Re: [PATCH v3 0/5] Fix OV5640 exposure & gain

[Steve Longerbeam]

Hi Hughes,

The whole series,

Acked-by: Steve Longerbeam 

and

Tested-by: Steve Longerbeam 
on i.MX6q SabreSD with MIPI CSI-2 OV5640 module


On 09/11/2018 06:48 AM, Hugues Fruchet wrote:

This patch serie fixes some problems around exposure & gain in OV5640 driver.

The 4th patch about autocontrols requires also a fix in v4l2-ctrls.c:
https://www.mail-archive.com/linux-media@vger.kernel.org/msg133164.html

Here is the test procedure used for exposure & gain controls check:
1) Preview in background
$> gst-launch-1.0 v4l2src ! "video/x-raw, width=640, Height=480" ! queue ! 
waylandsink -e &
2) Check gain & exposure values
$> v4l2-ctl --all | grep -e exposure -e gain | grep "(int)"
exposure (int): min=0 max=65535 step=1 default=0 
value=330 flags=inactive, volatile
gain (int): min=0 max=1023 step=1 default=0 
value=19 flags=inactive, volatile
3) Put finger in front of camera and check that gain/exposure values are 
changing:
$> v4l2-ctl --all | grep -e exposure -e gain | grep "(int)"
exposure (int): min=0 max=65535 step=1 default=0 
value=660 flags=inactive, volatile
gain (int): min=0 max=1023 step=1 default=0 
value=37 flags=inactive, volatile
4) switch to manual mode, image exposition must not change
$> v4l2-ctl --set-ctrl=gain_automatic=0
$> v4l2-ctl --set-ctrl=auto_exposure=1
Note the "1" for manual exposure.

5) Check current gain/exposure values:
$> v4l2-ctl --all | grep -e exposure -e gain | grep "(int)"
exposure (int): min=0 max=65535 step=1 default=0 
value=330
gain (int): min=0 max=1023 step=1 default=0 
value=20

6) Put finger behind camera and check that gain/exposure values are NOT 
changing:
$> v4l2-ctl --all | grep -e exposure -e gain | grep "(int)"
exposure (int): min=0 max=65535 step=1 default=0 
value=330
gain (int): min=0 max=1023 step=1 default=0 
value=20
7) Update exposure, check that it is well changed on display and that same 
value is returned:
$> v4l2-ctl --set-ctrl=exposure=100
$> v4l2-ctl --get-ctrl=exposure
exposure: 100

9) Update gain, check that it is well changed on display and that same value is 
returned:
$> v4l2-ctl --set-ctrl=gain=10
$> v4l2-ctl --get-ctrl=gain
gain: 10

10) Switch back to auto gain/exposure, verify that image is correct and values 
returned are correct:
$> v4l2-ctl --set-ctrl=gain_automatic=1
$> v4l2-ctl --set-ctrl=auto_exposure=0
$> v4l2-ctl --all | grep -e exposure -e gain | grep "(int)"
exposure (int): min=0 max=65535 step=1 default=0 
value=330 flags=inactive, volatile
gain (int): min=0 max=1023 step=1 default=0 
value=22 flags=inactive, volatile
Note the "0" for auto exposure.

===
= history =
===
version 3:
   - Change patch 5/5 by removing set_mode() orig_mode parameter as per jacopo' 
suggestion:
 https://www.spinics.net/lists/linux-media/msg139457.html

version 2:
   - Fix patch 3/5 commit comment and rename binning function as per jacopo' 
suggestion:
 https://www.mail-archive.com/linux-media@vger.kernel.org/msg133272.html

Hugues Fruchet (5):
   media: ov5640: fix exposure regression
   media: ov5640: fix auto gain & exposure when changing mode
   media: ov5640: fix wrong binning value in exposure calculation
   media: ov5640: fix auto controls values when switching to manual mode
   media: ov5640: fix restore of last mode set

  drivers/media/i2c/ov5640.c | 128 ++---
  1 file changed, 73 insertions(+), 55 deletions(-)

Re: [PATCH] media: imx: use well defined 32-bit RGB pixel format

[Steve Longerbeam]

On 09/18/2018 02:42 AM, Philipp Zabel wrote:

The documentation in Documentation/media/uapi/v4l/pixfmt-packed-rgb.rst
tells us that the V4L2_PIX_FMT_RGB32 format is deprecated and must not
be used by new drivers. Replace it with V4L2_PIX_FMT_XRGB32.

Signed-off-by: Philipp Zabel 


Acked-by: Steve Longerbeam 


---
  drivers/staging/media/imx/imx-media-utils.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-utils.c 
b/drivers/staging/media/imx/imx-media-utils.c
index 8aa13403b09d..0eaa353d5cb3 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -88,7 +88,7 @@ static const struct imx_media_pixfmt rgb_formats[] = {
.cs = IPUV3_COLORSPACE_RGB,
.bpp= 24,
}, {
-   .fourcc = V4L2_PIX_FMT_RGB32,
+   .fourcc = V4L2_PIX_FMT_XRGB32,
.codes  = {MEDIA_BUS_FMT_ARGB_1X32},
.cs = IPUV3_COLORSPACE_RGB,
.bpp= 32,
@@ -212,7 +212,7 @@ static const struct imx_media_pixfmt ipu_yuv_formats[] = {
  
  static const struct imx_media_pixfmt ipu_rgb_formats[] = {

{
-   .fourcc = V4L2_PIX_FMT_RGB32,
+   .fourcc = V4L2_PIX_FMT_XRGB32,
.codes  = {MEDIA_BUS_FMT_ARGB_1X32},
.cs = IPUV3_COLORSPACE_RGB,
.bpp= 32,

Re: [PATCH] media: staging/imx: Handle CSI->VDIC->PRPVF pipeline

[Steve Longerbeam]

On 09/17/2018 03:25 AM, Hans Verkuil wrote:

On 05/07/2018 11:55 AM, Hans Verkuil wrote:

On 07/04/18 15:05, Marek Vasut wrote:

In case the PRPVF is not connected directly to CSI, the PRPVF subdev
driver won't find the CSI subdev and will not configure the CSI input
mux. This is not noticable on the IPU1-CSI0 interface with parallel
camera, since the mux is set "correctly" by default and the parallel
camera will work just fine. This is however noticable on IPU2-CSI1,
where the mux is not set to the correct position by default and the
pipeline will fail.

Add similar code to what is in PRPVF to VDIC driver, so that the VDIC
can locate the CSI subdev and configure the mux correctly if the CSI
is connected to the VDIC. Make the PRPVF driver configure the CSI mux
only in case it's connected directly to CSI and not in case it is
connected to VDIC.

Signed-off-by: Marek Vasut 
Cc: Philipp Zabel 
Cc: Steve Longerbeam 

Same here, I cannot merge with out Acks since I don't know the details
of the imx hardware.

I'm marking this patch as Obsoleted since there has been no activity for a long
time.


Hi Hans, yes that's fine. IIRC this issue has been fixed a different way.

Steve





---
  drivers/staging/media/imx/imx-ic-prp.c |  6 ++
  drivers/staging/media/imx/imx-media-vdic.c | 24 
  2 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prp.c 
b/drivers/staging/media/imx/imx-ic-prp.c
index 98923fc844ce..84fa66dae21a 100644
--- a/drivers/staging/media/imx/imx-ic-prp.c
+++ b/drivers/staging/media/imx/imx-ic-prp.c
@@ -72,14 +72,12 @@ static inline struct prp_priv *sd_to_priv(struct 
v4l2_subdev *sd)
  static int prp_start(struct prp_priv *priv)
  {
struct imx_ic_priv *ic_priv = priv->ic_priv;
-   bool src_is_vdic;
  
  	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
  
  	/* set IC to receive from CSI or VDI depending on source */

-   src_is_vdic = !!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_VDIC);
-
-   ipu_set_ic_src_mux(priv->ipu, priv->csi_id, src_is_vdic);
+   if (!(priv->src_sd->grp_id & IMX_MEDIA_GRP_ID_VDIC))
+   ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
  
  	return 0;

  }
diff --git a/drivers/staging/media/imx/imx-media-vdic.c 
b/drivers/staging/media/imx/imx-media-vdic.c
index b538bbebedc5..e660911e7024 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -117,6 +117,9 @@ struct vdic_priv {
  
  	bool csi_direct;  /* using direct CSI->VDIC->IC pipeline */
  
+	/* the CSI id at link validate */

+   int csi_id;
+
/* motion select control */
struct v4l2_ctrl_handler ctrl_hdlr;
enum ipu_motion_sel motion;
@@ -388,6 +391,9 @@ static int vdic_start(struct vdic_priv *priv)
if (ret)
return ret;
  
+	/* set IC to receive from CSI or VDI depending on source */

+   ipu_set_ic_src_mux(priv->ipu, priv->csi_id, true);
+
/*
 * init the VDIC.
 *
@@ -778,6 +784,7 @@ static int vdic_link_validate(struct v4l2_subdev *sd,
  struct v4l2_subdev_format *sink_fmt)
  {
struct vdic_priv *priv = v4l2_get_subdevdata(sd);
+   struct imx_media_subdev *csi;
int ret;
  
  	ret = v4l2_subdev_link_validate_default(sd, link,

@@ -785,6 +792,23 @@ static int vdic_link_validate(struct v4l2_subdev *sd,
if (ret)
return ret;
  
+	csi = imx_media_find_upstream_subdev(priv->md, priv->src,

+IMX_MEDIA_GRP_ID_CSI);
+   if (!IS_ERR(csi)) {
+   switch (csi->sd->grp_id) {
+   case IMX_MEDIA_GRP_ID_CSI0:
+   priv->csi_id = 0;
+   break;
+   case IMX_MEDIA_GRP_ID_CSI1:
+   priv->csi_id = 1;
+   break;
+   default:
+   ret = -EINVAL;
+   }
+   } else {
+   priv->csi_id = 0;
+   }
+
mutex_lock(&priv->lock);
  
  	if (priv->csi_direct && priv->motion != HIGH_MOTION) {

Re: [PATCH] media: imx: Skip every second frame in VDIC DIRECT mode

[Steve Longerbeam]

On 09/17/2018 03:27 AM, Hans Verkuil wrote:

On 05/07/2018 11:54 AM, Hans Verkuil wrote:

On 07/04/18 15:04, Marek Vasut wrote:

In VDIC direct mode, the VDIC applies combing filter during and
doubles the framerate, that is, after the first two half-frames
are received and the first frame is emitted by the VDIC, every
subsequent half-frame is patched into the result and a full frame
is produced. The half-frame order in the full frames is as follows
12 32 34 54 etc.

Drop every second frame to trim the framerate back to the original
one of the signal and skip the odd patched frames.

Signed-off-by: Marek Vasut 
Cc: Steve Longerbeam 
Cc: Philipp Zabel 

Steve, Philipp,

I saw there was a discussion about this patch, but no clear answer whether
or not this patch is OK. If it is, then please Ack this patch.

Marking this patch as Obsoleted since I have no seen any activity for a long 
time.


Hi Hans, yes that's fine.

This needs to be re-worked to allow configuration of input/output 
frame-rates

from the VDIC via [gs]_frame_interval.

Steve





---
  drivers/staging/media/imx/imx-media-vdic.c | 3 +++
  1 file changed, 3 insertions(+)

diff --git a/drivers/staging/media/imx/imx-media-vdic.c 
b/drivers/staging/media/imx/imx-media-vdic.c
index 482250d47e7c..b538bbebedc5 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -289,6 +289,7 @@ static int vdic_setup_direct(struct vdic_priv *priv)
/* set VDIC to receive from CSI for direct path */
ipu_fsu_link(priv->ipu, IPUV3_CHANNEL_CSI_DIRECT,
 IPUV3_CHANNEL_CSI_VDI_PREV);
+   ipu_set_vdi_skip(priv->ipu, 0x2);
  
  	return 0;

  }
@@ -313,6 +314,8 @@ static int vdic_setup_indirect(struct vdic_priv *priv)
const struct imx_media_pixfmt *incc;
int in_size, ret;
  
+	ipu_set_vdi_skip(priv->ipu, 0x0);

+
infmt = &priv->format_mbus[VDIC_SINK_PAD_IDMAC];
incc = priv->cc[VDIC_SINK_PAD_IDMAC];
  

[PATCH v7 00/17] media: imx: Switch to subdev notifiers

[Steve Longerbeam]

This patchset converts the imx-media driver and its dependent
subdevs to use subdev notifiers.

There are a couple shortcomings in v4l2-core that prevented
subdev notifiers from working correctly in imx-media:

1. v4l2_async_notifier_fwnode_parse_endpoint() treats a fwnode
   endpoint that is not connected to a remote device as an error.
   But in the case of the video-mux subdev, this is not an error,
   it is OK if some of the mux inputs have no connection. Also,
   Documentation/devicetree/bindings/media/video-interfaces.txt explicitly
   states that the 'remote-endpoint' property is optional. So the first
   patch is a small modification to ignore empty endpoints in
   v4l2_async_notifier_fwnode_parse_endpoint() and allow
   __v4l2_async_notifier_parse_fwnode_endpoints() to continue to
   parse the remaining port endpoints of the device.

2. In the imx-media graph, multiple subdevs will encounter the same
   upstream subdev (such as the imx6-mipi-csi2 receiver), and so
   v4l2_async_notifier_parse_fwnode_endpoints() will add imx6-mipi-csi2
   multiple times. This is treated as an error by
   v4l2_async_notifier_register() later.

   To get around this problem, add an v4l2_async_notifier_add_subdev()
   which first verifies the provided asd does not already exist in the
   given notifier asd list or in other registered notifiers. If the asd
   exists, the function returns -EEXIST and it's up to the caller to
   decide if that is an error (in imx-media case it is never an error).

   Patches 2-5 deal with adding that support.

3. Patch 6 adds v4l2_async_register_fwnode_subdev(), which is a
   convenience function for parsing a subdev's fwnode port endpoints
   for connected remote subdevs, registering a subdev notifier, and
   then registering the sub-device itself.

4. Patches 7-14 update the subdev drivers to register a subdev notifier
   with endpoint parsing, and the changes to imx-media to support that.

5. Finally, the last 3 patches endeavor to completely remove support for
   the notifier->subdevs[] array in platform drivers and v4l2 core. All
   platform drivers are modified to make use of
   v4l2_async_notifier_add_subdev() and its related convenience functions
   to add asd's to the notifier @asd_list, and any allocation or reference
   to the notifier->subdevs[] array removed. After that large patch,
   notifier->subdevs[] array is stripped from v4l2-async and v4l2-subdev
   docs are updated to reflect the new method of adding asd's to notifiers.


Signed-off-by: Steve Longerbeam 

Patches 07-14 (video-mux and the imx patches) are
Reviewed-by: Philipp Zabel 

Patches 01-14 are
Tested-by: Philipp Zabel 
on i.MX6 with Toshiba TC358743 connected via MIPI CSI-2.

History:

v7:
- Comment, error message, and whitespace fixes in patch 02/17.
  Suggested by Mauro Chehab.
- Fixed merge conflict in drivers/media/platform/ti-vpe/cal.c.
- Make patch author and SoB email addresses the same.

v6:
- Export v4l2_async_notifier_init(), which must be called by all
  drivers before the first call to v4l2_async_notifier_add_subdev().
  Suggested by Sakari Ailus.
- Removed @num_subdevs from struct v4l2_async_notifier, and the macro
  V4L2_MAX_SUBDEVS. Sugested by Sakari.
- Fixed a double device node put in vpif_capture.c. Reported by Sakari.
- Fixed wrong printk format qualifiers in xilinx-vipp.c. Reported by
  Dan Carpenter.

v5:
- see point 5 above.

v4:
- small non-functional code cleanup in video-mux.c.
- strip TODO for comparing custom asd's for equivalence.
- add three more convenience functions: v4l2_async_notifier_add_fwnode_subdev,
  v4l2_async_notifier_add_i2c_subdev, v4l2_async_notifier_add_devname_subdev.
- strip support in v4l2_async_register_fwnode_subdev for sub-devices
  that register from port nodes.

v3:
- code optimization in asd_equal(), and remove unneeded braces,
  suggested by Sakari Ailus.
- add a NULL asd pointer check to v4l2_async_notifier_asd_valid().
- fix an error-out path in v4l2_async_register_fwnode_subdev() that
  forgot to put device.

v2:
- don't pass an empty endpoint to the parse_endpoint callback, 
  v4l2_async_notifier_fwnode_parse_endpoint() now just ignores them
  and returns success.
- Fix a couple compile warnings and errors seen in i386 and sh archs.


Steve Longerbeam (17):
  media: v4l2-fwnode: ignore endpoints that have no remote port parent
  media: v4l2: async: Allow searching for asd of any type
  media: v4l2: async: Add v4l2_async_notifier_add_subdev
  media: v4l2: async: Add convenience functions to allocate and add
asd's
  media: v4l2-fwnode: Switch to v4l2_async_notifier_add_subdev
  media: v4l2-fwnode: Add a convenience function for registering subdevs
with notifiers
  media: platform: video-mux: Register a subdev notifier
  media: imx: csi: Register a subdev notifier
  media: imx: mipi csi-2: Register a subdev notifier
  media: staging/imx: of: Remove recursive graph walk
  media: staging/imx: L

[RESEND PATCH v7 00/17] media: imx: Switch to subdev notifiers

[Steve Longerbeam]

This patchset converts the imx-media driver and its dependent
subdevs to use subdev notifiers.

There are a couple shortcomings in v4l2-core that prevented
subdev notifiers from working correctly in imx-media:

1. v4l2_async_notifier_fwnode_parse_endpoint() treats a fwnode
   endpoint that is not connected to a remote device as an error.
   But in the case of the video-mux subdev, this is not an error,
   it is OK if some of the mux inputs have no connection. Also,
   Documentation/devicetree/bindings/media/video-interfaces.txt explicitly
   states that the 'remote-endpoint' property is optional. So the first
   patch is a small modification to ignore empty endpoints in
   v4l2_async_notifier_fwnode_parse_endpoint() and allow
   __v4l2_async_notifier_parse_fwnode_endpoints() to continue to
   parse the remaining port endpoints of the device.

2. In the imx-media graph, multiple subdevs will encounter the same
   upstream subdev (such as the imx6-mipi-csi2 receiver), and so
   v4l2_async_notifier_parse_fwnode_endpoints() will add imx6-mipi-csi2
   multiple times. This is treated as an error by
   v4l2_async_notifier_register() later.

   To get around this problem, add an v4l2_async_notifier_add_subdev()
   which first verifies the provided asd does not already exist in the
   given notifier asd list or in other registered notifiers. If the asd
   exists, the function returns -EEXIST and it's up to the caller to
   decide if that is an error (in imx-media case it is never an error).

   Patches 2-5 deal with adding that support.

3. Patch 6 adds v4l2_async_register_fwnode_subdev(), which is a
   convenience function for parsing a subdev's fwnode port endpoints
   for connected remote subdevs, registering a subdev notifier, and
   then registering the sub-device itself.

4. Patches 7-14 update the subdev drivers to register a subdev notifier
   with endpoint parsing, and the changes to imx-media to support that.

5. Finally, the last 3 patches endeavor to completely remove support for
   the notifier->subdevs[] array in platform drivers and v4l2 core. All
   platform drivers are modified to make use of
   v4l2_async_notifier_add_subdev() and its related convenience functions
   to add asd's to the notifier @asd_list, and any allocation or reference
   to the notifier->subdevs[] array removed. After that large patch,
   notifier->subdevs[] array is stripped from v4l2-async and v4l2-subdev
   docs are updated to reflect the new method of adding asd's to notifiers.


Signed-off-by: Steve Longerbeam 

Patches 07-14 (video-mux and the imx patches) are
Reviewed-by: Philipp Zabel 

Patches 01-14 are
Tested-by: Philipp Zabel 
on i.MX6 with Toshiba TC358743 connected via MIPI CSI-2.

History:

v7:
- Comment, error message, and whitespace fixes in patch 02/17.
  Suggested by Mauro Chehab.
- Fixed merge conflict in drivers/media/platform/ti-vpe/cal.c.
- Make patch author and SoB email addresses the same.

v6:
- Export v4l2_async_notifier_init(), which must be called by all
  drivers before the first call to v4l2_async_notifier_add_subdev().
  Suggested by Sakari Ailus.
- Removed @num_subdevs from struct v4l2_async_notifier, and the macro
  V4L2_MAX_SUBDEVS. Sugested by Sakari.
- Fixed a double device node put in vpif_capture.c. Reported by Sakari.
- Fixed wrong printk format qualifiers in xilinx-vipp.c. Reported by
  Dan Carpenter.

v5:
- see point 5 above.

v4:
- small non-functional code cleanup in video-mux.c.
- strip TODO for comparing custom asd's for equivalence.
- add three more convenience functions: v4l2_async_notifier_add_fwnode_subdev,
  v4l2_async_notifier_add_i2c_subdev, v4l2_async_notifier_add_devname_subdev.
- strip support in v4l2_async_register_fwnode_subdev for sub-devices
  that register from port nodes.

v3:
- code optimization in asd_equal(), and remove unneeded braces,
  suggested by Sakari Ailus.
- add a NULL asd pointer check to v4l2_async_notifier_asd_valid().
- fix an error-out path in v4l2_async_register_fwnode_subdev() that
  forgot to put device.

v2:
- don't pass an empty endpoint to the parse_endpoint callback, 
  v4l2_async_notifier_fwnode_parse_endpoint() now just ignores them
  and returns success.
- Fix a couple compile warnings and errors seen in i386 and sh archs.


Steve Longerbeam (17):
  media: v4l2-fwnode: ignore endpoints that have no remote port parent
  media: v4l2: async: Allow searching for asd of any type
  media: v4l2: async: Add v4l2_async_notifier_add_subdev
  media: v4l2: async: Add convenience functions to allocate and add
asd's
  media: v4l2-fwnode: Switch to v4l2_async_notifier_add_subdev
  media: v4l2-fwnode: Add a convenience function for registering subdevs
with notifiers
  media: platform: video-mux: Register a subdev notifier
  media: imx: csi: Register a subdev notifier
  media: imx: mipi csi-2: Register a subdev notifier
  media: staging/imx: of: Remove recursive graph walk
  media: staging/imx: L

[RESEND PATCH v7 13/17] media: staging/imx: Switch to v4l2_async_notifier_add_*_subdev

[Steve Longerbeam]

Switch to v4l2_async_notifier_add_*_subdev() when adding async subdevs
to the imx-media root notifier. This removes the need to check for
an already added asd, since v4l2_async_notifier_add_*_subdev() does this
check. Also no need to allocate a subdevs array when registering the
root notifier, or keeping an internal master asd_list, since this is
moved to the notifier's asd_list.

Signed-off-by: Steve Longerbeam 
---
Changes since v6:
- none
Changes since v5:
- remove reference to notifier.num_subdevs and call
  v4l2_async_notifier_init(). Suggested by Sakari Ailus.
---
 drivers/staging/media/imx/imx-media-dev.c | 121 +-
 .../staging/media/imx/imx-media-internal-sd.c |   5 +-
 drivers/staging/media/imx/imx-media.h |   4 +-
 3 files changed, 36 insertions(+), 94 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index a9faee4b2495..481840195071 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -33,43 +33,10 @@ static inline struct imx_media_dev *notifier2dev(struct 
v4l2_async_notifier *n)
 }
 
 /*
- * Find an asd by fwnode or device name. This is called during
- * driver load to form the async subdev list and bind them.
- */
-static struct v4l2_async_subdev *
-find_async_subdev(struct imx_media_dev *imxmd,
- struct fwnode_handle *fwnode,
- const char *devname)
-{
-   struct imx_media_async_subdev *imxasd;
-   struct v4l2_async_subdev *asd;
-
-   list_for_each_entry(imxasd, &imxmd->asd_list, list) {
-   asd = &imxasd->asd;
-   switch (asd->match_type) {
-   case V4L2_ASYNC_MATCH_FWNODE:
-   if (fwnode && asd->match.fwnode == fwnode)
-   return asd;
-   break;
-   case V4L2_ASYNC_MATCH_DEVNAME:
-   if (devname && !strcmp(asd->match.device_name,
-  devname))
-   return asd;
-   break;
-   default:
-   break;
-   }
-   }
-
-   return NULL;
-}
-
-
-/*
- * Adds a subdev to the async subdev list. If fwnode is non-NULL, adds
- * the async as a V4L2_ASYNC_MATCH_FWNODE match type, otherwise as
- * a V4L2_ASYNC_MATCH_DEVNAME match type using the dev_name of the
- * given platform_device. This is called during driver load when
+ * Adds a subdev to the root notifier's async subdev list. If fwnode is
+ * non-NULL, adds the async as a V4L2_ASYNC_MATCH_FWNODE match type,
+ * otherwise as a V4L2_ASYNC_MATCH_DEVNAME match type using the dev_name
+ * of the given platform_device. This is called during driver load when
  * forming the async subdev list.
  */
 int imx_media_add_async_subdev(struct imx_media_dev *imxmd,
@@ -80,47 +47,34 @@ int imx_media_add_async_subdev(struct imx_media_dev *imxmd,
struct imx_media_async_subdev *imxasd;
struct v4l2_async_subdev *asd;
const char *devname = NULL;
-   int ret = 0;
-
-   mutex_lock(&imxmd->mutex);
+   int ret;
 
-   if (pdev)
+   if (fwnode) {
+   asd = v4l2_async_notifier_add_fwnode_subdev(
+   &imxmd->notifier, fwnode, sizeof(*imxasd));
+   } else {
devname = dev_name(&pdev->dev);
-
-   /* return -EEXIST if this asd already added */
-   if (find_async_subdev(imxmd, fwnode, devname)) {
-   dev_dbg(imxmd->md.dev, "%s: already added %s\n",
-   __func__, np ? np->name : devname);
-   ret = -EEXIST;
-   goto out;
+   asd = v4l2_async_notifier_add_devname_subdev(
+   &imxmd->notifier, devname, sizeof(*imxasd));
}
 
-   imxasd = devm_kzalloc(imxmd->md.dev, sizeof(*imxasd), GFP_KERNEL);
-   if (!imxasd) {
-   ret = -ENOMEM;
-   goto out;
-   }
-   asd = &imxasd->asd;
-
-   if (fwnode) {
-   asd->match_type = V4L2_ASYNC_MATCH_FWNODE;
-   asd->match.fwnode = fwnode;
-   } else {
-   asd->match_type = V4L2_ASYNC_MATCH_DEVNAME;
-   asd->match.device_name = devname;
-   imxasd->pdev = pdev;
+   if (IS_ERR(asd)) {
+   ret = PTR_ERR(asd);
+   if (ret == -EEXIST)
+   dev_dbg(imxmd->md.dev, "%s: already added %s\n",
+   __func__, np ? np->name : devname);
+   return ret;
}
 
-   list_add_tail(&imxasd->list, &imxmd->asd_list);
+   imxasd = to_imx_media_asd(asd);
 
-   imxmd->notifier.num_subdevs++;
+   if (devname)
+   imxasd->pdev = pdev;
 
dev_dbg(i

[RESEND PATCH v7 01/17] media: v4l2-fwnode: ignore endpoints that have no remote port parent

[Steve Longerbeam]

Documentation/devicetree/bindings/media/video-interfaces.txt states that
the 'remote-endpoint' property is optional.

So v4l2_async_notifier_fwnode_parse_endpoint() should not return error
if the endpoint has no remote port parent. Just ignore the endpoint,
skip adding an asd to the notifier and return 0.
__v4l2_async_notifier_parse_fwnode_endpoints() will then continue
parsing the remaining port endpoints of the device.

Signed-off-by: Steve Longerbeam 
Acked-by: Hans Verkuil 
---
Changes since v6:
- none
Changes since v5:
- none
Changes since v4:
- none
Changes since v3:
- none
Changes since v2:
- none
Changes since v1:
- don't pass an empty endpoint to the parse_endpoint callback,
  v4l2_async_notifier_fwnode_parse_endpoint() now just ignores them
  and returns success. The current users of
  v4l2_async_notifier_parse_fwnode_endpoints() (omap3isp, rcar-vin,
  intel-ipu3) no longer need modification.
---
 drivers/media/v4l2-core/v4l2-fwnode.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-fwnode.c 
b/drivers/media/v4l2-core/v4l2-fwnode.c
index 169bdbb1f61a..0b8c736b1606 100644
--- a/drivers/media/v4l2-core/v4l2-fwnode.c
+++ b/drivers/media/v4l2-core/v4l2-fwnode.c
@@ -367,7 +367,7 @@ static int v4l2_async_notifier_fwnode_parse_endpoint(
fwnode_graph_get_remote_port_parent(endpoint);
if (!asd->match.fwnode) {
dev_warn(dev, "bad remote port parent\n");
-   ret = -EINVAL;
+   ret = -ENOTCONN;
goto out_err;
}
 
-- 
2.17.1

Re: [PATCH v3 00/14] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

Hi Hans,


On 10/01/2018 03:07 AM, Hans Verkuil wrote:

Hi Steve,

On 08/01/2018 09:12 PM, Steve Longerbeam wrote:

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

I reviewed this series and it looks fine to me.


Cool.



It appears that the ipu* patches are already merged, so can you rebase and
repost?


Done. There are still two ipu* patches that still need a merge:

gpu: ipu-csi: Swap fields according to input/output field types
gpu: ipu-v3: Add planar support to interlaced scan

so those will still be included in the v4 submission.



I would also like to see the 'v4l2-compliance -f' for an interlaced source,
if at all possible.


Sure, I've run 'v4l2-compliance -f' on two configured pipelines: unprocessed
capture (no scaling, CSC, rotation using ipu), and a VDIC de-interlace 
pipeline.


I have the text output, the output is huge but here is the abbreviated 
results:


Unprocessed pipeline:

root@mx6q:/home/fu# v4l2-compliance -d4 -f
v4l2-compliance SHA   : 2d35de61ac90b030fe15439809b807014e9751fe

test VIDIOC_G/S/ENUMINPUT: FAIL

test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL


Total: 715, Succeeded: 713, Failed: 2, Warnings: 0


VDIC de-interlace pipeline:

root@mx6q:/home/fu# v4l2-compliance -d1 -f
v4l2-compliance SHA   : 2d35de61ac90b030fe15439809b807014e9751fe

test VIDIOC_G/S/ENUMINPUT: FAIL

test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

test VIDIOC_G/S_PARM: FAIL


Total: 50, Succeeded: 47, Failed: 3, Warnings: 1

I will send you the full output privately.




For that matter, were you able to test all the field formats?


Yes. I've tested on imx6q SabreAuto with the ADV7180 alternate source,
all of the following are tested and produce good video:

ntsc alternate -> interlaced-tb
ntsc alternate -> interlaced-bt
ntsc alternate -> none (VDIC pipeline)
ntsc alternate -> none (VDIC pipeline)

pal alternate -> interlaced-tb
pal alternate -> interlaced-bt
pal alternate -> none (VDIC pipeline)
pal alternate -> none (VDIC pipeline)

Steve





History:
v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
   ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
   Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
   CSI (and PRPENCVF) at the sink and source pads. In v3, this
   has been moved one hop downstream: interweave is now determined
   using field type at source pad, and field type selected at
   capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
   type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
   and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
   Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Philipp Zabel (1):
   gpu: ipu-v3: Allow negative offsets for interlaced scanning

Steve Longerbeam (13):
   media: videodev2.h: Add more field helper macros
   gpu: ipu-csi: Check for field type alternate
   gpu: ipu-csi: Swap fields according to input/output field types
   gpu: ipu-v3: Fix U/V offset macros for planar 4:2:0
   gpu: ipu-v3: Add planar support to interlaced scan
   media: imx: Fix field negotiation
   media: imx-csi: Double crop height for alternate fields at sink
   media: imx: interweave and odd-chroma-row skip are incompatible
   media: imx-csi: Allow skipping odd chroma rows for YVU420
   media: imx: vdic: rely on VDIC for correct field order
   media: imx-csi: Move crop/compose reset after filling default mbus
 fields
   media: imx: Allow interweave with top/bottom lines swapped
   media: imx.rst: Update doc to reflect fixes to interlaced capture

  Documentation/media/v4l-drivers/imx.rst   |  93 ++-
  drivers/gpu/ipu-v3/ipu-cpmem.c|  45 ++-
  drivers/gpu/ipu-v3/ipu-csi.c  | 136 ++---
  drivers/staging/media/imx/imx-ic-prpencvf.c   |  48 ++--
  drivers/staging/media/imx/imx-media-capture.c |  14 +++
  drivers/staging/media/imx/imx-media-csi.c | 166 ++
  drivers/staging/media/imx/imx-media-vdic.c|  12 +-
  include/uapi/linux/videodev2.h|   7 ++
  include/video/imx-ipu-v3.h|   6 +-
  9 files changed, 377 insertions(+), 150 deletions(-)

Re: [PATCH v3 00/14] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

On 10/04/2018 06:41 AM, Hans Verkuil wrote:

On 10/04/18 01:21, Steve Longerbeam wrote:

Hi Hans,


On 10/01/2018 03:07 AM, Hans Verkuil wrote:

Hi Steve,

On 08/01/2018 09:12 PM, Steve Longerbeam wrote:

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

I reviewed this series and it looks fine to me.

Cool.


It appears that the ipu* patches are already merged, so can you rebase and
repost?

Done. There are still two ipu* patches that still need a merge:

gpu: ipu-csi: Swap fields according to input/output field types
gpu: ipu-v3: Add planar support to interlaced scan

so those will still be included in the v4 submission.


I would also like to see the 'v4l2-compliance -f' for an interlaced source,
if at all possible.

Sure, I've run 'v4l2-compliance -f' on two configured pipelines: unprocessed
capture (no scaling, CSC, rotation using ipu), and a VDIC de-interlace
pipeline.

I have the text output, the output is huge but here is the abbreviated
results:

Unprocessed pipeline:

root@mx6q:/home/fu# v4l2-compliance -d4 -f
v4l2-compliance SHA   : 2d35de61ac90b030fe15439809b807014e9751fe

test VIDIOC_G/S/ENUMINPUT: FAIL

test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

This looks like something that should work. Not relevant for this patch
series, but something you should look into.


Yes, I've been meaning to implement (UN)SUBSCRIBE_EVENT/DQEVENT
at the capture interface. I'll send a patch soon.






Total: 715, Succeeded: 713, Failed: 2, Warnings: 0


VDIC de-interlace pipeline:

root@mx6q:/home/fu# v4l2-compliance -d1 -f
v4l2-compliance SHA   : 2d35de61ac90b030fe15439809b807014e9751fe

test VIDIOC_G/S/ENUMINPUT: FAIL

test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: FAIL

test VIDIOC_G/S_PARM: FAIL

Same here: this appears to be an actual bug. But also not related to this
patch series.


It's because the capture interface passes vidioc_[gs]_parm down to its
connected source subdevice as [gs]_frame_interval, which in this case
is PRPVF, which just accepts whatever frame interval is requested. Not
sure why v4l2-compliance reports an error, but perhaps [gs]_frame_interval
should be chained, until it reaches a subdevice that actually cares about
frame intervals (in this case CSI and VDIC), similar to how [gs]_stream is
chained. Anyway something else to look at later.

Steve

[PATCH v4 00/11] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

History:
v4:
- rebased to latest media-tree master branch.
- Make patch author and SoB email addresses the same.

v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
  ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
  Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
  CSI (and PRPENCVF) at the sink and source pads. In v3, this
  has been moved one hop downstream: interweave is now determined
  using field type at source pad, and field type selected at
  capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
  type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
  and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
  Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Steve Longerbeam (11):
  media: videodev2.h: Add more field helper macros
  gpu: ipu-csi: Swap fields according to input/output field types
  gpu: ipu-v3: Add planar support to interlaced scan
  media: imx: Fix field negotiation
  media: imx-csi: Double crop height for alternate fields at sink
  media: imx: interweave and odd-chroma-row skip are incompatible
  media: imx-csi: Allow skipping odd chroma rows for YVU420
  media: imx: vdic: rely on VDIC for correct field order
  media: imx-csi: Move crop/compose reset after filling default mbus
fields
  media: imx: Allow interweave with top/bottom lines swapped
  media: imx.rst: Update doc to reflect fixes to interlaced capture

 Documentation/media/v4l-drivers/imx.rst   |  93 ++
 drivers/gpu/ipu-v3/ipu-cpmem.c|  26 ++-
 drivers/gpu/ipu-v3/ipu-csi.c  | 132 ++
 drivers/staging/media/imx/imx-ic-prpencvf.c   |  48 +++--
 drivers/staging/media/imx/imx-media-capture.c |  14 ++
 drivers/staging/media/imx/imx-media-csi.c | 166 --
 drivers/staging/media/imx/imx-media-vdic.c|  12 +-
 include/uapi/linux/videodev2.h|   7 +
 include/video/imx-ipu-v3.h|   6 +-
 9 files changed, 359 insertions(+), 145 deletions(-)

-- 
2.17.1

Re: [PATCH v4 00/11] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

On 10/04/2018 12:34 PM, Hans Verkuil wrote:

On 10/04/2018 08:53 PM, Steve Longerbeam wrote:

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

History:
v4:
- rebased to latest media-tree master branch.
- Make patch author and SoB email addresses the same.

v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
   ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
   Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
   CSI (and PRPENCVF) at the sink and source pads. In v3, this
   has been moved one hop downstream: interweave is now determined
   using field type at source pad, and field type selected at
   capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
   type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
   and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
   Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Steve Longerbeam (11):
   media: videodev2.h: Add more field helper macros
   gpu: ipu-csi: Swap fields according to input/output field types
   gpu: ipu-v3: Add planar support to interlaced scan

What should I do with these patches? Do they go through us? Or the drm
subsystem (or whoever handles this)?

If it goes through another subsystem, then I can Ack them.


Hi Hans, sorry you are right. Philipp Zabel needs to merge these
to his imx-drm/fixes tree. Then we need to wait for them to filter
over to media-tree. Same old slow process, I wish this were faster,
but that is the drawback of changes that span subsystems.

I will submit the above patches to dri-devel ML. And resubmit this
series once they hit media-tree.

Steve





   media: imx: Fix field negotiation
   media: imx-csi: Double crop height for alternate fields at sink
   media: imx: interweave and odd-chroma-row skip are incompatible
   media: imx-csi: Allow skipping odd chroma rows for YVU420
   media: imx: vdic: rely on VDIC for correct field order
   media: imx-csi: Move crop/compose reset after filling default mbus
 fields
   media: imx: Allow interweave with top/bottom lines swapped
   media: imx.rst: Update doc to reflect fixes to interlaced capture

  Documentation/media/v4l-drivers/imx.rst   |  93 ++
  drivers/gpu/ipu-v3/ipu-cpmem.c|  26 ++-
  drivers/gpu/ipu-v3/ipu-csi.c  | 132 ++
  drivers/staging/media/imx/imx-ic-prpencvf.c   |  48 +++--
  drivers/staging/media/imx/imx-media-capture.c |  14 ++
  drivers/staging/media/imx/imx-media-csi.c | 166 --
  drivers/staging/media/imx/imx-media-vdic.c|  12 +-
  include/uapi/linux/videodev2.h|   7 +
  include/video/imx-ipu-v3.h|   6 +-
  9 files changed, 359 insertions(+), 145 deletions(-)

Re: [PATCH v3 00/16] i.MX media mem2mem scaler

[Steve Longerbeam]

 gpu: ipu-v3: image-convert: prepare for per-tile configuration
   gpu: ipu-v3: image-convert: calculate per-tile resize coefficients
   gpu: ipu-v3: image-convert: reconfigure IC per tile
   gpu: ipu-v3: image-convert: store tile top/left position
   gpu: ipu-v3: image-convert: calculate tile dimensions and offsets
 outside fill_image
   gpu: ipu-v3: image-convert: move tile alignment helpers
   gpu: ipu-v3: image-convert: select optimal seam positions
   gpu: ipu-v3: image-convert: fix debug output for varying tile sizes
   gpu: ipu-v3: image-convert: relax alignment restrictions
   gpu: ipu-v3: image-convert: fix bytesperline adjustment
   gpu: ipu-v3: image-convert: add some ASCII art to the exposition
   gpu: ipu-v3: image-convert: disable double buffering if necessary
   gpu: ipu-v3: image-convert: allow three rows or columns

Steve Longerbeam (1):
   gpu: ipu-cpmem: add WARN_ON_ONCE() for unaligned dma buffers

  drivers/gpu/ipu-v3/ipu-cpmem.c|   6 +
  drivers/gpu/ipu-v3/ipu-ic.c   |  52 +-
  drivers/gpu/ipu-v3/ipu-image-convert.c| 919 +++---
  drivers/staging/media/imx/Kconfig |   1 +
  drivers/staging/media/imx/Makefile|   1 +
  drivers/staging/media/imx/imx-media-dev.c |  11 +
  drivers/staging/media/imx/imx-media-mem2mem.c | 873 +
  drivers/staging/media/imx/imx-media.h |  10 +
  include/video/imx-ipu-v3.h|   6 +
  9 files changed, 1727 insertions(+), 152 deletions(-)
  create mode 100644 drivers/staging/media/imx/imx-media-mem2mem.c

Re: [PATCH v3 10/16] gpu: ipu-v3: image-convert: select optimal seam positions

[Steve Longerbeam]

On 09/18/2018 02:34 AM, Philipp Zabel wrote:



+/*
+ * Tile left edges are required to be aligned to multiples of 8 bytes
+ * by the IDMAC.
+ */
+static inline u32 tile_left_align(const struct ipu_image_pixfmt *fmt)
+{
+   return fmt->planar ? 8 * fmt->uv_width_dec : 64 / fmt->bpp;
+}



As I indicated, shouldn't this be

return fmt->planar ? 8 * fmt->uv_width_dec : 8;

?

Just from a unit analysis perspective, "64 / fmt->bp" has
units of pixels / 8-bytes, it should have units of bytes.

Steve

[PATCH v5 00/12] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

History:
v5:
- Added a regression fix to allow empty endpoints to CSI (fix for imx6q
  SabreAuto).
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
  by Philipp Zabel.
- Fixed a regression in csi_setup(), caught by Philipp.
- Removed interweave_offset and replace with boolean interweave_swap,
  suggested by Philipp.
- Make clear that it is IDMAC channel that does pixel reordering and
  interweave, not the CSI, in the imx.rst doc, caught by Philipp.

v4:
- rebased to latest media-tree master branch.
- Make patch author and SoB email addresses the same.

v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
  ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
  Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
  CSI (and PRPENCVF) at the sink and source pads. In v3, this
  has been moved one hop downstream: interweave is now determined
  using field type at source pad, and field type selected at
  capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
  type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
  and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
  Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Steve Longerbeam (12):
  media: videodev2.h: Add more field helper macros
  gpu: ipu-csi: Swap fields according to input/output field types
  gpu: ipu-v3: Add planar support to interlaced scan
  media: imx: Fix field negotiation
  media: imx-csi: Input connections to CSI should be optional
  media: imx-csi: Double crop height for alternate fields at sink
  media: imx: interweave and odd-chroma-row skip are incompatible
  media: imx-csi: Allow skipping odd chroma rows for YVU420
  media: imx: vdic: rely on VDIC for correct field order
  media: imx-csi: Move crop/compose reset after filling default mbus
fields
  media: imx: Allow interweave with top/bottom lines swapped
  media: imx.rst: Update doc to reflect fixes to interlaced capture

 Documentation/media/v4l-drivers/imx.rst   | 103 +++
 drivers/gpu/ipu-v3/ipu-cpmem.c|  26 ++-
 drivers/gpu/ipu-v3/ipu-csi.c  | 119 +
 drivers/staging/media/imx/imx-ic-prpencvf.c   |  46 +++--
 drivers/staging/media/imx/imx-media-capture.c |  14 ++
 drivers/staging/media/imx/imx-media-csi.c | 168 +-
 drivers/staging/media/imx/imx-media-vdic.c|  12 +-
 include/uapi/linux/videodev2.h|   7 +
 include/video/imx-ipu-v3.h|   6 +-
 9 files changed, 354 insertions(+), 147 deletions(-)

-- 
2.17.1

[PATCH v5 01/12] media: videodev2.h: Add more field helper macros

[Steve Longerbeam]

Adds two helper macros:

V4L2_FIELD_IS_SEQUENTIAL: returns true if the given field type is
'sequential', that is a full frame is transmitted, or exists in
memory, as all top field lines followed by all bottom field lines,
or vice-versa.

V4L2_FIELD_IS_INTERLACED: returns true if the given field type is
'interlaced', that is a full frame is transmitted, or exists in
memory, as top field lines interlaced with bottom field lines.

Signed-off-by: Steve Longerbeam 
---
Changes since v3:
- none
Changes since v2:
- none
Changes since v1:
- add the complement macro V4L2_FIELD_IS_INTERLACED
- remove V4L2_FIELD_ALTERNATE from V4L2_FIELD_IS_SEQUENTIAL macro.
- moved new macros past end of existing V4L2_FIELD_HAS_* macros.
---
 include/uapi/linux/videodev2.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 29729d580452..f7f031736d91 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -130,6 +130,13 @@ enum v4l2_field {
((field) == V4L2_FIELD_BOTTOM ||\
 (field) == V4L2_FIELD_TOP ||\
 (field) == V4L2_FIELD_ALTERNATE)
+#define V4L2_FIELD_IS_INTERLACED(field) \
+   ((field) == V4L2_FIELD_INTERLACED ||\
+(field) == V4L2_FIELD_INTERLACED_TB ||\
+(field) == V4L2_FIELD_INTERLACED_BT)
+#define V4L2_FIELD_IS_SEQUENTIAL(field) \
+   ((field) == V4L2_FIELD_SEQ_TB ||\
+(field) == V4L2_FIELD_SEQ_BT)
 
 enum v4l2_buf_type {
V4L2_BUF_TYPE_VIDEO_CAPTURE= 1,
-- 
2.17.1

Re: i.MX6 IPU CSI analog video input on Ventana

[Steve Longerbeam]

On 10/17/18 4:05 PM, Tim Harvey wrote:

On Wed, Oct 17, 2018 at 2:33 PM Steve Longerbeam  wrote:

Hi Tim,

On 10/17/18 1:38 PM, Tim Harvey wrote:

On Mon, Jun 4, 2018 at 1:58 AM Krzysztof Hałasa  wrote:

I've just tested the PAL setup: in currect situation (v4.17 + Steve's
fix-csi-interlaced.2 + "media: adv7180: fix field type" + a small cheap
PAL camera) the following produces bottom-first interlaced frames:

media-ctl -r -l '"adv7180 2-0020":0->"ipu2_csi1_mux":1[1],
  "ipu2_csi1_mux":2->"ipu2_csi1":0[1],
  "ipu2_csi1":2->"ipu2_csi1 capture":0[1]'

media-ctl -V "'adv7180 2-0020':0 [fmt:UYVY2X8/720x576 field:alternate]"
media-ctl -V "'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x576]"
media-ctl -V "'ipu2_csi1':2 [fmt:AYUV32/720x576 field:interlaced]"

"adv7180 2-0020":0 [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1_mux":1  [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1_mux":2  [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1":0  [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1":2  [fmt:AYUV32/720x576 field:interlaced]

I think it would be great if these changes make their way upstream.
The details could be refined then.

Krzysztof / Steve / Philipp,

I jumped back onto IMX6 video capture from the adv7180 the other day
trying to help out a customer that's using mainline and found things
are still not working right. Where is all of this at these days?

If I use v4.19 with Steves 'imx-media: Fixes for interlaced capture'
v3 series (https://patchwork.kernel.org/cover/10626499/) I
rolling/split (un-synchronized) video using:

# Setup links
media-ctl -r
media-ctl -l '"adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]'
media-ctl -l '"ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]'
media-ctl -l '"ipu2_csi1":1 -> "ipu2_ic_prp":0[1]'
media-ctl -l '"ipu2_ic_prp":2 -> "ipu2_ic_prpvf":0[1]'
media-ctl -l '"ipu2_ic_prpvf":1 -> "ipu2_ic_prpvf capture":0[1]'
# Configure pads
media-ctl -V "'adv7180 2-0020':0 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x480 field:interlaced]"
media-ctl -V "'ipu2_csi1':1 [fmt:UYVY2X8/720x480 field:interlaced]"
media-ctl -V "'ipu2_ic_prp':2 [fmt:UYVY2X8/720x480 field:interlaced]"
media-ctl -V "'ipu2_ic_prpvf':1 [fmt:UYVY2X8/720x480 field:none]"
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=UYVY !
jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device
'/dev/video3' does not support progressive interlacing

I'm doing the above on a Gateworks GW5404 IMXQ which has a tda1997x
HDMI receiver sensor and an adv7180 Analog CVBS sensor - media graph
is here: http://dev.gateworks.com/docs/linux/media/imx6q-gw54xx-media.png

Are there other patches I need or different field formats above with
4.19? Do any of the other kernels work without patchsets that you know
of between 4.16 and 4.19?


First, the v3 series is out of date. Please apply the latest v5 posting
of that series. See the imx.rst doc regarding field type negotiation,
all pads starting at ipu2_csi1:1 should be 'seq-bt' or 'seq-tb' until the
capture device, which should be set to 'interlaced' to enable IDMAC
interweave. The ADV7180 now correctly sets its field type to alternate,
which imx-media-csi.c translates to seq-tb or seq-bt at its output pad.

See the SabreAuto examples in the doc.

For the rolling/split image problem, try the attached somewhat hackish patch.
There used to be code in imx-media-csi.c that searched for the backend sensor
and queries via .g_skip_frames whether the sensor produces bad frames at first
stream-on. But there was push-back on that, so the attached is another
approach that doesn't require searching for a backend sensor.

Steve,

Thanks - I hadn't noticed the updated series. I've built it on top of
linux-media/master and tested with:

- Testing linux-media/master + your v5 now:

# Use simple interweaving
media-ctl -r
# Setup links
media-ctl -l '"adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]'
media-ctl -l '"ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]'
media-ctl -l '"ipu2_csi1":2 -> "ipu2_csi1 capture":0[1]'
# Configure pads
media-ctl -V "'adv7180 2-0020':0 [fmt:UYVY2X8/720x480 field:seq-bt]"
media-ctl -V "'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu2_csi1':1 [fmt:AYUV3

Re: [PATCH v3 10/16] gpu: ipu-v3: image-convert: select optimal seam positions

[Steve Longerbeam]

On 10/17/18 4:10 AM, Philipp Zabel wrote:

On Fri, 2018-10-12 at 17:33 -0700, Steve Longerbeam wrote:

On 09/18/2018 02:34 AM, Philipp Zabel wrote:



+/*
+ * Tile left edges are required to be aligned to multiples of 8 bytes
+ * by the IDMAC.
+ */
+static inline u32 tile_left_align(const struct ipu_image_pixfmt *fmt)
+{
+   return fmt->planar ? 8 * fmt->uv_width_dec : 64 / fmt->bpp;
+}



As I indicated, shouldn't this be

return fmt->planar ? 8 * fmt->uv_width_dec : 8;

?

Just from a unit analysis perspective, "64 / fmt->bp" has
units of pixels / 8-bytes, it should have units of bytes.

The tile alignment is in pixels, not in bytes.



Ah, yes of course you are right, I used to know this :) I am
loosing track of this code.



  For 16-bit and 32-bit
packed formats, we only need to align to 4 or 2 pixels, respectively,
as the LCM of 8-byte alignment and 2-byte or 4-byte pixel size is
always 8 bytes.



Yes I agree, the LCM of 8-byte alignment and bytes-per-pixel should
be the tile left edge alignment in pixels.



But now that you pointed it out, it is quite obvious that this can't
work for 24-bit packed formats. Here the LCM of 8-byte alignment and 3-
byte pixels is 24 bytes, or 8 pixels.

How about:

if (fmt->planar)
return fmt->uv_packed ? 8 : 8 * fmt->uv_width_dec;
else
return fmt->bpp == 32 ? 2 : fmt->bpp == 16 ? 4 : 8;



Yep, that looks better. I tested this and it works fine.

Steve

Re: [PATCH v3 00/16] i.MX media mem2mem scaler

[Steve Longerbeam]

Hi Philipp,

On 10/12/18 5:29 PM, Steve Longerbeam wrote:



But one last thing. Conversions to and from YV12 are producing images
with wrong colors, it looks like the .uv_swapped boolean needs to be 
checked

additionally somewhere. Any ideas?



Sorry, this was my fault. I fixed this in

"gpu: ipu-v3: Add chroma plane offset overrides to ipu_cpmem_set_image()"

in my fork g...@github.com:slongerbeam/mediatree.git, branch imx-mem2mem.3.

Steve

Re: [PATCH v4 00/22] i.MX media mem2mem scaler

[Steve Longerbeam]

Awesome, thanks Philipp.

For the whole series:

Acked-by: Steve Longerbeam 
Tested-by: Steve Longerbeam 
on i.MX6q SabreSD.


On 10/19/18 5:15 AM, Philipp Zabel wrote:

Hi,

this is the fourth version of the i.MX mem2mem scaler series.

An alignment issue with 24-bit RGB formats has been corrected in the
seam position selection patch and a few new fixes by Steve have been
added. If there are no more issues, I'll pick up the ipu-v3 patches
via imx-drm/next. The first patch could be merged via the media tree
independently.

Changes since v3:
  - Fix tile_left_align for 24-bit RGB formats and reduce alignment
restrictions for U/V packed planar YUV formats
  - Catch unaligned tile offsets in image-convert
  - Add chroma plane offset overrides to ipu_cpmem_set_image() to
prevent a false positive warning in some cases
  - Fix a race between run and unprepare and make abort reentrant.


Changes since v2:
  - Rely on ipu_image_convert_adjust() in mem2mem_try_fmt() for format
adjustments. This makes the mem2mem driver mostly a V4L2 mem2mem API
wrapper around the IPU image converter, and independent of the
internal image converter implementation.
  - Remove the source and destination buffers on error in device_run().
Otherwise the conversion is re-attempted apparently over and over
again (with WARN() backtraces).
  - Allow subscribing to control changes.
  - Fix seam position selection for more corner cases:
 - Switch width/height properly and align tile top left positions to 8x8
   IRT block size when rotating.
 - Align input width to input burst length in case the scaling step
   flips horizontally.
 - Fix bottom edge calculation.

Changes since v1:
  - Fix inverted allow_overshoot logic
  - Correctly switch horizontal / vertical tile alignment when
determining seam positions with the 90° rotator active.
  - Fix SPDX-License-Identifier and remove superfluous license
text.
  - Fix uninitialized walign in try_fmt

Previous cover letter:

we have image conversion code for scaling and colorspace conversion in
the IPUv3 base driver for a while. Since the IC hardware can only write
up to 1024x1024 pixel buffers, it scales to larger output buffers by
splitting the input and output frame into similarly sized tiles.

This causes the issue that the bilinear interpolation resets at the tile
boundary: instead of smoothly interpolating across the seam, there is a
jump in the input sample position that is very apparent for high
upscaling factors. This can be avoided by slightly changing the scaling
coefficients to let the left/top tiles overshoot their input sampling
into the first pixel / line of their right / bottom neighbors. The error
can be further reduced by letting tiles be differently sized and by
selecting seam positions that minimize the input sampling position error
at tile boundaries.
This is complicated by different DMA start address, burst size, and
rotator block size alignment requirements, depending on the input and
output pixel formats, and the fact that flipping happens in different
places depending on the rotation.

This series implements optimal seam position selection and seam hiding
with per-tile resizing coefficients and adds a scaling mem2mem device
to the imx-media driver.

regards
Philipp

Philipp Zabel (15):
   media: imx: add mem2mem device
   gpu: ipu-v3: ipu-ic: allow to manually set resize coefficients
   gpu: ipu-v3: image-convert: prepare for per-tile configuration
   gpu: ipu-v3: image-convert: calculate per-tile resize coefficients
   gpu: ipu-v3: image-convert: reconfigure IC per tile
   gpu: ipu-v3: image-convert: store tile top/left position
   gpu: ipu-v3: image-convert: calculate tile dimensions and offsets
 outside fill_image
   gpu: ipu-v3: image-convert: move tile alignment helpers
   gpu: ipu-v3: image-convert: select optimal seam positions
   gpu: ipu-v3: image-convert: fix debug output for varying tile sizes
   gpu: ipu-v3: image-convert: relax alignment restrictions
   gpu: ipu-v3: image-convert: fix bytesperline adjustment
   gpu: ipu-v3: image-convert: add some ASCII art to the exposition
   gpu: ipu-v3: image-convert: disable double buffering if necessary
   gpu: ipu-v3: image-convert: allow three rows or columns

Steve Longerbeam (7):
   gpu: ipu-cpmem: add WARN_ON_ONCE() for unaligned dma buffers
   gpu: ipu-v3: Add chroma plane offset overrides to
 ipu_cpmem_set_image()
   gpu: ipu-v3: image-convert: Prevent race between run and unprepare
   gpu: ipu-v3: image-convert: Only wait for abort completion if active
 run
   gpu: ipu-v3: image-convert: Allow reentrancy into abort
   gpu: ipu-v3: image-convert: Remove need_abort flag
   gpu: ipu-v3: image-convert: Catch unaligned tile offsets

  drivers/gpu/ipu-v3/ipu-cpmem.c|   52 +-
  drivers/gpu/ipu-v3/ipu-ic.c   |   52 +-
  drivers/gpu/ipu-v3/ipu-image-convert.c| 1019 ++---
  drivers/staging/media/imx/Kc

Re: i.MX6 IPU CSI analog video input on Ventana

[Steve Longerbeam]

On 10/18/18 10:56 AM, Tim Harvey wrote:

On Wed, Oct 17, 2018 at 4:37 PM Steve Longerbeam  wrote:


On 10/17/18 4:05 PM, Tim Harvey wrote:

On Wed, Oct 17, 2018 at 2:33 PM Steve Longerbeam  wrote:

Hi Tim,

On 10/17/18 1:38 PM, Tim Harvey wrote:

On Mon, Jun 4, 2018 at 1:58 AM Krzysztof Hałasa  wrote:

I've just tested the PAL setup: in currect situation (v4.17 + Steve's
fix-csi-interlaced.2 + "media: adv7180: fix field type" + a small cheap
PAL camera) the following produces bottom-first interlaced frames:

media-ctl -r -l '"adv7180 2-0020":0->"ipu2_csi1_mux":1[1],
   "ipu2_csi1_mux":2->"ipu2_csi1":0[1],
   "ipu2_csi1":2->"ipu2_csi1 capture":0[1]'

media-ctl -V "'adv7180 2-0020':0 [fmt:UYVY2X8/720x576 field:alternate]"
media-ctl -V "'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x576]"
media-ctl -V "'ipu2_csi1':2 [fmt:AYUV32/720x576 field:interlaced]"

"adv7180 2-0020":0 [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1_mux":1  [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1_mux":2  [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1":0  [fmt:UYVY2X8/720x576 field:alternate]
"ipu2_csi1":2  [fmt:AYUV32/720x576 field:interlaced]

I think it would be great if these changes make their way upstream.
The details could be refined then.

Krzysztof / Steve / Philipp,

I jumped back onto IMX6 video capture from the adv7180 the other day
trying to help out a customer that's using mainline and found things
are still not working right. Where is all of this at these days?

If I use v4.19 with Steves 'imx-media: Fixes for interlaced capture'
v3 series (https://patchwork.kernel.org/cover/10626499/) I
rolling/split (un-synchronized) video using:

# Setup links
media-ctl -r
media-ctl -l '"adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]'
media-ctl -l '"ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]'
media-ctl -l '"ipu2_csi1":1 -> "ipu2_ic_prp":0[1]'
media-ctl -l '"ipu2_ic_prp":2 -> "ipu2_ic_prpvf":0[1]'
media-ctl -l '"ipu2_ic_prpvf":1 -> "ipu2_ic_prpvf capture":0[1]'
# Configure pads
media-ctl -V "'adv7180 2-0020':0 [fmt:UYVY2X8/720x480]"
media-ctl -V "'ipu2_csi1_mux':2 [fmt:UYVY2X8/720x480 field:interlaced]"
media-ctl -V "'ipu2_csi1':1 [fmt:UYVY2X8/720x480 field:interlaced]"
media-ctl -V "'ipu2_ic_prp':2 [fmt:UYVY2X8/720x480 field:interlaced]"
media-ctl -V "'ipu2_ic_prpvf':1 [fmt:UYVY2X8/720x480 field:none]"
# stream JPEG/RTP/UDP
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=UYVY !
jpegenc ! rtpjpegpay ! udpsink host=$SERVER port=$PORT
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device
'/dev/video3' does not support progressive interlacing

I'm doing the above on a Gateworks GW5404 IMXQ which has a tda1997x
HDMI receiver sensor and an adv7180 Analog CVBS sensor - media graph
is here: http://dev.gateworks.com/docs/linux/media/imx6q-gw54xx-media.png

Are there other patches I need or different field formats above with
4.19? Do any of the other kernels work without patchsets that you know
of between 4.16 and 4.19?


First, the v3 series is out of date. Please apply the latest v5 posting
of that series. See the imx.rst doc regarding field type negotiation,
all pads starting at ipu2_csi1:1 should be 'seq-bt' or 'seq-tb' until the
capture device, which should be set to 'interlaced' to enable IDMAC
interweave. The ADV7180 now correctly sets its field type to alternate,
which imx-media-csi.c translates to seq-tb or seq-bt at its output pad.

See the SabreAuto examples in the doc.

For the rolling/split image problem, try the attached somewhat hackish patch.
There used to be code in imx-media-csi.c that searched for the backend sensor
and queries via .g_skip_frames whether the sensor produces bad frames at first
stream-on. But there was push-back on that, so the attached is another
approach that doesn't require searching for a backend sensor.

Steve,

Thanks - I hadn't noticed the updated series. I've built it on top of
linux-media/master and tested with:

- Testing linux-media/master + your v5 now:

# Use simple interweaving
media-ctl -r
# Setup links
media-ctl -l '"adv7180 2-0020":0 -> "ipu2_csi1_mux":1[1]'
media-ctl -l '"ipu2_csi1_mux":2 -> "ipu2_csi1":0[1]'
media-ctl -l '"ipu2_csi1":2 -> "ipu2_csi1 capture":0[1]'
# Configure pads
media-ctl -V "'adv7180 2-0020':0 [fmt:UYVY2X8/720x480 field:seq-bt]"
media-ctl -V "'

Re: [PATCH v3 01/16] media: imx: add mem2mem device

[Steve Longerbeam]

On 10/19/18 2:53 AM, Philipp Zabel wrote:

Hi Tim,

On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote:
[...]

Philipp,

Thanks for submitting this!

I'm hoping this lets us use non-IMX capture devices along with the IMX
media controller entities to so we can use hardware
CSC,scaling,pixel-format-conversions and ultimately coda based encode.

I've built this on top of linux-media and see that it registers as
/dev/video8 but I'm not clear how to use it? I don't see it within the
media controller graph.

It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform
element, for example. GStreamer should create a v4l2video8convert
element of that type.

The mem2mem device is not part of the media controller graph on purpose.
There is no interaction with any of the entities in the media controller
graph apart from the fact that the IC PP task we are using for mem2mem
scaling is sharing hardware resources with the IC PRP tasks used for the
media controller scaler entitites.



It would be nice in the future to link mem2mem output-side to the ipu_vdic:1
pad, to make use of h/w VDIC de-interlace as part of mem2mem operations.
The progressive output from a new ipu_vdic:3 pad can then be sent to the
image_convert APIs by the mem2mem driver for further tiled scaling, CSC,
and rotation by the IC PP task. The ipu_vdic:1 pad makes use of pure 
DMA-based

de-interlace, that is, all input frames (N-1, N, N+1) to the VDIC are sent
from DMA buffers, and this VDIC mode of operation is well understood
and produces clean de-interlace output. The risk is that this would require
iDMAC channel 5 for ipu_vdic:3, which IFAIK is not verified to work yet.


The other problem with that currently is that mem2mem would have to be 
split into
separate device nodes: a /dev/videoN for output-side (linked to 
ipu_vdic:1), and
a /dev/videoM for capture-side (linked from ipu_vdic:3). And then it no 
longer

presents to userspace as a mem2mem device with a single device node for both
output and capture sides.


Or is there another way? I recall work to integrate mem2mem with media 
control.
There is v4l2_m2m_register_media_controller(), but that create three 
entities:

source, processing, and sink. The VDIC entity would be part of mem2mem
processing but this entity already exists for the current graph. This 
function

could however be used as a guide to incorporate the VDIC entity into m2m
device.


Steve

Re: [PATCH v3 01/16] media: imx: add mem2mem device

[Steve Longerbeam]

(resending as plain text)


On 10/21/18 10:43 AM, Philipp Zabel wrote:

On Fri, Oct 19, 2018 at 01:19:10PM -0700, Steve Longerbeam wrote:

On 10/19/18 2:53 AM, Philipp Zabel wrote:

Hi Tim,

On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote:
[...]

Philipp,

Thanks for submitting this!

I'm hoping this lets us use non-IMX capture devices along with the IMX
media controller entities to so we can use hardware
CSC,scaling,pixel-format-conversions and ultimately coda based encode.

I've built this on top of linux-media and see that it registers as
/dev/video8 but I'm not clear how to use it? I don't see it within the
media controller graph.

It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform
element, for example. GStreamer should create a v4l2video8convert
element of that type.

The mem2mem device is not part of the media controller graph on purpose.
There is no interaction with any of the entities in the media controller
graph apart from the fact that the IC PP task we are using for mem2mem
scaling is sharing hardware resources with the IC PRP tasks used for the
media controller scaler entitites.

It would be nice in the future to link mem2mem output-side to the ipu_vdic:1
pad, to make use of h/w VDIC de-interlace as part of mem2mem operations.
The progressive output from a new ipu_vdic:3 pad can then be sent to the
image_convert APIs by the mem2mem driver for further tiled scaling, CSC,
and rotation by the IC PP task. The ipu_vdic:1 pad makes use of pure
DMA-based de-interlace, that is, all input frames (N-1, N, N+1) to the
VDIC are sent from DMA buffers, and this VDIC mode of operation is
well understood and produces clean de-interlace output. The risk is
that this would require iDMAC channel 5 for ipu_vdic:3, which IFAIK is
not verified to work yet.

Tiled mem2mem deinterlacing support would be nice, I'm not sure yet how
though. I'd limit media controller links to marking VDIC as unavailable
for the capture pipeline. The V4L2 subdev API is too lowlevel for tiling
mem2mem purposes, as we'd need to change the subdev format multiple
times per frame.



I wasn't considering tiled deinterlacing, only deinterlacing at the 
hardware limited input frame size to the VDIC.




Also I'd like to keep the option of scheduling tile jobs to both IPUs on
i.MX6Q, which will become difficult to describe via MC, as both IPUs'
ipu_vdics would have to be involved.


Agreed, it would be good to add to the mem2mem driver the ability to 
schedule jobs to whichever IPU is least busy.




The other problem with that currently is that mem2mem would have to be split
into separate device nodes: a /dev/videoN for output-side (linked to
ipu_vdic:1), and a /dev/videoM for capture-side (linked from
ipu_vdic:3). And then it no longer presents to userspace as a mem2mem
device with a single device node for both output and capture sides.

I don't understand why we'd need separate video devices for output and
capture, deinterlacing is still single input single (double rate)
output. As soon as we begin tiling, we are one layer of abstraction
away from the hardware pads anyway. Now if we want to support combining
on the other hand...



Again I wasn't thinking of doing tiled deinterlace.



Or is there another way? I recall work to integrate mem2mem with media
control. There is v4l2_m2m_register_media_controller(), but that
create three
entities:
source, processing, and sink. The VDIC entity would be part of mem2mem
processing but this entity already exists for the current graph. This
function could however be used as a guide to incorporate the VDIC
entity into m2m device.

I'm not sure if this is the right abstraction. Without tiling or
multi-IPU scheduling, sure. But the mem2mem driver does not directly
describe hardware operation anyway.



What I'm thinking is separate mem2mem devices from this proposed 
tiled-scaling post-processing mem2mem, that solely do motion compensated 
deinterlace using the VDIC as the processing entity.


[1] is the dot graph that demonstrates the idea, on imx6q SabreSD. Two 
new mem2mem devices are attached to VDIC sink and source IDMAC pads. 
Also the PP tiled scaling mem2mem is shown in the graph.


As of now, the VDIC is only available for video capture pipelines, so 
the advantage would be to allow the use of hardware deinterlace 
gstreamer pipelines from other types of interlaced sources like file or 
network streams.


So something like the following example which would do hardware 
deinterlace, followed by tiled scaling/CSC/rotation, then h.264 encode, 
the VDIC mem2mem device is at /dev/video0, and the PP mem2mem device is 
at /dev/video12 in this example:


gst-launch-1.0 \
v4l2src ! \
v4l2video0convert output-io-mode=dmabuf-import ! \
v4l2video12convert output-io-mode=dmabuf-import ! \
v4l2h264enc output-io-mode=dmabuf-import ! \
h264parse ! \
matroskamux ! \
filesink

I'm probably missing some 

Re: i.MX6: can't capture on MIPI-CSI2 with DS90UB954

[Steve Longerbeam]

Hi Jean-Michel,

We've done some work with another FPD-Link de-serializer (ds90ux940) and 
IIRC we had some trouble figuring out how to coax the lanes into LP-11 
state. But on the ds90ux940 it can be done by setting bit 7 in the CSI 
Enable Port registers (offsets 0x13 and 0x14). But the "imx6-mipi-csi2: 
clock lane timeout" message is something else and indicates the 
de-serializer is not activating the clock lane during its s_stream(ON) 
subdev call.


Hope that helps,
Steve


On 10/30/18 9:41 AM, Jean-Michel Hautbois wrote:

Hi there,

I am using the i.MX6D from Digi (connect core 6 sbc) with a mailine
kernel (well, 4.14 right now) and have an issue with mipi-csi2
capture.
First I will give brief explanation of my setup, and then I will
detail the issue.
I have a camera sensor (OV2732, but could be any other sensor)
connected on a DS90UB953 FPD-Link III serializer.
Then a coax cable propagates the signal to a DS90UB954 FPD-Link III
deserializer.

The DS90UB954 has the ability to work in a pattern generation mode,
and I will use it for the rest of the discussion.
It is an I²C device, and I have written a basic driver (for the moment
;)) in order to make it visible on the imx6-mipi-csi2 bus as a camera
sensor.
I can give an access to the driver if necessary.

I then program the MC pipeline :
media-ctl -l "'ds90ub954 2-0034':0 -> 'imx6-mipi-csi2':0[1]" -v
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]" -v
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" -v
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]"
media-ctl -V "'ds90ub954 2-0034':0 [fmt:RGB888_1X24/1280x720 field:none]"
media-ctl -V "'imx6-mipi-csi2':1 [fmt:RGB888_1X24/1280x720 field:none]"
media-ctl -V "'ipu1_csi0_mux':2 [fmt:RGB888_1X24/1280x720 field:none]"
media-ctl -V "'ipu1_csi0':2 [fmt:RGB888_1X24/1280x720 field:none]"

Everything works fine, all nodes are correctly configured, and the
DS90UB954 is normaly sending data on 2 lanes, with VC-ID=0.
The pattern is 1280x720@30 RGB888.

Then, I start a Gstreamer pipeline (I tried with v4l2-ctl and have the
same issue though) :
GST_DEBUG="v4l2:5" gst-launch-1.0 v4l2src device=/dev/video4 !
video/x-raw, width=1280, height=720, format=RGB ! fakesink

And... well, I had to use this patch
https://lkml.org/lkml/2017/3/11/270 in order to go further, but I am
finishing into :
[  164.077302] imx-ipuv3-csi imx-ipuv3-csi.0: stream ON
[  164.097955] imx-ipuv3-csi imx-ipuv3-csi.0: FI=3 usec
[  165.129424] ipu1_csi0: EOF timeout
[  165.142395] imx-ipuv3-csi imx-ipuv3-csi.0: stream OFF
[  166.169299] ipu1_csi0: wait last EOF timeout

Sounds like a recurrent issue on this ML :).
I can think of several things which could explain this, but I tried a
lot and am a bit stuck.
The clock is set to 800MHz on DS90UB954 side.
=> Should CSI2_PHY_TST_CTRL1 be 0x32 ? 0x12 ? or 0x4a (ie 400MHz) ?
I think I have tried all but still the same issue.

Maybe this could be a hint, when booting, the first stream-on leads to:
  imx6-mipi-csi2: LP-11 timeout, phy_state = 0x0200 => just a warning now
  imx6-mipi-csi2: clock lane timeout, phy_state = 0x0230
The next one leads to the EOF timeout.

Here is the dts part BTW :
&i2c3 {
 status = "okay";

 ds90ub954: camera@34 {
 compatible = "ti,ds90ub954";
 status = "okay";
 reg = <0x34>;
 clocks = <&clks IMX6QDL_CLK_CKO>;
 clock-names = "xclk";
 port {
 #address-cells = <1>;
 #size-cells = <0>;

 ds90ub954_out0: endpoint {
 remote-endpoint = <&mipi_csi2_in>;
 clock-lanes = <0>;
 data-lanes = <1 2>;
 };
 };
 };
};

&mipi_csi {
 status = "okay";

 port@0 {
 reg = <0>;

 mipi_csi2_in: endpoint {
 remote-endpoint = <&ds90ub954_out0>;
 clock-lanes = <0>;
 data-lanes = <1 2>;
 };
 };
};


If ayone has a suggestion...
Thanks a lot !

Regards,
JM

Re: [RFC] media: imx: queue subdevice events on the video device in the same pipeline

[Steve Longerbeam]

Hi Philipp,

Thanks, I've been meaning this too. Comments below.


On 11/5/18 7:03 AM, Philipp Zabel wrote:

While subdevice and video device are in the same pipeline, pass
subdevice events on to userspace via the video device node.

Signed-off-by: Philipp Zabel 
---
This would allow to see source change events from the source subdevice
on the video device node, for example.
---
  drivers/staging/media/imx/imx-media-dev.c | 18 ++
  1 file changed, 18 insertions(+)

diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index 4b344a4a3706..2fe6fdf2faf1 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -442,6 +442,23 @@ static const struct media_device_ops imx_media_md_ops = {
.link_notify = imx_media_link_notify,
  };
  
+static void imx_media_notify(struct v4l2_subdev *sd, unsigned int notification,

+void *arg)
+{
+   struct imx_media_dev *imxmd;
+   struct imx_media_video_dev *vdev;
+
+   imxmd = container_of(sd->v4l2_dev, struct imx_media_dev, v4l2_dev);
+   list_for_each_entry(vdev, &imxmd->vdev_list, list) {
+   if (sd->entity.pipe &&
+   sd->entity.pipe == vdev->vfd->entity.pipe &&


The problem with this check is that a sensor can be streaming to 
multiple video capture devices simultaneously (to prpenc and prpvf). The 
media framework doesn't support an entity being a member of multiple 
pipelines (there's only one pipe object in 'struct media_entity') so the 
pipe object ends up just pointing to whichever stream was started last 
that included that entity. The result being the event will only get 
queued to whichever video device who's stream was enabled last.


So I think there should be no if statement in the loop. Also it's better 
to loop through the sub-devices vdev_list, because that list contains 
only the video devices reachable from that sub-device. So the function 
should read:


static void imx_media_notify(struct v4l2_subdev *sd,
             unsigned int notification,
             void *arg)
{
    struct media_entity *entity = &sd->entity;
    int i;

    if (notification != V4L2_DEVICE_NOTIFY_EVENT)
        return;

    for (i = 0; i < entity->num_pads; i++) {
        struct media_pad *pad = &entity->pads[i];
        struct imx_media_pad_vdev *pad_vdev;
        struct list_head *pad_vdev_list;

        pad_vdev_list = to_pad_vdev_list(sd, pad->index);
        if (!pad_vdev_list)
            continue;
        list_for_each_entry(pad_vdev, pad_vdev_list, list)
            v4l2_event_queue(pad_vdev->vdev->vfd, arg);
    }
}

I posted this to my media-tree fork, see

da05ccab97 ("media: imx: queue subdevice events to the reachable video devices")

and this Note in the commit header:

Note this will queue the event to a video device even if there is
no actual _enabled_ path from the sub-device to the video device.
So a future fix is to skip the video device if there is no enabled
path to it from the sub-device. The entity->pipe pointer can't be
used for this check because in imx-media a sub-device can be a
member to more than one streaming pipeline at a time.


Steve
 


+   notification == V4L2_DEVICE_NOTIFY_EVENT) {
+   v4l2_event_queue(vdev->vfd, arg);
+   break;
+   }
+   }
+}
+
  static int imx_media_probe(struct platform_device *pdev)
  {
struct device *dev = &pdev->dev;
@@ -464,6 +481,7 @@ static int imx_media_probe(struct platform_device *pdev)
imxmd->v4l2_dev.mdev = &imxmd->md;
strscpy(imxmd->v4l2_dev.name, "imx-media",
sizeof(imxmd->v4l2_dev.name));
+   imxmd->v4l2_dev.notify = imx_media_notify;
  
  	media_device_init(&imxmd->md);
  

Re: [PATCH 2/3] media: imx: set compose rectangle to mbus format

[Steve Longerbeam]

Hi Philipp,


On 11/5/18 7:20 AM, Philipp Zabel wrote:

Prepare for mbus format being smaller than the written rectangle
due to burst size.

Signed-off-by: Philipp Zabel 
---
  drivers/staging/media/imx/imx-media-capture.c | 55 +--
  1 file changed, 38 insertions(+), 17 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index cace8a51aca8..2d49d9573056 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -203,21 +203,14 @@ static int capture_g_fmt_vid_cap(struct file *file, void 
*fh,
return 0;
  }
  
-static int capture_try_fmt_vid_cap(struct file *file, void *fh,

-  struct v4l2_format *f)
+static int __capture_try_fmt_vid_cap(struct capture_priv *priv,
+struct v4l2_subev_format *fmt_src,



typo: struct v4l2_subdev_format *fmt_src,



+struct v4l2_format *f)
  {
struct capture_priv *priv = video_drvdata(file);
-   struct v4l2_subdev_format fmt_src;
const struct imx_media_pixfmt *cc, *cc_src;
-   int ret;
-
-   fmt_src.pad = priv->src_sd_pad;
-   fmt_src.which = V4L2_SUBDEV_FORMAT_ACTIVE;
-   ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL, &fmt_src);
-   if (ret)
-   return ret;
  
-	cc_src = imx_media_find_ipu_format(fmt_src.format.code, CS_SEL_ANY);

+   cc_src = imx_media_find_ipu_format(fmt_src->format.code, CS_SEL_ANY);
if (cc_src) {
u32 fourcc, cs_sel;
  
@@ -231,7 +224,7 @@ static int capture_try_fmt_vid_cap(struct file *file, void *fh,

cc = imx_media_find_format(fourcc, cs_sel, false);
}
} else {
-   cc_src = imx_media_find_mbus_format(fmt_src.format.code,
+   cc_src = imx_media_find_mbus_format(fmt_src->format.code,
CS_SEL_ANY, true);
if (WARN_ON(!cc_src))
return -EINVAL;
@@ -239,15 +232,32 @@ static int capture_try_fmt_vid_cap(struct file *file, 
void *fh,
cc = cc_src;
}
  
-	imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, &fmt_src.format, cc);

+   imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, &fmt_src->format, cc);
  
  	return 0;

  }
  
+static int capture_try_fmt_vid_cap(struct file *file, void *fh,

+  struct v4l2_format *f)
+{
+   struct capture_priv *priv = video_drvdata(file);
+   struct v4l2_subdev_format fmt_src;
+   int ret;
+
+   fmt_src.pad = priv->src_sd_pad;
+   fmt_src.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+   ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL, &fmt_src);
+   if (ret)
+   return ret;
+
+   return __capture_try_fmt(priv, &fmt_src, f);



typo: return __capture_try_fmt_vid_cap(priv, &fmt_src, f);



+}
+
  static int capture_s_fmt_vid_cap(struct file *file, void *fh,
 struct v4l2_format *f)
  {
struct capture_priv *priv = video_drvdata(file);
+   struct v4l2_subdev_format fmt_src;
int ret;
  
  	if (vb2_is_busy(&priv->q)) {

@@ -255,7 +265,13 @@ static int capture_s_fmt_vid_cap(struct file *file, void 
*fh,
return -EBUSY;
}
  
-	ret = capture_try_fmt_vid_cap(file, priv, f);

+   fmt_src.pad = priv->src_sd_pad;
+   fmt_src.which = V4L2_SUBDEV_FORMAT_ACTIVE;
+   ret = v4l2_subdev_call(priv->src_sd, pad, get_fmt, NULL, &fmt_src);
+   if (ret)
+   return ret;
+
+   ret = __capture_try_fmt_vid_cap(priv, &fmt_src, f);
if (ret)
return ret;
  
@@ -264,8 +280,8 @@ static int capture_s_fmt_vid_cap(struct file *file, void *fh,

  CS_SEL_ANY, true);
priv->vdev.compose.left = 0;
priv->vdev.compose.top = 0;
-   priv->vdev.compose.width = f->fmt.fmt.pix.width;
-   priv->vdev.compose.height = f->fmt.fmt.pix.height;
+   priv->vdev.compose.width = fmt_src.width;
+   priv->vdev.compose.height = fmt_src.height;
  
  	return 0;

  }
@@ -307,9 +323,14 @@ static int capture_g_selection(struct file *file, void *fh,
case V4L2_SEL_TGT_COMPOSE:
case V4L2_SEL_TGT_COMPOSE_DEFAULT:
case V4L2_SEL_TGT_COMPOSE_BOUNDS:
-   case V4L2_SEL_TGT_COMPOSE_PADDED:
s->r = priv->vdev.compose;
break;
+   case V4L2_SEL_TGT_COMPOSE_PADDED:
+   s->r.left = 0;
+   s->r.top = 0;
+   s->r.width = priv->vdev.fmt.fmt.pix.width;
+   s->r.height = priv->vdev.fmt.fmt.pix.height;
+   break;
default:
return -EINVAL;
}

Re: [PATCH 1/3] media: imx: add capture compose rectangle

[Steve Longerbeam]

Hi Philipp,

On 11/5/18 7:20 AM, Philipp Zabel wrote:

Allowing to compose captured images into larger memory buffers
will let us lift alignment restrictions on CSI crop width.

Signed-off-by: Philipp Zabel 
---
  drivers/staging/media/imx/imx-ic-prpencvf.c   |  3 +-
  drivers/staging/media/imx/imx-media-capture.c | 38 +++
  drivers/staging/media/imx/imx-media-csi.c |  3 +-
  drivers/staging/media/imx/imx-media-vdic.c|  4 +-
  drivers/staging/media/imx/imx-media.h |  2 +
  5 files changed, 44 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index 28f41caba05d..fe5a77baa592 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -366,8 +366,7 @@ static int prp_setup_channel(struct prp_priv *priv,
  
  	memset(&image, 0, sizeof(image));

image.pix = vdev->fmt.fmt.pix;
-   image.rect.width = image.pix.width;
-   image.rect.height = image.pix.height;
+   image.rect = vdev->compose;
  
  	if (rot_swap_width_height) {

swap(image.pix.width, image.pix.height);
diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index b37e1186eb2f..cace8a51aca8 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -262,6 +262,10 @@ static int capture_s_fmt_vid_cap(struct file *file, void 
*fh,
priv->vdev.fmt.fmt.pix = f->fmt.pix;
priv->vdev.cc = imx_media_find_format(f->fmt.pix.pixelformat,
  CS_SEL_ANY, true);
+   priv->vdev.compose.left = 0;
+   priv->vdev.compose.top = 0;
+   priv->vdev.compose.width = f->fmt.fmt.pix.width;
+   priv->vdev.compose.height = f->fmt.fmt.pix.height;



this should be:

priv->vdev.compose.width = fmt_src.format.width;
priv->vdev.compose.height = fmt_src.format.height;

(corrected in the next patches but needs to be corrected here).

  
  	return 0;

  }
@@ -290,6 +294,35 @@ static int capture_s_std(struct file *file, void *fh, 
v4l2_std_id std)
return v4l2_subdev_call(priv->src_sd, video, s_std, std);
  }
  
+static int capture_g_selection(struct file *file, void *fh,

+  struct v4l2_selection *s)
+{
+   struct capture_priv *priv = video_drvdata(file);
+
+   switch (s->target) {
+   case V4L2_SEL_TGT_CROP:
+   case V4L2_SEL_TGT_CROP_DEFAULT:
+   case V4L2_SEL_TGT_CROP_BOUNDS:
+   case V4L2_SEL_TGT_NATIVE_SIZE:
+   case V4L2_SEL_TGT_COMPOSE:
+   case V4L2_SEL_TGT_COMPOSE_DEFAULT:
+   case V4L2_SEL_TGT_COMPOSE_BOUNDS:
+   case V4L2_SEL_TGT_COMPOSE_PADDED:
+   s->r = priv->vdev.compose;
+   break;
+   default:
+   return -EINVAL;
+   }
+
+   return 0;
+}
+
+static int capture_s_selection(struct file *file, void *fh,
+  struct v4l2_selection *s)
+{
+   return capture_g_selection(file, fh, s);
+}
+
  static int capture_g_parm(struct file *file, void *fh,
  struct v4l2_streamparm *a)
  {
@@ -350,6 +383,9 @@ static const struct v4l2_ioctl_ops capture_ioctl_ops = {
.vidioc_g_std   = capture_g_std,
.vidioc_s_std   = capture_s_std,
  
+	.vidioc_g_selection	= capture_g_selection,

+   .vidioc_s_selection = capture_s_selection,
+
.vidioc_g_parm  = capture_g_parm,
.vidioc_s_parm  = capture_s_parm,
  
@@ -687,6 +723,8 @@ int imx_media_capture_device_register(struct imx_media_video_dev *vdev)

vdev->fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
imx_media_mbus_fmt_to_pix_fmt(&vdev->fmt.fmt.pix,
  &fmt_src.format, NULL);
+   vdev->compose.width = fmt_src.format.width;
+   vdev->compose.height = fmt_src.format.height;
vdev->cc = imx_media_find_format(vdev->fmt.fmt.pix.pixelformat,
 CS_SEL_ANY, false);
  
diff --git a/drivers/staging/media/imx/imx-media-csi.c b/drivers/staging/media/imx/imx-media-csi.c

index 4223f8d418ae..c4523afe7b48 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -413,8 +413,7 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
  
  	memset(&image, 0, sizeof(image));

image.pix = vdev->fmt.fmt.pix;
-   image.rect.width = image.pix.width;
-   image.rect.height = image.pix.height;
+   image.rect = vdev->compose;
  
  	csi_idmac_setup_vb2_buf(priv, phys);
  
diff --git a/drivers/staging/media/imx/imx-media-vdic.c b/drivers/staging/media/imx/imx-media-vdic.c

index 482250d47e7c..e08d296cf4eb 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -263,10 +263,10 @@ static int setup_vdi_channel(struct vdic_priv *priv,
  
  	mem

Re: [PATCH 3/3] media: imx: lift CSI width alignment restriction

[Steve Longerbeam]

On 11/5/18 7:20 AM, Philipp Zabel wrote:

The CSI subdevice shouldn't have to care about IDMAC line start
address alignment. With compose rectangle support in the capture
driver, it doesn't have to anymore.

Signed-off-by: Philipp Zabel 
---
  drivers/staging/media/imx/imx-media-capture.c |  9 -
  drivers/staging/media/imx/imx-media-csi.c |  2 +-
  drivers/staging/media/imx/imx-media-utils.c   | 15 ---
  3 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index 2d49d9573056..f87d6e8019e5 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -204,10 +204,9 @@ static int capture_g_fmt_vid_cap(struct file *file, void 
*fh,
  }
  
  static int __capture_try_fmt_vid_cap(struct capture_priv *priv,

-struct v4l2_subev_format *fmt_src,
+struct v4l2_subdev_format *fmt_src,
 struct v4l2_format *f)
  {
-   struct capture_priv *priv = video_drvdata(file);
const struct imx_media_pixfmt *cc, *cc_src;
  
  	cc_src = imx_media_find_ipu_format(fmt_src->format.code, CS_SEL_ANY);

@@ -250,7 +249,7 @@ static int capture_try_fmt_vid_cap(struct file *file, void 
*fh,
if (ret)
return ret;
  
-	return __capture_try_fmt(priv, &fmt_src, f);

+   return __capture_try_fmt_vid_cap(priv, &fmt_src, f);
  }
  
  static int capture_s_fmt_vid_cap(struct file *file, void *fh,

@@ -280,8 +279,8 @@ static int capture_s_fmt_vid_cap(struct file *file, void 
*fh,
  CS_SEL_ANY, true);
priv->vdev.compose.left = 0;
priv->vdev.compose.top = 0;
-   priv->vdev.compose.width = fmt_src.width;
-   priv->vdev.compose.height = fmt_src.height;
+   priv->vdev.compose.width = fmt_src.format.width;
+   priv->vdev.compose.height = fmt_src.format.height;
  
  	return 0;

  }
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index c4523afe7b48..d39682192a67 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -41,7 +41,7 @@
  #define MIN_H   144
  #define MAX_W  4096
  #define MAX_H  4096
-#define W_ALIGN4 /* multiple of 16 pixels */
+#define W_ALIGN1 /* multiple of 2 pixels */



This works for the IDMAC output pad because the channel's cpmem width 
and stride can be rounded up, but width align at the CSI sink still 
needs to be 8 pixels when directed to the IC via the CSI_SRC_PAD_DIRECT 
pad, in order to support the 8x8 block rotator in the IC PRP, and 
there's no way AFAIK to do the same trick of rounding up width and 
stride for non-IDMAC direct paths through the IPU.


Also, the imx-ic-prpencvf.c W_ALIGN_SRC can be relaxed to 2 pixels as well.

Steve



  #define H_ALIGN1 /* multiple of 2 lines */
  #define S_ALIGN1 /* multiple of 2 */
  
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c

index 0eaa353d5cb3..5f110d90a4ef 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -580,6 +580,7 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format 
*pix,
  struct v4l2_mbus_framefmt *mbus,
  const struct imx_media_pixfmt *cc)
  {
+   u32 width;
u32 stride;
  
  	if (!cc) {

@@ -602,9 +603,16 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format 
*pix,
cc = imx_media_find_mbus_format(code, CS_SEL_YUV, false);
}
  
-	stride = cc->planar ? mbus->width : (mbus->width * cc->bpp) >> 3;

+   /* Round up width for minimum burst size */
+   width = round_up(mbus->width, 8);
  
-	pix->width = mbus->width;

+   /* Round up stride for IDMAC line start address alignment */
+   if (cc->planar)
+   stride = round_up(width, 16);
+   else
+   stride = round_up((width * cc->bpp) >> 3, 8);
+
+   pix->width = width;
pix->height = mbus->height;
pix->pixelformat = cc->fourcc;
pix->colorspace = mbus->colorspace;
@@ -613,7 +621,8 @@ int imx_media_mbus_fmt_to_pix_fmt(struct v4l2_pix_format 
*pix,
pix->quantization = mbus->quantization;
pix->field = mbus->field;
pix->bytesperline = stride;
-   pix->sizeimage = (pix->width * pix->height * cc->bpp) >> 3;
+   pix->sizeimage = cc->planar ? ((stride * pix->height * cc->bpp) >> 3) :
+stride * pix->height;
  
  	return 0;

  }

Re: [PATCH 3/3] media: imx: lift CSI width alignment restriction

[Steve Longerbeam]

On 11/9/18 6:50 AM, Philipp Zabel wrote:

On Thu, 2018-11-08 at 21:46 -0800, Steve Longerbeam wrote:

On 11/5/18 7:20 AM, Philipp Zabel wrote:

The CSI subdevice shouldn't have to care about IDMAC line start
address alignment. With compose rectangle support in the capture
driver, it doesn't have to anymore.

Signed-off-by: Philipp Zabel 
---
   drivers/staging/media/imx/imx-media-capture.c |  9 -
   drivers/staging/media/imx/imx-media-csi.c |  2 +-
   drivers/staging/media/imx/imx-media-utils.c   | 15 ---
   3 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index 2d49d9573056..f87d6e8019e5 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -204,10 +204,9 @@ static int capture_g_fmt_vid_cap(struct file *file, void 
*fh,
   }
   
   static int __capture_try_fmt_vid_cap(struct capture_priv *priv,

-struct v4l2_subev_format *fmt_src,
+struct v4l2_subdev_format *fmt_src,
 struct v4l2_format *f)
   {
-   struct capture_priv *priv = video_drvdata(file);
const struct imx_media_pixfmt *cc, *cc_src;
   
   	cc_src = imx_media_find_ipu_format(fmt_src->format.code, CS_SEL_ANY);

@@ -250,7 +249,7 @@ static int capture_try_fmt_vid_cap(struct file *file, void 
*fh,
if (ret)
return ret;
   
-	return __capture_try_fmt(priv, &fmt_src, f);

+   return __capture_try_fmt_vid_cap(priv, &fmt_src, f);
   }
   
   static int capture_s_fmt_vid_cap(struct file *file, void *fh,

@@ -280,8 +279,8 @@ static int capture_s_fmt_vid_cap(struct file *file, void 
*fh,
  CS_SEL_ANY, true);
priv->vdev.compose.left = 0;
priv->vdev.compose.top = 0;
-   priv->vdev.compose.width = fmt_src.width;
-   priv->vdev.compose.height = fmt_src.height;
+   priv->vdev.compose.width = fmt_src.format.width;
+   priv->vdev.compose.height = fmt_src.format.height;
   
   	return 0;

   }
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index c4523afe7b48..d39682192a67 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -41,7 +41,7 @@
   #define MIN_H   144
   #define MAX_W  4096
   #define MAX_H  4096
-#define W_ALIGN4 /* multiple of 16 pixels */
+#define W_ALIGN1 /* multiple of 2 pixels */


This works for the IDMAC output pad because the channel's cpmem width
and stride can be rounded up, but width align at the CSI sink still
needs to be 8 pixels when directed to the IC via the CSI_SRC_PAD_DIRECT
pad, in order to support the 8x8 block rotator in the IC PRP, and
there's no way AFAIK to do the same trick of rounding up width and
stride for non-IDMAC direct paths through the IPU.

Actually, this is not necessary at all. csi_try_crop takes care of this
by setting:
crop->width &= ~0x7;
Which is then used to set compose rectangle and source pad formats.


Ah you are right, I had forgotten about that line.



So this should be relaxed as well, if the SRC_DIRECT pad is not enabled.


Agreed, crop->width align can be relaxed but only if SRC_DIRECT pad
not enabled.


And further, I think there is no reason to align crop->left to multiples
of 4 pixels?


Honestly, I don't know the reason for that either. Along with crop->top 
this defines the CSI active image frame position (HSC/VSC fields in 
IPU_CSI_OUT_FRM_CTRL register), and I don't see any h/w restrictions on 
HSC in the ref manual. I do remember that this alignment exists in 
FSL/NXP BSP's though, so maybe it is an undocumented restriction.


Steve

Re: 'bad remote port parent' warnings

[Steve Longerbeam]

Hi Fabio,

On 11/22/18 11:17 AM, Fabio Estevam wrote:

Hi Philipp,

On Thu, Nov 22, 2018 at 2:27 PM Philipp Zabel  wrote:


There are empty endpoint nodes (without remote-endpoint property)
labeled ipu1_csi[01]_mux_from_parallel_sensor in the i.MX6 device trees
for board DT implementers' convenience. See commit 2539f517acbdc ("ARM:
dts: imx6qdl: Add video multiplexers, mipi_csi, and their connections").

We had a discussion about this issue in February when this caused a
probing error: https://patchwork.kernel.org/patch/10234469/

Thanks for the clarification.

  We could demote the warning to a debug message, make the wording a bit

less misleading (there is no bad remote port parent, there is just no
remote endpoint at all), or we could just accept the error message for

Something like this?

--- a/drivers/media/v4l2-core/v4l2-fwnode.c
+++ b/drivers/media/v4l2-core/v4l2-fwnode.c
@@ -613,7 +613,7 @@ v4l2_async_notifier_fwnode_parse_endpoint(struct
device *dev,
 asd->match.fwnode =
 fwnode_graph_get_remote_port_parent(endpoint);
 if (!asd->match.fwnode) {
-   dev_warn(dev, "bad remote port parent\n");
+   dev_dbg(dev, "no remote endpoint found\n");
 ret = -ENOTCONN;
 goto out_err;
 }

And how should we treat these error probes?

[3.449564] imx-ipuv3 240.ipu: driver could not parse
port@1/endpoint@0 (-22)
[3.457342] imx-ipuv3-csi: probe of imx-ipuv3-csi.1 failed with error -22
[3.464498] imx-ipuv3 280.ipu: driver could not parse
port@0/endpoint@0 (-22)
[3.472120] imx-ipuv3-csi: probe of imx-ipuv3-csi.4 failed with error -22


Yes, this is a regression caused by the imx subdev notifier patches. 
I've already sent a patch to the list for this, see


https://www.spinics.net/lists/linux-media/msg141809.html

Steve

Re: [PATCH] media: v4l2-fwnode: Demote warning to debug level

[Steve Longerbeam]

Reviewed-by: Steve Longerbeam 


On 11/23/18 4:50 AM, Fabio Estevam wrote:

On a imx6q-wandboard the following warnings are observed:

[4.327794] video-mux 20e.iomuxc-gpr:ipu1_csi0_mux: bad remote port 
parent
[4.336118] video-mux 20e.iomuxc-gpr:ipu2_csi1_mux: bad remote port 
parent

As explained by Philipp Zabel:

"There are empty endpoint nodes (without remote-endpoint property)
labeled ipu1_csi[01]_mux_from_parallel_sensor in the i.MX6 device trees
for board DT implementers' convenience. See commit 2539f517acbdc ("ARM:
dts: imx6qdl: Add video multiplexers, mipi_csi, and their connections")."

So demote the warning to debug level and make the wording a bit
less misleading.

Suggested-by: Philipp Zabel 
Signed-off-by: Fabio Estevam 
---
  drivers/media/v4l2-core/v4l2-fwnode.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-fwnode.c 
b/drivers/media/v4l2-core/v4l2-fwnode.c
index 218f0da..7a3cc10 100644
--- a/drivers/media/v4l2-core/v4l2-fwnode.c
+++ b/drivers/media/v4l2-core/v4l2-fwnode.c
@@ -613,7 +613,7 @@ v4l2_async_notifier_fwnode_parse_endpoint(struct device 
*dev,
asd->match.fwnode =
fwnode_graph_get_remote_port_parent(endpoint);
if (!asd->match.fwnode) {
-   dev_warn(dev, "bad remote port parent\n");
+   dev_dbg(dev, "no remote endpoint found\n");
ret = -ENOTCONN;
goto out_err;
}

Re: [PATCH v5] media: imx: add mem2mem device

[Steve Longerbeam]

Hi Hans, Philipp,

One comment on my side...

On 12/3/18 7:21 AM, Hans Verkuil wrote:



+void imx_media_mem2mem_device_unregister(struct imx_media_video_dev *vdev)
+{
+   struct mem2mem_priv *priv = to_mem2mem_priv(vdev);
+   struct video_device *vfd = priv->vdev.vfd;
+
+   mutex_lock(&priv->mutex);
+
+   if (video_is_registered(vfd)) {
+   video_unregister_device(vfd);
+   media_entity_cleanup(&vfd->entity);

Is this needed?

If this is to be part of the media controller, then I expect to see a call
to v4l2_m2m_register_media_controller() somewhere.



Yes, I agree there should be a call to 
v4l2_m2m_register_media_controller(). This driver does not connect with 
any of the imx-media entities, but calling it will at least make the 
mem2mem output/capture device entities (and processing entity) visible 
in the media graph.


Philipp, can you pick/squash the following from my media-tree github fork?

6fa05f5170 ("media: imx: mem2mem: Add missing media-device header")
d355bf8b15 ("media: imx: Add missing unregister and remove of mem2mem 
device")

6787a50cdc ("media: imx: mem2mem: Register with media control")

Steve

Re: [PATCH v5] media: imx: add mem2mem device

[Steve Longerbeam]

On 12/5/18 10:50 AM, Hans Verkuil wrote:

On 12/05/2018 02:20 AM, Steve Longerbeam wrote:

Hi Hans, Philipp,

One comment on my side...

On 12/3/18 7:21 AM, Hans Verkuil wrote:



+void imx_media_mem2mem_device_unregister(struct imx_media_video_dev *vdev)
+{
+   struct mem2mem_priv *priv = to_mem2mem_priv(vdev);
+   struct video_device *vfd = priv->vdev.vfd;
+
+   mutex_lock(&priv->mutex);
+
+   if (video_is_registered(vfd)) {
+   video_unregister_device(vfd);
+   media_entity_cleanup(&vfd->entity);

Is this needed?

If this is to be part of the media controller, then I expect to see a call
to v4l2_m2m_register_media_controller() somewhere.


Yes, I agree there should be a call to
v4l2_m2m_register_media_controller(). This driver does not connect with
any of the imx-media entities, but calling it will at least make the
mem2mem output/capture device entities (and processing entity) visible
in the media graph.

Philipp, can you pick/squash the following from my media-tree github fork?

6fa05f5170 ("media: imx: mem2mem: Add missing media-device header")
d355bf8b15 ("media: imx: Add missing unregister and remove of mem2mem
device")
6787a50cdc ("media: imx: mem2mem: Register with media control")

Steve


Why is this driver part of the imx driver? Since it doesn't connect with
any of the imx-media entities, doesn't that mean that this is really a
stand-alone driver?


It is basically a stand-alone m2m driver, but it makes use of some 
imx-media utility functions like imx_media_enum_format(). Also making it 
a true stand-alone driver would require creating a second /dev/mediaN 
device.


Steve

Re: [PATCH v5] media: imx: add mem2mem device

[Steve Longerbeam]

Hi Hans,

On 12/6/18 4:32 AM, Hans Verkuil wrote:

On 12/06/18 00:13, Steve Longerbeam wrote:


On 12/5/18 10:50 AM, Hans Verkuil wrote:

On 12/05/2018 02:20 AM, Steve Longerbeam wrote:

Hi Hans, Philipp,

One comment on my side...

On 12/3/18 7:21 AM, Hans Verkuil wrote:



+void imx_media_mem2mem_device_unregister(struct imx_media_video_dev *vdev)
+{
+   struct mem2mem_priv *priv = to_mem2mem_priv(vdev);
+   struct video_device *vfd = priv->vdev.vfd;
+
+   mutex_lock(&priv->mutex);
+
+   if (video_is_registered(vfd)) {
+   video_unregister_device(vfd);
+   media_entity_cleanup(&vfd->entity);

Is this needed?

If this is to be part of the media controller, then I expect to see a call
to v4l2_m2m_register_media_controller() somewhere.


Yes, I agree there should be a call to
v4l2_m2m_register_media_controller(). This driver does not connect with
any of the imx-media entities, but calling it will at least make the
mem2mem output/capture device entities (and processing entity) visible
in the media graph.

Philipp, can you pick/squash the following from my media-tree github fork?

6fa05f5170 ("media: imx: mem2mem: Add missing media-device header")
d355bf8b15 ("media: imx: Add missing unregister and remove of mem2mem
device")
6787a50cdc ("media: imx: mem2mem: Register with media control")

Steve


Why is this driver part of the imx driver? Since it doesn't connect with
any of the imx-media entities, doesn't that mean that this is really a
stand-alone driver?

It is basically a stand-alone m2m driver, but it makes use of some
imx-media utility functions like imx_media_enum_format(). Also making it
a true stand-alone driver would require creating a second /dev/mediaN
device.

If it is standalone, is it reused in newer iMX versions? (7 or 8)


No, this driver makes use of the Image Converter in IPUv3, so it will 
only run on iMX 5/6. The IPU has been dropped in iMX 7 and 8.



And if it is just a regular m2m device, then it doesn't need to create a
media device either (doesn't hurt, but it is not required).


Ok, I'll leave that up to Philipp. I don't mind either way whether it is 
folded into imx-media device, or whether it is made stand-alone with or 
without a new media device.


Steve

Re: [PATCH v5 00/12] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

Hi Hans,

On 12/7/18 5:35 AM, Hans Verkuil wrote:

Hi Steve,

How to proceed with this w.r.t. the two gpu ipu patches? Are those going
in first through the gpu tree? Or do they have to go in through our tree?


There is only one remaining gpu ipu patch in this series that is 
awaiting an ack from Philipp:


"gpu: ipu-csi: Swap fields according to input/output field types"

I pinged him again.

Philipp agreed to allow the two ipu patches in the series to be merged 
to media tree.


Steve



In that case I need Acks from whoever maintains that code.

Regards,

Hans

On 10/17/2018 02:00 AM, Steve Longerbeam wrote:

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

History:
v5:
- Added a regression fix to allow empty endpoints to CSI (fix for imx6q
   SabreAuto).
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
   by Philipp Zabel.
- Fixed a regression in csi_setup(), caught by Philipp.
- Removed interweave_offset and replace with boolean interweave_swap,
   suggested by Philipp.
- Make clear that it is IDMAC channel that does pixel reordering and
   interweave, not the CSI, in the imx.rst doc, caught by Philipp.

v4:
- rebased to latest media-tree master branch.
- Make patch author and SoB email addresses the same.

v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
   ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
   Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
   CSI (and PRPENCVF) at the sink and source pads. In v3, this
   has been moved one hop downstream: interweave is now determined
   using field type at source pad, and field type selected at
   capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
   type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
   and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
   Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Steve Longerbeam (12):
   media: videodev2.h: Add more field helper macros
   gpu: ipu-csi: Swap fields according to input/output field types
   gpu: ipu-v3: Add planar support to interlaced scan
   media: imx: Fix field negotiation
   media: imx-csi: Input connections to CSI should be optional
   media: imx-csi: Double crop height for alternate fields at sink
   media: imx: interweave and odd-chroma-row skip are incompatible
   media: imx-csi: Allow skipping odd chroma rows for YVU420
   media: imx: vdic: rely on VDIC for correct field order
   media: imx-csi: Move crop/compose reset after filling default mbus
 fields
   media: imx: Allow interweave with top/bottom lines swapped
   media: imx.rst: Update doc to reflect fixes to interlaced capture

  Documentation/media/v4l-drivers/imx.rst   | 103 +++
  drivers/gpu/ipu-v3/ipu-cpmem.c|  26 ++-
  drivers/gpu/ipu-v3/ipu-csi.c  | 119 +
  drivers/staging/media/imx/imx-ic-prpencvf.c   |  46 +++--
  drivers/staging/media/imx/imx-media-capture.c |  14 ++
  drivers/staging/media/imx/imx-media-csi.c | 168 +-
  drivers/staging/media/imx/imx-media-vdic.c|  12 +-
  include/uapi/linux/videodev2.h|   7 +
  include/video/imx-ipu-v3.h|   6 +-
  9 files changed, 354 insertions(+), 147 deletions(-)

Re: [PATCH v4 02/11] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

Hi Philipp,


On 10/05/2018 02:44 AM, Philipp Zabel wrote:

Hi Steve,

On Thu, 2018-10-04 at 11:53 -0700, Steve Longerbeam wrote:



+
+   /* framelines for NTSC / PAL */
+   height = (std & V4L2_STD_525_60) ? 525 : 625;

I think this is a bit convoluted. Instead of initializing std, then
possibly changing it, and then comparing to the inital value, and then
checking it again to determine the new height, why not just:

if (width == 720 && height == 480) {
std = V4L2_STD_NTSC;
height = 525;
} else if (width == 720 && height == 576) {
std = V4L2_STD_PAL;
height = 625;
} else {
dev_err(csi->ipu->dev,
"Unsupported interlaced video mode\n");
ret = -EINVAL;
goto out_unlock;
}

?


Yes that was a bit convoluted, fixed.



  
  	/*

 * if cycles is set, we need to handle this over multiple cycles as
 * generic/bayer data
 */
-   if (is_parallel_bus(&priv->upstream_ep) && incc->cycles) {
-   if_fmt.width *= incc->cycles;

If the input format width passed to ipu_csi_init_interface is not
multiplied by the number of cycles per pixel anymore, width in the
CSI_SENS_FRM_SIZE register will be set to the unmultiplied value from
infmt.
This breaks 779680e2e793 ("media: imx: add support for RGB565_2X8 on
parallel bus").


Oops, that was a mistake, thanks for catching, fixed.

Steve

Re: [PATCH v4 03/11] gpu: ipu-v3: Add planar support to interlaced scan

[Steve Longerbeam]

On 10/05/2018 02:48 AM, Philipp Zabel wrote:

On Thu, 2018-10-04 at 11:53 -0700, Steve Longerbeam wrote:

To support interlaced scan with planar formats, cpmem SLUV must
be programmed with the correct chroma line stride. For full and
partial planar 4:2:2 (YUV422P, NV16), chroma line stride must
be doubled. For full and partial planar 4:2:0 (YUV420, YVU420, NV12),
chroma line stride must _not_ be doubled, since a single chroma line
is shared by two luma lines.

Signed-off-by: Steve Longerbeam 
---
  drivers/gpu/ipu-v3/ipu-cpmem.c  | 26 +++--
  drivers/staging/media/imx/imx-ic-prpencvf.c |  3 ++-
  drivers/staging/media/imx/imx-media-csi.c   |  3 ++-
  include/video/imx-ipu-v3.h  |  3 ++-
  4 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-cpmem.c b/drivers/gpu/ipu-v3/ipu-cpmem.c
index a9d2501500a1..d41df8034c5b 100644
--- a/drivers/gpu/ipu-v3/ipu-cpmem.c
+++ b/drivers/gpu/ipu-v3/ipu-cpmem.c
@@ -273,9 +273,10 @@ void ipu_cpmem_set_uv_offset(struct ipuv3_channel *ch, u32 
u_off, u32 v_off)
  }
  EXPORT_SYMBOL_GPL(ipu_cpmem_set_uv_offset);
  
-void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride)

+void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride,
+  u32 pixelformat)
  {
-   u32 ilo, sly;
+   u32 ilo, sly, sluv;
  
  	if (stride < 0) {

stride = -stride;
@@ -286,9 +287,30 @@ void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, 
int stride)
  
  	sly = (stride * 2) - 1;
  
+	switch (pixelformat) {

+   case V4L2_PIX_FMT_YUV420:
+   case V4L2_PIX_FMT_YVU420:
+   sluv = stride / 2 - 1;
+   break;
+   case V4L2_PIX_FMT_NV12:
+   sluv = stride - 1;
+   break;
+   case V4L2_PIX_FMT_YUV422P:
+   sluv = stride - 1;
+   break;
+   case V4L2_PIX_FMT_NV16:
+   sluv = stride * 2 - 1;
+   break;
+   default:
+   sluv = 0;
+   break;
+   }
+
ipu_ch_param_write_field(ch, IPU_FIELD_SO, 1);
ipu_ch_param_write_field(ch, IPU_FIELD_ILO, ilo);
ipu_ch_param_write_field(ch, IPU_FIELD_SLY, sly);
+   if (sluv)
+   ipu_ch_param_write_field(ch, IPU_FIELD_SLUV, sluv);
  };
  EXPORT_SYMBOL_GPL(ipu_cpmem_interlaced_scan);

[...]

Reviewed-by: Philipp Zabel 

and

Acked-by: Philipp Zabel 

to be merged with the rest of the series via the media tree. I'll take
care not to introduce nontrivial conflicts in imx-drm.


Ok thanks.

Hans, for v5 I will just include the two IPU patches as before. As Philipp
stated, he is OK with merging them to the media tree (after his ack of
course), along with the rest of the patches in this series.

Steve

Re: [PATCH v4 10/11] media: imx: Allow interweave with top/bottom lines swapped

[Steve Longerbeam]

Hi Philipp,


On 10/05/2018 03:43 AM, Philipp Zabel wrote:

Hi Steve,

On Thu, 2018-10-04 at 11:54 -0700, Steve Longerbeam wrote:

Allow sequential->interlaced interweaving but with top/bottom
lines swapped to the output buffer.

This can be accomplished by adding one line length to IDMAC output
channel address, with a negative line length for the interlace offset.

This is to allow the seq-bt -> interlaced-bt transformation, where
bottom lines are still dominant (older in time) but with top lines
first in the interweaved output buffer.

With this support, the CSI can now allow seq-bt at its source pads,
e.g. the following transformations are allowed in CSI from sink to
source:

seq-tb -> seq-bt
seq-bt -> seq-bt
alternate -> seq-bt

Suggested-by: Philipp Zabel 
Signed-off-by: Steve Longerbeam 
---
  drivers/staging/media/imx/imx-ic-prpencvf.c | 17 +++-
  drivers/staging/media/imx/imx-media-csi.c   | 46 +
  2 files changed, 53 insertions(+), 10 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index cf76b0432371..1499b0c62d74 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -106,6 +106,8 @@ struct prp_priv {
u32 frame_sequence; /* frame sequence counter */
bool last_eof;  /* waiting for last EOF at stream off */
bool nfb4eof;/* NFB4EOF encountered during streaming */
+   u32 interweave_offset; /* interweave line offset to swap
+ top/bottom lines */

We have to store this instead of using vdev->fmt.fmt.bytesperline
because this potentially is the pre-rotation stride instead?


interweave_offset was used by prp_vb2_buf_done() below, but in fact
that function is passed the non-rotation IDMAC channel (priv->out_ch)
_only_ if rotation is not enabled, so it is actually safe to use
vdev->fmt.fmt.bytesperline for the interweave offset in
prp_vb2_buf_done().

So I've gotten rid of interweave_offset in both imx-ic-prpencvf.c and
imx-media-csi.c, and replaced with a boolean interweave_swap as you
suggested. I agree it is much cleaner.
  



diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 679295da5dde..592f7d6edec1 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -114,6 +114,8 @@ struct csi_priv {
u32 frame_sequence; /* frame sequence counter */
bool last_eof;   /* waiting for last EOF at stream off */
bool nfb4eof;/* NFB4EOF encountered during streaming */
+   u32 interweave_offset; /* interweave line offset to swap
+ top/bottom lines */

This doesn't seem necessary. Since there is no rotation here, the offset
is just vdev->fmt.fmt.pix.bytesperline if interweave_swap is enabled.
Maybe turn this into a bool interweave_swap?


Agreed.




struct completion last_eof_comp;
  };
  
@@ -286,7 +288,8 @@ static void csi_vb2_buf_done(struct csi_priv *priv)

if (ipu_idmac_buffer_is_ready(priv->idmac_ch, priv->ipu_buf_num))
ipu_idmac_clear_buffer(priv->idmac_ch, priv->ipu_buf_num);
  
-	ipu_cpmem_set_buffer(priv->idmac_ch, priv->ipu_buf_num, phys);

+   ipu_cpmem_set_buffer(priv->idmac_ch, priv->ipu_buf_num,
+phys + priv->interweave_offset);
  }
  
  static irqreturn_t csi_idmac_eof_interrupt(int irq, void *dev_id)

@@ -396,10 +399,10 @@ static void csi_idmac_unsetup_vb2_buf(struct csi_priv 
*priv,
  static int csi_idmac_setup_channel(struct csi_priv *priv)
  {
struct imx_media_video_dev *vdev = priv->vdev;
+   bool passthrough, interweave, interweave_swap;
const struct imx_media_pixfmt *incc;
struct v4l2_mbus_framefmt *infmt;
struct v4l2_mbus_framefmt *outfmt;
-   bool passthrough, interweave;
struct ipu_image image;
u32 passthrough_bits;
u32 passthrough_cycles;
@@ -433,6 +436,8 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 */
interweave = V4L2_FIELD_IS_INTERLACED(image.pix.field) &&
V4L2_FIELD_IS_SEQUENTIAL(outfmt->field);
+   interweave_swap = interweave &&
+   image.pix.field == V4L2_FIELD_INTERLACED_BT;

Although this could just as well be recalculated in csi_vb2_buf_done.


In the future yes, when we add support for alternate mode (I assume
that's what you are getting at?).



Apart from that,

Reviewed-by: Philipp Zabel 


Thanks.

Steve

Re: [PATCH v4 11/11] media: imx.rst: Update doc to reflect fixes to interlaced capture

[Steve Longerbeam]

On 10/05/2018 03:52 AM, Philipp Zabel wrote:

Hi Steve,

On Thu, 2018-10-04 at 11:54 -0700, Steve Longerbeam wrote:


+or bottom-top or alternate, and the capture interface field type is set
+to interlaced (t-b, b-t, or unqualified interlaced). The capture interface
+will enforce the same field order if the source pad field type is seq-bt
+or seq-tb. However if the source pad's field type is alternate, any
+interlaced type at the capture interface will be accepted.

This part is fine, though, as are the following changes. I'd just like
to avoid giving the wrong impression that the CSI does line interweaving
or pixel reordering into the output pixel format.


Agreed, I made the wording more clear.

Steve

[PATCH v5 02/12] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

The function ipu_csi_init_interface() was inverting the F-bit for
NTSC case, in the CCIR_CODE_1/2 registers. The result being that
for NTSC bottom-top field order, the CSI would swap fields and
capture in top-bottom order.

Instead, base field swap on the field order of the input to the CSI,
and the field order of the requested output. If the input/output
fields are sequential but different, swap fields, otherwise do
not swap. This requires passing both the input and output mbus
frame formats to ipu_csi_init_interface().

Move this code to a new private function ipu_csi_set_bt_interlaced_codes()
that programs the CCIR_CODE_1/2 registers for interlaced BT.656 (and
possibly interlaced BT.1120 in the future).

When detecting input video standard from the input frame width/height,
make sure to double height if input field type is alternate, since
in that case input height only includes lines for one field.

Signed-off-by: Steve Longerbeam 
---
Changes since v4:
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
  by Philipp Zabel.
- Fixed a regression in csi_setup(), caught by Philipp.
---
 drivers/gpu/ipu-v3/ipu-csi.c  | 119 +++---
 drivers/staging/media/imx/imx-media-csi.c |  17 +---
 include/video/imx-ipu-v3.h|   3 +-
 3 files changed, 88 insertions(+), 51 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index aa0e30a2ba18..4a15e513fa05 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -325,6 +325,15 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config 
*cfg, u32 mbus_code,
return 0;
 }
 
+/* translate alternate field mode based on given standard */
+static inline enum v4l2_field
+ipu_csi_translate_field(enum v4l2_field field, v4l2_std_id std)
+{
+   return (field != V4L2_FIELD_ALTERNATE) ? field :
+   ((std & V4L2_STD_525_60) ?
+V4L2_FIELD_SEQ_BT : V4L2_FIELD_SEQ_TB);
+}
+
 /*
  * Fill a CSI bus config struct from mbus_config and mbus_framefmt.
  */
@@ -374,22 +383,75 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config 
*csicfg,
return 0;
 }
 
+static int ipu_csi_set_bt_interlaced_codes(struct ipu_csi *csi,
+  struct v4l2_mbus_framefmt *infmt,
+  struct v4l2_mbus_framefmt *outfmt,
+  v4l2_std_id std)
+{
+   enum v4l2_field infield, outfield;
+   bool swap_fields;
+
+   /* get translated field type of input and output */
+   infield = ipu_csi_translate_field(infmt->field, std);
+   outfield = ipu_csi_translate_field(outfmt->field, std);
+
+   /*
+* Write the H-V-F codes the CSI will match against the
+* incoming data for start/end of active and blanking
+* field intervals. If input and output field types are
+* sequential but not the same (one is SEQ_BT and the other
+* is SEQ_TB), swap the F-bit so that the CSI will capture
+* field 1 lines before field 0 lines.
+*/
+   swap_fields = (V4L2_FIELD_IS_SEQUENTIAL(infield) &&
+  V4L2_FIELD_IS_SEQUENTIAL(outfield) &&
+  infield != outfield);
+
+   if (!swap_fields) {
+   /*
+* Field0BlankEnd  = 110, Field0BlankStart  = 010
+* Field0ActiveEnd = 100, Field0ActiveStart = 000
+* Field1BlankEnd  = 111, Field1BlankStart  = 011
+* Field1ActiveEnd = 101, Field1ActiveStart = 001
+*/
+   ipu_csi_write(csi, 0x40596 | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0xD07DF, CSI_CCIR_CODE_2);
+   } else {
+   dev_dbg(csi->ipu->dev, "capture field swap\n");
+
+   /* same as above but with F-bit inverted */
+   ipu_csi_write(csi, 0xD07DF | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0x40596, CSI_CCIR_CODE_2);
+   }
+
+   ipu_csi_write(csi, 0xFF, CSI_CCIR_CODE_3);
+
+   return 0;
+}
+
+
 int ipu_csi_init_interface(struct ipu_csi *csi,
   struct v4l2_mbus_config *mbus_cfg,
-  struct v4l2_mbus_framefmt *mbus_fmt)
+  struct v4l2_mbus_framefmt *infmt,
+  struct v4l2_mbus_framefmt *outfmt)
 {
struct ipu_csi_bus_config cfg;
unsigned long flags;
u32 width, height, data = 0;
+   v4l2_std_id std;
int ret;
 
-   ret = fill_csi_bus_cfg(&cfg, mbus_cfg, mbus_fmt);
+   ret = fill_csi_bus_cfg(&cfg, mbus_cfg, infmt);
if (ret < 0)
return ret;
 
/* set default sensor frame width and height */
-   width = mbus_fmt->width;
-   height = mbus_fmt->height;
+

[PATCH v5 07/12] media: imx: interweave and odd-chroma-row skip are incompatible

[Steve Longerbeam]

If IDMAC interweaving is enabled in a write channel, the channel must
write the odd chroma rows for 4:2:0 formats. Skipping writing the odd
chroma rows produces corrupted captured 4:2:0 images when interweave
is enabled.

Reported-by: Krzysztof Hałasa 
Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-ic-prpencvf.c | 9 +++--
 drivers/staging/media/imx/imx-media-csi.c   | 8 ++--
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index 1a03d4c9d7b8..cf76b0432371 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -391,12 +391,17 @@ static int prp_setup_channel(struct prp_priv *priv,
image.phys0 = addr0;
image.phys1 = addr1;
 
-   if (channel == priv->out_ch || channel == priv->rot_out_ch) {
+   /*
+* Skip writing U and V components to odd rows in the output
+* channels for planar 4:2:0 (but not when enabling IDMAC
+* interweaving, they are incompatible).
+*/
+   if (!interweave && (channel == priv->out_ch ||
+   channel == priv->rot_out_ch)) {
switch (image.pix.pixelformat) {
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_YVU420:
case V4L2_PIX_FMT_NV12:
-   /* Skip writing U and V components to odd rows */
ipu_cpmem_skip_odd_chroma_rows(channel);
break;
}
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index d5b0f8a66750..7e648fc9626a 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -457,8 +457,12 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 ((image.pix.width & 0x1f) ?
  ((image.pix.width & 0xf) ? 8 : 16) : 32) : 64;
passthrough_bits = 16;
-   /* Skip writing U and V components to odd rows */
-   ipu_cpmem_skip_odd_chroma_rows(priv->idmac_ch);
+   /*
+* Skip writing U and V components to odd rows (but not
+* when enabling IDMAC interweaving, they are incompatible).
+*/
+   if (!interweave)
+   ipu_cpmem_skip_odd_chroma_rows(priv->idmac_ch);
break;
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
-- 
2.17.1

[PATCH v5 11/12] media: imx: Allow interweave with top/bottom lines swapped

[Steve Longerbeam]

Allow sequential->interlaced interweaving but with top/bottom
lines swapped to the output buffer.

This can be accomplished by adding one line length to IDMAC output
channel address, with a negative line length for the interlace offset.

This is to allow the seq-bt -> interlaced-bt transformation, where
bottom lines are still dominant (older in time) but with top lines
first in the interweaved output buffer.

With this support, the CSI can now allow seq-bt at its source pads,
e.g. the following transformations are allowed in CSI from sink to
source:

seq-tb -> seq-bt
seq-bt -> seq-bt
alternate -> seq-bt

Suggested-by: Philipp Zabel 
Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
Changes since v4:
- Removed interweave_offset and replace with boolean interweave_swap,
  suggested by Philipp Zabel.
---
 drivers/staging/media/imx/imx-ic-prpencvf.c | 25 +
 drivers/staging/media/imx/imx-media-csi.c   | 40 ++---
 2 files changed, 54 insertions(+), 11 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index cf76b0432371..33ada6612fee 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -106,6 +106,7 @@ struct prp_priv {
u32 frame_sequence; /* frame sequence counter */
bool last_eof;  /* waiting for last EOF at stream off */
bool nfb4eof;/* NFB4EOF encountered during streaming */
+   bool interweave_swap; /* swap top/bottom lines when interweaving */
struct completion last_eof_comp;
 };
 
@@ -235,6 +236,9 @@ static void prp_vb2_buf_done(struct prp_priv *priv, struct 
ipuv3_channel *ch)
if (ipu_idmac_buffer_is_ready(ch, priv->ipu_buf_num))
ipu_idmac_clear_buffer(ch, priv->ipu_buf_num);
 
+   if (priv->interweave_swap && ch == priv->out_ch)
+   phys += vdev->fmt.fmt.pix.bytesperline;
+
ipu_cpmem_set_buffer(ch, priv->ipu_buf_num, phys);
 }
 
@@ -376,8 +380,9 @@ static int prp_setup_channel(struct prp_priv *priv,
 * the IDMAC output channel.
 */
interweave = V4L2_FIELD_IS_INTERLACED(image.pix.field) &&
-   V4L2_FIELD_IS_SEQUENTIAL(outfmt->field) &&
-   channel == priv->out_ch;
+   V4L2_FIELD_IS_SEQUENTIAL(outfmt->field);
+   priv->interweave_swap = interweave &&
+   image.pix.field == V4L2_FIELD_INTERLACED_BT;
 
if (rot_swap_width_height) {
swap(image.pix.width, image.pix.height);
@@ -388,6 +393,11 @@ static int prp_setup_channel(struct prp_priv *priv,
(image.pix.width * outcc->bpp) >> 3;
}
 
+   if (priv->interweave_swap && channel == priv->out_ch) {
+   /* start interweave scan at 1st top line (2nd line) */
+   image.rect.top = 1;
+   }
+
image.phys0 = addr0;
image.phys1 = addr1;
 
@@ -396,8 +406,8 @@ static int prp_setup_channel(struct prp_priv *priv,
 * channels for planar 4:2:0 (but not when enabling IDMAC
 * interweaving, they are incompatible).
 */
-   if (!interweave && (channel == priv->out_ch ||
-   channel == priv->rot_out_ch)) {
+   if ((channel == priv->out_ch && !interweave) ||
+   channel == priv->rot_out_ch) {
switch (image.pix.pixelformat) {
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_YVU420:
@@ -424,8 +434,11 @@ static int prp_setup_channel(struct prp_priv *priv,
if (rot_mode)
ipu_cpmem_set_rotation(channel, rot_mode);
 
-   if (interweave)
-   ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline,
+   if (interweave && channel == priv->out_ch)
+   ipu_cpmem_interlaced_scan(channel,
+ priv->interweave_swap ?
+ -image.pix.bytesperline :
+ image.pix.bytesperline,
  image.pix.pixelformat);
 
ret = ipu_ic_task_idma_init(priv->ic, channel,
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 0d494a7db211..73c9f3ae4221 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -114,6 +114,7 @@ struct csi_priv {
u32 frame_sequence; /* frame sequence counter */
bool last_eof;   /* waiting for last EOF at stream off */
bool nfb4eof;/* NFB4EOF encountered during streaming */
+   bool interweave_swap; /* swap top/bottom lines when interweaving */
struct completion last_eof_comp;
 };
 
@@ -286,6 +287,9 @@ static void csi_vb2_buf_done(struct csi_priv *priv)
 

[PATCH v5 10/12] media: imx-csi: Move crop/compose reset after filling default mbus fields

[Steve Longerbeam]

If caller passes un-initialized field type V4L2_FIELD_ANY to CSI
sink pad, the reset CSI crop window would not be correct, because
the crop window depends on a valid input field type. To fix move
the reset of crop and compose windows to after the call to
imx_media_fill_default_mbus_fields().

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-media-csi.c | 27 ---
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index c9110dd39a49..0d494a7db211 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1407,19 +1407,6 @@ static void csi_try_fmt(struct csi_priv *priv,
  W_ALIGN, &sdformat->format.height,
  MIN_H, MAX_H, H_ALIGN, S_ALIGN);
 
-   /* Reset crop and compose rectangles */
-   crop->left = 0;
-   crop->top = 0;
-   crop->width = sdformat->format.width;
-   crop->height = sdformat->format.height;
-   if (sdformat->format.field == V4L2_FIELD_ALTERNATE)
-   crop->height *= 2;
-   csi_try_crop(priv, crop, cfg, &sdformat->format, upstream_ep);
-   compose->left = 0;
-   compose->top = 0;
-   compose->width = crop->width;
-   compose->height = crop->height;
-
*cc = imx_media_find_mbus_format(sdformat->format.code,
 CS_SEL_ANY, true);
if (!*cc) {
@@ -1435,6 +1422,20 @@ static void csi_try_fmt(struct csi_priv *priv,
imx_media_fill_default_mbus_fields(
&sdformat->format, infmt,
priv->active_output_pad == CSI_SRC_PAD_DIRECT);
+
+   /* Reset crop and compose rectangles */
+   crop->left = 0;
+   crop->top = 0;
+   crop->width = sdformat->format.width;
+   crop->height = sdformat->format.height;
+   if (sdformat->format.field == V4L2_FIELD_ALTERNATE)
+   crop->height *= 2;
+   csi_try_crop(priv, crop, cfg, &sdformat->format, upstream_ep);
+   compose->left = 0;
+   compose->top = 0;
+   compose->width = crop->width;
+   compose->height = crop->height;
+
break;
}
 }
-- 
2.17.1

[PATCH v5 06/12] media: imx-csi: Double crop height for alternate fields at sink

[Steve Longerbeam]

If the incoming sink field type is alternate, the reset crop height
and crop height bounds must be set to twice the incoming height,
because in alternate field mode, upstream will report only the
lines for a single field, and the CSI captures the whole frame.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-media-csi.c | 20 +++-
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 8f52428d2c75..d5b0f8a66750 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1140,6 +1140,8 @@ static void csi_try_crop(struct csi_priv *priv,
 struct v4l2_mbus_framefmt *infmt,
 struct v4l2_fwnode_endpoint *upstream_ep)
 {
+   u32 in_height;
+
crop->width = min_t(__u32, infmt->width, crop->width);
if (crop->left + crop->width > infmt->width)
crop->left = infmt->width - crop->width;
@@ -1147,6 +1149,10 @@ static void csi_try_crop(struct csi_priv *priv,
crop->left &= ~0x3;
crop->width &= ~0x7;
 
+   in_height = infmt->height;
+   if (infmt->field == V4L2_FIELD_ALTERNATE)
+   in_height *= 2;
+
/*
 * FIXME: not sure why yet, but on interlaced bt.656,
 * changing the vertical cropping causes loss of vertical
@@ -1156,12 +1162,12 @@ static void csi_try_crop(struct csi_priv *priv,
if (upstream_ep->bus_type == V4L2_MBUS_BT656 &&
(V4L2_FIELD_HAS_BOTH(infmt->field) ||
 infmt->field == V4L2_FIELD_ALTERNATE)) {
-   crop->height = infmt->height;
-   crop->top = (infmt->height == 480) ? 2 : 0;
+   crop->height = in_height;
+   crop->top = (in_height == 480) ? 2 : 0;
} else {
-   crop->height = min_t(__u32, infmt->height, crop->height);
-   if (crop->top + crop->height > infmt->height)
-   crop->top = infmt->height - crop->height;
+   crop->height = min_t(__u32, in_height, crop->height);
+   if (crop->top + crop->height > in_height)
+   crop->top = in_height - crop->height;
}
 }
 
@@ -1401,6 +1407,8 @@ static void csi_try_fmt(struct csi_priv *priv,
crop->top = 0;
crop->width = sdformat->format.width;
crop->height = sdformat->format.height;
+   if (sdformat->format.field == V4L2_FIELD_ALTERNATE)
+   crop->height *= 2;
csi_try_crop(priv, crop, cfg, &sdformat->format, upstream_ep);
compose->left = 0;
compose->top = 0;
@@ -1528,6 +1536,8 @@ static int csi_get_selection(struct v4l2_subdev *sd,
sel->r.top = 0;
sel->r.width = infmt->width;
sel->r.height = infmt->height;
+   if (infmt->field == V4L2_FIELD_ALTERNATE)
+   sel->r.height *= 2;
break;
case V4L2_SEL_TGT_CROP:
sel->r = *crop;
-- 
2.17.1

[PATCH v5 09/12] media: imx: vdic: rely on VDIC for correct field order

[Steve Longerbeam]

prepare_vdi_in_buffers() was setting up the dma pointers as if the
VDIC is always programmed to receive the fields in bottom-top order,
i.e. as if ipu_vdi_set_field_order() only programs BT order in the VDIC.
But that's not true, ipu_vdi_set_field_order() is working correctly.

So fix prepare_vdi_in_buffers() to give the VDIC the fields in whatever
order they were received by the video source, and rely on the VDIC to
sort out which is top and which is bottom.

Signed-off-by: Steve Longerbeam 
---
 drivers/staging/media/imx/imx-media-vdic.c | 12 ++--
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-vdic.c 
b/drivers/staging/media/imx/imx-media-vdic.c
index 482250d47e7c..4a890714193e 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -219,26 +219,18 @@ static void __maybe_unused prepare_vdi_in_buffers(struct 
vdic_priv *priv,
 
switch (priv->fieldtype) {
case V4L2_FIELD_SEQ_TB:
-   prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0);
-   curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + fs;
-   next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
-   break;
case V4L2_FIELD_SEQ_BT:
prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0) + fs;
curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + fs;
break;
+   case V4L2_FIELD_INTERLACED_TB:
case V4L2_FIELD_INTERLACED_BT:
+   case V4L2_FIELD_INTERLACED:
prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0) + is;
curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + is;
break;
-   default:
-   /* assume V4L2_FIELD_INTERLACED_TB */
-   prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0);
-   curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + is;
-   next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
-   break;
}
 
ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
-- 
2.17.1

[PATCH v5 12/12] media: imx.rst: Update doc to reflect fixes to interlaced capture

[Steve Longerbeam]

Also add an example pipeline for unconverted capture with interweave
on SabreAuto.

Cleanup some language in various places in the process.

Signed-off-by: Steve Longerbeam 
---
Changes since v4:
- Make clear that it is IDMAC channel that does pixel reordering and
  interweave, not the CSI. Caught by Philipp Zabel.
Changes since v3:
- none.
Changes since v2:
- expand on idmac interweave behavior in CSI subdev.
- switch second SabreAuto pipeline example to PAL to give
  both NTSC and PAL examples.
- Cleanup some language in various places.
---
 Documentation/media/v4l-drivers/imx.rst | 103 +++-
 1 file changed, 66 insertions(+), 37 deletions(-)

diff --git a/Documentation/media/v4l-drivers/imx.rst 
b/Documentation/media/v4l-drivers/imx.rst
index 65d3d15eb159..45f3b68865dd 100644
--- a/Documentation/media/v4l-drivers/imx.rst
+++ b/Documentation/media/v4l-drivers/imx.rst
@@ -22,8 +22,8 @@ memory. Various dedicated DMA channels exist for both video 
capture and
 display paths. During transfer, the IDMAC is also capable of vertical
 image flip, 8x8 block transfer (see IRT description), pixel component
 re-ordering (for example UYVY to YUYV) within the same colorspace, and
-even packed <--> planar conversion. It can also perform a simple
-de-interlacing by interleaving even and odd lines during transfer
+packed <--> planar conversion. The IDMAC can also perform a simple
+de-interlacing by interweaving even and odd lines during transfer
 (without motion compensation which requires the VDIC).
 
 The CSI is the backend capture unit that interfaces directly with
@@ -173,15 +173,21 @@ via the SMFC and an IDMAC channel, bypassing IC 
pre-processing. This
 source pad is routed to a capture device node, with a node name of the
 format "ipuX_csiY capture".
 
-Note that since the IDMAC source pad makes use of an IDMAC channel, it
-can do pixel reordering within the same colorspace. For example, the
-sink pad can take UYVY2X8, but the IDMAC source pad can output YUYV2X8.
-If the sink pad is receiving YUV, the output at the capture device can
-also be converted to a planar YUV format such as YUV420.
-
-It will also perform simple de-interlace without motion compensation,
-which is activated if the sink pad's field type is an interlaced type,
-and the IDMAC source pad field type is set to none.
+Note that since the IDMAC source pad makes use of an IDMAC channel,
+pixel reordering within the same colorspace can be carried out by the
+IDMAC channel. For example, if the CSI sink pad is receiving in UYVY
+order, the capture device linked to the IDMAC source pad can capture
+in YUYV order. Also, if the CSI sink pad is receiving a packed YUV
+format, the capture device can capture a planar YUV format such as
+YUV420.
+
+The IDMAC channel at the IDMAC source pad also supports simple
+interweave without motion compensation, which is activated if the source
+pad's field type is sequential top-bottom or bottom-top, and the
+requested capture interface field type is set to interlaced (t-b, b-t,
+or unqualified interlaced). The capture interface will enforce the same
+field order as the source pad field order (interlaced-bt if source pad
+is seq-bt, interlaced-tb if source pad is seq-tb).
 
 This subdev can generate the following event when enabling the second
 IDMAC source pad:
@@ -323,14 +329,14 @@ ipuX_vdic
 
 The VDIC carries out motion compensated de-interlacing, with three
 motion compensation modes: low, medium, and high motion. The mode is
-specified with the menu control V4L2_CID_DEINTERLACING_MODE. It has
-two sink pads and a single source pad.
+specified with the menu control V4L2_CID_DEINTERLACING_MODE. The VDIC
+has two sink pads and a single source pad.
 
 The direct sink pad receives from an ipuX_csiY direct pad. With this
 link the VDIC can only operate in high motion mode.
 
 When the IDMAC sink pad is activated, it receives from an output
-or mem2mem device node. With this pipeline, it can also operate
+or mem2mem device node. With this pipeline, the VDIC can also operate
 in low and medium modes, because these modes require receiving
 frames from memory buffers. Note that an output or mem2mem device
 is not implemented yet, so this sink pad currently has no links.
@@ -343,8 +349,8 @@ ipuX_ic_prp
 This is the IC pre-processing entity. It acts as a router, routing
 data from its sink pad to one or both of its source pads.
 
-It has a single sink pad. The sink pad can receive from the ipuX_csiY
-direct pad, or from ipuX_vdic.
+This entity has a single sink pad. The sink pad can receive from the
+ipuX_csiY direct pad, or from ipuX_vdic.
 
 This entity has two source pads. One source pad routes to the
 pre-process encode task entity (ipuX_ic_prpenc), the other to the
@@ -367,8 +373,8 @@ color-space conversion, resizing (downscaling and 
upscaling),
 horizontal and vertical flip, and 90/270 degree rotation. Flip
 and rotation are provided via standard V4L2 controls.
 

[PATCH v5 05/12] media: imx-csi: Input connections to CSI should be optional

[Steve Longerbeam]

Some imx platforms do not have fwnode connections to all CSI input
ports, and should not be treated as an error. This includes the
imx6q SabreAuto, which has no connections to ipu1_csi1 and ipu2_csi0.
Return -ENOTCONN in imx_csi_parse_endpoint() so that v4l2-fwnode
endpoint parsing will not treat an unconnected endpoint as an error.

Fixes: c893500a16baf ("media: imx: csi: Register a subdev notifier")

Signed-off-by: Steve Longerbeam 
---
 drivers/staging/media/imx/imx-media-csi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 176978c7dfe7..8f52428d2c75 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1813,7 +1813,7 @@ static int imx_csi_parse_endpoint(struct device *dev,
  struct v4l2_fwnode_endpoint *vep,
  struct v4l2_async_subdev *asd)
 {
-   return fwnode_device_is_available(asd->match.fwnode) ? 0 : -EINVAL;
+   return fwnode_device_is_available(asd->match.fwnode) ? 0 : -ENOTCONN;
 }
 
 static int imx_csi_async_register(struct csi_priv *priv)
-- 
2.17.1

[PATCH v5 04/12] media: imx: Fix field negotiation

[Steve Longerbeam]

IDMAC interlaced scan, a.k.a. interweave, should be enabled in the
IDMAC output channels only if the IDMAC output pad field type is
'seq-bt' or 'seq-tb', and field type at the capture interface is
'interlaced*'.

V4L2_FIELD_HAS_BOTH() macro should not be used on the input to determine
enabling interlaced/interweave scan. That macro includes the 'interlaced'
field types, and in those cases the data is already interweaved with
top/bottom field lines.

The CSI will capture whole frames when the source specifies alternate
field mode. So the CSI also enables interweave for alternate input
field type and the field type at capture interface is interlaced.

Fix the logic for setting field type in try_fmt in CSI entity.
The behavior should be:

- No restrictions on field type at sink pad.

- At the output pads, allow sequential fields in TB order, if the sink pad
  field type is sequential or alternate. Otherwise passthrough the field
  type from sink to source pad.

Move this logic to new function csi_try_field().

These changes result in the following allowed field transformations
from CSI sink -> source pads (all other field types at sink are passed
through to source):

seq-tb -> seq-tb
seq-bt -> seq-tb
alternate -> seq-tb

In a future patch, the CSI sink -> source will allow:

seq-tb -> seq-bt
seq-bt -> seq-bt
alternate -> seq-bt

This will require supporting interweave with top/bottom line swapping.
Until then seq-bt is not allowed at the CSI source pad because there is
no way to swap top/bottom lines when interweaving to INTERLACED_BT --
note that despite the name, INTERLACED_BT is top-bottom order in memory.
The BT in this case refers to field dominance: the bottom lines are
older in time than the top lines.

The capture interface device allows selecting IDMAC interweave by
choosing INTERLACED_TB if the CSI/PRPENCVF source pad is seq-tb and
INTERLACED_BT if the source pad is seq-bt (for future support of seq-bt).

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-ic-prpencvf.c   | 21 --
 drivers/staging/media/imx/imx-media-capture.c | 14 
 drivers/staging/media/imx/imx-media-csi.c | 64 ++-
 3 files changed, 76 insertions(+), 23 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index af7224846bd5..1a03d4c9d7b8 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -354,12 +354,13 @@ static int prp_setup_channel(struct prp_priv *priv,
 {
struct imx_media_video_dev *vdev = priv->vdev;
const struct imx_media_pixfmt *outcc;
-   struct v4l2_mbus_framefmt *infmt;
+   struct v4l2_mbus_framefmt *outfmt;
unsigned int burst_size;
struct ipu_image image;
+   bool interweave;
int ret;
 
-   infmt = &priv->format_mbus[PRPENCVF_SINK_PAD];
+   outfmt = &priv->format_mbus[PRPENCVF_SRC_PAD];
outcc = vdev->cc;
 
ipu_cpmem_zero(channel);
@@ -369,6 +370,15 @@ static int prp_setup_channel(struct prp_priv *priv,
image.rect.width = image.pix.width;
image.rect.height = image.pix.height;
 
+   /*
+* If the field type at capture interface is interlaced, and
+* the output IDMAC pad is sequential, enable interweave at
+* the IDMAC output channel.
+*/
+   interweave = V4L2_FIELD_IS_INTERLACED(image.pix.field) &&
+   V4L2_FIELD_IS_SEQUENTIAL(outfmt->field) &&
+   channel == priv->out_ch;
+
if (rot_swap_width_height) {
swap(image.pix.width, image.pix.height);
swap(image.rect.width, image.rect.height);
@@ -409,9 +419,7 @@ static int prp_setup_channel(struct prp_priv *priv,
if (rot_mode)
ipu_cpmem_set_rotation(channel, rot_mode);
 
-   if (image.pix.field == V4L2_FIELD_NONE &&
-   V4L2_FIELD_HAS_BOTH(infmt->field) &&
-   channel == priv->out_ch)
+   if (interweave)
ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline,
  image.pix.pixelformat);
 
@@ -839,8 +847,7 @@ static void prp_try_fmt(struct prp_priv *priv,
infmt = __prp_get_fmt(priv, cfg, PRPENCVF_SINK_PAD, sdformat->which);
 
if (sdformat->pad == PRPENCVF_SRC_PAD) {
-   if (sdformat->format.field != V4L2_FIELD_NONE)
-   sdformat->format.field = infmt->field;
+   sdformat->format.field = infmt->field;
 
prp_bound_align_output(&sdformat->format, infmt,
   priv->rot_mode);
diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index b37e1186eb2f..01ec9443de55 100644
--- a/d

[PATCH v5 08/12] media: imx-csi: Allow skipping odd chroma rows for YVU420

[Steve Longerbeam]

Skip writing U/V components to odd rows for YVU420 in addition to
YUV420 and NV12.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-media-csi.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 7e648fc9626a..c9110dd39a49 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -452,6 +452,7 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
passthrough_bits = 16;
break;
case V4L2_PIX_FMT_YUV420:
+   case V4L2_PIX_FMT_YVU420:
case V4L2_PIX_FMT_NV12:
burst_size = (image.pix.width & 0x3f) ?
 ((image.pix.width & 0x1f) ?
-- 
2.17.1

[PATCH v5 03/12] gpu: ipu-v3: Add planar support to interlaced scan

[Steve Longerbeam]

To support interlaced scan with planar formats, cpmem SLUV must
be programmed with the correct chroma line stride. For full and
partial planar 4:2:2 (YUV422P, NV16), chroma line stride must
be doubled. For full and partial planar 4:2:0 (YUV420, YVU420, NV12),
chroma line stride must _not_ be doubled, since a single chroma line
is shared by two luma lines.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
Acked-by: Philipp Zabel 
---
 drivers/gpu/ipu-v3/ipu-cpmem.c  | 26 +++--
 drivers/staging/media/imx/imx-ic-prpencvf.c |  3 ++-
 drivers/staging/media/imx/imx-media-csi.c   |  3 ++-
 include/video/imx-ipu-v3.h  |  3 ++-
 4 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-cpmem.c b/drivers/gpu/ipu-v3/ipu-cpmem.c
index a9d2501500a1..d41df8034c5b 100644
--- a/drivers/gpu/ipu-v3/ipu-cpmem.c
+++ b/drivers/gpu/ipu-v3/ipu-cpmem.c
@@ -273,9 +273,10 @@ void ipu_cpmem_set_uv_offset(struct ipuv3_channel *ch, u32 
u_off, u32 v_off)
 }
 EXPORT_SYMBOL_GPL(ipu_cpmem_set_uv_offset);
 
-void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride)
+void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride,
+  u32 pixelformat)
 {
-   u32 ilo, sly;
+   u32 ilo, sly, sluv;
 
if (stride < 0) {
stride = -stride;
@@ -286,9 +287,30 @@ void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, 
int stride)
 
sly = (stride * 2) - 1;
 
+   switch (pixelformat) {
+   case V4L2_PIX_FMT_YUV420:
+   case V4L2_PIX_FMT_YVU420:
+   sluv = stride / 2 - 1;
+   break;
+   case V4L2_PIX_FMT_NV12:
+   sluv = stride - 1;
+   break;
+   case V4L2_PIX_FMT_YUV422P:
+   sluv = stride - 1;
+   break;
+   case V4L2_PIX_FMT_NV16:
+   sluv = stride * 2 - 1;
+   break;
+   default:
+   sluv = 0;
+   break;
+   }
+
ipu_ch_param_write_field(ch, IPU_FIELD_SO, 1);
ipu_ch_param_write_field(ch, IPU_FIELD_ILO, ilo);
ipu_ch_param_write_field(ch, IPU_FIELD_SLY, sly);
+   if (sluv)
+   ipu_ch_param_write_field(ch, IPU_FIELD_SLUV, sluv);
 };
 EXPORT_SYMBOL_GPL(ipu_cpmem_interlaced_scan);
 
diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index 28f41caba05d..af7224846bd5 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -412,7 +412,8 @@ static int prp_setup_channel(struct prp_priv *priv,
if (image.pix.field == V4L2_FIELD_NONE &&
V4L2_FIELD_HAS_BOTH(infmt->field) &&
channel == priv->out_ch)
-   ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+   ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline,
+ image.pix.pixelformat);
 
ret = ipu_ic_task_idma_init(priv->ic, channel,
image.pix.width, image.pix.height,
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 7ecbd4d76d09..4aa20ae72608 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -512,7 +512,8 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
if (image.pix.field == V4L2_FIELD_NONE &&
V4L2_FIELD_HAS_BOTH(infmt->field))
ipu_cpmem_interlaced_scan(priv->idmac_ch,
- image.pix.bytesperline);
+ image.pix.bytesperline,
+ image.pix.pixelformat);
 
ipu_idmac_set_double_buffer(priv->idmac_ch, true);
 
diff --git a/include/video/imx-ipu-v3.h b/include/video/imx-ipu-v3.h
index f44a35192313..e888c66b9d9d 100644
--- a/include/video/imx-ipu-v3.h
+++ b/include/video/imx-ipu-v3.h
@@ -255,7 +255,8 @@ void ipu_cpmem_set_stride(struct ipuv3_channel *ch, int 
stride);
 void ipu_cpmem_set_high_priority(struct ipuv3_channel *ch);
 void ipu_cpmem_set_buffer(struct ipuv3_channel *ch, int bufnum, dma_addr_t 
buf);
 void ipu_cpmem_set_uv_offset(struct ipuv3_channel *ch, u32 u_off, u32 v_off);
-void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride);
+void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride,
+  u32 pixelformat);
 void ipu_cpmem_set_axi_id(struct ipuv3_channel *ch, u32 id);
 int ipu_cpmem_get_burstsize(struct ipuv3_channel *ch);
 void ipu_cpmem_set_burstsize(struct ipuv3_channel *ch, int burstsize);
-- 
2.17.1

Re: i.MX6 MIPI-CSI2 OV5640 Camera testing on Mainline Linux

[Steve Longerbeam]

Hi Adam,


On 10/16/18 12:46 PM, Adam Ford wrote:

On Thu, Sep 20, 2018 at 9:58 AM jacopo mondi  wrote:

Hi imx6 people,

On Thu, May 31, 2018 at 08:39:20PM +0530, Jagan Teki wrote:

Hi All,

I'm trying to verify MIPI-CSI2 OV5640 camera on i.MX6 platform with
Mainline Linux.

Sorry to resurect this, but before diving deep into details, do anyone
of you verified JPEG capture with ov5640 and i.MX6 platforms, and has
maybe a pipeline configuration to share :) ?


I have a 4.14 kernel for my i.MX6D/Q using an ov5640 connected in a
similar way as the sabresd and I'm getting similar timeouts.
when executing

media-ctl -l "'ov5640 2-0010':0 -> 'imx6-mipi-csi2':0[1]"
media-ctl -l "'imx6-mipi-csi2':2 -> 'ipu1_csi1':0[1]"



You're routing through imx6-mipi-csi2 pad 2, which is CSI-2 virtual
channel 1, so make sure the ov5640 is transmitting on that channel,
see virtual_channel module parameter.



media-ctl -l "'ipu1_csi1':1 -> 'ipu1_ic_prp':0[1]"
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]"
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]"


media-ctl -V "'ov5640 2-0010':0 [fmt:UYVY2X8/640x480 field:none]"
media-ctl -V "'imx6-mipi-csi2':2 [fmt:UYVY2X8/640x480 field:none]"
media-ctl -V "'ipu1_csi1':1 [fmt:AYUV32/640x480 field:none]"
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480 field:none]"
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:AYUV32/640x480 field:none]"


   gst-launch-1.0 -v v4l2src num-buffers=1 device=/dev/video0 ! jpegenc
! filesink location=test.jpg

[   72.799015] ipu1_ic_prpenc: EOF timeout
[   73.838985] ipu1_ic_prpenc: wait last EOF timeout

When I try to jump directly to 4.19-RC8, I get errors regarding memory
allocation, so I think there might be something else there I am
missing.

Has anyone tried this camera module on a 4.14 kernel?  I noticed there
are a bunch of driver updates, and I was hoping there might be some
patches that could be be backported to the 4.14.y stable branch.


I would suggest backporting all the ov5640 commits. You can also
backport the imx-media commits, but that shouldn't be the cause
of the timeouts you are seeing.


Steve





thanks for any suggestions to try.

adam


Thanks
j


I've followed these[1] instructions to configure MC links and pads
based on the probing details from dmesg and trying to capture
ipu1_ic_prpenc capture (/dev/video1) but it's not working.

Can anyone help me to verify whether I configured all the details
properly if not please suggest.

I'm pasting full log here, so-that anyone can comment in line and dt
changes are at [2]

Log:
-

[1.211866] etnaviv-gpu 2204000.gpu: Ignoring GPU with VG and FE2.0
[1.220211] [drm] Initialized etnaviv 1.2.0 20151214 for etnaviv on minor 0
[1.230344] imx-ipuv3 240.ipu: IPUv3H probed
[1.237170] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[1.243920] [drm] No driver support for vblank timestamp query.
[1.250831] imx-drm display-subsystem: bound imx-ipuv3-crtc.2 (ops
ipu_crtc_ops)
[1.258503] imx-drm display-subsystem: bound imx-ipuv3-crtc.3 (ops
ipu_crtc_ops)
[1.266293] imx-drm display-subsystem: bound imx-ipuv3-crtc.6 (ops
ipu_crtc_ops)
[1.274027] imx-drm display-subsystem: bound imx-ipuv3-crtc.7 (ops
ipu_crtc_ops)
[1.282304] dwhdmi-imx 12.hdmi: Detected HDMI TX controller
v1.30a with HDCP (DWC HDMI 3D TX PHY)
[1.295722] imx-drm display-subsystem: bound 12.hdmi (ops
dw_hdmi_imx_ops)
[1.373615] Console: switching to colour frame buffer device 128x48
[1.396495] imx-drm display-subsystem: fb0:  frame buffer device
[1.404620] [drm] Initialized imx-drm 1.0.0 20120507 for
display-subsystem on minor 1
[1.412763] imx-ipuv3 280.ipu: IPUv3H probed
[1.439673] brd: module loaded
[1.469099] loop: module loaded
[1.480324] nand: No NAND device found
[1.487768] libphy: Fixed MDIO Bus: probed
[1.493034] CAN device driver interface
[1.499057] fec 2188000.ethernet: 2188000.ethernet supply phy not
found, using dummy regulator
[1.511633] pps pps0: new PPS source ptp0
[1.516928] fec 2188000.ethernet (unnamed net_device)
(uninitialized): Invalid MAC address: 00:00:00:00:00:00
[1.527177] fec 2188000.ethernet (unnamed net_device)
(uninitialized): Using random MAC address: f2:5a:6d:a6:90:74
[1.543567] libphy: fec_enet_mii_bus: probed
[1.549138] fec 2188000.ethernet eth0: registered PHC device 0
[1.556499] usbcore: registered new interface driver asix
[1.562066] usbcore: registered new interface driver ax88179_178a
[1.568259] usbcore: registered new interface driver cdc_ether
[1.574276] usbcore: registered new interface driver net1080
[1.580097] usbcore: registered new interface driver cdc_subset
[1.586144] usbcore: registered new interface driver zaurus
[1.591910] usbcore: registered new interface driver cdc_ncm
[1.597589] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[1.604209] ehci-pci: EHCI PCI platform driver
[1.6087

Re: i.MX6 MIPI-CSI2 OV5640 Camera testing on Mainline Linux

[Steve Longerbeam]

Hi Adam,

On 10/23/18 8:19 AM, Adam Ford wrote:

On Mon, Oct 22, 2018 at 7:40 AM Fabio Estevam  wrote:

Hi Adam,

On Mon, Oct 22, 2018 at 9:37 AM Adam Ford  wrote:


Thank you!  This tutorial web site is exactly what I need.  The
documentation page in Linux touched on the media-ctl links, but it
didn't explain the syntax or the mapping.  This graphical
interpretation really helps it make more sense.

Is capturing working well on your i.MX6 board now?

Fabio,

Unfortunately, no.  I built the rootfs based on Jagan's instructions
at https://openedev.amarulasolutions.com/display/ODWIKI/i.CoreM6+1.5

I tried building both the 4.15-RC6 kernel, a 4.19 kernel and a 4.14 LTS kernel.

Using the suggested method of generating the graphical display of the
pipeline options, I am able to enable various pipeline options
connecting different /dev/videoX options tot he camera.  I have tried
both the  suggested method above as well as the instructions found in
Documentation/media/v4l-drivers/imx.rst for their respective kernels,
and I have tried multiple options to capture through
ipu1_csi1_capture, ipu2_csi1_capture, and ip1_ic_prepenc capture, and
all yield a broken pipe.

libv4l2: error turning on stream: Broken pipe
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could
not read from resource.
Additional debug info:
gstv4l2bufferpool.c(1064): gst_v4l2_buffer_pool_poll ():
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
poll error 1: Broken pipe (32)

I can hear the camera click when I start gstreamer and click again
when it stops trying to stream.

dmesg indicates a broken pipe as well..

[ 2419.851502] ipu2_csi1: pipeline start failed with -32

might you have any suggestions?



This -EPIPE error might mean you have a mis-match of resolution, pixel 
format, or field type between one of the source->sink pad links. You can 
find out which pads have a mis-match by enabling dynamic debug in the 
kernel function __media_pipeline_start.


Also make sure you are attempting to stream from the correct /dev/videoN.

Steve

Re: i.MX6 MIPI-CSI2 OV5640 Camera testing on Mainline Linux

[Steve Longerbeam]

On 10/23/18 10:34 AM, Adam Ford wrote:

On Tue, Oct 23, 2018 at 11:36 AM Steve Longerbeam
 wrote:

Hi Adam,

On 10/23/18 8:19 AM, Adam Ford wrote:

On Mon, Oct 22, 2018 at 7:40 AM Fabio Estevam  wrote:

Hi Adam,

On Mon, Oct 22, 2018 at 9:37 AM Adam Ford  wrote:


Thank you!  This tutorial web site is exactly what I need.  The
documentation page in Linux touched on the media-ctl links, but it
didn't explain the syntax or the mapping.  This graphical
interpretation really helps it make more sense.

Is capturing working well on your i.MX6 board now?

Fabio,

Unfortunately, no.  I built the rootfs based on Jagan's instructions
at https://openedev.amarulasolutions.com/display/ODWIKI/i.CoreM6+1.5

I tried building both the 4.15-RC6 kernel, a 4.19 kernel and a 4.14 LTS kernel.

Using the suggested method of generating the graphical display of the
pipeline options, I am able to enable various pipeline options
connecting different /dev/videoX options tot he camera.  I have tried
both the  suggested method above as well as the instructions found in
Documentation/media/v4l-drivers/imx.rst for their respective kernels,
and I have tried multiple options to capture through
ipu1_csi1_capture, ipu2_csi1_capture, and ip1_ic_prepenc capture, and
all yield a broken pipe.

libv4l2: error turning on stream: Broken pipe
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could
not read from resource.
Additional debug info:
gstv4l2bufferpool.c(1064): gst_v4l2_buffer_pool_poll ():
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
poll error 1: Broken pipe (32)

I can hear the camera click when I start gstreamer and click again
when it stops trying to stream.

dmesg indicates a broken pipe as well..

[ 2419.851502] ipu2_csi1: pipeline start failed with -32

might you have any suggestions?


This -EPIPE error might mean you have a mis-match of resolution, pixel
format, or field type between one of the source->sink pad links. You can
find out which pads have a mis-match by enabling dynamic debug in the
kernel function __media_pipeline_start.

Following Jagan's suggestion, I tried to make sure all the resolution
and pixel formats were set the same between each source and sink.

media-ctl --set-v4l2 "'ov5640 2-0010':0[fmt:UYVY2X8/640x480
field:none]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/640x480
field:none]"
media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/640x480
field:none]"
media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/640x480 field:none]"


Also make sure you are attempting to stream from the correct /dev/videoN.

I have graphically plotted the pipeline using media-ctl --print-dot
and I can see the proper video is routed, but your dynamic debug
suggestion yielded something:

imx-media capture-subsystem: link validation failed for 'ov5640
2-0010':0 -> 'imx6-mipi-csi2':0, error -32



It's what I expected, you have a format mismatch between those pads.

Steve




I am assume this means the interface between the camera and the csi2
isn't working.  I am going to double check the power rails and the
clocks.  i can hear it click when activated and deactivated, so
something is happening.

adam


Steve

Re: i.MX6 MIPI-CSI2 OV5640 Camera testing on Mainline Linux

[Steve Longerbeam]

On 10/23/18 10:54 AM, Adam Ford wrote:

On Tue, Oct 23, 2018 at 12:39 PM Steve Longerbeam
 wrote:


On 10/23/18 10:34 AM, Adam Ford wrote:

On Tue, Oct 23, 2018 at 11:36 AM Steve Longerbeam
 wrote:

Hi Adam,

On 10/23/18 8:19 AM, Adam Ford wrote:

On Mon, Oct 22, 2018 at 7:40 AM Fabio Estevam  wrote:

Hi Adam,

On Mon, Oct 22, 2018 at 9:37 AM Adam Ford  wrote:


Thank you!  This tutorial web site is exactly what I need.  The
documentation page in Linux touched on the media-ctl links, but it
didn't explain the syntax or the mapping.  This graphical
interpretation really helps it make more sense.

Is capturing working well on your i.MX6 board now?

Fabio,

Unfortunately, no.  I built the rootfs based on Jagan's instructions
at https://openedev.amarulasolutions.com/display/ODWIKI/i.CoreM6+1.5

I tried building both the 4.15-RC6 kernel, a 4.19 kernel and a 4.14 LTS kernel.

Using the suggested method of generating the graphical display of the
pipeline options, I am able to enable various pipeline options
connecting different /dev/videoX options tot he camera.  I have tried
both the  suggested method above as well as the instructions found in
Documentation/media/v4l-drivers/imx.rst for their respective kernels,
and I have tried multiple options to capture through
ipu1_csi1_capture, ipu2_csi1_capture, and ip1_ic_prepenc capture, and
all yield a broken pipe.

libv4l2: error turning on stream: Broken pipe
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Could
not read from resource.
Additional debug info:
gstv4l2bufferpool.c(1064): gst_v4l2_buffer_pool_poll ():
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
poll error 1: Broken pipe (32)

I can hear the camera click when I start gstreamer and click again
when it stops trying to stream.

dmesg indicates a broken pipe as well..

[ 2419.851502] ipu2_csi1: pipeline start failed with -32

might you have any suggestions?

This -EPIPE error might mean you have a mis-match of resolution, pixel
format, or field type between one of the source->sink pad links. You can
find out which pads have a mis-match by enabling dynamic debug in the
kernel function __media_pipeline_start.

Following Jagan's suggestion, I tried to make sure all the resolution
and pixel formats were set the same between each source and sink.

media-ctl --set-v4l2 "'ov5640 2-0010':0[fmt:UYVY2X8/640x480
field:none]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY2X8/640x480
field:none]"
media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY2X8/640x480
field:none]"
media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/640x480 field:none]"


Also make sure you are attempting to stream from the correct /dev/videoN.

I have graphically plotted the pipeline using media-ctl --print-dot
and I can see the proper video is routed, but your dynamic debug
suggestion yielded something:

 imx-media capture-subsystem: link validation failed for 'ov5640
2-0010':0 -> 'imx6-mipi-csi2':0, error -32


It's what I expected, you have a format mismatch between those pads.

Is the mismatch something I am doing wrong with:

media-ctl --set-v4l2 "'ov5640 2-0010':0[fmt:UYVY2X8/640x480 field:none]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':2[fmt:UYVY2X8/640x480 field:none]"

or is there something else I need to do?  I just used Jagan's suggestion.



Yeah, that looks correct, media-ctl _should_ automatically propagate the 
format at 'ov5640 2-0010':0 to 'imx6-mipi-csi2':0. What version of 
media-ctl are you using? If you have an older version of media-ctl tool 
that doesn't propagate formats from source->sink pads, you'll have to 
explicitly set the format at 'imx6-mipi-csi2':0, and at all the other 
sink pads in your pipeline. Better yet, upgrade! :)


Steve





I am assume this means the interface between the camera and the csi2
isn't working.  I am going to double check the power rails and the
clocks.  i can hear it click when activated and deactivated, so
something is happening.

adam


Steve

Re: Possible regression in v4l2-async

[Steve Longerbeam]

Hi Niklas,

On 11/29/18 10:47 AM, Niklas Söderlund wrote:

Hi Steve, Sakari and Hans,

I have been made aware of a possible regression by a few users of
rcar-vin and I'm a bit puzzled how to best handle it. Maybe you can help
me out?

The issue is visible when running with LOCKDEP enabled and it prints a
warning about a possible circular locking dependency, see end of mail.
The warning is triggered because rcar-vin takes a mutex (group->lock) in
its async bound call back while the async framework already holds one
(lisk_lock).


I see two possible solutions to this:

A. Remove acquiring the list_lock in v4l2_async_notifier_init().

B. Move the call to v4l2_async_notifier_init()**to the top of 
rvin_mc_parse_of_graph() (before acquiring group->lock).


It's most likely safe to remove the list_lock from 
v4l2_async_notifier_init(), because all drivers should be calling that 
function at probe start, before it begins to add async subdev 
descriptors to their notifiers. But just the same, I think it would be 
safer to keep list_lock in v4l2_async_notifier_init(), just in case of 
some strange corner case (such as a driver that adds descriptors in a 
separate thread from the thread that calls v4l2_async_notifier_init()).


So I would prefer B, but I'm open to either solution.

**


I traced the issue back to [1]. I don't believe this is any real trouble
here unless any of the async callbacks where to call into the async
framework themself which would trigger further calls to driver
callbacks, or maybe I'm naive and this is a real problem today.

Even if it's no real problem today I'm not sure this is never going to
be a problem, it would be nice if this warning could be handled somehow.
It is my understanding that any implementation of the async callbacks
who take a driver specific lock would trigger this warning which is not
nice.


It has always been the case that v4l2-async holds the list_lock when it 
calls a driver's async bound callback.


So the problem is only that v4l2_async_notifier_init() acquires the 
list_lock, which can create the reverse lock order driver-specific-lock 
-> list_lock. Which is solved by either A or B above.


Steve


Any suggestions or hints on how to move forward with this would be
appreciated.

1. eae2aed1eab9bf08 ("media: v4l2-fwnode: Switch to 
v4l2_async_notifier_add_subdev")

>> Warning output <<

  ==
  WARNING: possible circular locking dependency detected
  4.19.0-rc1-arm64-renesas-00212-geae2aed1eab9bf08 #56 Not tainted
  --
  swapper/0/1 is trying to acquire lock:
  (ptrval) (&group->lock){+.+.}, at: rvin_group_notify_bound+0x30/0xa8

  but task is already holding lock:
  (ptrval) (list_lock){+.+.}, at: 
__v4l2_async_notifier_register+0x54/0x1b0

  which lock already depends on the new lock.


  the existing dependency chain (in reverse order) is:

  -> #1 (list_lock){+.+.}:
 __mutex_lock+0x70/0x7f0
 mutex_lock_nested+0x1c/0x28
 v4l2_async_notifier_init+0x28/0x48
 rcar_vin_probe+0x13c/0x630
 platform_drv_probe+0x50/0xa0
 really_probe+0x1e0/0x298
 driver_probe_device+0x54/0xe8
 __driver_attach+0xf0/0xf8
 bus_for_each_dev+0x70/0xc0
 driver_attach+0x20/0x28
 bus_add_driver+0x1d4/0x200
 driver_register+0x60/0x110
 __platform_driver_register+0x44/0x50
 rcar_vin_driver_init+0x18/0x20
 do_one_initcall+0x180/0x35c
 kernel_init_freeable+0x454/0x4f8
 kernel_init+0x10/0xfc
 ret_from_fork+0x10/0x1c

  -> #0 (&group->lock){+.+.}:
 lock_acquire+0xc8/0x238
 __mutex_lock+0x70/0x7f0
 mutex_lock_nested+0x1c/0x28
 rvin_group_notify_bound+0x30/0xa8
 v4l2_async_match_notify+0x50/0x138
 v4l2_async_notifier_try_all_subdevs+0x58/0xb8
 __v4l2_async_notifier_register+0xdc/0x1b0
 v4l2_async_notifier_register+0x38/0x58
 rcar_vin_probe+0x1b8/0x630
 platform_drv_probe+0x50/0xa0
 really_probe+0x1e0/0x298
 driver_probe_device+0x54/0xe8
 __driver_attach+0xf0/0xf8
 bus_for_each_dev+0x70/0xc0
 driver_attach+0x20/0x28
 bus_add_driver+0x1d4/0x200
 driver_register+0x60/0x110
 __platform_driver_register+0x44/0x50
 rcar_vin_driver_init+0x18/0x20
 do_one_initcall+0x180/0x35c
 kernel_init_freeable+0x454/0x4f8
 kernel_init+0x10/0xfc
 ret_from_fork+0x10/0x1c

  other info that might help us debug this:

   Possible unsafe locking scenario:

 CPU0CPU1
 
lock(list_lock);
 lock(&group->lock);
 lock(list_lock);
lock(&group->lock);

   *** DEADLOCK ***

  2 locks held by swapper/0/1:
   #0: (ptrval) (&dev->mutex){}, at: __dr

Re: Possible regression in v4l2-async

[Steve Longerbeam]

On 11/29/18 11:26 AM, Steve Longerbeam wrote:

Hi Niklas,

On 11/29/18 10:47 AM, Niklas Söderlund wrote:

Hi Steve, Sakari and Hans,

I have been made aware of a possible regression by a few users of
rcar-vin and I'm a bit puzzled how to best handle it. Maybe you can help
me out?

The issue is visible when running with LOCKDEP enabled and it prints a
warning about a possible circular locking dependency, see end of mail.
The warning is triggered because rcar-vin takes a mutex (group->lock) in
its async bound call back while the async framework already holds one
(lisk_lock).


I see two possible solutions to this:

A. Remove acquiring the list_lock in v4l2_async_notifier_init().

B. Move the call to v4l2_async_notifier_init()**to the top of 
rvin_mc_parse_of_graph() (before acquiring group->lock).


It's most likely safe to remove the list_lock from 
v4l2_async_notifier_init(), because all drivers should be calling that 
function at probe start, before it begins to add async subdev 
descriptors to their notifiers. But just the same, I think it would be 
safer to keep list_lock in v4l2_async_notifier_init(), just in case of 
some strange corner case (such as a driver that adds descriptors in a 
separate thread from the thread that calls v4l2_async_notifier_init()).


Well, on second thought that's probably a lame example, no driver should 
be doing that. So removing the list_lock from v4l2_async_notifier_init() 
is probably safe. The notifier is not registered with v4l2-async at that 
point.


Steve



So I would prefer B, but I'm open to either solution.

**


I traced the issue back to [1]. I don't believe this is any real trouble
here unless any of the async callbacks where to call into the async
framework themself which would trigger further calls to driver
callbacks, or maybe I'm naive and this is a real problem today.

Even if it's no real problem today I'm not sure this is never going to
be a problem, it would be nice if this warning could be handled somehow.
It is my understanding that any implementation of the async callbacks
who take a driver specific lock would trigger this warning which is not
nice.


It has always been the case that v4l2-async holds the list_lock when 
it calls a driver's async bound callback.


So the problem is only that v4l2_async_notifier_init() acquires the 
list_lock, which can create the reverse lock order 
driver-specific-lock -> list_lock. Which is solved by either A or B 
above.


Steve


Any suggestions or hints on how to move forward with this would be
appreciated.

1. eae2aed1eab9bf08 ("media: v4l2-fwnode: Switch to 
v4l2_async_notifier_add_subdev")


>> Warning output <<

  ==
  WARNING: possible circular locking dependency detected
  4.19.0-rc1-arm64-renesas-00212-geae2aed1eab9bf08 #56 Not tainted
  --
  swapper/0/1 is trying to acquire lock:
  (ptrval) (&group->lock){+.+.}, at: 
rvin_group_notify_bound+0x30/0xa8


  but task is already holding lock:
  (ptrval) (list_lock){+.+.}, at: 
__v4l2_async_notifier_register+0x54/0x1b0


  which lock already depends on the new lock.


  the existing dependency chain (in reverse order) is:

  -> #1 (list_lock){+.+.}:
 __mutex_lock+0x70/0x7f0
 mutex_lock_nested+0x1c/0x28
 v4l2_async_notifier_init+0x28/0x48
 rcar_vin_probe+0x13c/0x630
 platform_drv_probe+0x50/0xa0
 really_probe+0x1e0/0x298
 driver_probe_device+0x54/0xe8
 __driver_attach+0xf0/0xf8
 bus_for_each_dev+0x70/0xc0
 driver_attach+0x20/0x28
 bus_add_driver+0x1d4/0x200
 driver_register+0x60/0x110
 __platform_driver_register+0x44/0x50
 rcar_vin_driver_init+0x18/0x20
 do_one_initcall+0x180/0x35c
 kernel_init_freeable+0x454/0x4f8
 kernel_init+0x10/0xfc
 ret_from_fork+0x10/0x1c

  -> #0 (&group->lock){+.+.}:
 lock_acquire+0xc8/0x238
 __mutex_lock+0x70/0x7f0
 mutex_lock_nested+0x1c/0x28
 rvin_group_notify_bound+0x30/0xa8
 v4l2_async_match_notify+0x50/0x138
 v4l2_async_notifier_try_all_subdevs+0x58/0xb8
 __v4l2_async_notifier_register+0xdc/0x1b0
 v4l2_async_notifier_register+0x38/0x58
 rcar_vin_probe+0x1b8/0x630
 platform_drv_probe+0x50/0xa0
 really_probe+0x1e0/0x298
 driver_probe_device+0x54/0xe8
 __driver_attach+0xf0/0xf8
 bus_for_each_dev+0x70/0xc0
 driver_attach+0x20/0x28
 bus_add_driver+0x1d4/0x200
 driver_register+0x60/0x110
 __platform_driver_register+0x44/0x50
 rcar_vin_driver_init+0x18/0x20
 do_one_initcall+0x180/0x35c
 kernel_init_freeable+0x454/0x4f8
 kernel_init+0x10/0xfc
 ret_from_fork+0x10/0x1c

Re: Possible regression in v4l2-async

[Steve Longerbeam]

Hi Niklas,

On 11/29/18 6:25 PM, Niklas Söderlund wrote:

Hi Sakari, Steve,

Thanks for your quick response.

On 2018-11-29 22:37:53 +0200, Sakari Ailus wrote:

Hi Steve, Niklas,

On Thu, Nov 29, 2018 at 11:41:32AM -0800, Steve Longerbeam wrote:

On 11/29/18 11:26 AM, Steve Longerbeam wrote:

Hi Niklas,

On 11/29/18 10:47 AM, Niklas Söderlund wrote:

Hi Steve, Sakari and Hans,

I have been made aware of a possible regression by a few users of
rcar-vin and I'm a bit puzzled how to best handle it. Maybe you can help
me out?

The issue is visible when running with LOCKDEP enabled and it prints a
warning about a possible circular locking dependency, see end of mail.
The warning is triggered because rcar-vin takes a mutex (group->lock) in
its async bound call back while the async framework already holds one
(lisk_lock).

I see two possible solutions to this:

A. Remove acquiring the list_lock in v4l2_async_notifier_init().

B. Move the call to v4l2_async_notifier_init()**to the top of
rvin_mc_parse_of_graph() (before acquiring group->lock).

It's most likely safe to remove the list_lock from
v4l2_async_notifier_init(), because all drivers should be calling that
function at probe start, before it begins to add async subdev
descriptors to their notifiers. But just the same, I think it would be
safer to keep list_lock in v4l2_async_notifier_init(), just in case of
some strange corner case (such as a driver that adds descriptors in a
separate thread from the thread that calls v4l2_async_notifier_init()).

Well, on second thought that's probably a lame example, no driver should be
doing that. So removing the list_lock from v4l2_async_notifier_init() is
probably safe. The notifier is not registered with v4l2-async at that point.

I agree, apart from "probably". It is safe.

Niklas: would you like to send a patch? :-)

Thanks for your suggestions, I have sent a patch [1] for Steves
alternative A as I agree that is the correct approach to cure the
symptom of the problem and have value in its own right. Unfortunately
this do not cover the core of the problem.

As I tried to describe in my first mail, maybe I could have described it
better, sorry about that. This problem comes from the adding of
subdevices to notifiers using a list instead of a array allocated and
handled by the driver. Even with [1] applied a LOCKDEP warning is still
trigged (see end of mail) as one thread could be busy adding subdevs to
it's notifier using the fwnode helpers who take the list_lock as another
thread is busy processing and binding subdevices also holding the
list_lock. Then if a async callback implemented by a driver also takes a
driver local lock the warning is still triggered as described in my
first mail.


Understood. So the root issue is that by acquiring list_lock in
v4l2_async_notifier_add_subdev(), it can create the reverse locking
order driver-specific-lock -> list_lock in some drivers.

My first thought was a new mutex who's sole use is to protect the
notifier->asd_list, instead of using the global list_lock, in
in v4l2_async_notifier_add_subdev().

But that won't work. The list_lock is needed in 
v4l2_async_notifier_add_subdev(),

because the latter calls v4l2_async_notifier_has_async_subdev() which
requires holding the list_lock in order to scan all notifier's waiting and
done lists.

Is it possible rcar-vin can use the graph_mutex in place of the group->lock?
It seems the group->lock is meant to protect changes to the media graph, so
could the driver use the graph_mutex for that purpose instead?

Steve



1. [PATCH] v4l2: async: remove locking when initializing async notifier

>> warning output <<
  ==
  WARNING: possible circular locking dependency detected
  4.20.0-rc1-arm64-renesas-00141-gcadfba6a1339544c #58 Not tainted
  --
  swapper/0/1 is trying to acquire lock:
  (ptrval) (&group->lock){+.+.}, at: rvin_group_notify_bound+0x30/0xa8
  
  but task is already holding lock:

  (ptrval) (list_lock){+.+.}, at: 
__v4l2_async_notifier_register+0x4c/0x140
  
  which lock already depends on the new lock.
  
  
  the existing dependency chain (in reverse order) is:
  
  -> #1 (list_lock){+.+.}:

 __mutex_lock+0x70/0x7e0
 mutex_lock_nested+0x1c/0x28
 v4l2_async_notifier_add_subdev+0x2c/0x78
 __v4l2_async_notifier_parse_fwnode_ep+0x17c/0x310
 v4l2_async_notifier_parse_fwnode_endpoints_by_port+0x14/0x20
 rcar_vin_probe+0x174/0x638
 platform_drv_probe+0x50/0xa0
 really_probe+0x1c8/0x2a8
 driver_probe_device+0x54/0xe8
 __driver_attach+0xf0/0xf8
 bus_for_each_dev+0x70/0xc0
 driver_attach+0x20/0x28
 bus_add_driver+0x1d4/0x200
 driver_register+0x60/0x110
 __platform_driver_register+0

Re: [PATCH v5 02/12] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

Hi Philipp, can you review this patch and give it your ack?

Thanks,
Steve


On 10/16/18 5:00 PM, Steve Longerbeam wrote:

The function ipu_csi_init_interface() was inverting the F-bit for
NTSC case, in the CCIR_CODE_1/2 registers. The result being that
for NTSC bottom-top field order, the CSI would swap fields and
capture in top-bottom order.

Instead, base field swap on the field order of the input to the CSI,
and the field order of the requested output. If the input/output
fields are sequential but different, swap fields, otherwise do
not swap. This requires passing both the input and output mbus
frame formats to ipu_csi_init_interface().

Move this code to a new private function ipu_csi_set_bt_interlaced_codes()
that programs the CCIR_CODE_1/2 registers for interlaced BT.656 (and
possibly interlaced BT.1120 in the future).

When detecting input video standard from the input frame width/height,
make sure to double height if input field type is alternate, since
in that case input height only includes lines for one field.

Signed-off-by: Steve Longerbeam 
---
Changes since v4:
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
   by Philipp Zabel.
- Fixed a regression in csi_setup(), caught by Philipp.
---
  drivers/gpu/ipu-v3/ipu-csi.c  | 119 +++---
  drivers/staging/media/imx/imx-media-csi.c |  17 +---
  include/video/imx-ipu-v3.h|   3 +-
  3 files changed, 88 insertions(+), 51 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index aa0e30a2ba18..4a15e513fa05 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -325,6 +325,15 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config 
*cfg, u32 mbus_code,
return 0;
  }
  
+/* translate alternate field mode based on given standard */

+static inline enum v4l2_field
+ipu_csi_translate_field(enum v4l2_field field, v4l2_std_id std)
+{
+   return (field != V4L2_FIELD_ALTERNATE) ? field :
+   ((std & V4L2_STD_525_60) ?
+V4L2_FIELD_SEQ_BT : V4L2_FIELD_SEQ_TB);
+}
+
  /*
   * Fill a CSI bus config struct from mbus_config and mbus_framefmt.
   */
@@ -374,22 +383,75 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config 
*csicfg,
return 0;
  }
  
+static int ipu_csi_set_bt_interlaced_codes(struct ipu_csi *csi,

+  struct v4l2_mbus_framefmt *infmt,
+  struct v4l2_mbus_framefmt *outfmt,
+  v4l2_std_id std)
+{
+   enum v4l2_field infield, outfield;
+   bool swap_fields;
+
+   /* get translated field type of input and output */
+   infield = ipu_csi_translate_field(infmt->field, std);
+   outfield = ipu_csi_translate_field(outfmt->field, std);
+
+   /*
+* Write the H-V-F codes the CSI will match against the
+* incoming data for start/end of active and blanking
+* field intervals. If input and output field types are
+* sequential but not the same (one is SEQ_BT and the other
+* is SEQ_TB), swap the F-bit so that the CSI will capture
+* field 1 lines before field 0 lines.
+*/
+   swap_fields = (V4L2_FIELD_IS_SEQUENTIAL(infield) &&
+  V4L2_FIELD_IS_SEQUENTIAL(outfield) &&
+  infield != outfield);
+
+   if (!swap_fields) {
+   /*
+* Field0BlankEnd  = 110, Field0BlankStart  = 010
+* Field0ActiveEnd = 100, Field0ActiveStart = 000
+* Field1BlankEnd  = 111, Field1BlankStart  = 011
+* Field1ActiveEnd = 101, Field1ActiveStart = 001
+*/
+   ipu_csi_write(csi, 0x40596 | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0xD07DF, CSI_CCIR_CODE_2);
+   } else {
+   dev_dbg(csi->ipu->dev, "capture field swap\n");
+
+   /* same as above but with F-bit inverted */
+   ipu_csi_write(csi, 0xD07DF | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0x40596, CSI_CCIR_CODE_2);
+   }
+
+   ipu_csi_write(csi, 0xFF, CSI_CCIR_CODE_3);
+
+   return 0;
+}
+
+
  int ipu_csi_init_interface(struct ipu_csi *csi,
   struct v4l2_mbus_config *mbus_cfg,
-  struct v4l2_mbus_framefmt *mbus_fmt)
+  struct v4l2_mbus_framefmt *infmt,
+  struct v4l2_mbus_framefmt *outfmt)
  {
struct ipu_csi_bus_config cfg;
unsigned long flags;
u32 width, height, data = 0;
+   v4l2_std_id std;
int ret;
  
-	ret = fill_csi_bus_cfg(&cfg, mbus_cfg, mbus_fmt);

+   ret = fill_csi_bus_cfg(&cfg, mbus_cfg, infmt);
if (ret < 0)
return re

[PATCH] media: imx: queue subdev events to reachable video devices

[Steve Longerbeam]

From: Steve Longerbeam 

Forward events from a sub-device to its list of reachable video
devices.

Note this will queue the event to a video device even if there is
no actual _enabled_ media path from the sub-device to the video device.
So a future improvement is to skip the video device if there is no enabled
path to it from the sub-device. The entity->pipe pointer can't be
used for this check because in imx-media a sub-device can be a
member to more than one streaming pipeline at a time.

Signed-off-by: Steve Longerbeam 
---
 drivers/staging/media/imx/imx-media-capture.c | 18 ++
 drivers/staging/media/imx/imx-media-dev.c | 24 +++
 2 files changed, 42 insertions(+)

diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index b37e1186eb2f..4dfbe05d203e 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -335,6 +335,21 @@ static int capture_s_parm(struct file *file, void *fh,
return 0;
 }
 
+static int capture_subscribe_event(struct v4l2_fh *fh,
+  const struct v4l2_event_subscription *sub)
+{
+   switch (sub->type) {
+   case V4L2_EVENT_IMX_FRAME_INTERVAL_ERROR:
+   return v4l2_event_subscribe(fh, sub, 0, NULL);
+   case V4L2_EVENT_SOURCE_CHANGE:
+   return v4l2_src_change_event_subscribe(fh, sub);
+   case V4L2_EVENT_CTRL:
+   return v4l2_ctrl_subscribe_event(fh, sub);
+   default:
+   return -EINVAL;
+   }
+}
+
 static const struct v4l2_ioctl_ops capture_ioctl_ops = {
.vidioc_querycap= vidioc_querycap,
 
@@ -362,6 +377,9 @@ static const struct v4l2_ioctl_ops capture_ioctl_ops = {
.vidioc_expbuf  = vb2_ioctl_expbuf,
.vidioc_streamon= vb2_ioctl_streamon,
.vidioc_streamoff   = vb2_ioctl_streamoff,
+
+   .vidioc_subscribe_event = capture_subscribe_event,
+   .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
 };
 
 /*
diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index 4b344a4a3706..25e916562c66 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -442,6 +442,29 @@ static const struct media_device_ops imx_media_md_ops = {
.link_notify = imx_media_link_notify,
 };
 
+static void imx_media_notify(struct v4l2_subdev *sd,
+unsigned int notification,
+void *arg)
+{
+   struct media_entity *entity = &sd->entity;
+   int i;
+
+   if (notification != V4L2_DEVICE_NOTIFY_EVENT)
+   return;
+
+   for (i = 0; i < entity->num_pads; i++) {
+   struct media_pad *pad = &entity->pads[i];
+   struct imx_media_pad_vdev *pad_vdev;
+   struct list_head *pad_vdev_list;
+
+   pad_vdev_list = to_pad_vdev_list(sd, pad->index);
+   if (!pad_vdev_list)
+   continue;
+   list_for_each_entry(pad_vdev, pad_vdev_list, list)
+   v4l2_event_queue(pad_vdev->vdev->vfd, arg);
+   }
+}
+
 static int imx_media_probe(struct platform_device *pdev)
 {
struct device *dev = &pdev->dev;
@@ -462,6 +485,7 @@ static int imx_media_probe(struct platform_device *pdev)
mutex_init(&imxmd->mutex);
 
imxmd->v4l2_dev.mdev = &imxmd->md;
+   imxmd->v4l2_dev.notify = imx_media_notify;
strscpy(imxmd->v4l2_dev.name, "imx-media",
sizeof(imxmd->v4l2_dev.name));
 
-- 
2.17.1

Re: [PATCH v5 02/12] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

On 12/13/18 4:59 AM, Philipp Zabel wrote:

Hi Steve,

On Tue, 2018-10-16 at 17:00 -0700, Steve Longerbeam wrote:

The function ipu_csi_init_interface() was inverting the F-bit for
NTSC case, in the CCIR_CODE_1/2 registers. The result being that
for NTSC bottom-top field order, the CSI would swap fields and
capture in top-bottom order.

Instead, base field swap on the field order of the input to the CSI,
and the field order of the requested output. If the input/output
fields are sequential but different, swap fields, otherwise do
not swap. This requires passing both the input and output mbus
frame formats to ipu_csi_init_interface().

Move this code to a new private function ipu_csi_set_bt_interlaced_codes()
that programs the CCIR_CODE_1/2 registers for interlaced BT.656 (and
possibly interlaced BT.1120 in the future).

When detecting input video standard from the input frame width/height,
make sure to double height if input field type is alternate, since
in that case input height only includes lines for one field.

Signed-off-by: Steve Longerbeam 
---
Changes since v4:
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
   by Philipp Zabel.
- Fixed a regression in csi_setup(), caught by Philipp.
---
  drivers/gpu/ipu-v3/ipu-csi.c  | 119 +++---
  drivers/staging/media/imx/imx-media-csi.c |  17 +---
  include/video/imx-ipu-v3.h|   3 +-
  3 files changed, 88 insertions(+), 51 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index aa0e30a2ba18..4a15e513fa05 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -325,6 +325,15 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config 
*cfg, u32 mbus_code,
return 0;
  }
  
+/* translate alternate field mode based on given standard */

+static inline enum v4l2_field
+ipu_csi_translate_field(enum v4l2_field field, v4l2_std_id std)
+{
+   return (field != V4L2_FIELD_ALTERNATE) ? field :
+   ((std & V4L2_STD_525_60) ?
+V4L2_FIELD_SEQ_BT : V4L2_FIELD_SEQ_TB);
+}
+
  /*
   * Fill a CSI bus config struct from mbus_config and mbus_framefmt.
   */
@@ -374,22 +383,75 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config 
*csicfg,
return 0;
  }
  
+static int ipu_csi_set_bt_interlaced_codes(struct ipu_csi *csi,

+  struct v4l2_mbus_framefmt *infmt,
+  struct v4l2_mbus_framefmt *outfmt,

infmt and outfmt parameters could be const.


Agreed, I will convert these pointer args to const. And since we are 
changing the API to ipu_csi_init_interface() anyway, I went ahead and 
converted the mbus_cfg, infmt, and outfmt pointer args to const there as 
well.




+  v4l2_std_id std)
+{
+   enum v4l2_field infield, outfield;
+   bool swap_fields;
+
+   /* get translated field type of input and output */
+   infield = ipu_csi_translate_field(infmt->field, std);
+   outfield = ipu_csi_translate_field(outfmt->field, std);
+
+   /*
+* Write the H-V-F codes the CSI will match against the
+* incoming data for start/end of active and blanking
+* field intervals. If input and output field types are
+* sequential but not the same (one is SEQ_BT and the other
+* is SEQ_TB), swap the F-bit so that the CSI will capture
+* field 1 lines before field 0 lines.
+*/
+   swap_fields = (V4L2_FIELD_IS_SEQUENTIAL(infield) &&
+  V4L2_FIELD_IS_SEQUENTIAL(outfield) &&
+  infield != outfield);
+
+   if (!swap_fields) {
+   /*
+* Field0BlankEnd  = 110, Field0BlankStart  = 010
+* Field0ActiveEnd = 100, Field0ActiveStart = 000
+* Field1BlankEnd  = 111, Field1BlankStart  = 011
+* Field1ActiveEnd = 101, Field1ActiveStart = 001
+*/
+   ipu_csi_write(csi, 0x40596 | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0xD07DF, CSI_CCIR_CODE_2);
+   } else {
+   dev_dbg(csi->ipu->dev, "capture field swap\n");
+
+   /* same as above but with F-bit inverted */
+   ipu_csi_write(csi, 0xD07DF | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0x40596, CSI_CCIR_CODE_2);
+   }
+
+   ipu_csi_write(csi, 0xFF, CSI_CCIR_CODE_3);
+
+   return 0;
+}
+
+
  int ipu_csi_init_interface(struct ipu_csi *csi,
   struct v4l2_mbus_config *mbus_cfg,
-  struct v4l2_mbus_framefmt *mbus_fmt)
+  struct v4l2_mbus_framefmt *infmt,
+  struct v4l2_mbus_framefmt *outfmt)
  {
struct ipu_csi_bus_config cfg;
unsi

[PATCH v6] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

The function ipu_csi_init_interface() was inverting the F-bit for
NTSC case, in the CCIR_CODE_1/2 registers. The result being that
for NTSC bottom-top field order, the CSI would swap fields and
capture in top-bottom order.

Instead, base field swap on the field order of the input to the CSI,
and the field order of the requested output. If the input/output
fields are sequential but different, swap fields, otherwise do
not swap. This requires passing both the input and output mbus
frame formats to ipu_csi_init_interface().

Move this code to a new private function ipu_csi_set_bt_interlaced_codes()
that programs the CCIR_CODE_1/2 registers for interlaced BT.656 (and
possibly interlaced BT.1120 in the future).

When detecting input video standard from the input frame width/height,
make sure to double height if input field type is alternate, since
in that case input height only includes lines for one field.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
Changes since v5:
- Convert to const the infmt, outfmt, and mbus_cfg pointer args to
  ipu_csi_init_interface(), suggested by Philipp Zabel.
- Bring back if_fmt local var and don't copy outfmt to local stack in
  csi_setup(), suggested by Philipp.

Changes since v4:
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
  by Philipp.
- Fixed a regression in csi_setup(), caught by Philipp.
---
 drivers/gpu/ipu-v3/ipu-csi.c  | 126 +++---
 drivers/staging/media/imx/imx-media-csi.c |   7 +-
 include/video/imx-ipu-v3.h|   5 +-
 3 files changed, 89 insertions(+), 49 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index aa0e30a2ba18..d1e575571a8d 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -325,12 +325,21 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config 
*cfg, u32 mbus_code,
return 0;
 }
 
+/* translate alternate field mode based on given standard */
+static inline enum v4l2_field
+ipu_csi_translate_field(enum v4l2_field field, v4l2_std_id std)
+{
+   return (field != V4L2_FIELD_ALTERNATE) ? field :
+   ((std & V4L2_STD_525_60) ?
+V4L2_FIELD_SEQ_BT : V4L2_FIELD_SEQ_TB);
+}
+
 /*
  * Fill a CSI bus config struct from mbus_config and mbus_framefmt.
  */
 static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
-struct v4l2_mbus_config *mbus_cfg,
-struct v4l2_mbus_framefmt *mbus_fmt)
+   const struct v4l2_mbus_config *mbus_cfg,
+   const struct v4l2_mbus_framefmt *mbus_fmt)
 {
int ret;
 
@@ -374,22 +383,76 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config 
*csicfg,
return 0;
 }
 
+static int
+ipu_csi_set_bt_interlaced_codes(struct ipu_csi *csi,
+   const struct v4l2_mbus_framefmt *infmt,
+   const struct v4l2_mbus_framefmt *outfmt,
+   v4l2_std_id std)
+{
+   enum v4l2_field infield, outfield;
+   bool swap_fields;
+
+   /* get translated field type of input and output */
+   infield = ipu_csi_translate_field(infmt->field, std);
+   outfield = ipu_csi_translate_field(outfmt->field, std);
+
+   /*
+* Write the H-V-F codes the CSI will match against the
+* incoming data for start/end of active and blanking
+* field intervals. If input and output field types are
+* sequential but not the same (one is SEQ_BT and the other
+* is SEQ_TB), swap the F-bit so that the CSI will capture
+* field 1 lines before field 0 lines.
+*/
+   swap_fields = (V4L2_FIELD_IS_SEQUENTIAL(infield) &&
+  V4L2_FIELD_IS_SEQUENTIAL(outfield) &&
+  infield != outfield);
+
+   if (!swap_fields) {
+   /*
+* Field0BlankEnd  = 110, Field0BlankStart  = 010
+* Field0ActiveEnd = 100, Field0ActiveStart = 000
+* Field1BlankEnd  = 111, Field1BlankStart  = 011
+* Field1ActiveEnd = 101, Field1ActiveStart = 001
+*/
+   ipu_csi_write(csi, 0x40596 | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0xD07DF, CSI_CCIR_CODE_2);
+   } else {
+   dev_dbg(csi->ipu->dev, "capture field swap\n");
+
+   /* same as above but with F-bit inverted */
+   ipu_csi_write(csi, 0xD07DF | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0x40596, CSI_CCIR_CODE_2);
+   }
+
+   ipu_csi_write(csi, 0xFF, CSI_CCIR_CODE_3);
+
+   return 0;
+}
+
+
 int ipu_csi_init_interface(struct ipu_csi *csi,
-  struct v4l2_mbus_config *mbus_cfg,
-  

[RFC PATCH] media: rcar-vin: Allow independent VIN link enablement

[Steve Longerbeam]

There is a block of code in rvin_group_link_notify() that prevents
enabling a link to a VIN node if any entity in the media graph is
in use. This prevents enabling a VIN link even if there is an in-use
entity somewhere in the graph that is independent of the link's
pipeline.

For example, the code block will prevent enabling a link from
the first rcar-csi2 receiver to a VIN node even if there is an
enabled link somewhere far upstream on the second independent
rcar-csi2 receiver pipeline.

If this code block is meant to prevent modifying a link if the
link is actively involved in streaming, there is already such a
check in __media_entity_setup_link() that verifies the stream_count
of the link's source and sink entities are both zero.

Remove the code block so that VIN node links can be enabled even if
there are other independent in-use entities.

Fixes: c0cc5aef31 ("media: rcar-vin: add link notify for Gen3")

Signed-off-by: Steve Longerbeam 
---
 drivers/media/platform/rcar-vin/rcar-core.c | 6 --
 1 file changed, 6 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-core.c 
b/drivers/media/platform/rcar-vin/rcar-core.c
index f0719ce24b97..b2c9a876969e 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -116,7 +116,6 @@ static int rvin_group_link_notify(struct media_link *link, 
u32 flags,
struct rvin_group, mdev);
unsigned int master_id, channel, mask_new, i;
unsigned int mask = ~0;
-   struct media_entity *entity;
struct video_device *vdev;
struct media_pad *csi_pad;
struct rvin_dev *vin = NULL;
@@ -131,11 +130,6 @@ static int rvin_group_link_notify(struct media_link *link, 
u32 flags,
!is_media_entity_v4l2_video_device(link->sink->entity))
return 0;
 
-   /* If any entity is in use don't allow link changes. */
-   media_device_for_each_entity(entity, &group->mdev)
-   if (entity->use_count)
-   return -EBUSY;
-
mutex_lock(&group->lock);
 
/* Find the master VIN that controls the routes. */
-- 
2.17.1

Re: [RFC PATCH] media: rcar-vin: Allow independent VIN link enablement

[Steve Longerbeam]

Hi Niklas,

On 12/26/18 4:51 PM, Niklas Söderlund wrote:

Hi Steve,

Thanks for your patch.

On 2018-12-25 15:27:25 -0800, Steve Longerbeam wrote:

There is a block of code in rvin_group_link_notify() that prevents
enabling a link to a VIN node if any entity in the media graph is
in use. This prevents enabling a VIN link even if there is an in-use
entity somewhere in the graph that is independent of the link's
pipeline.

For example, the code block will prevent enabling a link from
the first rcar-csi2 receiver to a VIN node even if there is an
enabled link somewhere far upstream on the second independent
rcar-csi2 receiver pipeline.

Unfortunately this is by design and needed due to the hardware design.
The different VIN endpoints shares a configuration register which
controls the routing from the CSI-2 receivers to the VIN (register name
CHSEL). Modifying the CHSEL register which is what happens when a link
is enabled/disabled can have side effects on running streams even if
they are not shown to be dependent in the media graph.


Ok, understood, modifying CHSEL register can adversely affect running 
streams.




There is a CHSEL register in VIN0 which controls the routing from all
CSI-2 receivers to VIN0-3 and a CHSEL register in VIN4 which controls
the same for VIN4-7.


If this code block is meant to prevent modifying a link if the
link is actively involved in streaming, there is already such a
check in __media_entity_setup_link() that verifies the stream_count
of the link's source and sink entities are both zero.

For the reason above the check in __media_entity_setup_link() is not
enough :-( This register sharing is my least favorite thing about the
VIN on Gen3 and forces the driver to become more complex as all VIN
instances needs to know about each other and interact.



Given above I understand why the stream count checks in 
__media_entity_setup_link() are insufficient, because only the requested 
link's source stream count is checked, and not the other CSI-2 receiver 
for example.


But why check the use counts of every entity upstream from the VIN 
sources? Why not check only the VIN source entities stream counts (both 
CSI-2 receivers and/or parallel devices), and ignore entities upstream 
from those?


And why are the use counts checked, it seems it should be the stream 
counts that should be checked.



Remove the code block so that VIN node links can be enabled even if
there are other independent in-use entities.

There is room for some improvement in this area disregarding the odd
hardware design. It *could* be allowed to change a link terminating in
VIN4-7 even if there is a stream running for one or more in VIN0-3.

I would be interested to test such a patch but to allow any link change
which is allowed by __media_entity_setup_link() is unfortunately not
possible, as I understand it. Maybe someone more clever then me can find
ways to unlock even more then just the split between VIN0-3 and VIn4-7.

In essence the CHSEL register can not be changed if it's involved in a
running pipeline even if the end result would be that the running
pipeline would look the same. This is possible as there are multiple
CHSEL settings where the same source is connected to a specific VIN
while other members of the "subgroup of VINs" (e.g. VIN0-3) is routed to
something else for the two CHSEL settings.


Right, so rvin_group_link_notify() determines whether the requested VIN 
link enable will result in a valid set of CSI2->VIN links for the given 
hardware, using the CHSEL bitmask tables. Which is why it seems it is 
the stream counts that should be checked as mentioned above, rather than 
the use counts, because the CHSEL bitmask checks are validating the set 
of enabled links, and the only remaining checks are to verify no streams 
are running on either CSI-2 receiver.


Steve



Fixes: c0cc5aef31 ("media: rcar-vin: add link notify for Gen3")

Signed-off-by: Steve Longerbeam 
---
  drivers/media/platform/rcar-vin/rcar-core.c | 6 --
  1 file changed, 6 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-core.c 
b/drivers/media/platform/rcar-vin/rcar-core.c
index f0719ce24b97..b2c9a876969e 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -116,7 +116,6 @@ static int rvin_group_link_notify(struct media_link *link, 
u32 flags,
struct rvin_group, mdev);
unsigned int master_id, channel, mask_new, i;
unsigned int mask = ~0;
-   struct media_entity *entity;
struct video_device *vdev;
struct media_pad *csi_pad;
struct rvin_dev *vin = NULL;
@@ -131,11 +130,6 @@ static int rvin_group_link_notify(struct media_link *link, 
u32 flags,
!is_media_entity_v4l2_video_device(link->sink->entity))
return 0;
  
-	/* If any entity is in use don't allow link changes. */

-  

Re: [RFC PATCH] media: rcar-vin: Allow independent VIN link enablement

[Steve Longerbeam]

Hi Niklas,

How about a patch that simply replaces the entity use_count check with a 
stream_count check?


As in:

diff --git a/drivers/media/platform/rcar-vin/rcar-core.c 
b/drivers/media/platform/rcar-vin/rcar-core.c

index f0719ce24b97..aef8d8dab6ab 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -131,9 +131,13 @@ static int rvin_group_link_notify(struct media_link 
*link, u32 flags,

!is_media_entity_v4l2_video_device(link->sink->entity))
     return 0;

-    /* If any entity is in use don't allow link changes. */
+    /*
+     * Don't allow link changes if any entity in the graph is
+     * streaming, modifying the CHSEL register fields can disrupt
+     * running streams.
+     */
 media_device_for_each_entity(entity, &group->mdev)
-        if (entity->use_count)
+        if (entity->stream_count)
         return -EBUSY;

 mutex_lock(&group->lock);


And that might be overkilll, maybe only the stream_count's of the VIN 
entities need to be checked.


Steve



On 12/29/18 3:37 PM, Steve Longerbeam wrote:

Hi Niklas,

On 12/26/18 4:51 PM, Niklas Söderlund wrote:

Hi Steve,

Thanks for your patch.

On 2018-12-25 15:27:25 -0800, Steve Longerbeam wrote:

There is a block of code in rvin_group_link_notify() that prevents
enabling a link to a VIN node if any entity in the media graph is
in use. This prevents enabling a VIN link even if there is an in-use
entity somewhere in the graph that is independent of the link's
pipeline.

For example, the code block will prevent enabling a link from
the first rcar-csi2 receiver to a VIN node even if there is an
enabled link somewhere far upstream on the second independent
rcar-csi2 receiver pipeline.

Unfortunately this is by design and needed due to the hardware design.
The different VIN endpoints shares a configuration register which
controls the routing from the CSI-2 receivers to the VIN (register name
CHSEL). Modifying the CHSEL register which is what happens when a link
is enabled/disabled can have side effects on running streams even if
they are not shown to be dependent in the media graph.


Ok, understood, modifying CHSEL register can adversely affect running 
streams.




There is a CHSEL register in VIN0 which controls the routing from all
CSI-2 receivers to VIN0-3 and a CHSEL register in VIN4 which controls
the same for VIN4-7.


If this code block is meant to prevent modifying a link if the
link is actively involved in streaming, there is already such a
check in __media_entity_setup_link() that verifies the stream_count
of the link's source and sink entities are both zero.

For the reason above the check in __media_entity_setup_link() is not
enough :-( This register sharing is my least favorite thing about the
VIN on Gen3 and forces the driver to become more complex as all VIN
instances needs to know about each other and interact.



Given above I understand why the stream count checks in 
__media_entity_setup_link() are insufficient, because only the 
requested link's source stream count is checked, and not the other 
CSI-2 receiver for example.


But why check the use counts of every entity upstream from the VIN 
sources? Why not check only the VIN source entities stream counts 
(both CSI-2 receivers and/or parallel devices), and ignore entities 
upstream from those?


And why are the use counts checked, it seems it should be the stream 
counts that should be checked.



Remove the code block so that VIN node links can be enabled even if
there are other independent in-use entities.

There is room for some improvement in this area disregarding the odd
hardware design. It *could* be allowed to change a link terminating in
VIN4-7 even if there is a stream running for one or more in VIN0-3.

I would be interested to test such a patch but to allow any link change
which is allowed by __media_entity_setup_link() is unfortunately not
possible, as I understand it. Maybe someone more clever then me can find
ways to unlock even more then just the split between VIN0-3 and VIn4-7.

In essence the CHSEL register can not be changed if it's involved in a
running pipeline even if the end result would be that the running
pipeline would look the same. This is possible as there are multiple
CHSEL settings where the same source is connected to a specific VIN
while other members of the "subgroup of VINs" (e.g. VIN0-3) is routed to
something else for the two CHSEL settings.


Right, so rvin_group_link_notify() determines whether the requested 
VIN link enable will result in a valid set of CSI2->VIN links for the 
given hardware, using the CHSEL bitmask tables. Which is why it seems 
it is the stream counts that should be checked as mentioned above, 
rather than the use counts, because the CHSEL bitmask checks are 
validating the set of enabled links, and the only remaining checks are 
to verify no streams are running on 

[RFC PATCH v2] media: rcar-vin: Allow independent VIN link enablement

[Steve Longerbeam]

There is a block of code in rvin_group_link_notify() that loops through
all entities in the media graph, and prevents enabling a link to a VIN
node if any entity is in use. This prevents enabling a VIN link even if
there is an in-use entity somewhere in the graph that is independent of
the link's pipeline.

For example, the code will prevent enabling a link from the first
rcar-csi2 receiver to a VIN node even if there is an enabled link
somewhere far upstream on the second independent rcar-csi2 receiver
pipeline.

If this code is meant to prevent modifying a link if any entity in the
graph is actively involved in streaming (because modifying the CHSEL
register fields can disrupt any/all running streams), then the entities
stream counts should be checked rather than the use counts.

(There is already such a check in __media_entity_setup_link() that verifies
the stream_count of the link's source and sink entities are both zero,
but that is insufficient, since there should be no running streams in
the entire graph).

Modify the media_device_for_each_entity() loop to check the entity
stream_count instead of the use_count, and elaborate on the comment.
VIN node links can now be enabled even if there are other independent
in-use entities that are not streaming.

Fixes: c0cc5aef31 ("media: rcar-vin: add link notify for Gen3")

Signed-off-by: Steve Longerbeam 
---
Changes in v2:
- bring back the media_device_for_each_entity() loop but check the
  stream_count not the use_count.
---
 drivers/media/platform/rcar-vin/rcar-core.c | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-core.c 
b/drivers/media/platform/rcar-vin/rcar-core.c
index f0719ce24b97..6dd6b11c1b2b 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -131,9 +131,13 @@ static int rvin_group_link_notify(struct media_link *link, 
u32 flags,
!is_media_entity_v4l2_video_device(link->sink->entity))
return 0;
 
-   /* If any entity is in use don't allow link changes. */
+   /*
+* Don't allow link changes if any entity in the graph is
+* streaming, because modifying the CHSEL register fields
+* can disrupt running streams.
+*/
media_device_for_each_entity(entity, &group->mdev)
-   if (entity->use_count)
+   if (entity->stream_count)
return -EBUSY;
 
mutex_lock(&group->lock);
-- 
2.17.1

Re: [PATCH] media: imx: queue subdev events to reachable video devices

[Steve Longerbeam]

Hi Hans,

On 1/8/19 5:26 AM, Hans Verkuil wrote:

On 12/09/18 20:57, Steve Longerbeam wrote:

From: Steve Longerbeam 

Forward events from a sub-device to its list of reachable video
devices.

Note this will queue the event to a video device even if there is
no actual _enabled_ media path from the sub-device to the video device.
So a future improvement is to skip the video device if there is no enabled
path to it from the sub-device. The entity->pipe pointer can't be
used for this check because in imx-media a sub-device can be a
member to more than one streaming pipeline at a time.

You explain what you are doing, but I am missing an explanation of *why*
this is needed.


Ok, I'll add more explanation.




Signed-off-by: Steve Longerbeam 
---
  drivers/staging/media/imx/imx-media-capture.c | 18 ++
  drivers/staging/media/imx/imx-media-dev.c | 24 +++
  2 files changed, 42 insertions(+)

diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index b37e1186eb2f..4dfbe05d203e 100644
--- a/drivers/staging/media/imx/imx-media-capture.c
+++ b/drivers/staging/media/imx/imx-media-capture.c
@@ -335,6 +335,21 @@ static int capture_s_parm(struct file *file, void *fh,
return 0;
  }
  
+static int capture_subscribe_event(struct v4l2_fh *fh,

+  const struct v4l2_event_subscription *sub)
+{
+   switch (sub->type) {
+   case V4L2_EVENT_IMX_FRAME_INTERVAL_ERROR:
+   return v4l2_event_subscribe(fh, sub, 0, NULL);
+   case V4L2_EVENT_SOURCE_CHANGE:
+   return v4l2_src_change_event_subscribe(fh, sub);
+   case V4L2_EVENT_CTRL:
+   return v4l2_ctrl_subscribe_event(fh, sub);
+   default:
+   return -EINVAL;
+   }
+}
+
  static const struct v4l2_ioctl_ops capture_ioctl_ops = {
.vidioc_querycap= vidioc_querycap,
  
@@ -362,6 +377,9 @@ static const struct v4l2_ioctl_ops capture_ioctl_ops = {

.vidioc_expbuf  = vb2_ioctl_expbuf,
.vidioc_streamon= vb2_ioctl_streamon,
.vidioc_streamoff   = vb2_ioctl_streamoff,
+
+   .vidioc_subscribe_event = capture_subscribe_event,
+   .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
  };

This part of the patch adds event support to the capture device, can't this be
split up into a separate patch? It seems to be useful in its own right.


Ok, I will split this up. The first patch will add the 
(un)subscribe_event callbacks to the capture device, and the second 
patch will forward subdev events.


  
  /*

diff --git a/drivers/staging/media/imx/imx-media-dev.c 
b/drivers/staging/media/imx/imx-media-dev.c
index 4b344a4a3706..25e916562c66 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -442,6 +442,29 @@ static const struct media_device_ops imx_media_md_ops = {
.link_notify = imx_media_link_notify,
  };
  
+static void imx_media_notify(struct v4l2_subdev *sd,

+unsigned int notification,
+void *arg)
+{
+   struct media_entity *entity = &sd->entity;
+   int i;
+
+   if (notification != V4L2_DEVICE_NOTIFY_EVENT)
+   return;
+
+   for (i = 0; i < entity->num_pads; i++) {
+   struct media_pad *pad = &entity->pads[i];
+   struct imx_media_pad_vdev *pad_vdev;
+   struct list_head *pad_vdev_list;
+
+   pad_vdev_list = to_pad_vdev_list(sd, pad->index);
+   if (!pad_vdev_list)
+   continue;
+   list_for_each_entry(pad_vdev, pad_vdev_list, list)
+   v4l2_event_queue(pad_vdev->vdev->vfd, arg);

Which events do you want to forward?


Shouldn't any/all subdev events be forwarded?

I would also prefer to allow userland to subscribe to any events that a 
subdevice might generate. In other words, imx-media will allow the user 
to subscribe to, and receive any events that might be generated by 
subdevices.




  E.g. forwarding control events
doesn't seem right, but other events may be useful.


The imx capture devices inherit controls from the subdevices. If an 
inherited control is changed via the capture device, will _two_ control 
events be generated (one from the subdevice and one from the capture 
device)? Or will only one event get generated to the capture device? 
Same question goes when changing an inherited control from the subdevice 
nodes.




  Are those events
also appearing on the v4l-subdevX device? And if so, should they?


How do we know what subdevices a future imx-based system might attach to 
the media graph? That's why I think it makes sense to forward any/all 
events from subdevices.


Steve




+   }
+}
+
  static int imx_media_probe(struct platform_device *pdev)
  {
struct device *dev = 

Re: [PATCH v6] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

Please disregard. This patch can't be submitted stand-alone, I will 
re-submit as part of a v6 of "imx-media: Fixes for interlaced capture" 
patchset.


Steve


On 12/14/18 3:46 PM, Steve Longerbeam wrote:

The function ipu_csi_init_interface() was inverting the F-bit for
NTSC case, in the CCIR_CODE_1/2 registers. The result being that
for NTSC bottom-top field order, the CSI would swap fields and
capture in top-bottom order.

Instead, base field swap on the field order of the input to the CSI,
and the field order of the requested output. If the input/output
fields are sequential but different, swap fields, otherwise do
not swap. This requires passing both the input and output mbus
frame formats to ipu_csi_init_interface().

Move this code to a new private function ipu_csi_set_bt_interlaced_codes()
that programs the CCIR_CODE_1/2 registers for interlaced BT.656 (and
possibly interlaced BT.1120 in the future).

When detecting input video standard from the input frame width/height,
make sure to double height if input field type is alternate, since
in that case input height only includes lines for one field.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
Changes since v5:
- Convert to const the infmt, outfmt, and mbus_cfg pointer args to
   ipu_csi_init_interface(), suggested by Philipp Zabel.
- Bring back if_fmt local var and don't copy outfmt to local stack in
   csi_setup(), suggested by Philipp.

Changes since v4:
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
   by Philipp.
- Fixed a regression in csi_setup(), caught by Philipp.
---
  drivers/gpu/ipu-v3/ipu-csi.c  | 126 +++---
  drivers/staging/media/imx/imx-media-csi.c |   7 +-
  include/video/imx-ipu-v3.h|   5 +-
  3 files changed, 89 insertions(+), 49 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index aa0e30a2ba18..d1e575571a8d 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -325,12 +325,21 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config 
*cfg, u32 mbus_code,
return 0;
  }
  
+/* translate alternate field mode based on given standard */

+static inline enum v4l2_field
+ipu_csi_translate_field(enum v4l2_field field, v4l2_std_id std)
+{
+   return (field != V4L2_FIELD_ALTERNATE) ? field :
+   ((std & V4L2_STD_525_60) ?
+V4L2_FIELD_SEQ_BT : V4L2_FIELD_SEQ_TB);
+}
+
  /*
   * Fill a CSI bus config struct from mbus_config and mbus_framefmt.
   */
  static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
-struct v4l2_mbus_config *mbus_cfg,
-struct v4l2_mbus_framefmt *mbus_fmt)
+   const struct v4l2_mbus_config *mbus_cfg,
+   const struct v4l2_mbus_framefmt *mbus_fmt)
  {
int ret;
  
@@ -374,22 +383,76 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,

return 0;
  }
  
+static int

+ipu_csi_set_bt_interlaced_codes(struct ipu_csi *csi,
+   const struct v4l2_mbus_framefmt *infmt,
+   const struct v4l2_mbus_framefmt *outfmt,
+   v4l2_std_id std)
+{
+   enum v4l2_field infield, outfield;
+   bool swap_fields;
+
+   /* get translated field type of input and output */
+   infield = ipu_csi_translate_field(infmt->field, std);
+   outfield = ipu_csi_translate_field(outfmt->field, std);
+
+   /*
+* Write the H-V-F codes the CSI will match against the
+* incoming data for start/end of active and blanking
+* field intervals. If input and output field types are
+* sequential but not the same (one is SEQ_BT and the other
+* is SEQ_TB), swap the F-bit so that the CSI will capture
+* field 1 lines before field 0 lines.
+*/
+   swap_fields = (V4L2_FIELD_IS_SEQUENTIAL(infield) &&
+  V4L2_FIELD_IS_SEQUENTIAL(outfield) &&
+  infield != outfield);
+
+   if (!swap_fields) {
+   /*
+* Field0BlankEnd  = 110, Field0BlankStart  = 010
+* Field0ActiveEnd = 100, Field0ActiveStart = 000
+* Field1BlankEnd  = 111, Field1BlankStart  = 011
+* Field1ActiveEnd = 101, Field1ActiveStart = 001
+*/
+   ipu_csi_write(csi, 0x40596 | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0xD07DF, CSI_CCIR_CODE_2);
+   } else {
+   dev_dbg(csi->ipu->dev, "capture field swap\n");
+
+   /* same as above but with F-bit inverted */
+   ipu_csi_write(csi, 0xD07DF | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0x40596, CSI

[PATCH v6 04/12] media: imx: Fix field negotiation

[Steve Longerbeam]

IDMAC interlaced scan, a.k.a. interweave, should be enabled in the
IDMAC output channels only if the IDMAC output pad field type is
'seq-bt' or 'seq-tb', and field type at the capture interface is
'interlaced*'.

V4L2_FIELD_HAS_BOTH() macro should not be used on the input to determine
enabling interlaced/interweave scan. That macro includes the 'interlaced'
field types, and in those cases the data is already interweaved with
top/bottom field lines.

The CSI will capture whole frames when the source specifies alternate
field mode. So the CSI also enables interweave for alternate input
field type and the field type at capture interface is interlaced.

Fix the logic for setting field type in try_fmt in CSI entity.
The behavior should be:

- No restrictions on field type at sink pad.

- At the output pads, allow sequential fields in TB order, if the sink pad
  field type is sequential or alternate. Otherwise passthrough the field
  type from sink to source pad.

Move this logic to new function csi_try_field().

These changes result in the following allowed field transformations
from CSI sink -> source pads (all other field types at sink are passed
through to source):

seq-tb -> seq-tb
seq-bt -> seq-tb
alternate -> seq-tb

In a future patch, the CSI sink -> source will allow:

seq-tb -> seq-bt
seq-bt -> seq-bt
alternate -> seq-bt

This will require supporting interweave with top/bottom line swapping.
Until then seq-bt is not allowed at the CSI source pad because there is
no way to swap top/bottom lines when interweaving to INTERLACED_BT --
note that despite the name, INTERLACED_BT is top-bottom order in memory.
The BT in this case refers to field dominance: the bottom lines are
older in time than the top lines.

The capture interface device allows selecting IDMAC interweave by
choosing INTERLACED_TB if the CSI/PRPENCVF source pad is seq-tb and
INTERLACED_BT if the source pad is seq-bt (for future support of seq-bt).

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-ic-prpencvf.c   | 21 --
 drivers/staging/media/imx/imx-media-capture.c | 14 
 drivers/staging/media/imx/imx-media-csi.c | 64 ++-
 3 files changed, 76 insertions(+), 23 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index af7224846bd5..1a03d4c9d7b8 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -354,12 +354,13 @@ static int prp_setup_channel(struct prp_priv *priv,
 {
struct imx_media_video_dev *vdev = priv->vdev;
const struct imx_media_pixfmt *outcc;
-   struct v4l2_mbus_framefmt *infmt;
+   struct v4l2_mbus_framefmt *outfmt;
unsigned int burst_size;
struct ipu_image image;
+   bool interweave;
int ret;
 
-   infmt = &priv->format_mbus[PRPENCVF_SINK_PAD];
+   outfmt = &priv->format_mbus[PRPENCVF_SRC_PAD];
outcc = vdev->cc;
 
ipu_cpmem_zero(channel);
@@ -369,6 +370,15 @@ static int prp_setup_channel(struct prp_priv *priv,
image.rect.width = image.pix.width;
image.rect.height = image.pix.height;
 
+   /*
+* If the field type at capture interface is interlaced, and
+* the output IDMAC pad is sequential, enable interweave at
+* the IDMAC output channel.
+*/
+   interweave = V4L2_FIELD_IS_INTERLACED(image.pix.field) &&
+   V4L2_FIELD_IS_SEQUENTIAL(outfmt->field) &&
+   channel == priv->out_ch;
+
if (rot_swap_width_height) {
swap(image.pix.width, image.pix.height);
swap(image.rect.width, image.rect.height);
@@ -409,9 +419,7 @@ static int prp_setup_channel(struct prp_priv *priv,
if (rot_mode)
ipu_cpmem_set_rotation(channel, rot_mode);
 
-   if (image.pix.field == V4L2_FIELD_NONE &&
-   V4L2_FIELD_HAS_BOTH(infmt->field) &&
-   channel == priv->out_ch)
+   if (interweave)
ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline,
  image.pix.pixelformat);
 
@@ -839,8 +847,7 @@ static void prp_try_fmt(struct prp_priv *priv,
infmt = __prp_get_fmt(priv, cfg, PRPENCVF_SINK_PAD, sdformat->which);
 
if (sdformat->pad == PRPENCVF_SRC_PAD) {
-   if (sdformat->format.field != V4L2_FIELD_NONE)
-   sdformat->format.field = infmt->field;
+   sdformat->format.field = infmt->field;
 
prp_bound_align_output(&sdformat->format, infmt,
   priv->rot_mode);
diff --git a/drivers/staging/media/imx/imx-media-capture.c 
b/drivers/staging/media/imx/imx-media-capture.c
index b37e1186eb2f..01ec9443de55 100644
--- a/d

[PATCH v6 12/12] media: imx.rst: Update doc to reflect fixes to interlaced capture

[Steve Longerbeam]

Also add an example pipeline for unconverted capture with interweave
on SabreAuto.

Cleanup some language in various places in the process.

Signed-off-by: Steve Longerbeam 
---
Changes since v4:
- Make clear that it is IDMAC channel that does pixel reordering and
  interweave, not the CSI. Caught by Philipp Zabel.
Changes since v3:
- none.
Changes since v2:
- expand on idmac interweave behavior in CSI subdev.
- switch second SabreAuto pipeline example to PAL to give
  both NTSC and PAL examples.
- Cleanup some language in various places.
---
 Documentation/media/v4l-drivers/imx.rst | 103 +++-
 1 file changed, 66 insertions(+), 37 deletions(-)

diff --git a/Documentation/media/v4l-drivers/imx.rst 
b/Documentation/media/v4l-drivers/imx.rst
index 6922dde4a82b..9314af00d067 100644
--- a/Documentation/media/v4l-drivers/imx.rst
+++ b/Documentation/media/v4l-drivers/imx.rst
@@ -24,8 +24,8 @@ memory. Various dedicated DMA channels exist for both video 
capture and
 display paths. During transfer, the IDMAC is also capable of vertical
 image flip, 8x8 block transfer (see IRT description), pixel component
 re-ordering (for example UYVY to YUYV) within the same colorspace, and
-even packed <--> planar conversion. It can also perform a simple
-de-interlacing by interleaving even and odd lines during transfer
+packed <--> planar conversion. The IDMAC can also perform a simple
+de-interlacing by interweaving even and odd lines during transfer
 (without motion compensation which requires the VDIC).
 
 The CSI is the backend capture unit that interfaces directly with
@@ -175,15 +175,21 @@ via the SMFC and an IDMAC channel, bypassing IC 
pre-processing. This
 source pad is routed to a capture device node, with a node name of the
 format "ipuX_csiY capture".
 
-Note that since the IDMAC source pad makes use of an IDMAC channel, it
-can do pixel reordering within the same colorspace. For example, the
-sink pad can take UYVY2X8, but the IDMAC source pad can output YUYV2X8.
-If the sink pad is receiving YUV, the output at the capture device can
-also be converted to a planar YUV format such as YUV420.
-
-It will also perform simple de-interlace without motion compensation,
-which is activated if the sink pad's field type is an interlaced type,
-and the IDMAC source pad field type is set to none.
+Note that since the IDMAC source pad makes use of an IDMAC channel,
+pixel reordering within the same colorspace can be carried out by the
+IDMAC channel. For example, if the CSI sink pad is receiving in UYVY
+order, the capture device linked to the IDMAC source pad can capture
+in YUYV order. Also, if the CSI sink pad is receiving a packed YUV
+format, the capture device can capture a planar YUV format such as
+YUV420.
+
+The IDMAC channel at the IDMAC source pad also supports simple
+interweave without motion compensation, which is activated if the source
+pad's field type is sequential top-bottom or bottom-top, and the
+requested capture interface field type is set to interlaced (t-b, b-t,
+or unqualified interlaced). The capture interface will enforce the same
+field order as the source pad field order (interlaced-bt if source pad
+is seq-bt, interlaced-tb if source pad is seq-tb).
 
 This subdev can generate the following event when enabling the second
 IDMAC source pad:
@@ -325,14 +331,14 @@ ipuX_vdic
 
 The VDIC carries out motion compensated de-interlacing, with three
 motion compensation modes: low, medium, and high motion. The mode is
-specified with the menu control V4L2_CID_DEINTERLACING_MODE. It has
-two sink pads and a single source pad.
+specified with the menu control V4L2_CID_DEINTERLACING_MODE. The VDIC
+has two sink pads and a single source pad.
 
 The direct sink pad receives from an ipuX_csiY direct pad. With this
 link the VDIC can only operate in high motion mode.
 
 When the IDMAC sink pad is activated, it receives from an output
-or mem2mem device node. With this pipeline, it can also operate
+or mem2mem device node. With this pipeline, the VDIC can also operate
 in low and medium modes, because these modes require receiving
 frames from memory buffers. Note that an output or mem2mem device
 is not implemented yet, so this sink pad currently has no links.
@@ -345,8 +351,8 @@ ipuX_ic_prp
 This is the IC pre-processing entity. It acts as a router, routing
 data from its sink pad to one or both of its source pads.
 
-It has a single sink pad. The sink pad can receive from the ipuX_csiY
-direct pad, or from ipuX_vdic.
+This entity has a single sink pad. The sink pad can receive from the
+ipuX_csiY direct pad, or from ipuX_vdic.
 
 This entity has two source pads. One source pad routes to the
 pre-process encode task entity (ipuX_ic_prpenc), the other to the
@@ -369,8 +375,8 @@ color-space conversion, resizing (downscaling and 
upscaling),
 horizontal and vertical flip, and 90/270 degree rotation. Flip
 and rotation are provided via standard V4L2 controls.
 

[PATCH v6 10/12] media: imx-csi: Move crop/compose reset after filling default mbus fields

[Steve Longerbeam]

If caller passes un-initialized field type V4L2_FIELD_ANY to CSI
sink pad, the reset CSI crop window would not be correct, because
the crop window depends on a valid input field type. To fix move
the reset of crop and compose windows to after the call to
imx_media_fill_default_mbus_fields().

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-media-csi.c | 27 ---
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 47568ec43f4a..7afb7e367d76 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1409,19 +1409,6 @@ static void csi_try_fmt(struct csi_priv *priv,
  W_ALIGN, &sdformat->format.height,
  MIN_H, MAX_H, H_ALIGN, S_ALIGN);
 
-   /* Reset crop and compose rectangles */
-   crop->left = 0;
-   crop->top = 0;
-   crop->width = sdformat->format.width;
-   crop->height = sdformat->format.height;
-   if (sdformat->format.field == V4L2_FIELD_ALTERNATE)
-   crop->height *= 2;
-   csi_try_crop(priv, crop, cfg, &sdformat->format, upstream_ep);
-   compose->left = 0;
-   compose->top = 0;
-   compose->width = crop->width;
-   compose->height = crop->height;
-
*cc = imx_media_find_mbus_format(sdformat->format.code,
 CS_SEL_ANY, true);
if (!*cc) {
@@ -1437,6 +1424,20 @@ static void csi_try_fmt(struct csi_priv *priv,
imx_media_fill_default_mbus_fields(
&sdformat->format, infmt,
priv->active_output_pad == CSI_SRC_PAD_DIRECT);
+
+   /* Reset crop and compose rectangles */
+   crop->left = 0;
+   crop->top = 0;
+   crop->width = sdformat->format.width;
+   crop->height = sdformat->format.height;
+   if (sdformat->format.field == V4L2_FIELD_ALTERNATE)
+   crop->height *= 2;
+   csi_try_crop(priv, crop, cfg, &sdformat->format, upstream_ep);
+   compose->left = 0;
+   compose->top = 0;
+   compose->width = crop->width;
+   compose->height = crop->height;
+
break;
}
 }
-- 
2.17.1

[PATCH v6 11/12] media: imx: Allow interweave with top/bottom lines swapped

[Steve Longerbeam]

Allow sequential->interlaced interweaving but with top/bottom
lines swapped to the output buffer.

This can be accomplished by adding one line length to IDMAC output
channel address, with a negative line length for the interlace offset.

This is to allow the seq-bt -> interlaced-bt transformation, where
bottom lines are still dominant (older in time) but with top lines
first in the interweaved output buffer.

With this support, the CSI can now allow seq-bt at its source pads,
e.g. the following transformations are allowed in CSI from sink to
source:

seq-tb -> seq-bt
seq-bt -> seq-bt
alternate -> seq-bt

Suggested-by: Philipp Zabel 
Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
Changes since v4:
- Removed interweave_offset and replace with boolean interweave_swap,
  suggested by Philipp Zabel.
---
 drivers/staging/media/imx/imx-ic-prpencvf.c | 25 +
 drivers/staging/media/imx/imx-media-csi.c   | 40 ++---
 2 files changed, 54 insertions(+), 11 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index cf76b0432371..33ada6612fee 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -106,6 +106,7 @@ struct prp_priv {
u32 frame_sequence; /* frame sequence counter */
bool last_eof;  /* waiting for last EOF at stream off */
bool nfb4eof;/* NFB4EOF encountered during streaming */
+   bool interweave_swap; /* swap top/bottom lines when interweaving */
struct completion last_eof_comp;
 };
 
@@ -235,6 +236,9 @@ static void prp_vb2_buf_done(struct prp_priv *priv, struct 
ipuv3_channel *ch)
if (ipu_idmac_buffer_is_ready(ch, priv->ipu_buf_num))
ipu_idmac_clear_buffer(ch, priv->ipu_buf_num);
 
+   if (priv->interweave_swap && ch == priv->out_ch)
+   phys += vdev->fmt.fmt.pix.bytesperline;
+
ipu_cpmem_set_buffer(ch, priv->ipu_buf_num, phys);
 }
 
@@ -376,8 +380,9 @@ static int prp_setup_channel(struct prp_priv *priv,
 * the IDMAC output channel.
 */
interweave = V4L2_FIELD_IS_INTERLACED(image.pix.field) &&
-   V4L2_FIELD_IS_SEQUENTIAL(outfmt->field) &&
-   channel == priv->out_ch;
+   V4L2_FIELD_IS_SEQUENTIAL(outfmt->field);
+   priv->interweave_swap = interweave &&
+   image.pix.field == V4L2_FIELD_INTERLACED_BT;
 
if (rot_swap_width_height) {
swap(image.pix.width, image.pix.height);
@@ -388,6 +393,11 @@ static int prp_setup_channel(struct prp_priv *priv,
(image.pix.width * outcc->bpp) >> 3;
}
 
+   if (priv->interweave_swap && channel == priv->out_ch) {
+   /* start interweave scan at 1st top line (2nd line) */
+   image.rect.top = 1;
+   }
+
image.phys0 = addr0;
image.phys1 = addr1;
 
@@ -396,8 +406,8 @@ static int prp_setup_channel(struct prp_priv *priv,
 * channels for planar 4:2:0 (but not when enabling IDMAC
 * interweaving, they are incompatible).
 */
-   if (!interweave && (channel == priv->out_ch ||
-   channel == priv->rot_out_ch)) {
+   if ((channel == priv->out_ch && !interweave) ||
+   channel == priv->rot_out_ch) {
switch (image.pix.pixelformat) {
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_YVU420:
@@ -424,8 +434,11 @@ static int prp_setup_channel(struct prp_priv *priv,
if (rot_mode)
ipu_cpmem_set_rotation(channel, rot_mode);
 
-   if (interweave)
-   ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline,
+   if (interweave && channel == priv->out_ch)
+   ipu_cpmem_interlaced_scan(channel,
+ priv->interweave_swap ?
+ -image.pix.bytesperline :
+ image.pix.bytesperline,
  image.pix.pixelformat);
 
ret = ipu_ic_task_idma_init(priv->ic, channel,
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index 7afb7e367d76..e18f58f56dfb 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -114,6 +114,7 @@ struct csi_priv {
u32 frame_sequence; /* frame sequence counter */
bool last_eof;   /* waiting for last EOF at stream off */
bool nfb4eof;/* NFB4EOF encountered during streaming */
+   bool interweave_swap; /* swap top/bottom lines when interweaving */
struct completion last_eof_comp;
 };
 
@@ -286,6 +287,9 @@ static void csi_vb2_buf_done(struct csi_priv *priv)
 

[PATCH v6 00/12] imx-media: Fixes for interlaced capture

[Steve Longerbeam]

From: Steve Longerbeam 

A set of patches that fixes some bugs with capturing from an
interlaced source, and incompatibilites between IDMAC interlace
interweaving and 4:2:0 data write reduction.

History:
v6:
- Changes to patch "gpu: ipu-csi: Swap fields according to input/output
  field types" suggested by Philipp Zabel.

v5:
- Added a regression fix to allow empty endpoints to CSI (fix for imx6q
  SabreAuto).
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
  by Philipp Zabel.
- Fixed a regression in csi_setup(), caught by Philipp.
- Removed interweave_offset and replace with boolean interweave_swap,
  suggested by Philipp.
- Make clear that it is IDMAC channel that does pixel reordering and
  interweave, not the CSI, in the imx.rst doc, caught by Philipp.

v4:
- rebased to latest media-tree master branch.
- Make patch author and SoB email addresses the same.

v3:
- add support for/fix interweaved scan with YUV planar output.
- fix bug in 4:2:0 U/V offset macros.
- add patch that generalizes behavior of field swap in
  ipu_csi_init_interface().
- add support for interweaved scan with field order swap.
  Suggested by Philipp Zabel.
- in v2, inteweave scan was determined using field types of
  CSI (and PRPENCVF) at the sink and source pads. In v3, this
  has been moved one hop downstream: interweave is now determined
  using field type at source pad, and field type selected at
  capture interface. Suggested by Philipp.
- make sure to double CSI crop target height when input field
  type in alternate.
- more updates to media driver doc to reflect above.

v2:
- update media driver doc.
- enable idmac interweave only if input field is sequential/alternate,
  and output field is 'interlaced*'.
- move field try logic out of *try_fmt and into separate function.
- fix bug with resetting crop/compose rectangles.
- add a patch that fixes a field order bug in VDIC indirect mode.
- remove alternate field type from V4L2_FIELD_IS_SEQUENTIAL() macro
  Suggested-by: Nicolas Dufresne .
- add macro V4L2_FIELD_IS_INTERLACED().


Steve Longerbeam (12):
  media: videodev2.h: Add more field helper macros
  gpu: ipu-csi: Swap fields according to input/output field types
  gpu: ipu-v3: Add planar support to interlaced scan
  media: imx: Fix field negotiation
  media: imx-csi: Input connections to CSI should be optional
  media: imx-csi: Double crop height for alternate fields at sink
  media: imx: interweave and odd-chroma-row skip are incompatible
  media: imx-csi: Allow skipping odd chroma rows for YVU420
  media: imx: vdic: rely on VDIC for correct field order
  media: imx-csi: Move crop/compose reset after filling default mbus
fields
  media: imx: Allow interweave with top/bottom lines swapped
  media: imx.rst: Update doc to reflect fixes to interlaced capture

 Documentation/media/v4l-drivers/imx.rst   | 103 
 drivers/gpu/ipu-v3/ipu-cpmem.c|  26 ++-
 drivers/gpu/ipu-v3/ipu-csi.c  | 126 +-
 drivers/staging/media/imx/imx-ic-prpencvf.c   |  46 +++--
 drivers/staging/media/imx/imx-media-capture.c |  14 ++
 drivers/staging/media/imx/imx-media-csi.c | 158 +-
 drivers/staging/media/imx/imx-media-vdic.c|  12 +-
 include/uapi/linux/videodev2.h|   7 +
 include/video/imx-ipu-v3.h|   8 +-
 9 files changed, 355 insertions(+), 145 deletions(-)

-- 
2.17.1

[PATCH v6 08/12] media: imx-csi: Allow skipping odd chroma rows for YVU420

[Steve Longerbeam]

Skip writing U/V components to odd rows for YVU420 in addition to
YUV420 and NV12.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-media-csi.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index dbc5a92ec073..47568ec43f4a 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -452,6 +452,7 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
passthrough_bits = 16;
break;
case V4L2_PIX_FMT_YUV420:
+   case V4L2_PIX_FMT_YVU420:
case V4L2_PIX_FMT_NV12:
burst_size = (image.pix.width & 0x3f) ?
 ((image.pix.width & 0x1f) ?
-- 
2.17.1

[PATCH v6 09/12] media: imx: vdic: rely on VDIC for correct field order

[Steve Longerbeam]

prepare_vdi_in_buffers() was setting up the dma pointers as if the
VDIC is always programmed to receive the fields in bottom-top order,
i.e. as if ipu_vdi_set_field_order() only programs BT order in the VDIC.
But that's not true, ipu_vdi_set_field_order() is working correctly.

So fix prepare_vdi_in_buffers() to give the VDIC the fields in whatever
order they were received by the video source, and rely on the VDIC to
sort out which is top and which is bottom.

Signed-off-by: Steve Longerbeam 
---
 drivers/staging/media/imx/imx-media-vdic.c | 12 ++--
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-vdic.c 
b/drivers/staging/media/imx/imx-media-vdic.c
index 482250d47e7c..4a890714193e 100644
--- a/drivers/staging/media/imx/imx-media-vdic.c
+++ b/drivers/staging/media/imx/imx-media-vdic.c
@@ -219,26 +219,18 @@ static void __maybe_unused prepare_vdi_in_buffers(struct 
vdic_priv *priv,
 
switch (priv->fieldtype) {
case V4L2_FIELD_SEQ_TB:
-   prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0);
-   curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + fs;
-   next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
-   break;
case V4L2_FIELD_SEQ_BT:
prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0) + fs;
curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + fs;
break;
+   case V4L2_FIELD_INTERLACED_TB:
case V4L2_FIELD_INTERLACED_BT:
+   case V4L2_FIELD_INTERLACED:
prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0) + is;
curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + is;
break;
-   default:
-   /* assume V4L2_FIELD_INTERLACED_TB */
-   prev_phys = vb2_dma_contig_plane_dma_addr(prev_vb, 0);
-   curr_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0) + is;
-   next_phys = vb2_dma_contig_plane_dma_addr(curr_vb, 0);
-   break;
}
 
ipu_cpmem_set_buffer(priv->vdi_in_ch_p, 0, prev_phys);
-- 
2.17.1

[PATCH v6 07/12] media: imx: interweave and odd-chroma-row skip are incompatible

[Steve Longerbeam]

If IDMAC interweaving is enabled in a write channel, the channel must
write the odd chroma rows for 4:2:0 formats. Skipping writing the odd
chroma rows produces corrupted captured 4:2:0 images when interweave
is enabled.

Reported-by: Krzysztof Hałasa 
Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-ic-prpencvf.c | 9 +++--
 drivers/staging/media/imx/imx-media-csi.c   | 8 ++--
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index 1a03d4c9d7b8..cf76b0432371 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -391,12 +391,17 @@ static int prp_setup_channel(struct prp_priv *priv,
image.phys0 = addr0;
image.phys1 = addr1;
 
-   if (channel == priv->out_ch || channel == priv->rot_out_ch) {
+   /*
+* Skip writing U and V components to odd rows in the output
+* channels for planar 4:2:0 (but not when enabling IDMAC
+* interweaving, they are incompatible).
+*/
+   if (!interweave && (channel == priv->out_ch ||
+   channel == priv->rot_out_ch)) {
switch (image.pix.pixelformat) {
case V4L2_PIX_FMT_YUV420:
case V4L2_PIX_FMT_YVU420:
case V4L2_PIX_FMT_NV12:
-   /* Skip writing U and V components to odd rows */
ipu_cpmem_skip_odd_chroma_rows(channel);
break;
}
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index c40c3262038e..dbc5a92ec073 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -457,8 +457,12 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
 ((image.pix.width & 0x1f) ?
  ((image.pix.width & 0xf) ? 8 : 16) : 32) : 64;
passthrough_bits = 16;
-   /* Skip writing U and V components to odd rows */
-   ipu_cpmem_skip_odd_chroma_rows(priv->idmac_ch);
+   /*
+* Skip writing U and V components to odd rows (but not
+* when enabling IDMAC interweaving, they are incompatible).
+*/
+   if (!interweave)
+   ipu_cpmem_skip_odd_chroma_rows(priv->idmac_ch);
break;
case V4L2_PIX_FMT_YUYV:
case V4L2_PIX_FMT_UYVY:
-- 
2.17.1

[PATCH v6 02/12] gpu: ipu-csi: Swap fields according to input/output field types

[Steve Longerbeam]

The function ipu_csi_init_interface() was inverting the F-bit for
NTSC case, in the CCIR_CODE_1/2 registers. The result being that
for NTSC bottom-top field order, the CSI would swap fields and
capture in top-bottom order.

Instead, base field swap on the field order of the input to the CSI,
and the field order of the requested output. If the input/output
fields are sequential but different, swap fields, otherwise do
not swap. This requires passing both the input and output mbus
frame formats to ipu_csi_init_interface().

Move this code to a new private function ipu_csi_set_bt_interlaced_codes()
that programs the CCIR_CODE_1/2 registers for interlaced BT.656 (and
possibly interlaced BT.1120 in the future).

When detecting input video standard from the input frame width/height,
make sure to double height if input field type is alternate, since
in that case input height only includes lines for one field.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
Changes since v5:
- Convert to const the infmt, outfmt, and mbus_cfg pointer args to
  ipu_csi_init_interface(), suggested by Philipp Zabel.
- Bring back if_fmt local var and don't copy outfmt to local stack in
  csi_setup(), suggested by Philipp.

Changes since v4:
- Cleaned up some convoluted code in ipu_csi_init_interface(), suggested
  by Philipp.
- Fixed a regression in csi_setup(), caught by Philipp.
---
 drivers/gpu/ipu-v3/ipu-csi.c  | 126 +++---
 drivers/staging/media/imx/imx-media-csi.c |   7 +-
 include/video/imx-ipu-v3.h|   5 +-
 3 files changed, 89 insertions(+), 49 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index aa0e30a2ba18..d1e575571a8d 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -325,12 +325,21 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config 
*cfg, u32 mbus_code,
return 0;
 }
 
+/* translate alternate field mode based on given standard */
+static inline enum v4l2_field
+ipu_csi_translate_field(enum v4l2_field field, v4l2_std_id std)
+{
+   return (field != V4L2_FIELD_ALTERNATE) ? field :
+   ((std & V4L2_STD_525_60) ?
+V4L2_FIELD_SEQ_BT : V4L2_FIELD_SEQ_TB);
+}
+
 /*
  * Fill a CSI bus config struct from mbus_config and mbus_framefmt.
  */
 static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
-struct v4l2_mbus_config *mbus_cfg,
-struct v4l2_mbus_framefmt *mbus_fmt)
+   const struct v4l2_mbus_config *mbus_cfg,
+   const struct v4l2_mbus_framefmt *mbus_fmt)
 {
int ret;
 
@@ -374,22 +383,76 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config 
*csicfg,
return 0;
 }
 
+static int
+ipu_csi_set_bt_interlaced_codes(struct ipu_csi *csi,
+   const struct v4l2_mbus_framefmt *infmt,
+   const struct v4l2_mbus_framefmt *outfmt,
+   v4l2_std_id std)
+{
+   enum v4l2_field infield, outfield;
+   bool swap_fields;
+
+   /* get translated field type of input and output */
+   infield = ipu_csi_translate_field(infmt->field, std);
+   outfield = ipu_csi_translate_field(outfmt->field, std);
+
+   /*
+* Write the H-V-F codes the CSI will match against the
+* incoming data for start/end of active and blanking
+* field intervals. If input and output field types are
+* sequential but not the same (one is SEQ_BT and the other
+* is SEQ_TB), swap the F-bit so that the CSI will capture
+* field 1 lines before field 0 lines.
+*/
+   swap_fields = (V4L2_FIELD_IS_SEQUENTIAL(infield) &&
+  V4L2_FIELD_IS_SEQUENTIAL(outfield) &&
+  infield != outfield);
+
+   if (!swap_fields) {
+   /*
+* Field0BlankEnd  = 110, Field0BlankStart  = 010
+* Field0ActiveEnd = 100, Field0ActiveStart = 000
+* Field1BlankEnd  = 111, Field1BlankStart  = 011
+* Field1ActiveEnd = 101, Field1ActiveStart = 001
+*/
+   ipu_csi_write(csi, 0x40596 | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0xD07DF, CSI_CCIR_CODE_2);
+   } else {
+   dev_dbg(csi->ipu->dev, "capture field swap\n");
+
+   /* same as above but with F-bit inverted */
+   ipu_csi_write(csi, 0xD07DF | CSI_CCIR_ERR_DET_EN,
+ CSI_CCIR_CODE_1);
+   ipu_csi_write(csi, 0x40596, CSI_CCIR_CODE_2);
+   }
+
+   ipu_csi_write(csi, 0xFF, CSI_CCIR_CODE_3);
+
+   return 0;
+}
+
+
 int ipu_csi_init_interface(struct ipu_csi *csi,
-  struct v4l2_mbus_config *mbus_cfg,
-  

[PATCH v6 05/12] media: imx-csi: Input connections to CSI should be optional

[Steve Longerbeam]

Some imx platforms do not have fwnode connections to all CSI input
ports, and should not be treated as an error. This includes the
imx6q SabreAuto, which has no connections to ipu1_csi1 and ipu2_csi0.
Return -ENOTCONN in imx_csi_parse_endpoint() so that v4l2-fwnode
endpoint parsing will not treat an unconnected endpoint as an error.

Fixes: c893500a16baf ("media: imx: csi: Register a subdev notifier")

Signed-off-by: Steve Longerbeam 
---
 drivers/staging/media/imx/imx-media-csi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index e3a4f39dbf73..b276672cae1d 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1815,7 +1815,7 @@ static int imx_csi_parse_endpoint(struct device *dev,
  struct v4l2_fwnode_endpoint *vep,
  struct v4l2_async_subdev *asd)
 {
-   return fwnode_device_is_available(asd->match.fwnode) ? 0 : -EINVAL;
+   return fwnode_device_is_available(asd->match.fwnode) ? 0 : -ENOTCONN;
 }
 
 static int imx_csi_async_register(struct csi_priv *priv)
-- 
2.17.1

[PATCH v6 06/12] media: imx-csi: Double crop height for alternate fields at sink

[Steve Longerbeam]

If the incoming sink field type is alternate, the reset crop height
and crop height bounds must be set to twice the incoming height,
because in alternate field mode, upstream will report only the
lines for a single field, and the CSI captures the whole frame.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
---
 drivers/staging/media/imx/imx-media-csi.c | 20 +++-
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index b276672cae1d..c40c3262038e 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1142,6 +1142,8 @@ static void csi_try_crop(struct csi_priv *priv,
 struct v4l2_mbus_framefmt *infmt,
 struct v4l2_fwnode_endpoint *upstream_ep)
 {
+   u32 in_height;
+
crop->width = min_t(__u32, infmt->width, crop->width);
if (crop->left + crop->width > infmt->width)
crop->left = infmt->width - crop->width;
@@ -1149,6 +1151,10 @@ static void csi_try_crop(struct csi_priv *priv,
crop->left &= ~0x3;
crop->width &= ~0x7;
 
+   in_height = infmt->height;
+   if (infmt->field == V4L2_FIELD_ALTERNATE)
+   in_height *= 2;
+
/*
 * FIXME: not sure why yet, but on interlaced bt.656,
 * changing the vertical cropping causes loss of vertical
@@ -1158,12 +1164,12 @@ static void csi_try_crop(struct csi_priv *priv,
if (upstream_ep->bus_type == V4L2_MBUS_BT656 &&
(V4L2_FIELD_HAS_BOTH(infmt->field) ||
 infmt->field == V4L2_FIELD_ALTERNATE)) {
-   crop->height = infmt->height;
-   crop->top = (infmt->height == 480) ? 2 : 0;
+   crop->height = in_height;
+   crop->top = (in_height == 480) ? 2 : 0;
} else {
-   crop->height = min_t(__u32, infmt->height, crop->height);
-   if (crop->top + crop->height > infmt->height)
-   crop->top = infmt->height - crop->height;
+   crop->height = min_t(__u32, in_height, crop->height);
+   if (crop->top + crop->height > in_height)
+   crop->top = in_height - crop->height;
}
 }
 
@@ -1403,6 +1409,8 @@ static void csi_try_fmt(struct csi_priv *priv,
crop->top = 0;
crop->width = sdformat->format.width;
crop->height = sdformat->format.height;
+   if (sdformat->format.field == V4L2_FIELD_ALTERNATE)
+   crop->height *= 2;
csi_try_crop(priv, crop, cfg, &sdformat->format, upstream_ep);
compose->left = 0;
compose->top = 0;
@@ -1530,6 +1538,8 @@ static int csi_get_selection(struct v4l2_subdev *sd,
sel->r.top = 0;
sel->r.width = infmt->width;
sel->r.height = infmt->height;
+   if (infmt->field == V4L2_FIELD_ALTERNATE)
+   sel->r.height *= 2;
break;
case V4L2_SEL_TGT_CROP:
sel->r = *crop;
-- 
2.17.1

[PATCH v6 01/12] media: videodev2.h: Add more field helper macros

[Steve Longerbeam]

Adds two helper macros:

V4L2_FIELD_IS_SEQUENTIAL: returns true if the given field type is
'sequential', that is a full frame is transmitted, or exists in
memory, as all top field lines followed by all bottom field lines,
or vice-versa.

V4L2_FIELD_IS_INTERLACED: returns true if the given field type is
'interlaced', that is a full frame is transmitted, or exists in
memory, as top field lines interlaced with bottom field lines.

Signed-off-by: Steve Longerbeam 
---
Changes since v3:
- none
Changes since v2:
- none
Changes since v1:
- add the complement macro V4L2_FIELD_IS_INTERLACED
- remove V4L2_FIELD_ALTERNATE from V4L2_FIELD_IS_SEQUENTIAL macro.
- moved new macros past end of existing V4L2_FIELD_HAS_* macros.
---
 include/uapi/linux/videodev2.h | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index d6eed479c3a6..bca07d35ea09 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -130,6 +130,13 @@ enum v4l2_field {
((field) == V4L2_FIELD_BOTTOM ||\
 (field) == V4L2_FIELD_TOP ||\
 (field) == V4L2_FIELD_ALTERNATE)
+#define V4L2_FIELD_IS_INTERLACED(field) \
+   ((field) == V4L2_FIELD_INTERLACED ||\
+(field) == V4L2_FIELD_INTERLACED_TB ||\
+(field) == V4L2_FIELD_INTERLACED_BT)
+#define V4L2_FIELD_IS_SEQUENTIAL(field) \
+   ((field) == V4L2_FIELD_SEQ_TB ||\
+(field) == V4L2_FIELD_SEQ_BT)
 
 enum v4l2_buf_type {
V4L2_BUF_TYPE_VIDEO_CAPTURE= 1,
-- 
2.17.1

[PATCH v6 03/12] gpu: ipu-v3: Add planar support to interlaced scan

[Steve Longerbeam]

To support interlaced scan with planar formats, cpmem SLUV must
be programmed with the correct chroma line stride. For full and
partial planar 4:2:2 (YUV422P, NV16), chroma line stride must
be doubled. For full and partial planar 4:2:0 (YUV420, YVU420, NV12),
chroma line stride must _not_ be doubled, since a single chroma line
is shared by two luma lines.

Signed-off-by: Steve Longerbeam 
Reviewed-by: Philipp Zabel 
Acked-by: Philipp Zabel 
---
 drivers/gpu/ipu-v3/ipu-cpmem.c  | 26 +++--
 drivers/staging/media/imx/imx-ic-prpencvf.c |  3 ++-
 drivers/staging/media/imx/imx-media-csi.c   |  3 ++-
 include/video/imx-ipu-v3.h  |  3 ++-
 4 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/ipu-v3/ipu-cpmem.c b/drivers/gpu/ipu-v3/ipu-cpmem.c
index 163fadb8a33a..d047a6867c59 100644
--- a/drivers/gpu/ipu-v3/ipu-cpmem.c
+++ b/drivers/gpu/ipu-v3/ipu-cpmem.c
@@ -277,9 +277,10 @@ void ipu_cpmem_set_uv_offset(struct ipuv3_channel *ch, u32 
u_off, u32 v_off)
 }
 EXPORT_SYMBOL_GPL(ipu_cpmem_set_uv_offset);
 
-void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride)
+void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride,
+  u32 pixelformat)
 {
-   u32 ilo, sly;
+   u32 ilo, sly, sluv;
 
if (stride < 0) {
stride = -stride;
@@ -290,9 +291,30 @@ void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, 
int stride)
 
sly = (stride * 2) - 1;
 
+   switch (pixelformat) {
+   case V4L2_PIX_FMT_YUV420:
+   case V4L2_PIX_FMT_YVU420:
+   sluv = stride / 2 - 1;
+   break;
+   case V4L2_PIX_FMT_NV12:
+   sluv = stride - 1;
+   break;
+   case V4L2_PIX_FMT_YUV422P:
+   sluv = stride - 1;
+   break;
+   case V4L2_PIX_FMT_NV16:
+   sluv = stride * 2 - 1;
+   break;
+   default:
+   sluv = 0;
+   break;
+   }
+
ipu_ch_param_write_field(ch, IPU_FIELD_SO, 1);
ipu_ch_param_write_field(ch, IPU_FIELD_ILO, ilo);
ipu_ch_param_write_field(ch, IPU_FIELD_SLY, sly);
+   if (sluv)
+   ipu_ch_param_write_field(ch, IPU_FIELD_SLUV, sluv);
 };
 EXPORT_SYMBOL_GPL(ipu_cpmem_interlaced_scan);
 
diff --git a/drivers/staging/media/imx/imx-ic-prpencvf.c 
b/drivers/staging/media/imx/imx-ic-prpencvf.c
index 28f41caba05d..af7224846bd5 100644
--- a/drivers/staging/media/imx/imx-ic-prpencvf.c
+++ b/drivers/staging/media/imx/imx-ic-prpencvf.c
@@ -412,7 +412,8 @@ static int prp_setup_channel(struct prp_priv *priv,
if (image.pix.field == V4L2_FIELD_NONE &&
V4L2_FIELD_HAS_BOTH(infmt->field) &&
channel == priv->out_ch)
-   ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+   ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline,
+ image.pix.pixelformat);
 
ret = ipu_ic_task_idma_init(priv->ic, channel,
image.pix.width, image.pix.height,
diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index c2a8d9cd31b7..da4808348845 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -512,7 +512,8 @@ static int csi_idmac_setup_channel(struct csi_priv *priv)
if (image.pix.field == V4L2_FIELD_NONE &&
V4L2_FIELD_HAS_BOTH(infmt->field))
ipu_cpmem_interlaced_scan(priv->idmac_ch,
- image.pix.bytesperline);
+ image.pix.bytesperline,
+ image.pix.pixelformat);
 
ipu_idmac_set_double_buffer(priv->idmac_ch, true);
 
diff --git a/include/video/imx-ipu-v3.h b/include/video/imx-ipu-v3.h
index bbc8481f567d..c887f4bee5f8 100644
--- a/include/video/imx-ipu-v3.h
+++ b/include/video/imx-ipu-v3.h
@@ -258,7 +258,8 @@ void ipu_cpmem_set_stride(struct ipuv3_channel *ch, int 
stride);
 void ipu_cpmem_set_high_priority(struct ipuv3_channel *ch);
 void ipu_cpmem_set_buffer(struct ipuv3_channel *ch, int bufnum, dma_addr_t 
buf);
 void ipu_cpmem_set_uv_offset(struct ipuv3_channel *ch, u32 u_off, u32 v_off);
-void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride);
+void ipu_cpmem_interlaced_scan(struct ipuv3_channel *ch, int stride,
+  u32 pixelformat);
 void ipu_cpmem_set_axi_id(struct ipuv3_channel *ch, u32 id);
 int ipu_cpmem_get_burstsize(struct ipuv3_channel *ch);
 void ipu_cpmem_set_burstsize(struct ipuv3_channel *ch, int burstsize);
-- 
2.17.1

Re: [PATCH v6 05/12] media: imx-csi: Input connections to CSI should be optional

[Steve Longerbeam]

On 1/9/19 3:13 AM, Philipp Zabel wrote:

On Tue, 2019-01-08 at 16:15 -0800, Steve Longerbeam wrote:

Some imx platforms do not have fwnode connections to all CSI input
ports, and should not be treated as an error. This includes the
imx6q SabreAuto, which has no connections to ipu1_csi1 and ipu2_csi0.
Return -ENOTCONN in imx_csi_parse_endpoint() so that v4l2-fwnode
endpoint parsing will not treat an unconnected endpoint as an error.

Fixes: c893500a16baf ("media: imx: csi: Register a subdev notifier")

Signed-off-by: Steve Longerbeam 
---
  drivers/staging/media/imx/imx-media-csi.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/media/imx/imx-media-csi.c 
b/drivers/staging/media/imx/imx-media-csi.c
index e3a4f39dbf73..b276672cae1d 100644
--- a/drivers/staging/media/imx/imx-media-csi.c
+++ b/drivers/staging/media/imx/imx-media-csi.c
@@ -1815,7 +1815,7 @@ static int imx_csi_parse_endpoint(struct device *dev,
  struct v4l2_fwnode_endpoint *vep,
  struct v4l2_async_subdev *asd)
  {
-   return fwnode_device_is_available(asd->match.fwnode) ? 0 : -EINVAL;
+   return fwnode_device_is_available(asd->match.fwnode) ? 0 : -ENOTCONN;
  }
  
  static int imx_csi_async_register(struct csi_priv *priv)

Is this something that should be applied as a fix, separately from this
series?


Yes. I'll submit it separately and send a v7 of this series without it.

Steve