Thanks Chen, will apply these suggestions as soon as I get past the cabac issues. As I mentioned, the patch is missing some fixes and results in CABAC errors whenever intra NxN is chosen in case that MinCbSizeY != 8.
Any suggestions on where else to look is greatly appreciated *Qiwei Wen* *Software Developer* *Exablaze Pty Ltd* ✆ 0430872689 LinkedIn <https://www.linkedin.com/in/qiwei-wen-9905578a/> On Fri, Dec 19, 2025, 2:58 AM chen <[email protected]> wrote: > Hi Wen, > > Thank for the patch. > > I have some comments, > > (1) access global variant is higher cost, how about change below? > cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] > (1 << cuGeom.log2CUSize) == m_param->minCUSize > > (2) looks mixed another fix on array index typo, suggest separate patch > m_lumaIntraDir[absPartIdx] > > Regards, > Chen > > > At 2025-12-18 21:26:30, "Qiwei Wen" <[email protected]> wrote: > >Dear Gurus, > > > >As you know, the HEVC spec allows NxN partition mode for intra-coded > >CUs when the CU is of size MinCbSizeY. I assume either for coding > >efficiency or historical reasons, x265 won't explore PART_NxN if the > >minimum CU size is not 8. I'm working on a project that involves using > >x265 to prototype intra-coding algorithms and it'd be ideal for my use > >case if the limitation could be lifted. > > > >I realise that there are a few places in the code base that assume > >that intra NxN implies a CU size of 8. Below is my attempt at fixing > >it, the result of which is a non-local CABAC error, where the first > >few intra NxN CU is coded correctly, but the CABAC state starts to > >diverge to the point of making the bitstream undecodeable a few CUs > >further down. > > > >I'd appreciate any pointers on if there are more places in the code > >that makes the assumption and what the possible causes could be. > > > >Thanks, > >Qiwei > > > >> diff --git a/source/encoder/analysis.cpp b/source/encoder/analysis.cpp > >> index b219d5da4..4c42faba6 100644 > >> --- a/source/encoder/analysis.cpp > >> +++ b/source/encoder/analysis.cpp > >> @@ -580,7 +580,7 @@ uint64_t Analysis::compressIntraCU(const CUData& > >> parentCTU, const CUGeom& cuGeom > >> checkIntra(md.pred[PRED_INTRA], cuGeom, SIZE_2Nx2N); > >> checkBestMode(md.pred[PRED_INTRA], depth); > >> > >> - if (cuGeom.log2CUSize == 3 && > >> m_slice->m_sps->quadtreeTULog2MinSize < 3) > >> + if (cuGeom.log2CUSize == g_log2Size[m_param->minCUSize] && > >> m_slice->m_sps->quadtreeTULog2MinSize < g_log2Size[m_param->minCUSize]) > >> { > >> md.pred[PRED_INTRA_NxN].cu.initSubCU(parentCTU, cuGeom, qp); > >> checkIntra(md.pred[PRED_INTRA_NxN], cuGeom, SIZE_NxN); > >> diff --git a/source/encoder/entropy.cpp b/source/encoder/entropy.cpp > >> index d4bf73d53..9aecbb5e5 100644 > >> --- a/source/encoder/entropy.cpp > >> +++ b/source/encoder/entropy.cpp > >> @@ -1328,7 +1328,7 @@ void Entropy::encodeTransform(const CUData& cu, > >> uint32_t absPartIdx, uint32_t cu > >> > >> /* in each of these conditions, the subdiv flag is implied and not > >> signaled, > >> * so we have checks to make sure the implied value matches our > >> intentions */ > >> - if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N > >> && log2CurSize == MIN_LOG2_CU_SIZE) > >> + if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N > >> && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize]) > >> { > >> X265_CHECK(subdiv, "intra NxN requires TU depth below CU > >> depth\n"); > >> } > >> @@ -1463,7 +1463,7 @@ void Entropy::encodeTransformLuma(const CUData& cu, > >> uint32_t absPartIdx, uint32_ > >> > >> /* in each of these conditions, the subdiv flag is implied and not > >> signaled, > >> * so we have checks to make sure the implied value matches our > >> intentions */ > >> - if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N > >> && log2CurSize == MIN_LOG2_CU_SIZE) > >> + if (cu.isIntra(absPartIdx) && cu.m_partSize[absPartIdx] != SIZE_2Nx2N > >> && log2CurSize == g_log2Size[cu.m_encData->m_param->minCUSize]) > >> { > >> X265_CHECK(subdiv, "intra NxN requires TU depth below CU > >> depth\n"); > >> } > >> diff --git a/source/encoder/search.cpp b/source/encoder/search.cpp > >> index 0522f52cc..3d61caabf 100644 > >> --- a/source/encoder/search.cpp > >> +++ b/source/encoder/search.cpp > >> @@ -1541,8 +1541,8 @@ sse_t Search::estIntraPredQT(Mode &intraMode, const > >> CUGeom& cuGeom, const uint32 > >> { > >> uint32_t bmode = 0; > >> > >> - if (intraMode.cu.m_lumaIntraDir[puIdx] != (uint8_t)ALL_IDX) > >> - bmode = intraMode.cu.m_lumaIntraDir[puIdx]; > >> + if (intraMode.cu.m_lumaIntraDir[absPartIdx] != (uint8_t)ALL_IDX) > >> + bmode = intraMode.cu.m_lumaIntraDir[absPartIdx]; > >> else > >> { > >> uint64_t candCostList[MAX_RD_INTRA_MODES]; > > > >-- > >Qiwei Wen > >Software Developer > >Exablaze Pty Ltd > >✆ 0430872689 > >LinkedIn > >_______________________________________________ > >x265-devel mailing list > >[email protected] > >https://mailman.videolan.org/listinfo/x265-devel > >
_______________________________________________ x265-devel mailing list [email protected] https://mailman.videolan.org/listinfo/x265-devel
