[
https://issues.apache.org/jira/browse/SPARK-30332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040560#comment-17040560
]
Hyukjin Kwon commented on SPARK-30332:
--------------------------------------
Where is error message, and how do you create the tables?
There are many ways to narrow down the problem. You can remove the columns one
by one, and see if it still reproduces the issue.
Simplifying the multiple CSV files. Unifying the file names for readability.
> When running sql query with limit catalyst throw StackOverFlow exception
> -------------------------------------------------------------------------
>
> Key: SPARK-30332
> URL: https://issues.apache.org/jira/browse/SPARK-30332
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Environment: spark version 3.0.0-preview
> Reporter: Izek Greenfield
> Priority: Major
> Attachments: AGGR_41380.csv, AGGR_41390.csv, AGGR_41406.csv,
> AGGR_41406.csv, AGGR_41410.csv, AGGR_41418.csv, PORTFOLIO_41446.csv,
> T_41233.csv
>
>
> Running that SQL:
> {code:sql}
> SELECT BT_capital.asof_date,
> BT_capital.run_id,
> BT_capital.v,
> BT_capital.id,
> BT_capital.entity,
> BT_capital.level_1,
> BT_capital.level_2,
> BT_capital.level_3,
> BT_capital.level_4,
> BT_capital.level_5,
> BT_capital.level_6,
> BT_capital.path_bt_capital,
> BT_capital.line_item,
> t0.target_line_item,
> t0.line_description,
> BT_capital.col_item,
> BT_capital.rep_amount,
> root.orgUnitId,
> root.cptyId,
> root.instId,
> root.startDate,
> root.maturityDate,
> root.amount,
> root.nominalAmount,
> root.quantity,
> root.lkupAssetLiability,
> root.lkupCurrency,
> root.lkupProdType,
> root.interestResetDate,
> root.interestResetTerm,
> root.noticePeriod,
> root.historicCostAmount,
> root.dueDate,
> root.lkupResidence,
> root.lkupCountryOfUltimateRisk,
> root.lkupSector,
> root.lkupIndustry,
> root.lkupAccountingPortfolioType,
> root.lkupLoanDepositTerm,
> root.lkupFixedFloating,
> root.lkupCollateralType,
> root.lkupRiskType,
> root.lkupEligibleRefinancing,
> root.lkupHedging,
> root.lkupIsOwnIssued,
> root.lkupIsSubordinated,
> root.lkupIsQuoted,
> root.lkupIsSecuritised,
> root.lkupIsSecuritisedServiced,
> root.lkupIsSyndicated,
> root.lkupIsDeRecognised,
> root.lkupIsRenegotiated,
> root.lkupIsTransferable,
> root.lkupIsNewBusiness,
> root.lkupIsFiduciary,
> root.lkupIsNonPerforming,
> root.lkupIsInterGroup,
> root.lkupIsIntraGroup,
> root.lkupIsRediscounted,
> root.lkupIsCollateral,
> root.lkupIsExercised,
> root.lkupIsImpaired,
> root.facilityId,
> root.lkupIsOTC,
> root.lkupIsDefaulted,
> root.lkupIsSavingsPosition,
> root.lkupIsForborne,
> root.lkupIsDebtRestructuringLoan,
> root.interestRateAAR,
> root.interestRateAPRC,
> root.custom1,
> root.custom2,
> root.custom3,
> root.lkupSecuritisationType,
> root.lkupIsCashPooling,
> root.lkupIsEquityParticipationGTE10,
> root.lkupIsConvertible,
> root.lkupEconomicHedge,
> root.lkupIsNonCurrHeldForSale,
> root.lkupIsEmbeddedDerivative,
> root.lkupLoanPurpose,
> root.lkupRegulated,
> root.lkupRepaymentType,
> root.glAccount,
> root.lkupIsRecourse,
> root.lkupIsNotFullyGuaranteed,
> root.lkupImpairmentStage,
> root.lkupIsEntireAmountWrittenOff,
> root.lkupIsLowCreditRisk,
> root.lkupIsOBSWithinIFRS9,
> root.lkupIsUnderSpecialSurveillance,
> root.lkupProtection,
> root.lkupIsGeneralAllowance,
> root.lkupSectorUltimateRisk,
> root.cptyOrgUnitId,
> root.name,
> root.lkupNationality,
> root.lkupSize,
> root.lkupIsSPV,
> root.lkupIsCentralCounterparty,
> root.lkupIsMMRMFI,
> root.lkupIsKeyManagement,
> root.lkupIsOtherRelatedParty,
> root.lkupResidenceProvince,
> root.lkupIsTradingBook,
> root.entityHierarchy_entityId,
> root.entityHierarchy_Residence,
> root.lkupLocalCurrency,
> root.cpty_entityhierarchy_entityId,
> root.lkupRelationship,
> root.cpty_lkupRelationship,
> root.entityNationality,
> root.lkupRepCurrency,
> root.startDateFinancialYear,
> root.numEmployees,
> root.numEmployeesTotal,
> root.collateralAmount,
> root.guaranteeAmount,
> root.impairmentSpecificIndividual,
> root.impairmentSpecificCollective,
> root.impairmentGeneral,
> root.creditRiskAmount,
> root.provisionSpecificIndividual,
> root.provisionSpecificCollective,
> root.provisionGeneral,
> root.writeOffAmount,
> root.interest,
> root.fairValueAmount,
> root.grossCarryingAmount,
> root.carryingAmount,
> root.code,
> root.lkupInstrumentType,
> root.price,
> root.amountAtIssue,
> root.yield,
> root.totalFacilityAmount,
> root.facility_rate,
> root.spec_indiv_est,
> root.spec_coll_est,
> root.coll_inc_loss,
> root.impairment_amount,
> root.provision_amount,
> root.accumulated_impairment,
> root.exclusionFlag,
> root.lkupIsHoldingCompany,
> root.instrument_startDate,
> root.entityResidence,
> fxRate.enumerator,
> fxRate.lkupFromCurrency,
> fxRate.rate,
> fxRate.custom1,
> fxRate.custom2,
> fxRate.custom3,
> GB_position.lkupIsECGDGuaranteed,
> GB_position.lkupIsMultiAcctOffsetMortgage,
> GB_position.lkupIsIndexLinked,
> GB_position.lkupIsRetail,
> GB_position.lkupCollateralLocation,
> GB_position.percentAboveBBR,
> GB_position.lkupIsMoreInArrears,
> GB_position.lkupIsArrearsCapitalised,
> GB_position.lkupCollateralPossession,
> GB_position.lkupIsLifetimeMortgage,
> GB_position.lkupLoanConcessionType,
> GB_position.lkupIsMultiCurrency,
> GB_position.lkupIsJointIncomeBasis,
> GB_position.ratioIncomeMultiple,
> GB_position.interestRate,
> GB_position.exclusionFlag,
> GB_position.lkupFDIDirection,
> GB_position.lkupIsRTGS,
> GB_positionExtended.nonRecourseFinanceAmount,
> GB_positionExtended.arrearsAmount,
> GB_Counterparty.lkupIsClearingFirm,
> GB_Counterparty.lkupIsIntermediateFinCorp,
> GB_Counterparty.lkupIsImpairedCreditHistory,
> GB_Counterparty.lkupFDIRelationship FROM portfolio_41446 BT_capital
> JOIN aggr_41390 root ON (root.id = BT_capital.id AND root.entity =
> BT_capital.entity AND (root.instance_id =
> 'e3b82807-9371-44f4-9c97-d63cde21dbf2'))
> JOIN aggr_41380 fxRate ON (fxRate.lkupToCurrency = root.lkupRepCurrency AND
> (root.lkupCurrency = fxRate.lkupFromCurrency) AND (fxRate.instance_id =
> '641f3e44-8d0a-4040-8f26-1d459c5a47a3'))
> LEFT OUTER JOIN aggr_41418 GB_position ON (GB_position.id = root.id AND
> (GB_position.instance_id = 'a8eb71eb-906b-4b76-a977-589618f5f1b5'))
> LEFT OUTER JOIN aggr_41406 GB_positionExtended ON (GB_positionExtended.id =
> GB_position.id AND (GB_positionExtended.instance_id =
> 'bc7415ca-8cda-4c64-a323-fdfe72b00d54'))
> LEFT OUTER JOIN aggr_41410 GB_Counterparty ON (GB_Counterparty.id =
> root.cptyId AND (GB_Counterparty.instance_id =
> '6e0d1229-c142-4411-81e2-c1cc5f636843'))
> LEFT OUTER JOIN t_41233 t0 ON (t0.line_item = BT_capital.line_item)
> WHERE ((BT_capital.instance_id = '4fa9a3a5-67e2-4c78-9708-76849d424123')) AND
> (root.exclusionFlag!='yes'
> ) AND (GB_position.exclusionFlag!='yes')
> AND
> ((((NVL(t0.target_line_item,BT_capital.line_item) = '10') AND
> (BT_capital.col_item = 'Euro') AND
> (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '10') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '10') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '11A') AND
> (BT_capital.col_item = 'Euro') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '11A') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '11A') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '11B') AND
> (BT_capital.col_item = 'Euro') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '11B') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '11B') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19A') AND
> (BT_capital.col_item = 'Euro') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19A') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19A') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19B') AND
> (BT_capital.col_item = 'Euro') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19B') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19B') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19BL') AND
> (BT_capital.col_item = 'Euro') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19BL') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19BL') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19CCA') AND
> (BT_capital.col_item = 'Euro') AND (BT_capital.entity = 'E002')) OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19CCA') AND
> (BT_capital.col_item = 'Other currencies') AND (BT_capital.entity = 'E002'))
> OR
> ((NVL(t0.target_line_item,BT_capital.line_item) = '19CCA') AND
> (BT_capital.col_item = 'Sterling') AND (BT_capital.entity = 'E002'))) AND
> (BT_capital.level_1 != 'Orphans')) limit 101
> {code}
> I get a StackOverflow exception. remove the LIMIT at the end and all is
> working.
> with spark version 2.4.4 it does not happen
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
