[jira] [Created] (SPARK-41415) SASL Request Retry
Aravind Patnam created SPARK-41415: -- Summary: SASL Request Retry Key: SPARK-41415 URL: https://issues.apache.org/jira/browse/SPARK-41415 Project: Spark Issue Type: Task Components: Shuffle Affects Versions: 3.2.4 Reporter: Aravind Patnam -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41415) SASL Request Retries
[ https://issues.apache.org/jira/browse/SPARK-41415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravind Patnam updated SPARK-41415: --- Summary: SASL Request Retries (was: SASL Request Retry) > SASL Request Retries > > > Key: SPARK-41415 > URL: https://issues.apache.org/jira/browse/SPARK-41415 > Project: Spark > Issue Type: Task > Components: Shuffle >Affects Versions: 3.2.4 >Reporter: Aravind Patnam >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2
Aravind Patnam created SPARK-36926:
--
Summary: Discrepancy in Q22 of TPCH for Spark 3.2
Key: SPARK-36926
URL: https://issues.apache.org/jira/browse/SPARK-36926
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.0
Reporter: Aravind Patnam
When running TPCH scale 100 against 3.2, Query 22 has a discrepancy in the
number of rows returned by the query. This was tested with both AQE on and off.
All the other queries were matching in results. Below is the results that we
got when testing Q22 on 3.2:
{code:java}
"results": [
{
"name": "Q22",
"mode": "collect",
"parameters": {},
"joinTypes": [
"SortMergeJoin"
],
"tables": [
"customer"
],
"parsingTime": 0.016522,
"analysisTime": 0.004132,
"optimizationTime": 39.173868,
"planningTime": 23.10939,
"executionTime": 13762.183844,
"result": 0,
"breakDown": [],
"queryExecution": "== Parsed Logical Plan ==\n'Sort ['cntrycode ASC NULLS
FIRST], true\n+- 'Aggregate ['cntrycode], ['cntrycode, 'count(1) AS
numcust#150, 'sum('c_acctbal) AS totacctbal#151]\n +- 'SubqueryAlias
custsale\n +- 'Project ['substring('c_phone, 1, 2) AS cntrycode#147,
'c_acctbal]\n +- 'Filter (('substring('c_phone, 1, 2) IN
(13,31,23,29,30,18,17) AND ('c_acctbal > scalar-subquery#148 [])) AND NOT
exists#149 [])\n: :- 'Project [unresolvedalias('avg('c_acctbal),
None)]\n: : +- 'Filter (('c_acctbal > 0.00) AND
'substring('c_phone, 1, 2) IN (13,31,23,29,30,18,17))\n: : +-
'UnresolvedRelation [customer], [], false\n: +- 'Project [*]\n
: +- 'Filter ('o_custkey = 'c_custkey)\n:+-
'UnresolvedRelation [orders], [], false\n+- 'UnresolvedRelation
[customer], [], false\n\n== Analyzed Logical Plan ==\ncntrycode: string,
numcust: bigint, totacctbal: decimal(22,2)\nSort [cntrycode#147 ASC NULLS
FIRST], true\n+- Aggregate [cntrycode#147], [cntrycode#147, count(1) AS
numcust#150L, sum(c_acctbal#11) AS totacctbal#151]\n +- SubqueryAlias
custsale\n +- Project [substring(c_phone#10, 1, 2) AS cntrycode#147,
c_acctbal#11]\n +- Filter ((substring(c_phone#10, 1, 2) IN
(13,31,23,29,30,18,17) AND (cast(c_acctbal#11 as decimal(16,6)) >
cast(scalar-subquery#148 [] as decimal(16,6 AND NOT exists#149
[c_custkey#6L])\n: :- Aggregate [avg(c_acctbal#160) AS
avg(c_acctbal)#154]\n: : +- Filter ((cast(c_acctbal#160 as
decimal(12,2)) > cast(0.00 as decimal(12,2))) AND substring(c_phone#159, 1, 2)
IN (13,31,23,29,30,18,17))\n: : +- SubqueryAlias
spark_catalog.tpch_data_orc_100.customer\n: :+- Relation
tpch_data_orc_100.customer[c_custkey#155L,c_name#156,c_address#157,c_nationkey#158L,c_phone#159,c_acctbal#160,c_comment#161,c_mktsegment#162]
orc\n: +- Project [o_orderkey#16L, o_custkey#17L,
o_orderstatus#18, o_totalprice#19, o_orderpriority#20, o_clerk#21,
o_shippriority#22, o_comment#23, o_orderdate#24]\n: +- Filter
(o_custkey#17L = outer(c_custkey#6L))\n:+- SubqueryAlias
spark_catalog.tpch_data_orc_100.orders\n: +- Relation
tpch_data_orc_100.orders[o_orderkey#16L,o_custkey#17L,o_orderstatus#18,o_totalprice#19,o_orderpriority#20,o_clerk#21,o_shippriority#22,o_comment#23,o_orderdate#24]
orc\n+- SubqueryAlias spark_catalog.tpch_data_orc_100.customer\n
+- Relation
tpch_data_orc_100.customer[c_custkey#6L,c_name#7,c_address#8,c_nationkey#9L,c_phone#10,c_acctbal#11,c_comment#12,c_mktsegment#13]
orc\n\n== Optimized Logical Plan ==\nSort [cntrycode#147 ASC NULLS FIRST],
true\n+- Aggregate [cntrycode#147], [cntrycode#147, count(1) AS numcust#150L,
sum(c_acctbal#11) AS totacctbal#151]\n +- Project [substring(c_phone#10, 1,
2) AS cntrycode#147, c_acctbal#11]\n +- Join LeftAnti, (o_custkey#17L =
c_custkey#6L)\n :- Project [c_custkey#6L, c_phone#10, c_acctbal#11]\n
: +- Filter ((isnotnull(c_acctbal#11) AND substring(c_phone#10, 1, 2) IN
(13,31,23,29,30,18,17)) AND (cast(c_acctbal#11 as decimal(16,6)) >
scalar-subquery#148 []))\n : : +- Aggregate [avg(c_acctbal#160) AS
avg(c_acctbal)#154]\n : : +- Project [c_acctbal#160]\n
: :+- Filter (isnotnull(c_acctbal#160) AND ((c_acctbal#160 > 0.00)
AND substring(c_phone#159, 1, 2) IN (13,31,23,29,30,18,17)))\n : :
+- Relation
tpch_data_orc_100.customer[c_custkey#155L,c_name#156,c_address#157,c_nationkey#158L,c_phone#159,c_acctbal#160,c_comment#161,c_mktsegment#162]
orc\n : +- Relation
tpch_data_orc_100.customer[c_custkey#6L,c_name#7,c_address#8,c_nationkey#9L,c_phone#10,c_acctbal#11,c_comment#12,c_mk
[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2
[
https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424058#comment-17424058
]
Aravind Patnam commented on SPARK-36926:
cc: [~vsowrirajan] [~mridulm80]
> Discrepancy in Q22 of TPCH for Spark 3.2
>
>
> Key: SPARK-36926
> URL: https://issues.apache.org/jira/browse/SPARK-36926
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Aravind Patnam
>Priority: Major
>
> When running TPCH scale 100 against 3.2, Query 22 has a discrepancy in the
> number of rows returned by the query. This was tested with both AQE on and
> off. All the other queries were matching in results. Below is the results
> that we got when testing Q22 on 3.2:
>
> {code:java}
> "results": [
> {
> "name": "Q22",
> "mode": "collect",
> "parameters": {},
> "joinTypes": [
> "SortMergeJoin"
> ],
> "tables": [
> "customer"
> ],
> "parsingTime": 0.016522,
> "analysisTime": 0.004132,
> "optimizationTime": 39.173868,
> "planningTime": 23.10939,
> "executionTime": 13762.183844,
> "result": 0,
> "breakDown": [],
> "queryExecution": "== Parsed Logical Plan ==\n'Sort ['cntrycode ASC
> NULLS FIRST], true\n+- 'Aggregate ['cntrycode], ['cntrycode, 'count(1) AS
> numcust#150, 'sum('c_acctbal) AS totacctbal#151]\n +- 'SubqueryAlias
> custsale\n +- 'Project ['substring('c_phone, 1, 2) AS cntrycode#147,
> 'c_acctbal]\n +- 'Filter (('substring('c_phone, 1, 2) IN
> (13,31,23,29,30,18,17) AND ('c_acctbal > scalar-subquery#148 [])) AND NOT
> exists#149 [])\n: :- 'Project [unresolvedalias('avg('c_acctbal),
> None)]\n: : +- 'Filter (('c_acctbal > 0.00) AND
> 'substring('c_phone, 1, 2) IN (13,31,23,29,30,18,17))\n: :
> +- 'UnresolvedRelation [customer], [], false\n: +- 'Project
> [*]\n: +- 'Filter ('o_custkey = 'c_custkey)\n:
> +- 'UnresolvedRelation [orders], [], false\n+-
> 'UnresolvedRelation [customer], [], false\n\n== Analyzed Logical Plan
> ==\ncntrycode: string, numcust: bigint, totacctbal: decimal(22,2)\nSort
> [cntrycode#147 ASC NULLS FIRST], true\n+- Aggregate [cntrycode#147],
> [cntrycode#147, count(1) AS numcust#150L, sum(c_acctbal#11) AS
> totacctbal#151]\n +- SubqueryAlias custsale\n +- Project
> [substring(c_phone#10, 1, 2) AS cntrycode#147, c_acctbal#11]\n +-
> Filter ((substring(c_phone#10, 1, 2) IN (13,31,23,29,30,18,17) AND
> (cast(c_acctbal#11 as decimal(16,6)) > cast(scalar-subquery#148 [] as
> decimal(16,6 AND NOT exists#149 [c_custkey#6L])\n: :-
> Aggregate [avg(c_acctbal#160) AS avg(c_acctbal)#154]\n: : +-
> Filter ((cast(c_acctbal#160 as decimal(12,2)) > cast(0.00 as decimal(12,2)))
> AND substring(c_phone#159, 1, 2) IN (13,31,23,29,30,18,17))\n: :
> +- SubqueryAlias spark_catalog.tpch_data_orc_100.customer\n:
> :+- Relation
> tpch_data_orc_100.customer[c_custkey#155L,c_name#156,c_address#157,c_nationkey#158L,c_phone#159,c_acctbal#160,c_comment#161,c_mktsegment#162]
> orc\n: +- Project [o_orderkey#16L, o_custkey#17L,
> o_orderstatus#18, o_totalprice#19, o_orderpriority#20, o_clerk#21,
> o_shippriority#22, o_comment#23, o_orderdate#24]\n: +- Filter
> (o_custkey#17L = outer(c_custkey#6L))\n:+- SubqueryAlias
> spark_catalog.tpch_data_orc_100.orders\n: +- Relation
> tpch_data_orc_100.orders[o_orderkey#16L,o_custkey#17L,o_orderstatus#18,o_totalprice#19,o_orderpriority#20,o_clerk#21,o_shippriority#22,o_comment#23,o_orderdate#24]
> orc\n+- SubqueryAlias spark_catalog.tpch_data_orc_100.customer\n
> +- Relation
> tpch_data_orc_100.customer[c_custkey#6L,c_name#7,c_address#8,c_nationkey#9L,c_phone#10,c_acctbal#11,c_comment#12,c_mktsegment#13]
> orc\n\n== Optimized Logical Plan ==\nSort [cntrycode#147 ASC NULLS FIRST],
> true\n+- Aggregate [cntrycode#147], [cntrycode#147, count(1) AS numcust#150L,
> sum(c_acctbal#11) AS totacctbal#151]\n +- Project [substring(c_phone#10, 1,
> 2) AS cntrycode#147, c_acctbal#11]\n +- Join LeftAnti, (o_custkey#17L =
> c_custkey#6L)\n :- Project [c_custkey#6L, c_phone#10, c_acctbal#11]\n
> : +- Filter ((isnotnull(c_acctbal#11) AND substring(c_phone#10, 1,
> 2) IN (13,31,23,29,30,18,17)) AND (cast(c_acctbal#11 as decimal(16,6)) >
> scalar-subquery#148 []))\n : : +- Aggregate [avg(c_acctbal#160)
> AS avg(c_acctbal)#154]\n : : +- Project [c_acctbal#160]\n
> : :+- Filter (isnotnull(c_acctbal#160) AND ((c
[jira] [Created] (SPARK-38356) Print resolved versions of platform JARs used by Spark in Yarn client
Aravind Patnam created SPARK-38356: -- Summary: Print resolved versions of platform JARs used by Spark in Yarn client Key: SPARK-38356 URL: https://issues.apache.org/jira/browse/SPARK-38356 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 3.3.0 Reporter: Aravind Patnam It would be nice if we could print the resolved versions. I think a sensible approach would be to resolve the symlinks in yarn.Client before uploading. This would also make the localized file name include the version number, so we could see it easily on the Environment SHS tab. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38356) Print resolved versions of platform JARs used by Spark in Yarn client
[ https://issues.apache.org/jira/browse/SPARK-38356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499214#comment-17499214 ] Aravind Patnam commented on SPARK-38356: working on this > Print resolved versions of platform JARs used by Spark in Yarn client > - > > Key: SPARK-38356 > URL: https://issues.apache.org/jira/browse/SPARK-38356 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.3.0 >Reporter: Aravind Patnam >Priority: Minor > > It would be nice if we could print the resolved versions. I think a sensible > approach would be to resolve the symlinks in yarn.Client before uploading. > This would also make the localized file name include the version number, so > we could see it easily on the Environment SHS tab. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36419) Move final aggregation in RDD.treeAggregate to executor
Aravind Patnam created SPARK-36419: -- Summary: Move final aggregation in RDD.treeAggregate to executor Key: SPARK-36419 URL: https://issues.apache.org/jira/browse/SPARK-36419 Project: Spark Issue Type: Improvement Components: Spark Core, Tests Affects Versions: 3.0.0 Reporter: Aravind Patnam For the last iteration in RDD.treeAggregate, spark relies on RDD.fold as an implementation detail. RDD.fold pulls all shuffle partitions to the driver to merge the result. There are two concerns with this: a) Shuffle machinery at executors is much more robust/fault tolerant compared to fetching results to driver. b) Driver is single point of failure in a spark application. When this results in nontrivial increase in memory pressure while pulling partitions to driver or increased memory usage as part of computing the aggregated state (in user code), it can result in driver failures. For treeAggregate, instead of relying on fold for the last iteration, we should (optionally) do the computation at a single reducer - and fetch the final result to driver. Additional cost: one extra stage with a single (resulting) partition. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
