[jira] [Comment Edited] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-19 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383707#comment-17383707
 ] 

Hyukjin Kwon edited comment on SPARK-36218 at 7/20/21, 2:44 AM:


cc [~maropu], [~cloud_fan], [~dongjoon] FYI.

Actually, I faced this issue in our internal repo a while ago, and just added a 
hacky fix by adding an explicit GC:

{code}
  if (tpcdsDataPath.nonEmpty) {
tpcdsQueries
  .foreach { name =>
  val queryString = resourceToString(s"tpcds/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(name) {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v1_4", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
tpcdsQueriesV2_7_0
  .foreach { name =>
  val queryString = resourceToString(s"tpcds-v2.7.0/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(s"$name-v2.7") {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v2_7", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
  } else {
ignore("skipped because env `SPARK_TPCDS_DATA` is not set") {}
  }
}
{code}

Oh wait, let me take this back. TPC-DS became flaky now in our internal repo 
even with the fix ^.


was (Author: hyukjin.kwon):
cc [~maropu], [~cloud_fan], [~dongjoon] FYI.

Actually, I faced this issue in our internal repo a while ago, and just added a 
hacky fix by adding an explicit GC:

{code}
  if (tpcdsDataPath.nonEmpty) {
tpcdsQueries
  .foreach { name =>
  val queryString = resourceToString(s"tpcds/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(name) {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v1_4", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
tpcdsQueriesV2_7_0
  .foreach { name =>
  val queryString = resourceToString(s"tpcds-v2.7.0/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(s"$name-v2.7") {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v2_7", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
  } else {
ignore("skipped because env `SPARK_TPCDS_DATA` is not set") {}
  }
}
{code}

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-19 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383707#comment-17383707
 ] 

Hyukjin Kwon edited comment on SPARK-36218 at 7/20/21, 2:41 AM:


cc [~maropu], [~cloud_fan], [~dongjoon] FYI.

Actually, I faced this issue in our internal repo a while ago, and just added a 
hacky fix by adding an explicit GC:

{code}
  if (tpcdsDataPath.nonEmpty) {
tpcdsQueries
  .foreach { name =>
  val queryString = resourceToString(s"tpcds/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(name) {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v1_4", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
tpcdsQueriesV2_7_0
  .foreach { name =>
  val queryString = resourceToString(s"tpcds-v2.7.0/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(s"$name-v2.7") {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v2_7", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
  } else {
ignore("skipped because env `SPARK_TPCDS_DATA` is not set") {}
  }
}
{code}


was (Author: hyukjin.kwon):
cc [~maropu], [~cloud_fan], [~dongjoon] FYI.

Actually, I faced this issue in our internal repo a while ago, and just added a 
hacky fix by adding an explicit GC:

{code}
  if (tpcdsDataPath.nonEmpty) {
tpcdsQueries
  .filter(_ != "q95") // TODO(SC-75125)
  .filter(_ != "q75") // TODO(SC-75127)
  .filter(_ != "q64") // TODO(SC-75126)
  .foreach { name =>
  val queryString = resourceToString(s"tpcds/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(name) {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v1_4", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
tpcdsQueriesV2_7_0
  .filter(_ != "q95") // TODO(SC-75125)
  .filter(_ != "q75") // TODO(SC-75127)
  .filter(_ != "q64") // TODO(SC-75126)
  .foreach { name =>
  val queryString = resourceToString(s"tpcds-v2.7.0/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(s"$name-v2.7") {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v2_7", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
  } else {
ignore("skipped because env `SPARK_TPCDS_DATA` is not set") {}
  }
}
{code}

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org