[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-25 Thread Takeshi Yamamuro (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386980#comment-17386980
 ] 

Takeshi Yamamuro commented on SPARK-36218:
--

Ah, okay. I got it now. Thanks.

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-25 Thread Takeshi Yamamuro (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386979#comment-17386979
 ] 

Takeshi Yamamuro commented on SPARK-36218:
--

Ah, okay. Is it okay to close this as resolved?

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-25 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386976#comment-17386976
 ] 

Hyukjin Kwon commented on SPARK-36218:
--

Oh actually I think this is now fixed via SPARK-36270

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-25 Thread Takeshi Yamamuro (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17386972#comment-17386972
 ] 

Takeshi Yamamuro commented on SPARK-36218:
--

Sorry for my late work.., has anyone tried to make  `spark.driver.memory` 
larger, e.g.,, 1g? Spark might not have enough memory for running 
TPCDSQueryTestSuite. Anyway, I'll check this today.

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-20 Thread Dongjoon Hyun (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384495#comment-17384495
 ] 

Dongjoon Hyun commented on SPARK-36218:
---

Oh, thank you for sharing, [~hyukjin.kwon].

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-19 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383709#comment-17383709
 ] 

Hyukjin Kwon commented on SPARK-36218:
--

Let me create a Pr for now as a temporary workaround ...

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-19 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383708#comment-17383708
 ] 

Hyukjin Kwon commented on SPARK-36218:
--

If we're not sure, It think we can land the same hacky fix for now ... 
[~maropu] do you have any idea about this?

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36218) Flaky Test: TPC-DS in PR builder

2021-07-19 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383707#comment-17383707
 ] 

Hyukjin Kwon commented on SPARK-36218:
--

cc [~maropu], [~cloud_fan], [~dongjoon] FYI.

Actually, I faced this issue in our internal repo a while ago, and just added a 
hacky fix by adding an explicit GC:

{code}
  if (tpcdsDataPath.nonEmpty) {
tpcdsQueries
  .filter(_ != "q95") // TODO(SC-75125)
  .filter(_ != "q75") // TODO(SC-75127)
  .filter(_ != "q64") // TODO(SC-75126)
  .foreach { name =>
  val queryString = resourceToString(s"tpcds/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(name) {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v1_4", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
tpcdsQueriesV2_7_0
  .filter(_ != "q95") // TODO(SC-75125)
  .filter(_ != "q75") // TODO(SC-75127)
  .filter(_ != "q64") // TODO(SC-75126)
  .foreach { name =>
  val queryString = resourceToString(s"tpcds-v2.7.0/$name.sql",
classLoader = Thread.currentThread().getContextClassLoader)
  test(s"$name-v2.7") {
+ // SPARK-36218: workaround to prevent unexpected failure related to 
resource usage.
+ System.gc()
val goldenFile = new File(s"$baseResourcePath/v2_7", s"$name.sql.out")
runQuery(queryString, goldenFile)
  }
}
  } else {
ignore("skipped because env `SPARK_TPCDS_DATA` is not set") {}
  }
}
{code}

> Flaky Test: TPC-DS in PR builder
> 
>
> Key: SPARK-36218
> URL: https://issues.apache.org/jira/browse/SPARK-36218
> Project: Spark
>  Issue Type: Test
>  Components: SQL, Tests
>Affects Versions: 3.0.3, 3.1.2, 3.2.0, 3.3.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> {code}
> [info] - q1 (9 seconds, 603 milliseconds)
> [info] - q2 (5 seconds, 860 milliseconds)
> [info] - q3 (1 second, 777 milliseconds)
> [info] - q4 (31 seconds, 951 milliseconds)
> [info] - q5 (4 seconds, 561 milliseconds)
> [info] - q7 (2 seconds, 471 milliseconds)
> [info] - q8 (2 seconds, 74 milliseconds)
> [info] - q9 (4 seconds, 402 milliseconds)
> [info] - q10 (4 seconds, 618 milliseconds)
> /home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 
> Killed  "$@"
> Error: Process completed with exit code 137.
> {code}
> It dies in the middle: https://github.com/apache/spark/runs/3109502701



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org