[
https://issues.apache.org/jira/browse/SPARK-30599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk updated SPARK-30599:
-------------------------------
Description:
The LogAppender has a limit for the number of logged event. By default, it is
100. All tests that use LogAppender finish successfully with this limit but
sometime the test fail with the exception like:
{code:java}
java.lang.IllegalStateException: Number of events reached the limit of 100
while logging CSV header matches to schema w/ enforceSchema.
at
org.apache.spark.SparkFunSuite$LogAppender.append(SparkFunSuite.scala:200)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
{code}
For example, the CSV test "SPARK-23786: warning should be printed if CSV header
doesn't conform to schema" uses 2 log appenders. The `success_trace.txt`
contains stored log event in normal runs, another one `fail_trace.txt` I got
while re-running the test in a loop.
was:
The LogAppender has a limit for the number of logged event. By default, it is
100. All tests that use LogAppender finish successfully with this limit but
sometime the test fail with the exception like:
{code:java}
java.lang.IllegalStateException: Number of events reached the limit of 100
while logging CSV header matches to schema w/ enforceSchema.
at
org.apache.spark.SparkFunSuite$LogAppender.append(SparkFunSuite.scala:200)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
{code}
For example, the CSV test "SPARK-23786: warning should be printed if CSV header
doesn't conform to schema" uses 2 log appenders.
> SparkFunSuite.LogAppender throws java.lang.IllegalStateException
> ----------------------------------------------------------------
>
> Key: SPARK-30599
> URL: https://issues.apache.org/jira/browse/SPARK-30599
> Project: Spark
> Issue Type: Test
> Components: Spark Core, SQL, Tests
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Minor
> Attachments: fail_trace.txt, success_trace.txt
>
>
> The LogAppender has a limit for the number of logged event. By default, it is
> 100. All tests that use LogAppender finish successfully with this limit but
> sometime the test fail with the exception like:
> {code:java}
> java.lang.IllegalStateException: Number of events reached the limit of 100
> while logging CSV header matches to schema w/ enforceSchema.
> at
> org.apache.spark.SparkFunSuite$LogAppender.append(SparkFunSuite.scala:200)
> at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> at org.apache.log4j.Category.callAppenders(Category.java:206)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> {code}
> For example, the CSV test "SPARK-23786: warning should be printed if CSV
> header doesn't conform to schema" uses 2 log appenders. The
> `success_trace.txt` contains stored log event in normal runs, another one
> `fail_trace.txt` I got while re-running the test in a loop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
