[
https://issues.apache.org/jira/browse/SPARK-30599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020465#comment-17020465
]
Maxim Gekk commented on SPARK-30599:
------------------------------------
I see at least 2 solutions:
1. Increase the limit from 100 to something bigger number. For example, 1000
2. Pass to LogAppender's constructor a sequence of strings, and filter incoming
log events by the strings in the `append()` method.
[~cloud_fan] [~dongjoon] [~hyukjin.kwon] [~maropu] [~srowen] WDYT?
> SparkFunSuite.LogAppender throws java.lang.IllegalStateException
> ----------------------------------------------------------------
>
> Key: SPARK-30599
> URL: https://issues.apache.org/jira/browse/SPARK-30599
> Project: Spark
> Issue Type: Test
> Components: Spark Core, SQL, Tests
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Minor
> Attachments: fail_trace.txt, success_trace.txt
>
>
> The LogAppender has a limit for the number of logged event. By default, it is
> 100. All tests that use LogAppender finish successfully with this limit but
> sometime the test fail with the exception like:
> {code:java}
> java.lang.IllegalStateException: Number of events reached the limit of 100
> while logging CSV header matches to schema w/ enforceSchema.
> at
> org.apache.spark.SparkFunSuite$LogAppender.append(SparkFunSuite.scala:200)
> at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> at org.apache.log4j.Category.callAppenders(Category.java:206)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> {code}
> For example, the CSV test "SPARK-23786: warning should be printed if CSV
> header doesn't conform to schema" uses 2 log appenders. The
> `success_trace.txt` contains stored log event in normal runs, another one
> `fail_trace.txt` I got while re-running the test in a loop.
> The difference is in:
> {code}
> [35] Block broadcast_222 stored as values in memory (estimated size 286.9
> KiB, free 1998.5 MiB)
> [36] Removed broadcast_214_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2003.8 MiB)
> [37] Removed broadcast_204_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2003.9 MiB)
> [38] Removed broadcast_200_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2003.9 MiB)
> [39] Removed broadcast_207_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2003.9 MiB)
> [40] Removed broadcast_208_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2003.9 MiB)
> [41] Removed broadcast_194_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2003.9 MiB)
> [42] Removed broadcast_216_piece0 on 192.168.1.66:52354 in memory (size: 53.3
> KiB, free: 2004.0 MiB)
> [43] Removed broadcast_195_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2004.0 MiB)
> [44] Removed broadcast_219_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2004.0 MiB)
> [45] Removed broadcast_198_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.0 MiB)
> [46] Removed broadcast_201_piece0 on 192.168.1.66:52354 in memory (size: 53.4
> KiB, free: 2004.0 MiB)
> [47] Removed broadcast_205_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2004.0 MiB)
> [48] Removed broadcast_197_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.1 MiB)
> [49] Removed broadcast_189_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2004.1 MiB)
> [50] Removed broadcast_193_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.1 MiB)
> [51] Removed broadcast_196_piece0 on 192.168.1.66:52354 in memory (size: 53.3
> KiB, free: 2004.2 MiB)
> [52] Removed broadcast_206_piece0 on 192.168.1.66:52354 in memory (size: 53.4
> KiB, free: 2004.2 MiB)
> [53] Removed broadcast_209_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2004.2 MiB)
> [54] Removed broadcast_192_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.2 MiB)
> [55] Removed broadcast_215_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2004.2 MiB)
> [56] Removed broadcast_210_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2004.2 MiB)
> [57] Removed broadcast_199_piece0 on 192.168.1.66:52354 in memory (size: 5.7
> KiB, free: 2004.2 MiB)
> [58] Removed broadcast_221_piece0 on 192.168.1.66:52354 in memory (size: 53.4
> KiB, free: 2004.3 MiB)
> [59] Removed broadcast_213_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.3 MiB)
> [60] Removed broadcast_191_piece0 on 192.168.1.66:52354 in memory (size: 53.4
> KiB, free: 2004.4 MiB)
> [61] Removed broadcast_220_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2004.4 MiB)
> [62] Block broadcast_222_piece0 stored as bytes in memory (estimated size
> 24.2 KiB, free 2002.0 MiB)
> [63] Added broadcast_222_piece0 in memory on 192.168.1.66:52354 (size: 24.2
> KiB, free: 2004.4 MiB)
> [64] Removed broadcast_203_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.4 MiB)
> [65] Created broadcast 222 from apply at OutcomeOf.scala:85
> [66] Removed broadcast_218_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.4 MiB)
> [67] Removed broadcast_211_piece0 on 192.168.1.66:52354 in memory (size: 53.4
> KiB, free: 2004.5 MiB)
> [68] Removed broadcast_190_piece0 on 192.168.1.66:52354 in memory (size: 3.7
> KiB, free: 2004.5 MiB)
> [69] Block broadcast_223 stored as values in memory (estimated size 286.9
> KiB, free 2002.5 MiB)
> [70] Removed broadcast_212_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.5 MiB)
> [71] Removed broadcast_188_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.5 MiB)
> [72] Removed broadcast_187_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.5 MiB)
> [73] Removed broadcast_202_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.6 MiB)
> [74] Removed broadcast_217_piece0 on 192.168.1.66:52354 in memory (size: 24.2
> KiB, free: 2004.6 MiB)
> [75] Block broadcast_223_piece0 stored as bytes in memory (estimated size
> 24.2 KiB, free 2004.0 MiB)
> [76] Added broadcast_223_piece0 in memory on 192.168.1.66:52354 (size: 24.2
> KiB, free: 2004.6 MiB)
> [77] Created broadcast 223 from apply at OutcomeOf.scala:85
> [78] Starting job: apply at OutcomeOf.scala:85
> [79] Got job 134 (apply at OutcomeOf.scala:85) with 1 output partitions
> [80] Final stage: ResultStage 134 (apply at OutcomeOf.scala:85)
> [81] Parents of final stage: List()
> [82] Missing parents: List()
> [83] Submitting ResultStage 134 (MapPartitionsRDD[809] at apply at
> OutcomeOf.scala:85), which has no missing parents
> [84] Block broadcast_224 stored as values in memory (estimated size 11.0 KiB,
> free 2004.0 MiB)
> [85] Block broadcast_224_piece0 stored as bytes in memory (estimated size 5.7
> KiB, free 2004.0 MiB)
> [86] Added broadcast_224_piece0 in memory on 192.168.1.66:52354 (size: 5.7
> KiB, free: 2004.5 MiB)
> [87] Created broadcast 224 from broadcast at DAGScheduler.scala:1200
> [88] Submitting 1 missing tasks from ResultStage 134 (MapPartitionsRDD[809]
> at apply at OutcomeOf.scala:85) (first 15 tasks are for partitions Vector(0))
> [89] Adding task set 134.0 with 1 tasks
> [90] Starting task 0.0 in stage 134.0 (TID 134, 192.168.1.66, executor
> driver, partition 0, PROCESS_LOCAL, 8029 bytes)
> [91] Running task 0.0 in stage 134.0 (TID 134)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
