Re: ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception
Thanks.
This error I see is happening in Google Dataproc with Spark 3.1.1. It used
to be Spark 3.1.1-rc2 but now upgraded to 3.1.1. This is provided as a
service.
Unfortunately this error comes back on occasions when the job is running:
*** price for ticker MKS is 703.00 >= 676.96 SELL ***
21/03/18 22:14:13 ERROR org.apache.spark.scheduler.AsyncEventQueue:
Listener EventLoggingListener threw an exception
java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
at
scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:424)
at
scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:420)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.mutable.MapLike.toSeq(MapLike.scala:75)
at scala.collection.mutable.MapLike.toSeq$(MapLike.scala:72)
at scala.collection.mutable.AbstractMap.toSeq(Map.scala:84)
at
org.apache.spark.scheduler.EventLoggingListener.redactProperties(EventLoggingListener.scala:290)
at
org.apache.spark.scheduler.EventLoggingListener.onJobStart(EventLoggingListener.scala:162)
at
org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37)
at
org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at
org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at
org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at
org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
at
org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
at
org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
at
org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
at
scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.scheduler.AsyncEventQueue.org
$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
at
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
at
org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381)
at
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
*** price for ticker SAP is 156.83 >= 150.39 SELL ***
*** price for ticker TSCO is 182.30 <= 540.84 BUY ***
I assume there is no fix for this at the moment?
Thanks
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 18 Mar 2021 at 22:14, Jungtaek Lim
wrote:
> We've fixed the single case for "onJobStart", please check SPARK-34731
> [1]. The patch will be available in Spark 3.1.2 / 3.2.0, but if someone
> reports the same for lower version lines I think we could port back to
> lower version lines as well.
>
> 1. https://issues.apache.org/jira/browse/SPARK-34731
>
> On Fri, Mar 19, 2021 at 5:00 AM Mich Talebzadeh
> wrote:
>
>>
>> Recall this was the error
>>
>> 21/03/18 16:53:38 ERROR org.apache.spark.scheduler.AsyncEventQueue:
>> Listener EventLoggingListener threw an exception
>>
>> java.util.ConcurrentModificationException
>>
>> at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
>>
>> I resolved this error message by setting:
>>
>> spark.conf.set("spark.eventLog.enabled", "false")
>>
>> This happens when Spark writes to Google BigQuery table. The error is not
>> a show stopper
>>
>> There is also a recent reference to event logging listener exception in
>> this jira
>>
>> Getting event logging listener exception · Issue #439 · delta-io/delta
>> (github.com) <https://github.com/delta-io/delta/issues/439>
>>
>>
>> HTH
>>
>>
>>view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Discl
Re: ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception
We've fixed the single case for "onJobStart", please check SPARK-34731 [1].
The patch will be available in Spark 3.1.2 / 3.2.0, but if someone reports
the same for lower version lines I think we could port back to lower
version lines as well.
1. https://issues.apache.org/jira/browse/SPARK-34731
On Fri, Mar 19, 2021 at 5:00 AM Mich Talebzadeh
wrote:
>
> Recall this was the error
>
> 21/03/18 16:53:38 ERROR org.apache.spark.scheduler.AsyncEventQueue:
> Listener EventLoggingListener threw an exception
>
> java.util.ConcurrentModificationException
>
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
>
> I resolved this error message by setting:
>
> spark.conf.set("spark.eventLog.enabled", "false")
>
> This happens when Spark writes to Google BigQuery table. The error is not
> a show stopper
>
> There is also a recent reference to event logging listener exception in
> this jira
>
> Getting event logging listener exception · Issue #439 · delta-io/delta
> (github.com) <https://github.com/delta-io/delta/issues/439>
>
>
> HTH
>
>
>view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 18 Mar 2021 at 17:39, Mich Talebzadeh
> wrote:
>
>> This is an intermittent error. Full error is this
>>
>> 21/03/18 17:35:12 ERROR org.apache.spark.scheduler.AsyncEventQueue:
>> Listener EventLoggingListener threw an exception
>> java.util.ConcurrentModificationException
>> at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
>> at
>> scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:424)
>> at
>> scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:420)
>> at scala.collection.Iterator.foreach(Iterator.scala:943)
>> at scala.collection.Iterator.foreach$(Iterator.scala:943)
>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
>> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
>> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
>> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
>> at scala.collection.mutable.MapLike.toSeq(MapLike.scala:75)
>> at scala.collection.mutable.MapLike.toSeq$(MapLike.scala:72)
>> at scala.collection.mutable.AbstractMap.toSeq(Map.scala:84)
>> at
>> org.apache.spark.scheduler.EventLoggingListener.redactProperties(EventLoggingListener.scala:290)
>> at
>> org.apache.spark.scheduler.EventLoggingListener.onJobStart(EventLoggingListener.scala:162)
>> at
>> org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37)
>> at
>> org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
>> at
>> org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
>> at
>> org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
>> at
>> org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
>> at
>> org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
>> at
>> org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
>> at
>> org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
>> at
>> scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
>> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
>> at org.apache.spark.scheduler.AsyncEventQueue.org
>> $apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
>> at
>> org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
>> at
>> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381)
>> at
>> org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
>>
>> There is a reference to this error here.
>>
>>
>> [SPARK-32027] EventLoggingListener threw
>> java.util.ConcurrentModificationException - ASF JIRA (apache
Re: ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception
Recall this was the error
21/03/18 16:53:38 ERROR org.apache.spark.scheduler.AsyncEventQueue:
Listener EventLoggingListener threw an exception
java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
I resolved this error message by setting:
spark.conf.set("spark.eventLog.enabled", "false")
This happens when Spark writes to Google BigQuery table. The error is not a
show stopper
There is also a recent reference to event logging listener exception in
this jira
Getting event logging listener exception · Issue #439 · delta-io/delta
(github.com) <https://github.com/delta-io/delta/issues/439>
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 18 Mar 2021 at 17:39, Mich Talebzadeh
wrote:
> This is an intermittent error. Full error is this
>
> 21/03/18 17:35:12 ERROR org.apache.spark.scheduler.AsyncEventQueue:
> Listener EventLoggingListener threw an exception
> java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1387)
> at
> scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:424)
> at
> scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:420)
> at scala.collection.Iterator.foreach(Iterator.scala:943)
> at scala.collection.Iterator.foreach$(Iterator.scala:943)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
> at scala.collection.IterableLike.foreach(IterableLike.scala:74)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
> at scala.collection.mutable.MapLike.toSeq(MapLike.scala:75)
> at scala.collection.mutable.MapLike.toSeq$(MapLike.scala:72)
> at scala.collection.mutable.AbstractMap.toSeq(Map.scala:84)
> at
> org.apache.spark.scheduler.EventLoggingListener.redactProperties(EventLoggingListener.scala:290)
> at
> org.apache.spark.scheduler.EventLoggingListener.onJobStart(EventLoggingListener.scala:162)
> at
> org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37)
> at
> org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
> at
> org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
> at
> org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
> at
> org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
> at
> org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
> at
> org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
> at
> org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
> at
> scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
> at org.apache.spark.scheduler.AsyncEventQueue.org
> $apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
> at
> org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
> at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381)
> at
> org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
>
> There is a reference to this error here.
>
>
> [SPARK-32027] EventLoggingListener threw
> java.util.ConcurrentModificationException - ASF JIRA (apache.org)
> <https://issues.apache.org/jira/browse/SPARK-32027>
>
>
> HTH
>
>
>view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 18 Mar 2021 at 17:02, Mich Talebzadeh
> wrote:
>
>> Hi,
>>
>>
>> Does anyone know about the cause of this error in Spark structured
>> streaming?
>>
>
Re: ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception
This is an intermittent error. Full error is this 21/03/18 17:35:12 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception java.util.ConcurrentModificationException at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:424) at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$6.next(Wrappers.scala:420) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.mutable.MapLike.toSeq(MapLike.scala:75) at scala.collection.mutable.MapLike.toSeq$(MapLike.scala:72) at scala.collection.mutable.AbstractMap.toSeq(Map.scala:84) at org.apache.spark.scheduler.EventLoggingListener.redactProperties(EventLoggingListener.scala:290) at org.apache.spark.scheduler.EventLoggingListener.onJobStart(EventLoggingListener.scala:162) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37) at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at org.apache.spark.scheduler.AsyncEventQueue.org $apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381) at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) There is a reference to this error here. [SPARK-32027] EventLoggingListener threw java.util.ConcurrentModificationException - ASF JIRA (apache.org) <https://issues.apache.org/jira/browse/SPARK-32027> HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Thu, 18 Mar 2021 at 17:02, Mich Talebzadeh wrote: > Hi, > > > Does anyone know about the cause of this error in Spark structured > streaming? > > > Spark version 3.1.1 > > > > 21/03/18 16:53:38 ERROR org.apache.spark.scheduler.AsyncEventQueue: > Listener EventLoggingListener threw an exception > > java.util.ConcurrentModificationException > > at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) > > Thanks > > >view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > >
ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception
Hi, Does anyone know about the cause of this error in Spark structured streaming? Spark version 3.1.1 21/03/18 16:53:38 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exception java.util.ConcurrentModificationException at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) Thanks view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
