[jira] [Commented] (SPARK-36548) Throwing NoClassDefFoundError for Logging$class

2021-08-31 Thread Hyukjin Kwon (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407551#comment-17407551
 ] 

Hyukjin Kwon commented on SPARK-36548:
--

This isn't an API by design. Won't be fixed.

> Throwing NoClassDefFoundError for Logging$class
> ---
>
> Key: SPARK-36548
> URL: https://issues.apache.org/jira/browse/SPARK-36548
> Project: Spark
>  Issue Type: Bug
>  Components: Kubernetes, Spark Core
>Affects Versions: 3.1.1
>Reporter: sadagopan kalyanaraman
>Priority: Major
>
> I'm getting NoClassDefFoundError. is it removed in 3.1.1?
>  
> https://spark.apache.org/docs/3.1.1/api/java/org/apache/spark/internal/Logging.html
>  
> *https://spark.apache.org/docs/2.4.7/api/java/org/apache/spark/internal/Logging.html*
>  
> Exception in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiatedException in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiated at java.base/java.util.ServiceLoader.fail(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.get(Unknown Source) at 
> java.base/java.util.ServiceLoader$3.next(Unknown Source) at 
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44) at 
> scala.collection.Iterator.foreach(Iterator.scala:941) at 
> scala.collection.Iterator.foreach$(Iterator.scala:941) at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at 
> scala.collection.IterableLike.foreach(IterableLike.scala:74) at 
> scala.collection.IterableLike.foreach$(IterableLike.scala:73) at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:56) at 
> scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255) at 
> scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249) at 
> scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108) at 
> scala.collection.TraversableLike.filter(TraversableLike.scala:347) at 
> scala.collection.TraversableLike.filter$(TraversableLike.scala:347) at 
> scala.collection.AbstractTraversable.filter(Traversable.scala:108) at 
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:659)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.loadInternal(DataStreamReader.scala:209)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:195)
>  at 
> com.xyz.xyz.xyz.AbstractCdcMessageToSqJob.runJob(AbstractCdcMessageToSqJob.java:94)
>  at 
> com.xyz.xyz.xyz.CdcMessageToSqlBillerJob.main(CdcMessageToSqlBillerJob.java:30)
>  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method) at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
> at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
>  at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at 
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at 
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) 
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at 
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: 
> java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala:34)
>  at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:43)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:50)
>  at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method) at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>  Source) at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>  Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown 
> Source) ... 33 moreCaused by: java.lang.ClassNotFoundException: 
> org.apache.spark.internal.Logging$class at 
>

[jira] [Commented] (SPARK-36548) Throwing NoClassDefFoundError for Logging$class

2021-08-31 Thread sadagopan kalyanaraman (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407481#comment-17407481
 ] 

sadagopan kalyanaraman commented on SPARK-36548:


[~gurwls223], why was this moved to resolved? was it fixed 

> Throwing NoClassDefFoundError for Logging$class
> ---
>
> Key: SPARK-36548
> URL: https://issues.apache.org/jira/browse/SPARK-36548
> Project: Spark
>  Issue Type: Bug
>  Components: Kubernetes, Spark Core
>Affects Versions: 3.1.1
>Reporter: sadagopan kalyanaraman
>Priority: Major
>
> I'm getting NoClassDefFoundError. is it removed in 3.1.1?
>  
> https://spark.apache.org/docs/3.1.1/api/java/org/apache/spark/internal/Logging.html
>  
> *https://spark.apache.org/docs/2.4.7/api/java/org/apache/spark/internal/Logging.html*
>  
> Exception in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiatedException in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiated at java.base/java.util.ServiceLoader.fail(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.get(Unknown Source) at 
> java.base/java.util.ServiceLoader$3.next(Unknown Source) at 
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44) at 
> scala.collection.Iterator.foreach(Iterator.scala:941) at 
> scala.collection.Iterator.foreach$(Iterator.scala:941) at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at 
> scala.collection.IterableLike.foreach(IterableLike.scala:74) at 
> scala.collection.IterableLike.foreach$(IterableLike.scala:73) at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:56) at 
> scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255) at 
> scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249) at 
> scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108) at 
> scala.collection.TraversableLike.filter(TraversableLike.scala:347) at 
> scala.collection.TraversableLike.filter$(TraversableLike.scala:347) at 
> scala.collection.AbstractTraversable.filter(Traversable.scala:108) at 
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:659)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.loadInternal(DataStreamReader.scala:209)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:195)
>  at 
> com.xyz.xyz.xyz.AbstractCdcMessageToSqJob.runJob(AbstractCdcMessageToSqJob.java:94)
>  at 
> com.xyz.xyz.xyz.CdcMessageToSqlBillerJob.main(CdcMessageToSqlBillerJob.java:30)
>  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method) at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
> at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
>  at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at 
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at 
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) 
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at 
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: 
> java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala:34)
>  at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:43)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:50)
>  at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method) at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>  Source) at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>  Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown 
> Source) ... 33 moreCaused by: java.lang.ClassNotFoundException: 
> org.apache.spark.internal.Logging$class at 
>

[jira] [Commented] (SPARK-36548) Throwing NoClassDefFoundError for Logging$class

2021-08-20 Thread sadagopan kalyanaraman (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402394#comment-17402394
 ] 

sadagopan kalyanaraman commented on SPARK-36548:


[~sarutak]  i agree on the doc but the no class def error?

> Throwing NoClassDefFoundError for Logging$class
> ---
>
> Key: SPARK-36548
> URL: https://issues.apache.org/jira/browse/SPARK-36548
> Project: Spark
>  Issue Type: Bug
>  Components: Kubernetes, Spark Core
>Affects Versions: 3.1.1
>Reporter: sadagopan kalyanaraman
>Priority: Major
>
> I'm getting NoClassDefFoundError. is it removed in 3.1.1?
>  
> https://spark.apache.org/docs/3.1.1/api/java/org/apache/spark/internal/Logging.html
>  
> *https://spark.apache.org/docs/2.4.7/api/java/org/apache/spark/internal/Logging.html*
>  
> Exception in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiatedException in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiated at java.base/java.util.ServiceLoader.fail(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.get(Unknown Source) at 
> java.base/java.util.ServiceLoader$3.next(Unknown Source) at 
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44) at 
> scala.collection.Iterator.foreach(Iterator.scala:941) at 
> scala.collection.Iterator.foreach$(Iterator.scala:941) at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at 
> scala.collection.IterableLike.foreach(IterableLike.scala:74) at 
> scala.collection.IterableLike.foreach$(IterableLike.scala:73) at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:56) at 
> scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255) at 
> scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249) at 
> scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108) at 
> scala.collection.TraversableLike.filter(TraversableLike.scala:347) at 
> scala.collection.TraversableLike.filter$(TraversableLike.scala:347) at 
> scala.collection.AbstractTraversable.filter(Traversable.scala:108) at 
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:659)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.loadInternal(DataStreamReader.scala:209)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:195)
>  at 
> com.xyz.xyz.xyz.AbstractCdcMessageToSqJob.runJob(AbstractCdcMessageToSqJob.java:94)
>  at 
> com.xyz.xyz.xyz.CdcMessageToSqlBillerJob.main(CdcMessageToSqlBillerJob.java:30)
>  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method) at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
> at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
>  at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at 
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at 
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) 
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at 
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: 
> java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala:34)
>  at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:43)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:50)
>  at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method) at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>  Source) at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>  Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown 
> Source) ... 33 moreCaused by: java.lang.ClassNotFoundException: 
> org.apache.spark.internal.Logging$class at 
>

[jira] [Commented] (SPARK-36548) Throwing NoClassDefFoundError for Logging$class

2021-08-20 Thread Kousuke Saruta (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402364#comment-17402364
 ] 

Kousuke Saruta commented on SPARK-36548:


[~sadagopan]
It was wrongly exposed before. Logging is for internal use.
As of 3.0.0, internally used classes are not in the doc.
See SPARK-30779.

> Throwing NoClassDefFoundError for Logging$class
> ---
>
> Key: SPARK-36548
> URL: https://issues.apache.org/jira/browse/SPARK-36548
> Project: Spark
>  Issue Type: Bug
>  Components: Kubernetes, Spark Core
>Affects Versions: 3.1.1
>Reporter: sadagopan kalyanaraman
>Priority: Major
>
> I'm getting NoClassDefFoundError. is it removed in 3.1.1?
>  
> https://spark.apache.org/docs/3.1.1/api/java/org/apache/spark/internal/Logging.html
>  
> *https://spark.apache.org/docs/2.4.7/api/java/org/apache/spark/internal/Logging.html*
>  
> Exception in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiatedException in thread "main" java.util.ServiceConfigurationError: 
> org.apache.spark.sql.sources.DataSourceRegister: Provider 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be 
> instantiated at java.base/java.util.ServiceLoader.fail(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) at 
> java.base/java.util.ServiceLoader$ProviderImpl.get(Unknown Source) at 
> java.base/java.util.ServiceLoader$3.next(Unknown Source) at 
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44) at 
> scala.collection.Iterator.foreach(Iterator.scala:941) at 
> scala.collection.Iterator.foreach$(Iterator.scala:941) at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at 
> scala.collection.IterableLike.foreach(IterableLike.scala:74) at 
> scala.collection.IterableLike.foreach$(IterableLike.scala:73) at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:56) at 
> scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255) at 
> scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249) at 
> scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108) at 
> scala.collection.TraversableLike.filter(TraversableLike.scala:347) at 
> scala.collection.TraversableLike.filter$(TraversableLike.scala:347) at 
> scala.collection.AbstractTraversable.filter(Traversable.scala:108) at 
> org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:659)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.loadInternal(DataStreamReader.scala:209)
>  at 
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:195)
>  at 
> com.xyz.xyz.xyz.AbstractCdcMessageToSqJob.runJob(AbstractCdcMessageToSqJob.java:94)
>  at 
> com.xyz.xyz.xyz.CdcMessageToSqlBillerJob.main(CdcMessageToSqlBillerJob.java:30)
>  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method) at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source) at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 
> at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
>  at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at 
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at 
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030) 
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039) at 
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: 
> java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala:34)
>  at 
> com.google.cloud.spark.bigquery.BigQueryUtilScala$.(BigQueryUtil.scala)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:43)
>  at 
> com.google.cloud.spark.bigquery.BigQueryRelationProvider.(BigQueryRelationProvider.scala:50)
>  at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>  Method) at 
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>  Source) at 
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>  Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown 
> Source) ... 33 moreCaused by: