You may be compiling your app against 3.0.1 JARs but submitting to 3.1.1. You do not in general modify the Spark libs. You need to package libs like this with your app at the correct version.
On Tue, Apr 6, 2021 at 6:42 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks Gabor. > > All nodes are running Spark /spark-3.1.1-bin-hadoop3.2 > > So $SPARK_HOME/jars contains all the required jars on all nodes including > the jar file commons-pool2-2.9.0.jar as well. > > They are installed identically on all nodes. > > I have looked at the Spark environment for classpath. Still I don't see > the reason why Spark 3.1.1 fails with spark-sql-kafka-0-10_2.12-3.1.1.jar > but works ok with spark-sql-kafka-0-10_2.12-3.1.0.jar > > Anyway I unzipped the tarball for Spark-3.1.1 and there is > no spark-sql-kafka-0-10_2.12-3.0.1.jar even > > I had to add spark-sql-kafka-0-10_2.12-3.0.1.jar to make it work. Then I > enquired the availability of new version from Maven that pointed to > *spark-sql-kafka-0-10_2.12-3.1.1.jar* > > So to confirm Spark out of the tarball does not have any > > ltr spark-sql-kafka-* > ls: cannot access spark-sql-kafka-*: No such file or directory > > > For SSS, I had to add these > > add commons-pool2-2.9.0.jar. The one shipped is commons-pool-1.5.4.jar! > > add kafka-clients-2.7.0.jar Did not have any > > add spark-sql-kafka-0-10_2.12-3.0.1.jar Did not have any > > I gather from your second mail, there seems to be an issue with > spark-sql-kafka-0-10_2.12-3.*1*.1.jar ? > > HTH > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Tue, 6 Apr 2021 at 11:54, Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > >> Since you've not shared too much details I presume you've updated the >> spark-sql-kafka >> jar only. >> KafkaTokenUtil is in the token provider jar. >> >> As a general note if I'm right, please update Spark as a whole on all >> nodes and not just jars independently. >> >> BR, >> G >> >> >> On Tue, Apr 6, 2021 at 10:21 AM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> >>> Hi, >>> >>> >>> Any chance of someone testing the latest >>> spark-sql-kafka-0-10_2.12-3.1.1.jar >>> for Spark. It throws >>> >>> >>> java.lang.NoSuchMethodError: >>> org.apache.spark.kafka010.KafkaTokenUtil$.needTokenUpdate(Ljava/util/Map;Lscala/Option;)Z >>> >>> >>> However, the previous version spark-sql-kafka-0-10_2.12-3.0.1.jar works >>> fine >>> >>> >>> Thanks >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>
