[jira] [Commented] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.
[
https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504246#comment-16504246
]
Prashant Sharma commented on SPARK-24279:
-
Thanks a lot, that was the mistake.
> Incompatible byte code errors, when using test-jar of spark sql.
>
>
> Key: SPARK-24279
> URL: https://issues.apache.org/jira/browse/SPARK-24279
> Project: Spark
> Issue Type: Bug
> Components: SQL, Structured Streaming, Tests
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Using test libraries available with spark sql streaming, produces weird
> incompatible byte code errors. It is already tested on a different virtual
> box instance to make sure, that it is not related to my system environment. A
> reproducer is uploaded to github.
> [https://github.com/ScrapCodes/spark-bug-reproducer]
>
> Doing a clean build reproduces the error.
>
> Verbatim paste of the error.
> {code:java}
> [INFO] Compiling 1 source files to
> /home/prashant/work/test/target/test-classes at 1526380360990
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'QueryTest.class'.
> [INFO] Could not access type PlanTest in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'QueryTest.class' was compiled against an
> incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'SQLTestUtilsBase.class'.
> [INFO] Could not access type PlanTestBase in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled
> against an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'SQLTestUtils.class'.
> [INFO] Could not access type PlanTest in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against
> an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25:
> error: Unable to find encoder for type stored in a Dataset. Primitive types
> (Int, String, etc) and Product types (case classes) are supported by
> importing spark.implicits._ Support for serializing other types will be added
> in future releases.
> [ERROR] val inputData = MemoryStream[Int]
> [ERROR] ^
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30:
> error: Unable to find encoder for type stored in a Dataset. Primitive types
> (Int, String, etc) and Product types (case classes) are supported by
> importing spark.implicits._ Support for serializing other types will be added
> in future releases.
> [ERROR] CheckAnswer(2, 3, 4))
> [ERROR] ^
> [ERROR] 5 errors found
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.
[
https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503630#comment-16503630
]
Shixiong Zhu commented on SPARK-24279:
--
Just did a quick look at your pom.xml. I think it's missing the following
catalyst test jar.
{code}
org.apache.spark
spark-catalyst_${scala.binary.version}
${project.version}
test-jar
test
{code}
> Incompatible byte code errors, when using test-jar of spark sql.
>
>
> Key: SPARK-24279
> URL: https://issues.apache.org/jira/browse/SPARK-24279
> Project: Spark
> Issue Type: Bug
> Components: SQL, Structured Streaming, Tests
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Using test libraries available with spark sql streaming, produces weird
> incompatible byte code errors. It is already tested on a different virtual
> box instance to make sure, that it is not related to my system environment. A
> reproducer is uploaded to github.
> [https://github.com/ScrapCodes/spark-bug-reproducer]
>
> Doing a clean build reproduces the error.
>
> Verbatim paste of the error.
> {code:java}
> [INFO] Compiling 1 source files to
> /home/prashant/work/test/target/test-classes at 1526380360990
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'QueryTest.class'.
> [INFO] Could not access type PlanTest in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'QueryTest.class' was compiled against an
> incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'SQLTestUtilsBase.class'.
> [INFO] Could not access type PlanTestBase in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtilsBase.class' was compiled
> against an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] error: missing or invalid dependency detected while loading class
> file 'SQLTestUtils.class'.
> [INFO] Could not access type PlanTest in package
> org.apache.spark.sql.catalyst.plans,
> [INFO] because it (or its dependencies) are missing. Check your build
> definition for
> [INFO] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to
> see the problematic classpath.)
> [INFO] A full rebuild may help if 'SQLTestUtils.class' was compiled against
> an incompatible version of org.apache.spark.sql.catalyst.plans.
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:25:
> error: Unable to find encoder for type stored in a Dataset. Primitive types
> (Int, String, etc) and Product types (case classes) are supported by
> importing spark.implicits._ Support for serializing other types will be added
> in future releases.
> [ERROR] val inputData = MemoryStream[Int]
> [ERROR] ^
> [ERROR] /home/prashant/work/test/src/test/scala/SparkStreamingTests.scala:30:
> error: Unable to find encoder for type stored in a Dataset. Primitive types
> (Int, String, etc) and Product types (case classes) are supported by
> importing spark.implicits._ Support for serializing other types will be added
> in future releases.
> [ERROR] CheckAnswer(2, 3, 4))
> [ERROR] ^
> [ERROR] 5 errors found
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
