Maciej Bryński created SPARK-21507: --------------------------------------
Summary: Exception when using spark.jars.packages Key: SPARK-21507 URL: https://issues.apache.org/jira/browse/SPARK-21507 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.2.0 Reporter: Maciej Bryński Priority: Minor When more than one process is using packages option it's possible to create exception {code} [2017-07-21 18:14:18,356] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:18,356] {bash_operator.py:94} INFO - Ivy Default Cache set to: /home/bi/.ivy2/cache [2017-07-21 18:14:18,357] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:18,356] {bash_operator.py:94} INFO - The jars for the packages stored in: /home/bi/.ivy2/jars [2017-07-21 18:14:18,406] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:18,405] {bash_operator.py:94} INFO - :: loading settings :: url = jar:file:/opt/spark-2.2.0/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml [2017-07-21 18:14:18,735] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:18,735] {bash_operator.py:94} INFO - datastax#spark-cassandra-connector added as a dependency [2017-07-21 18:14:18,737] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:18,737] {bash_operator.py:94} INFO - :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 [2017-07-21 18:14:18,738] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:18,737] {bash_operator.py:94} INFO - confs: [default] [2017-07-21 18:14:19,284] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,283] {bash_operator.py:94} INFO - found datastax#spark-cassandra-connector;2.0.3-s_2.11 in spark-packages [2017-07-21 18:14:19,379] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,379] {bash_operator.py:94} INFO - found commons-beanutils#commons-beanutils;1.9.3 in central [2017-07-21 18:14:19,415] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,415] {bash_operator.py:94} INFO - found commons-collections#commons-collections;3.2.2 in local-m2-cache [2017-07-21 18:14:19,469] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,469] {bash_operator.py:94} INFO - found joda-time#joda-time;2.3 in central [2017-07-21 18:14:19,518] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,517] {bash_operator.py:94} INFO - found org.joda#joda-convert;1.2 in central [2017-07-21 18:14:19,583] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,582] {bash_operator.py:94} INFO - found com.twitter#jsr166e;1.1.0 in central [2017-07-21 18:14:19,666] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,666] {bash_operator.py:94} INFO - found io.netty#netty-all;4.0.33.Final in central [2017-07-21 18:14:19,732] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,732] {bash_operator.py:94} INFO - found org.scala-lang#scala-reflect;2.11.8 in local-m2-cache [2017-07-21 18:14:19,831] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,830] {bash_operator.py:94} INFO - :: resolution report :: resolve 1042ms :: artifacts dl 51ms [2017-07-21 18:14:19,831] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,831] {bash_operator.py:94} INFO - :: modules in use: [2017-07-21 18:14:19,834] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,832] {bash_operator.py:94} INFO - com.twitter#jsr166e;1.1.0 from central in [default] [2017-07-21 18:14:19,834] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,832] {bash_operator.py:94} INFO - commons-beanutils#commons-beanutils;1.9.3 from central in [default] [2017-07-21 18:14:19,834] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,832] {bash_operator.py:94} INFO - commons-collections#commons-collections;3.2.2 from local-m2-cache in [default] [2017-07-21 18:14:19,834] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,832] {bash_operator.py:94} INFO - datastax#spark-cassandra-connector;2.0.3-s_2.11 from spark-packages in [default] [2017-07-21 18:14:19,834] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,832] {bash_operator.py:94} INFO - io.netty#netty-all;4.0.33.Final from central in [default] [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - joda-time#joda-time;2.3 from central in [default] [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - org.joda#joda-convert;1.2 from central in [default] [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - org.scala-lang#scala-reflect;2.11.8 from local-m2-cache in [default] [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - --------------------------------------------------------------------- [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - | | modules || artifacts | [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - | conf | number| search|dwnlded|evicted|| number|dwnlded| [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - --------------------------------------------------------------------- [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,833] {bash_operator.py:94} INFO - | default | 8 | 0 | 0 | 0 || 8 | 0 | [2017-07-21 18:14:19,835] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,834] {bash_operator.py:94} INFO - --------------------------------------------------------------------- [2017-07-21 18:14:19,852] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,851] {bash_operator.py:94} INFO - :: retrieving :: org.apache.spark#spark-submit-parent [2017-07-21 18:14:19,852] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,852] {bash_operator.py:94} INFO - confs: [default] [2017-07-21 18:14:19,891] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,891] {bash_operator.py:94} INFO - Exception in thread "main" java.lang.RuntimeException: problem during retrieve of org.apache.spark#spark-submit-parent: java.text.ParseException: failed to parse report: /home/bi/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml: Premature end of file. [2017-07-21 18:14:19,892] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,892] {bash_operator.py:94} INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:249) [2017-07-21 18:14:19,892] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,892] {bash_operator.py:94} INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:83) [2017-07-21 18:14:19,892] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,892] {bash_operator.py:94} INFO - at org.apache.ivy.Ivy.retrieve(Ivy.java:551) [2017-07-21 18:14:19,893] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,893] {bash_operator.py:94} INFO - at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1180) [2017-07-21 18:14:19,893] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,893] {bash_operator.py:94} INFO - at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:298) [2017-07-21 18:14:19,893] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,893] {bash_operator.py:94} INFO - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153) [2017-07-21 18:14:19,893] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,893] {bash_operator.py:94} INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) [2017-07-21 18:14:19,894] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,894] {bash_operator.py:94} INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [2017-07-21 18:14:19,894] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,894] {bash_operator.py:94} INFO - Caused by: java.text.ParseException: failed to parse report: /home/bi/.ivy2/cache/org.apache.spark-spark-submit-parent-default.xml: Premature end of file. [2017-07-21 18:14:19,894] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,894] {bash_operator.py:94} INFO - at org.apache.ivy.plugins.report.XmlReportParser.parse(XmlReportParser.java:293) [2017-07-21 18:14:19,894] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,894] {bash_operator.py:94} INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.determineArtifactsToCopy(RetrieveEngine.java:329) [2017-07-21 18:14:19,895] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,894] {bash_operator.py:94} INFO - at org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:118) [2017-07-21 18:14:19,895] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,895] {bash_operator.py:94} INFO - ... 7 more [2017-07-21 18:14:19,895] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,895] {bash_operator.py:94} INFO - Caused by: org.xml.sax.SAXParseException; Premature end of file. [2017-07-21 18:14:19,895] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,895] {bash_operator.py:94} INFO - at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) [2017-07-21 18:14:19,895] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,895] {bash_operator.py:94} INFO - at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source) [2017-07-21 18:14:19,901] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,901] {bash_operator.py:94} INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) [2017-07-21 18:14:19,902] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,901] {bash_operator.py:94} INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) [2017-07-21 18:14:19,902] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,902] {bash_operator.py:94} INFO - at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) [2017-07-21 18:14:19,902] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,902] {bash_operator.py:94} INFO - at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source) [2017-07-21 18:14:19,902] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,902] {bash_operator.py:94} INFO - at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [2017-07-21 18:14:19,903] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,903] {bash_operator.py:94} INFO - at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [2017-07-21 18:14:19,903] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,903] {bash_operator.py:94} INFO - at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) [2017-07-21 18:14:19,904] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,903] {bash_operator.py:94} INFO - at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) [2017-07-21 18:14:19,904] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,904] {bash_operator.py:94} INFO - at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) [2017-07-21 18:14:19,904] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,904] {bash_operator.py:94} INFO - at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) [2017-07-21 18:14:19,905] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,905] {bash_operator.py:94} INFO - at javax.xml.parsers.SAXParser.parse(SAXParser.java:328) [2017-07-21 18:14:19,905] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,905] {bash_operator.py:94} INFO - at org.apache.ivy.plugins.report.XmlReportParser$SaxXmlReportParser.parse(XmlReportParser.java:249) [2017-07-21 18:14:19,905] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,905] {bash_operator.py:94} INFO - at org.apache.ivy.plugins.report.XmlReportParser.parse(XmlReportParser.java:291) [2017-07-21 18:14:19,906] {base_task_runner.py:95} INFO - Subtask: [2017-07-21 18:14:19,906] {bash_operator.py:94} INFO - ... 9 more {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org