[
https://issues.apache.org/jira/browse/SIS-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Desruisseaux closed SIS-491.
-----------------------------------
> sis-embedded-data database fails to start on Apache Spark
> ---------------------------------------------------------
>
> Key: SIS-491
> URL: https://issues.apache.org/jira/browse/SIS-491
> Project: Spatial Information Systems
> Issue Type: Bug
> Components: Metadata
> Affects Versions: 0.8, 1.0
> Environment: Spark version 2.4.4
> Hadoop version 2.7.6
> Scala version 2.11.12
> Java version 1.8.0_242
> OS: Linux <computer> 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC
> 2020 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Derrick Oswald
> Assignee: Martin Desruisseaux
> Priority: Minor
> Fix For: 1.1
>
>
> Operation _CRS_.forCode fails when used within the Apache Spark environment.
> Steps to reproduce (see full log below):
> 1) Start spark-shell with the --packages argument specifying the
> sis-embedded-data artifact
> 2) Execute CRS.forCode ("EPSG:21781") {color:#808080}// Swiss coordinate
> system [https://epsg.io/21781]{color}
> {color:#172b4d}Result:{color}
> {color:#172b4d} NoSuchAuthorityCodeException (with a prior warning
> message: Failed to start database
> 'classpath:SIS_DATA/Databases/SpatialMetadata'){color}
> {color:#172b4d}Expected result:{color}
> {color:#172b4d} Returns a valid CoordinateReferenceSystem for
> "EPSG:21781".{color}
> {color:#172b4d}Workaround:{color}
> {color:#172b4d} Use _CRS.fromWKT_ instead.{color}
> {color:#172b4d}Note:{color}
> {{{color:#172b4d}It works fine in a stand-alone Scala program with
> build.sbt:{color}}}
> {{{color:#660e7a}name {color}:= {color:#008000}"ApacheSISTest"{color}}}
> {{{color:#660e7a}version {color}:=
> {color:#008000}"0.1"{color}{color:#660e7a}scalaVersion {color}:=
> {color:#008000}"2.11.12"{color}}}
> {{{color:#660e7a}libraryDependencies {color}+=
> {color:#008000}"org.apache.sis.core" {color}%
> {color:#008000}"sis-referencing" {color}% {color:#008000}"1.0"{color}+}}
> {{+{color:#660e7a}libraryDependencies{color}+ =
> {color:#008000}"org.apache.sis.non-free" {color}%
> {color:#008000}"sis-embedded-data" {color}% {color:#008000}"1.0"{color}}}
> and code
> {color:#000080}import {color}org.apache.sis.geometry.DirectPosition2D
> {color:#000080}import {color}org.apache.sis.referencing.CRS
> {color:#000080}object {color}Test {color:#000080}extends {color}App {
> {color:#000080}val {color}{color:#660e7a}position {color}=
> {color:#000080}new {color}DirectPosition2D
> ({color:#0000ff}678544.6095273666{color},
> {color:#0000ff}248462.881516265{color})
> println ({color:#008000}"Source: " {color}+
> {color:#660e7a}position{color})
> {color:#000080}val {color}{color:#660e7a}sourceCRS {color}= CRS.forCode
> ({color:#008000}"EPSG:21781"{color}) {color:#808080}// Swiss coordinate
> system [https://epsg.io/21781]{color}
> {color:#000080}val {color}{color:#660e7a}targetCRS {color}= CRS.forCode
> ({color:#008000}"EPSG:4326"{color}) {color:#808080}// WGS 84
> [https://epsg.io/4326]{color}
> {color:#000080}val {color}{color:#660e7a}operation {color}=
> CRS.findOperation ({color:#660e7a}sourceCRS{color},
> {color:#660e7a}targetCRS{color}, {color:#000080}null{color})
> {color:#000080}val {color}{color:#660e7a}transform {color}=
> {color:#660e7a}operation{color}.getMathTransform
> {color:#000080}val {color}{color:#660e7a}latlong {color}=
> {color:#660e7a}transform{color}.transform ({color:#660e7a}position{color},
> {color:#000080}null{color}) {color:#808080}// °N, °E{color}
> println ({color:#008000}"Target: " {color}+ {color:#660e7a}latlong{color})
> }
>
> {color:#172b4d}Full Spark log:{color}
> $ *spark-shell --master local[*] --verbose --packages
> org.apache.sis.core:sis-referencing:1.0,org.apache.sis.non-free:sis-embedded-data:1.0*
> Using properties file: null
> 20/03/08 15:41:42 WARN Utils: Your hostname, <computer> resolves to a
> loopback address: 127.0.1.1; using 192.168.10.221 instead (on interface
> wlp4s0)
> 20/03/08 15:41:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
> Parsed arguments:
> master local[*]
> deployMode null
> executorMemory null
> executorCores null
> totalExecutorCores null
> propertiesFile null
> driverMemory null
> driverCores null
> driverExtraClassPath null
> driverExtraLibraryPath null
> driverExtraJavaOptions null
> supervise false
> queue null
> numExecutors null
> files null
> pyFiles null
> archives null
> mainClass org.apache.spark.repl.Main
> primaryResource spark-shell
> name Spark shell
> childArgs []
> jars null
> packages
> org.apache.sis.core:sis-referencing:1.0,org.apache.sis.non-free:sis-embedded-data:1.0
> packagesExclusions null
> repositories null
> verbose true
> Spark properties used, including those specified through
> --conf and those from the properties file null:
> Ivy Default Cache set to: /home/<user>/.ivy2/cache
> The jars for the packages stored in: /home/<user>/.ivy2/jars
> :: loading settings :: url =
> jar:[file:/home/|file:///home/]<user>/spark/spark-2.4.4-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
> org.apache.sis.core#sis-referencing added as a dependency
> org.apache.sis.non-free#sis-embedded-data added as a dependency
> :: resolving dependencies ::
> org.apache.spark#spark-submit-parent-83046344-c80a-4a8a-9d41-e6145875a27a;1.0
> confs: [default]
> found org.apache.sis.core#sis-referencing;1.0 in central
> found org.apache.sis.core#sis-utility;1.0 in central
> found javax.measure#unit-api;1.0 in central
> found org.opengis#geoapi;3.0.1 in central
> found org.apache.sis.core#sis-metadata;1.0 in central
> found org.apache.sis.non-free#sis-embedded-data;1.0 in central
> found org.apache.derby#derby;10.14.2.0 in central
> downloading
> [https://repo1.maven.org/maven2/org/apache/sis/core/sis-referencing/1.0/sis-referencing-1.0.jar]
> ...
> [SUCCESSFUL ] org.apache.sis.core#sis-referencing;1.0!sis-referencing.jar
> (245ms)
> downloading
> [https://repo1.maven.org/maven2/org/apache/sis/non-free/sis-embedded-data/1.0/sis-embedded-data-1.0.jar]
> ...
> [SUCCESSFUL ]
> org.apache.sis.non-free#sis-embedded-data;1.0!sis-embedded-data.jar (489ms)
> downloading
> [https://repo1.maven.org/maven2/org/apache/sis/core/sis-utility/1.0/sis-utility-1.0.jar]
> ...
> [SUCCESSFUL ] org.apache.sis.core#sis-utility;1.0!sis-utility.jar (92ms)
> downloading
> [https://repo1.maven.org/maven2/org/apache/sis/core/sis-metadata/1.0/sis-metadata-1.0.jar]
> ...
> [SUCCESSFUL ] org.apache.sis.core#sis-metadata;1.0!sis-metadata.jar (134ms)
> :: resolution report :: resolve 7312ms :: artifacts dl 970ms
> :: modules in use:
> javax.measure#unit-api;1.0 from central in [default]
> org.apache.derby#derby;10.14.2.0 from central in [default]
> org.apache.sis.core#sis-metadata;1.0 from central in [default]
> org.apache.sis.core#sis-referencing;1.0 from central in [default]
> org.apache.sis.core#sis-utility;1.0 from central in [default]
> org.apache.sis.non-free#sis-embedded-data;1.0 from central in [default]
> org.opengis#geoapi;3.0.1 from central in [default]
> ---------------------------------------------------------------------
> | |modules||artifacts||
> |conf|number|search|dwnlded|evicted||number|dwnlded|
> ---------------------------------------------------------------------
> |default|7|6|6|0||7|4|
> ---------------------------------------------------------------------
> :: problems summary ::
> :::: ERRORS
> unknown resolver null
> unknown resolver null
> unknown resolver null
> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> :: retrieving ::
> org.apache.spark#spark-submit-parent-83046344-c80a-4a8a-9d41-e6145875a27a
> confs: [default]
> 7 artifacts copied, 0 already retrieved (10240kB/20ms)
> Main class:
> org.apache.spark.repl.Main
> Arguments:
> Spark config:
>
> (spark.jars,[file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-referencing-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.apache.sis.non-free_sis-embedded-data-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-utility-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-metadata-1.0.jar,[file:///home/]<user>/.ivy2/jars/javax.measure_unit-api-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.opengis_geoapi-3.0.1.jar,[file:///home/]<user>/.ivy2/jars/org.apache.derby_derby-10.14.2.0.jar)
> (spark.app.name,Spark shell)
> (spark.ui.showConsoleProgress,true)
> (spark.submit.deployMode,client)
> (spark.master,local[*])
>
> (spark.repl.local.jars,[file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-referencing-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.apache.sis.non-free_sis-embedded-data-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-utility-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-metadata-1.0.jar,[file:///home/]<user>/.ivy2/jars/javax.measure_unit-api-1.0.jar,[file:///home/]<user>/.ivy2/jars/org.opengis_geoapi-3.0.1.jar,[file:///home/]<user>/.ivy2/jars/org.apache.derby_derby-10.14.2.0.jar)
> Classpath elements:
> [file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-referencing-1.0.jar
>
> [file:///home/]<user>/.ivy2/jars/org.apache.sis.non-free_sis-embedded-data-1.0.jar
> [file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-utility-1.0.jar
> [file:///home/]<user>/.ivy2/jars/org.apache.sis.core_sis-metadata-1.0.jar
> [file:///home/]<user>/.ivy2/jars/javax.measure_unit-api-1.0.jar
> [file:///home/]<user>/.ivy2/jars/org.opengis_geoapi-3.0.1.jar
> [file:///home/]<user>/.ivy2/jars/org.apache.derby_derby-10.14.2.0.jar
> 20/03/08 15:41:58 WARN Utils: Service 'SparkUI' could not bind on port 4040.
> Attempting port 4041.
> 20/03/08 15:41:58 WARN Utils: Service 'SparkUI' could not bind on port 4041.
> Attempting port 4042.
> Spark context Web UI available at
> [http://192.168.10.221:4042|http://192.168.10.221:4042/]
> Spark context available as 'sc' (master = local[*], app id =
> local-1583678518329).
> Spark session available as 'spark'.
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/_,_/_/ /_/_\ version 2.4.4
> /_/
> Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> *import org.apache.sis.geometry.DirectPosition2D*
> import org.apache.sis.geometry.DirectPosition2D
> scala> *import org.apache.sis.referencing.CRS*
> import org.apache.sis.referencing.CRS
> scala> *val position = new DirectPosition2D (678544.6095273666,
> 248462.881516265)*
> position: org.apache.sis.geometry.DirectPosition2D = POINT(678544.6095273666
> 248462.881516265)
> scala> *println ("Source: " + position)*
> Source: POINT(678544.6095273666 248462.881516265)
> scala>
> scala> *val sourceCRS = CRS.forCode ("EPSG:21781")*
> 20/03/08 15:43:14 WARN factory: Can not use the EPSG geodetic parameters:
> Failed to start database 'classpath:SIS_DATA/Databases/SpatialMetadata' with
> class loader sun.misc.Launcher$AppClassLoader@f2a0b8e, see the next exception
> for details.
> org.opengis.referencing.NoSuchAuthorityCodeException: No
> ‘CoordinateReferenceSystem’ object found for code “21781”. However only a
> subset of the EPSG geodetic dataset has been queried. See
> [http://sis.apache.org/epsg.html] for instruction about how to install the
> full EPSG database.
> at
> org.apache.sis.referencing.EPSGFactoryFallback.predefined(EPSGFactoryFallback.java:355)
> at
> org.apache.sis.referencing.EPSGFactoryFallback.createCoordinateReferenceSystem(EPSGFactoryFallback.java:231)
> at org.apache.sis.referencing.CRS.forCode(CRS.java:236)
> ... 49 elided
> scala> *val targetCRS = CRS.forCode ("EPSG:4326")*
> 20/03/08 15:43:17 WARN system: Can not connect to “jdbc/SpatialMetadata”.
> java.sql.SQLException: Failed to start database
> 'classpath:SIS_DATA/Databases/SpatialMetadata' with class loader
> sun.misc.Launcher$AppClassLoader@f2a0b8e, see the next exception for details.
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
> at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
> at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
> at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
> at org.apache.derby.jdbc.BasicEmbeddedDataSource40.getConnection(Unknown
> Source)
> at org.apache.derby.jdbc.BasicEmbeddedDataSource40.getConnection(Unknown
> Source)
> at
> org.apache.sis.metadata.sql.MetadataSource.connection(MetadataSource.java:490)
> at
> org.apache.sis.metadata.sql.MetadataSource.install(MetadataSource.java:457)
> at
> org.apache.sis.metadata.sql.MetadataSource.getProvided(MetadataSource.java:318)
> at
> org.apache.sis.internal.simple.CitationConstant.delegate(CitationConstant.java:177)
> at
> org.apache.sis.internal.simple.CitationConstant.getIdentifiers(CitationConstant.java:204)
> at
> org.apache.sis.internal.metadata.Identifiers.getIdentifier(Identifiers.java:117)
> at
> org.apache.sis.referencing.ImmutableIdentifier.formatTo(ImmutableIdentifier.java:517)
> at org.apache.sis.io.wkt.Formatter.append(Formatter.java:691)
> at org.apache.sis.io.wkt.Formatter.appendComplement(Formatter.java:849)
> at org.apache.sis.io.wkt.Formatter.append(Formatter.java:738)
> at
> org.apache.sis.io.wkt.FormattableObject.formatWKT(FormattableObject.java:200)
> at
> org.apache.sis.io.wkt.FormattableObject.toString(FormattableObject.java:126)
> at
> scala.runtime.ScalaRunTime$.scala$runtime$ScalaRunTime$$inner$1(ScalaRunTime.scala:332)
> at scala.runtime.ScalaRunTime$.stringOf(ScalaRunTime.scala:337)
> at scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345)
> at $line21.$eval$.$print$lzycompute(<console>:10)
> at $line21.$eval$.$print(<console>:6)
> at $line21.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
> at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
> at
> scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
> at
> scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
> at
> scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
> at
> scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
> at
> scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
> at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:819)
> at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:691)
> at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
> at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:425)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:285)
> at org.apache.spark.repl.SparkILoop.runClosure(SparkILoop.scala:159)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:182)
> at org.apache.spark.repl.Main$.doMain(Main.scala:78)
> at org.apache.spark.repl.Main$.main(Main.scala:58)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: ERROR XJ040: Failed to start database
> 'classpath:SIS_DATA/Databases/SpatialMetadata' with class loader
> sun.misc.Launcher$AppClassLoader@f2a0b8e, see the next exception for details.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> ... 65 more
> Caused by: ERROR XCL20: Catalogs at version level 'null' cannot be upgraded
> to version level '10.12'.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> at org.apache.derby.impl.sql.catalog.DD_Version.upgradeIfNeeded(Unknown
> Source)
> at
> org.apache.derby.impl.sql.catalog.DataDictionaryImpl.loadDictionaryTables(Unknown
> Source)
> at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
> Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown
> Source)
> at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown
> Source)
> at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown
> Source)
> at org.apache.derby.impl.db.BasicDatabase$5.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.apache.derby.impl.db.BasicDatabase.bootServiceModule(Unknown Source)
> at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
> at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
> Source)
> at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown
> Source)
> at
> org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown
> Source)
> at
> org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown
> Source)
> at
> org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown
> Source)
> at
> org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedConnection$4.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.apache.derby.impl.jdbc.EmbedConnection.startPersistentService(Unknown
> Source)
> ... 62 more
> targetCRS: org.opengis.referencing.crs.CoordinateReferenceSystem =
> GeodeticCRS["WGS 84",
> Datum["World Geodetic System 1984",
> Ellipsoid["WGS 84", 6378137.0, 298.257223563]],
> CS[ellipsoidal, 2],
> Axis["Geodetic latitude (φ)", north],
> Axis["Geodetic longitude (λ)", east],
> Unit["degree", 0.017453292519943295],
> Scope["Horizontal component of 3D system."],
> Area["World"],
> BBox[-90.00, -180.00, 90.00, 180.00],
> Id["EPSG", 4326, URI["urn:ogc:def:crs:EPSG::4326"]]]
> scala>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
