fstolba opened a new issue #8858: export-metadata throws java.lang.NullPointerException URL: https://github.com/apache/incubator-druid/issues/8858 I'm trying to migrate my segments from local storage to S3. When running `export-metadata` with parameters `-b`/`-k` (rewriting deep storage location to S3), it crashes while exporting the segments table. Exporting without rewriting works fine. I suspect it might be due to the fact that not all segments in the table refer to the same storage location. Recently created and new segments are actually stored in the target bucket already (storage location in `common/runtime.conf` was changed to reflect the S3 configuration). ### Affected Version 0.16 ### Description ``` bash-4.2# java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[] org.apache.druid.cli.Main tools export-metadata --connectURI "jdbc:derby://localhost:1527/var/druid/metadata.db;" -o /tmp -b druid-bucket -k druid/segments Nov 12, 2019 10:11:27 AM org.hibernate.validator.internal.util.Version <clinit> INFO: HV000001: Hibernate Validator 5.1.3.Final 2019-11-12T10:11:27,504 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.common.config.NullValueHandlingConfig] from props[druid.generic.] as [org.apache.druid.common.config.NullValueHandlingConfig@3da05287] 2019-11-12T10:11:27,552 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='extensions', useExtensionClassloaderFirst=false, hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[]}] 2019-11-12T10:11:27,640 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}] 2019-11-12T10:11:27,795 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.aws.AWSModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension 2019-11-12T10:11:27,797 INFO [main] org.apache.druid.initialization.Initialization - Adding implementation [org.apache.druid.common.gcp.GcpModule] for class [interface org.apache.druid.initialization.DruidModule] from classpath extension 2019-11-12T10:11:28,042 INFO [main] org.apache.druid.guice.Jerseys - Adding Jersey resource: org.apache.druid.server.StatusResource 2019-11-12T10:11:28,476 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[interface org.apache.druid.server.security.Escalator] from props[druid.escalator.] as [NoopEscalator{}] 2019-11-12T10:11:28,486 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.CuratorConfig] from props[druid.zk.service.] as [org.apache.druid.curator.CuratorConfig@3015db78] 2019-11-12T10:11:28,490 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [org.apache.druid.curator.ExhibitorConfig@6b760460] 2019-11-12T10:11:28,543 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode 2019-11-12T10:11:28,545 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29 2019-11-12T10:11:28,583 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [org.apache.druid.server.initialization.ZkPathsConfig@58d3f4be] 2019-11-12T10:11:28,591 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain=null, authorizers=null, unsecuredPaths=[], allowUnauthenticatedHttpOptions=false}] 2019-11-12T10:11:28,642 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [org.apache.druid.server.metrics.DruidMonitorSchedulerConfig@7bc6d27a] 2019-11-12T10:11:28,646 INFO [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}] 2019-11-12T10:11:28,647 INFO [main] org.apache.druid.server.emitter.EmitterModule - Underlying emitter for ServiceEmitter: org.apache.druid.java.util.emitter.core.NoopEmitter@1ac45389 2019-11-12T10:11:28,647 INFO [main] org.apache.druid.server.emitter.EmitterModule - Extra service dimensions: {version=0.16.0-incubating} 2019-11-12T10:11:28,648 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.query.ExecutorServiceMonitor@10bd9df0] 2019-11-12T10:11:28,648 INFO [main] org.apache.druid.server.metrics.MetricsModule - Adding monitor[org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@732c9b5c] 2019-11-12T10:11:28,720 INFO [main] org.apache.druid.metadata.storage.derby.DerbyConnector - Derby connector instantiated with metadata storage [org.apache.druid.metadata.NoopMetadataStorageProvider$1]. 2019-11-12T10:11:28,720 INFO [main] org.apache.druid.cli.ExportMetadata - Exporting datasource table: druid_dataSource 2019-11-12T10:11:28,862 INFO [main] org.apache.druid.cli.ExportMetadata - Exporting segments table: druid_segments Exception in thread "main" java.lang.NullPointerException at org.apache.druid.cli.ExportMetadata.getUniqueIDFromLocalLoadSpec(ExportMetadata.java:537) at org.apache.druid.cli.ExportMetadata.makePayloadWithConvertedLoadSpec(ExportMetadata.java:429) at org.apache.druid.cli.ExportMetadata.rewriteSegmentsExport(ExportMetadata.java:407) at org.apache.druid.cli.ExportMetadata.run(ExportMetadata.java:225) at org.apache.druid.cli.Main.main(Main.java:113) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
