[ https://issues.apache.org/jira/browse/SPARK-36875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-36875. ---------------------------------- Resolution: Invalid Apparently It failed to connect to the Yarn cluster. Seems like an environment issue > 21/09/28 11:18:51 WARN YarnScheduler: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient resources > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-36875 > URL: https://issues.apache.org/jira/browse/SPARK-36875 > Project: Spark > Issue Type: Bug > Components: Spark Core, Spark Submit > Affects Versions: 3.1.2 > Environment: Eclipse > Hadoop 3.3 > Spark3.1.2-hadoop3.2 > Dependencies > > {noformat} > <dependency> > <groupId>org.apache.spark</groupId> > <!-- scala version 2.12 --> > <artifactId>spark-core_2.12</artifactId> > <version>3.1.2</version> > </dependency> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-sql_2.12</artifactId> > <version>3.1.2</version> <exclusions> > <exclusion> > <artifactId>janino</artifactId> > <groupId>org.codehaus.janino</groupId> > </exclusion> </exclusions> </dependency> > <dependency> <groupId>org.codehaus.janino</groupId> > <artifactId>janino</artifactId> > <version>3.0.8</version> > </dependency> > <dependency> <groupId>org.apache.spark</groupId> > <artifactId>spark-yarn_2.12</artifactId> > <version>3.1.2</version> <scope>provided</scope> > </dependency> > <dependency> > <groupId>org.scala-lang</groupId> > <artifactId>scala-library</artifactId> > <version>2.12.13</version> > </dependency> > Enviroment Variables set in eclipse: SPARK_HOME path/to/my/sparkfolder > OS Linux with UBUNTU 20 > The test is launched on my first user davben. > Spark folder and hadoop are on my second user hadoop{noformat} > > > Reporter: Davide Benedetto > Priority: Major > Labels: Eclipse, YARN, dataset, hadoop, spark-conf, spark-core > Original Estimate: 48h > Remaining Estimate: 48h > > Hi, I am running a spark job with yarn programmatically using Eclipse IDE. > Here I > 1: open the spark session passing a SparkConf as input parameter, > > {quote} > {code:java} > System.setProperty("hadoop.home.dir", "/home/hadoop/hadoop"); > System.setProperty("hadoop.home.dir", "/home/hadoop/hadoop"); > System.setProperty("SPARK_YARN_MODE", "yarn"); > System.setProperty("HADOOP_USER_NAME", "hadoop"); > SparkConf sparkConf = new > SparkConf().setAppName("simpleTest2").setMaster("yarn") > .set("spark.executor.memory", "1g") > .set("deploy.mode", "cluster") > .set("spark.yarn.stagingDir", "hdfs://localhost:9000/user/hadoop/") > .set("spark.yarn.am.memory", "512m") > .set("spark.dynamicAllocation.minExecutors","1") > .set("spark.dynamicAllocation.maxExecutors","40") > .set("spark.dynamicAllocation.initialExecutors","2") > .set("spark.shuffle.service.enabled", "true") > .set("spark.dynamicAllocation.enabled", "false") > .set("spark.cores.max", "1") > .set("spark.yarn.executor.memoryOverhead", "500m") > .set("spark.executor.instances","2") > .set("spark.executor.memory","500m") > .set("spark.num.executors","2") > .set("spark.executor.cores","1") > .set("spark.worker.instances","1") > .set("spark.worker.memory","512m") > .set("spark.worker.max.heapsize","512m") > .set("spark.worker.cores","1") > .set("maximizeResourceAllocation", "true") > .set("spark.yarn.nodemanager.resource.cpu-vcores","4") > .set("spark.yarn.submit.file.replication", "1") > SparkSession spark = SparkSession.builder().config(sparkConf).getOrCreate(); > {code} > > {quote} > 2: Create a dataset of two Rows and i Show them > {code:java} > List<Row> rows = new ArrayList<>(); List<Row> rows = new ArrayList<>(); > rows.add(RowFactory.create("a", "b")); rows.add(RowFactory.create("b", "c")); > rows.add(RowFactory.create("a", "a")); > StructType structType = new StructType(); structType = > structType.add("edge_1", DataTypes.StringType, false); structType = > structType.add("edge_2", DataTypes.StringType, false); ExpressionEncoder<Row> > edgeEncoder = RowEncoder.apply(structType); > Dataset<Row> edge = spark.createDataset(rows, edgeEncoder); edge.show(); > > {code} > {{From now it is all Ok, the job is submitted on hadoop and the rows are > showed correctly}} > {{3: I perform a Map that upper cases the elements in the row}} > > {quote} > {code:java} > Dataset<Row> edge2 = edge.map(new MyFunction2(), edgeEncoder); Dataset<Row> > edge2 = edge.map(new MyFunction2(), edgeEncoder);{code} > {quote} > > {quote}{{ public static class MyFunction2 implements MapFunction<Row, Row> { > public static class MyFunction2 implements MapFunction<Row, Row> {}} > {{ /** * */ private static final long serialVersionUID = 1L;}} > {{ @Override public Row call(Row v1) throws Exception \{ String el1 = > v1.get(0).toString().toUpperCase(); String el2 = > v1.get(1).toString().toUpperCase(); return RowFactory.create(el1,el2); }}} > {{ }}} > {quote} > {{4: Then I show the dataset after map is performed}} > {quote} > {code:java} > edge2.show();{code} > {quote} > {{And precisely here the log Loops saying }} > {{21/09/28 11:18:51 WARN YarnScheduler: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient resources}} > > {quote}{{Here is the Full log}} > {{log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN No > appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please > initialize the log4j system properly.log4j:WARN See > http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Using > Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties21/09/28 03:05:16 WARN Utils: Your > hostname, davben-lubuntu resolves to a loopback address: 127.0.1.1; using > 192.168.1.36 instead (on interface wlo1)21/09/28 03:05:16 WARN Utils: Set > SPARK_LOCAL_IP if you need to bind to another addressWARNING: An illegal > reflective access operation has occurredWARNING: Illegal reflective access by > org.apache.spark.unsafe.Platform > (file:/home/davben/.m2/repository/org/apache/spark/spark-unsafe_2.12/3.1.2/spark-unsafe_2.12-3.1.2.jar) > to constructor java.nio.DirectByteBuffer(long,int)WARNING: Please consider > reporting this to the maintainers of org.apache.spark.unsafe.PlatformWARNING: > Use --illegal-access=warn to enable warnings of further illegal reflective > access operationsWARNING: All illegal access operations will be denied in a > future release21/09/28 03:05:16 WARN SparkConf: The configuration key > 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and > may be removed in the future. Please use the new key > 'spark.executor.memoryOverhead' instead.21/09/28 03:05:16 WARN SparkConf: The > configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as > of Spark 2.3 and may be removed in the future. Please use the new key > 'spark.executor.memoryOverhead' instead.21/09/28 03:05:16 INFO SparkContext: > Running Spark version 3.1.221/09/28 03:05:16 INFO ResourceUtils: > ==============================================================21/09/28 > 03:05:16 INFO ResourceUtils: No custom resources configured for > spark.driver.21/09/28 03:05:16 INFO ResourceUtils: > ==============================================================21/09/28 > 03:05:16 INFO SparkContext: Submitted application: simpleTest221/09/28 > 03:05:16 INFO ResourceProfile: Default ResourceProfile created, executor > resources: Map(memoryOverhead -> name: memoryOverhead, amount: 500, script: , > vendor: , cores -> name: cores, amount: 1, script: , vendor: , memory -> > name: memory, amount: 500, script: , vendor: , offHeap -> name: offHeap, > amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, > amount: 1.0)21/09/28 03:05:16 INFO ResourceProfile: Limiting resource is cpus > at 1 tasks per executor21/09/28 03:05:16 INFO ResourceProfileManager: Added > ResourceProfile id: 021/09/28 03:05:16 INFO SecurityManager: Changing view > acls to: davben,hadoop21/09/28 03:05:16 INFO SecurityManager: Changing modify > acls to: davben,hadoop21/09/28 03:05:16 INFO SecurityManager: Changing view > acls groups to: 21/09/28 03:05:16 INFO SecurityManager: Changing modify acls > groups to: 21/09/28 03:05:16 INFO SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(davben, hadoop); groups with view permissions: Set(); users with modify > permissions: Set(davben, hadoop); groups with modify permissions: > Set()21/09/28 03:05:16 INFO Utils: Successfully started service 'sparkDriver' > on port 38955.21/09/28 03:05:17 INFO SparkEnv: Registering > MapOutputTracker21/09/28 03:05:17 INFO SparkEnv: Registering > BlockManagerMaster21/09/28 03:05:17 INFO BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information21/09/28 03:05:17 INFO BlockManagerMasterEndpoint: > BlockManagerMasterEndpoint up21/09/28 03:05:17 INFO SparkEnv: Registering > BlockManagerMasterHeartbeat21/09/28 03:05:17 INFO DiskBlockManager: Created > local directory at /tmp/blockmgr-33cfa046-2c7b-45b5-9cf1-c0f20195a98421/09/28 > 03:05:17 INFO MemoryStore: MemoryStore started with capacity 994.8 > MiB21/09/28 03:05:17 INFO SparkEnv: Registering > OutputCommitCoordinator21/09/28 03:05:17 INFO Utils: Successfully started > service 'SparkUI' on port 4040.21/09/28 03:05:17 INFO SparkUI: Bound SparkUI > to 0.0.0.0, and started at http://davben-lubuntu.home:404021/09/28 03:05:17 > INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:803221/09/28 03:05:18 > INFO Client: Requesting a new application from cluster with 1 > NodeManagers21/09/28 03:05:18 INFO Configuration: resource-types.xml not > found21/09/28 03:05:18 INFO ResourceUtils: Unable to find > 'resource-types.xml'.21/09/28 03:05:18 INFO Client: Verifying our application > has not requested more than the maximum memory capability of the cluster > (8192 MB per container)21/09/28 03:05:18 INFO Client: Will allocate AM > container, with 896 MB memory including 384 MB overhead21/09/28 03:05:18 INFO > Client: Setting up container launch context for our AM21/09/28 03:05:18 INFO > Client: Setting up the launch environment for our AM container21/09/28 > 03:05:18 INFO Client: Preparing resources for our AM container21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/HikariCP-2.5.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/JLargeArrays-1.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/JTransforms-3.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/RoaringBitmap-0.9.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/ST4-4.0.4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/accessors-smart-1.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/activation-1.1.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/aircompressor-0.10.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/algebra_2.12-2.0.0-M2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/antlr-runtime-3.5.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/antlr4-runtime-4.8-1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/aopalliance-1.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/aopalliance-repackaged-2.6.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/arpack_combined_all-0.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/arrow-format-2.0.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/arrow-memory-core-2.0.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/arrow-memory-netty-2.0.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/arrow-vector-2.0.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/audience-annotations-0.5.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/automaton-1.11-8.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/avro-1.8.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/avro-ipc-1.8.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/avro-mapred-1.8.2-hadoop2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/bonecp-0.8.0.RELEASE.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/breeze-macros_2.12-1.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/breeze_2.12-1.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/cats-kernel_2.12-2.0.0-M4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/chill-java-0.9.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/chill_2.12-0.9.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-beanutils-1.9.4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-cli-1.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-codec-1.10.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-collections-3.2.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-compiler-3.0.16.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-compress-1.20.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-configuration2-2.1.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-crypto-1.1.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-daemon-1.0.13.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-dbcp-1.4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-httpclient-3.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-io-2.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-lang-2.6.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-lang3-3.10.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-logging-1.1.3.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-math3-3.4.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-net-3.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-pool-1.5.4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/commons-text-1.6.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/compress-lzf-1.0.3.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/core-1.1.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/curator-client-2.13.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/curator-framework-2.13.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/curator-recipes-2.13.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/datanucleus-api-jdo-4.2.4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/datanucleus-core-4.1.17.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/datanucleus-rdbms-4.1.19.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/derby-10.12.1.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/dnsjava-2.1.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/ehcache-3.3.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/flatbuffers-java-1.9.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/generex-1.0.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/geronimo-jcache_1.0_spec-1.0-alpha-1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/gson-2.2.4.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/guava-14.0.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/guice-4.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/guice-servlet-4.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-annotations-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-auth-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-client-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-common-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-hdfs-client-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-mapreduce-client-common-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-mapreduce-client-core-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-mapreduce-client-jobclient-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-yarn-api-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-yarn-client-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-yarn-common-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-yarn-registry-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-yarn-server-common-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hadoop-yarn-server-web-proxy-3.2.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-beeline-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-cli-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-common-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-exec-2.3.7-core.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-jdbc-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-llap-common-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-metastore-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-serde-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-service-rpc-3.1.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-shims-0.23-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-shims-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-shims-common-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-shims-scheduler-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-storage-api-2.7.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hive-vector-code-gen-2.3.7.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hk2-api-2.6.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hk2-locator-2.6.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/hk2-utils-2.6.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/htrace-core4-4.1.0-incubating.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/httpclient-4.5.6.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/httpcore-4.4.12.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/istack-commons-runtime-3.0.8.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/ivy-2.4.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-annotations-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-core-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-core-asl-1.9.13.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-databind-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-dataformat-yaml-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-datatype-jsr310-2.11.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-jaxrs-base-2.9.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-jaxrs-json-provider-2.9.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-mapper-asl-1.9.13.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-module-jaxb-annotations-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-module-paranamer-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jackson-module-scala_2.12-2.10.0.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.activation-api-1.2.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.annotation-api-1.3.5.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.inject-2.6.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.servlet-api-4.0.3.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.validation-api-2.0.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.ws.rs-api-2.1.6.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jakarta.xml.bind-api-2.3.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/janino-3.0.16.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/javassist-3.25.0-GA.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/javax.inject-1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/javax.jdo-3.2.0-m3.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/javolution-5.5.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jaxb-api-2.2.11.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jaxb-runtime-2.3.2.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jcip-annotations-1.0-1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jcl-over-slf4j-1.7.30.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jdo-api-3.0.1.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-client-2.30.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-common-2.30.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-container-servlet-2.30.jar21/09/28 > 03:05:18 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-container-servlet-core-2.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-hk2-2.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-media-jaxb-2.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jersey-server-2.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jline-2.14.6.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/joda-time-2.10.5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jodd-core-3.5.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/jpam-1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/json-1.8.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/json-smart-2.3.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/json4s-ast_2.12-3.7.0-M5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/json4s-core_2.12-3.7.0-M5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/json4s-jackson_2.12-3.7.0-M5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/json4s-scalap_2.12-3.7.0-M5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/jsp-api-2.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jsr305-3.0.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/jta-1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/jul-to-slf4j-1.7.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-admin-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-client-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-common-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-core-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-crypto-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-identity-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-server-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-simplekdc-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerb-util-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerby-asn1-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerby-config-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerby-pkix-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerby-util-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kerby-xdr-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kryo-shaded-4.0.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-client-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-admissionregistration-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-apiextensions-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-apps-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-autoscaling-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-batch-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-certificates-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-common-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-coordination-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-core-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-discovery-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-events-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-extensions-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-metrics-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-networking-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-policy-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-rbac-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-scheduling-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-settings-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/kubernetes-model-storageclass-4.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/leveldbjni-all-1.8.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/libfb303-0.9.3.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/libthrift-0.12.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/log4j-1.2.17.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/logging-interceptor-3.12.12.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/lz4-java-1.7.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/machinist_2.12-0.6.8.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/macro-compat_2.12-1.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/mesos-1.4.0-shaded-protobuf.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/metrics-core-4.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/metrics-graphite-4.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/metrics-jmx-4.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/metrics-json-4.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/metrics-jvm-4.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/minlog-1.3.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/netty-all-4.1.51.Final.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/nimbus-jose-jwt-4.41.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/objenesis-2.6.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/okhttp-2.7.5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/okhttp-3.12.12.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/okio-1.14.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/opencsv-2.3.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/orc-core-1.5.12.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/orc-mapreduce-1.5.12.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/orc-shims-1.5.12.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/oro-2.0.8.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/osgi-resource-locator-1.0.3.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/paranamer-2.8.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/parquet-column-1.10.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/parquet-common-1.10.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/parquet-encoding-1.10.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/parquet-format-2.4.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/parquet-hadoop-1.10.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/parquet-jackson-1.10.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/protobuf-java-2.5.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/py4j-0.10.9.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/pyrolite-4.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/re2j-1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/scala-collection-compat_2.12-2.1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/scala-compiler-2.12.10.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/scala-library-2.12.10.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/scala-parser-combinators_2.12-1.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/scala-reflect-2.12.10.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/scala-xml_2.12-1.2.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/shapeless_2.12-2.3.3.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/shims-0.9.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/slf4j-api-1.7.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/slf4j-log4j12-1.7.30.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/snakeyaml-1.24.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/snappy-java-1.1.8.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-catalyst_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-core_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-graphx_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-hive-thriftserver_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-hive_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-kubernetes_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-kvstore_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-launcher_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/spark-libs.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-mesos_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-mllib-local_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-mllib_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-network-common_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-network-shuffle_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-repl_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-sketch_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-sql_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-streaming_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-tags_2.12-3.1.2-tests.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-tags_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-unsafe_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spark-yarn_2.12-3.1.2.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spire-macros_2.12-0.17.0-M1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spire-platform_2.12-0.17.0-M1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spire-util_2.12-0.17.0-M1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/spire_2.12-0.17.0-M1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/stax-api-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/stax2-api-3.1.4.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/stream-2.9.6.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/super-csv-2.2.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/threeten-extra-1.5.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/token-provider-1.0.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/transaction-api-1.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/univocity-parsers-2.9.1.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/velocity-1.5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/woodstox-core-5.0.3.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/xbean-asm7-shaded-4.15.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying hdfs://localhost:9000/user/hadoop/davben/jars/xz-1.5.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/zjsonpatch-0.3.0.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/zookeeper-3.4.14.jar21/09/28 > 03:05:19 INFO Client: Source and destination file systems are the same. Not > copying > hdfs://localhost:9000/user/hadoop/davben/jars/zstd-jni-1.4.8-1.jar21/09/28 > 03:05:19 INFO Client: Uploading resource > file:/tmp/spark-3b097b0a-1f72-4cb1-9dd1-f419cdbd645f/__spark_conf__17045972699920737203.zip > -> > hdfs://localhost:9000/user/hadoop/hadoop/.sparkStaging/application_1632774234019_0016/__spark_conf__.zip21/09/28 > 03:05:19 INFO SecurityManager: Changing view acls to: davben,hadoop21/09/28 > 03:05:19 INFO SecurityManager: Changing modify acls to: davben,hadoop21/09/28 > 03:05:19 INFO SecurityManager: Changing view acls groups to: 21/09/28 > 03:05:19 INFO SecurityManager: Changing modify acls groups to: 21/09/28 > 03:05:19 INFO SecurityManager: SecurityManager: authentication disabled; ui > acls disabled; users with view permissions: Set(davben, hadoop); groups with > view permissions: Set(); users with modify permissions: Set(davben, hadoop); > groups with modify permissions: Set()21/09/28 03:05:19 INFO Client: > Submitting application application_1632774234019_0016 to > ResourceManager21/09/28 03:05:19 INFO YarnClientImpl: Submitted application > application_1632774234019_001621/09/28 03:05:20 INFO Client: Application > report for application_1632774234019_0016 (state: ACCEPTED)21/09/28 03:05:20 > INFO Client: client token: N/A diagnostics: AM container is launched, > waiting for AM container to Register with RM ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 queue: default start time: 1632791119751 final > status: UNDEFINED tracking URL: > http://davben-lubuntu:8088/proxy/application_1632774234019_0016/ user: > hadoop21/09/28 03:05:21 INFO Client: Application report for > application_1632774234019_0016 (state: ACCEPTED)21/09/28 03:05:22 INFO > Client: Application report for application_1632774234019_0016 (state: > ACCEPTED)21/09/28 03:05:23 INFO Client: Application report for > application_1632774234019_0016 (state: RUNNING)21/09/28 03:05:23 INFO Client: > client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.1.36 > ApplicationMaster RPC port: -1 queue: default start time: 1632791119751 final > status: UNDEFINED tracking URL: > http://davben-lubuntu:8088/proxy/application_1632774234019_0016/ user: > hadoop21/09/28 03:05:23 INFO YarnClientSchedulerBackend: Application > application_1632774234019_0016 has started running.21/09/28 03:05:23 INFO > Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port > 33377.21/09/28 03:05:23 INFO NettyBlockTransferService: Server created on > davben-lubuntu.home:3337721/09/28 03:05:23 INFO BlockManager: Using > org.apache.spark.storage.RandomBlockReplicationPolicy for block replication > policy21/09/28 03:05:23 INFO BlockManagerMaster: Registering BlockManager > BlockManagerId(driver, davben-lubuntu.home, 33377, None)21/09/28 03:05:23 > INFO BlockManagerMasterEndpoint: Registering block manager > davben-lubuntu.home:33377 with 994.8 MiB RAM, BlockManagerId(driver, > davben-lubuntu.home, 33377, None)21/09/28 03:05:23 INFO BlockManagerMaster: > Registered BlockManager BlockManagerId(driver, davben-lubuntu.home, 33377, > None)21/09/28 03:05:23 INFO BlockManager: external shuffle service port = > 733721/09/28 03:05:23 INFO BlockManager: Initialized BlockManager: > BlockManagerId(driver, davben-lubuntu.home, 33377, None)21/09/28 03:05:23 > INFO YarnClientSchedulerBackend: Add WebUI Filter. > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS > -> davben-lubuntu, PROXY_URI_BASES -> > http://davben-lubuntu:8088/proxy/application_1632774234019_0016), > /proxy/application_1632774234019_001621/09/28 03:05:24 INFO ServerInfo: > Adding filter to /metrics/json: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter21/09/28 03:05:25 > INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered > as NettyRpcEndpointRef(spark-client://YarnAM)21/09/28 03:05:47 INFO > YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling > beginning after waiting maxRegisteredResourcesWaitingTime: > 30000000000(ns)21/09/28 03:05:49 INFO SharedState: Setting > hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir > ('file:/home/davben/prometheux/projects/spark-reasoner/spark-warehouse').21/09/28 > 03:05:49 INFO SharedState: Warehouse path is > 'file:/home/davben/prometheux/projects/spark-reasoner/spark-warehouse'.21/09/28 > 03:05:49 INFO ServerInfo: Adding filter to /SQL: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter21/09/28 03:05:49 > INFO ServerInfo: Adding filter to /SQL/json: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter21/09/28 03:05:49 > INFO ServerInfo: Adding filter to /SQL/execution: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter21/09/28 03:05:49 > INFO ServerInfo: Adding filter to /SQL/execution/json: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter21/09/28 03:05:49 > INFO ServerInfo: Adding filter to /static/sql: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter21/09/28 03:05:50 > INFO CodeGenerator: Code generated in 169.2098 ms21/09/28 03:05:50 INFO > CodeGenerator: Code generated in 13.789217 ms21/09/28 03:05:51 INFO > CodeGenerator: Code generated in 17.287752 ms}} > {{+------+------+}} > |edge_1|edge_2| > {{+------+------+| }} > {{a| b|| }} > {{a| }}{{a|}} > {{+------+------+}} > {{SHOW21/09/28 03:05:51 INFO CodeGenerator: Code generated in 45.072695 > ms21/09/28 03:05:51 INFO SparkContext: Starting job: show at > TrivialTests.java:124621/09/28 03:05:51 INFO DAGScheduler: Got job 0 (show at > TrivialTests.java:1246) with 1 output partitions21/09/28 03:05:51 INFO > DAGScheduler: Final stage: ResultStage 0 (show at > TrivialTests.java:1246)21/09/28 03:05:51 INFO DAGScheduler: Parents of final > stage: List()21/09/28 03:05:51 INFO DAGScheduler: Missing parents: > List()21/09/28 03:05:51 INFO DAGScheduler: Submitting ResultStage 0 > (MapPartitionsRDD[2] at show at TrivialTests.java:1246), which has no missing > parents21/09/28 03:05:51 INFO MemoryStore: Block broadcast_0 stored as values > in memory (estimated size 13.3 KiB, free 994.8 MiB)21/09/28 03:05:51 INFO > MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated > size 5.3 KiB, free 994.8 MiB)21/09/28 03:05:51 INFO BlockManagerInfo: Added > broadcast_0_piece0 in memory on davben-lubuntu.home:33377 (size: 5.3 KiB, > free: 994.8 MiB)21/09/28 03:05:51 INFO SparkContext: Created broadcast 0 from > broadcast at DAGScheduler.scala:138821/09/28 03:05:51 INFO DAGScheduler: > Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at show at > TrivialTests.java:1246) (first 15 tasks are for partitions Vector(0))21/09/28 > 03:05:51 INFO YarnScheduler: Adding task set 0.0 with 1 tasks resource > profile 021/09/28 03:06:06 WARN YarnScheduler: Initial job has not accepted > any resources; check your cluster UI to ensure that workers are registered > and have sufficient resources21/09/28 03:06:21 WARN YarnScheduler: Initial > job has not accepted any resources; check your cluster UI to ensure that > workers are registered and have sufficient resources21/09/28 03:06:36 WARN > YarnScheduler: Initial job has not accepted any resources; check your cluster > UI to ensure that workers are registered and have sufficient > resources21/09/28 03:06:51 WARN YarnScheduler: Initial job has not accepted > any resources; check your cluster UI to ensure that workers are registered > and have sufficient resources}} > {quote} > > {{It is strange because if the same job is run locally (setMaster(local[*])) > all works properly.}} > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org