[jira] [Created] (FLINK-27130) unable to read custom System properties in job class
Adrian Zhong created FLINK-27130: Summary: unable to read custom System properties in job class Key: FLINK-27130 URL: https://issues.apache.org/jira/browse/FLINK-27130 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.13.6, 1.13.0 Reporter: Adrian Zhong I'm using Flink YARN-PER-JOB mode to submit a job. I'm wondering what is wrong, and how flink CliFrontend prevent job class to read system properties which seems to be per-jvm. I have searched all related issues, read unit tests for CliFrontend and DynamicProperties, however, I can't figure out. Here is my job class: ``` public static void main(String[] args) { String property = System.getProperty("kafka.start_from_timestamp"); if (property == null) { //-Dkafka.start_from_timestamp=164882880 System.err.println("-Dkafka.start_from_timestamp Not found"); System.err.println("This are Properties Found in this JVM:"); System.err.println(System.getProperties().stringPropertyNames()); } else { System.err.println("-Dkafka.start_from_timestamp is" + property); } // } ``` outputs: ``` -Dkafka.start_from_timestamp Not found This are Properties Found in this JVM: [zookeeper.sasl.client, java.runtime.name, sun.boot.library.path, java.vm.version, java.vm.vendor, java.vendor.url, path.separator, java.vm.name, file.encoding.pkg, user.country, sun.java.launcher, sun.os.patch.level, java.vm.specification.name, user.dir, java.runtime.version, java.awt.graphicsenv, java.endorsed.dirs, os.arch, java.io.tmpdir, line.separator, java.vm.specification.vendor, os.name, log4j.configuration, sun.jnu.encoding, java.library.path, java.specification.name, java.class.version, sun.management.compiler, os.version, user.home, user.timezone, java.awt.printerjob, file.encoding, java.specification.version, log4j.configurationFile, user.name, java.class.path, log.file, java.vm.specification.version, sun.arch.data.model, java.home, sun.java.command, java.specification.vendor, user.language, awt.toolkit, java.vm.info, java.version, java.ext.dirs, sun.boot.class.path, java.vendor, logback.configurationFile, java.security.auth.login.config, file.separator, java.vendor.url.bug, sun.cpu.endian, sun.io.unicode.encoding, sun.cpu.isalist] ``` Environment: JDK: Oracle 1.8/25.121-b13 Flink flink-1.13.0 What I have tried: ``` -Denv.java.opts.client="-Dkafka.start_from_timestamp=164882880" -Denv.java.opts="-Dkafka.start_from_timestamp=164882881" -Dkafka.start_from_timestamp=164882882 -yD env.java.opts="kafka.start_from_timestamp=164882883" ``` submit command: ``` bin/flink run --yarnjobManagerMemory 1G --yarntaskManagerMemory 1G --yarnqueue root.users.appuser --yarnslots 1 --yarnname SocketWindowWordCount -m yarn-cluster --class com.slankka.learn.rtc.SocketWindowWordCount -Denv.java.opts="-Dkafka.start_from_timestamp=164882881" -Dkafka.start_from_timestamp=164882882 -yD env.java.opts="kafka.start_from_timestamp=164882883" -d /data/files_upload/socketWindowWordCount.jar --hostname 10.11.159.156 --port 7890 ``` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-22769) yarnship do not support symbolic directory
Adrian Zhong created FLINK-22769: Summary: yarnship do not support symbolic directory Key: FLINK-22769 URL: https://issues.apache.org/jira/browse/FLINK-22769 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.13.0, 1.12.2, 1.13.1 Reporter: Adrian Zhong If we pass `-yt ` a symbolic directory, we will get an exception: {code:java} Caused by: java.lang.IllegalArgumentException: File to copy cannot be a directory: file:/path/to/sql/connectors/Caused by: java.lang.IllegalArgumentException: File to copy cannot be a directory: file:/path/to/sql/connectors/ at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138) at org.apache.flink.yarn.YarnApplicationFileUploader.uploadLocalFileToRemote(YarnApplicationFileUploader.java:197) at org.apache.flink.yarn.YarnApplicationFileUploader.registerSingleLocalResource(YarnApplicationFileUploader.java:179) at org.apache.flink.yarn.YarnApplicationFileUploader.registerMultipleLocalResources(YarnApplicationFileUploader.java:289) at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:870) {code} The root cause is not here but in : {code:java} final java.nio.file.Path shipPath = file.toPath(); final java.nio.file.Path parentPath = shipPath.getParent(); Files.walkFileTree( shipPath.toRealPath(), new SimpleFileVisitor() { @Override public FileVisitResult visitFile( java.nio.file.Path file, BasicFileAttributes attrs) { localPaths.add(new Path(file.toUri())); //root cause: localPath is an symbolic file, without any file under real path. relativePaths.add( new Path( localResourcesDirectory, parentPath.relativize(file).toString())); return FileVisitResult.CONTINUE; } }); {code} Please assign to me, I'd like to be a contributor. Any other please *do not implement* or create pr before this issue is assigned. Note: this bug is not os platform specific. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22639) ClassLoaderUtil cannot print classpath of FlinkUserCodeClassLoader
Adrian Zhong created FLINK-22639: Summary: ClassLoaderUtil cannot print classpath of FlinkUserCodeClassLoader Key: FLINK-22639 URL: https://issues.apache.org/jira/browse/FLINK-22639 Project: Flink Issue Type: Improvement Components: Runtime / Configuration, Table SQL / Runtime Affects Versions: 1.13.0 Reporter: Adrian Zhong Hello, community. I found FlinkUserCodeClassLoader is wrapping by SafetyNetWrapperClassLoader, but it cut getURL invoking chain. ClassLoaderUtil.getUserCodeClassLoaderInfo: {code:java} public static String getUserCodeClassLoaderInfo(ClassLoader loader) { if (loader instanceof URLClassLoader) { URLClassLoader cl = (URLClassLoader) loader; try { StringBuilder bld = new StringBuilder(); if (cl == ClassLoader.getSystemClassLoader()) { bld.append("System ClassLoader: "); } else { bld.append("URL ClassLoader:"); } for (URL url : cl.getURLs()) { } }{code} {code:java} SafetyNetWrapperClassLoader(FlinkUserCodeClassLoader inner, ClassLoader parent) { super(new URL[0], parent); this.inner = inner; } {code} The url passing empty array, SafetyNetWrapperClassLoader.getURL should dispatch this invocation. {code:java} @Override public URL[] getURLs() { return inner.getURLs(); } {code} Otherwise, {code:java} ClassLoaderUtil.getUserCodeClassLoaderInfo(theJarClassLoader); {code} will print empty, like below: {code:java} URL ClassLoader: {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)