[
https://issues.apache.org/jira/browse/FLINK-7643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193175#comment-16193175
]
ASF GitHub Bot commented on FLINK-7643:
---------------------------------------
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4776#discussion_r142989415
--- Diff:
flink-core/src/main/java/org/apache/flink/core/fs/factories/HadoopFileSystemFactoryLoader.java
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.fs.factories;
+
+import org.apache.flink.core.fs.FileSystemFactory;
+
+/**
+ * A
+ */
+public class HadoopFileSystemFactoryLoader {
+
+ private static final String FACTORY_CLASS =
"org.apache.flink.runtime.fs.hdfs.HadoopFsFactory";
+
+ private static final String HADOOP_CONFIG_CLASS =
"org.apache.hadoop.conf.Configuration";
+
+ private static final String HADOOP_FS_CLASS =
"org.apache.hadoop.fs.FileSystem";
+
+
+ public static FileSystemFactory loadFactory() {
+ final ClassLoader cl =
HadoopFileSystemFactoryLoader.class.getClassLoader();
+
+ // first, see if the Flink runtime classes are available
+ final Class<? extends FileSystemFactory> factoryClass;
+ try {
+ factoryClass = Class.forName(FACTORY_CLASS, false,
cl).asSubclass(FileSystemFactory.class);
+ }
+ catch (ClassNotFoundException e) {
+ return new UnsupportedSchemeFactory("Flink runtime
classes missing in classpath/dependencies.");
+ }
+ catch (Exception | LinkageError e) {
--- End diff --
ditto
> Configure FileSystems only once
> -------------------------------
>
> Key: FLINK-7643
> URL: https://issues.apache.org/jira/browse/FLINK-7643
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
> Affects Versions: 1.4.0
> Reporter: Ufuk Celebi
> Assignee: Stephan Ewen
>
> HadoopFileSystem always reloads GlobalConfiguration, which potentially leads
> to a lot of noise in the logs, because this happens on each checkpoint.
> Instead, file systems should be configured once upon process startup, when
> the configuration is loaded.
> This will also increase efficiency of checkpoints, as it avoids redundant
> parsing for each data chunk.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
