paul-rogers commented on a change in pull request #1987: DRILL-7589: Set
temporary tests folder for UDF_DIRECTORY_LOCAL, fix allocators closing in
BloomFilterTest, fix permissions issue for TestGracefulShutdown tests
URL: https://github.com/apache/drill/pull/1987#discussion_r380845219
##########
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/udf/dynamic/TestDynamicUDFSupport.java
##########
@@ -104,8 +104,10 @@ public static void buildAndStoreDefaultJars() throws
IOException {
@Before
public void setupNewDrillbit() throws Exception {
udfDir = dirTestWatcher.makeSubDir(Paths.get("udf"));
+ File udfLocalDir = dirTestWatcher.makeSubDir(Paths.get("udf", "local"));
Properties overrideProps = new Properties();
overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_ROOT,
udfDir.getAbsolutePath());
+ overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_LOCAL,
udfLocalDir.getAbsolutePath());
Review comment:
@vvysotskyi, glad you were able to simplify the test. I realize you didn't
write them, so I appreciate taking the time to clean them up.
The random thought on the directories was that we have 4 or 5 (more) local
directories that we set up. It seems we configure each separately. For example,
from `drill-module.conf`:
```
drill.tmp-dir: "/tmp"
drill.tmp-dir: ${?DRILL_TMP_DIR}
...
sys.store.provider: {
local: {
path: "/tmp/drill",
}
trace: {
directory: "/tmp/drill-trace",
filesystem: "file:///"
},
tmp: {
directories: ["/tmp/drill"],
filesystem: "drill-local:///"
},
compile: {
code_dir: "/tmp/drill/codegen"
...
spill: {
// *** Options common to all the operators that may spill
// File system to use. Local file system by default.
fs: "file:///",
// List of directories to use. Directories are created
// if they do not exist.
directories: [ "/tmp/drill/spill" ]
...
udf: {
directory: {
// Base directory for remote and local udf directories, unique among
clusters.
```
And probably more. To move where Drill stores temp files, the user must
change all of these properties.
Fortunately, @arina-ielchiieva did a nice job with the UDF directories: they
all are computed from the base directory:
```
directory: {
// Base directory for remote and local udf directories, unique among
clusters.
base: ${drill.exec.zk.root}"/udf",
// Path to local udf directory, always created on local file system.
// Root for these directory is generated at runtime unless Drill
temporary directory is set.
local: ${drill.exec.udf.directory.base}"/udf/local",
// Set this property if custom file system should be used to create
remote directories, ex: fs: "file:///".
// fs: "",
// Set this property if custom absolute root should be used for remote
directories, ex: root: "/app/drill".
// root: "",
// Relative path to all remote udf directories.
// Directories are created under default file system taken from Hadoop
configuration
// unless ${drill.exec.udf.directory.fs} is set.
// User home directory is used as root unless
${drill.exec.udf.directory.root} is set.
staging: ${drill.exec.udf.directory.base}"/staging",
registry: ${drill.exec.udf.directory.base}"/registry",
tmp: ${drill.exec.udf.directory.base}"/tmp"
}
```
So, my question was, can we do the same thing for all the other local
directories? Allow each to be custom-set, but default them to be computed from
a single base directory. That way, if a unit test or install wants to move the
Drill local directories to, say, `/var/drill/tmp`, they only need change a
single config line and everything else follows automatically.
This can be done in the existing conf file as was done for UDFs. And, I
guess to preserve compatibility, we'd have to leave the properties where they
are; we'd just change their values.
If you feel this is a bit out of scope for this PR then we can file a Jira
for the change.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
