[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Internal Jenkins has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 10 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. IMPALA-3983/IMPALA-3974: Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Reviewed-on: http://gerrit.cloudera.org:8080/4617 Reviewed-by: Bharath Vissapragada Tested-by: Internal Jenkins --- M be/src/catalog/catalog.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/custom_cluster/test_permanent_udfs.py 6 files changed, 42 insertions(+), 11 deletions(-) Approvals: Bharath Vissapragada: Looks good to me, approved Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 10: Code-Review+2 Giving GVO another shot. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 10 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Internal Jenkins has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 9: Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/396/ -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Internal Jenkins has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 9: Verified-1 Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/395/ -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 9: Code-Review+2 Carry +2, rebased. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 8: The previous build was manually aborted because the master branch was locked. Can I run the GVO now? -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Lars Volker has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 8: Is this ready to be merged? Do we know why the previous build had failed? -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Lars Volker Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Internal Jenkins has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 8: Verified-1 Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/372/ -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 8: Code-Review+2 Made a minor change to the test to remove --local_library_dir override in impalad_args. The reason the test failed was because the lib-cache doesn't deterministically remove a lib-cache-entry by the end of test. Given this change only relates to the Catalog side changes, I've retained --catalogd_args. Carrying +2 and kicking off GVO again. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Hello Henry Robinson, Internal Jenkins, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4617 to look at the new patch set (#8). Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. IMPALA-3983/IMPALA-3974: Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 --- M be/src/catalog/catalog.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/custom_cluster/test_permanent_udfs.py 6 files changed, 42 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/4617/8 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Internal Jenkins has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 7: Verified-1 Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/364/ -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Internal Jenkins Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 7: Code-Review+2 Rebased. Carrying +2. Thanks Alex and Henry. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Henry Robinson has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/4617/5/tests/custom_cluster/test_permanent_udfs.py File tests/custom_cluster/test_permanent_udfs.py: PS5, Line 39: LOCAL_LIBRARY_DIR = mkdtemp(dir="/tmp") > use mkdtemp? Done PS5, Line 237: assert len(glob.glob(self.LOCAL_LIBRARY_DIR + "/*.jar")) == 0 > comment what this is checking Done -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Hello Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4617 to look at the new patch set (#6). Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. IMPALA-3983/IMPALA-3974: Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 --- M be/src/catalog/catalog.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/custom_cluster/test_permanent_udfs.py 6 files changed, 43 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/4617/6 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Henry Robinson has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 5: You're right, of course, I was confused. Ignore my previous comment. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 5: bq: I'm a bit worried about the compatibility of this change: if someone has set --local_library_dir to some local staging dir, will they be surprised when that dir is emptied after this patch? I wonder if we need to hide this behaviour behind a flag. Not sure I follow you. We only delete the temporary jar that our code adds to that directory (CatalogServiceCatalog.java). I'm guessing that your observation is based on the test case I added (please correct me if I'm wrong). That behavior is specific to the test runs where that directory is a random one created on the fly and won't have any other jars added to it and the assert holds. In practice that directory can have any other stuff too, that we don't touch. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Henry Robinson has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 5: (2 comments) I'm a bit worried about the compatibility of this change: if someone has set --local_library_dir to some local staging dir, will they be surprised when that dir is emptied after this patch? I wonder if we need to hide this behaviour behind a flag. http://gerrit.cloudera.org:8080/#/c/4617/5/tests/custom_cluster/test_permanent_udfs.py File tests/custom_cluster/test_permanent_udfs.py: PS5, Line 39: LOCAL_LIBRARY_DIR = "/tmp/" + get_random_id(5) use mkdtemp? PS5, Line 237: assert len(glob.glob(self.LOCAL_LIBRARY_DIR + "/*.jar")) == 0 comment what this is checking -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 5: Henry, do you have any further comments on this one? Thanks. -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Hello Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4617 to look at the new patch set (#5). Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. IMPALA-3983/IMPALA-3974: Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 --- M be/src/catalog/catalog.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/custom_cluster/test_permanent_udfs.py 6 files changed, 43 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/4617/5 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 3: (1 comment) Added a test. http://gerrit.cloudera.org:8080/#/c/4617/2/be/src/catalog/catalog.cc File be/src/catalog/catalog.cc: PS2, Line 48: String > long line Done -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Hello Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4617 to look at the new patch set (#4). Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. IMPALA-3983/IMPALA-3974: Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 --- M be/src/catalog/catalog.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M tests/custom_cluster/test_permanent_udfs.py 6 files changed, 42 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/4617/4 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Henry Robinson has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 2: (1 comment) Any easy way to test this? Can you add some UDFs and then check the local directory for any leftover jar files? http://gerrit.cloudera.org:8080/#/c/4617/2/be/src/catalog/catalog.cc File be/src/catalog/catalog.cc: PS2, Line 48: String long line -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Henry Robinson Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Bharath Vissapragada has posted comments on this change. Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. Patch Set 2: (1 comment) Thanks Alex. Carrying +1. http://gerrit.cloudera.org:8080/#/c/4617/2//COMMIT_MSG Commit Message: Line 7: IMPALA-3983: Delete function jar resources after load > add other JIRA Done -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3983/IMPALA-3974: Delete function jar resources after load
Hello Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4617 to look at the new patch set (#3). Change subject: IMPALA-3983/IMPALA-3974: Delete function jar resources after load .. IMPALA-3983/IMPALA-3974: Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 --- M be/src/catalog/catalog.cc M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java 5 files changed, 32 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/4617/3 -- To view, visit http://gerrit.cloudera.org:8080/4617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada
