Josh Wills created CRUNCH-669:
---------------------------------
Summary: Add an option to preserve Crunch temp directories
Key: CRUNCH-669
URL: https://issues.apache.org/jira/browse/CRUNCH-669
Project: Crunch
Issue Type: Improvement
Reporter: Josh Wills
I have a problem where a Crunch client can potentially get killed through no
fault of its own (e.g., an Airflow task failing a heartbeat check), which will
kill the client, but leave the MR job itself still running. The problem is that
the finalize() operation can delete the temp directory that contains the config
info for Crunch, which will make any subsequent tasks fail to run to completion
b/c their config info has been removed, or because some temp output in the
directory that would have been useful can be deleted. I'd like to add an option
to override the automatic delete of the temp directory during the finalize()
operation for these situations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
