[
https://issues.apache.org/jira/browse/FALCON-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688823#comment-15688823
]
Srikanth Sundarrajan commented on FALCON-1406:
----------------------------------------------
[~ajayyadava], This is a much cleaner way of retaining history than the current
scheme. What happens in the current scheme is that users simply delete the old
entity, create new one with the same name with a different time range. This
change would allow us to track history of changes for more cleanly. Also when
you go back and run an older instance (applicable on old data), corresponding
version of the app is used to process the data. Cleanup of entities isn't
really possible through a tool because no one is going to declare (there isn't
a mechanism either) to say this is a new processing entity created for fixing
issues (backfills, code bug or any others) with another process. Someone has
to really sit down and cleanup manually. This feature should pave the way for
us to have a single process entity corresponding to a processing need and allow
for natural evolution with system tracking and maintaining history of those
changes. The status quo is quite broken IMHO.
>> a lot of features developed in falcon took the immutability of entity
>> definition for past instances as given
I think this assumption isn't valid. Past instances are mutable. People do
rerun older instances, even worse they create newer process entities to re-run
old instances.
> Effective time in Entity updates.
> ---------------------------------
>
> Key: FALCON-1406
> URL: https://issues.apache.org/jira/browse/FALCON-1406
> Project: Falcon
> Issue Type: New Feature
> Reporter: sandeep samudrala
> Assignee: sandeep samudrala
> Attachments: FALCON-1406-initial.patch,
> effective_time_in_entity_updates.pdf
>
>
> Effective time with entity updates needs to be provided even with past time
> too. There was effective time capability provided in the past which gives the
> functionality to set an effective time for an entity with only current or
> future time(now + delay), which could not solve all the issues.
> Following are few scenarios which would require effective time to be
> available with time back in past.
> a) New code being deployed for an incompatible input data set which would
> leave instances with old code and new data.
> b) Bad code being pushed for which, the entity should be able to go back in
> time to replay(rerun) with new code.
> c) Orchestration level changes(good/bad) would need functionality to go back
> in time to start with.
> For reference: Linking all the Jiras that have been worked upon around
> effective time .
> https://issues.apache.org/jira/browse/FALCON-374
> https://issues.apache.org/jira/browse/FALCON-297
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
