That would be close to serialization which you could do with marshmallow (which works better than pickle).
B. Sent from my iPhone > On 25 Apr 2017, at 00:07, Alex Guziel <[email protected]> wrote: > > You can also use reflection in Python to read the modules all the way down. > > On Mon, Apr 24, 2017 at 3:05 PM, Dan Davydov <[email protected] >> wrote: > >> Was talking with Alex about the DB case offline, for those we could support >> a force refresh arg with an interval param. >> >> Manifests would need to be hierarchal but I feel like it would spin out >> into a full blown build system inevitably. >> >> On Mon, Apr 24, 2017 at 3:02 PM, Arthur Wiedmer <[email protected]> >> wrote: >> >>> What if the DAG actually depends on configuration that only exists in a >>> database and is retrieved by the Python code generating the DAG? >>> >>> Just asking because we have this case in production here. It is slowly >>> changing, so still fits within the Airflow framework, but you cannot just >>> watch a file... >>> >>> Best, >>> Arthur >>> >>> On Mon, Apr 24, 2017 at 2:55 PM, Bolke de Bruin <[email protected]> >> wrote: >>> >>>> Inotify can work without a daemon. Just fire a call to the API when a >>> file >>>> changes. Just a few lines in bash. >>>> >>>> If you bundle you dependencies in a zip you should be fine with the >>> above. >>>> Or if we start using manifests that list the files that are needed in a >>>> dag... >>>> >>>> >>>> Sent from my iPhone >>>> >>>>> On 24 Apr 2017, at 22:46, Dan Davydov <[email protected]. >> INVALID> >>>> wrote: >>>>> >>>>> One idea to solve this is to use a daemon that uses inotify to watch >>> for >>>>> changes in files and then reprocesses just those files. The hard part >>> is >>>>> without any kind of dependency/build system for DAGs it can be hard >> to >>>> tell >>>>> which DAGs depend on which files. >>>>> >>>>> On Mon, Apr 24, 2017 at 1:21 PM, Gerard Toonstra < >> [email protected]> >>>>> wrote: >>>>> >>>>>> Hey, >>>>>> >>>>>> I've seen some people complain about DAG file processing times. An >>> issue >>>>>> was raised about this today: >>>>>> >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1139 >>>>>> >>>>>> I attempted to provide a good explanation what's going on. Feel free >>> to >>>>>> validate and comment. >>>>>> >>>>>> >>>>>> I'm noticing that the file processor is a bit naive in the way it >>>>>> reprocesses DAGs. It doesn't look at the DAG interval for example, >> so >>> it >>>>>> looks like it reprocesses all files continuously in one big batch, >>> even >>>> if >>>>>> we can determine that the next "schedule" for all its dags are in >> the >>>>>> future? >>>>>> >>>>>> >>>>>> Wondering if a change in the DagFileProcessingManager could optimize >>>> things >>>>>> a bit here. >>>>>> >>>>>> In the part where it gets the simple_dags from a file it's currently >>>>>> processing: >>>>>> >>>>>> for simple_dag in processor.result: >>>>>> simple_dags.append(simple_dag) >>>>>> >>>>>> the file_path is in the context and the simple_dags should be able >> to >>>>>> provide the next interval date for each dag in the file. >>>>>> >>>>>> The idea is to add files to a sorted deque by >> "next_schedule_datetime" >>>> (the >>>>>> minimum next interval date), so that when we build the list >>>>>> "files_paths_to_queue", it can remove files that have dags that we >>> know >>>>>> won't have a new dagrun for a while. >>>>>> >>>>>> One gotcha to resolve after that is to deal with files getting >> updated >>>> with >>>>>> new dags or changed dag definitions and renames and different >> interval >>>>>> schedules. >>>>>> >>>>>> Worth a PR to glance over? >>>>>> >>>>>> Rgds, >>>>>> >>>>>> Gerard >>>>>> >>>> >>> >>
