This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-3-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 63ee7a624ac1a0b7f23a4f6a9e741943ff6b0ab3 Author: Ephraim Anierobi <splendidzig...@gmail.com> AuthorDate: Thu Apr 14 19:37:04 2022 +0100 Add 2.3.0b1 changelog to CHANGELOG.txt --- CHANGELOG.txt | 331 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 331 insertions(+) diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 5289344fad..a7532ad720 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,3 +1,334 @@ +Airflow 2.3.0b1, 2022-04-14 +--------------------------- + +New Features +"""""""""""" +- Add dynamic task mapping (`AIP-42 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42>`_) +- New Grid View replaces Tree View (#18675) +- Move the database configuration to a new section (#22284) +- Add SmoothOperator (#22813) +- make operator's execution_timeout configurable (#22389) +- Events Timetable (#22332) +- Support dag serialization with custom ti_deps rules (#22698) +- Support log download in task log view (#22804) +- Add summary of runs to dag details (#22766) +- support for continue backfill on failures (#22697) +- Add dag-processor cli command (#22305) +- Add possibility to create users in LDAP mode (#22619) +- Add ignore_first_depends_on_past for scheduled jobs (#22491) +- Update base sensor operator to support XCOM return value (#20656) +- add an option for run id in the ui trigger screen (#21851) +- Enable JSON serialization for connections (#19857) +- Add REST API endpoint for bulk update of DAGs (#19758) +- Add queue button to click-on-DagRun interface. (#21555) +- Add ``list-import-errors`` to ``airflow dags`` command (#22084) +- Store callbacks in database if standalone_dag_processor config is True. (#21731) +- Local kubernetes executor (#19729) +- add celery.task_timeout_error metric (#21602) +- Airflow ``db downgrade`` cli command (#21596) +- Add ALL_SKIPPED trigger rule (#21662) +- Add ``db clean`` CLI command for purging old data (#20838) +- Add celery_logging_level (#21506) +- Support different timeout value for dag file parsing (#21501) +- Support generating SQL script for upgrades (#20962) +- Add option to compress Serialized dag data (#21332) +- Branch python operator decorator (#20860) +- Add Audit Log View to Dag View (#20733) +- Add missing statsd metric for failing SLA Callback notification (#20924) +- Add ShortCircuitOperator configurability for respecting downstream trigger rules (#20044) +- Allow using Markup in page title in Webserver (#20888) +- Add Listener Plugin API that tracks TaskInstance state changes (#20443) +- #16037 Templated requirements.txt in Python operators (#17349) +- Add context var hook to inject more env vars (#20361) +- Add a button to set all tasks to skipped (#20455) +- Cleanup pending pods (#20438) +- Add config to warn public deployment exposure in UI (#18557) +- Log filename template records (#20165) +- Added windows extensions (#16110) +- Showing approximate time until next dag_run in Airflow (#20273) +- Extend config window on UI (#20052) +- Add show dag dependencies feature to CLI (#19985) +- Add cli command for ``airflow dags reserialize`` (#19471) +- Add missing description field to Pool schema(REST API) (#19841) +- Introduce DagRun action to change state to queued. (#19353) +- Add DAG run details page (#19705) +- Add role export/import to cli tools (#18916) +- Adding dag_id_pattern parameter to the /dags endpoint (#18924) + +Improvements +"""""""""""" +- Update ImportError items instead of deleting and recreating them (#22928) +- Add option ``--skip-init`` to db reset command (#22989) +- Support importing connections from files with ".yml" extension (#22872) +- Support glob syntax in ``.airflowignore`` files (#21392) (#22051) +- Hide pagination when data is a single page (#22963) +- Support for sorting DAGs in the web UI (#22671) +- Speed up ``has_access`` decorator by ~200ms (#22858) +- Add XComArg to lazy-imported list of Airflow module (#22862) +- Add more fields to REST API dags/dag_id/details endpoint (#22756) +- Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812) +- No need to load whole ti in current_state (#22764) +- Pickle dag exception string fix (#22760) +- Better verification of Localexecutor's parallelism option (#22711) +- log backfill exceptions to sentry (#22704) +- retry commit on MySQL deadlocks during backfill (#22696) +- Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637) +- Use timetable to generate planned days for current year (#22055) +- Disable connection pool for celery worker (#22493) +- Make date picker label visible in trigger dag view (#22379) +- Expose try_number in airflow vars (#22297) +- Add generic connection type (#22310) +- add a few more fields to the taskinstance finished log message (#22262) +- Pause autorefresh if scheduler isn't running (#22151) +- Show DagModel details. (#21868) +- Add pip_install_options to PythonVirtualenvOperator (#22158) +- Show import error for ``airflow dags list`` CLI command (#21991) +- Pause auto-refresh when page is hidden (#21904) +- Default args type check (#21809) +- Enhance magic methods on XComArg for UX (#21882) +- py files doesn't have to be checked is_zipfiles in refresh_dag (#21926) +- Fix TaskDecorator type hints (#21881) +- Add 'Show record' option for variables (#21342) +- Use DB where possible for quicker ``airflow dag`` subcommands (#21793) +- REST API: add rendered fields in task instance. (#21741) +- Change the default auth backend to session (#21640) +- Refactor TriggerRule & WeightRule classes to inherit from Enum (#21264) +- Don't check if `py` DAG files are zipped during parsing (#21538) +- Switch XCom implementation to use run_id (#20975) +- Action log on Browse Views (#21569) +- Implement multiple API auth backends (#21472) +- Change logging level details of connection info in `get_connection()` (#21162) +- Support mssql in airflow db shell (#21511) +- Support config worker_enable_remote_control for celery (#21507) +- Log memory usage in CgroupTaskRunner (#21481) +- Modernize DAG-related URL routes and rename "tree" to "grid" (#20730) +- Move Zombie detection to SchedulerJob (#21181) +- Improve speed to run ``airflow`` by 6x (#21438) +- Add more SQL template fields renderers (#21237) +- Simplify fab has access lookup (#19294) +- Log context only for default method (#21244) +- Log trigger status only if at least one is running (#21191) +- Add optional features in providers. (#21074) +- Better multiple_outputs inferral for @task.python (#20800) +- Improve handling of string type and non-attribute ``template_fields`` (#21054) +- Remove un-needed deps/version requirements (#20979) +- Correctly specify overloads for TaskFlow API for type-hinting (#20933) +- Introduce notification_sent to SlaMiss view (#20923) +- Rewrite the task decorator as a composition (#20868) +- Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798) +- Rewrite DAG run retrieval in task command (#20737) +- Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722) +- Make native environment Airflow-flavoured like sandbox (#20704) +- Better error when param value has unexpected type (#20648) +- Add filter by state in DagRun REST API (List Dag Runs) (#20485) +- Prevent exponential memory growth in Tasks with custom logging handler (#20541) +- Set default logger in logging Mixin (#20355) +- Reduce deprecation warnings from www (#20378) +- Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002) +- Add specific warning when Task asks for more slots than pool defined with (#20178) +- UI: Update duration column for better human readability (#20112) +- Use Viewer role as example public role (#19215) +- Properly implement DAG param dict copying (#20216) +- #16692 show schedule_interval/timetable description in UI (#16931) +- ShortCircuitOperator push XCom by returnung python_callable result (#20071) +- Add clear logging to tasks killed due to a Dagrun timeout (#19950) +- Change log level for Zombie detection messages (#20204) +- Better confirmation prompts (#20183) +- Only execute TIs of running DagRuns (#20182) +- Check and run migration in commands if necessary (#18439) +- Log only when Zombies exists (#20118) +- Inrease length of the email and username (#19932) +- Add more filtering options for TI's in the UI (#19910) +- Dynamically enable "Test Connection" button by connection type (#19792) +- Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842) +- [19458] Added column duration to DAG runs view (#19482) +- Renamed Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849) +- Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431) +- Speed up webserver boot time by delaying provider initialization (#19709) +- Configurable logging of XCOM value in PythonOperator (#19378) +- Minimze production js files (#19658) +- Add hook_params in BaseSqlOperator (#18718) +- Add missing ``end_date`` to hash components (#19281) +- More friendly output of the airflow plugins command + add timetables (#19298) +- Add sensor default timeout config (#19119) +- Update taskinstance REST API schema to include dag_run_id field (#19105) +- Adding feature in bash operator to append the user defined env variable to system env variable (#18944) +- Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161) + +Bug Fixes +""""""""" +- Change trigger dropdown left position (#23013) +- Don't add planned tasks for legacy DAG runs (#23007) +- Add dangling rows check for TaskInstance references (#22924) +- formatting fix (#22688) +- Fix trigger event payload is not persisted in db (#22944) +- Drop "airflow moved" tables in command ``db reset`` (#22990) +- Add max width to task group tooltips (#22978) +- Add template support for external_task_ids. (#22809) +- Allow DagParam to hold falsy values (#22964) +- Fix regression in pool metrics (#22939) +- Priority order tasks even when using pools (#22483) +- Do not clear XCom when resuming from deferral (#22932) +- Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898) +- Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850) +- Fixed backfill interference with scheduler (#22701) +- Support conf param override for backfill runs (#22837) +- Correctly interpolate pool name in PoolSlotsAvailableDep statues (#22807) +- Fix ``email_on_failure`` with ``render_template_as_native_obj`` (#22770) +- Fix processor cleanup on DagFileProcessorManager (#22685) +- Prevent meta name clash for task instances (#22783) +- remove json parse for gantt chart (#22780) +- Check for missing dagrun should know version (#22752) +- Fixes ScheduleInterval spec (#22635) +- Fixing task status for non-running and non-committed tasks (#22410) +- Do not log the hook connection details even at DEBUG level (#22627) +- Stop crashing when empty logs are received from kubernetes client (#22566) +- Fix bugs about timezone change (#22525) +- Fix entire DAG stops when one task has end_date (#20920) +- Use logger to print message during task execution. (#22488) +- Make sure finalizers are not skipped during exception handling (#22475) +- update smart sensor docs and minor fix on is_smart_sensor_compatible() (#22386) +- Fix ``run_id`` k8s and elasticsearch compatibility with Airflow 2.1 (#22385) +- Allow to except_skip None on BranchPythonOperator (#20411) +- Fix incorrect datetime details (DagRun views) (#21357) +- Remove incorrect deprecation warning in secrets backend (#22326) +- Remove RefreshConfiguration workaround for K8s token refreshing (#20759) +- Masking extras in GET /connections/<connection> endpoint (#22227) +- Set queued_dttm when submitting task to directly to executor (#22259) +- Addressed some issues in the tutorial mentioned in discussion #22233 (#22236) +- Change default python executable to python3 for docker decorator (#21973) +- Don't validate that Params are JSON when NOTSET (#22000) +- Add per-DAG delete permissions (#21938) +- Fix handling some None parameters in kubernetes 23 libs. (#21905) +- EdgeModifier refactoring (#21404) +- Fix handling of empty (None) tags in ``bulk_write_to_db`` (#21757) +- Fix DAG date range bug (#20507) +- Removed ``request.referrer`` from views.py (#21751) +- Make DbApiHook use get_uri from Connection (#21764) +- Fix some migrations (#21670) +- [de]serialize resources on task correctly (#21445) +- Add params dag_id, task_id etc to XCom.serialize_value (#19505) +- Update test connection functionality to use custom form fields (#21330) +- fix all "high" npm vulnerabilities (#21526) +- Fix bug incorrectly removing action from role, rather than permission. (#21483) +- Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296) +- Use Identity instead of Seqence in SQLAlchemy 1.4 for MSSQL (#21238) +- Ensure ``on_task_instance_running`` listener can get at task (#21157) +- Return to the same place when triggering a DAG (#20955) +- Fix task ID deduplication in @task_group (#20870) +- Add downgrade to some FAB migrations (#20874) +- Only validate Params when DAG is triggered (#20802) +- Fix airflow trigger cli (#20781) +- Fix task instances iteration in a pool to prevent blocking (#20816) +- Allow depending to a @task_group as a whole (#20671) +- Use original task's ``start_date`` if a task continues after deferral (#20062) +- Disabled edit button in task instances list view page (#20659) +- Fix a package name import error (#20519) (#20519) +- Remove ``execution_date`` label when get cleanup pods list (#20417) +- Remove unneeded FAB REST API endpoints (#20487) +- Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700) +- Sanity check for MySQL's TIMESTAMP column (#19821) +- Allow using default celery command group with executors subclassed from Celery-based executors. (#18189) +- Move class_permission_name to mixin so it applies to all auths (#18749) +- Adjust trimmed_pod_id and replace '.' with '-' (#19036) +- Pass custom_headers to send_email and send_email_smtp (#19009) +- Ensure ``catchup=False`` is used in example dags (#19396) +- Edit permalinks in OpenApi description file (#19244) +- Navigate directly to DAG when selecting from search typeahead list (#18991) +- [Minor] Fix padding on home page (#19025) + +Doc only changes +"""""""""""""""" +- Fix wrong reference in tracking-user-activity.rst (#22745) +- Remove references to ``rbac = True`` from docs (#22725) +- Doc: Update description for executor-bound dependencies (#22601) +- Update check-health.rst (#22372) +- Stronger language about Docker Compose customizability (#22304) +- Update logging-tasks.rst (#22116) +- Add example config of sql_alchemy_connect_args (#22045) +- Update best-practices.rst (#22053) +- Add information on DAG pausing/deactivation/deletion (#22025) +- Add brief examples of integration test dags you might want (#22009) +- Run inclusive language check on CHANGELOG (#21980) +- Add detailed email docs for Sendgrid (#21958) +- Add docs for ``db upgrade`` / ``db downgrade`` (#21879) +- Update modules_management.rst (#21889) +- Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887) +- Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446) +- Fix doc - replace decreasing by increasing (#21805) +- Add another way to dynamically generate DAGs to docs (#21297) +- Add extra information about time synchronization needed (#21685) +- Update debug.rst docs (#21246) +- Replaces the usage of postgres:// with postgresql:// (#21205) +- Fix task execution process in ``CeleryExecutor`` docs (#20783) + +Misc/Internal +""""""""""""" +- Replace usage of ``DummyOperator`` with ``EmptyOperator`` (#22974) +- Deprecate ``DummyOperator`` in favor of ``EmptyOperator`` (#22832) +- Remove unnecessary python 3.6 conditionals (#20549) +- Bump moment from 2.29.1 to 2.29.2 in /airflow/www (#22873) +- Bump prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823) +- Bump nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803) +- Bump minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798) +- Remove dag parsing from db init command (#22531) +- removed unnecessary dependence on bcrypt (#22498) +- Update our approach for executor-bound dependencies (#22573) +- Use Airflow.Base.metadata in FAB models (#22353) +- Update black precommit (#22521) +- New design of system tests (#22311) +- Limit Docutils to make our documentation pretty again (#22420) +- Add Python 3.10 support (#22050) +- [FEATURE] add 1.22 1.23 K8S support (#21902) +- Remove pandas upper limit now that SQLA is 1.4+ (#22162) +- Patch sql_alchemy_conn if old postgres scheme used (#22333) +- Protect against accidental misuse of XCom.get_value() (#22244) +- Remove Snowflake limits (#22181) +- Order filenames for migrations (#22168) +- Don't try to create automigration for celery tables (#22120) +- Require SQLAlchemy 1.4 (#22114) +- bump moto version (#22099) +- bump sphinx-jinja (#22101) +- Upgrade ``moto`` library to version 3.0 (#22005) +- Simplify tests for CLI connections commands (#21983) +- Add test to run DB downgrade in the CI (#21273) +- Add compat shim for SQLAlchemy to avoid warnings (#21959) +- Rename ``xcom.dagrun_id`` to ``xcom.dag_run_id`` (#21806) +- Deprecate non-JSON conn.extra (#21816) +- Autogenerate migration reference doc (#21601) +- Limits GitHub3.py in order to avoid backtracking (#21824) +- Bump upper bound version of jsonschema to 5.0 (#21712) +- Deprecate helper utility ``days_ago`` (#21653) +- Remove ``:type`` lines now sphinx-autoapi supports typehints (#20951) +- Quelch deprecation warning in tests (#20900) +- Use ``DagRun.run_id`` instead of ``execution_date`` when updating state of TIs(UI & REST API) (#18724) +- Add Context stub to Airflow packages (#20817) +- Update Kubernetes library version (#18797) +- Rename ``PodLauncher`` to ``PodManager`` (#20576) +- Removes Python 3.6 support (#20467) +- Add deprecation warnining for non-json-serializable params (#20174) +- Rename TaskMixin to DependencyMixin (#20297) +- Deprecate passing execution_date to XCom methods (#19825) +- Remove get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961) +- Remove postgres 9.6 support (#19987) +- Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639) +- add kubernetes 1.21 support (#19557) +- Add FAB base class and set import_name explicitly. (#19667) +- Remove remaining ``pylint: disable`` comments (#19541) +- Removes unused state transitions to handle auto-changing view permissions. (#19153) +- Chore: Use enum for "__var" and "__type" members (#19303) +- Use fab models (#19121) +- Consolidate method names between Airflow Security Manager and FAB default (#18726) +- Fixup string concatenations (#19099) +- Remove distutils usages for Python 3.10 (#19064) +- Removing redundant max_tis_per_query initialisation on SchedulerJob (#19020) +- More f-strings (#18855) +- Add pre-commit hook for common misspelling check in files (#18964) +- Remove deprecated usage of init_role() from API (#18820) +- Ignore License check in ``airflow/www/static/robots.txt`` (#18886) +- Remove duplicate code on dbapi hook (#18821) + + Airflow 2.2.5, 2022-04-04 -------------------------