kevin-lann opened a new pull request, #35920: URL: https://github.com/apache/superset/pull/35920
<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly --> ### SUMMARY Upgrades marshmallow version to >= 4.0 - Flask-AppBuilder 5.0.0 auto-generates schema fields for SQLAlchemy relationships that don't exist in marshmallow 4.x's stricter field validation, causing `KeyError` during schema initialization. - Our solution patches marshmallow's `Schema._init_fields` method to detect missing fields that Flask-AppBuilder 5.0.0 auto-generates, and then adds these missing fields dynamically as raw fields with appropriate flags & retry initialization until all missing fields are resolved. - So, for example, it can handle multiple fields in a sequence _(permission_id, view_menu_id, db_id, chart_id, dashboard_id, user_id)_. All of this logic for this bullet point is encapsulated in the `marshmallow_fix.py` module, whose job is to specifically address the Flask-AppBuilder 5.0.0 incompatibility with marshmallow 4.x, and then `superset_config.py` applies the patch by importing it from `marshmallow_fix.py`. - All changes in files other than `superset_config.py` and `marshmallow_fix.py` are minor syntax changes that are required for the Marshmallow upgrade to >= 4.0.0 (See https://marshmallow.readthedocs.io/en/stable/upgrading.html for more info) ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF BEFORE: Upon pinning `marshmallow>=4` in `pyproject.toml`and initializing the superset app, an initialization error occurs.  AFTER: App runs without error <img width="1182" height="773" alt="Screenshot 2025-10-30 at 9 39 23 PM" src="https://github.com/user-attachments/assets/42a8dc37-33e2-49cd-87dd-653b877e2e84"; /> ### TESTING INSTRUCTIONS Start superset application, see that no marshmallow related errors show up. Run tests with `docker compose run --rm superset python -m pytest tests/unit_tests/test_marshmallow_compatibility.py -v` **Unit tests created:** 1. _test_patch_marshmallow_for_flask_appbuilder_applies_patch_ - Verifies the patch function correctly modifies the Schema class 2. _test_patch_functionality_with_real_schema_creation_ - Tests actual schema creation works with the patch 3. _test_patch_handles_schema_with_no_fields_ - Ensures empty schemas work properly 4. _test_raw_field_creation_and_configuration_ - Validates Raw field configuration 5. _test_print_function_can_be_mocked_ - Tests logging functionality can be tested 6. _test_keyerror_exception_handling_ - Verifies KeyError exception handling 7. _test_schema_declared_fields_manipulation_ - Tests field manipulation capabilities 8. _test_flask_appbuilder_field_names_list_ - Validates the field names our fix handles 9. _test_patch_function_is_callable_ - Ensures the patch function works correctly 10. _test_marshmallow_schema_basic_functionality_ - Confirms basic marshmallow functionality still works ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [x] Has associated issue: Fixes issue #33162 - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
