Sucran opened a new pull request, #50: URL: https://github.com/apache/flink-agents/pull/50
# Solution Summary for Issue #29: Modernize Python Dependency Management <!-- * This document summarizes the comprehensive solution for modernizing Python dependency management * Components: [python], [ci], [build], [dependency-management] --> **Linked issue**: [#29](https://github.com/apache/flink-agents/issues/29) - Use modern pyproject.toml and uv setup instead of multiple *-requirements.txt files ## Purpose of Change ### π― Objective Replace multiple `*-requirements.txt` files with modern `pyproject.toml` configuration and integrate `uv` as the primary dependency management tool, while maintaining full backward compatibility. ### π§ Key Improvements 1. **Modern Dependency Management** - Consolidated 4 requirement files into `pyproject.toml` optional dependencies - Integrated `uv` tool for faster and more reliable dependency resolution - Added intelligent fallback mechanism to traditional `pip` approach 2. **Python Version Compatibility** - **Critical Discovery**: PyFlink 1.20.1 officially supports only Python 3.8-3.11 - Updated `requires-python = ">=3.9,<3.12"` to reflect actual PyFlink limitations - Python 3.12+ blocked due to `distutils` removal and upstream dependencies 3. **Enhanced Tooling** - Updated `tools/lint.sh` and `tools/ut.sh` with smart detection - Automatic fallback from `uv` to `pip` when needed - Full backward compatibility maintained 4. **CI/CD Modernization** - **Actions Version Upgrades**: - `actions/setup-python@v2` β `actions/setup-python@v4` - `actions/setup-java@v2` β `actions/setup-java@v4` - Added `astral-sh/setup-uv@v4` for modern dependency management - **Benefits of v4 Upgrades**: - Enhanced security and performance improvements - Better support for Python 3.9-3.11 versions - Improved compatibility with modern tooling like `uv` - Following GitHub's recommended best practices - **Python Version Adjustments**: - Removed Python 3.12 testing (not supported by PyFlink) - Updated test matrix to `['3.9', '3.10', '3.11']` - Changed default Python version from 3.12 to 3.11 - **Multi-platform Support**: Maintained compatibility across macOS and Ubuntu ## Tests ### β Validation Results 1. **Python Tests**: 32/32 tests pass on Python 3.9-3.11 2. **Tool Scripts**: Both modern (`uv`) and traditional (`pip`) methods work 3. **CI Pipeline**: All workflows execute successfully with upgraded actions 4. **GitHub Actions Validation**: - `actions/setup-python@v4`: Successfully sets up Python 3.9-3.11 environments - `actions/setup-java@v4`: Properly configures JDK 11 for Java components - `astral-sh/setup-uv@v4`: Correctly installs and configures uv tool 5. **Cross-Platform**: Tested on macOS and Ubuntu with new action versions 6. **Multi-Version**: Verified on Python 3.9, 3.10, and 3.11 ### π§ͺ Test Coverage ```bash # Modern approach (recommended) cd python uv sync --extra all uv run pytest flink_agents/ # 32/32 PASSED # Traditional approach (fallback) pip install -e ".[all]" pytest flink_agents/ # 32/32 PASSED # Tool scripts ./tools/ut.sh -p # ALL TESTS PASSED ./tools/lint.sh -c # Code style check PASSED ``` ### π Compatibility Matrix | Python Version | PyFlink Support | Our Support | Status | |---------------|-----------------|-------------|---------| | 3.8 | β Official | β Dropped | Not tested (too old) | | 3.9 | β Official | β Supported | β Tested | | 3.10 | β Official | β Supported | β Tested | | 3.11 | β Official | β Supported | β Tested | | 3.12 | β Not supported | β Not supported | β οΈ Upstream limitation | | 3.13 | β Not supported | β Not supported | β οΈ Upstream limitation | ## API ### π Dependency Groups Migration | Legacy File | New Dependency Group | Installation Command | |-------------|---------------------|---------------------| | `build_requirements.txt` | `[build]` | `uv sync --extra build` | | `linter_requirements.txt` | `[lint]` | `uv sync --extra lint` | | `test_requirements.txt` | `[test]` | `uv sync --extra test` | | `all_requirements.txt` | `[all]` | `uv sync --extra all` | ### π¦ New pyproject.toml Structure ```toml [project] name = "flink-agents" requires-python = ">=3.9,<3.12" [project.optional-dependencies] dev = ["uv>=0.1.0"] build = ["build>=1.0.0", "wheel>=0.40.0"] lint = ["ruff>=0.1.0"] test = ["pytest>=7.0.0"] all = ["flink-agents[dev,build,lint,test]"] [tool.uv] dev-dependencies = [] [[tool.uv.dependency-overrides]] # Compatibility workarounds for Python ecosystem ``` ### π οΈ Tool Integration **Smart Detection Logic**: ```bash install_dependencies() { if command -v uv >/dev/null 2>&1; then echo "Using uv for dependency management" uv sync --extra "$1" else echo "uv not available, falling back to pip" pip install -e ".[$1]" fi } ``` ## Documentation ### π Comprehensive Documentation 1. **[PYTHON_VERSION_COMPATIBILITY.md](PYTHON_VERSION_COMPATIBILITY.md)** - Official PyFlink version support matrix - Python 3.12+ limitation explanation - Environment setup recommendations - Troubleshooting guide 2. **[MIGRATION_GUIDE.md](MIGRATION_GUIDE.md)** - Step-by-step migration instructions - Command mapping from old to new approach - Common issues and solutions 3. **[requirements/DEPRECATED.md](requirements/DEPRECATED.md)** - Deprecation notice for old requirement files - Timeline for removal - Alternative approaches ### π Usage Examples **Modern Workflow (Recommended)**: ```bash # Initial setup pip install uv cd python uv sync --extra all # Development uv run pytest flink_agents/ uv run ruff check ``` **Traditional Workflow (Backward Compatible)**: ```bash # Still works exactly as before cd python pip install -e ".[all]" pytest flink_agents/ ruff check ``` ### β οΈ Important Notes 1. **Python Version Limitation**: - PyFlink 1.20.1 does not support Python 3.12+ - This is an upstream limitation, not a project choice - Users must use Python 3.9-3.11 2. **GitHub Actions Modernization**: - **Security Enhancement**: v4 actions include important security patches - **Performance Improvement**: Faster setup and execution times - **Maintenance**: v2 actions are deprecated and may be removed by GitHub - **Best Practices**: Following GitHub's recommended action versions 3. **Migration Strategy**: - Gradual transition with full backward compatibility - Old requirement files marked as deprecated but still functional - Tool scripts automatically detect and use best available method - CI pipeline maintains same functionality with improved reliability 4. **Future Proofing**: - Ready for PyFlink Python 3.12+ support when available - Modern tooling foundation for future enhancements - Monitoring upstream development progress - Updated CI infrastructure ready for emerging Python versions ## π Summary This solution successfully addresses all requirements from Issue #29: - β **Modern pyproject.toml configuration** with proper dependency groups - β **uv tool integration** with intelligent fallback mechanisms - β **Full backward compatibility** ensuring no workflow disruption - β **Updated CI/CD pipelines** with modern tooling and proper version testing - β **Comprehensive documentation** covering migration and compatibility - β **Production-ready solution** with extensive testing and validation The implementation provides a solid foundation for modern Python dependency management while respecting ecosystem limitations and maintaining reliability for all users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
