SameerMesiah97 opened a new pull request, #72081: URL: https://github.com/apache/airflow/pull/72081
**Description** This change introduces a new operator, `AthenaSparkOperator`, to submit Apache Spark calculations to an existing Amazon Athena Spark session. The operator delegates to new `AthenaHook` methods for starting, polling, inspecting, and stopping Athena Spark calculations through the Athena calculation APIs. It waits until the calculation reaches a terminal state and returns calculation metadata, including the calculation execution ID, state, session ID, working directory, and result S3 URIs. **Rationale** Amazon Athena supports Apache Spark through session-based calculation APIs, but the Amazon provider does not currently expose an Airflow operator for submitting Spark code to an existing Athena Spark session. Adding `AthenaSparkOperator` provides DAG authors with a provider-native way to run Athena Spark calculations from Airflow without writing custom `PythonOperator` code around boto3. The implementation reuses the existing `AthenaHook` because Spark calculations use the same Athena client and AWS connection configuration as existing Athena query operations. **Tests** Added unit tests verifying that: * `AthenaHook` correctly starts, inspects, checks, and stops Athena Spark calculations. * `AthenaHook` handles cached calculation execution responses. * `AthenaHook` returns `None` for malformed or missing calculation status fields. * `AthenaSparkOperator` submits Spark code using the hook and polls until completion. * `AthenaSparkOperator` raises on failed, canceled, timed-out, or malformed calculation states. * `AthenaSparkOperator` attempts to stop the calculation when the task is killed. * Template fields are correctly configured. **Example DAG** Added a new example DAG, `example_athena_spark`, demonstrating how to submit Spark code to an existing Athena Spark session using `AthenaSparkOperator`. **Documentation** Added documentation for `AthenaSparkOperator`, including prerequisites, supported parameters, guidance on using an existing Athena Spark session, and a usage example. **Backwards Compatibility** This is a new operator and only adds new Athena hook methods. Existing Athena query functionality is not modified. No breaking changes are introduced. **Acknowledgements** This change builds on the Athena Spark provider support previously explored in PR #66576 by @Andisha2004 which proposed adding an Athena Spark hook, sensor, and operator to the Amazon provider. This PR carries that direction forward with a narrower initial scope focused on `AthenaSparkOperator` and the supporting `AthenaHook` calculation methods, along with updated tests, documentation, and an example DAG aligned with current provider conventions. ###### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: [GPT 5.5] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
