guan404ming commented on code in PR #54195:
URL: https://github.com/apache/airflow/pull/54195#discussion_r2280881246
##########
providers/amazon/src/airflow/providers/amazon/aws/transfers/sql_to_s3.py:
##########
@@ -220,17 +244,23 @@ def execute(self, context: Context) -> None:
file_obj=buf, key=object_key, bucket_name=self.s3_bucket,
replace=self.replace
)
- def _partition_dataframe(self, df: pd.DataFrame) -> Iterable[tuple[str,
pd.DataFrame]]:
+ def _partition_dataframe(self, df: pd.DataFrame | pl.DataFrame) ->
Iterable[tuple[str, pd.DataFrame]]:
"""Partition dataframe using pandas groupby() method."""
try:
import secrets
import string
import numpy as np
+ import polars as pl
except ImportError:
pass
+
+ if isinstance(df, pl.DataFrame):
+ df = df.to_pandas()
Review Comment:
You're right, `to_pandas()` still need pandas installed. That means it need
re-implementation for this function for supporting two libs differently and I
would open a PR for it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
