Dian Fu created FLINK-40194:
-------------------------------
Summary: Add drop_duplicates to DataFrame API
Key: FLINK-40194
URL: https://issues.apache.org/jira/browse/FLINK-40194
Project: Flink
Issue Type: Sub-task
Components: API / Python
Reporter: Dian Fu
Fix For: 2.4.0
def drop_duplicates(
self,
subset: Union[str, List[str]] = None,
*,
keep: str = "first",
order_by: Union[str, Expression, List[Union[str, Expression]]] = None,
nulls_first: Union[bool, List[bool]] = None,
) -> "DataFrame"
Remove duplicate rows. When subset is omitted, fully identical rows are
dropped; when subset is given, rows are deduplicated by those key columns,
keeping one row per key. distinct and unique are provided as aliases of this
method, matching the common naming convention used by other DataFrame libraries.
* subset — a column name or list of column names that define a duplicate. When
omitted (default), all columns are considered (whole-row distinct).
* keep — "first" keeps the earliest row, "last" keeps the latest row within
each duplicate group.
* order_by — a column name or Expression (or a list of them) that defines the
order in which keep selects the surviving row. When omitted, processing time is
used, so keep="first"/"last" keep the first/last row to arrive. Ignored when
subset is omitted.
* nulls_first — a bool, or a list of bool aligned with order_by, controlling
where NULLs rank when keep selects the surviving row. When omitted, Flink's
default null ordering applies. Ignored when order_by is omitted.
Example:
{code:python}
# Whole-row deduplication: drop fully identical rows
df.drop_duplicates()
# Keep the earliest row per device by arrival (defaults to processing time)
df.drop_duplicates(subset="device_id")
# Keep the earliest event per order, ordered by event time
df.drop_duplicates(subset=["order_id"], order_by="event_time")
# Keep the latest record per user, ordered by event time
df.drop_duplicates(subset="user_id", order_by="event_time", keep="last")
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
