ManvithPanyam commented on PR #39851: URL: https://github.com/apache/beam/pull/39851#issuecomment-5452493914
Not really an active Beam DataFrames user day-to-day — found this while working through the dataframe module after my last PR (#39581, the empty-CSV restriction tracker fix). This was the next issue that stood out while digging deeper into the module. Thanks for the catch on `dtype.type()` — confirmed it breaks on Categorical (`TypeError: type.__new__() takes exactly 3 arguments`) and tz-aware datetime (`TypeError: function missing required argument 'year'`). Applied your `reindex()` approach for the DataFrame branch — works cleanly and is simpler than the manual per-column construction. For the Series branch (single-match → scalar case), `reindex().iloc[0]` alone silently changes dtype on plain numeric types (e.g. int64 → float64, since reindex introduces NaN), so I kept `dtype.type()` as the primary path there and only fall back to `reindex().iloc[0]` on the `TypeError` extension-dtype case. Also had to handle proxy indexes with duplicate labels — `reindex()` raises on non-unique index, so I drop duplicates on the proxy copy before reindexing (doesn't affect real execution, this is proxy-only). Added regression tests for Categorical, tz-aware datetime, and nullable Int64 columns. Full suite: 452 passed, 19 skipped, zero regressions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
