Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
timsaucer commented on issue #862: URL: https://github.com/apache/datafusion-python/issues/862#issuecomment-4215961689 I believe this is now resolved as I no longer get the error on the most recent release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
timsaucer closed issue #862: Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data URL: https://github.com/apache/datafusion-python/issues/862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
rickspencer3 commented on issue #862:
URL:
https://github.com/apache/datafusion-python/issues/862#issuecomment-2336472790
> Oh, I just realized that might not work since `functions.when` was
recently exposed. You would need to do something like `F.case(col("count") !=
lit(0)).when(lit(True), col("count")).end()` instead
Thank you so much for the help. Much appreciated. Please let me know if
there is anything that I can do to help. It sounds like, if all else fails,
there is a workaround coming with ```pa.table(df)```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
timsaucer commented on issue #862:
URL:
https://github.com/apache/datafusion-python/issues/862#issuecomment-2336458296
Oh, I just realized that might not work since `functions.when` was recently
exposed. You would need to do something like `F.case(col("count") !=
lit(0)).when(lit(True), col("count")).end()` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
timsaucer commented on issue #862:
URL:
https://github.com/apache/datafusion-python/issues/862#issuecomment-2336452591
It's super hacky:
```
df = df.select("time_window", F.when(col("count") != lit(0),
col("count")).end().alias("count"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
timsaucer commented on issue #862: URL: https://github.com/apache/datafusion-python/issues/862#issuecomment-2336437952 Also, when I coerced the `count` field to be nullable it was able to work, so I've got a strong suspicion about that mixture problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Can't convert date_bin aggregated with count(*) to arrow if some windows contain null data [datafusion-python]
rickspencer3 commented on issue #862: URL: https://github.com/apache/datafusion-python/issues/862#issuecomment-2336431893 When looking at the dataframe under the debugger, it looks like it skipped the null values ok: ``` df DataFrame() +-+---+ | time_window | count | +-+---+ | 2024-09-07T10:01:00 | 2 | | 2024-09-07T10:03:00 | 2 | +-+---+ df.schema() time_window: timestamp[ns] count: int64 not null ``` So I am pretty confused about the error message. ``` return self.df.to_arrow_table() File "pyarrow/table.pxi", line 4865, in pyarrow.lib.Table.from_batches File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Schema at index 0 was different: time_window: timestamp[ns] count: int64 not null vs time_window: timestamp[ns] count: int64 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
