bumping this up again for suggestions?.. Is the official recommendation to not have *int* or *date* typed partition columns?
On Wed, 12 Apr 2023 at 10:44, Charles vinodh <mig.flan...@gmail.com> wrote: > There are other distributed execution engines (like hive, trino) that do > support non-string data types for partition columns such as date and > integer. > Any idea why this restriction exists in Spark? .. > > > On Tue, 11 Apr 2023 at 20:34, Chitral Verma <chitralve...@gmail.com> > wrote: > >> Because the name of the directory cannot be an object, it has to be a >> string to create partitioned dirs like "date=2023-04-10" >> >> On Tue, 11 Apr, 2023, 8:27 pm Charles vinodh, <mig.flan...@gmail.com> >> wrote: >> >>> >>> Hi Team, >>> >>> We are running into the below error when we are trying to run a simple >>> query a partitioned table in Spark. >>> >>> *MetaException(message:Filtering is supported only on partition keys of >>> type string) >>> * >>> >>> >>> Our the partition column has been to type *date *instead of string and >>> query is a very simple SQL as shown below. >>> >>> *SELECT * FROM my_table WHERE partition_col = date '2023-04-11'* >>> >>> Any idea why spark mandates partition columns to be of type string?. Is >>> there a recommended work around for this issue? >>> >>> >>>
