Hi Mich,
FYI we're using spark
operator(https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) to build
stateful structured streaming on k8s for a year. Haven't test it using
non-operator way.
Besides that, the main contributor of the spark operator, Yinan Li, has been
inactive for qui
try explain codegen on your DF and then pardee the string
On Fri, 7 Apr, 2023, 3:53 pm Chenghao Lyu, wrote:
> Hi,
>
> The detailed stage page shows the involved WholeStageCodegen Ids in its
> DAG visualization from the Spark UI when running a SparkSQL. (e.g., under
> the link
> node:18088/histor
Because the name of the directory cannot be an object, it has to be a
string to create partitioned dirs like "date=2023-04-10"
On Tue, 11 Apr, 2023, 8:27 pm Charles vinodh, wrote:
>
> Hi Team,
>
> We are running into the below error when we are trying to run a simple
> query a partitioned table
Hi Team,
We are running into the below error when we are trying to run a simple
query a partitioned table in Spark.
*MetaException(message:Filtering is supported only on partition keys
of type string)
*
Our the partition column has been to type *date *instead of string and
query is a very simpl