DataFrameReader: timestampFormat default value

2024-04-24 Thread keen
Is anyone familiar with [Datetime patterns]( https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html) and `TimestampType` parsing in PySpark? When reading CSV or JSON files, timestamp columns need to be parsed. via datasource property `timestampFormat`. [According to documentation](

Re: [Feature Request] create *permanent* Spark View from DataFrame via PySpark

2023-06-04 Thread keen
Do Spark **devs** read this mailing list? Is there another/a better way to make feature requests? I tried in the past to write a mail to the dev mailing list but it did not show at all. Cheers keen schrieb am Do., 1. Juni 2023, 07:11: > Hi all, > currently only *temporary* Spark Vie

[Feature Request] create *permanent* Spark View from DataFrame via PySpark

2023-06-01 Thread keen
Hi all, currently only *temporary* Spark Views can be created from a DataFrame (df.createOrReplaceTempView or df.createOrReplaceGlobalTempView). When I want a *permanent* Spark View I need to specify it via Spark SQL (CREATE VIEW AS SELECT ...). Sometimes it is easier to specify the desired

Slack for PySpark users

2023-03-27 Thread keen
Hi all, I really like *Slack *as communication channel for a tech community. There is a Slack workspace for *delta lake users* (https://go.delta.io/slack) that I enjoy a lot. I was wondering if there is something similar for PySpark users. If not, would there be anything wrong with creating a new