dongjoon-hyun commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1994934200
Thank you for reverting, @yaooqinn and all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993888126
This is reverted this
by[e170252](https://github.com/apache/spark/commit/e170252714e3662c7354321f78a3250114ea7e9e).
After an offline discussion with @cloud-fan, we have reached a
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993350663
Just for FYI, after looking up the pgjdbc type conversion, timestamps both
w/ tz and w/o tz are java.sql.Types.TIMESTAMP. Thus, we have already done such
things for reading TIMESTAMP
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993294887
+1 for reverting this if the doc is incorrect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
cloud-fan commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993290677
> inferring a value which can be either Timestamp or TimestampNTZ
I don't think JDBC TIMESTAMP TZ can be either Timestamp or TimestampNTZ. It
can only be Timestamp.
--
This is
gengliangwang commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993287394
Well, the intention of option `preferTimestampNTZ` is for inferring a value
which can be either Timestamp or TimestampNTZ. We use it in file sources like
JSON/CSV.
I found
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993247657
> When the option is set to true, all timestamps are inferred
as TIMESTAMP WITHOUT TIME ZONE.
Otherwise, timestamps are read as TIMESTAMP with local time zone.
I guess
cloud-fan commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993225902
This looks very confusing to end users, as TIMESTAMP WITHOUT TIMEZONE is
wall clock time but TIMESTAMP TZ is not. What's wrong with reading TIMESTAMP TZ
as TIMESTAMP LTZ?
BTW
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993214195
Many systems supporting TIMESTAMP WITH TIME ZONE will drop the time zone
info and use the JVM timezone. Since we get the value from JDBC API with
getTimestamp, the conversion is valid?
cloud-fan commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1993168674
How can we read TIMESTAMP WITH TIMEZONE as TIMESTAMP WITHOUT TIMEZONE? This
doesn't make sense.
--
This is an automated message from the Apache Git Service.
To respond to the
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1991907943
Thank you @dongjoon-hyun ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
dongjoon-hyun commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1991894812
BTW, congratulations for becoming ASF member, @yaooqinn !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
yaooqinn commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1991887238
Thank you @dongjoon-hyun for creating the umbrella. It helps a lot for
curating recent JDBC data source related change logs.
--
This is an automated message from the Apache Git
dongjoon-hyun commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1991829064
Hi, @yaooqinn .
I created an umbrella JIRA issue, SPARK-47361, to give a more visibility to
the community JDBC improvement of Apache Spark 4.0.0.
After collecting all
dongjoon-hyun commented on PR #45471:
URL: https://github.com/apache/spark/pull/45471#issuecomment-1991796835
Merged to master for Apache Spark 4.0.0.
Thank you, @yaooqinn .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
dongjoon-hyun closed pull request #45471: [SPARK-47342][SQL] Support
TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE
URL: https://github.com/apache/spark/pull/45471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
yaooqinn opened a new pull request, #45471:
URL: https://github.com/apache/spark/pull/45471
### What changes were proposed in this pull request?
This PR Supports TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE when
`preferTimestampNTZ` option is set to true by users
17 matches
Mail list logo
