[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593200#comment-16593200
]
Chenxiao Mao commented on SPARK-25175:
--
Also here is the similar investigation I did for parquet
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chenxiao Mao reopened SPARK-25175:
--
> Case-insensitive field resolution when reading from ORC
>
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593194#comment-16593194
]
Chenxiao Mao commented on SPARK-25175:
--
[~dongjoon] [~yucai] Here is a brief summary. We can see
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593185#comment-16593185
]
Chenxiao Mao commented on SPARK-25175:
--
Thorough investigation about ORC tables
{code}
val data =
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593176#comment-16593176
]
Apache Spark commented on SPARK-25248:
--
User 'mengxr' has created a pull request for this issue:
Xiangrui Meng created SPARK-25248:
-
Summary: Audit barrier APIs for Spark 2.4
Key: SPARK-25248
URL: https://issues.apache.org/jira/browse/SPARK-25248
Project: Spark
Issue Type: Story
Xiangrui Meng created SPARK-25247:
-
Summary: Make RDDBarrier configurable
Key: SPARK-25247
URL: https://issues.apache.org/jira/browse/SPARK-25247
Project: Spark
Issue Type: Story
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593156#comment-16593156
]
Dongjoon Hyun commented on SPARK-25175:
---
Thanks, [~yucai]. I'm highly interested in this case.
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-25175.
---
Resolution: Cannot Reproduce
I followed the same direction given by SPARK-25132, but I
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593152#comment-16593152
]
yucai commented on SPARK-25175:
---
I pinged [~seancxmao] offline, he will give more details.
>
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Summary: wrong records are returned when Hive metastore schema and parquet
schema are in different letter
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Summary: data issue when Hive metastore schema and parquet schema are in
different letter cases (was: data
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Summary: data issue when Hive metastore schema and parquet schema have
different letter case (was: data
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Summary: data issue when (was: data issue because wrong column is
pushdown for parquet)
> data issue when
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593126#comment-16593126
]
yucai edited comment on SPARK-25206 at 8/27/18 2:27 AM:
[~dongjoon], because of
[
https://issues.apache.org/jira/browse/SPARK-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593128#comment-16593128
]
Dongjoon Hyun commented on SPARK-25207:
---
[~yucai]. My bad. Please ignore that. It was based on the
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593126#comment-16593126
]
yucai commented on SPARK-25206:
---
[~dongjoon], because of the below root cause
{quote}Spark pushdowns
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Description:
In current Spark 2.3.1, below query returns wrong data silently.
{code:java}
[
https://issues.apache.org/jira/browse/SPARK-25221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saisai Shao updated SPARK-25221:
Target Version/s: (was: 2.3.2, 2.4.0)
> [DEPLOY] Consistent trailing whitespace treatment of
[
https://issues.apache.org/jira/browse/SPARK-25221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593124#comment-16593124
]
Saisai Shao commented on SPARK-25221:
-
I'm going to remove the target version, I don't think it is a
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yucai updated SPARK-25206:
--
Summary: data issue because wrong column is pushdown for parquet (was:
Wrong data may be returned for
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593117#comment-16593117
]
Hyukjin Kwon commented on SPARK-25206:
--
[~yucai], mind fixing the JIRA title?
> Wrong data may be
[
https://issues.apache.org/jira/browse/SPARK-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593110#comment-16593110
]
yucai commented on SPARK-25207:
---
[~dongjoon] , sorry if I am confusing you.
This bug is created for
[
https://issues.apache.org/jira/browse/SPARK-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593108#comment-16593108
]
Dongjoon Hyun commented on SPARK-25207:
---
According to the PR, this seems to be a new regression
[
https://issues.apache.org/jira/browse/SPARK-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25207:
--
Attachment: image.png
> Case-insensitve field resolution for filter pushdown when reading
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593102#comment-16593102
]
yucai commented on SPARK-25206:
---
I am OK with "known correctness bug in 2.3" way, just raise some concern
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593100#comment-16593100
]
yucai commented on SPARK-25206:
---
[~smilegator] , sure, I will add tests.
If we don't backport
[
https://issues.apache.org/jira/browse/SPARK-25236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593098#comment-16593098
]
holdenk commented on SPARK-25236:
-
Probably. The only thing would be probably wanting to pass log level
[
https://issues.apache.org/jira/browse/SPARK-25236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593097#comment-16593097
]
Liang-Chi Hsieh commented on SPARK-25236:
-
hmm, maybe dumb question, can't we use {{logging}} to
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593096#comment-16593096
]
Wenchen Fan commented on SPARK-25206:
-
I'm fine to mark it as a known correctness bug in Spark 2.2,
[
https://issues.apache.org/jira/browse/SPARK-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593085#comment-16593085
]
Michail Giannakopoulos edited comment on SPARK-24826 at 8/27/18 12:53 AM:
[
https://issues.apache.org/jira/browse/SPARK-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593085#comment-16593085
]
Michail Giannakopoulos commented on SPARK-24826:
[~dongjoon] I will and let you know...
[
https://issues.apache.org/jira/browse/SPARK-19355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593080#comment-16593080
]
Apache Spark commented on SPARK-19355:
--
User 'viirya' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25207:
--
Description:
Currently, filter pushdown will not work if Parquet schema and Hive metastore
[
https://issues.apache.org/jira/browse/SPARK-24766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-24766:
--
Labels: Parquet (was: )
> CreateHiveTableAsSelect and InsertIntoHiveDir won't generate
[
https://issues.apache.org/jira/browse/SPARK-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593067#comment-16593067
]
Dongjoon Hyun commented on SPARK-24826:
---
Hi, [~miccagiann]. Could you try that in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-25132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25132:
--
Labels: Parquet (was: )
> Case-insensitive field resolution when reading from Parquet
>
[
https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25135:
--
Labels: Parquet correctness (was: correctness)
> insert datasource table may all null when
[
https://issues.apache.org/jira/browse/SPARK-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25207:
--
Labels: Parquet (was: )
> Case-insensitve field resolution for filter pushdown when reading
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25206:
--
Labels: Parquet correctness (was: correctness)
> Wrong data may be returned for Parquet
>
[
https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593066#comment-16593066
]
Dongjoon Hyun commented on SPARK-25135:
---
[~yumwang]. Could you update your PR according to this
[
https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25135:
--
Description:
This happens on parquet.
How to reproduce in parquet.
{code:scala}
val path =
[
https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25135:
--
Description:
How to reproduce:
{code:scala}
val path = "/tmp/spark/parquet"
val cnt = 30
[
https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25135:
--
Summary: insert datasource table may all null when select from view on
parquet (was: insert
[
https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593061#comment-16593061
]
Dongjoon Hyun commented on SPARK-25091:
---
Hi, [~Chao Fang]. Could you remove `Spark Thrift Server:
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593041#comment-16593041
]
Xiao Li edited comment on SPARK-25206 at 8/26/18 10:45 PM:
---
Currently, we do
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593041#comment-16593041
]
Xiao Li commented on SPARK-25206:
-
Previously, we do not have a good test coverage when the physical
[
https://issues.apache.org/jira/browse/SPARK-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593040#comment-16593040
]
shahid commented on SPARK-25246:
I am working on it :)
> When the spark.eventLog.compress is enabled,
shahid created SPARK-25246:
--
Summary: When the spark.eventLog.compress is enabled, the
Application is not showing in the History server UI ('incomplete application'
page), initially.
Key: SPARK-25246
URL:
[
https://issues.apache.org/jira/browse/SPARK-25245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-25245:
Assignee: Apache Spark
> Explain regarding limiting modification on
[
https://issues.apache.org/jira/browse/SPARK-25245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593034#comment-16593034
]
Apache Spark commented on SPARK-25245:
--
User 'HeartSaVioR' has created a pull request for this
[
https://issues.apache.org/jira/browse/SPARK-25245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-25245:
Assignee: (was: Apache Spark)
> Explain regarding limiting modification on
Jungtaek Lim created SPARK-25245:
Summary: Explain regarding limiting modification on
"spark.sql.shuffle.partitions" for structured streaming
Key: SPARK-25245
URL:
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593024#comment-16593024
]
Dongjoon Hyun commented on SPARK-25206:
---
Hi, [~yucai], [~cloud_fan], [~smilegator],
[
https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593013#comment-16593013
]
Dongjoon Hyun commented on SPARK-25175:
---
[~seancxmao]. If there is no example, we can not help
[
https://issues.apache.org/jira/browse/SPARK-25244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Daitche updated SPARK-25244:
--
Description:
The setting `spark.sql.session.timeZone` is respected by PySpark when
Anton Daitche created SPARK-25244:
-
Summary: [Python] Setting `spark.sql.session.timeZone` only
partially respected
Key: SPARK-25244
URL: https://issues.apache.org/jira/browse/SPARK-25244
Project:
[
https://issues.apache.org/jira/browse/SPARK-25244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Daitche updated SPARK-25244:
--
Description:
The setting `spark.sql.session.timeZone` is respected by PySpark when
[
https://issues.apache.org/jira/browse/SPARK-25243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-25243:
Assignee: Apache Spark
> Use FailureSafeParser in from_json
>
[
https://issues.apache.org/jira/browse/SPARK-25243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592940#comment-16592940
]
Apache Spark commented on SPARK-25243:
--
User 'MaxGekk' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-25243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-25243:
Assignee: (was: Apache Spark)
> Use FailureSafeParser in from_json
>
Maxim Gekk created SPARK-25243:
--
Summary: Use FailureSafeParser in from_json
Key: SPARK-25243
URL: https://issues.apache.org/jira/browse/SPARK-25243
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-23707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-23707.
--
Resolution: Cannot Reproduce
> Don't need shuffle exchange with single partition for
[
https://issues.apache.org/jira/browse/SPARK-25013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-25013.
--
Resolution: Won't Fix
I wouldn't add this into Spark for now unless there's strong request
[
https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-10697:
Assignee: Apache Spark
> Lift Calculation in Association Rule mining
>
[
https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592899#comment-16592899
]
Apache Spark commented on SPARK-10697:
--
User 'mgaido91' has created a pull request for this issue:
[
https://issues.apache.org/jira/browse/SPARK-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-10697:
Assignee: (was: Apache Spark)
> Lift Calculation in Association Rule mining
>
[
https://issues.apache.org/jira/browse/SPARK-23792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-23792.
---
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 20901
[
https://issues.apache.org/jira/browse/SPARK-23792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen reassigned SPARK-23792:
-
Assignee: A Bradbury
> Documentation improvements for datetime functions
>
[
https://issues.apache.org/jira/browse/SPARK-25080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-25080.
--
Resolution: Cannot Reproduce
> NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110)
>
[
https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592893#comment-16592893
]
Hyukjin Kwon commented on SPARK-25206:
--
Please fix the JIRA title to reflect more precisely rather
[
https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592890#comment-16592890
]
Yuming Wang commented on SPARK-25135:
-
Another serious case:
{code:scala}
withTempDir { dir =>
[
https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592848#comment-16592848
]
Apache Spark commented on SPARK-23698:
--
User 'HyukjinKwon' has created a pull request for this
80 matches
Mail list logo
