[jira] [Comment Edited] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761535#comment-16761535 ] Mitesh edited comment on SPARK-19981 at 2/6/19 6:54 AM: Ping any updates here?

[jira] [Commented] (SPARK-19981) Sort-Merge join inserts shuffles when joining dataframes with aliased columns

[ https://issues.apache.org/jira/browse/SPARK-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761535#comment-16761535 ] Mitesh commented on SPARK-19981: Ping any updates here? This still is an issue in 2.3.2. > Sort-Merge

[jira] [Comment Edited] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761289#comment-16761289 ] Bruce Robbins edited comment on SPARK-26708 at 2/6/19 12:41 AM: How does

[jira] [Resolved] (SPARK-26733) Clean up entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-26733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26733. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23655

[jira] [Assigned] (SPARK-26733) Clean up entrypoint.sh

[ https://issues.apache.org/jira/browse/SPARK-26733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26733: -- Assignee: Marcelo Vanzin > Clean up entrypoint.sh > -- > >

[jira] [Commented] (SPARK-17636) Parquet predicate pushdown for nested fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761312#comment-16761312 ] DB Tsai commented on SPARK-17636: - [~MasterDDT] This is different from [SPARK-4502]. One is predicate

[jira] [Commented] (SPARK-17636) Parquet predicate pushdown for nested fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761311#comment-16761311 ] Mitesh commented on SPARK-17636: Should this be closed, as a duplicate of SPARK-4502? > Parquet

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761309#comment-16761309 ] Mitesh commented on SPARK-19468: Also this may be a dupe of SPARK-19981 > Dataset slow because of

[jira] [Commented] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761289#comment-16761289 ] Bruce Robbins commented on SPARK-26708: --- How does one hit this issue? > Incorrect result caused

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761262#comment-16761262 ] Mitesh commented on SPARK-19468: Also curious why in the fix for SPARK-19931, it was only fixed for

[jira] [Comment Edited] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761207#comment-16761207 ] Mitesh edited comment on SPARK-19468 at 2/5/19 8:59 PM: +1 I'm seeing the same

[jira] [Comment Edited] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761207#comment-16761207 ] Mitesh edited comment on SPARK-19468 at 2/5/19 8:59 PM: +1 I'm seeing the same

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761207#comment-16761207 ] Mitesh commented on SPARK-19468: +1 I'm seeing the same behavior. It seems like any physical operator

[jira] [Assigned] (SPARK-26768) Remove useless code in BlockManager

[ https://issues.apache.org/jira/browse/SPARK-26768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26768: - Assignee: liupengcheng > Remove useless code in BlockManager >

[jira] [Resolved] (SPARK-26768) Remove useless code in BlockManager

[ https://issues.apache.org/jira/browse/SPARK-26768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26768. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23693

[jira] [Resolved] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite.fetchBothChunks

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25692. --- Resolution: Fixed Issue resolved by pull request 23700 [https://github.com/apache/spark/pull/23700]

[jira] [Assigned] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite.fetchBothChunks

[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25692: - Assignee: Sanket Reddy > Flaky test: ChunkFetchIntegrationSuite.fetchBothChunks >

[jira] [Comment Edited] (SPARK-26821) filters not working with char datatype when querying against hive table

[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761086#comment-16761086 ] Sujith edited comment on SPARK-26821 at 2/5/19 6:27 PM: Yeah with spaces it will

[jira] [Commented] (SPARK-26821) filters not working with char datatype when querying against hive table

[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761086#comment-16761086 ] Sujith commented on SPARK-26821: Yeah with spaces it will work fine, will try to check the behavior in

[jira] [Created] (SPARK-26829) In place standard scaler so the column remains same after transformation

Santokh Singh created SPARK-26829: - Summary: In place standard scaler so the column remains same after transformation Key: SPARK-26829 URL: https://issues.apache.org/jira/browse/SPARK-26829 Project:

[jira] [Comment Edited] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761072#comment-16761072 ] Dhruve Ashar edited comment on SPARK-26827 at 2/5/19 6:08 PM: -- Resolution :

[jira] [Updated] (SPARK-26828) Coalesce to reduce partitions before writing to hive is not working

[ https://issues.apache.org/jira/browse/SPARK-26828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anusha Buchireddygari updated SPARK-26828: -- Priority: Minor (was: Major) > Coalesce to reduce partitions before writing

[jira] [Resolved] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhruve Ashar resolved SPARK-26827. -- Resolution: Workaround Pass the same archive with py-files and archives option. > Support

[jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761069#comment-16761069 ] Dhruve Ashar commented on SPARK-26827: -- Thanks for the response [~irashid] and [~hyukjin.kwon].

[jira] [Created] (SPARK-26828) Coalesce to reduce partitions before writing to hive is not working

Anusha Buchireddygari created SPARK-26828: - Summary: Coalesce to reduce partitions before writing to hive is not working Key: SPARK-26828 URL: https://issues.apache.org/jira/browse/SPARK-26828

[jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761050#comment-16761050 ] Hyukjin Kwon commented on SPARK-26827: -- The workaround sounds pretty okay as is. > Support

[jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761034#comment-16761034 ] Dhruve Ashar commented on SPARK-26827: -- [~holden.ka...@gmail.com] , [~irashid] any thoughts on this

[jira] [Commented] (SPARK-26827) Support importing python modules having shared objects(.so)

[ https://issues.apache.org/jira/browse/SPARK-26827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761039#comment-16761039 ] Imran Rashid commented on SPARK-26827: -- I don't know enough about python use cases to have an

[jira] [Created] (SPARK-26827) Support importing python modules having shared objects(.so)

Dhruve Ashar created SPARK-26827: Summary: Support importing python modules having shared objects(.so) Key: SPARK-26827 URL: https://issues.apache.org/jira/browse/SPARK-26827 Project: Spark

[jira] [Updated] (SPARK-26826) Array indexing functions array_allpositions and array_select

[ https://issues.apache.org/jira/browse/SPARK-26826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-26826: -- Description: This ticket proposes two extra array functions: {{array_allpositions}} (named

[jira] [Assigned] (SPARK-26826) Array indexing functions array_allpositions and array_select

[ https://issues.apache.org/jira/browse/SPARK-26826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26826: Assignee: Apache Spark > Array indexing functions array_allpositions and array_select >

[jira] [Assigned] (SPARK-26826) Array indexing functions array_allpositions and array_select

[ https://issues.apache.org/jira/browse/SPARK-26826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26826: Assignee: (was: Apache Spark) > Array indexing functions array_allpositions and

[jira] [Created] (SPARK-26826) Array indexing functions array_allpositions and array_select

Petar Zecevic created SPARK-26826: - Summary: Array indexing functions array_allpositions and array_select Key: SPARK-26826 URL: https://issues.apache.org/jira/browse/SPARK-26826 Project: Spark

[jira] [Commented] (SPARK-26825) Spark Structure Streaming job failing when submitted in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-26825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760901#comment-16760901 ] Andre Araujo commented on SPARK-26825: -- Thanks a lot, [~gsomogyi] > Spark Structure Streaming job

[jira] [Commented] (SPARK-26821) filters not working with char datatype when querying against hive table

[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760897#comment-16760897 ] Sean Owen commented on SPARK-26821: --- Can you find the result when filtering for "ds " with spaces?

[jira] [Comment Edited] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : wo

[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760799#comment-16760799 ] M. Le Bihan edited comment on SPARK-26819 at 2/5/19 1:30 PM: - You made me

[jira] [Commented] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : working

[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760799#comment-16760799 ] M. Le Bihan commented on SPARK-26819: - You made me search a lot...   In this stacktrace part that 

[jira] [Commented] (SPARK-24284) java.util.NoSuchElementException in Spark Streaming with Kafka

[ https://issues.apache.org/jira/browse/SPARK-24284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760769#comment-16760769 ] Gabor Somogyi commented on SPARK-24284: --- [~ujjalsatpa...@gmail.com] On 1.6.3 CachedKafkaConsumer

[jira] [Commented] (SPARK-26825) Spark Structure Streaming job failing when submitted in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-26825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760777#comment-16760777 ] Gabor Somogyi commented on SPARK-26825: --- There is another PR from me which modifies this part but

[jira] [Commented] (SPARK-24284) java.util.NoSuchElementException in Spark Streaming with Kafka

[ https://issues.apache.org/jira/browse/SPARK-24284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760775#comment-16760775 ] Gabor Somogyi commented on SPARK-24284: --- This code part has been rewritten in 2.4. Is it possible

[jira] [Commented] (SPARK-26825) Spark Structure Streaming job failing when submitted in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-26825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760702#comment-16760702 ] Gabor Somogyi commented on SPARK-26825: --- [~asdaraujo] excellent analysis! One minor correction: *

[jira] [Commented] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : working

[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760670#comment-16760670 ] Hyukjin Kwon commented on SPARK-26819: -- Is it basically a dependency management issue in your side?

[jira] [Updated] (SPARK-26823) SBT Build Warnings

[ https://issues.apache.org/jira/browse/SPARK-26823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26823: - Target Version/s: (was: 2.4.0) > SBT Build Warnings > --- > >

[jira] [Comment Edited] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760604#comment-16760604 ] Laszlo Rigo edited comment on SPARK-26727 at 2/5/19 10:05 AM: -- For me these

[jira] [Comment Edited] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755846#comment-16755846 ] Laszlo Rigo edited comment on SPARK-26727 at 2/5/19 9:29 AM: - [~dongjoon],

[jira] [Comment Edited] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755846#comment-16755846 ] Laszlo Rigo edited comment on SPARK-26727 at 2/5/19 9:28 AM: - [~dongjoon],

[jira] [Commented] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760604#comment-16760604 ] Laszlo Rigo commented on SPARK-26727: - For me these calls still seem to be asynchronous: {noformat}