[jira] [Comment Edited] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

2019-03-03 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783074#comment-16783074 ] Gabor Somogyi edited comment on SPARK-26727 at 3/4/19 7:55 AM: --- [~Udbhav

[jira] [Commented] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

2019-03-03 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783074#comment-16783074 ] Gabor Somogyi commented on SPARK-26727: --- I was dealing with "failOnDataLoss=false should not

[jira] [Assigned] (SPARK-27038) Rack resolving takes a long time when initializing TaskSetManager

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27038: Assignee: Apache Spark > Rack resolving takes a long time when initializing

[jira] [Assigned] (SPARK-27038) Rack resolving takes a long time when initializing TaskSetManager

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27038: Assignee: (was: Apache Spark) > Rack resolving takes a long time when initializing

[jira] [Issue Comment Deleted] (SPARK-26961) Found Java-level deadlock in Spark Driver

2019-03-03 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S updated SPARK-26961: Comment: was deleted (was: The problem is here org.apache.spark.util.MutableURLClassLoader (entire

[jira] [Comment Edited] (SPARK-26961) Found Java-level deadlock in Spark Driver

2019-03-03 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783031#comment-16783031 ] Ajith S edited comment on SPARK-26961 at 3/4/19 6:53 AM: - The problem is here

[jira] [Comment Edited] (SPARK-26961) Found Java-level deadlock in Spark Driver

2019-03-03 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783031#comment-16783031 ] Ajith S edited comment on SPARK-26961 at 3/4/19 6:54 AM: - The problem is here

[jira] [Comment Edited] (SPARK-26961) Found Java-level deadlock in Spark Driver

2019-03-03 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783031#comment-16783031 ] Ajith S edited comment on SPARK-26961 at 3/4/19 6:52 AM: - The problem is here

[jira] [Comment Edited] (SPARK-26961) Found Java-level deadlock in Spark Driver

2019-03-03 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783031#comment-16783031 ] Ajith S edited comment on SPARK-26961 at 3/4/19 6:51 AM: - The problem is here

[jira] [Commented] (SPARK-26961) Found Java-level deadlock in Spark Driver

2019-03-03 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783031#comment-16783031 ] Ajith S commented on SPARK-26961: - The problem is here org.apache.spark.util.MutableURLClassLoader

[jira] [Commented] (SPARK-27020) Unable to insert data with partial dynamic partition with Spark & Hive 3

2019-03-03 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783002#comment-16783002 ] sandeep katta commented on SPARK-27020: --- Hi Can you please provide some details  1.what is the

[jira] [Resolved] (SPARK-26956) remove streaming output mode from data source v2 APIs

2019-03-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-26956. - Resolution: Fixed Fix Version/s: 3.0.0 > remove streaming output mode from data source v2 APIs >

[jira] [Issue Comment Deleted] (SPARK-27020) Unable to insert data with partial dynamic partition with Spark & Hive 3

2019-03-03 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-27020: -- Comment: was deleted (was: I would like to take up this jira, will be working on this jira)

[jira] [Updated] (SPARK-27015) spark-submit does not properly escape arguments sent to Mesos dispatcher

2019-03-03 Thread Martin Loncaric (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Loncaric updated SPARK-27015: Affects Version/s: (was: 2.5.0) (was: 3.0.0)

[jira] [Updated] (SPARK-27015) spark-submit does not properly escape arguments sent to Mesos dispatcher

2019-03-03 Thread Martin Loncaric (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Loncaric updated SPARK-27015: Fix Version/s: 3.0.0 2.5.0 > spark-submit does not properly escape

[jira] [Updated] (SPARK-27015) spark-submit does not properly escape arguments sent to Mesos dispatcher

2019-03-03 Thread Martin Loncaric (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Loncaric updated SPARK-27015: Affects Version/s: (was: 2.3.3) (was: 2.4.0)

[jira] [Assigned] (SPARK-26893) Allow partition pruning with subquery filters on file source

2019-03-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26893: --- Assignee: Peter Toth > Allow partition pruning with subquery filters on file source >

[jira] [Commented] (SPARK-27020) Unable to insert data with partial dynamic partition with Spark & Hive 3

2019-03-03 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782980#comment-16782980 ] sandeep katta commented on SPARK-27020: --- I would like to take up this jira, will be working on

[jira] [Resolved] (SPARK-26893) Allow partition pruning with subquery filters on file source

2019-03-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26893. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23802

[jira] [Commented] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

2019-03-03 Thread Udbhav Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782975#comment-16782975 ] Udbhav Agrawal commented on SPARK-26727: [~gsomogyi] can you tell me the details of the test  

[jira] [Commented] (SPARK-26964) to_json/from_json do not match JSON spec due to not supporting scalars

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782961#comment-16782961 ] Hyukjin Kwon commented on SPARK-26964: -- I resolved it as Later mainly due to no feedback. I think

[jira] [Comment Edited] (SPARK-26964) to_json/from_json do not match JSON spec due to not supporting scalars

2019-03-03 Thread Huon Wilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782958#comment-16782958 ] Huon Wilson edited comment on SPARK-26964 at 3/4/19 4:45 AM: - I see. Could

[jira] [Commented] (SPARK-26964) to_json/from_json do not match JSON spec due to not supporting scalars

2019-03-03 Thread Huon Wilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782958#comment-16782958 ] Huon Wilson commented on SPARK-26964: - I see. Could you say why you're resolving it as Later? I'm

[jira] [Resolved] (SPARK-27032) Flaky test: org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.HDFSMetadataLog: metadata directory collision

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27032. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23937

[jira] [Updated] (SPARK-27027) from_avro function does not deserialize the Avro record of a struct column type correctly

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27027: - Description: {{from_avro}} function produces wrong output of a struct field.  See the output

[jira] [Commented] (SPARK-27030) DataFrameWriter.insertInto fails when writing in parallel to a hive table

2019-03-03 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782943#comment-16782943 ] Shivu Sondur commented on SPARK-27030: -- I am checking this issue > DataFrameWriter.insertInto

[jira] [Created] (SPARK-27038) Rack resolving takes a long time when initializing TaskSetManager

2019-03-03 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-27038: -- Summary: Rack resolving takes a long time when initializing TaskSetManager Key: SPARK-27038 URL: https://issues.apache.org/jira/browse/SPARK-27038 Project: Spark

[jira] [Commented] (SPARK-27028) PySpark read .dat file. Multiline issue

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782941#comment-16782941 ] Hyukjin Kwon commented on SPARK-27028: -- If newline is included in the data, it should be quoted to

[jira] [Resolved] (SPARK-27037) Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27037. -- Resolution: Not A Problem > Pyspark Row .asDict() cannot handle MapType with a Struct as the

[jira] [Commented] (SPARK-27037) Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782940#comment-16782940 ] Hyukjin Kwon commented on SPARK-27037: -- Use {{asDict(recursive=True)}} {code} >>>

[jira] [Resolved] (SPARK-26964) to_json/from_json do not match JSON spec due to not supporting scalars

2019-03-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26964. -- Resolution: Later Let me leave this resolved as Later for now. > to_json/from_json do not

[jira] [Resolved] (SPARK-27001) Refactor "serializerFor" method between ScalaReflection and JavaTypeInference

2019-03-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27001. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23908

[jira] [Assigned] (SPARK-27001) Refactor "serializerFor" method between ScalaReflection and JavaTypeInference

2019-03-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27001: --- Assignee: Jungtaek Lim > Refactor "serializerFor" method between ScalaReflection and

[jira] [Updated] (SPARK-27037) Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value

2019-03-03 Thread Tanjin Panna (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanjin Panna updated SPARK-27037: - Description: When we have a tuple as the key or value in a {{MapType}} and call the

[jira] [Updated] (SPARK-27037) Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value

2019-03-03 Thread Tanjin Panna (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanjin Panna updated SPARK-27037: - Description: When we have a tuple as the key or value in a {{MapType}} and call the

[jira] [Created] (SPARK-27037) Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value

2019-03-03 Thread Tanjin Panna (JIRA)
Tanjin Panna created SPARK-27037: Summary: Pyspark Row .asDict() cannot handle MapType with a Struct as the key or value Key: SPARK-27037 URL: https://issues.apache.org/jira/browse/SPARK-27037

[jira] [Assigned] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala:

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25863: Assignee: (was: Apache Spark) > java.lang.UnsupportedOperationException: empty.max

[jira] [Commented] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala

2019-03-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782903#comment-16782903 ] Takeshi Yamamuro commented on SPARK-25863: -- Yea, returning 0 sounds reasonable to me, too. >

[jira] [Commented] (SPARK-21871) Check actual bytecode size when compiling generated code

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782904#comment-16782904 ] Apache Spark commented on SPARK-21871: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-21871) Check actual bytecode size when compiling generated code

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782905#comment-16782905 ] Apache Spark commented on SPARK-21871: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala:

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25863: Assignee: Apache Spark > java.lang.UnsupportedOperationException: empty.max at >

[jira] [Commented] (SPARK-26918) All .md should have ASF license header

2019-03-03 Thread Mani M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782900#comment-16782900 ] Mani M commented on SPARK-26918: Hi [~felixcheung] Just to check how to remove rat filter for .md

[jira] [Commented] (SPARK-26247) SPIP - ML Model Extension for no-Spark MLLib Online Serving

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782876#comment-16782876 ] Sean Owen commented on SPARK-26247: --- There are two issues here -- load time of the model, and scoring

[jira] [Commented] (SPARK-26918) All .md should have ASF license header

2019-03-03 Thread Mani M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782874#comment-16782874 ] Mani M commented on SPARK-26918: Ok will add and raise the PR > All .md should have ASF license header

[jira] [Commented] (SPARK-26247) SPIP - ML Model Extension for no-Spark MLLib Online Serving

2019-03-03 Thread Anne Holler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782863#comment-16782863 ] Anne Holler commented on SPARK-26247: - Hi, [~skonto] and [~srowen], Thank you for your comments! 

[jira] [Commented] (SPARK-25130) [Python] Wrong timestamp returned by toPandas

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782846#comment-16782846 ] Sean Owen commented on SPARK-25130: --- [~maxgekk] is this likely fixed by your overhaul of time parsing?

[jira] [Resolved] (SPARK-25201) Synchronization performed on AtomicReference in LevelDB class

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25201. --- Resolution: Invalid I don't see a problem statement here > Synchronization performed on

[jira] [Commented] (SPARK-25350) Spark Serving

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782844#comment-16782844 ] Sean Owen commented on SPARK-25350: --- I think this kind of thing is great, but belongs outside the

[jira] [Resolved] (SPARK-25405) Saving RDD with new Hadoop API file as a Sequence File too restrictive

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25405. --- Resolution: Not A Problem Looks like you are using the old Mapreduce OutputFormat classes with the

[jira] [Resolved] (SPARK-25441) calculate term frequency in CountVectorizer()

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25441. --- Resolution: Won't Fix What you have there is already term frequency. If you want to normalize it to

[jira] [Resolved] (SPARK-25552) Upgrade from Spark 1.6.3 to 2.3.0 seems to make jobs use about 50% more memory

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25552. --- Resolution: Invalid This is too broad. Literally 1 things change from 1.6 to 2.3. You'd have to

[jira] [Updated] (SPARK-25466) Documentation does not specify how to set Kafka consumer cache capacity for SS

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25466: -- Labels: (was: doc ss) Priority: Minor (was: Major) Component/s: Documentation

[jira] [Commented] (SPARK-25544) Slow/failed convergence in Spark ML models due to internal predictor scaling

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782841#comment-16782841 ] Sean Owen commented on SPARK-25544: --- I think this is a reasonable change -- you can test it in a PR if

[jira] [Commented] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-03 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782840#comment-16782840 ] Sujith commented on SPARK-27036: It seems to be the problem area is   BroadcastExchangeExec  in driver

[jira] [Resolved] (SPARK-25550) [Spark Job History] Environment Page of Spark Job History UI showing wrong value for spark.ui.retainedJobs

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25550. --- Resolution: Won't Fix > [Spark Job History] Environment Page of Spark Job History UI showing wrong

[jira] [Resolved] (SPARK-25733) The method toLocalIterator() with dataframe doesn't work

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25733. --- Resolution: Cannot Reproduce I can't reproduce this; with a simple local test (and Spark unit

[jira] [Resolved] (SPARK-25562) The Spark add audit log

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25562. --- Resolution: Invalid > The Spark add audit log > --- > > Key:

[jira] [Resolved] (SPARK-25633) Performance Improvement for Drools Spark Jobs.

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25633. --- Resolution: Invalid I can't make out a specific issue here. JIRA isn't for tech support questions;

[jira] [Updated] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-03 Thread Babulal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated SPARK-27036: Attachment: image-2019-03-04-00-39-38-779.png > Even Broadcast thread is timed out, BroadCast Job is not

[jira] [Commented] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

2019-03-03 Thread Martin Loncaric (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782821#comment-16782821 ] Martin Loncaric commented on SPARK-26555: - Yes - when I take away any randomness and use the

[jira] [Updated] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-03 Thread Babulal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated SPARK-27036: Description: During broadcast table job is execution if broadcast timeout (spark.sql.broadcastTimeout)

[jira] [Updated] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-03 Thread Babulal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated SPARK-27036: Attachment: image-2019-03-04-00-39-12-210.png > Even Broadcast thread is timed out, BroadCast Job is not

[jira] [Updated] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-03 Thread Babulal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated SPARK-27036: Attachment: image-2019-03-04-00-38-52-401.png > Even Broadcast thread is timed out, BroadCast Job is not

[jira] [Created] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-03 Thread Babulal (JIRA)
Babulal created SPARK-27036: --- Summary: Even Broadcast thread is timed out, BroadCast Job is not aborted. Key: SPARK-27036 URL: https://issues.apache.org/jira/browse/SPARK-27036 Project: Spark

[jira] [Resolved] (SPARK-25853) Parts of spark components (DAG Visualizationand executors page) not available in Internet Explorer

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25853. --- Resolution: Won't Fix It looks like recent versions of Internet Explorer, as Edge, do support this.

[jira] [Comment Edited] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

2019-03-03 Thread Martin Loncaric (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782821#comment-16782821 ] Martin Loncaric edited comment on SPARK-26555 at 3/3/19 6:56 PM: - Yes -

[jira] [Commented] (SPARK-26984) Incompatibility between Spark releases - Some(null)

2019-03-03 Thread Gerard Alexander (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782815#comment-16782815 ] Gerard Alexander commented on SPARK-26984: -- Sean Owen: you are right of course it should be

[jira] [Resolved] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2019-03-03 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid resolved SPARK-23134. Resolution: Duplicate > WebUI is showing the cache table details even after cache idle timeout >

[jira] [Issue Comment Deleted] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2019-03-03 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-23134: --- Comment: was deleted (was: Will resolve once the Jira SPARK-27012 merged) > WebUI is showing the cache

[jira] [Commented] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782810#comment-16782810 ] Sean Owen commented on SPARK-25863: --- Returning 0 seems like the correct thing to do, locally.

[jira] [Reopened] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2019-03-03 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid reopened SPARK-23134: > WebUI is showing the cache table details even after cache idle timeout >

[jira] [Resolved] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2019-03-03 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid resolved SPARK-23134. Resolution: Duplicate > WebUI is showing the cache table details even after cache idle timeout >

[jira] [Commented] (SPARK-25982) Dataframe write is non blocking in fair scheduling mode

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782808#comment-16782808 ] Sean Owen commented on SPARK-25982: --- Can you clarify with a more complete example? what is running in

[jira] [Commented] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782806#comment-16782806 ] Sean Owen commented on SPARK-26555: --- To be clear, is there a data set that works only when not run in

[jira] [Reopened] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2019-03-03 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid reopened SPARK-23134: Will resolve once the Jira SPARK-27012 merged > WebUI is showing the cache table details even after cache

[jira] [Resolved] (SPARK-23134) WebUI is showing the cache table details even after cache idle timeout

2019-03-03 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid resolved SPARK-23134. Resolution: Duplicate > WebUI is showing the cache table details even after cache idle timeout >

[jira] [Commented] (SPARK-26881) Scaling issue with Gramian computation for RowMatrix: too many results sent to driver

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782803#comment-16782803 ] Sean Owen commented on SPARK-26881: --- [~gagafunctor] would you like to open a pull request? I think the

[jira] [Resolved] (SPARK-26906) Pyspark RDD Replication Potentially Not Working

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26906. --- Resolution: Cannot Reproduce > Pyspark RDD Replication Potentially Not Working >

[jira] [Resolved] (SPARK-26980) Kryo deserialization not working with KryoSerializable class

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26980. --- Resolution: Not A Problem Yes, my guess is it's because you're using Spark's Kryo and config, and

[jira] [Commented] (SPARK-26991) Investigate difference of `returnNullable` between ScalaReflection.deserializerFor and JavaTypeInference.deserializerFor

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782796#comment-16782796 ] Sean Owen commented on SPARK-26991: --- This is a philosophical question about what JIRA is for. JIRA was

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782789#comment-16782789 ] Sean Owen commented on SPARK-27025: --- It's an interesting question; let's break it down. Calling

[jira] [Commented] (SPARK-26016) Encoding not working when using a map / mapPartitions call

2019-03-03 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782778#comment-16782778 ] Maxim Gekk commented on SPARK-26016: > nothing reinterprets the bytes according to a different

[jira] [Resolved] (SPARK-26620) DataFrameReader.json and csv in Python should accept DataFrame.

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26620. --- Resolution: Not A Problem > DataFrameReader.json and csv in Python should accept DataFrame. >

[jira] [Resolved] (SPARK-26610) Fix inconsistency between toJSON Method in Python and Scala

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26610. --- Resolution: Not A Problem > Fix inconsistency between toJSON Method in Python and Scala >

[jira] [Commented] (SPARK-26016) Encoding not working when using a map / mapPartitions call

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782774#comment-16782774 ] Sean Owen commented on SPARK-26016: --- [~maxgekk] I see, but am I correct that in the text source,

[jira] [Reopened] (SPARK-26918) All .md should have ASF license header

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-26918: --- Huh, OK. I had thought all these headers were actually conveniences, and technically redundant with

[jira] [Resolved] (SPARK-24778) DateTimeUtils.getTimeZone method returns GMT time if timezone cannot be parsed

2019-03-03 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk resolved SPARK-24778. Resolution: Fixed Fix Version/s: 3.0.0 The issue has been fixed already by using ZoneId.of

[jira] [Resolved] (SPARK-27016) Treat all antlr warnings as errors while generating parser from the sql grammar file.

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27016. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23925

[jira] [Assigned] (SPARK-27016) Treat all antlr warnings as errors while generating parser from the sql grammar file.

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27016: - Assignee: Dilip Biswal > Treat all antlr warnings as errors while generating parser from the

[jira] [Resolved] (SPARK-26274) Download page must link to https://www.apache.org/dist/spark for current releases

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26274. --- Resolution: Fixed Assignee: Sean Owen Fix Version/s: 2.3.3 > Download page must

[jira] [Commented] (SPARK-26146) CSV wouln't be ingested in Spark 2.4.0 with Scala 2.12

2019-03-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782759#comment-16782759 ] Sean Owen commented on SPARK-26146: --- Wait a sec, here's the explanation:

[jira] [Commented] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

2019-03-03 Thread Mohamed Mehdi BEN AISSA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782748#comment-16782748 ] Mohamed Mehdi BEN AISSA commented on SPARK-26795: - I have the same issue with spark

[jira] [Commented] (SPARK-24346) Executors are unable to fetch remote cache blocks

2019-03-03 Thread Mohamed Mehdi BEN AISSA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782739#comment-16782739 ] Mohamed Mehdi BEN AISSA commented on SPARK-24346: - Many thanks [~kien_truong] !  

[jira] [Commented] (SPARK-24346) Executors are unable to fetch remote cache blocks

2019-03-03 Thread Truong Duc Kien (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782723#comment-16782723 ] Truong Duc Kien commented on SPARK-24346: - We never got to find out the cause of this problem.

[jira] [Commented] (SPARK-24346) Executors are unable to fetch remote cache blocks

2019-03-03 Thread Mohamed Mehdi BEN AISSA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782678#comment-16782678 ] Mohamed Mehdi BEN AISSA commented on SPARK-24346: - Any news !?  I have exactly the same

[jira] [Commented] (SPARK-24346) Executors are unable to fetch remote cache blocks

2019-03-03 Thread Mohamed Mehdi BEN AISSA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782679#comment-16782679 ] Mohamed Mehdi BEN AISSA commented on SPARK-24346: -

[jira] [Assigned] (SPARK-27035) Current time with microsecond resolution

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27035: Assignee: (was: Apache Spark) > Current time with microsecond resolution >

[jira] [Assigned] (SPARK-27035) Current time with microsecond resolution

2019-03-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27035: Assignee: Apache Spark > Current time with microsecond resolution >

[jira] [Updated] (SPARK-27035) Current time with microsecond resolution

2019-03-03 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-27035: --- Description: Currently,  the CurrentTimestamp expression uses 

[jira] [Created] (SPARK-27035) Current time with microsecond resolution

2019-03-03 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27035: -- Summary: Current time with microsecond resolution Key: SPARK-27035 URL: https://issues.apache.org/jira/browse/SPARK-27035 Project: Spark Issue Type: Improvement

  1   2   >