[jira] [Updated] (SPARK-43947) Incorrect SparkException when missing config in resources in Stage-Level Scheduling

2023-06-02 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-43947: Summary: Incorrect SparkException when missing config in resources in Stage-Level

[jira] [Created] (SPARK-43947) Incorrect SparkException when missing amount in resources in Stage-Level Scheduling

2023-06-02 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-43947: --- Summary: Incorrect SparkException when missing amount in resources in Stage-Level Scheduling Key: SPARK-43947 URL: https://issues.apache.org/jira/browse/SPARK-43947

[jira] [Created] (SPARK-43912) Incorrect SparkException for Stage-Level Scheduling in local mode

2023-06-01 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-43912: --- Summary: Incorrect SparkException for Stage-Level Scheduling in local mode Key: SPARK-43912 URL: https://issues.apache.org/jira/browse/SPARK-43912 Project:

[jira] [Updated] (SPARK-43152) User-defined output metadata path (_spark_metadata)

2023-04-20 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-43152: Summary: User-defined output metadata path (_spark_metadata) (was: Parametrisable output

[jira] [Commented] (SPARK-42977) spark sql Disable vectorized faild

2023-03-31 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707256#comment-17707256 ] Jacek Laskowski commented on SPARK-42977: - Unless you can reproduce it without Iceberg, it's

[jira] [Updated] (SPARK-42496) Introducting Spark Connect at main page

2023-03-14 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-42496: Summary: Introducting Spark Connect at main page (was: Introduction Spark Connect at

[jira] [Updated] (SPARK-40821) Fix late record filtering to support chaining of stateful operators

2022-10-25 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-40821: Summary: Fix late record filtering to support chaining of stateful operators (was: Fix

[jira] [Updated] (SPARK-40807) "RocksDB: commit - pause bg time total" metric always 0

2022-10-15 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-40807: Summary: "RocksDB: commit - pause bg time total" metric always 0 (was: RocksDBStateStore

[jira] [Updated] (SPARK-40807) RocksDBStateStore always 0 for "RocksDB: commit - pause bg time total" metric

2022-10-15 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-40807: Summary: RocksDBStateStore always 0 for "RocksDB: commit - pause bg time total" metric

[jira] [Updated] (SPARK-40807) RocksDBStateStore always 0 for "RocksDB: commit - pause bg time total" metric

2022-10-15 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-40807: Attachment: spark-streams-commit-pause-bg-time.png > RocksDBStateStore always 0 for

[jira] [Created] (SPARK-40807) RocksDBStateStore always 0 for pause bg time total metric

2022-10-15 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-40807: --- Summary: RocksDBStateStore always 0 for pause bg time total metric Key: SPARK-40807 URL: https://issues.apache.org/jira/browse/SPARK-40807 Project: Spark

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2022-05-08 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533465#comment-17533465 ] Jacek Laskowski commented on SPARK-17556: - Given: # "I'm running a large query with over

[jira] [Resolved] (SPARK-36904) The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH

2021-10-17 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski resolved SPARK-36904. - Resolution: Invalid I finally managed to find the root cause of the issue which is 

[jira] [Commented] (SPARK-36904) The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH

2021-09-30 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422784#comment-17422784 ] Jacek Laskowski commented on SPARK-36904: - The table does not exist. I simply tried to execute a

[jira] [Commented] (SPARK-36904) The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH

2021-09-30 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422727#comment-17422727 ] Jacek Laskowski commented on SPARK-36904: - The solution from [this answer on

[jira] [Updated] (SPARK-36904) The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH

2021-09-30 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-36904: Attachment: exception.txt > The specified datastore driver ("org.postgresql.Driver") was

[jira] [Updated] (SPARK-36904) The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH

2021-09-30 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-36904: Environment: Spark 3.2.0 (RC6) {code:java} $ ./bin/spark-shell --version

[jira] [Created] (SPARK-36904) The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH

2021-09-30 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-36904: --- Summary: The specified datastore driver ("org.postgresql.Driver") was not found in the CLASSPATH Key: SPARK-36904 URL: https://issues.apache.org/jira/browse/SPARK-36904

[jira] [Resolved] (SPARK-34351) Running into "Py4JJavaError" while counting to text file or list using Pyspark, Jupyter notebook

2021-02-04 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski resolved SPARK-34351. - Resolution: Invalid Please use StackOverflow or the user@spark.a.o mailing list to ask

[jira] [Created] (SPARK-34264) Prevent incomplete master URLs for Spark on Kubernetes early

2021-01-27 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-34264: --- Summary: Prevent incomplete master URLs for Spark on Kubernetes early Key: SPARK-34264 URL: https://issues.apache.org/jira/browse/SPARK-34264 Project: Spark

[jira] [Commented] (SPARK-32333) Drop references to Master

2021-01-22 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270058#comment-17270058 ] Jacek Laskowski commented on SPARK-32333: - Just today when I was reading about a new ASF project

[jira] [Created] (SPARK-34158) Incorrect url of the only developer Matei in pom.xml

2021-01-19 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-34158: --- Summary: Incorrect url of the only developer Matei in pom.xml Key: SPARK-34158 URL: https://issues.apache.org/jira/browse/SPARK-34158 Project: Spark

[jira] [Created] (SPARK-34131) NPE when driver.podTemplateFile defines no containers

2021-01-15 Thread Jacek Laskowski (Jira)
Jacek Laskowski created SPARK-34131: --- Summary: NPE when driver.podTemplateFile defines no containers Key: SPARK-34131 URL: https://issues.apache.org/jira/browse/SPARK-34131 Project: Spark

[jira] [Resolved] (SPARK-34024) datasourceV1 VS dataSourceV2

2021-01-06 Thread Jacek Laskowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski resolved SPARK-34024. - Resolution: Invalid Please post questions to the u...@spark.apache.org mailing list or

[jira] [Commented] (SPARK-27708) Add documentation for v2 data sources

2019-06-10 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860013#comment-16860013 ] Jacek Laskowski commented on SPARK-27708: - [~rdblue] Mind if I asked you to update the

[jira] [Created] (SPARK-27977) MicroBatchWriter should use StreamWriter for human-friendly textual representation (toString)

2019-06-07 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-27977: --- Summary: MicroBatchWriter should use StreamWriter for human-friendly textual representation (toString) Key: SPARK-27977 URL:

[jira] [Created] (SPARK-27975) ConsoleSink should display alias and options for streaming progress

2019-06-07 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-27975: --- Summary: ConsoleSink should display alias and options for streaming progress Key: SPARK-27975 URL: https://issues.apache.org/jira/browse/SPARK-27975 Project:

[jira] [Updated] (SPARK-27933) Extracting common purge "behaviour" to the parent StreamExecution

2019-06-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-27933: Summary: Extracting common purge "behaviour" to the parent StreamExecution (was:

[jira] [Created] (SPARK-27933) Introduce StreamExecution.purge for removing entries from metadata logs

2019-06-03 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-27933: --- Summary: Introduce StreamExecution.purge for removing entries from metadata logs Key: SPARK-27933 URL: https://issues.apache.org/jira/browse/SPARK-27933

[jira] [Commented] (SPARK-27708) Add documentation for v2 data sources

2019-05-14 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839846#comment-16839846 ] Jacek Laskowski commented on SPARK-27708: - What's really needed? I've been reviewing the code of

[jira] [Updated] (SPARK-27708) Add documentation for v2 data sources

2019-05-14 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-27708: Labels: documentation (was: docuentation) > Add documentation for v2 data sources >

[jira] [Commented] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2019-02-12 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765971#comment-16765971 ] Jacek Laskowski commented on SPARK-20597: - [~nimfadora] sure. go ahead. > KafkaSourceProvider

[jira] [Created] (SPARK-26063) CatalystDataToAvro gives "UnresolvedException: Invalid call to dataType on unresolved object" when requested for numberedTreeString

2018-11-14 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-26063: --- Summary: CatalystDataToAvro gives "UnresolvedException: Invalid call to dataType on unresolved object" when requested for numberedTreeString Key: SPARK-26063 URL:

[jira] [Created] (SPARK-26062) Rename spark-avro external module to spark-sql-avro (to match spark-sql-kafka)

2018-11-14 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-26062: --- Summary: Rename spark-avro external module to spark-sql-avro (to match spark-sql-kafka) Key: SPARK-26062 URL: https://issues.apache.org/jira/browse/SPARK-26062

[jira] [Updated] (SPARK-25278) Number of output rows metric of union of views is multiplied by their occurrences

2018-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-25278: Description: When you use a view in a union multiple times (self-union), the {{number of

[jira] [Updated] (SPARK-25278) Number of output rows metric of union of views is multiplied by their occurrences

2018-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-25278: Description: When you use a view in a union multiple times (self-union), the {{number of

[jira] [Updated] (SPARK-25278) Number of output rows metric of union of views is multiplied by their occurrences

2018-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-25278: Description: When you use a view in a union multiple times (self-union), the {{number of

[jira] [Updated] (SPARK-25278) Number of output rows metric of union of views is multiplied by their occurrences

2018-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-25278: Attachment: union-3-views.png > Number of output rows metric of union of views is

[jira] [Created] (SPARK-25278) Number of output rows metric of union of views is multiplied by their occurrences

2018-08-30 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-25278: --- Summary: Number of output rows metric of union of views is multiplied by their occurrences Key: SPARK-25278 URL: https://issues.apache.org/jira/browse/SPARK-25278

[jira] [Updated] (SPARK-25278) Number of output rows metric of union of views is multiplied by their occurrences

2018-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-25278: Attachment: union-2-views.png > Number of output rows metric of union of views is

[jira] [Commented] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2018-07-27 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559412#comment-16559412 ] Jacek Laskowski commented on SPARK-20597: - Sorry [~Satyajit] for not responding earlier. I'd

[jira] [Created] (SPARK-24899) Add example of monotonically_increasing_id standard function to scaladoc

2018-07-24 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-24899: --- Summary: Add example of monotonically_increasing_id standard function to scaladoc Key: SPARK-24899 URL: https://issues.apache.org/jira/browse/SPARK-24899

[jira] [Updated] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-24408: Summary: Move abs function to math_funcs group (was: Move abs, bitwiseNOT, isnan, nanvl

[jira] [Created] (SPARK-24490) Use WebUI.addStaticHandler in web UIs

2018-06-07 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-24490: --- Summary: Use WebUI.addStaticHandler in web UIs Key: SPARK-24490 URL: https://issues.apache.org/jira/browse/SPARK-24490 Project: Spark Issue Type:

[jira] [Created] (SPARK-24408) Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group

2018-05-29 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-24408: --- Summary: Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group Key: SPARK-24408 URL: https://issues.apache.org/jira/browse/SPARK-24408 Project:

[jira] [Commented] (SPARK-24025) Join of bucketed and non-bucketed tables can give two exchanges and sorts for non-bucketed side

2018-04-20 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445520#comment-16445520 ] Jacek Laskowski commented on SPARK-24025: - it seems related or duplicated > Join of bucketed and

[jira] [Commented] (SPARK-24025) Join of bucketed and non-bucketed tables can give two exchanges and sorts for non-bucketed side

2018-04-20 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445512#comment-16445512 ] Jacek Laskowski commented on SPARK-24025: - I was about to have closed this as a duplicate, but

[jira] [Commented] (SPARK-24025) Join of bucketed and non-bucketed tables can give two exchanges and sorts for non-bucketed side

2018-04-19 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444998#comment-16444998 ] Jacek Laskowski commented on SPARK-24025: - The other issue seems similar. > Join of bucketed and

[jira] [Updated] (SPARK-24025) Join of bucketed and non-bucketed tables can give two exchanges and sorts for non-bucketed side

2018-04-19 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-24025: Attachment: join-jira.png > Join of bucketed and non-bucketed tables can give two

[jira] [Created] (SPARK-24025) Join of bucketed and non-bucketed tables can give two exchanges and sorts for non-bucketed side

2018-04-19 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-24025: --- Summary: Join of bucketed and non-bucketed tables can give two exchanges and sorts for non-bucketed side Key: SPARK-24025 URL:

[jira] [Commented] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-04-18 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442488#comment-16442488 ] Jacek Laskowski commented on SPARK-23830: - It's about how easy it is to find out that the issue

[jira] [Created] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-03-30 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-23830: --- Summary: Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object Key: SPARK-23830 URL:

[jira] [Created] (SPARK-23731) FileSourceScanExec throws NullPointerException in subexpression elimination

2018-03-18 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-23731: --- Summary: FileSourceScanExec throws NullPointerException in subexpression elimination Key: SPARK-23731 URL: https://issues.apache.org/jira/browse/SPARK-23731

[jira] [Commented] (SPARK-20536) Extend ColumnName to create StructFields with explicit nullable

2018-03-14 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398899#comment-16398899 ] Jacek Laskowski commented on SPARK-20536: - I'm not sure how meaningful it still is, but given

[jira] [Created] (SPARK-23229) Dataset.hint should use planWithBarrier logical plan

2018-01-26 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-23229: --- Summary: Dataset.hint should use planWithBarrier logical plan Key: SPARK-23229 URL: https://issues.apache.org/jira/browse/SPARK-23229 Project: Spark

[jira] [Commented] (SPARK-22457) Tables are supposed to be MANAGED only taking into account whether a path is provided

2018-01-16 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326959#comment-16326959 ] Jacek Laskowski commented on SPARK-22457: -  That should be fairly easy to fix _iff_ we want to

[jira] [Created] (SPARK-22954) ANALYZE TABLE fails with NoSuchTableException for temporary tables (but should have reported "not supported on views")

2018-01-04 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-22954: --- Summary: ANALYZE TABLE fails with NoSuchTableException for temporary tables (but should have reported "not supported on views") Key: SPARK-22954 URL:

[jira] [Commented] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-02 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308363#comment-16308363 ] Jacek Laskowski commented on SPARK-22935: - It does not seem to be the case as described in

[jira] [Commented] (SPARK-22929) Short name for "kafka" doesn't work in pyspark with packages

2017-12-31 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307149#comment-16307149 ] Jacek Laskowski commented on SPARK-22929: - When I saw the issue I was so much surprised as that's

[jira] [Updated] (SPARK-22048) Show id, runId, batch in Description column in SQL tab for streaming queries (as in Jobs)

2017-09-18 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-22048: Attachment: webui-jobs-description.png webui-sql-description.png > Show

[jira] [Created] (SPARK-22048) Show id, runId, batch in Description column in SQL tab for streaming queries (as in Jobs)

2017-09-18 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-22048: --- Summary: Show id, runId, batch in Description column in SQL tab for streaming queries (as in Jobs) Key: SPARK-22048 URL: https://issues.apache.org/jira/browse/SPARK-22048

[jira] [Created] (SPARK-22044) explain function with codegen and cost parameters

2017-09-17 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-22044: --- Summary: explain function with codegen and cost parameters Key: SPARK-22044 URL: https://issues.apache.org/jira/browse/SPARK-22044 Project: Spark

[jira] [Commented] (SPARK-22040) current_date function with timezone id

2017-09-17 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169215#comment-16169215 ] Jacek Laskowski commented on SPARK-22040: - That'd be awesome! It's yours, [~mgaido] >

[jira] [Created] (SPARK-22040) current_date function with timezone id

2017-09-16 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-22040: --- Summary: current_date function with timezone id Key: SPARK-22040 URL: https://issues.apache.org/jira/browse/SPARK-22040 Project: Spark Issue Type:

[jira] [Created] (SPARK-21901) Define toString for StateOperatorProgress

2017-09-03 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21901: --- Summary: Define toString for StateOperatorProgress Key: SPARK-21901 URL: https://issues.apache.org/jira/browse/SPARK-21901 Project: Spark Issue Type:

[jira] [Created] (SPARK-21886) Use SparkSession.internalCreateDataFrame to create Dataset with LogicalRDD logical operator

2017-08-31 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21886: --- Summary: Use SparkSession.internalCreateDataFrame to create Dataset with LogicalRDD logical operator Key: SPARK-21886 URL: https://issues.apache.org/jira/browse/SPARK-21886

[jira] [Updated] (SPARK-21728) Allow SparkSubmit to use logging

2017-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-21728: Attachment: logging.patch sparksubmit.patch > Allow SparkSubmit to use

[jira] [Commented] (SPARK-21728) Allow SparkSubmit to use logging

2017-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147957#comment-16147957 ] Jacek Laskowski commented on SPARK-21728: - After I changed your change, I could see the logs

[jira] [Commented] (SPARK-21728) Allow SparkSubmit to use logging

2017-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147840#comment-16147840 ] Jacek Laskowski commented on SPARK-21728: - The idea behind the custom {{conf/log4j.properties}}

[jira] [Commented] (SPARK-21728) Allow SparkSubmit to use logging

2017-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147643#comment-16147643 ] Jacek Laskowski commented on SPARK-21728: - Thanks [~vanzin] for the prompt response! I'm stuck

[jira] [Commented] (SPARK-21728) Allow SparkSubmit to use logging

2017-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147086#comment-16147086 ] Jacek Laskowski commented on SPARK-21728: - Thanks [~sowen]. I'll label it as such when I know if

[jira] [Commented] (SPARK-21728) Allow SparkSubmit to use logging

2017-08-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146780#comment-16146780 ] Jacek Laskowski commented on SPARK-21728: - I think the change is user-visible and therefore

[jira] [Commented] (SPARK-21765) Ensure all leaf nodes that are derived from streaming sources have isStreaming=true

2017-08-26 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142921#comment-16142921 ] Jacek Laskowski commented on SPARK-21765: - BTW, *Assignee* field of the JIRA is empty, but should

[jira] [Commented] (SPARK-21765) Ensure all leaf nodes that are derived from streaming sources have isStreaming=true

2017-08-26 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142919#comment-16142919 ] Jacek Laskowski commented on SPARK-21765: - I think {{TextSocketSource}} was missed in the change

[jira] [Commented] (SPARK-21667) ConsoleSink should not fail streaming query with checkpointLocation option

2017-08-09 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120576#comment-16120576 ] Jacek Laskowski commented on SPARK-21667: - Oh, what an offer! Couldn't have thought of a better

[jira] [Created] (SPARK-21667) ConsoleSink should not fail streaming query with checkpointLocation option

2017-08-08 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21667: --- Summary: ConsoleSink should not fail streaming query with checkpointLocation option Key: SPARK-21667 URL: https://issues.apache.org/jira/browse/SPARK-21667

[jira] [Updated] (SPARK-21546) dropDuplicates with watermark yields RuntimeException due to binding failure

2017-07-27 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-21546: Description: With today's master... The following streaming query with watermark and

[jira] [Updated] (SPARK-21546) dropDuplicates with watermark yields RuntimeException due to binding failure

2017-07-27 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-21546: Summary: dropDuplicates with watermark yields RuntimeException due to binding failure

[jira] [Created] (SPARK-21546) dropDuplicates followed by select yields RuntimeException due to binding failure

2017-07-27 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21546: --- Summary: dropDuplicates followed by select yields RuntimeException due to binding failure Key: SPARK-21546 URL: https://issues.apache.org/jira/browse/SPARK-21546

[jira] [Created] (SPARK-21429) show on structured Dataset is equivalent to writeStream to console once

2017-07-16 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21429: --- Summary: show on structured Dataset is equivalent to writeStream to console once Key: SPARK-21429 URL: https://issues.apache.org/jira/browse/SPARK-21429

[jira] [Created] (SPARK-21427) Describe mapGroupsWithState and flatMapGroupsWithState for stateful aggregation in Structured Streaming

2017-07-16 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21427: --- Summary: Describe mapGroupsWithState and flatMapGroupsWithState for stateful aggregation in Structured Streaming Key: SPARK-21427 URL:

[jira] [Created] (SPARK-21329) Make EventTimeWatermarkExec explicitly UnaryExecNode

2017-07-06 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21329: --- Summary: Make EventTimeWatermarkExec explicitly UnaryExecNode Key: SPARK-21329 URL: https://issues.apache.org/jira/browse/SPARK-21329 Project: Spark

[jira] [Created] (SPARK-21313) ConsoleSink's string representation

2017-07-05 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21313: --- Summary: ConsoleSink's string representation Key: SPARK-21313 URL: https://issues.apache.org/jira/browse/SPARK-21313 Project: Spark Issue Type:

[jira] [Created] (SPARK-21285) VectorAssembler should report the column name when data type used is not supported

2017-07-03 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21285: --- Summary: VectorAssembler should report the column name when data type used is not supported Key: SPARK-21285 URL: https://issues.apache.org/jira/browse/SPARK-21285

[jira] [Commented] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2017-06-30 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070865#comment-16070865 ] Jacek Laskowski commented on SPARK-20597: - Go for it, [~Satyajit]! > KafkaSourceProvider falls

[jira] [Commented] (SPARK-20997) spark-submit's --driver-cores marked as "YARN-only" but listed under "Spark standalone with cluster deploy mode only"

2017-06-07 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040594#comment-16040594 ] Jacek Laskowski commented on SPARK-20997: - Go ahead! Thanks [~guoxiaolongzte]! > spark-submit's

[jira] [Created] (SPARK-20997) spark-submit's --driver-cores marked as "YARN-only" but listed under "Spark standalone with cluster deploy mode only"

2017-06-06 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20997: --- Summary: spark-submit's --driver-cores marked as "YARN-only" but listed under "Spark standalone with cluster deploy mode only" Key: SPARK-20997 URL:

[jira] [Commented] (SPARK-20782) Dataset's isCached operator

2017-06-02 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035316#comment-16035316 ] Jacek Laskowski commented on SPARK-20782: - Just stumbled upon {{CatalogImpl.isCached}} that could

[jira] [Updated] (SPARK-20937) Describe spark.sql.parquet.writeLegacyFormat property in Spark SQL, DataFrames and Datasets Guide

2017-05-31 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-20937: Description: As a follow-up to SPARK-20297 (and SPARK-10400) in which

[jira] [Created] (SPARK-20937) Describe spark.sql.parquet.writeLegacyFormat property in Spark SQL, DataFrames and Datasets Guide

2017-05-31 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20937: --- Summary: Describe spark.sql.parquet.writeLegacyFormat property in Spark SQL, DataFrames and Datasets Guide Key: SPARK-20937 URL:

[jira] [Resolved] (SPARK-20865) caching dataset throws "Queries with streaming sources must be executed with writeStream.start()"

2017-05-31 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski resolved SPARK-20865. - Resolution: Won't Fix Fix Version/s: 2.3.0 2.2.0 {{cache}} is

[jira] [Created] (SPARK-20927) Add cache operator to Unsupported Operations in Structured Streaming

2017-05-30 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20927: --- Summary: Add cache operator to Unsupported Operations in Structured Streaming Key: SPARK-20927 URL: https://issues.apache.org/jira/browse/SPARK-20927 Project:

[jira] [Commented] (SPARK-20912) map function with columns as strings

2017-05-29 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028285#comment-16028285 ] Jacek Laskowski commented on SPARK-20912: - I can and I did, but the point is that it's not

[jira] [Commented] (SPARK-20912) map function with columns as strings

2017-05-29 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028208#comment-16028208 ] Jacek Laskowski commented on SPARK-20912: - Nope as it would create a map of the two literals but

[jira] [Created] (SPARK-20912) map function with columns as strings

2017-05-29 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20912: --- Summary: map function with columns as strings Key: SPARK-20912 URL: https://issues.apache.org/jira/browse/SPARK-20912 Project: Spark Issue Type:

[jira] [Created] (SPARK-20782) Dataset's isCached operator

2017-05-17 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20782: --- Summary: Dataset's isCached operator Key: SPARK-20782 URL: https://issues.apache.org/jira/browse/SPARK-20782 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-4570) Add broadcast join to left semi join

2017-05-16 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-4570: --- Description: For now, spark use broadcast join instead of hash join to optimize {{inner

[jira] [Updated] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-12 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-20600: Description: Executing the following batch query gives the default stringified *internal

[jira] [Created] (SPARK-20691) Difference between Storage Memory as seen internally and in web UI

2017-05-10 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20691: --- Summary: Difference between Storage Memory as seen internally and in web UI Key: SPARK-20691 URL: https://issues.apache.org/jira/browse/SPARK-20691 Project:

[jira] [Commented] (SPARK-20630) Thread Dump link available in Executors tab irrespective of spark.ui.threadDumpsEnabled

2017-05-08 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001259#comment-16001259 ] Jacek Laskowski commented on SPARK-20630: - Go [~ajbozarth], go! > Thread Dump link available in

  1   2   3   >