[jira] [Resolved] (SPARK-23993) Support DESC FORMATTED table_name column_name
[
https://issues.apache.org/jira/browse/SPARK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Glushak resolved SPARK-23993.
---
Resolution: Duplicate
Fix Version/s: 2.3.0
Duplicate of SPARK-17642.
Thanks [~ksunitha] for investigation.
> Support DESC FORMATTED table_name column_name
> -
>
> Key: SPARK-23993
> URL: https://issues.apache.org/jira/browse/SPARK-23993
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.1.2
>Reporter: Volodymyr Glushak
>Priority: Major
> Fix For: 2.3.0
>
>
> Hive and Spark both supports:
> {code}
> DESC FORMATTED table_name{code}
> which gives table metadata.
> If you want to get metadata for particular column in hive you can execute:
> {code}
> DESC FORMATTED table_name column_name{code}
> Thos is not supported in Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24133) Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException
Ala Luszczak created SPARK-24133: Summary: Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException Key: SPARK-24133 URL: https://issues.apache.org/jira/browse/SPARK-24133 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.3.0 Reporter: Ala Luszczak ColumnVectors store string data in one big byte array. Since the array size is capped at just under Integer.MAX_VALUE, a single ColumnVector cannot store more than 2GB of string data. However, since the Parquet files commonly contain large blobs stored as strings, and ColumnVectors by default carry 4096 values, it's entirely possible to go past that limit. In such cases a negative capacity is requested from WritableColumnVector.reserve(). The call succeeds (requested capacity is smaller than already allocated), and consequently java.lang.ArrayIndexOutOfBoundsException is thrown when the reader actually attempts to put the data into the array. This behavior is hard to troubleshoot for the users. Spark should instead check for negative requested capacity in WritableColumnVector.reserve() and throw more informative error, instructing the user to tweak ColumnarBatch size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24061) [SS]TypedFilter is not supported in Continuous Processing
[ https://issues.apache.org/jira/browse/SPARK-24061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-24061. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21136 [https://github.com/apache/spark/pull/21136] > [SS]TypedFilter is not supported in Continuous Processing > - > > Key: SPARK-24061 > URL: https://issues.apache.org/jira/browse/SPARK-24061 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wang Yanlin >Priority: Major > Fix For: 2.4.0 > > Attachments: TypedFilter_error.png > > > using filter with filter function in continuous processing application ,cause > error -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24061) [SS]TypedFilter is not supported in Continuous Processing
[ https://issues.apache.org/jira/browse/SPARK-24061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-24061: Assignee: Wang Yanlin > [SS]TypedFilter is not supported in Continuous Processing > - > > Key: SPARK-24061 > URL: https://issues.apache.org/jira/browse/SPARK-24061 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Wang Yanlin >Assignee: Wang Yanlin >Priority: Major > Fix For: 2.4.0 > > Attachments: TypedFilter_error.png > > > using filter with filter function in continuous processing application ,cause > error -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24133) Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/SPARK-24133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459562#comment-16459562 ] Apache Spark commented on SPARK-24133: -- User 'ala' has created a pull request for this issue: https://github.com/apache/spark/pull/21206 > Reading Parquet files containing large strings can fail with > java.lang.ArrayIndexOutOfBoundsException > - > > Key: SPARK-24133 > URL: https://issues.apache.org/jira/browse/SPARK-24133 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ala Luszczak >Priority: Major > > ColumnVectors store string data in one big byte array. Since the array size > is capped at just under Integer.MAX_VALUE, a single ColumnVector cannot store > more than 2GB of string data. > However, since the Parquet files commonly contain large blobs stored as > strings, and ColumnVectors by default carry 4096 values, it's entirely > possible to go past that limit. > In such cases a negative capacity is requested from > WritableColumnVector.reserve(). The call succeeds (requested capacity is > smaller than already allocated), and consequently > java.lang.ArrayIndexOutOfBoundsException is thrown when the reader actually > attempts to put the data into the array. > This behavior is hard to troubleshoot for the users. Spark should instead > check for negative requested capacity in WritableColumnVector.reserve() and > throw more informative error, instructing the user to tweak ColumnarBatch > size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24133) Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/SPARK-24133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24133: Assignee: (was: Apache Spark) > Reading Parquet files containing large strings can fail with > java.lang.ArrayIndexOutOfBoundsException > - > > Key: SPARK-24133 > URL: https://issues.apache.org/jira/browse/SPARK-24133 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ala Luszczak >Priority: Major > > ColumnVectors store string data in one big byte array. Since the array size > is capped at just under Integer.MAX_VALUE, a single ColumnVector cannot store > more than 2GB of string data. > However, since the Parquet files commonly contain large blobs stored as > strings, and ColumnVectors by default carry 4096 values, it's entirely > possible to go past that limit. > In such cases a negative capacity is requested from > WritableColumnVector.reserve(). The call succeeds (requested capacity is > smaller than already allocated), and consequently > java.lang.ArrayIndexOutOfBoundsException is thrown when the reader actually > attempts to put the data into the array. > This behavior is hard to troubleshoot for the users. Spark should instead > check for negative requested capacity in WritableColumnVector.reserve() and > throw more informative error, instructing the user to tweak ColumnarBatch > size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24133) Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/SPARK-24133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24133: Assignee: Apache Spark > Reading Parquet files containing large strings can fail with > java.lang.ArrayIndexOutOfBoundsException > - > > Key: SPARK-24133 > URL: https://issues.apache.org/jira/browse/SPARK-24133 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Ala Luszczak >Assignee: Apache Spark >Priority: Major > > ColumnVectors store string data in one big byte array. Since the array size > is capped at just under Integer.MAX_VALUE, a single ColumnVector cannot store > more than 2GB of string data. > However, since the Parquet files commonly contain large blobs stored as > strings, and ColumnVectors by default carry 4096 values, it's entirely > possible to go past that limit. > In such cases a negative capacity is requested from > WritableColumnVector.reserve(). The call succeeds (requested capacity is > smaller than already allocated), and consequently > java.lang.ArrayIndexOutOfBoundsException is thrown when the reader actually > attempts to put the data into the array. > This behavior is hard to troubleshoot for the users. Spark should instead > check for negative requested capacity in WritableColumnVector.reserve() and > throw more informative error, instructing the user to tweak ColumnarBatch > size. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
Xiaodong DENG created SPARK-24134:
-
Summary: A missing full-stop in doc "Tuning Spark"
Key: SPARK-24134
URL: https://issues.apache.org/jira/browse/SPARK-24134
Project: Spark
Issue Type: Documentation
Components: Documentation
Affects Versions: 2.3.0
Reporter: Xiaodong DENG
In the document [Tuning Spark -> Determining Memory Consumption,
(https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
a full stop was missing in the second paragraph.
It's "{{*...use SizeEstimator’s estimate method This is useful for
experimenting...*"}}, while there is supposed to be a full stop before
"{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
[
https://issues.apache.org/jira/browse/SPARK-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-24134:
Assignee: (was: Apache Spark)
> A missing full-stop in doc "Tuning Spark"
> -
>
> Key: SPARK-24134
> URL: https://issues.apache.org/jira/browse/SPARK-24134
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Affects Versions: 2.3.0
>Reporter: Xiaodong DENG
>Priority: Minor
> Labels: documentation
> Attachments: screenshot.png
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> In the document [Tuning Spark -> Determining Memory Consumption,
> (https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
> a full stop was missing in the second paragraph.
> It's "{{*...use SizeEstimator’s estimate method This is useful for
> experimenting...*"}}, while there is supposed to be a full stop before
> "{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
[
https://issues.apache.org/jira/browse/SPARK-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated SPARK-24134:
--
Attachment: screenshot.png
> A missing full-stop in doc "Tuning Spark"
> -
>
> Key: SPARK-24134
> URL: https://issues.apache.org/jira/browse/SPARK-24134
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Affects Versions: 2.3.0
>Reporter: Xiaodong DENG
>Priority: Minor
> Labels: documentation
> Attachments: screenshot.png
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> In the document [Tuning Spark -> Determining Memory Consumption,
> (https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
> a full stop was missing in the second paragraph.
> It's "{{*...use SizeEstimator’s estimate method This is useful for
> experimenting...*"}}, while there is supposed to be a full stop before
> "{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
[
https://issues.apache.org/jira/browse/SPARK-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated SPARK-24134:
--
Attachment: (was: 39468206-778e3d8a-4d64-11e8-8a92-38464952b54b.png)
> A missing full-stop in doc "Tuning Spark"
> -
>
> Key: SPARK-24134
> URL: https://issues.apache.org/jira/browse/SPARK-24134
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Affects Versions: 2.3.0
>Reporter: Xiaodong DENG
>Priority: Minor
> Labels: documentation
> Attachments: screenshot.png
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> In the document [Tuning Spark -> Determining Memory Consumption,
> (https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
> a full stop was missing in the second paragraph.
> It's "{{*...use SizeEstimator’s estimate method This is useful for
> experimenting...*"}}, while there is supposed to be a full stop before
> "{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
[
https://issues.apache.org/jira/browse/SPARK-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated SPARK-24134:
--
Attachment: 39468206-778e3d8a-4d64-11e8-8a92-38464952b54b.png
> A missing full-stop in doc "Tuning Spark"
> -
>
> Key: SPARK-24134
> URL: https://issues.apache.org/jira/browse/SPARK-24134
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Affects Versions: 2.3.0
>Reporter: Xiaodong DENG
>Priority: Minor
> Labels: documentation
> Attachments: screenshot.png
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> In the document [Tuning Spark -> Determining Memory Consumption,
> (https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
> a full stop was missing in the second paragraph.
> It's "{{*...use SizeEstimator’s estimate method This is useful for
> experimenting...*"}}, while there is supposed to be a full stop before
> "{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
[
https://issues.apache.org/jira/browse/SPARK-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-24134:
Assignee: Apache Spark
> A missing full-stop in doc "Tuning Spark"
> -
>
> Key: SPARK-24134
> URL: https://issues.apache.org/jira/browse/SPARK-24134
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Affects Versions: 2.3.0
>Reporter: Xiaodong DENG
>Assignee: Apache Spark
>Priority: Minor
> Labels: documentation
> Attachments: screenshot.png
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> In the document [Tuning Spark -> Determining Memory Consumption,
> (https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
> a full stop was missing in the second paragraph.
> It's "{{*...use SizeEstimator’s estimate method This is useful for
> experimenting...*"}}, while there is supposed to be a full stop before
> "{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24134) A missing full-stop in doc "Tuning Spark"
[
https://issues.apache.org/jira/browse/SPARK-24134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459664#comment-16459664
]
Apache Spark commented on SPARK-24134:
--
User 'XD-DENG' has created a pull request for this issue:
https://github.com/apache/spark/pull/21205
> A missing full-stop in doc "Tuning Spark"
> -
>
> Key: SPARK-24134
> URL: https://issues.apache.org/jira/browse/SPARK-24134
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
>Affects Versions: 2.3.0
>Reporter: Xiaodong DENG
>Priority: Minor
> Labels: documentation
> Attachments: screenshot.png
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> In the document [Tuning Spark -> Determining Memory Consumption,
> (https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption|https://spark.apache.org/docs/latest/tuning.html#determining-memory-consumption]),
> a full stop was missing in the second paragraph.
> It's "{{*...use SizeEstimator’s estimate method This is useful for
> experimenting...*"}}, while there is supposed to be a full stop before
> "{{*This*"}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
Matt Cheah created SPARK-24135:
--
Summary: [K8s] Executors that fail to start up because of
init-container errors are not retried and limit the executor pool size
Key: SPARK-24135
URL: https://issues.apache.org/jira/browse/SPARK-24135
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.3.0
Reporter: Matt Cheah
In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
When executors fail in these ways, they are removed from the pending executors
pool and the driver should retry requesting these executors.
However, the driver does not handle a different class of error: when the pod
enters the {{Init:Error}} state. This state comes up when the executor fails to
launch because one of its init-containers fails. Spark itself doesn't attach
any init-containers to the executors. However, custom web hooks can run on the
cluster and attach init-containers to the executor pods. Additionally, pod
presets can specify init containers to run on these pods. Therefore Spark
should be handling the {{Init:Error}} cases regardless if Spark itself is aware
of init-containers or not.
This class of error is particularly bad because when we hit this state, the
failed executor will never start, but it's still seen as pending by the
executor allocator. The executor allocator won't request more rounds of
executors because its current batch hasn't been resolved to either running or
failed. Therefore we end up with being stuck with the number of executors that
successfully started before the faulty one failed to start, potentially
creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459754#comment-16459754
]
Matt Cheah commented on SPARK-24135:
[~foxish] [~eje] [~liyinan926] wanted to get feedback on this - the code change
is relatively straightforward, but is this the right stance / opinion to take
in Spark?
> [K8s] Executors that fail to start up because of init-container errors are
> not retried and limit the executor pool size
> ---
>
> Key: SPARK-24135
> URL: https://issues.apache.org/jira/browse/SPARK-24135
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
> having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
> When executors fail in these ways, they are removed from the pending
> executors pool and the driver should retry requesting these executors.
> However, the driver does not handle a different class of error: when the pod
> enters the {{Init:Error}} state. This state comes up when the executor fails
> to launch because one of its init-containers fails. Spark itself doesn't
> attach any init-containers to the executors. However, custom web hooks can
> run on the cluster and attach init-containers to the executor pods.
> Additionally, pod presets can specify init containers to run on these pods.
> Therefore Spark should be handling the {{Init:Error}} cases regardless if
> Spark itself is aware of init-containers or not.
> This class of error is particularly bad because when we hit this state, the
> failed executor will never start, but it's still seen as pending by the
> executor allocator. The executor allocator won't request more rounds of
> executors because its current batch hasn't been resolved to either running or
> failed. Therefore we end up with being stuck with the number of executors
> that successfully started before the faulty one failed to start, potentially
> creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-23941) Mesos task failed on specific spark app name
[
https://issues.apache.org/jira/browse/SPARK-23941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-23941.
Resolution: Fixed
Fix Version/s: 2.3.1
2.4.0
2.2.2
Issue resolved by pull request 21014
[https://github.com/apache/spark/pull/21014]
> Mesos task failed on specific spark app name
>
>
> Key: SPARK-23941
> URL: https://issues.apache.org/jira/browse/SPARK-23941
> Project: Spark
> Issue Type: Bug
> Components: Mesos, Spark Submit
>Affects Versions: 2.2.1, 2.3.0
> Environment: OS: Ubuntu 16.0.4
> Spark: 2.3.0
> Mesos: 1.5.0
>Reporter: bounkong khamphousone
>Priority: Major
> Fix For: 2.2.2, 2.4.0, 2.3.1
>
>
> It seems to be a bug related to spark's MesosClusterDispatcher. In order to
> reproduce the bug, you need to have mesos and mesos dispatcher running.
> I'm currently running mesos 1.5 and spark 2.3.0 (tried with 2.2.1 as well).
> If you launch the following program:
>
> {code:java}
> spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class
> org.apache.spark.examples.SparkPi --name "my favorite task (myId = 123-456)"
> /home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar 100
> {code}
> , then the task fails with the following output :
>
> {code:java}
> I0409 11:00:35.360352 22726 fetcher.cpp:551] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/tiboun","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"\/home\/tiboun\/tools\/spark\/examples\/jars\/spark-examples_2.11-2.3.0.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/0262246c-14a3-4408-9b74-5e3b65dc1344-S0\/frameworks\/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014\/executors\/driver-20180409110035-0004\/runs\/8ac20902-74e1-45c4-9ab6-c52a79940189","user":"tiboun"}
> I0409 11:00:35.363119 22726 fetcher.cpp:450] Fetching URI
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
> I0409 11:00:35.363143 22726 fetcher.cpp:291] Fetching directly into the
> sandbox directory
> I0409 11:00:35.363168 22726 fetcher.cpp:225] Fetching URI
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
> W0409 11:00:35.366839 22726 fetcher.cpp:330] Copying instead of extracting
> resource from URI with 'extract' flag, because it does not seem to be an
> archive: /home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar
> I0409 11:00:35.366873 22726 fetcher.cpp:603] Fetched
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar' to
> '/var/lib/mesos/slaves/0262246c-14a3-4408-9b74-5e3b65dc1344-S0/frameworks/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014/executors/driver-20180409110035-0004/runs/8ac20902-74e1-45c4-9ab6-c52a79940189/spark-examples_2.11-2.3.0.jar'
> I0409 11:00:35.366878 22726 fetcher.cpp:608] Successfully fetched all URIs
> into
> '/var/lib/mesos/slaves/0262246c-14a3-4408-9b74-5e3b65dc1344-S0/frameworks/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014/executors/driver-20180409110035-0004/runs/8ac20902-74e1-45c4-9ab6-c52a79940189'
> I0409 11:00:35.438725 22733 exec.cpp:162] Version: 1.5.0
> I0409 11:00:35.440770 22734 exec.cpp:236] Executor registered on agent
> 0262246c-14a3-4408-9b74-5e3b65dc1344-S0
> I0409 11:00:35.441388 22733 executor.cpp:171] Received SUBSCRIBED event
> I0409 11:00:35.441586 22733 executor.cpp:175] Subscribed executor on
> tiboun-Dell-Precision-M3800
> I0409 11:00:35.441643 22733 executor.cpp:171] Received LAUNCH event
> I0409 11:00:35.441767 22733 executor.cpp:638] Starting task
> driver-20180409110035-0004
> I0409 11:00:35.445050 22733 executor.cpp:478] Running
> '/usr/libexec/mesos/mesos-containerizer launch '
> I0409 11:00:35.445770 22733 executor.cpp:651] Forked command at 22743
> sh: 1: Syntax error: "(" unexpected
> I0409 11:00:35.538661 22736 executor.cpp:938] Command exited with status 2
> (pid: 22743)
> I0409 11:00:36.541016 22739 process.cpp:887] Failed to accept socket: future
> discarded
> {code}
> If you remove the parentheses, you get the following result:
>
> {code:java}
> I0409 11:03:02.023701 23085 fetcher.cpp:551] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/tiboun","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"\/home\/tiboun\/tools\/spark\/examples\/jars\/spark-examples_2.11-2.3.0.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/0262246c-14a3-4408-9b74-5e3b65dc1344-S0\/frameworks\/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014\/executors\/driver-20180409110301-0006\/runs\/f887c0ab-b48f-4382-850c-383c1c944269","user":"tiboun"}
> I0409 11:03:02.028268 23085 fetcher.cpp:450] Fetching URI
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
> I0409 11:03:02.028302 23085 fetcher.cpp:291] Fetching directly in
[jira] [Assigned] (SPARK-23941) Mesos task failed on specific spark app name
[
https://issues.apache.org/jira/browse/SPARK-23941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin reassigned SPARK-23941:
--
Assignee: bounkong khamphousone
> Mesos task failed on specific spark app name
>
>
> Key: SPARK-23941
> URL: https://issues.apache.org/jira/browse/SPARK-23941
> Project: Spark
> Issue Type: Bug
> Components: Mesos, Spark Submit
>Affects Versions: 2.2.1, 2.3.0
> Environment: OS: Ubuntu 16.0.4
> Spark: 2.3.0
> Mesos: 1.5.0
>Reporter: bounkong khamphousone
>Assignee: bounkong khamphousone
>Priority: Major
> Fix For: 2.2.2, 2.3.1, 2.4.0
>
>
> It seems to be a bug related to spark's MesosClusterDispatcher. In order to
> reproduce the bug, you need to have mesos and mesos dispatcher running.
> I'm currently running mesos 1.5 and spark 2.3.0 (tried with 2.2.1 as well).
> If you launch the following program:
>
> {code:java}
> spark-submit --master mesos://127.0.1.1:7077 --deploy-mode cluster --class
> org.apache.spark.examples.SparkPi --name "my favorite task (myId = 123-456)"
> /home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar 100
> {code}
> , then the task fails with the following output :
>
> {code:java}
> I0409 11:00:35.360352 22726 fetcher.cpp:551] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/tiboun","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"\/home\/tiboun\/tools\/spark\/examples\/jars\/spark-examples_2.11-2.3.0.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/0262246c-14a3-4408-9b74-5e3b65dc1344-S0\/frameworks\/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014\/executors\/driver-20180409110035-0004\/runs\/8ac20902-74e1-45c4-9ab6-c52a79940189","user":"tiboun"}
> I0409 11:00:35.363119 22726 fetcher.cpp:450] Fetching URI
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
> I0409 11:00:35.363143 22726 fetcher.cpp:291] Fetching directly into the
> sandbox directory
> I0409 11:00:35.363168 22726 fetcher.cpp:225] Fetching URI
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
> W0409 11:00:35.366839 22726 fetcher.cpp:330] Copying instead of extracting
> resource from URI with 'extract' flag, because it does not seem to be an
> archive: /home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar
> I0409 11:00:35.366873 22726 fetcher.cpp:603] Fetched
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar' to
> '/var/lib/mesos/slaves/0262246c-14a3-4408-9b74-5e3b65dc1344-S0/frameworks/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014/executors/driver-20180409110035-0004/runs/8ac20902-74e1-45c4-9ab6-c52a79940189/spark-examples_2.11-2.3.0.jar'
> I0409 11:00:35.366878 22726 fetcher.cpp:608] Successfully fetched all URIs
> into
> '/var/lib/mesos/slaves/0262246c-14a3-4408-9b74-5e3b65dc1344-S0/frameworks/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014/executors/driver-20180409110035-0004/runs/8ac20902-74e1-45c4-9ab6-c52a79940189'
> I0409 11:00:35.438725 22733 exec.cpp:162] Version: 1.5.0
> I0409 11:00:35.440770 22734 exec.cpp:236] Executor registered on agent
> 0262246c-14a3-4408-9b74-5e3b65dc1344-S0
> I0409 11:00:35.441388 22733 executor.cpp:171] Received SUBSCRIBED event
> I0409 11:00:35.441586 22733 executor.cpp:175] Subscribed executor on
> tiboun-Dell-Precision-M3800
> I0409 11:00:35.441643 22733 executor.cpp:171] Received LAUNCH event
> I0409 11:00:35.441767 22733 executor.cpp:638] Starting task
> driver-20180409110035-0004
> I0409 11:00:35.445050 22733 executor.cpp:478] Running
> '/usr/libexec/mesos/mesos-containerizer launch '
> I0409 11:00:35.445770 22733 executor.cpp:651] Forked command at 22743
> sh: 1: Syntax error: "(" unexpected
> I0409 11:00:35.538661 22736 executor.cpp:938] Command exited with status 2
> (pid: 22743)
> I0409 11:00:36.541016 22739 process.cpp:887] Failed to accept socket: future
> discarded
> {code}
> If you remove the parentheses, you get the following result:
>
> {code:java}
> I0409 11:03:02.023701 23085 fetcher.cpp:551] Fetcher Info:
> {"cache_directory":"\/tmp\/mesos\/fetch\/tiboun","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"\/home\/tiboun\/tools\/spark\/examples\/jars\/spark-examples_2.11-2.3.0.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/0262246c-14a3-4408-9b74-5e3b65dc1344-S0\/frameworks\/edff1a6f-38c6-46e0-a3c1-62a8fbfc2b5d-0014\/executors\/driver-20180409110301-0006\/runs\/f887c0ab-b48f-4382-850c-383c1c944269","user":"tiboun"}
> I0409 11:03:02.028268 23085 fetcher.cpp:450] Fetching URI
> '/home/tiboun/tools/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
> I0409 11:03:02.028302 23085 fetcher.cpp:291] Fetching directly into the
> sandbox directory
> I0409 11:03:02.028336 23085 fetcher.cpp:225] Fetching URI
> '/home
[jira] [Assigned] (SPARK-22885) ML test for StructuredStreaming: spark.ml.tuning
[ https://issues.apache.org/jira/browse/SPARK-22885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-22885: - Assignee: Weichen Xu > ML test for StructuredStreaming: spark.ml.tuning > > > Key: SPARK-22885 > URL: https://issues.apache.org/jira/browse/SPARK-22885 > Project: Spark > Issue Type: Test > Components: ML, Tests >Affects Versions: 2.3.0 >Reporter: Joseph K. Bradley >Assignee: Weichen Xu >Priority: Major > > Task for adding Structured Streaming tests for all Models/Transformers in a > sub-module in spark.ml > For an example, see LinearRegressionSuite.scala in > https://github.com/apache/spark/pull/19843 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22885) ML test for StructuredStreaming: spark.ml.tuning
[ https://issues.apache.org/jira/browse/SPARK-22885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-22885: -- Shepherd: Joseph K. Bradley > ML test for StructuredStreaming: spark.ml.tuning > > > Key: SPARK-22885 > URL: https://issues.apache.org/jira/browse/SPARK-22885 > Project: Spark > Issue Type: Test > Components: ML, Tests >Affects Versions: 2.3.0 >Reporter: Joseph K. Bradley >Priority: Major > > Task for adding Structured Streaming tests for all Models/Transformers in a > sub-module in spark.ml > For an example, see LinearRegressionSuite.scala in > https://github.com/apache/spark/pull/19843 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available
Arun Mahadevan created SPARK-24136:
--
Summary: MemoryStreamDataReader.next should skip sleeping if
record is available
Key: SPARK-24136
URL: https://issues.apache.org/jira/browse/SPARK-24136
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Arun Mahadevan
Currently the code sleeps 10ms for each invocation of the next even if the
record is available.
{code:java}
override def next(): Boolean = {
current = None
while (current.isEmpty) {
Thread.sleep(10)
current = endpoint.askSync[Option[Row]](
GetRecord(ContinuousMemoryStreamPartitionOffset(partition, currentOffset)))
}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available
[
https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-24136:
Assignee: (was: Apache Spark)
> MemoryStreamDataReader.next should skip sleeping if record is available
> ---
>
> Key: SPARK-24136
> URL: https://issues.apache.org/jira/browse/SPARK-24136
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
>Affects Versions: 2.4.0
>Reporter: Arun Mahadevan
>Priority: Minor
>
> Currently the code sleeps 10ms for each invocation of the next even if the
> record is available.
> {code:java}
> override def next(): Boolean = {
> current = None
> while (current.isEmpty) {
> Thread.sleep(10)
> current = endpoint.askSync[Option[Row]](
> GetRecord(ContinuousMemoryStreamPartitionOffset(partition, currentOffset)))
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available
[
https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-24136:
Assignee: Apache Spark
> MemoryStreamDataReader.next should skip sleeping if record is available
> ---
>
> Key: SPARK-24136
> URL: https://issues.apache.org/jira/browse/SPARK-24136
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
>Affects Versions: 2.4.0
>Reporter: Arun Mahadevan
>Assignee: Apache Spark
>Priority: Minor
>
> Currently the code sleeps 10ms for each invocation of the next even if the
> record is available.
> {code:java}
> override def next(): Boolean = {
> current = None
> while (current.isEmpty) {
> Thread.sleep(10)
> current = endpoint.askSync[Option[Row]](
> GetRecord(ContinuousMemoryStreamPartitionOffset(partition, currentOffset)))
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24136) MemoryStreamDataReader.next should skip sleeping if record is available
[
https://issues.apache.org/jira/browse/SPARK-24136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459844#comment-16459844
]
Apache Spark commented on SPARK-24136:
--
User 'arunmahadevan' has created a pull request for this issue:
https://github.com/apache/spark/pull/21207
> MemoryStreamDataReader.next should skip sleeping if record is available
> ---
>
> Key: SPARK-24136
> URL: https://issues.apache.org/jira/browse/SPARK-24136
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
>Affects Versions: 2.4.0
>Reporter: Arun Mahadevan
>Priority: Minor
>
> Currently the code sleeps 10ms for each invocation of the next even if the
> record is available.
> {code:java}
> override def next(): Boolean = {
> current = None
> while (current.isEmpty) {
> Thread.sleep(10)
> current = endpoint.askSync[Option[Row]](
> GetRecord(ContinuousMemoryStreamPartitionOffset(partition, currentOffset)))
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
Matt Cheah created SPARK-24137: -- Summary: [K8s] Mount temporary directories in emptydir volumes Key: SPARK-24137 URL: https://issues.apache.org/jira/browse/SPARK-24137 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 2.3.0 Reporter: Matt Cheah Currently the Spark local directories do not get any volumes and volume mounts, which means we're writing Spark shuffle and cache contents to the file system mounted by Docker. This can be terribly inefficient. We should use emptydir volumes for these directories instead for significant performance improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459860#comment-16459860 ] Matt Cheah commented on SPARK-24137: [~foxish] [~liyinan926] for SA. We actually did this on our prototype fork, but looks like it got lost when we implemented this stuff in mainline. https://github.com/apache-spark-on-k8s/spark/pull/522 > [K8s] Mount temporary directories in emptydir volumes > - > > Key: SPARK-24137 > URL: https://issues.apache.org/jira/browse/SPARK-24137 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Matt Cheah >Priority: Major > > Currently the Spark local directories do not get any volumes and volume > mounts, which means we're writing Spark shuffle and cache contents to the > file system mounted by Docker. This can be terribly inefficient. We should > use emptydir volumes for these directories instead for significant > performance improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459869#comment-16459869 ] Matt Cheah commented on SPARK-24137: Benchmark results on this subject are here: https://github.com/apache-spark-on-k8s/spark/pull/486#issuecomment-335635260 > [K8s] Mount temporary directories in emptydir volumes > - > > Key: SPARK-24137 > URL: https://issues.apache.org/jira/browse/SPARK-24137 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Matt Cheah >Priority: Major > > Currently the Spark local directories do not get any volumes and volume > mounts, which means we're writing Spark shuffle and cache contents to the > file system mounted by Docker. This can be terribly inefficient. We should > use emptydir volumes for these directories instead for significant > performance improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459892#comment-16459892
]
Yinan Li commented on SPARK-24135:
--
I think it's fine detecting and deleting the executor pods that failed
initialization. But I'm not sure how much this buys us because very likely the
newly requested executors will fail to be initialized, in particular if the
init-container is added by an external webhook or an initializer. The job won't
be able to proceed in case of this and effectively the bottleneck still exists.
> [K8s] Executors that fail to start up because of init-container errors are
> not retried and limit the executor pool size
> ---
>
> Key: SPARK-24135
> URL: https://issues.apache.org/jira/browse/SPARK-24135
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
> having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
> When executors fail in these ways, they are removed from the pending
> executors pool and the driver should retry requesting these executors.
> However, the driver does not handle a different class of error: when the pod
> enters the {{Init:Error}} state. This state comes up when the executor fails
> to launch because one of its init-containers fails. Spark itself doesn't
> attach any init-containers to the executors. However, custom web hooks can
> run on the cluster and attach init-containers to the executor pods.
> Additionally, pod presets can specify init containers to run on these pods.
> Therefore Spark should be handling the {{Init:Error}} cases regardless if
> Spark itself is aware of init-containers or not.
> This class of error is particularly bad because when we hit this state, the
> failed executor will never start, but it's still seen as pending by the
> executor allocator. The executor allocator won't request more rounds of
> executors because its current batch hasn't been resolved to either running or
> failed. Therefore we end up with being stuck with the number of executors
> that successfully started before the faulty one failed to start, potentially
> creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
[
https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459900#comment-16459900
]
Yinan Li commented on SPARK-24137:
--
Yeah, {{LocalDirectoryMountConfigurationStep}} was missed in the upstream PRs.
We probably should try to get it into 2.3.1.
> [K8s] Mount temporary directories in emptydir volumes
> -
>
> Key: SPARK-24137
> URL: https://issues.apache.org/jira/browse/SPARK-24137
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> Currently the Spark local directories do not get any volumes and volume
> mounts, which means we're writing Spark shuffle and cache contents to the
> file system mounted by Docker. This can be terribly inefficient. We should
> use emptydir volumes for these directories instead for significant
> performance improvements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459954#comment-16459954
]
Erik Erlandson commented on SPARK-24135:
I think it makes sense to detect these failure states. Even if they won't
resolve by requesting replacement executors, reporting the specific failure
mode in the error logs should aid in debugging. It could optionally be used as
grounds for job failure, in the case of repeating executor failures.
> [K8s] Executors that fail to start up because of init-container errors are
> not retried and limit the executor pool size
> ---
>
> Key: SPARK-24135
> URL: https://issues.apache.org/jira/browse/SPARK-24135
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
> having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
> When executors fail in these ways, they are removed from the pending
> executors pool and the driver should retry requesting these executors.
> However, the driver does not handle a different class of error: when the pod
> enters the {{Init:Error}} state. This state comes up when the executor fails
> to launch because one of its init-containers fails. Spark itself doesn't
> attach any init-containers to the executors. However, custom web hooks can
> run on the cluster and attach init-containers to the executor pods.
> Additionally, pod presets can specify init containers to run on these pods.
> Therefore Spark should be handling the {{Init:Error}} cases regardless if
> Spark itself is aware of init-containers or not.
> This class of error is particularly bad because when we hit this state, the
> failed executor will never start, but it's still seen as pending by the
> executor allocator. The executor allocator won't request more rounds of
> executors because its current batch hasn't been resolved to either running or
> failed. Therefore we end up with being stuck with the number of executors
> that successfully started before the faulty one failed to start, potentially
> creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23074) Dataframe-ified zipwithindex
[
https://issues.apache.org/jira/browse/SPARK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459979#comment-16459979
]
Reynold Xin commented on SPARK-23074:
-
Can your problem be solved by monotonically_increasing_id, rather than index?
Index is expensive.
> Dataframe-ified zipwithindex
>
>
> Key: SPARK-23074
> URL: https://issues.apache.org/jira/browse/SPARK-23074
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Ruslan Dautkhanov
>Priority: Minor
> Labels: dataframe, rdd
>
> Would be great to have a daraframe-friendly equivalent of rdd.zipWithIndex():
> {code:java}
> import org.apache.spark.sql.DataFrame
> import org.apache.spark.sql.types.{LongType, StructField, StructType}
> import org.apache.spark.sql.Row
> def dfZipWithIndex(
> df: DataFrame,
> offset: Int = 1,
> colName: String = "id",
> inFront: Boolean = true
> ) : DataFrame = {
> df.sqlContext.createDataFrame(
> df.rdd.zipWithIndex.map(ln =>
> Row.fromSeq(
> (if (inFront) Seq(ln._2 + offset) else Seq())
> ++ ln._1.toSeq ++
> (if (inFront) Seq() else Seq(ln._2 + offset))
> )
> ),
> StructType(
> (if (inFront) Array(StructField(colName,LongType,false)) else
> Array[StructField]())
> ++ df.schema.fields ++
> (if (inFront) Array[StructField]() else
> Array(StructField(colName,LongType,false)))
> )
> )
> }
> {code}
> credits:
> [https://stackoverflow.com/questions/30304810/dataframe-ified-zipwithindex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460047#comment-16460047
]
Matt Cheah commented on SPARK-24135:
_> But I'm not sure how much this buys us because very likely the newly
requested executors will fail to be initialized,_
That's entirely up to the behavior of the init container itself - there's many
reasons for one to believe that a given init container's logic can be flaky.
But it's not immediately obvious to me whether or not the init container's
failure should count towards a job failure. Job failures shouldn't be caused by
failures in the framework, and in this case, the framework has added the
init-container for these pods - in other words the user's code didn't directly
cause the job failure.
> [K8s] Executors that fail to start up because of init-container errors are
> not retried and limit the executor pool size
> ---
>
> Key: SPARK-24135
> URL: https://issues.apache.org/jira/browse/SPARK-24135
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
> having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
> When executors fail in these ways, they are removed from the pending
> executors pool and the driver should retry requesting these executors.
> However, the driver does not handle a different class of error: when the pod
> enters the {{Init:Error}} state. This state comes up when the executor fails
> to launch because one of its init-containers fails. Spark itself doesn't
> attach any init-containers to the executors. However, custom web hooks can
> run on the cluster and attach init-containers to the executor pods.
> Additionally, pod presets can specify init containers to run on these pods.
> Therefore Spark should be handling the {{Init:Error}} cases regardless if
> Spark itself is aware of init-containers or not.
> This class of error is particularly bad because when we hit this state, the
> failed executor will never start, but it's still seen as pending by the
> executor allocator. The executor allocator won't request more rounds of
> executors because its current batch hasn't been resolved to either running or
> failed. Therefore we end up with being stuck with the number of executors
> that successfully started before the faulty one failed to start, potentially
> creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460066#comment-16460066
]
Yinan Li commented on SPARK-24135:
--
I agree that we should add detection for initialization errors. But I'm not
sure if requesting new executors to replace the ones that failed initialization
is a good idea. External webhooks or initializers are typically installed by
cluster admins and there's always risks of bugs in the webhooks or initializers
that cause pods to fail initialization. In case of initializers, things are
worse as pods will not be able to get out of pending status if for whatever
reasons the controller that's handling a particular initializer is down. For
the reasons [~mcheah] mentioned above, it's not obvious if initialization
errors should count towards job failures. I think keeping track of how many
initialization errors are seen and stopping requesting new executors might be a
good idea.
> [K8s] Executors that fail to start up because of init-container errors are
> not retried and limit the executor pool size
> ---
>
> Key: SPARK-24135
> URL: https://issues.apache.org/jira/browse/SPARK-24135
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
> having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
> When executors fail in these ways, they are removed from the pending
> executors pool and the driver should retry requesting these executors.
> However, the driver does not handle a different class of error: when the pod
> enters the {{Init:Error}} state. This state comes up when the executor fails
> to launch because one of its init-containers fails. Spark itself doesn't
> attach any init-containers to the executors. However, custom web hooks can
> run on the cluster and attach init-containers to the executor pods.
> Additionally, pod presets can specify init containers to run on these pods.
> Therefore Spark should be handling the {{Init:Error}} cases regardless if
> Spark itself is aware of init-containers or not.
> This class of error is particularly bad because when we hit this state, the
> failed executor will never start, but it's still seen as pending by the
> executor allocator. The executor allocator won't request more rounds of
> executors because its current batch hasn't been resolved to either running or
> failed. Therefore we end up with being stuck with the number of executors
> that successfully started before the faulty one failed to start, potentially
> creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460066#comment-16460066
]
Yinan Li edited comment on SPARK-24135 at 5/1/18 7:53 PM:
--
I agree that we should add detection for initialization errors. But I'm not
sure if requesting new executors to replace the ones that failed initialization
is a good idea. External webhooks or initializers are typically installed by
cluster admins and there's always risks of bugs in the webhooks or initializers
that cause pods to fail initialization. In case of initializers, things are
worse as pods will not be able to get out of pending status if for whatever
reasons the controller that's handling a particular initializer is down. For
the reasons [~mcheah] mentioned above, it's not obvious if initialization
errors should count towards job failures. I think keeping track of how many
initialization errors are seen and stopping requesting new executors after
certain threshold might be a good idea.
was (Author: liyinan926):
I agree that we should add detection for initialization errors. But I'm not
sure if requesting new executors to replace the ones that failed initialization
is a good idea. External webhooks or initializers are typically installed by
cluster admins and there's always risks of bugs in the webhooks or initializers
that cause pods to fail initialization. In case of initializers, things are
worse as pods will not be able to get out of pending status if for whatever
reasons the controller that's handling a particular initializer is down. For
the reasons [~mcheah] mentioned above, it's not obvious if initialization
errors should count towards job failures. I think keeping track of how many
initialization errors are seen and stopping requesting new executors might be a
good idea.
> [K8s] Executors that fail to start up because of init-container errors are
> not retried and limit the executor pool size
> ---
>
> Key: SPARK-24135
> URL: https://issues.apache.org/jira/browse/SPARK-24135
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.3.0
>Reporter: Matt Cheah
>Priority: Major
>
> In KubernetesClusterSchedulerBackend, we detect if executors disconnect after
> having been started or if executors hit the {{ERROR}} or {{DELETED}} states.
> When executors fail in these ways, they are removed from the pending
> executors pool and the driver should retry requesting these executors.
> However, the driver does not handle a different class of error: when the pod
> enters the {{Init:Error}} state. This state comes up when the executor fails
> to launch because one of its init-containers fails. Spark itself doesn't
> attach any init-containers to the executors. However, custom web hooks can
> run on the cluster and attach init-containers to the executor pods.
> Additionally, pod presets can specify init containers to run on these pods.
> Therefore Spark should be handling the {{Init:Error}} cases regardless if
> Spark itself is aware of init-containers or not.
> This class of error is particularly bad because when we hit this state, the
> failed executor will never start, but it's still seen as pending by the
> executor allocator. The executor allocator won't request more rounds of
> executors because its current batch hasn't been resolved to either running or
> failed. Therefore we end up with being stuck with the number of executors
> that successfully started before the faulty one failed to start, potentially
> creating a fake resource bottleneck.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-6305: - Assignee: (was: Marcelo Vanzin) > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-6305) Add support for log4j 2.x to Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-6305: - Assignee: Marcelo Vanzin > Add support for log4j 2.x to Spark > -- > > Key: SPARK-6305 > URL: https://issues.apache.org/jira/browse/SPARK-6305 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Tal Sliwowicz >Assignee: Marcelo Vanzin >Priority: Minor > > log4j 2 requires replacing the slf4j binding and adding the log4j jars in the > classpath. Since there are shaded jars, it must be done during the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23074) Dataframe-ified zipwithindex
[
https://issues.apache.org/jira/browse/SPARK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460087#comment-16460087
]
Ruslan Dautkhanov commented on SPARK-23074:
---
[~rxin], monotonically_increasing_id wouldn't guarantee same physical order as
in file behind a dataframe (let's say it's a dataframe that was created with
spark-csv)? If not, then nope, we would still need dfZipWithIndex some way or
another for some of our use cases that require interpreting file rows content
based on their relative position to each other - that's why we need
zipwithindex here. Thanks.
> Dataframe-ified zipwithindex
>
>
> Key: SPARK-23074
> URL: https://issues.apache.org/jira/browse/SPARK-23074
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Ruslan Dautkhanov
>Priority: Minor
> Labels: dataframe, rdd
>
> Would be great to have a daraframe-friendly equivalent of rdd.zipWithIndex():
> {code:java}
> import org.apache.spark.sql.DataFrame
> import org.apache.spark.sql.types.{LongType, StructField, StructType}
> import org.apache.spark.sql.Row
> def dfZipWithIndex(
> df: DataFrame,
> offset: Int = 1,
> colName: String = "id",
> inFront: Boolean = true
> ) : DataFrame = {
> df.sqlContext.createDataFrame(
> df.rdd.zipWithIndex.map(ln =>
> Row.fromSeq(
> (if (inFront) Seq(ln._2 + offset) else Seq())
> ++ ln._1.toSeq ++
> (if (inFront) Seq() else Seq(ln._2 + offset))
> )
> ),
> StructType(
> (if (inFront) Array(StructField(colName,LongType,false)) else
> Array[StructField]())
> ++ df.schema.fields ++
> (if (inFront) Array[StructField]() else
> Array(StructField(colName,LongType,false)))
> )
> )
> }
> {code}
> credits:
> [https://stackoverflow.com/questions/30304810/dataframe-ified-zipwithindex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23074) Dataframe-ified zipwithindex
[
https://issues.apache.org/jira/browse/SPARK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460116#comment-16460116
]
Reynold Xin commented on SPARK-23074:
-
It should give you the same ordering - that's the "monotonically increasing"
part.
> Dataframe-ified zipwithindex
>
>
> Key: SPARK-23074
> URL: https://issues.apache.org/jira/browse/SPARK-23074
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Ruslan Dautkhanov
>Priority: Minor
> Labels: dataframe, rdd
>
> Would be great to have a daraframe-friendly equivalent of rdd.zipWithIndex():
> {code:java}
> import org.apache.spark.sql.DataFrame
> import org.apache.spark.sql.types.{LongType, StructField, StructType}
> import org.apache.spark.sql.Row
> def dfZipWithIndex(
> df: DataFrame,
> offset: Int = 1,
> colName: String = "id",
> inFront: Boolean = true
> ) : DataFrame = {
> df.sqlContext.createDataFrame(
> df.rdd.zipWithIndex.map(ln =>
> Row.fromSeq(
> (if (inFront) Seq(ln._2 + offset) else Seq())
> ++ ln._1.toSeq ++
> (if (inFront) Seq() else Seq(ln._2 + offset))
> )
> ),
> StructType(
> (if (inFront) Array(StructField(colName,LongType,false)) else
> Array[StructField]())
> ++ df.schema.fields ++
> (if (inFront) Array[StructField]() else
> Array(StructField(colName,LongType,false)))
> )
> )
> }
> {code}
> credits:
> [https://stackoverflow.com/questions/30304810/dataframe-ified-zipwithindex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23074) Dataframe-ified zipwithindex
[
https://issues.apache.org/jira/browse/SPARK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460137#comment-16460137
]
Ruslan Dautkhanov commented on SPARK-23074:
---
That's . When we have a "record" that spans multiple text lines, and it happens
that some lines are in one partitions, and rest of lines are in another
partition.. what would monotonically_increasing_id() return? It wouldn't be
consequential, right?
See [https://stackoverflow.com/a/48454000/470583] - people are creating quite
expensive workaround
{code:scala}
def zipWithIndex(df: DataFrame, offset: Long = 1, indexName: String = "index")
= {
val dfWithPartitionId = df.withColumn("partition_id",
spark_partition_id()).withColumn("inc_id", monotonically_increasing_id())
val partitionOffsets = dfWithPartitionId
.groupBy("partition_id")
.agg(count(lit(1)) as "cnt", first("inc_id") as "inc_id")
.orderBy("partition_id")
.select(sum("cnt").over(Window.orderBy("partition_id")) - col("cnt") -
col("inc_id") + lit(offset) as "cnt" )
.collect()
.map(_.getLong(0))
.toArray
dfWithPartitionId
.withColumn("partition_offset", udf((partitionId: Int) =>
partitionOffsets(partitionId), LongType)(col("partition_id")))
.withColumn(indexName, col("partition_offset") + col("inc_id"))
.drop("partition_id", "partition_offset", "inc_id")
}
{code}
Do you see an easier way to do this? Thanks!
> Dataframe-ified zipwithindex
>
>
> Key: SPARK-23074
> URL: https://issues.apache.org/jira/browse/SPARK-23074
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Ruslan Dautkhanov
>Priority: Minor
> Labels: dataframe, rdd
>
> Would be great to have a daraframe-friendly equivalent of rdd.zipWithIndex():
> {code:java}
> import org.apache.spark.sql.DataFrame
> import org.apache.spark.sql.types.{LongType, StructField, StructType}
> import org.apache.spark.sql.Row
> def dfZipWithIndex(
> df: DataFrame,
> offset: Int = 1,
> colName: String = "id",
> inFront: Boolean = true
> ) : DataFrame = {
> df.sqlContext.createDataFrame(
> df.rdd.zipWithIndex.map(ln =>
> Row.fromSeq(
> (if (inFront) Seq(ln._2 + offset) else Seq())
> ++ ln._1.toSeq ++
> (if (inFront) Seq() else Seq(ln._2 + offset))
> )
> ),
> StructType(
> (if (inFront) Array(StructField(colName,LongType,false)) else
> Array[StructField]())
> ++ df.schema.fields ++
> (if (inFront) Array[StructField]() else
> Array(StructField(colName,LongType,false)))
> )
> )
> }
> {code}
> credits:
> [https://stackoverflow.com/questions/30304810/dataframe-ified-zipwithindex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23074) Dataframe-ified zipwithindex
[
https://issues.apache.org/jira/browse/SPARK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460137#comment-16460137
]
Ruslan Dautkhanov edited comment on SPARK-23074 at 5/1/18 8:56 PM:
---
That's . When we have a "record" that spans multiple text lines, and it happens
that some lines are in one partitions, and rest of lines are in another
partition.. what would monotonically_increasing_id() return? -It wouldn't be
consequential, right?- I just fond answer to this question in the [code
|https://github.com/apache/spark/blob/7c1654e2159662e7e663ba141719d755002f770a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/MonotonicallyIncreasingID.scala#L27].
See [https://stackoverflow.com/a/48454000/470583] - people are creating quite
expensive workaround
{code}
def zipWithIndex(df: DataFrame, offset: Long = 1, indexName: String = "index")
= {
val dfWithPartitionId = df.withColumn("partition_id",
spark_partition_id()).withColumn("inc_id", monotonically_increasing_id())
val partitionOffsets = dfWithPartitionId
.groupBy("partition_id")
.agg(count(lit(1)) as "cnt", first("inc_id") as "inc_id")
.orderBy("partition_id")
.select(sum("cnt").over(Window.orderBy("partition_id")) - col("cnt") -
col("inc_id") + lit(offset) as "cnt" )
.collect()
.map(_.getLong(0))
.toArray
dfWithPartitionId
.withColumn("partition_offset", udf((partitionId: Int) =>
partitionOffsets(partitionId), LongType)(col("partition_id")))
.withColumn(indexName, col("partition_offset") + col("inc_id"))
.drop("partition_id", "partition_offset", "inc_id")
}
{code}
Do you see an easier way to do this? Thanks!
was (Author: tagar):
That's . When we have a "record" that spans multiple text lines, and it happens
that some lines are in one partitions, and rest of lines are in another
partition.. what would monotonically_increasing_id() return? It wouldn't be
consequential, right?
See [https://stackoverflow.com/a/48454000/470583] - people are creating quite
expensive workaround
{code:scala}
def zipWithIndex(df: DataFrame, offset: Long = 1, indexName: String = "index")
= {
val dfWithPartitionId = df.withColumn("partition_id",
spark_partition_id()).withColumn("inc_id", monotonically_increasing_id())
val partitionOffsets = dfWithPartitionId
.groupBy("partition_id")
.agg(count(lit(1)) as "cnt", first("inc_id") as "inc_id")
.orderBy("partition_id")
.select(sum("cnt").over(Window.orderBy("partition_id")) - col("cnt") -
col("inc_id") + lit(offset) as "cnt" )
.collect()
.map(_.getLong(0))
.toArray
dfWithPartitionId
.withColumn("partition_offset", udf((partitionId: Int) =>
partitionOffsets(partitionId), LongType)(col("partition_id")))
.withColumn(indexName, col("partition_offset") + col("inc_id"))
.drop("partition_id", "partition_offset", "inc_id")
}
{code}
Do you see an easier way to do this? Thanks!
> Dataframe-ified zipwithindex
>
>
> Key: SPARK-23074
> URL: https://issues.apache.org/jira/browse/SPARK-23074
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Ruslan Dautkhanov
>Priority: Minor
> Labels: dataframe, rdd
>
> Would be great to have a daraframe-friendly equivalent of rdd.zipWithIndex():
> {code:java}
> import org.apache.spark.sql.DataFrame
> import org.apache.spark.sql.types.{LongType, StructField, StructType}
> import org.apache.spark.sql.Row
> def dfZipWithIndex(
> df: DataFrame,
> offset: Int = 1,
> colName: String = "id",
> inFront: Boolean = true
> ) : DataFrame = {
> df.sqlContext.createDataFrame(
> df.rdd.zipWithIndex.map(ln =>
> Row.fromSeq(
> (if (inFront) Seq(ln._2 + offset) else Seq())
> ++ ln._1.toSeq ++
> (if (inFront) Seq() else Seq(ln._2 + offset))
> )
> ),
> StructType(
> (if (inFront) Array(StructField(colName,LongType,false)) else
> Array[StructField]())
> ++ df.schema.fields ++
> (if (inFront) Array[StructField]() else
> Array(StructField(colName,LongType,false)))
> )
> )
> }
> {code}
> credits:
> [https://stackoverflow.com/questions/30304810/dataframe-ified-zipwithindex]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation
[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460164#comment-16460164 ] Joseph K. Bradley commented on SPARK-23686: --- [~yogeshgarg] made the good point that we should not convert all uses of Logging to use Instrumentation: if logging happens on executors, then we should not use the (non-serializable) Instrumentation class. E.g.: https://github.com/apache/spark/blob/6782359a04356e4cde32940861bf2410ef37f445/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala#L1587 Also, these instances all seem to be at the DEBUG level, which is not really useful for users. > Make better usage of org.apache.spark.ml.util.Instrumentation > - > > Key: SPARK-23686 > URL: https://issues.apache.org/jira/browse/SPARK-23686 > Project: Spark > Issue Type: Improvement > Components: ML >Affects Versions: 2.3.0 >Reporter: Bago Amirbekian >Priority: Major > > This Jira is a bit high level and might require subtasks or other jiras for > more specific tasks. > I've noticed that we don't make the best usage of the instrumentation class. > Specifically sometimes we bypass the instrumentation class and use the > debugger instead. For example, > [https://github.com/apache/spark/blob/9b9827759af2ca3eea146a6032f9165f640ce152/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala#L143] > Also there are some things that might be useful to log in the instrumentation > class that we currently don't. For example: > number of training examples > mean/var of label (regression) > I know computing these things can be expensive in some cases, but especially > when this data is already available we can log it for free. For example, > Logistic Regression Summarizer computes some useful data including numRows > that we don't log. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15750) Constructing FPGrowth fails when no numPartitions specified in pyspark
[
https://issues.apache.org/jira/browse/SPARK-15750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph K. Bradley updated SPARK-15750:
--
Shepherd: Joseph K. Bradley
> Constructing FPGrowth fails when no numPartitions specified in pyspark
> --
>
> Key: SPARK-15750
> URL: https://issues.apache.org/jira/browse/SPARK-15750
> Project: Spark
> Issue Type: Bug
> Components: MLlib, PySpark
>Affects Versions: 2.0.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Major
>
> {code}
> >>> model1 = FPGrowth.train(rdd, 0.6)
> Traceback (most recent call last):
> File "", line 1, in
> File "/Users/jzhang/github/spark-2/python/pyspark/mllib/fpm.py", line 96,
> in train
> model = callMLlibFunc("trainFPGrowthModel", data, float(minSupport),
> int(numPartitions))
> File "/Users/jzhang/github/spark-2/python/pyspark/mllib/common.py", line
> 130, in callMLlibFunc
> return callJavaFunc(sc, api, *args)
> File "/Users/jzhang/github/spark-2/python/pyspark/mllib/common.py", line
> 123, in callJavaFunc
> return _java2py(sc, func(*args))
> File
> "/Users/jzhang/github/spark-2/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
> line 933, in __call__
> File "/Users/jzhang/github/spark-2/python/pyspark/sql/utils.py", line 79,
> in deco
> raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
> pyspark.sql.utils.IllegalArgumentException: u'requirement failed: Number of
> partitions must be positive but got -1'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-15750) Constructing FPGrowth fails when no numPartitions specified in pyspark
[
https://issues.apache.org/jira/browse/SPARK-15750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph K. Bradley reassigned SPARK-15750:
-
Assignee: Jeff Zhang
> Constructing FPGrowth fails when no numPartitions specified in pyspark
> --
>
> Key: SPARK-15750
> URL: https://issues.apache.org/jira/browse/SPARK-15750
> Project: Spark
> Issue Type: Bug
> Components: MLlib, PySpark
>Affects Versions: 2.0.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Major
>
> {code}
> >>> model1 = FPGrowth.train(rdd, 0.6)
> Traceback (most recent call last):
> File "", line 1, in
> File "/Users/jzhang/github/spark-2/python/pyspark/mllib/fpm.py", line 96,
> in train
> model = callMLlibFunc("trainFPGrowthModel", data, float(minSupport),
> int(numPartitions))
> File "/Users/jzhang/github/spark-2/python/pyspark/mllib/common.py", line
> 130, in callMLlibFunc
> return callJavaFunc(sc, api, *args)
> File "/Users/jzhang/github/spark-2/python/pyspark/mllib/common.py", line
> 123, in callJavaFunc
> return _java2py(sc, func(*args))
> File
> "/Users/jzhang/github/spark-2/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py",
> line 933, in __call__
> File "/Users/jzhang/github/spark-2/python/pyspark/sql/utils.py", line 79,
> in deco
> raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
> pyspark.sql.utils.IllegalArgumentException: u'requirement failed: Number of
> partitions must be positive but got -1'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23925) High-order function: repeat(element, count) → array
[ https://issues.apache.org/jira/browse/SPARK-23925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23925: Assignee: (was: Apache Spark) > High-order function: repeat(element, count) → array > --- > > Key: SPARK-23925 > URL: https://issues.apache.org/jira/browse/SPARK-23925 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Xiao Li >Priority: Major > > Ref: https://prestodb.io/docs/current/functions/array.html > Repeat element for count times. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23925) High-order function: repeat(element, count) → array
[ https://issues.apache.org/jira/browse/SPARK-23925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460174#comment-16460174 ] Apache Spark commented on SPARK-23925: -- User 'pepinoflo' has created a pull request for this issue: https://github.com/apache/spark/pull/21208 > High-order function: repeat(element, count) → array > --- > > Key: SPARK-23925 > URL: https://issues.apache.org/jira/browse/SPARK-23925 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Xiao Li >Priority: Major > > Ref: https://prestodb.io/docs/current/functions/array.html > Repeat element for count times. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23925) High-order function: repeat(element, count) → array
[ https://issues.apache.org/jira/browse/SPARK-23925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23925: Assignee: Apache Spark > High-order function: repeat(element, count) → array > --- > > Key: SPARK-23925 > URL: https://issues.apache.org/jira/browse/SPARK-23925 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Xiao Li >Assignee: Apache Spark >Priority: Major > > Ref: https://prestodb.io/docs/current/functions/array.html > Repeat element for count times. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation
[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460164#comment-16460164 ] Joseph K. Bradley edited comment on SPARK-23686 at 5/1/18 9:52 PM: --- [~yogeshgarg] and [~WeichenXu123] made the good point that some logging occurs on executors. This brings up the question: * Should we use Instrumentation on executors? * What levels of logging should we use on executors (in MLlib algorithms)? I figure it's safe to assume that executor logs should be more for developers than for users. (Current use in MLlib seems like this, e.g., for training of trees in https://github.com/apache/spark/pull/21163 ) These all seem to be at the DEBUG level, which is not really useful for users. Given that, I recommend: * We leave Instrumentation non-Serializable to avoid use on executors * We use regular Logging on executors. Developers who are debugging algorithms will presumably be running pretty isolated tests anyways. was (Author: josephkb): [~yogeshgarg] made the good point that we should not convert all uses of Logging to use Instrumentation: if logging happens on executors, then we should not use the (non-serializable) Instrumentation class. E.g.: https://github.com/apache/spark/blob/6782359a04356e4cde32940861bf2410ef37f445/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala#L1587 Also, these instances all seem to be at the DEBUG level, which is not really useful for users. > Make better usage of org.apache.spark.ml.util.Instrumentation > - > > Key: SPARK-23686 > URL: https://issues.apache.org/jira/browse/SPARK-23686 > Project: Spark > Issue Type: Improvement > Components: ML >Affects Versions: 2.3.0 >Reporter: Bago Amirbekian >Priority: Major > > This Jira is a bit high level and might require subtasks or other jiras for > more specific tasks. > I've noticed that we don't make the best usage of the instrumentation class. > Specifically sometimes we bypass the instrumentation class and use the > debugger instead. For example, > [https://github.com/apache/spark/blob/9b9827759af2ca3eea146a6032f9165f640ce152/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala#L143] > Also there are some things that might be useful to log in the instrumentation > class that we currently don't. For example: > number of training examples > mean/var of label (regression) > I know computing these things can be expensive in some cases, but especially > when this data is already available we can log it for free. For example, > Logistic Regression Summarizer computes some useful data including numRows > that we don't log. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23798) The CreateArray and ConcatArray should return the default array type when no children provided
[ https://issues.apache.org/jira/browse/SPARK-23798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460213#comment-16460213 ] Sunitha Kambhampati commented on SPARK-23798: - ArrayType.defaultConcreteType is ArrayType(NullType, containsNull = true) But this pr [https://github.com/apache/spark/pull/18516] made changes to have CreateArray return the default ArrayType of elementType to be StringType. cc [~maropu] > The CreateArray and ConcatArray should return the default array type when no > children provided > -- > > Key: SPARK-23798 > URL: https://issues.apache.org/jira/browse/SPARK-23798 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.0 >Reporter: Marek Novotny >Priority: Minor > > The expressions should return ArrayType.defaultConcreteType. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases
[
https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460239#comment-16460239
]
Aaron S. Hawley commented on SPARK-14146:
-
Should be resolved in Scala after this was merged:
[https://github.com/scala/scala/pull/5084]
That fix was released in Scala 2.11.11
[https://github.com/scala/scala/releases/tag/v2.11.11]
> Imported implicits can't be found in Spark REPL in some cases
> -
>
> Key: SPARK-14146
> URL: https://issues.apache.org/jira/browse/SPARK-14146
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.0.0
>Reporter: Wenchen Fan
>Priority: Major
>
> {code}
> class I(i: Int) {
> def double: Int = i * 2
> }
> class Context {
> implicit def toI(i: Int): I = new I(i)
> }
> val c = new Context
> import c._
> // OK
> 1.double
> // Fail
> class A; 1.double
> {code}
> The above code snippets can work in Scala REPL however.
> This will affect our Dataset functionality, for example:
> {code}
> class A; Seq(1 -> "a").toDS() // fail
> {code}
> or in paste mode:
> {code}
> :paste
> class A
> Seq(1 -> "a").toDS() // fail
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24138) Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite
Xiao Li created SPARK-24138: --- Summary: Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite Key: SPARK-24138 URL: https://issues.apache.org/jira/browse/SPARK-24138 Project: Spark Issue Type: Bug Components: SQL, Tests Affects Versions: 2.3.0 Reporter: Xiao Li Assignee: Dongjoon Hyun Keep hitting the flaky tests org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.6/4455/testReport/junit/org.apache.spark.sql.execution.datasources.orc/OrcQuerySuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24138) Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite
[ https://issues.apache.org/jira/browse/SPARK-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24138: Shepherd: Wenchen Fan > Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > > > Key: SPARK-24138 > URL: https://issues.apache.org/jira/browse/SPARK-24138 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 2.3.0 >Reporter: Xiao Li >Assignee: Dongjoon Hyun >Priority: Major > > Keep hitting the flaky tests > org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.6/4455/testReport/junit/org.apache.spark.sql.execution.datasources.orc/OrcQuerySuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-23197) Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
[ https://issues.apache.org/jira/browse/SPARK-23197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-23197: - > Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle" > --- > > Key: SPARK-23197 > URL: https://issues.apache.org/jira/browse/SPARK-23197 > Project: Spark > Issue Type: Test > Components: DStreams >Affects Versions: 2.3.0 >Reporter: Tathagata Das >Assignee: Tathagata Das >Priority: Major > Fix For: 2.3.0, 3.0.0 > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/109/testReport/junit/org.apache.spark.streaming/ReceiverSuite/receiver_life_cycle/history/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23197) Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle"
[ https://issues.apache.org/jira/browse/SPARK-23197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460275#comment-16460275 ] Xiao Li commented on SPARK-23197: - [~tdas] We are still hitting the same issue in the latest build tests. https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.7/4405/testReport/junit/org.apache.spark.streaming/ReceiverSuite/receiver_life_cycle/ > Flaky test: spark.streaming.ReceiverSuite."receiver_life_cycle" > --- > > Key: SPARK-23197 > URL: https://issues.apache.org/jira/browse/SPARK-23197 > Project: Spark > Issue Type: Test > Components: DStreams >Affects Versions: 2.3.0 >Reporter: Tathagata Das >Assignee: Tathagata Das >Priority: Major > Fix For: 2.3.0, 3.0.0 > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/109/testReport/junit/org.apache.spark.streaming/ReceiverSuite/receiver_life_cycle/history/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24138) Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite
[ https://issues.apache.org/jira/browse/SPARK-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460266#comment-16460266 ] Xiao Li commented on SPARK-24138: - cc [~dongjoon] Could you take a look at this? > Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > > > Key: SPARK-24138 > URL: https://issues.apache.org/jira/browse/SPARK-24138 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 2.3.0 >Reporter: Xiao Li >Assignee: Dongjoon Hyun >Priority: Major > > Keep hitting the flaky tests > org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.6/4455/testReport/junit/org.apache.spark.sql.execution.datasources.orc/OrcQuerySuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23798) The CreateArray and ConcatArray should return the default array type when no children provided
[ https://issues.apache.org/jira/browse/SPARK-23798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460329#comment-16460329 ] Takeshi Yamamuro commented on SPARK-23798: -- That pr was to make the behaviour along with the hive one. WDYT [~smilegator] ? > The CreateArray and ConcatArray should return the default array type when no > children provided > -- > > Key: SPARK-23798 > URL: https://issues.apache.org/jira/browse/SPARK-23798 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.0 >Reporter: Marek Novotny >Priority: Minor > > The expressions should return ArrayType.defaultConcreteType. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation
[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460164#comment-16460164 ] Joseph K. Bradley edited comment on SPARK-23686 at 5/2/18 12:21 AM: [~yogeshgarg] and [~WeichenXu123] made the good point that some logging occurs on executors. This brings up the question: * Should we use Instrumentation on executors? * What levels of logging should we use on executors (in MLlib algorithms)? I figure it's safe to assume that executor logs should be more for developers than for users. (Current use in MLlib seems like this, e.g., for training of trees in https://github.com/apache/spark/pull/21163 ) These all seem to be at the DEBUG level, which is not really useful for users. (UPDATED BELOW) Since it'd be handy to have prefixes on executor logs too (to link them with Estimators), let's use Instrumentation on executors. was (Author: josephkb): [~yogeshgarg] and [~WeichenXu123] made the good point that some logging occurs on executors. This brings up the question: * Should we use Instrumentation on executors? * What levels of logging should we use on executors (in MLlib algorithms)? I figure it's safe to assume that executor logs should be more for developers than for users. (Current use in MLlib seems like this, e.g., for training of trees in https://github.com/apache/spark/pull/21163 ) These all seem to be at the DEBUG level, which is not really useful for users. Given that, I recommend: * We leave Instrumentation non-Serializable to avoid use on executors * We use regular Logging on executors. Developers who are debugging algorithms will presumably be running pretty isolated tests anyways. > Make better usage of org.apache.spark.ml.util.Instrumentation > - > > Key: SPARK-23686 > URL: https://issues.apache.org/jira/browse/SPARK-23686 > Project: Spark > Issue Type: Improvement > Components: ML >Affects Versions: 2.3.0 >Reporter: Bago Amirbekian >Priority: Major > > This Jira is a bit high level and might require subtasks or other jiras for > more specific tasks. > I've noticed that we don't make the best usage of the instrumentation class. > Specifically sometimes we bypass the instrumentation class and use the > debugger instead. For example, > [https://github.com/apache/spark/blob/9b9827759af2ca3eea146a6032f9165f640ce152/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala#L143] > Also there are some things that might be useful to log in the instrumentation > class that we currently don't. For example: > number of training examples > mean/var of label (regression) > I know computing these things can be expensive in some cases, but especially > when this data is already available we can log it for free. For example, > Logistic Regression Summarizer computes some useful data including numRows > that we don't log. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24138) Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite
[ https://issues.apache.org/jira/browse/SPARK-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460335#comment-16460335 ] Dongjoon Hyun commented on SPARK-24138: --- I see. Sure. [~smilegator]. Actually, we observed this situation in our three test suites for both Parquet and ORC. - SPARK-23458 (ORC) - SPARK-23505 (Parquet) - SPARK-23390 (FileBasedDataSource for both ORC/Parquet) We reduced the number of points of failures with SPARK-23457 and SPARK-23399. But, for now, it seems that we need to reinvestigate this. For now, I'll merge the failure report to the existing one. > Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > > > Key: SPARK-24138 > URL: https://issues.apache.org/jira/browse/SPARK-24138 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 2.3.0 >Reporter: Xiao Li >Assignee: Dongjoon Hyun >Priority: Major > > Keep hitting the flaky tests > org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.6/4455/testReport/junit/org.apache.spark.sql.execution.datasources.orc/OrcQuerySuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24138) Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite
[ https://issues.apache.org/jira/browse/SPARK-24138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-24138. --- Resolution: Duplicate > Flaky test: org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > > > Key: SPARK-24138 > URL: https://issues.apache.org/jira/browse/SPARK-24138 > Project: Spark > Issue Type: Bug > Components: SQL, Tests >Affects Versions: 2.3.0 >Reporter: Xiao Li >Assignee: Dongjoon Hyun >Priority: Major > > Keep hitting the flaky tests > org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-sbt-hadoop-2.6/4455/testReport/junit/org.apache.spark.sql.execution.datasources.orc/OrcQuerySuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23458) Flaky test: OrcQuerySuite
[
https://issues.apache.org/jira/browse/SPARK-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23458:
--
Description:
Sometimes we have UT failures with the following stacktrace:
{code:java}
sbt.ForkMain$ForkError:
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
eventually never returned normally. Attempted 15 times over 10.01396221801
seconds. Last failure message: There are 1 possibly leaked file streams..
at
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:421)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:439)
at
org.apache.spark.sql.execution.datasources.orc.OrcTest.eventually(OrcTest.scala:45)
at
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:308)
at
org.apache.spark.sql.execution.datasources.orc.OrcTest.eventually(OrcTest.scala:45)
at
org.apache.spark.sql.test.SharedSparkSession$class.afterEach(SharedSparkSession.scala:114)
at
org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite.afterEach(OrcQuerySuite.scala:583)
at
org.scalatest.BeforeAndAfterEach$$anonfun$1.apply$mcV$sp(BeforeAndAfterEach.scala:234)
at
org.scalatest.Status$$anonfun$withAfterEffect$1.apply(Status.scala:379)
at
org.scalatest.Status$$anonfun$withAfterEffect$1.apply(Status.scala:375)
at org.scalatest.SucceededStatus$.whenCompleted(Status.scala:454)
at org.scalatest.Status$class.withAfterEffect(Status.scala:375)
at org.scalatest.SucceededStatus$.withAfterEffect(Status.scala:426)
at
org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:232)
at
org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite.runTest(OrcQuerySuite.scala:583)
at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
at org.scalatest.Suite$class.run(Suite.scala:1147)
at
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
at
org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:52)
at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: sbt.ForkMain$ForkError: java.lang.IllegalStateException: There are 1
possibly leaked file streams.
at
org.apache.spark.DebugFilesystem$.assertNoOpenStreams(DebugFilesystem.scala:54)
at
org.apache.spark.sql.test.SharedSparkSession$$anonfun$afterEach$1.apply$mcV$sp(SharedSparkSession.scala:115)
at
org.apache.spark.sql.test.SharedSparkSession$$anonfun$afterEach$1.apply(SharedSparkSession.scala:115)
at
org.apache.spark.sql.test.SharedSparkSession$$anonfun$afterEach$1.apply(SharedSparkSession.scala:115)
at
org.scalatest.concurrent.Eventually$class.makeAValiantAttempt$1(Eventually.scala:395)
at
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:409)
... 42 more
Caused by: sbt.ForkMain$ForkError: java.lang.Throwable: null
at
org.apache.spark.DebugFilesystem$.addOpenStre
[jira] [Commented] (SPARK-23458) Flaky test: OrcQuerySuite
[
https://issues.apache.org/jira/browse/SPARK-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460336#comment-16460336
]
Dongjoon Hyun commented on SPARK-23458:
---
cc [~smilegator]
> Flaky test: OrcQuerySuite
> --
>
> Key: SPARK-23458
> URL: https://issues.apache.org/jira/browse/SPARK-23458
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.4.0
> Environment: AMPLab Jenkins
>Reporter: Marco Gaido
>Priority: Major
>
> Sometimes we have UT failures with the following stacktrace:
> {code:java}
> sbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
> eventually never returned normally. Attempted 15 times over
> 10.01396221801 seconds. Last failure message: There are 1 possibly leaked
> file streams..
> at
> org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:421)
> at
> org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:439)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcTest.eventually(OrcTest.scala:45)
> at
> org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:308)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcTest.eventually(OrcTest.scala:45)
> at
> org.apache.spark.sql.test.SharedSparkSession$class.afterEach(SharedSparkSession.scala:114)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite.afterEach(OrcQuerySuite.scala:583)
> at
> org.scalatest.BeforeAndAfterEach$$anonfun$1.apply$mcV$sp(BeforeAndAfterEach.scala:234)
> at
> org.scalatest.Status$$anonfun$withAfterEffect$1.apply(Status.scala:379)
> at
> org.scalatest.Status$$anonfun$withAfterEffect$1.apply(Status.scala:375)
> at org.scalatest.SucceededStatus$.whenCompleted(Status.scala:454)
> at org.scalatest.Status$class.withAfterEffect(Status.scala:375)
> at org.scalatest.SucceededStatus$.withAfterEffect(Status.scala:426)
> at
> org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:232)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite.runTest(OrcQuerySuite.scala:583)
> at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite$class.run(Suite.scala:1147)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:52)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.lang.IllegalStateException: There are
> 1 possibly leaked file streams.
> at
> org.apache.spark.DebugFilesystem$.assertNoOpenStreams(DebugFilesystem.scala:54)
> at
> org.apache.spark.sql.test.SharedSparkSession$$anonfun$afterEach$1.apply$mcV$sp(SharedSparkSession.scala:115)
> at
> org.apache.spark
[jira] [Comment Edited] (SPARK-23458) Flaky test: OrcQuerySuite
[
https://issues.apache.org/jira/browse/SPARK-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460336#comment-16460336
]
Dongjoon Hyun edited comment on SPARK-23458 at 5/2/18 12:36 AM:
We reduced the number of points of failures by SPARK-23457 and SPARK-23399.
But, these are still reported. I'll reinvestigate the followings together.
- SPARK-23458 (ORC)
- SPARK-23505 (Parquet)
- SPARK-23390 (FileBasedDataSource for both ORC/Parquet)
BTW, these are hidden by another several flaky test failures; Hive test suites,
DirectKakfaStreamSuite, ReceiverSuite, DateTimeUtilsSuite.monthsBetween. We had
better resolve all of these before 2.3.1 if possible.
cc [~smilegator]
was (Author: dongjoon):
cc [~smilegator]
> Flaky test: OrcQuerySuite
> --
>
> Key: SPARK-23458
> URL: https://issues.apache.org/jira/browse/SPARK-23458
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.4.0
> Environment: AMPLab Jenkins
>Reporter: Marco Gaido
>Priority: Major
>
> Sometimes we have UT failures with the following stacktrace:
> {code:java}
> sbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
> eventually never returned normally. Attempted 15 times over
> 10.01396221801 seconds. Last failure message: There are 1 possibly leaked
> file streams..
> at
> org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:421)
> at
> org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:439)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcTest.eventually(OrcTest.scala:45)
> at
> org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:308)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcTest.eventually(OrcTest.scala:45)
> at
> org.apache.spark.sql.test.SharedSparkSession$class.afterEach(SharedSparkSession.scala:114)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite.afterEach(OrcQuerySuite.scala:583)
> at
> org.scalatest.BeforeAndAfterEach$$anonfun$1.apply$mcV$sp(BeforeAndAfterEach.scala:234)
> at
> org.scalatest.Status$$anonfun$withAfterEffect$1.apply(Status.scala:379)
> at
> org.scalatest.Status$$anonfun$withAfterEffect$1.apply(Status.scala:375)
> at org.scalatest.SucceededStatus$.whenCompleted(Status.scala:454)
> at org.scalatest.Status$class.withAfterEffect(Status.scala:375)
> at org.scalatest.SucceededStatus$.withAfterEffect(Status.scala:426)
> at
> org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:232)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite.runTest(OrcQuerySuite.scala:583)
> at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> at
> org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite$class.run(Suite.scala:1147)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> at
> org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:52)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent
[jira] [Created] (SPARK-24139) Fix Flaky Tests
Dongjoon Hyun created SPARK-24139: - Summary: Fix Flaky Tests Key: SPARK-24139 URL: https://issues.apache.org/jira/browse/SPARK-24139 Project: Spark Issue Type: Umbrella Components: Tests Affects Versions: 2.4.0 Reporter: Dongjoon Hyun As of today, master and branch-2.3 branches suffer from several flaky tests. This umbrella issue aims to give better visibility for the flaky tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24140) Flaky test: KMeansClusterSuite.task size should be small in both training and prediction
Dongjoon Hyun created SPARK-24140:
-
Summary: Flaky test: KMeansClusterSuite.task size should be small
in both training and prediction
Key: SPARK-24140
URL: https://issues.apache.org/jira/browse/SPARK-24140
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 2.4.0
Reporter: Dongjoon Hyun
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/4765/
{code}
Error Message
Job 0 cancelled because SparkContext was shut down
Stacktrace
org.apache.spark.SparkException: Job 0 cancelled because SparkContext was
shut down
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:837)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:835)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at
org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:835)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1841)
at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1754)
at
org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1927)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1303)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1926)
at
org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:574)
at
org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1907)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at
org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at
org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2030)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2051)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2070)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2095)
at org.apache.spark.rdd.RDD.count(RDD.scala:1162)
at org.apache.spark.rdd.RDD$$anonfun$takeSample$1.apply(RDD.scala:571)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.takeSample(RDD.scala:560)
at org.apache.spark.mllib.clustering.KMeans.initRandom(KMeans.scala:360)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24139) Fix Flaky Tests
[ https://issues.apache.org/jira/browse/SPARK-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24139: -- Description: As of today, master and branch-2.3 branches suffer from several flaky tests. This umbrella issue aims to give better visibility for the recent flaky tests. was: As of today, master and branch-2.3 branches suffer from several flaky tests. This umbrella issue aims to give better visibility for the flaky tests. > Fix Flaky Tests > --- > > Key: SPARK-24139 > URL: https://issues.apache.org/jira/browse/SPARK-24139 > Project: Spark > Issue Type: Umbrella > Components: Tests >Affects Versions: 2.4.0 >Reporter: Dongjoon Hyun >Priority: Major > > As of today, master and branch-2.3 branches suffer from several flaky tests. > This umbrella issue aims to give better visibility for the recent flaky tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23489:
--
Description:
I saw this error in an unrelated PR. It seems to me a bad configuration in the
Jenkins node where the tests are run.
{code}
Error Message
java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
"/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
Stacktrace
sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
"./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2, No
such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at
org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
at scala.collection.immutable.List.foreach(List.scala:381)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more
{code}
This is the link:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
was:
I saw this error in an unrelated PR. It seems to me a bad configuration in the
Jenkins node where the tests are run.
{code}
Error Message
java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
"/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
Stacktrace
sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
"./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2, No
such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at
org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
at scala.collection.immutable.List.foreach(List.scala:381)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
jav
[jira] [Updated] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky
[
https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23415:
--
Description:
The test suite fails due to 60-second timeout sometimes.
{code}
Error Message
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
failAfter did not complete within 60 seconds.
Stacktrace
sbt.ForkMain$ForkError:
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
failAfter did not complete within 60 seconds.
{code}
- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87380/
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/4206/
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/4759/
was:
The test suite fails due to 60-second timeout sometimes.
{code}
Error Message
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
failAfter did not complete within 60 seconds.
Stacktrace
sbt.ForkMain$ForkError:
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
failAfter did not complete within 60 seconds.
{code}
- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87380/
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/4206/
> BufferHolderSparkSubmitSuite is flaky
> -
>
> Key: SPARK-23415
> URL: https://issues.apache.org/jira/browse/SPARK-23415
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> The test suite fails due to 60-second timeout sometimes.
> {code}
> Error Message
> org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
> failAfter did not complete within 60 seconds.
> Stacktrace
> sbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to
> failAfter did not complete within 60 seconds.
> {code}
> - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87380/
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/4206/
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/4759/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23489:
--
Description:
I saw this error in an unrelated PR. It seems to me a bad configuration in the
Jenkins node where the tests are run.
{code}
Error Message
java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
"/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
Stacktrace
sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
"./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2, No
such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at
org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
at scala.collection.immutable.List.foreach(List.scala:381)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more
{code}
This is the link:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
*NOTE: This failure frequently looks as `Test Result (no failures)`*
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
was:
I saw this error in an unrelated PR. It seems to me a bad configuration in the
Jenkins node where the tests are run.
{code}
Error Message
java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
"/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
Stacktrace
sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
"./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2, No
such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at
org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
at scala.collection.immutable.List.foreach(List.scala:381)
at
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call
[jira] [Updated] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23489:
--
Issue Type: Bug (was: Task)
> HiveExternalCatalogVersionsSuite flaky test
> ---
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: Tests
>Affects Versions: 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24141) Fix bug in CoarseGrainedSchedulerBackend.killExecutors
wuyi created SPARK-24141: Summary: Fix bug in CoarseGrainedSchedulerBackend.killExecutors Key: SPARK-24141 URL: https://issues.apache.org/jira/browse/SPARK-24141 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.3.0 Reporter: wuyi Fix For: 2.3.0 In method _CoarseGrainedSchedulerBackend.killExecutors()_, *numPendingExecutors* should add *executorsToKill.size* rather than *knownExecutors.size* if we do not adjust target number of executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24141) Fix bug in CoarseGrainedSchedulerBackend.killExecutors
[ https://issues.apache.org/jira/browse/SPARK-24141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460404#comment-16460404 ] Apache Spark commented on SPARK-24141: -- User 'Ngone51' has created a pull request for this issue: https://github.com/apache/spark/pull/21209 > Fix bug in CoarseGrainedSchedulerBackend.killExecutors > -- > > Key: SPARK-24141 > URL: https://issues.apache.org/jira/browse/SPARK-24141 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: wuyi >Priority: Critical > Fix For: 2.3.0 > > > In method _CoarseGrainedSchedulerBackend.killExecutors()_, > *numPendingExecutors* should add > *executorsToKill.size* rather than *knownExecutors.size* if we do not adjust > target number of executors. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24141) Fix bug in CoarseGrainedSchedulerBackend.killExecutors
[ https://issues.apache.org/jira/browse/SPARK-24141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24141: Assignee: (was: Apache Spark) > Fix bug in CoarseGrainedSchedulerBackend.killExecutors > -- > > Key: SPARK-24141 > URL: https://issues.apache.org/jira/browse/SPARK-24141 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: wuyi >Priority: Critical > Fix For: 2.3.0 > > > In method _CoarseGrainedSchedulerBackend.killExecutors()_, > *numPendingExecutors* should add > *executorsToKill.size* rather than *knownExecutors.size* if we do not adjust > target number of executors. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24141) Fix bug in CoarseGrainedSchedulerBackend.killExecutors
[ https://issues.apache.org/jira/browse/SPARK-24141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24141: Assignee: Apache Spark > Fix bug in CoarseGrainedSchedulerBackend.killExecutors > -- > > Key: SPARK-24141 > URL: https://issues.apache.org/jira/browse/SPARK-24141 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: wuyi >Assignee: Apache Spark >Priority: Critical > Fix For: 2.3.0 > > > In method _CoarseGrainedSchedulerBackend.killExecutors()_, > *numPendingExecutors* should add > *executorsToKill.size* rather than *knownExecutors.size* if we do not adjust > target number of executors. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23489:
--
Component/s: SQL
> HiveExternalCatalogVersionsSuite flaky test
> ---
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24131) Add majorMinorVersion API to PySpark for determining Spark versions
[
https://issues.apache.org/jira/browse/SPARK-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-24131:
Assignee: Liang-Chi Hsieh
> Add majorMinorVersion API to PySpark for determining Spark versions
> ---
>
> Key: SPARK-24131
> URL: https://issues.apache.org/jira/browse/SPARK-24131
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 2.4.0
>Reporter: Liang-Chi Hsieh
>Assignee: Liang-Chi Hsieh
>Priority: Minor
>
> We need to determine Spark major and minor versions in PySpark. We can add a
> {{majorMinorVersion}} API to PySpark which is similar to the API in
> {{VersionUtils.majorMinorVersion}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24131) Add majorMinorVersion API to PySpark for determining Spark versions
[
https://issues.apache.org/jira/browse/SPARK-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-24131.
--
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 21203
[https://github.com/apache/spark/pull/21203]
> Add majorMinorVersion API to PySpark for determining Spark versions
> ---
>
> Key: SPARK-24131
> URL: https://issues.apache.org/jira/browse/SPARK-24131
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 2.4.0
>Reporter: Liang-Chi Hsieh
>Assignee: Liang-Chi Hsieh
>Priority: Minor
> Fix For: 2.4.0
>
>
> We need to determine Spark major and minor versions in PySpark. We can add a
> {{majorMinorVersion}} API to PySpark which is similar to the API in
> {{VersionUtils.majorMinorVersion}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24116) SparkSQL inserting overwrite table has inconsistent behavior regarding HDFS trash
[
https://issues.apache.org/jira/browse/SPARK-24116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460472#comment-16460472
]
Rui Li commented on SPARK-24116:
[~hyukjin.kwon], sorry for the late response. For example, assume we have two
non-partitioned tables, one is text table and the other is Parquet table. If we
insert overwrite the text table, old data will go to HDFS trash. But if we
insert overwrite the Parquet table, old data doesn't go to trash.
I believe SparkSQL has different code paths to load data into different kinds
of tables. And whether old data goes to trash is inconsistent among these code
paths. Specifically, {{Hive::loadTable}} moves old data to trash but seems
other code paths simply delete the old data. Ideally it's good if SparkSQL lets
user specify whether old data goes to trash when overwriting, some feature like
HIVE-15880.
> SparkSQL inserting overwrite table has inconsistent behavior regarding HDFS
> trash
> -
>
> Key: SPARK-24116
> URL: https://issues.apache.org/jira/browse/SPARK-24116
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Priority: Major
>
> When inserting overwrite a table, the old data may or may not go to trash
> based on:
> # Date format. E.g. text table may go to trash but parquet table doesn't.
> # Whether table is partitioned. E.g. partitioned text table doesn't go to
> trash while non-partitioned table does.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24116) SparkSQL inserting overwrite table has inconsistent behavior regarding HDFS trash
[ https://issues.apache.org/jira/browse/SPARK-24116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460473#comment-16460473 ] Hyukjin Kwon commented on SPARK-24116: -- [~lirui], mind if I ask a reproducer please? > SparkSQL inserting overwrite table has inconsistent behavior regarding HDFS > trash > - > > Key: SPARK-24116 > URL: https://issues.apache.org/jira/browse/SPARK-24116 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: Rui Li >Priority: Major > > When inserting overwrite a table, the old data may or may not go to trash > based on: > # Date format. E.g. text table may go to trash but parquet table doesn't. > # Whether table is partitioned. E.g. partitioned text table doesn't go to > trash while non-partitioned table does. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)
[
https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kumar resolved SPARK-24089.
---
Resolution: Not A Problem
Thanks for the clarification.
> DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)
> --
>
> Key: SPARK-24089
> URL: https://issues.apache.org/jira/browse/SPARK-24089
> Project: Spark
> Issue Type: Bug
> Components: Java API, Spark Core, SQL
>Affects Versions: 2.3.0
>Reporter: kumar
>Priority: Major
> Labels: bug
>
> I am completely stuck with this issue, unable to progress further. For more
> info pls refer this post :
> [https://stackoverflow.com/questions/49994085/spark-sql-2-3-dataframe-savemode-append-issue]
> I want to load multiple files one by one, don't want to load all files at a
> time. To achieve this i used SaveMode.Append, so that 2nd file data will be
> added to 1st file data in database, but it's throwing exception.
> {code:java}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: unresolved
> operator 'InsertIntoTable LogicalRDD [a1#4, b1#5, c1#6, d1#7], false, false,
> false;;
> 'InsertIntoTable LogicalRDD [a1#4, b1#5, c1#6, d1#7], false, false, false
> +- LogicalRDD [a1#22, b1#23, c1#24, d1#25], false
> {code}
> Code:
> {code:java}
> package com.log;
> import com.log.common.RegexMatch;
> import com.log.spark.SparkProcessor;
> import org.apache.spark.SparkContext;
> import org.apache.spark.api.java.JavaRDD;
> import org.apache.spark.api.java.function.Function;
> import org.apache.spark.sql.*;
> import org.apache.spark.sql.types.DataTypes;
> import org.apache.spark.sql.types.StructField;
> import org.apache.spark.sql.types.StructType;
> import org.apache.spark.storage.StorageLevel;
> import java.util.ArrayList;
> import java.util.List;
> public class TestApp {
> private SparkSession spark;
> private SparkContext sparkContext;
> private SQLContext sqlContext;
> public TestApp() {
> SparkSession spark = SparkSession.builder().appName("Simple
> Application")
> .config("spark.master", "local").getOrCreate();
> SparkContext sc = spark.sparkContext();
> this.spark = spark;
> this.sparkContext = sc;
> }
> public static void main(String[] args) {
> TestApp app = new TestApp();
> String[] afiles = {"C:\\Users\\test\\Desktop\\logs\\log1.txt",
> "C:\\Users\\test\\Desktop\\logs\\log2.txt"};
> for (String file : afiles) {
> app.writeFileToSchema(file);
> }
> }
> public void writeFileToSchema(String filePath) {
> StructType schema = getSchema();
> JavaRDD rowRDD = getRowRDD(filePath);
> if (spark.catalog().tableExists("mylogs")) {
> logDataFrame = spark.createDataFrame(rowRDD, schema);
> logDataFrame.createOrReplaceTempView("temptable");
>
> logDataFrame.write().mode(SaveMode.Append).insertInto("mylogs");//exception
> } else {
> logDataFrame = spark.createDataFrame(rowRDD, schema);
> logDataFrame.createOrReplaceTempView("mylogs");
> }
> Dataset results = spark.sql("SELECT count(b1) FROM mylogs");
> List allrows = results.collectAsList();
> System.out.println("Count:"+allrows);
> sqlContext = logDataFrame.sqlContext();
> }
> Dataset logDataFrame;
> public List getTagList() {
> Dataset results = sqlContext.sql("SELECT distinct(b1) FROM
> mylogs");
> List allrows = results.collectAsList();
> return allrows;
> }
> public StructType getSchema() {
> String schemaString = "a1 b1 c1 d1";
> List fields = new ArrayList<>();
> for (String fieldName : schemaString.split(" ")) {
> StructField field = DataTypes.createStructField(fieldName,
> DataTypes.StringType, true);
> fields.add(field);
> }
> StructType schema = DataTypes.createStructType(fields);
> return schema;
> }
> public JavaRDD getRowRDD(String filePath) {
> JavaRDD logRDD = sparkContext.textFile(filePath,
> 1).toJavaRDD();
> RegexMatch reg = new RegexMatch();
> JavaRDD rowRDD = logRDD
> .map((Function) line -> {
> String[] st = line.split(" ");
> return RowFactory.create(st[0], st[1], st[2], st[3]);
> });
> rowRDD.persist(StorageLevel.MEMORY_ONLY());
> return rowRDD;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24142) Add interpreted execution to SortPrefix expression
Bruce Robbins created SPARK-24142: - Summary: Add interpreted execution to SortPrefix expression Key: SPARK-24142 URL: https://issues.apache.org/jira/browse/SPARK-24142 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 2.3.0 Reporter: Bruce Robbins -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24119) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-24119: - Issue Type: Sub-task (was: Task) Parent: SPARK-23580 > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24119 > URL: https://issues.apache.org/jira/browse/SPARK-24119 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Bruce Robbins >Priority: Minor > > [~hvanhovell] [~kiszk] > I noticed SortPrefix did not support interpreted execution when I was testing > the PR for SPARK-24043. Somehow it was not covered by the umbrella Jira for > adding interpreted execution (SPARK-23580) > Since I had to implement interpreted execution for SortPrefix to complete > testing, I am creating this Jira. If there's no good reason why eval wasn't > implemented, I will make the PR in a few days. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24142) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-24142: -- Affects Version/s: (was: 2.3.0) 2.4.0 > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24142 > URL: https://issues.apache.org/jira/browse/SPARK-24142 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bruce Robbins >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24142) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460525#comment-16460525 ] Bruce Robbins commented on SPARK-24142: --- I opened another Jira on this a few days ago, but it was not a subtask. Opening this as a subtask of SPARK-23580. I will mark the other subtask as a duplicate,. I will make a PR in the next day or so. > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24142 > URL: https://issues.apache.org/jira/browse/SPARK-24142 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bruce Robbins >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24142) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-24142. --- Resolution: Duplicate > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24142 > URL: https://issues.apache.org/jira/browse/SPARK-24142 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bruce Robbins >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-23489:
Assignee: (was: Apache Spark)
> HiveExternalCatalogVersionsSuite flaky test
> ---
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460528#comment-16460528
]
Apache Spark commented on SPARK-23489:
--
User 'dongjoon-hyun' has created a pull request for this issue:
https://github.com/apache/spark/pull/21210
> HiveExternalCatalogVersionsSuite flaky test
> ---
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24119) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460527#comment-16460527 ] Bruce Robbins commented on SPARK-24119: --- [~maropu] Ahh... we crossed paths and I opened a second ticket as a subtask. I will try to remove that newer ticket. > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24119 > URL: https://issues.apache.org/jira/browse/SPARK-24119 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.3.0 >Reporter: Bruce Robbins >Priority: Minor > > [~hvanhovell] [~kiszk] > I noticed SortPrefix did not support interpreted execution when I was testing > the PR for SPARK-24043. Somehow it was not covered by the umbrella Jira for > adding interpreted execution (SPARK-23580) > Since I had to implement interpreted execution for SortPrefix to complete > testing, I am creating this Jira. If there's no good reason why eval wasn't > implemented, I will make the PR in a few days. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-23489) HiveExternalCatalogVersionsSuite flaky test
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-23489:
Assignee: Apache Spark
> HiveExternalCatalogVersionsSuite flaky test
> ---
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.4.0
>Reporter: Marco Gaido
>Assignee: Apache Spark
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24131) Add majorMinorVersion API to PySpark for determining Spark versions
[
https://issues.apache.org/jira/browse/SPARK-24131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460529#comment-16460529
]
Apache Spark commented on SPARK-24131:
--
User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/21211
> Add majorMinorVersion API to PySpark for determining Spark versions
> ---
>
> Key: SPARK-24131
> URL: https://issues.apache.org/jira/browse/SPARK-24131
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
>Affects Versions: 2.4.0
>Reporter: Liang-Chi Hsieh
>Assignee: Liang-Chi Hsieh
>Priority: Minor
> Fix For: 2.4.0
>
>
> We need to determine Spark major and minor versions in PySpark. We can add a
> {{majorMinorVersion}} API to PySpark which is similar to the API in
> {{VersionUtils.majorMinorVersion}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23489) Flaky Test: HiveExternalCatalogVersionsSuite
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23489:
--
Affects Version/s: 2.2.1
2.3.0
> Flaky Test: HiveExternalCatalogVersionsSuite
>
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.2.1, 2.3.0, 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23489) Flaky Test: HiveExternalCatalogVersionsSuite
[
https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-23489:
--
Summary: Flaky Test: HiveExternalCatalogVersionsSuite (was:
HiveExternalCatalogVersionsSuite flaky test)
> Flaky Test: HiveExternalCatalogVersionsSuite
>
>
> Key: SPARK-23489
> URL: https://issues.apache.org/jira/browse/SPARK-23489
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
>Affects Versions: 2.2.1, 2.3.0, 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> I saw this error in an unrelated PR. It seems to me a bad configuration in
> the Jenkins node where the tests are run.
> {code}
> Error Message
> java.io.IOException: Cannot run program "./bin/spark-submit" (in directory
> "/tmp/test-spark/spark-2.2.1"): error=2, No such file or directory
> Stacktrace
> sbt.ForkMain$ForkError: java.io.IOException: Cannot run program
> "./bin/spark-submit" (in directory "/tmp/test-spark/spark-2.2.1"): error=2,
> No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at
> org.apache.spark.sql.hive.SparkSubmitTestUtils$class.runSparkSubmit(SparkSubmitTestUtils.scala:73)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:43)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:176)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite$$anonfun$beforeAll$1.apply(HiveExternalCatalogVersionsSuite.scala:161)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at
> org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:161)
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:212)
> at
> org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:52)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: sbt.ForkMain$ForkError: java.io.IOException: error=2, No such file
> or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.(UNIXProcess.java:248)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 17 more
> {code}
> This is the link:
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87615/testReport/.
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/4389/testReport/junit/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
> *NOTE: This failure frequently looks as `Test Result (no failures)`*
> -
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/4811/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24142) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460532#comment-16460532 ] Bruce Robbins commented on SPARK-24142: --- [~maropu] I don't seem to have the Jira authority (or maybe I lack the knowledge) to remove this duplicate as a subtask from SPARK-23580. If you have that authority, it would be greatly appreciated if you could remove it. Thanks. > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24142 > URL: https://issues.apache.org/jira/browse/SPARK-24142 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bruce Robbins >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24143) filter empty blocks when convert mapstatus to (blockId, size) pair
jin xing created SPARK-24143: Summary: filter empty blocks when convert mapstatus to (blockId, size) pair Key: SPARK-24143 URL: https://issues.apache.org/jira/browse/SPARK-24143 Project: Spark Issue Type: New Feature Components: Spark Core Affects Versions: 2.3.0 Reporter: jin xing In current code(MapOutputTracker.convertMapStatuses), mapstatus are converted to (blockId, size) pair for all blocks -- no matter the block is empty or not, which result in OOM when there are lots of consecutive empty blocks, especially when adaptive execution is enabled. (blockId, size) pair is only used in ShuffleBlockFetcherIterator to control shuffle-read and only non-empty block request is sent. Can we just filter out the empty blocks in MapOutputTracker.convertMapStatuses and save memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24143) filter empty blocks when convert mapstatus to (blockId, size) pair
[ https://issues.apache.org/jira/browse/SPARK-24143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-24143: - Issue Type: Bug (was: New Feature) > filter empty blocks when convert mapstatus to (blockId, size) pair > -- > > Key: SPARK-24143 > URL: https://issues.apache.org/jira/browse/SPARK-24143 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: jin xing >Priority: Major > > In current code(MapOutputTracker.convertMapStatuses), mapstatus are converted > to (blockId, size) pair for all blocks -- no matter the block is empty or > not, which result in OOM when there are lots of consecutive empty blocks, > especially when adaptive execution is enabled. > (blockId, size) pair is only used in ShuffleBlockFetcherIterator to control > shuffle-read and only non-empty block request is sent. Can we just filter out > the empty blocks in MapOutputTracker.convertMapStatuses and save memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24144) monotonically_increasing_id on streaming dataFrames
Hemant Bhanawat created SPARK-24144: --- Summary: monotonically_increasing_id on streaming dataFrames Key: SPARK-24144 URL: https://issues.apache.org/jira/browse/SPARK-24144 Project: Spark Issue Type: New Feature Components: Structured Streaming Affects Versions: 2.3.0 Reporter: Hemant Bhanawat For our use case, we want to assign snapshot ids (incrementing counters) to the incoming records. In case of failures, the same record should get the same id after failure so that the downstream DB can handle the records in a correct manner. We were trying to do this by zipping the streaming rdds with that counter using a modified version of ZippedWithIndexRDD. There are other ways to do that but it turns out all ways are cumbersome and error prone in failure scenarios. As suggested on the spark user dev list, one way to do this would be to support monotonically_increasing_id on streaming dataFrames in Spark code base. This would ensure that counters are incrementing for the records of the stream. Also, since the counter can be checkpointed, it would work well in case of failure scenarios. Last but not the least, doing this in spark would be the most performance efficient way. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24144) monotonically_increasing_id on streaming dataFrames
[ https://issues.apache.org/jira/browse/SPARK-24144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemant Bhanawat updated SPARK-24144: Priority: Major (was: Minor) > monotonically_increasing_id on streaming dataFrames > --- > > Key: SPARK-24144 > URL: https://issues.apache.org/jira/browse/SPARK-24144 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Hemant Bhanawat >Priority: Major > > For our use case, we want to assign snapshot ids (incrementing counters) to > the incoming records. In case of failures, the same record should get the > same id after failure so that the downstream DB can handle the records in a > correct manner. > We were trying to do this by zipping the streaming rdds with that counter > using a modified version of ZippedWithIndexRDD. There are other ways to do > that but it turns out all ways are cumbersome and error prone in failure > scenarios. > As suggested on the spark user dev list, one way to do this would be to > support monotonically_increasing_id on streaming dataFrames in Spark code > base. This would ensure that counters are incrementing for the records of the > stream. Also, since the counter can be checkpointed, it would work well in > case of failure scenarios. Last but not the least, doing this in spark would > be the most performance efficient way. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24143) filter empty blocks when convert mapstatus to (blockId, size) pair
[ https://issues.apache.org/jira/browse/SPARK-24143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460548#comment-16460548 ] Apache Spark commented on SPARK-24143: -- User 'jinxing64' has created a pull request for this issue: https://github.com/apache/spark/pull/21212 > filter empty blocks when convert mapstatus to (blockId, size) pair > -- > > Key: SPARK-24143 > URL: https://issues.apache.org/jira/browse/SPARK-24143 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: jin xing >Priority: Major > > In current code(MapOutputTracker.convertMapStatuses), mapstatus are converted > to (blockId, size) pair for all blocks -- no matter the block is empty or > not, which result in OOM when there are lots of consecutive empty blocks, > especially when adaptive execution is enabled. > (blockId, size) pair is only used in ShuffleBlockFetcherIterator to control > shuffle-read and only non-empty block request is sent. Can we just filter out > the empty blocks in MapOutputTracker.convertMapStatuses and save memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24143) filter empty blocks when convert mapstatus to (blockId, size) pair
[ https://issues.apache.org/jira/browse/SPARK-24143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24143: Assignee: (was: Apache Spark) > filter empty blocks when convert mapstatus to (blockId, size) pair > -- > > Key: SPARK-24143 > URL: https://issues.apache.org/jira/browse/SPARK-24143 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: jin xing >Priority: Major > > In current code(MapOutputTracker.convertMapStatuses), mapstatus are converted > to (blockId, size) pair for all blocks -- no matter the block is empty or > not, which result in OOM when there are lots of consecutive empty blocks, > especially when adaptive execution is enabled. > (blockId, size) pair is only used in ShuffleBlockFetcherIterator to control > shuffle-read and only non-empty block request is sent. Can we just filter out > the empty blocks in MapOutputTracker.convertMapStatuses and save memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24143) filter empty blocks when convert mapstatus to (blockId, size) pair
[ https://issues.apache.org/jira/browse/SPARK-24143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24143: Assignee: Apache Spark > filter empty blocks when convert mapstatus to (blockId, size) pair > -- > > Key: SPARK-24143 > URL: https://issues.apache.org/jira/browse/SPARK-24143 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: jin xing >Assignee: Apache Spark >Priority: Major > > In current code(MapOutputTracker.convertMapStatuses), mapstatus are converted > to (blockId, size) pair for all blocks -- no matter the block is empty or > not, which result in OOM when there are lots of consecutive empty blocks, > especially when adaptive execution is enabled. > (blockId, size) pair is only used in ShuffleBlockFetcherIterator to control > shuffle-read and only non-empty block request is sent. Can we just filter out > the empty blocks in MapOutputTracker.convertMapStatuses and save memory? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes
[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460554#comment-16460554 ] Anirudh Ramanathan commented on SPARK-24137: SGTM. Let's try and share as much code as possible with https://issues.apache.org/jira/browse/SPARK-23529. > [K8s] Mount temporary directories in emptydir volumes > - > > Key: SPARK-24137 > URL: https://issues.apache.org/jira/browse/SPARK-24137 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Matt Cheah >Priority: Major > > Currently the Spark local directories do not get any volumes and volume > mounts, which means we're writing Spark shuffle and cache contents to the > file system mounted by Docker. This can be terribly inefficient. We should > use emptydir volumes for these directories instead for significant > performance improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24142) Add interpreted execution to SortPrefix expression
[ https://issues.apache.org/jira/browse/SPARK-24142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460555#comment-16460555 ] Takeshi Yamamuro commented on SPARK-24142: -- I think it is ok as it is, resolved as duplicate. > Add interpreted execution to SortPrefix expression > -- > > Key: SPARK-24142 > URL: https://issues.apache.org/jira/browse/SPARK-24142 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.4.0 >Reporter: Bruce Robbins >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
