[jira] [Commented] (SPARK-27499) Support mapping spark.local.dir to hostPath volume
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885666#comment-16885666 ] Junjie Chen commented on SPARK-27499: - Hi [~vanzin] There is an opened Jira SPARK-28042 for this. > Support mapping spark.local.dir to hostPath volume > -- > > Key: SPARK-27499 > URL: https://issues.apache.org/jira/browse/SPARK-27499 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Junjie Chen >Priority: Minor > > Currently, the k8s executor builder mount spark.local.dir as emptyDir or > memory, it should satisfy some small workload, while in some heavily workload > like TPCDS, both of them can have some problem, such as pods are evicted due > to disk pressure when using emptyDir, and OOM when using tmpfs. > In particular on cloud environment, users may allocate cluster with minimum > configuration and add cloud storage when running workload. In this case, we > can specify multiple elastic storage as spark.local.dir to accelerate the > spilling. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27499) Support mapping spark.local.dir to hostPath volume
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861707#comment-16861707 ] Junjie Chen commented on SPARK-27499: - Yes, In KubernetesExecutorBuilder.scala, the LocalDrisFeatureStep is built before MountVolumesFeatureStep which means we cannot use any volumes mount later. I think we should build localDirsFeature at last, so that we can check if directories in SPARK_LOCAL_DIRS are set to volumes mounted either hostPath, PV, or others may support later and use that as local storage. With that way, we can utilize specified media to improve the local storage performance instead of just emptyDir which is just a ephemeral directory on node. > Support mapping spark.local.dir to hostPath volume > -- > > Key: SPARK-27499 > URL: https://issues.apache.org/jira/browse/SPARK-27499 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Junjie Chen >Priority: Minor > > Currently, the k8s executor builder mount spark.local.dir as emptyDir or > memory, it should satisfy some small workload, while in some heavily workload > like TPCDS, both of them can have some problem, such as pods are evicted due > to disk pressure when using emptyDir, and OOM when using tmpfs. > In particular on cloud environment, users may allocate cluster with minimum > configuration and add cloud storage when running workload. In this case, we > can specify multiple elastic storage as spark.local.dir to accelerate the > spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27499) Support mapping spark.local.dir to hostPath volume
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860519#comment-16860519 ] Junjie Chen edited comment on SPARK-27499 at 6/11/19 5:20 AM: -- Hi, [~dongjoon], I know SPARK_LOCAL_DIRS can be mounted as emptyDir. However, emptyDir just one directory on node. I opened this Jira to track a feature to setting multiple directories to full utilize the nodes' disks bandwidth for spilling, which I think currently it can not be achieve through setting spark.local.dir. Even I set to multiple dirs, they still map to one directory on node. This Jira was intended to use hostPath volumes mounts as spark.local.dir, which needs build mountVolumeFeature to built before localDirFeature, while currently the localDirFeature is built before mountVolumeFeature. was (Author: junjie): Hi, [~dongjoon], I know SPARK_LOCAL_DIRS can be mounted as emptyDir. However, emptyDir just one directory on node. I opened this Jira to track a feature to setting multiple directories to full utilize the nodes' disks bandwidth for spilling, which I think currently it can not be achieve through setting spark.local.dir. Even I set to multiple dirs, they still map to one directory on node. This Jira is intended to use hostPath volumes mounts as spark.local.dir, for exmaple: spark.kubernetes.executor.volumes.hostPath.spark-local-dir-1.mount.path=/data/mnt-x > Support mapping spark.local.dir to hostPath volume > -- > > Key: SPARK-27499 > URL: https://issues.apache.org/jira/browse/SPARK-27499 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Junjie Chen >Priority: Minor > Fix For: 2.4.0 > > > Currently, the k8s executor builder mount spark.local.dir as emptyDir or > memory, it should satisfy some small workload, while in some heavily workload > like TPCDS, both of them can have some problem, such as pods are evicted due > to disk pressure when using emptyDir, and OOM when using tmpfs. > In particular on cloud environment, users may allocate cluster with minimum > configuration and add cloud storage when running workload. In this case, we > can specify multiple elastic storage as spark.local.dir to accelerate the > spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-27499) Support mapping spark.local.dir to hostPath volume
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junjie Chen reopened SPARK-27499: - > Support mapping spark.local.dir to hostPath volume > -- > > Key: SPARK-27499 > URL: https://issues.apache.org/jira/browse/SPARK-27499 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Junjie Chen >Priority: Minor > Fix For: 2.4.0 > > > Currently, the k8s executor builder mount spark.local.dir as emptyDir or > memory, it should satisfy some small workload, while in some heavily workload > like TPCDS, both of them can have some problem, such as pods are evicted due > to disk pressure when using emptyDir, and OOM when using tmpfs. > In particular on cloud environment, users may allocate cluster with minimum > configuration and add cloud storage when running workload. In this case, we > can specify multiple elastic storage as spark.local.dir to accelerate the > spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27499) Support mapping spark.local.dir to hostPath volume
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860519#comment-16860519 ] Junjie Chen commented on SPARK-27499: - Hi, [~dongjoon], I know SPARK_LOCAL_DIRS can be mounted as emptyDir. However, emptyDir just one directory on node. I opened this Jira to track a feature to setting multiple directories to full utilize the nodes' disks bandwidth for spilling, which I think currently it can not be achieve through setting spark.local.dir. Even I set to multiple dirs, they still map to one directory on node. This Jira is intended to use hostPath volumes mounts as spark.local.dir, for exmaple: spark.kubernetes.executor.volumes.hostPath.spark-local-dir-1.mount.path=/data/mnt-x > Support mapping spark.local.dir to hostPath volume > -- > > Key: SPARK-27499 > URL: https://issues.apache.org/jira/browse/SPARK-27499 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Junjie Chen >Priority: Minor > Fix For: 2.4.0 > > > Currently, the k8s executor builder mount spark.local.dir as emptyDir or > memory, it should satisfy some small workload, while in some heavily workload > like TPCDS, both of them can have some problem, such as pods are evicted due > to disk pressure when using emptyDir, and OOM when using tmpfs. > In particular on cloud environment, users may allocate cluster with minimum > configuration and add cloud storage when running workload. In this case, we > can specify multiple elastic storage as spark.local.dir to accelerate the > spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27499) Support mapping spark.local.dir to hostPath volume
Junjie Chen created SPARK-27499: --- Summary: Support mapping spark.local.dir to hostPath volume Key: SPARK-27499 URL: https://issues.apache.org/jira/browse/SPARK-27499 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 2.4.1 Reporter: Junjie Chen Currently, the k8s executor builder mount spark.local.dir as emptyDir or memory, it should satisfy some small workload, while in some heavily workload like TPCDS, both of them can have some problem, such as pods are evicted due to disk pressure when using emptyDir, and OOM when using tmpfs. In particular on cloud environment, users may allocate cluster with minimum configuration and add cloud storage when running workload. In this case, we can specify multiple elastic storage as spark.local.dir to accelerate the spilling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes
[
https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817499#comment-16817499
]
Junjie Chen commented on SPARK-25262:
-
Hi
Do we have plan to for second change mentioned above for checking existing
volume logic?
> Make Spark local dir volumes configurable with Spark on Kubernetes
> --
>
> Key: SPARK-25262
> URL: https://issues.apache.org/jira/browse/SPARK-25262
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 2.3.0, 2.3.1
>Reporter: Rob Vesse
>Priority: Major
>
> As discussed during review of the design document for SPARK-24434 while
> providing pod templates will provide more in-depth customisation for Spark on
> Kubernetes there are some things that cannot be modified because Spark code
> generates pod specs in very specific ways.
> The particular issue identified relates to handling on {{spark.local.dirs}}
> which is done by {{LocalDirsFeatureStep.scala}}. For each directory
> specified, or a single default if no explicit specification, it creates a
> Kubernetes {{emptyDir}} volume. As noted in the Kubernetes documentation
> this will be backed by the node storage
> (https://kubernetes.io/docs/concepts/storage/volumes/#emptydir). In some
> compute environments this may be extremely undesirable. For example with
> diskless compute resources the node storage will likely be a non-performant
> remote mounted disk, often with limited capacity. For such environments it
> would likely be better to set {{medium: Memory}} on the volume per the K8S
> documentation to use a {{tmpfs}} volume instead.
> Another closely related issue is that users might want to use a different
> volume type to back the local directories and there is no possibility to do
> that.
> Pod templates will not really solve either of these issues because Spark is
> always going to attempt to generate a new volume for each local directory and
> always going to set these as {{emptyDir}}.
> Therefore the proposal is to make two changes to {{LocalDirsFeatureStep}}:
> * Provide a new config setting to enable using {{tmpfs}} backed {{emptyDir}}
> volumes
> * Modify the logic to check if there is a volume already defined with the
> name and if so skip generating a volume definition for it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13331) AES support for over-the-wire encryption
[ https://issues.apache.org/jira/browse/SPARK-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613837#comment-15613837 ] Junjie Chen commented on SPARK-13331: - Hi [~vanzin] Do we need more review? > AES support for over-the-wire encryption > > > Key: SPARK-13331 > URL: https://issues.apache.org/jira/browse/SPARK-13331 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Dong Chen >Priority: Minor > > In network/common, SASL with DIGEST-MD5 authentication is used for > negotiating a secure communication channel. When SASL operation mode is > "auth-conf", the data transferred on the network is encrypted. DIGEST-MD5 > mechanism supports following encryption: 3DES, DES, and RC4. The negotiation > procedure will select one of them to encrypt / decrypt the data on the > channel. > However, 3des and rc4 are slow relatively. We could add code in the > negotiation to make it support AES for more secure and performance. > The proposed solution is: > When "auth-conf" is enabled, at the end of original negotiation, the > authentication succeeds and a secure channel is built. We could add one more > negotiation step: Client and server negotiate whether they both support AES. > If yes, the Key and IV used by AES will be generated by server and sent to > client through the already secure channel. Then update the encryption / > decryption handler to AES at both client and server side. Following data > transfer will use AES instead of original encryption algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13331) AES support for over-the-wire encryption
[ https://issues.apache.org/jira/browse/SPARK-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607851#comment-15607851 ] Junjie Chen commented on SPARK-13331: - Hi [~vanzin] The patch was updated according to your comments. > AES support for over-the-wire encryption > > > Key: SPARK-13331 > URL: https://issues.apache.org/jira/browse/SPARK-13331 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Dong Chen >Priority: Minor > > In network/common, SASL with DIGEST-MD5 authentication is used for > negotiating a secure communication channel. When SASL operation mode is > "auth-conf", the data transferred on the network is encrypted. DIGEST-MD5 > mechanism supports following encryption: 3DES, DES, and RC4. The negotiation > procedure will select one of them to encrypt / decrypt the data on the > channel. > However, 3des and rc4 are slow relatively. We could add code in the > negotiation to make it support AES for more secure and performance. > The proposed solution is: > When "auth-conf" is enabled, at the end of original negotiation, the > authentication succeeds and a secure channel is built. We could add one more > negotiation step: Client and server negotiate whether they both support AES. > If yes, the Key and IV used by AES will be generated by server and sent to > client through the already secure channel. Then update the encryption / > decryption handler to AES at both client and server side. Following data > transfer will use AES instead of original encryption algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13331) AES support for over-the-wire encryption
[ https://issues.apache.org/jira/browse/SPARK-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604582#comment-15604582 ] Junjie Chen commented on SPARK-13331: - Hi [~vanzin] The updated patch was committed according to your comments. I tried to change the negotiation by sending configuration to server without waiting for response, but server end does not get data as expected. From the description of TransportClient.send, it doesn't guarantee delivery. I think it should wait for response from server to do a handshake here, otherwise client will send encrypted data out and server may still not ready to accept encrypted data. isn't it? > AES support for over-the-wire encryption > > > Key: SPARK-13331 > URL: https://issues.apache.org/jira/browse/SPARK-13331 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Dong Chen >Priority: Minor > > In network/common, SASL with DIGEST-MD5 authentication is used for > negotiating a secure communication channel. When SASL operation mode is > "auth-conf", the data transferred on the network is encrypted. DIGEST-MD5 > mechanism supports following encryption: 3DES, DES, and RC4. The negotiation > procedure will select one of them to encrypt / decrypt the data on the > channel. > However, 3des and rc4 are slow relatively. We could add code in the > negotiation to make it support AES for more secure and performance. > The proposed solution is: > When "auth-conf" is enabled, at the end of original negotiation, the > authentication succeeds and a secure channel is built. We could add one more > negotiation step: Client and server negotiate whether they both support AES. > If yes, the Key and IV used by AES will be generated by server and sent to > client through the already secure channel. Then update the encryption / > decryption handler to AES at both client and server side. Following data > transfer will use AES instead of original encryption algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13331) AES support for over-the-wire encryption
[ https://issues.apache.org/jira/browse/SPARK-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15600705#comment-15600705 ] Junjie Chen commented on SPARK-13331: - Hi [~vanzin] I have updated the latest patch, Could you please help to review it? Due to an issue (CRYPTO-125) in Common Crypto, the patch has to use two helper channels. Once it be fixed and released, I will remove these channels. > AES support for over-the-wire encryption > > > Key: SPARK-13331 > URL: https://issues.apache.org/jira/browse/SPARK-13331 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Dong Chen >Priority: Minor > > In network/common, SASL with DIGEST-MD5 authentication is used for > negotiating a secure communication channel. When SASL operation mode is > "auth-conf", the data transferred on the network is encrypted. DIGEST-MD5 > mechanism supports following encryption: 3DES, DES, and RC4. The negotiation > procedure will select one of them to encrypt / decrypt the data on the > channel. > However, 3des and rc4 are slow relatively. We could add code in the > negotiation to make it support AES for more secure and performance. > The proposed solution is: > When "auth-conf" is enabled, at the end of original negotiation, the > authentication succeeds and a secure channel is built. We could add one more > negotiation step: Client and server negotiate whether they both support AES. > If yes, the Key and IV used by AES will be generated by server and sent to > client through the already secure channel. Then update the encryption / > decryption handler to AES at both client and server side. Following data > transfer will use AES instead of original encryption algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13331) AES support for over-the-wire encryption
[ https://issues.apache.org/jira/browse/SPARK-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512340#comment-15512340 ] Junjie Chen commented on SPARK-13331: - Hi [~vanzin] Could you help to review the code? > AES support for over-the-wire encryption > > > Key: SPARK-13331 > URL: https://issues.apache.org/jira/browse/SPARK-13331 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Dong Chen >Priority: Minor > > In network/common, SASL with DIGEST-MD5 authentication is used for > negotiating a secure communication channel. When SASL operation mode is > "auth-conf", the data transferred on the network is encrypted. DIGEST-MD5 > mechanism supports following encryption: 3DES, DES, and RC4. The negotiation > procedure will select one of them to encrypt / decrypt the data on the > channel. > However, 3des and rc4 are slow relatively. We could add code in the > negotiation to make it support AES for more secure and performance. > The proposed solution is: > When "auth-conf" is enabled, at the end of original negotiation, the > authentication succeeds and a secure channel is built. We could add one more > negotiation step: Client and server negotiate whether they both support AES. > If yes, the Key and IV used by AES will be generated by server and sent to > client through the already secure channel. Then update the encryption / > decryption handler to AES at both client and server side. Following data > transfer will use AES instead of original encryption algorithm. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
