+1 thanks Yikun for initiating this! Look forward to more progress made together by the community!
On Wed, Jan 5, 2022 at 4:52 PM Weiwei Yang <w...@apache.org> wrote: > +1 > > I think it is in a good shape to move forward > > On Wed, Jan 5, 2022 at 3:00 PM Bowen Li <b...@apache.org> wrote: > >> +1 for SPIP >> >> According our production experience, the default scheduler isn't meeting >> prod requirements on K8S, and such effort of integrating with batch-native >> schedulers makes running Spark natively on K8S much easier for users. >> >> Thanks, >> Bowen >> >> On Wed, Jan 5, 2022 at 11:52 AM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> +1 non-binding >>> >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Wed, 5 Jan 2022 at 19:16, Holden Karau <hol...@pigscanfly.ca> wrote: >>> >>>> Do we want to move the SPIP forward to a vote? It seems like we're >>>> mostly agreeing in principle? >>>> >>>> On Wed, Jan 5, 2022 at 11:12 AM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi Bo, >>>>> >>>>> Thanks for the info. Let me elaborate: >>>>> >>>>> In theory you can set the number of executors to multiple values of >>>>> Nodes. For example if you have a three node k8s cluster (in my case Google >>>>> GKE), you can set the number of executors to 6 and end up with six >>>>> executors queuing to start but ultimately you finish with two running >>>>> executors plus the driver in a 3 node cluster as shown below >>>>> >>>>> hduser@ctpvm: /home/hduser> k get pods -n spark >>>>> >>>>> NAME READY STATUS >>>>> RESTARTS AGE >>>>> >>>>> *randomdatabigquery-d42d067e2b91c88a-exec-1 1/1 Running 0 >>>>> 33s* >>>>> >>>>> *randomdatabigquery-d42d067e2b91c88a-exec-2 1/1 Running 0 >>>>> 33s* >>>>> >>>>> randomdatabigquery-d42d067e2b91c88a-exec-3 0/1 Pending 0 >>>>> 33s >>>>> >>>>> randomdatabigquery-d42d067e2b91c88a-exec-4 0/1 Pending 0 >>>>> 33s >>>>> >>>>> randomdatabigquery-d42d067e2b91c88a-exec-5 0/1 Pending 0 >>>>> 33s >>>>> >>>>> randomdatabigquery-d42d067e2b91c88a-exec-6 0/1 Pending 0 >>>>> 33s >>>>> >>>>> *sparkbq-0beda77e2b919e01-driver 1/1 Running 0 >>>>> 45s* >>>>> >>>>> hduser@ctpvm: /home/hduser> k get pods -n spark >>>>> >>>>> NAME READY STATUS >>>>> RESTARTS AGE >>>>> >>>>> randomdatabigquery-d42d067e2b91c88a-exec-1 1/1 Running 0 >>>>> 38s >>>>> >>>>> randomdatabigquery-d42d067e2b91c88a-exec-2 1/1 Running 0 >>>>> 38s >>>>> >>>>> sparkbq-0beda77e2b919e01-driver 1/1 Running 0 >>>>> 50s >>>>> >>>>> hduser@ctpvm: /home/hduser> k get pods -n spark >>>>> >>>>> *NAME READY STATUS >>>>> RESTARTS AGE* >>>>> >>>>> *randomdatabigquery-d42d067e2b91c88a-exec-1 1/1 Running 0 >>>>> 40s* >>>>> >>>>> *randomdatabigquery-d42d067e2b91c88a-exec-2 1/1 Running 0 >>>>> 40s* >>>>> >>>>> *sparkbq-0beda77e2b919e01-driver 1/1 Running 0 >>>>> 52s* >>>>> >>>>> So you end up with the three added executors dropping out. Hence the >>>>> conclusion seems to be you want to fit exactly one Spark executor pod >>>>> per Kubernetes node with the current model. >>>>> >>>>> HTH >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, 5 Jan 2022 at 17:01, bo yang <bobyan...@gmail.com> wrote: >>>>> >>>>>> Hi Mich, >>>>>> >>>>>> Curious what do you mean “The constraint seems to be that you can >>>>>> fit one Spark executor pod per Kubernetes node and from my tests you >>>>>> don't seem to be able to allocate more than 50% of RAM on the node >>>>>> to the container", Would you help to explain a bit? Asking this because >>>>>> there could be multiple executor pods running on a single Kuberentes >>>>>> node. >>>>>> >>>>>> Thanks, >>>>>> Bo >>>>>> >>>>>> >>>>>> On Wed, Jan 5, 2022 at 1:13 AM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> Thanks William for the info. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> The current model of Spark on k8s has certain drawbacks with pod >>>>>>> based scheduling as I tested it on Google Kubernetes Cluster (GKE). The >>>>>>> constraint seems to be that you can fit one Spark executor pod per >>>>>>> Kubernetes node and from my tests you don't seem to be able to allocate >>>>>>> more than 50% of RAM on the node to the container. >>>>>>> >>>>>>> >>>>>>> [image: gke_memoeyPlot.png] >>>>>>> >>>>>>> >>>>>>> Anymore results in the container never been created (stuck at pending) >>>>>>> >>>>>>> kubectl describe pod sparkbq-b506ac7dc521b667-driver -n spark >>>>>>> >>>>>>> Events: >>>>>>> >>>>>>> Type Reason Age From >>>>>>> Message >>>>>>> >>>>>>> ---- ------ ---- ---- >>>>>>> ------- >>>>>>> >>>>>>> Warning FailedScheduling 17m default-scheduler >>>>>>> 0/3 nodes are available: 3 Insufficient memory. >>>>>>> >>>>>>> Warning FailedScheduling 17m default-scheduler >>>>>>> 0/3 nodes are available: 3 Insufficient memory. >>>>>>> >>>>>>> Normal NotTriggerScaleUp 2m28s (x92 over 17m) cluster-autoscaler >>>>>>> pod didn't trigger scale-up: >>>>>>> >>>>>>> Obviously this is far from ideal and this model although works is >>>>>>> not efficient. >>>>>>> >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> >>>>>>> Mich >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>> for any loss, damage or destruction >>>>>>> >>>>>>> of data or any other property which may arise from relying on this >>>>>>> email's technical content is explicitly disclaimed. >>>>>>> >>>>>>> The author will in no case be liable for any monetary damages >>>>>>> arising from such >>>>>>> >>>>>>> loss, damage or destruction. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, 5 Jan 2022 at 03:55, William Wang <wang.platf...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Mich, >>>>>>>> >>>>>>>> Here are parts of performance indications in Volcano. >>>>>>>> 1. Scheduler throughput: 1.5k pod/s (default scheduler: 100 Pod/s) >>>>>>>> 2. Spark application performance improved 30%+ with minimal >>>>>>>> resource reservation feature in case of insufficient resource.(tested >>>>>>>> with >>>>>>>> TPC-DS) >>>>>>>> >>>>>>>> We are still working on more optimizations. Besides the >>>>>>>> performance, Volcano is continuously enhanced in below four directions >>>>>>>> to >>>>>>>> provide abilities that users care about. >>>>>>>> - Full lifecycle management for jobs >>>>>>>> - Scheduling policies for high-performance workloads(fair-share, >>>>>>>> topology, sla, reservation, preemption, backfill etc) >>>>>>>> - Support for heterogeneous hardware >>>>>>>> - Performance optimization for high-performance workloads >>>>>>>> >>>>>>>> Thanks >>>>>>>> LeiBo >>>>>>>> >>>>>>>> Mich Talebzadeh <mich.talebza...@gmail.com> 于2022年1月4日周二 18:12写道： >>>>>>>> >>>>>>> Interesting,thanks >>>>>>>>> >>>>>>>>> Do you have any indication of the ballpark figure (a rough >>>>>>>>> numerical estimate) of adding Volcano as an alternative scheduler >>>>>>>>> is going to improve Spark on k8s performance? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>>>> for any loss, damage or destruction >>>>>>>>> >>>>>>>>> of data or any other property which may arise from relying on this >>>>>>>>> email's technical content is explicitly disclaimed. >>>>>>>>> >>>>>>>>> The author will in no case be liable for any monetary damages >>>>>>>>> arising from such >>>>>>>>> >>>>>>>>> loss, damage or destruction. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, 4 Jan 2022 at 09:43, Yikun Jiang <yikunk...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, folks! Wishing you all the best in 2022. >>>>>>>>>> >>>>>>>>>> I'd like to share the current status on "Support Customized K8S >>>>>>>>>> Scheduler in Spark". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg/edit#heading=h.1quyr1r2kr5n >>>>>>>>>> >>>>>>>>>> Framework/Common support >>>>>>>>>> >>>>>>>>>> - Volcano and Yunikorn team join the discussion and complete the >>>>>>>>>> initial doc on framework/common part. >>>>>>>>>> >>>>>>>>>> - SPARK-37145 <https://issues.apache.org/jira/browse/SPARK-37145> >>>>>>>>>> (under reviewing): We proposed to extend the customized scheduler by >>>>>>>>>> just >>>>>>>>>> using a custom feature step, it will meet the requirement of >>>>>>>>>> customized >>>>>>>>>> scheduler after it gets merged. After this, the user can enable >>>>>>>>>> featurestep >>>>>>>>>> and scheduler like: >>>>>>>>>> >>>>>>>>>> spark-submit \ >>>>>>>>>> >>>>>>>>>> --conf spark.kubernete.scheduler.name volcano \ >>>>>>>>>> >>>>>>>>>> --conf spark.kubernetes.driver.pod.featureSteps >>>>>>>>>> org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep >>>>>>>>>> >>>>>>>>>> --conf spark.kubernete.job.queue xxx >>>>>>>>>> >>>>>>>>>> (such as above, the VolcanoFeatureStep will help to set the the >>>>>>>>>> spark scheduler queue according user specified conf) >>>>>>>>>> >>>>>>>>>> - SPARK-37331 <https://issues.apache.org/jira/browse/SPARK-37331>: >>>>>>>>>> Added the ability to create kubernetes resources before driver pod >>>>>>>>>> creation. >>>>>>>>>> >>>>>>>>>> - SPARK-36059 <https://issues.apache.org/jira/browse/SPARK-36059>: >>>>>>>>>> Add the ability to specify a scheduler in driver/executor >>>>>>>>>> >>>>>>>>>> After above all, the framework/common support would be ready for >>>>>>>>>> most of customized schedulers >>>>>>>>>> >>>>>>>>>> Volcano part: >>>>>>>>>> >>>>>>>>>> - SPARK-37258 <https://issues.apache.org/jira/browse/SPARK-37258>: >>>>>>>>>> Upgrade kubernetes-client to 5.11.1 to add volcano scheduler API >>>>>>>>>> support. >>>>>>>>>> >>>>>>>>>> - SPARK-36061 <https://issues.apache.org/jira/browse/SPARK-36061>: >>>>>>>>>> Add a VolcanoFeatureStep to help users to create a PodGroup with user >>>>>>>>>> specified minimum resources required, there is also a WIP commit >>>>>>>>>> to show the preview of this >>>>>>>>>> <https://github.com/Yikun/spark/pull/45/commits/81bf6f98edb5c00ebd0662dc172bc73f980b6a34> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> Yunikorn part: >>>>>>>>>> >>>>>>>>>> - @WeiweiYang is completing the doc of the Yunikorn part and >>>>>>>>>> implementing the Yunikorn part. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Yikun >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Weiwei Yang <w...@apache.org> 于2021年12月2日周四 02:00写道： >>>>>>>>>> >>>>>>>>>>> Thank you Yikun for the info, and thanks for inviting me to a >>>>>>>>>>> meeting to discuss this. >>>>>>>>>>> I appreciate your effort to put these together, and I agree that >>>>>>>>>>> the purpose is to make Spark easy/flexible enough to support other >>>>>>>>>>> K8s >>>>>>>>>>> schedulers (not just for Volcano). >>>>>>>>>>> As discussed, could you please help to abstract out the things >>>>>>>>>>> in common and allow Spark to plug different implementations? I'd be >>>>>>>>>>> happy >>>>>>>>>>> to work with you guys on this issue. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Nov 30, 2021 at 6:49 PM Yikun Jiang <yikunk...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> @Weiwei @Chenya >>>>>>>>>>>> >>>>>>>>>>>> > Thanks for bringing this up. This is quite interesting, we >>>>>>>>>>>> definitely should participate more in the discussions. >>>>>>>>>>>> >>>>>>>>>>>> Thanks for your reply and welcome to join the discussion, I >>>>>>>>>>>> think the input from Yunikorn is very critical. >>>>>>>>>>>> >>>>>>>>>>>> > The main thing here is, the Spark community should make Spark >>>>>>>>>>>> pluggable in order to support other schedulers, not just for >>>>>>>>>>>> Volcano. It >>>>>>>>>>>> looks like this proposal is pushing really hard for adopting >>>>>>>>>>>> PodGroup, >>>>>>>>>>>> which isn't part of K8s yet, that to me is problematic. >>>>>>>>>>>> >>>>>>>>>>>> Definitely yes, we are on the same page. >>>>>>>>>>>> >>>>>>>>>>>> I think we have the same goal: propose a general and reasonable >>>>>>>>>>>> mechanism to make spark on k8s with a custom scheduler more usable. >>>>>>>>>>>> >>>>>>>>>>>> But for the PodGroup, just allow me to do a brief introduction: >>>>>>>>>>>> - The PodGroup definition has been approved by Kubernetes >>>>>>>>>>>> officially in KEP-583. [1] >>>>>>>>>>>> - It can be regarded as a general concept/standard in >>>>>>>>>>>> Kubernetes rather than a specific concept in Volcano, there are >>>>>>>>>>>> also others >>>>>>>>>>>> to implement it, such as [2][3]. >>>>>>>>>>>> - Kubernetes recommends using CRD to do more extension to >>>>>>>>>>>> implement what they want. [4] >>>>>>>>>>>> - Volcano as extension provides an interface to maintain the >>>>>>>>>>>> life cycle PodGroup CRD and use volcano-scheduler to complete the >>>>>>>>>>>> scheduling. >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling >>>>>>>>>>>> [2] >>>>>>>>>>>> https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling#podgroup >>>>>>>>>>>> [3] https://github.com/kubernetes-sigs/kube-batch >>>>>>>>>>>> [4] >>>>>>>>>>>> https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Yikun >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Weiwei Yang <w...@apache.org> 于2021年12月1日周三 上午5:57写道： >>>>>>>>>>>> >>>>>>>>>>>>> Hi Chenya >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for bringing this up. This is quite interesting, we >>>>>>>>>>>>> definitely should participate more in the discussions. >>>>>>>>>>>>> The main thing here is, the Spark community should make Spark >>>>>>>>>>>>> pluggable in order to support other schedulers, not just for >>>>>>>>>>>>> Volcano. It >>>>>>>>>>>>> looks like this proposal is pushing really hard for adopting >>>>>>>>>>>>> PodGroup, >>>>>>>>>>>>> which isn't part of K8s yet, that to me is problematic. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Nov 30, 2021 at 9:21 AM Prasad Paravatha < >>>>>>>>>>>>> prasad.parava...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> This is a great feature/idea. >>>>>>>>>>>>>> I'd love to get involved in some form (testing and/or >>>>>>>>>>>>>> documentation). This could be my 1st contribution to Spark! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Nov 30, 2021 at 10:46 PM John Zhuge < >>>>>>>>>>>>>> jzh...@apache.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 Kudos to Yikun and the community for starting the >>>>>>>>>>>>>>> discussion! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang < >>>>>>>>>>>>>>> chenyazhangche...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks folks for bringing up the topic of natively >>>>>>>>>>>>>>>> integrating Volcano and other alternative schedulers into >>>>>>>>>>>>>>>> Spark! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +Weiwei, Wilfred, Chaoran. We would love to contribute to >>>>>>>>>>>>>>>> the discussion as well. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From our side, we have been using and improving on one >>>>>>>>>>>>>>>> alternative resource scheduler, Apache YuniKorn ( >>>>>>>>>>>>>>>> https://yunikorn.apache.org/), for Spark on Kubernetes in >>>>>>>>>>>>>>>> production at Apple with solid results in the past year. It is >>>>>>>>>>>>>>>> capable of >>>>>>>>>>>>>>>> supporting Gang scheduling (similar to PodGroups), >>>>>>>>>>>>>>>> multi-tenant resource >>>>>>>>>>>>>>>> queues (similar to YARN), FIFO, and other handy features like >>>>>>>>>>>>>>>> bin packing >>>>>>>>>>>>>>>> to enable efficient autoscaling, etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Natively integrating with Spark would provide more >>>>>>>>>>>>>>>> flexibility for users and reduce the extra cost and potential >>>>>>>>>>>>>>>> inconsistency >>>>>>>>>>>>>>>> of maintaining different layers of resource strategies. One >>>>>>>>>>>>>>>> interesting >>>>>>>>>>>>>>>> topic we hope to discuss more about is dynamic allocation, >>>>>>>>>>>>>>>> which would >>>>>>>>>>>>>>>> benefit from native coordination between Spark and resource >>>>>>>>>>>>>>>> schedulers in >>>>>>>>>>>>>>>> K8s & cloud environment for an optimal resource efficiency. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Nov 30, 2021 at 8:10 AM Holden Karau < >>>>>>>>>>>>>>>> hol...@pigscanfly.ca> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for putting this together, I’m really excited for >>>>>>>>>>>>>>>>> us to add better batch scheduling integrations. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang < >>>>>>>>>>>>>>>>> yikunk...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hey everyone, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'd like to start a discussion on "Support >>>>>>>>>>>>>>>>>> Volcano/Alternative Schedulers Proposal". >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This SPIP is proposed to make spark k8s schedulers >>>>>>>>>>>>>>>>>> provide more YARN like features (such as queues and minimum >>>>>>>>>>>>>>>>>> resources >>>>>>>>>>>>>>>>>> before scheduling jobs) that many folks want on Kubernetes. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The goal of this SPIP is to improve current spark k8s >>>>>>>>>>>>>>>>>> scheduler implementations, add the ability of batch >>>>>>>>>>>>>>>>>> scheduling and support >>>>>>>>>>>>>>>>>> volcano as one of implementations. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Design doc: >>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg >>>>>>>>>>>>>>>>>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057 >>>>>>>>>>>>>>>>>> Part of PRs: >>>>>>>>>>>>>>>>>> Ability to create resources >>>>>>>>>>>>>>>>>> https://github.com/apache/spark/pull/34599 >>>>>>>>>>>>>>>>>> Add PodGroupFeatureStep: >>>>>>>>>>>>>>>>>> https://github.com/apache/spark/pull/34456 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Yikun >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>>>>>>>>>> YouTube Live Streams: >>>>>>>>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> John Zhuge >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Prasad Paravatha >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>> >>>> -- >>>> Twitter: https://twitter.com/holdenkarau >>>> Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> >>>
