Re: [openstack-dev] [sahara]FFE Request for nfs-as-a-data-source
Hi, Currently, there is no plan for other plugin support in this feature. We would like to put this feature on the table at first and see if it can bring more customers who are interested in Big Data on Cloud and expecting to integrate Hadoop with different storage type support. However, it’s just a beginning and should be worth a shot to bring it in Mitaka. And to support any other plugin is also still open and on-demand in the future. From: Vitaly Gridnev [mailto:vgrid...@mirantis.com] Sent: Wednesday, March 2, 2016 3:31 PM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [sahara]FFE Request for nfs-as-a-data-source Hi, From my point of view, if we adding new type of the datasources (or configurations for that), it means that it should be supported in almost all plugins (at least in vanilla, spark, ambari, cdh I guess). Current implementation is nice, but it looks like it touches only vanilla 2.7.1 plugin which strange for me. Are there plans to add support for other plugins? If yes, then I think this feature should be done in Newton cycle to have complete picture of this support. If no, I think it's ok to land this code in RC with other improvements in validation. At conclusion I would like to say that from point of view we should collaborate actively to implement this support in early Newton-1 cycle, that would be a best choice. Thanks. On Wed, Mar 2, 2016 at 4:23 AM, Chen, Weiting <weiting.c...@intel.com<mailto:weiting.c...@intel.com>> wrote: Hi all, I would like to request a FFE for the feature “nfs-as-a-data-source”: BP: https://blueprints.launchpad.net/sahara/+spec/nfs-as-a-data-source BP Review: https://review.openstack.org/#/c/210839/ Sahara Code: https://review.openstack.org/#/c/218638/ Sahara Image Elements Code: https://review.openstack.org/#/c/218637/ Estimate Complete Time: The BP has been complete and the implementation has been complete as well. All the code is under code reviewing and since there is no big change or modify for the code we expect it can only take one weeks to be merged. The Benefits for this change: Provide NFS support in Sahara. The Risk: The risk would be low for this patch, since all the functions have been delivered. Thanks, Weiting(William) Chen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Vitaly Gridnev Mirantis, Inc __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [sahara]FFE Request for nfs-as-a-data-source
Hi, It's different between Sahara and Manila for this feature support. This feature is to put NetApp Hadoop NFS Connector into Hadoop Cluster and let Hadoop can support NFS protocol. And it can also work with Manila NFS driver, since Manila only need to expose the NFS address from the storage side. Hadoop cluster can use this connector to communicate with NFS protocol directly. For example: /bin/hadoop jar terasort nfs://input_file nfs://output_file -Original Message- From: Monty Taylor [mailto:mord...@inaugust.com] Sent: Wednesday, March 2, 2016 9:33 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [sahara]FFE Request for nfs-as-a-data-source On 03/01/2016 07:23 PM, Chen, Weiting wrote: > Hi all, > > I would like to request a FFE for the feature "nfs-as-a-data-source": > > BP: https://blueprints.launchpad.net/sahara/+spec/nfs-as-a-data-source > > BP Review: https://review.openstack.org/#/c/210839/ Please forgive me for not having been involved with this at all... Wouldn't it make more sense to use Manila for this? I mean, they've got vendor drivers already, and this code says "Setup NetApp ..." - I imagine there are other NFS providers you'd want to use, no? Or ignore me - no worries - just a mailing list driveby > Sahara Code: https://review.openstack.org/#/c/218638/ > > Sahara Image Elements Code: https://review.openstack.org/#/c/218637/ > > Estimate Complete Time: The BP has been complete and the > implementation has been complete as well. All the code is under code > reviewing and since there is no big change or modify for the code we > expect it can only take one weeks to be merged. > > The Benefits for this change: Provide NFS support in Sahara. > > The Risk: The risk would be low for this patch, since all the > functions have been delivered. > > Thanks, > > Weiting(William) Chen > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [sahara]FFE Request for nfs-as-a-data-source
Hi all, I would like to request a FFE for the feature "nfs-as-a-data-source": BP: https://blueprints.launchpad.net/sahara/+spec/nfs-as-a-data-source BP Review: https://review.openstack.org/#/c/210839/ Sahara Code: https://review.openstack.org/#/c/218638/ Sahara Image Elements Code: https://review.openstack.org/#/c/218637/ Estimate Complete Time: The BP has been complete and the implementation has been complete as well. All the code is under code reviewing and since there is no big change or modify for the code we expect it can only take one weeks to be merged. The Benefits for this change: Provide NFS support in Sahara. The Risk: The risk would be low for this patch, since all the functions have been delivered. Thanks, Weiting(William) Chen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [sahara] FFE request for nfs-as-a-data-source
Hi, all. I would like to request FFE for nfs as a data source for sahara. This bp originally should include a dashboard change to create nfs as a data source. I will register it as another bp and implement it in next version. However, these patches have already done to put nfs-driver into sahara-image-elements and enable it in the cluster. By using this way, the user can use nfs protocol via command line in Liberty release. Blueprint: https://blueprints.launchpad.net/sahara/+spec/nfs-as-a-data-source Spec: https://review.openstack.org/#/c/210839/ Patch: https://review.openstack.org/#/c/218637/ https://review.openstack.org/#/c/218638/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] need help regarding openstack
Could you provide more detail log in sahara? Your situation usually is because the VMs cannot be ssh, so they are waiting for the VMs get ready. One thing you can do is to make sure you can ssh into the VM using private ip/floating ip. From: Deepika Agrawal [mailto:deepika...@gmail.com] Sent: Wednesday, April 8, 2015 12:11 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] need help regarding openstack hi guys! I installed Openstack and in that i installed Hadoop i.e., sahara for distributed storage. But the problem I am facing is that when i am going to run node cluster, system will go in the waiting state because I have only 4GB RAM in my laptop. and my college also dint provide me sufficient space. This is for my Btech project. Can Someone tell me what I can be done with open stack so that I'll show this to my mentors as my Btech project. I need help. Please help me. -- Deepika Agrawal __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] About Sahara EDP New Ideas for Liberty
Hi Andrew. Thanks for response. My reply in line. From: Andrew Lazarev [mailto:alaza...@mirantis.com] Sent: Saturday, March 21, 2015 12:10 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] About Sahara EDP New Ideas for Liberty Hi Weiting, 1. Add a schedule feature to run the jobs on time: This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run. Looks like a great feature. And should be quite easy to implement. Feel free to create spec for that. [Weiting] We are working on the spec and the bp has already been registered in https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs. 2. A more complex workflow design in Sahara EDP: Current EDP only provide one job that is running on one cluster. Yes. And ability to run several jobs in one oozie workflow is discussed on every summit (e.g. 'coordinated jobs' at https://etherpad.openstack.org/p/kilo-summit-sahara-edp). But for now it was not a priority But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it.. It means that workflow manager should be on Sahara side. Looks like a complicated feature. But we would be happy to help with designing and implementing it. Please file proposal for design session on ongoing summit. Are you going to Vancouver? [Weiting] I’m not sure I will be there because the plan is still not ready yet. We are also looking for some customer’s real case in big data area and see how they are using data processing in current environment. However, for any idea we can update later. Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios. If workflow is on Sahara side it should work automatically for all engines. [Weiting] Yes, agree. Thanks, Andrew. On Sun, Mar 8, 2015 at 3:17 AM, Chen, Weiting weiting.c...@intel.commailto:weiting.c...@intel.com wrote: Hi all. We got several feedbacks about Sahara EDP’s future from some China customers. Here are some ideas we would like to share with you and need your input if we can implement them in Sahara(Liberty). 1. Add a schedule feature to run the jobs on time: This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run. 2. A more complex workflow design in Sahara EDP: Current EDP only provide one job that is running on one cluster. But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it. For example: Raw Data - Job A(Cluster A) - Job B(Cluster B) - Job C(Cluster A) - Result Actually in my opinion, this kind of job could be easy to implement by using Oozie as a workflow engine. But for current EDP, it doesn’t implement this kind of complex case. Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios. However, any suggestion is welcome. Thanks. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] About Sahara EDP New Ideas for Liberty
Hi all. We got several feedbacks about Sahara EDP's future from some China customers. Here are some ideas we would like to share with you and need your input if we can implement them in Sahara(Liberty). 1. Add a schedule feature to run the jobs on time: This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run. 2. A more complex workflow design in Sahara EDP: Current EDP only provide one job that is running on one cluster. But in a real case, it should be more complex, they usually use multiple jobs to calculate the data and may use several different type clusters to process it. For example: Raw Data - Job A(Cluster A) - Job B(Cluster B) - Job C(Cluster A) - Result Actually in my opinion, this kind of job could be easy to implement by using Oozie as a workflow engine. But for current EDP, it doesn't implement this kind of complex case. Another concern is about Spark, for Spark it cannot use Oozie to do this. So we need to create an abstract layer to help to implement this kind of scenarios. However, any suggestion is welcome. Thanks. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
