Hi Alexandr, Have you read the DLO/SLO support in Swift ? That might the thing you can leverage for managing chunks. http://docs.openstack.org/developer/swift/overview_large_objects.html?highlight=large
Hugo 2016-03-11 16:40 GMT+08:00 Alexandr Porunov <alexandr.poru...@gmail.com>: > Hello Hugo, > > Thank you for your response. > > It is example of my architecture of video upload: > [image: Inline image 1] > 1. User uploads video chunks to an upload server. > 2. The upload server uploads each chunk to the temporary storage. (If > connection breaks then the user will be able to recover it and continue > download his video because his chunks already saved in the temporary > storage) > 3. After the video is fully saved, a upload server will send a task to a > queue to transform the video. > 4. A worker takes the task and transforms the video. > 5. The worker uploads the video to the permanent storage. > 6. The worker removes the video from temporary storage. > > Each chunk of the video is about 1 mb. > Maximum size of the video is 8 GB. > The upload server and the worker are written on Java. > Permanent storage - OpenStack Swift. > Transformation will starts only when full video will be uploaded. > > It will be great to use Swift as a temporary storage but I am not sure > that I can manage chunks in Swift. > If it is possible, please, give me any link to read about it. > > Best Regards, Alexandr > > > On Fri, Mar 11, 2016 at 6:15 AM, Kuo Hugo <tonyt...@gmail.com> wrote: > >> Hi Alexandr, >> >> The OpenStack Swift is a good choice as permanent storage pool for your >> video case. >> As for temporary storage media in your case, Swift could be an option. >> But you need to consider several factors. >> >> - How will user upload video to your site ? >> - How’s the speed do you need to store video chunks into temporary >> tier ? >> - What protocol is supported to store video by your application ? >> - Does your transformation server pull the data to it’s memory/local >> media to process ? >> - How many data will you have in your estimation ? >> - How’s the size of each video ingeneral ? >> >> Here’s a list of benefit by using Swift as both temporary & permanent >> storage pool : >> >> - User can upload video to Swift directly via HTTP. No need an >> additional server between your Swift pool and client. >> - Swift has built-in function for authentication/authorization. >> - If you’d like to move video cross temporary and permanent >> locations, it could be done in swift’s internal network. >> - The metadata can associate with object itself. >> - HTTP protocol gives high flexibility for various ways. >> - No video size limitation. >> - You can have 2 copies policy for temporary tier and 3 copies for >> permanent tier to get better data availability and durability. >> >> Here’s a list of potential weaknesses for temporary tier : >> >> - If your transformation server needs to edit data online and change >> bits on each chunk, it’s not good for Swift. >> >> If the temporary tier doesn’t need too much space (lower than 3TB), you >> can just use several high speed disks as a raid0 pool to have nice IOPs. >> >> Regard // Hugo >> >> >> 2016-03-10 18:54 GMT+08:00 Alexandr Porunov <alexandr.poru...@gmail.com>: >> >>> Hello, >>> >>> I need a temporary storage for users videos. >>> This is my scenario: >>> 1. User uploads a video file chunk by chunk. >>> 2. My application stores each chunk in the temporary storage. >>> 3. When video loading is finished then my transformation server >>> transforms this video and stores into permanent storage. >>> 4. After video was saved in permanent storage it is deleted from >>> temporary storage. >>> >>> Which storage is the best for read/write/delete only once and has the >>> ability to save files chunk by chunk? Swift? Cinder? or something else? >>> >>> Sincerely, Alexandr >>> >>> _______________________________________________ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> >> > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack