and use ssd for maximum I/O as youd creating and deleting files a lot, move session from db to something like redis and python has concurrency libraries make use of them as well.
On Fri, 18 Oct 2019, 14:29 vineet daniel, <vineetdan...@gmail.com> wrote: > Dont forget inode and files limit if you are creating too many files real > quickly. > > On Fri, 18 Oct 2019, 13:15 James Gutu, <jmg...@gmail.com> wrote: > >> Jack, >> That sounds about right. >> >> On Thu, Oct 17, 2019 at 11:22 PM Jack Sundberg <jacksundberg...@gmail.com> >> wrote: >> >>> Thank you, James! This seems like a good fix. From reading through the >>> documentation (here >>> <https://docs.djangoproject.com/en/2.2/topics/http/sessions/>), this is >>> how I understand the workflow -- so correct me if I'm wrong please! >>> >>> > First, I can activate Django sessions by >>> adding 'django.contrib.sessions' to my installed apps. >>> > Django by-default stores session info in the SQLite database -- >>> therefore, each session entry will have an ID assigned to it automatically. >>> > I can use each session_id to create a temporary folder in some >>> directory (say /tmp) where the generated files can be made. So folder >>> structure will look like: >>> tmp/ >>> [session_id1]/ >>> file1.tlgf >>> file2.bin >>> [session_id2]/ >>> file1.tlgf >>> file2.bin >>> .... >>> > when sessions data hits a certain threshold, I can call >>> django.contrib.sessions.clearsessions which will delete expired session >>> entries in the database >>> > pull out which entries were deleted and then delete the corresponding >>> tmp/[session_id] directory >>> >>> Is that the best way to do it? >>> >>> -Jack >>> >>> >>> On Thursday, October 17, 2019 at 4:42:04 PM UTC-4, James Gutu wrote: >>>> >>>> Jack, >>>> >>>> From what I understand it looks like a conflict of uniquely named files >>>> when they are generated simultaneously. >>>> Could you resolve this by creating a folder named from the session_id, >>>> with the file inside of that. So that if two users request the same file, >>>> there will never be a conflict, as the unique file will be created in both >>>> folders. >>>> >>>> [user_1_session_id]/entry123.jpeg >>>> [user_2_session_id]/entry123.jpeg >>>> >>>> Though, this might seem like a fix, but the issue may be more >>>> fundamental than file names. >>>> Let me know what you think. >>>> >>>> Regards, >>>> James >>>> >>>> On Thu, Oct 17, 2019 at 4:23 PM Jack Sundberg <jacksun...@gmail.com> >>>> wrote: >>>> >>>>> Yes, I tried getting the idea across in layman terms, as I hope this >>>>> helps discussion of the issue -- and then I can later translate the >>>>> solution into django implementation. >>>>> >>>>> I avoided the full process in my original post for clarity. I used the >>>>> *.jpeg file as an illustrative example, when I'm actually generating >>>>> *.gltf >>>>> and *.bin files. These files are typically a couple hundred KB in size, >>>>> but >>>>> I will have thousands of them (millions as I add to my database). These >>>>> files are used in coordination with Verge3D - an app built on top of >>>>> Three.js for 3D rendering. Verge3D allows use of Blender for CAD, which is >>>>> a python-based program that my scripts are written in. Running of these >>>>> scripts are certainly cpu intensive; however I am going to get around this >>>>> by using an <iframe> of this Verge3D request. >>>>> >>>>> So to be more specific, my app does the following: >>>>> - the url mapping pulls a database entry >>>>> - the app serves the proper html template with an <iframe> embedded >>>>> in it. It displays a loading screen while the 3D model is created backend. >>>>> - the <iframe> calls a separate app using the identical data entry >>>>> - using the data entry, static files are created from the data >>>>> (Blender script creates a *.gltf and *.bin) >>>>> - the app serves the html template with the generated static files >>>>> embedded inside the <iframe> >>>>> >>>>> Is this helpful? Also thank you for the quick response. I really >>>>> appreciate the help. >>>>> >>>>> -Jack >>>>> >>>>> >>>>> On Thursday, October 17, 2019 at 10:56:56 AM UTC-4, vineet daniel >>>>> wrote: >>>>>> >>>>>> This problem can happen with any stack and shouldn't be considered as >>>>>> django specific. What you are looking for is a solution in a layman >>>>>> terms.which then can be translated to django implementation. >>>>>> >>>>>> Before that i would like to know image size and kind of image that >>>>>> you create. Is it some qr code or what? >>>>>> >>>>>> Please note that working with images on the fly is always a cpu >>>>>> intensive operation. Ill can come up with something once you share the >>>>>> details with me. >>>>>> >>>>>> On Thu, 17 Oct 2019, 20:00 Jack Sundberg, <jacksun...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I am currently writing an app that generates static files before >>>>>>> serving them, and I could use some input! Here's a step-by-step example >>>>>>> of >>>>>>> how the app is setup: >>>>>>> >>>>>>> - the url mapping pulls a database entry >>>>>>> - using the data entry, static files are created from the data (for >>>>>>> example, a *.jpeg or a *.gltf) >>>>>>> - the app serves the html template with the generated static files >>>>>>> embedded (for example, via an <img> tag) >>>>>>> >>>>>>> The issue comes with the creation of the unique static files. I can >>>>>>> have the app create a file in the static folder, send it off, then >>>>>>> delete >>>>>>> the file to ready for the next request -- but the webpage will run into >>>>>>> issues if multiple requests are made at the same time. If two people >>>>>>> request the same url at the same time, the app will fail because both >>>>>>> requests are trying to create/serve/delete an identical file (i.e. >>>>>>> entry123.jpeg) at the same time. Trying to mess with the filename to >>>>>>> avoid >>>>>>> this won't work here either (i.e. naming entry123.01.jpeg, >>>>>>> entry123.02.jpeg, entry123.03.jpeg, ...). >>>>>>> >>>>>>> I can think of two solutions, but they both are things I'd like to >>>>>>> avoid: >>>>>>> 1. have the files premade and stored -- I want to avoid this >>>>>>> because of the massive filesize required for all these files >>>>>>> 2. have the static files generated in a user-specific directory -- >>>>>>> I want to avoid this because I'd rather not have registered users on my >>>>>>> website >>>>>>> >>>>>>> Is there another solution to this? I'm still relatively new with >>>>>>> Django, so there might be something simple I'm missing. >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Django users" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to django...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/django-users/190637d2-bd9b-499a-a122-1dc7adf2aca5%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/django-users/190637d2-bd9b-499a-a122-1dc7adf2aca5%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Django users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to django...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/django-users/317c4ce6-5378-4e59-920e-c0fdcacd453a%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/django-users/317c4ce6-5378-4e59-920e-c0fdcacd453a%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-users+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-users/20e29fca-88db-4454-bb7d-d25bd8271ca3%40googlegroups.com >>> <https://groups.google.com/d/msgid/django-users/20e29fca-88db-4454-bb7d-d25bd8271ca3%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-users/CAOnWGOLGs3%2Bamtsvo_btgQ4RVL1fVsP-WO%2BQurTN050nXNgYJQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/django-users/CAOnWGOLGs3%2Bamtsvo_btgQ4RVL1fVsP-WO%2BQurTN050nXNgYJQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAJPbAW_A0SO1pUnO1Df0ED1ohtqgVd-vG4mO13PV8bDcbdTu_A%40mail.gmail.com.
