What kind of data do you store in that file? if this is someting that can be grouped by date, i'd say break it up into smaller files like yyyymm.json. you might want to check this out as well: http://cookbooks.adobe.com/post_Improving_speed_of_JSON_data_parsing-13226.html
On Tue, Dec 1, 2009 at 9:46 AM, waseem sabjee <waseemsab...@gmail.com>wrote: > if the user has to upload the file to you, you read it, edit and then save > it and promt the user to download it - i would say thats an effecient > process...until you deal with a huge file. > > is the user uploading the file to you the only method you are willing to > use to retreive the data ? > > if you know how to build windows or linux applications you could use that > so the user sends the data directly. > this could redice bandwidth usage. > > > On Tue, Dec 1, 2009 at 4:37 PM, km <srikrishnamo...@gmail.com> wrote: > >> well i got it. >> On the server side, I can open the json file (100MB) and read the data >> structure directly into python and process required fields; - then return >> them to the client. but 1) persistence ? - does the 100MB file be loaded in >> memory always ? or 2) loaded multiple times based on request ? >> so is there a memory efficient way of reading big json files ? >> >> thanks, >> Krishna >> >> >> On Tue, Dec 1, 2009 at 10:50 PM, waseem sabjee <waseemsab...@gmail.com>wrote: >> >>> one way this can be achieved. a mixture of server side and client side >>> code. >>> >>> set up a seperate file or web service to get your data. >>> >>> make an ajax call to this file passing your parameters via the call. >>> on success of that call, if the data your requested is returned, make >>> another ajax call with new parameters. >>> >>> say my first call was to get the data rows 1- 50. >>> my second was to ge rows 51 - 100 >>> >>> do you mean something like that ? >>> >>> 100Mb is a bit much for client side scripting though. >>> >>> i would suggest using a server side script and cleaning server side >>> memory when required. >>> here you would be just eating your users bandwidth...eating it like >>> theres no tomorow. >>> >>> >>> On Tue, Dec 1, 2009 at 3:44 PM, MorningZ <morni...@gmail.com> wrote: >>> >>>> Let's put it simply, JavaScript code just isn't that smart.... >>>> >>>> Your client-side code (1) makes a request, then (2) the server >>>> responds, ** that's it **.......... as the person above me suggests, >>>> use your server side code, the one providing the "big JSON" data to do >>>> the filtering >>>> >>>> On Dec 1, 3:44 am, km <srikrishnamo...@gmail.com> wrote: >>>> > Hi all, >>>> > >>>> > I am currently using $.getJSON to load a big JSON format file >>>> (100MB). >>>> > So is there a way to selectively parse a few fields of the JSON file >>>> so that >>>> > the full file doesnt get loaded in memory ? >>>> > In summary i am looking for parsing a few keys in the JSON file and >>>> fetch >>>> > those values only to display on the webpage. >>>> > >>>> > any ideas ? >>>> > thanks, >>>> > >>>> > regards >>>> > Krishna >>>> >>> >>> >> >
