Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

Thanks Abdur Rahmaan. I will give it a try ! Thanks Asif On Mon, Sep 30, 2024 at 11:19 AM Abdur-Rahmaan Janhangeer < arj.pyt...@gmail.com> wrote: > Idk if you tried Polars, but it seems to work well with JSON data > > import polars as pl > pl.read_json("file.json") > > Kind Regards, > > Abdur-Ra

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

Idk if you tried Polars, but it seems to work well with JSON data import polars as pl pl.read_json("file.json") Kind Regards, Abdur-Rahmaan Janhangeer about | blog github Mauritius On Mon,

Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

Dear Python Experts, I am working with the Kenna Application's API to retrieve vulnerability data. The API endpoint provides a single, massive JSON file in gzip format, approximately 60 GB in size. Handling such a large dataset in one go is proving to be quite challenging, especially in terms of m