Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-02 Thread Chris Angelico via Python-list
On Thu, 3 Oct 2024 at 08:48, Left Right wrote: > > > You can't validate an IP packet without having all of it. Your notion > > of "streaming" is nonsensical. > > Whoa, whoa, hold your horses! "nonsensical" needs a little bit of > justification :) > > It seems you don't understand the difference be

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-02 Thread Chris Angelico via Python-list
On Wed, 2 Oct 2024 at 23:53, Left Right via Python-list wrote: > In the same email you replied to, I gave examples of languages for > which parsers can be streaming (in general): SCSI or IP. You can't validate an IP packet without having all of it. Your notion of "streaming" is nonsensical. Chri

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-02 Thread Left Right via Python-list
parser that is only useful _sometimes_. And, in practice, languages like XML or JSON do well with streaming, even though in general it's impossible. I'm sorry if this comes as a surprise. On one hand I don't want to sound condescending, on the other hand, this is something that you&

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Greg Ewing via Python-list
On 2/10/24 12:26 pm, avi.e.gr...@gmail.com wrote: The real problem is how the JSON is set up. If you take umpteen data structures and wrap them all in something like a list, then it may be a tad hard to stream as you may not necessarily be examining the contents till the list finishes gigabytes

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Dan Sommers via Python-list
relevant; the point is that it is possible to parse integers from most significant digit to least significant digit under a streaming model (and if you're sufficiently clever, you can even write partial results to external storage and/or another transmission protocol, thus allowing for numbers

RE: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread AVI GROSS via Python-list
generators to give you as many primes as you want, and no more. So, if you can store arbitrary python code as part of your JSON, you can send quite a bit of somewhat compressed data. The real problem is how the JSON is set up. If you take umpteen data structures and wrap them all in something like

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Greg Ewing via Python-list
On 2/10/24 10:03 am, Left Right wrote: Consider also an interesting consequence of SCSI not being able to have infinite words: this means, besides other things that fsync() is nonsense! :) If you aren't familiar with the concept: UNIX filesystem API suggests that it's possible to destage arbitrar

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Greg Ewing via Python-list
On 1/10/24 8:34 am, Left Right wrote: You probably forgot that it has to be _streaming_. Suppose you parse the first digit: can you hand this information over to an external function to process the parsed data? -- No! because you don't know the magnitude yet. By that definition of "streaming",

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Left Right via Python-list
to sync _everything_ (and it hurts!) On Tue, Oct 1, 2024 at 5:49 PM Dan Sommers via Python-list wrote: > > On 2024-09-30 at 21:34:07 +0200, > Regarding "Re: Help with Streaming and Chunk Processing for Large JSON Data > (60 GB) from Kenna API," > Left Right via Python-lis

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Dan Sommers via Python-list
On 2024-09-30 at 21:34:07 +0200, Regarding "Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API," Left Right via Python-list wrote: > > What am I missing? Handwavingly, start with the first digit, and as > > long as the next character

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Dan Sommers via Python-list
On 2024-09-30 at 18:48:02 -0700, Keith Thompson via Python-list wrote: > 2qdxy4rzwzuui...@potatochowder.com writes: > [...] > > In Common Lisp, you can write integers as #nnR[digits], where nn is the > > decimal representation of the base (possibly without a leading zero), > > the # and the R are

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Left Right via Python-list
> What am I missing? Handwavingly, start with the first digit, and as > long as the next character is a digit, multipliy the accumulated result > by 10 (or the appropriate base) and add the next value. Oh, and handle > scientific notation as a special case, and perhaps fail spectacularly > instea

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-10-01 Thread Keith Thompson via Python-list
2qdxy4rzwzuui...@potatochowder.com writes: [...] > In Common Lisp, you can write integers as #nnR[digits], where nn is the > decimal representation of the base (possibly without a leading zero), > the # and the R are literal characters, and the digits are written in > the intended base. So the inp

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Dan Sommers via Python-list
On 2024-10-01 at 09:09:07 +1000, Chris Angelico via Python-list wrote: > On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list > wrote: > > > > On 2024-09-30, Dan Sommers via Python-list wrote: > > > > > In Common Lisp, integers can be written in any integer base from two > > > to thirty s

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Chris Angelico via Python-list
On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list wrote: > > On 2024-09-30, Dan Sommers via Python-list wrote: > > > In Common Lisp, integers can be written in any integer base from two > > to thirty six, inclusive. So knowing the last digit doesn't tell > > you whether an integer is ev

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Grant Edwards via Python-list
On 2024-09-30, Dan Sommers via Python-list wrote: > In Common Lisp, integers can be written in any integer base from two > to thirty six, inclusive. So knowing the last digit doesn't tell > you whether an integer is even or odd until you know the base > anyway. I had to think about that for an

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Dan Sommers via Python-list
ny integer base from two to thirty six, inclusive. So knowing the last digit doesn't tell you whether an integer is even or odd until you know the base anyway. Curiously, we agree: if you move the goal posts arbitrarily, then some algorithms that parse JSON numbers will fail. -- https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Thomas Passin via Python-list
streaming parser is required. There is also the json-stream library, on PyPi at https://pypi.org/project/json-stream/ -- https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Left Right via Python-list
> Streaming won't work because the file is gzipped. You have to receive > the whole thing before you can unzip it. Once unzipped it will be even > larger, and all in memory. GZip is specifically designed to be streamed. So, that's not a problem (in principle), but you would need to have a stream

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Thomas Passin via Python-list
On 9/30/2024 1:00 PM, Chris Angelico via Python-list wrote: On Tue, 1 Oct 2024 at 02:20, Thomas Passin via Python-list wrote: On 9/30/2024 11:30 AM, Barry via Python-list wrote: On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list wrote: import polars as pl pl.read_json("

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Chris Angelico via Python-list
On Tue, 1 Oct 2024 at 04:30, Dan Sommers via Python-list wrote: > > But why do I need to start with the least > significant digit? If you start from the most significant, you don't know anything about the number until you finish parsing it. There's almost nothing you can say about a number given

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Grant Edwards via Python-list
On 2024-09-30, Dan Sommers via Python-list wrote: > On 2024-09-30 at 11:44:50 -0400, > Grant Edwards via Python-list wrote: > >> On 2024-09-30, Left Right via Python-list wrote: >> > [...] >> > Imagine a pathological case of this shape: 1... <60GB of digits&g

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Dan Sommers via Python-list
On 2024-09-30 at 11:44:50 -0400, Grant Edwards via Python-list wrote: > On 2024-09-30, Left Right via Python-list wrote: > > Whether and to what degree you can stream JSON depends on JSON > > structure. In general, however, JSON cannot be streamed (but commonly > > it can b

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Chris Angelico via Python-list
On Tue, 1 Oct 2024 at 02:20, Thomas Passin via Python-list wrote: > > On 9/30/2024 11:30 AM, Barry via Python-list wrote: > > > > > >> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list > >> wrote: > >> > >> > >> import polars as pl > >> pl.read_json("file.json") > >> > >> > > >

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Thomas Passin via Python-list
On 9/30/2024 11:30 AM, Barry via Python-list wrote: On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list wrote: import polars as pl pl.read_json("file.json") This is not going to work unless the computer has a lot more the 60GiB of RAM. As later suggested a streaming par

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Grant Edwards via Python-list
On 2024-09-30, Left Right via Python-list wrote: > Whether and to what degree you can stream JSON depends on JSON > structure. In general, however, JSON cannot be streamed (but commonly > it can be). > > Imagine a pathological case of this shape: 1... <60GB of digits>. Th

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Barry via Python-list
> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list > wrote: > > > import polars as pl > pl.read_json("file.json") > > This is not going to work unless the computer has a lot more the 60GiB of RAM. As later suggested a streaming parser is required. Barry -- https://m

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-30 Thread Left Right via Python-list
Whether and to what degree you can stream JSON depends on JSON structure. In general, however, JSON cannot be streamed (but commonly it can be). Imagine a pathological case of this shape: 1... <60GB of digits>. This is still a valid JSON (it doesn't have any limits on how many digits a

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-29 Thread Asif Ali Hirekumbi via Python-list
Thanks Abdur Rahmaan. I will give it a try ! Thanks Asif On Mon, Sep 30, 2024 at 11:19 AM Abdur-Rahmaan Janhangeer < arj.pyt...@gmail.com> wrote: > Idk if you tried Polars, but it seems to work well with JSON data > > import polars as pl > pl.read_json("file.json")

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-29 Thread Abdur-Rahmaan Janhangeer via Python-list
Idk if you tried Polars, but it seems to work well with JSON data import polars as pl pl.read_json("file.json") Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://www.pythonkitchen.com> github <https://github.com/Abdur-Rah

Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

2024-09-29 Thread Asif Ali Hirekumbi via Python-list
Dear Python Experts, I am working with the Kenna Application's API to retrieve vulnerability data. The API endpoint provides a single, massive JSON file in gzip format, approximately 60 GB in size. Handling such a large dataset in one go is proving to be quite challenging, especially in ter

Marshmallow: json-to-schema helper?

2022-04-04 Thread Albert-Jan Roskam
Hi, I'm looking for a convenience function to convert a Marshmallow schema into a valid Python class definition. That is, I want to generate python code (class MySchema.. etc) that I could write to a .py file. Does this exist? I tried the code below, but that is not the intended use

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-03-02 Thread Loris Bennett
acters each per event. The traffic on the DB is going to be >>very low, creating maybe a dozen events a day, mainly triggered via a >>command-line interface, although I will probably set up one or two cron >>jobs, each of which might generate another 0 to maybe 5 records a day. &

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-03-01 Thread Loris Bennett
t; very low, creating maybe a dozen events a day, mainly triggered via a >> command-line interface, although I will probably set up one or two cron >> jobs, each of which might generate another 0 to maybe 5 records a day. >> >> I could go for JSON (or rather LONGSTRING, as J

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-03-01 Thread Robert Latest via Python-list
day, mainly triggered via a > command-line interface, although I will probably set up one or two cron > jobs, each of which might generate another 0 to maybe 5 records a day. > > I could go for JSON (or rather LONGSTRING, as JSON is just an alias for > LONGSTRING, but JSON is not

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-03-01 Thread Loris Bennett
hich I want to add in the field 'info'. This data never needs to be >>modified, once the event has been inserted into the DB. >> >>What type should the info field have? JSON, PickleType, String, or >>something else? > > I would use JSON, it expresses dicts well

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-02-28 Thread Cameron Simpson
ullable=False) > uid = Column('gid', String(64), ForeignKey('users.uid'), nullable=False) > info = ?? > >The event may have arbitrary, but dict-like data associated with it, >which I want to add in the field 'info'. This data never needs to be >mod

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-02-28 Thread Greg Ewing
ned to use JSON if the data is something that can be easily represented that way. -- Greg -- https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-02-28 Thread Robert Latest via Python-list
Albert-Jan Roskam wrote: > The event may have arbitrary, but dict-like data associated with it, > which I want to add in the field 'info'.  This data never needs to be > modified, once the event has been inserted into the DB. > > What type should t

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-02-28 Thread Albert-Jan Roskam
needs to be modified, once the event has been inserted into the DB. What type should the info field have?  JSON, PickleType, String, or something else? I couldn't find any really reliable sounding information about the relative pros and cons, apart from a Red

SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

2022-02-28 Thread Loris Bennett
ForeignKey('users.uid'), nullable=False) info = ?? The event may have arbitrary, but dict-like data associated with it, which I want to add in the field 'info'. This data never needs to be modified, once the event has been inserted into the DB. What type should the info field hav

Re: Fwd: How to specify JSON parameters in CallBack?

2020-12-13 Thread Chris Angelico
On Mon, Dec 14, 2020 at 4:57 AM Ethan Furman wrote: > > From: Caleb Gattegno > Date: Fri, Dec 11, 2020 at 5:02 PM > Subject: How to specify JSON parameters in CallBack? > > > Please can you suggest where should I look for advice on converting a old > style web app which v

Fwd: Fwd: How to specify JSON parameters in CallBack?

2020-12-13 Thread Ethan Furman
From: Caleb Gattegno Date: Fri, Dec 11, 2020 at 5:02 PM Subject: How to specify JSON parameters in CallBack? Please can you suggest where should I look for advice on converting a old style web app which vends whole pages of html with a cgi-bin/python script invoked bypython3 server.py, into

Re: Unable to pass dict from json.

2020-12-13 Thread Grant Edwards
On 2020-12-13, Bischoop wrote: > Here https://bpa.st/YBVA Don't do that. Include in your post a short example that illustrates your questions. > I've working code with dictionary only if used dict from the code > [...] -- https://mail.python.org/mailman/listinfo/python-list

Re: Unable to pass dict from json.

2020-12-12 Thread MRAB
On 2020-12-13 00:01, Bischoop wrote: Here https://bpa.st/YBVA I've working code with dictionary only if used dict from the code (it's commented now) but when I load it I can print it from load function (line 14) but at all have not a clue how to pass the data so could use them. I've learnt a

Re: Unable to pass dict from json.

2020-12-12 Thread Dan Stromberg
On Sat, Dec 12, 2020 at 4:05 PM Bischoop wrote: > > Here https://bpa.st/YBVA I've working code with dictionary only if used > dict from the code > (it's commented now) but when I load it I can print it from load > function (line 14) but at all have not a > clue how to pass the data so could use

Unable to pass dict from json.

2020-12-12 Thread Bischoop
Here https://bpa.st/YBVA I've working code with dictionary only if used dict from the code (it's commented now) but when I load it I can print it from load function (line 14) but at all have not a clue how to pass the data so could use them. I've learnt a lot when making it but here I'm comple

a bit feedback for JSON documents

2020-07-03 Thread 황병희
There is some comment for https://docs.python.org/3.10/library/json.html. The latest RFC number is 8259. Thanks, Sincerely, JSON fan Byung-Hee -- ^고맙습니다 _白衣從軍_ 감사합니다_^))// -- https://mail.python.org/mailman/listinfo/python-list

Re: How to load cookies from a json input in python-requests?

2019-08-12 Thread Alexandre Brault
On 2019-08-12 11:38 p.m., Peng Yu wrote: ``` import requests s = requests.Session() import json s.cookies.set_cookie(requests.utils.cookiejar_from_dict(json.load(sys.stdin))) ``` I used the above command to load cookies from a json file. But I got the following error. Does anybody know how to

How to load cookies from a json input in python-requests?

2019-08-12 Thread Peng Yu
``` import requests s = requests.Session() import json s.cookies.set_cookie(requests.utils.cookiejar_from_dict(json.load(sys.stdin))) ``` I used the above command to load cookies from a json file. But I got the following error. Does anybody know how to fix the error? Thanks. ``` Traceback (most

Re: Decoding a huge JSON file incrementally

2018-12-20 Thread Terry Reedy
On 12/20/2018 10:42 AM, Paul Moore wrote: I'm looking for a way to incrementally decode a JSON file. I know this has come up before, and in general the problem is not soluble (because in theory the JSON file could be a single object). AFAIK, a JSON file always represents a single JSON ite

Re: Decoding a huge JSON file incrementally

2018-12-20 Thread Chris Angelico
On Fri, Dec 21, 2018 at 4:30 AM Paul Moore wrote: > On Thu, 20 Dec 2018 at 17:22, Chris Angelico wrote: > > Proper error handling is left as an exercise for the reader, both in > > terms of JSON errors and file errors. Also, the code is completely > > untested. Have fun

Re: Decoding a huge JSON file incrementally

2018-12-20 Thread Paul Moore
(Sorry, hit "Send" too soon on the last try!) On Thu, 20 Dec 2018 at 17:22, Chris Angelico wrote: > > On Fri, Dec 21, 2018 at 2:44 AM Paul Moore wrote: > > > > I'm looking for a way to incrementally decode a JSON file. I know this > > has come up be

Re: Decoding a huge JSON file incrementally

2018-12-20 Thread Paul Moore
On Thu, 20 Dec 2018 at 17:22, Chris Angelico wrote: > > On Fri, Dec 21, 2018 at 2:44 AM Paul Moore wrote: > > > > I'm looking for a way to incrementally decode a JSON file. I know this > > has come up before, and in general the problem is not soluble (because > &g

Re: Decoding a huge JSON file incrementally

2018-12-20 Thread Chris Angelico
On Fri, Dec 21, 2018 at 2:44 AM Paul Moore wrote: > > I'm looking for a way to incrementally decode a JSON file. I know this > has come up before, and in general the problem is not soluble (because > in theory the JSON file could be a single object). In my particular > situati

Decoding a huge JSON file incrementally

2018-12-20 Thread Paul Moore
I'm looking for a way to incrementally decode a JSON file. I know this has come up before, and in general the problem is not soluble (because in theory the JSON file could be a single object). In my particular situation, though, I have a 9GB file containing a top-level array object, with

Re: multiple JSON documents in one file, change proposal

2018-12-03 Thread Rhodri James
On 01/12/2018 10:10, Marko Rauhamaa wrote: In my case, I need an explicit terminator marker to know when a JSON value is complete. For example, if I should read from a socket: 123 [snip] You're having to invent a transport protocol to encapsulate your data protocol because you

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Akkana Peck
of libraries to deal with them. Then the data files wouldn't be human readable, making debugging a lot more hassle. Cameron Simpson writes: > There's a common format called Newline Delimited JSON (NDJSON) for just this > need. > > Just format the outbound records as JSON

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Grant Edwards
On 2018-11-30, Marko Rauhamaa wrote: > Paul Rubin : >> Maybe someone can convince me I'm misusing JSON but I often want to >> write out a file containing multiple records, and it's convenient to >> use JSON to represent the record data. >> >> The obvious

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Ben Bacarisse
Marko Rauhamaa writes: > Chris Angelico : > >> On Sat, Dec 1, 2018 at 10:16 PM Marko Rauhamaa wrote: >>> and the framing format is HTTP. I will need to type something like this: >>> >>>POST / HTTP/1.1^M >>>Host: localhost^M >>>

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Marko Rauhamaa
Chris Angelico : > On Sat, Dec 1, 2018 at 10:16 PM Marko Rauhamaa wrote: >> and the framing format is HTTP. I will need to type something like this: >> >>POST / HTTP/1.1^M >>Host: localhost^M >> Content-type: application/json^M &g

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Chris Angelico
gt; > > > I dunno, I type HTTP manually often enough that it can't be all *that* > > horrible. > > Say I want to send this piece of JSON: > >{ >"msgtype": "echo-req", >"opid": 3487547843 >} > > an

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Marko Rauhamaa
x. HTTP would be >> horrible to have to type even without the content-length problem, but >> BEEP (RFC 3080) would suffer from the content-length (and CRLF!) >> issue as well. > > I dunno, I type HTTP manually often enough that it can't be all *that* > horrible. Say I

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Chris Angelico
On Sat, Dec 1, 2018 at 9:16 PM Marko Rauhamaa wrote: > > Paul Rubin : > > > Marko Rauhamaa writes: > >> Having rejected different options ( >> https://en.wikipedia.org/wiki/JSON_streaming>), I settled with > >> terminating each JSON value with an ASC

Re: multiple JSON documents in one file, change proposal

2018-12-01 Thread Marko Rauhamaa
Paul Rubin : > Marko Rauhamaa writes: >> Having rejected different options (> https://en.wikipedia.org/wiki/JSON_streaming>), I settled with >> terminating each JSON value with an ASCII NUL character, which is >> illegal in JSON proper. > > Thanks, that Wikipedi

Re: multiple JSON documents in one file, change proposal

2018-11-30 Thread Thomas Jollans
On 30/11/2018 23:40, Marko Rauhamaa wrote: > Paul Rubin : >> Maybe someone can convince me I'm misusing JSON but I often want to >> write out a file containing multiple records, and it's convenient to >> use JSON to represent the record data. >> >> The ob

Re: multiple JSON documents in one file, change proposal

2018-11-30 Thread Chris Angelico
On Sat, Dec 1, 2018 at 9:46 AM Marko Rauhamaa wrote: > > Paul Rubin : > > Maybe someone can convince me I'm misusing JSON but I often want to > > write out a file containing multiple records, and it's convenient to > > use JSON to represent the record data. >

Re: multiple JSON documents in one file, change proposal

2018-11-30 Thread Cameron Simpson
On 01Dec2018 00:40, Marko Rauhamaa wrote: Paul Rubin : Maybe someone can convince me I'm misusing JSON but I often want to write out a file containing multiple records, and it's convenient to use JSON to represent the record data. The obvious way to read a JSON doc from a fi

Re: multiple JSON documents in one file, change proposal

2018-11-30 Thread Marko Rauhamaa
Paul Rubin : > Maybe someone can convince me I'm misusing JSON but I often want to > write out a file containing multiple records, and it's convenient to > use JSON to represent the record data. > > The obvious way to read a JSON doc from a file is with "json.load(

Re: Data Integrity Parsing json

2018-04-26 Thread Alister via Python-list
On Wed, 25 Apr 2018 14:32:01 -0700, Sayth Renshaw wrote: > Hi > > Hoping for guidance trying to find some advanced articles or guides or > topics for json parsing. > > I can parse and extract json just dandy. > > What I am trying to figure out is how I give myself

Re: Data Integrity Parsing json

2018-04-25 Thread Sayth Renshaw
On Thursday, 26 April 2018 07:57:28 UTC+10, Paul Rubin wrote: > Sayth Renshaw writes: > > What I am trying to figure out is how I give myself surety that the > > data I parse out is correct or will fail in an expected way. > > JSON is messier than people think. Here&#

Re: Putting Unicode characters in JSON

2018-03-24 Thread Peter J. Holzer
On 2018-03-25 06:30:54 +1100, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 3:35 AM, Peter J. Holzer wrote: > > On 2018-03-24 11:21:09 +1100, Chris Angelico wrote: > >> If the database has been configured to use UTF-8 (as mentioned, that's > >> "utf8mb4" in MySQL), you won't get that byte sequen

Re: Putting Unicode characters in JSON

2018-03-24 Thread Chris Angelico
On Sun, Mar 25, 2018 at 3:35 AM, Peter J. Holzer wrote: > On 2018-03-24 11:21:09 +1100, Chris Angelico wrote: >> If the database has been configured to use UTF-8 (as mentioned, that's >> "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get >> back valid UTF-8. > > Actually (with

Re: Putting Unicode characters in JSON

2018-03-24 Thread Peter J. Holzer
On 2018-03-24 11:21:09 +1100, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 11:11 AM, Steven D'Aprano > wrote: > > On Fri, 23 Mar 2018 07:46:16 -0700, Tobiah wrote: > >> If I changed my database tables to all be UTF-8 would this work cleanly > >> without any decoding? > > > > Not reliably or saf

Re: Putting Unicode characters in JSON

2018-03-23 Thread Steven D'Aprano
On Sat, 24 Mar 2018 11:21:09 +1100, Chris Angelico wrote: >>> If I changed my database tables to all be UTF-8 would this work >>> cleanly without any decoding? >> >> Not reliably or safely. It will appear to work so long as you have only >> pure ASCII strings from the database, and then crash when

Re: Putting Unicode characters in JSON

2018-03-23 Thread Chris Angelico
(most recent call last): > File "", line 1, in > File "/usr/local/lib/python2.7/json/__init__.py", line 231, in dumps > return _default_encoder.encode(obj) > File "/usr/local/lib/python2.7/json/encoder.py", line 195, in encode > return e

Re: Putting Unicode characters in JSON

2018-03-23 Thread Steven D'Aprano
#x27;t: py> text_from_database = u"hello wörld".encode('latin1') py> print text_from_database hello w�rld py> json.dumps(text_from_database) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/json/__init__.py"

Re: Putting Unicode characters in JSON

2018-03-23 Thread Chris Angelico
On Sat, Mar 24, 2018 at 1:46 AM, Tobiah wrote: > On 03/22/2018 12:46 PM, Tobiah wrote: >> >> I have some mailing information in a Mysql database that has >> characters from various other countries. The table says that >> it's using latin-1 encoding. I want

Re: Putting Unicode characters in JSON

2018-03-23 Thread Grant Edwards
On 2018-03-23, Richard Damon wrote: > One comment on this whole argument, the original poster asked how to get > data from a database that WAS using Latin-1 encoding into JSON (which > wants UTF-8 encoding) and was asking if something needed to be done > beyond using .decode(&#x

Re: Putting Unicode characters in JSON

2018-03-23 Thread Grant Edwards
On 2018-03-23, Chris Angelico wrote: > On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano > wrote: >> On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> >>>> I was reading though, that JSON files must be encoded with UTF-8. So >>>> should

Re: Putting Unicode characters in JSON

2018-03-23 Thread Tobiah
On 03/22/2018 12:46 PM, Tobiah wrote: I have some mailing information in a Mysql database that has characters from various other countries.  The table says that it's using latin-1 encoding.  I want to send this data out as JSON. So I'm just taking each datum and doing 'name&#

Re: Putting Unicode characters in JSON

2018-03-23 Thread Richard Damon
with a maximum of three bytes per character", in contrast to "utf8mb4" which is, well, UTF-8. In any case, abusing "Latin-1" to store binary data is still wrong. That's what BLOB is for. ChrisA One comment on this whole argument, the original poster asked how to get

Re: Putting Unicode characters in JSON

2018-03-23 Thread Chris Angelico
On Fri, Mar 23, 2018 at 9:29 PM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote: > >> That doesn't seem to be a strictly-correct Latin-1 decoder, then. There >> are a number of unassigned byte values in ISO-8859-1. > > That's incorrect, but I don't blame you for

Re: Putting Unicode characters in JSON

2018-03-23 Thread Steven D'Aprano
On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote: > That doesn't seem to be a strictly-correct Latin-1 decoder, then. There > are a number of unassigned byte values in ISO-8859-1. That's incorrect, but I don't blame you for getting it wrong. Who thought that it was a good idea to disting

Re: Putting Unicode characters in JSON

2018-03-23 Thread Paul Moore
On 23 March 2018 at 00:27, Thomas Jollans wrote: > On 22/03/18 20:46, Tobiah wrote: >> I was reading though, that JSON files must be encoded with UTF-8. So >> should I be doing string.decode('latin-1').encode('utf-8')? Or does >> the json module do th

Re: Putting Unicode characters in JSON

2018-03-23 Thread Chris Angelico
On Fri, Mar 23, 2018 at 4:35 PM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 12:05:34 +1100, Chris Angelico wrote: > >> Latin-1 is not "arbitrary bytes". It is a very specific encoding that >> cannot decode every possible byte value. > > Yes it can. > > py> blob = bytes(range(256)) > py> len(blob

Re: Putting Unicode characters in JSON

2018-03-22 Thread Steven D'Aprano
On Fri, 23 Mar 2018 12:05:34 +1100, Chris Angelico wrote: > Latin-1 is not "arbitrary bytes". It is a very specific encoding that > cannot decode every possible byte value. Yes it can. py> blob = bytes(range(256)) py> len(blob) 256 py> blob[45:55] b'-./0123456' py> s = blob.decode('latin1') py>

Re: Putting Unicode characters in JSON

2018-03-22 Thread Chris Angelico
ut Python 2 style byte- > strings, and Latin-1 is just a convenient, easy way to get that that > ensures ASCII bytes look like ASCII characters. The OP is talking about JSON. Reason makes no sense in that context. And if it really is a byte string, why store it as a Latin-1 string? Store it as the

Re: Putting Unicode characters in JSON

2018-03-22 Thread Steven D'Aprano
On Fri, 23 Mar 2018 11:08:56 +1100, Chris Angelico wrote: > On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano > wrote: >> On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> >>>> I was reading though, that JSON files must be encoded with UTF-8. So &

Re: Putting Unicode characters in JSON

2018-03-22 Thread Chris Angelico
On Fri, Mar 23, 2018 at 11:39 AM, Ben Finney wrote: > Chris Angelico writes: > >> There is NOT always a good reason for a suboptimal configuration. > > True. Did anyone claim otherwise? > > What I saw Steven responding to was your claim that there is *never* a > good reason to do it. > > To refut

Re: Putting Unicode characters in JSON

2018-03-22 Thread Thomas Jollans
On 22/03/18 20:46, Tobiah wrote: > I was reading though, that JSON files must be encoded with UTF-8.  So > should I be doing string.decode('latin-1').encode('utf-8')?  Or does > the json module do that for me when I give it a unicode object? Definitely not.

Re: Putting Unicode characters in JSON

2018-03-22 Thread Ben Finney
Chris Angelico writes: > There is NOT always a good reason for a suboptimal configuration. True. Did anyone claim otherwise? What I saw Steven responding to was your claim that there is *never* a good reason to do it. To refute that, it's sufficient to show that good reason can exist in some c

Re: Putting Unicode characters in JSON

2018-03-22 Thread Chris Angelico
On Fri, Mar 23, 2018 at 11:25 AM, Ben Finney wrote: > Chris Angelico writes: > >> On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano >> wrote: >> > On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> >> Reconfigure your MySQL database to use UTF-8. There is no reason to >> >> use Latin-1

Re: Putting Unicode characters in JSON

2018-03-22 Thread Ben Finney
Chris Angelico writes: > On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano > wrote: > > On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: > >> Reconfigure your MySQL database to use UTF-8. There is no reason to > >> use Latin-1 in the database. > > > > You don't know that. You don't know

Re: Putting Unicode characters in JSON

2018-03-22 Thread Chris Angelico
On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: > >>> I was reading though, that JSON files must be encoded with UTF-8. So >>> should I be doing string.decode('latin-1').encode('utf-8

Re: Putting Unicode characters in JSON

2018-03-22 Thread Steven D'Aprano
On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> I was reading though, that JSON files must be encoded with UTF-8. So >> should I be doing string.decode('latin-1').encode('utf-8')? Or does >> the json module do that for me when I give it a unico

Re: Putting Unicode characters in JSON

2018-03-22 Thread Tobiah
On 03/22/2018 01:09 PM, Chris Angelico wrote: On Fri, Mar 23, 2018 at 6:46 AM, Tobiah wrote: I have some mailing information in a Mysql database that has characters from various other countries. The table says that it's using latin-1 encoding. I want to send this data out as JSON. S

Re: Putting Unicode characters in JSON

2018-03-22 Thread Chris Angelico
On Fri, Mar 23, 2018 at 6:46 AM, Tobiah wrote: > I have some mailing information in a Mysql database that has > characters from various other countries. The table says that > it's using latin-1 encoding. I want to send this data out > as JSON. > > So I'm just takin

Putting Unicode characters in JSON

2018-03-22 Thread Tobiah
I have some mailing information in a Mysql database that has characters from various other countries. The table says that it's using latin-1 encoding. I want to send this data out as JSON. So I'm just taking each datum and doing 'name'.decode('latin-1') and addi

Re: Looping on a list in json

2017-11-04 Thread Sayth Renshaw
no doubt tho after playing with this is that enumerate value ends up in the output which is a dictionary. The enumerate has no key which makes it invalid json if dumped. Not massive issue but getting the effect of enumerate without polluting output would be the winner. >runner_li

  1   2   3   4   5   6   7   >