doRe: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Greg Ewing via Python-list]

On 3/10/24 11:48 am, Left Right wrote:

So, streaming parsers (eg. SAX) are written for a regular language
that approximates XML.


SAX doesn't parse a whole XML document, it parses small pieces of it
independently and passes them on. It's more like a lexical analyser than
a parser in that respect.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Ethan Furman via Python-list]

This thread is derailing.

Please consider it closed.

--
~Ethan~
Moderator
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

> You can't validate an IP packet without having all of it. Your notion
> of "streaming" is nonsensical.

Whoa, whoa, hold your horses! "nonsensical" needs a little bit of
justification :)

It seems you don't understand the difference between words and
languages! In my examples, IP _protocol_ is the language, sequences of
IP packets are the words in the language. A language is amenable to
streaming if the words of the language are repetition of sequences of
symbols of the alphabet of fixed length.  This is, essentially, like
saying that the words themselves are regular.

So, the follow-up question from you to me should be: how come strictly
context-free languages can still be parsed with streaming parsers? --
And the answer to that is it's possible to approximate context-free
languages with regular languages.  In fact, this is a very interesting
subject, which unfortunately is usually overlooked in automata
classes.  It's interesting in a sense that it's very accessible to the
students who already mastered the understanding of regular and
context-free formalisms.

So, streaming parsers (eg. SAX) are written for a regular language
that approximates XML.  This is because in practice we will almost
never encounter more than N nesting levels in an XML, more than N
characters in an element name etc. (for some large enough N).
Something which allows us to create a regular language from a
context-free one.

NB. "Nonsensical" has a very precise meaning, when it comes to
discussing the truth value of a proposition, which I think you also
somehow didn't know about.  You seem to use "nonsensical" as a synonym
to "wrong".  But, unbeknownst to you, you said something else.  You
actually implied that there's no way to tell if my notion of streaming
is correct or not.

But, for the future reference: my notion of streaming is correct, and
you would do better learning some materials about it before jumping to
conclusions.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

> One single IP packet is all you can parse.

I worked for an undisclosed company which manufactures h/w for ISPs
(4- and 8-unit boxes you mount on a rack in a datacenter).
Essentially, big-big routers.  So, I had the pleasure of writing
software that parses IP _protocol_, and let me tell you: you have no
idea what you just wrote.

But, like I wrote earlier: you don't understand the distinction
between languages and words.  And in general, are just being stubborn
and rude because you are trying to prove a point to someone you don't
like, but, in reality, you just look more and more ridiculous.

On Thu, Oct 3, 2024 at 12:51 AM Chris Angelico  wrote:
>
> On Thu, 3 Oct 2024 at 08:48, Left Right  wrote:
> >
> > > You can't validate an IP packet without having all of it. Your notion
> > > of "streaming" is nonsensical.
> >
> > Whoa, whoa, hold your horses! "nonsensical" needs a little bit of
> > justification :)
> >
> > It seems you don't understand the difference between words and
> > languages! In my examples, IP _protocol_ is the language, sequences of
> > IP packets are the words in the language. A language is amenable to
> > streaming if the words of the language are repetition of sequences of
> > symbols of the alphabet of fixed length.  This is, essentially, like
> > saying that the words themselves are regular.
>
> One single IP packet is all you can parse. You're playing shenanigans
> with words the way Humpty Dumpty does. IP packets are not sequences,
> they are individuals.
>
> ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Chris Angelico via Python-list]

On Thu, 3 Oct 2024 at 08:48, Left Right  wrote:
>
> > You can't validate an IP packet without having all of it. Your notion
> > of "streaming" is nonsensical.
>
> Whoa, whoa, hold your horses! "nonsensical" needs a little bit of
> justification :)
>
> It seems you don't understand the difference between words and
> languages! In my examples, IP _protocol_ is the language, sequences of
> IP packets are the words in the language. A language is amenable to
> streaming if the words of the language are repetition of sequences of
> symbols of the alphabet of fixed length.  This is, essentially, like
> saying that the words themselves are regular.

One single IP packet is all you can parse. You're playing shenanigans
with words the way Humpty Dumpty does. IP packets are not sequences,
they are individuals.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Chris Angelico via Python-list]

On Wed, 2 Oct 2024 at 23:53, Left Right via Python-list
 wrote:
> In the same email you replied to, I gave examples of languages for
> which parsers can be streaming (in general): SCSI or IP.

You can't validate an IP packet without having all of it. Your notion
of "streaming" is nonsensical.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

> By that definition of "streaming", no parser can ever be streaming,
> because there will be some constructs that must be read in their
> entirety before a suitably-structured piece of output can be
> emitted.

In the same email you replied to, I gave examples of languages for
which parsers can be streaming (in general): SCSI or IP. For some
languages (eg. everything in the context-free family) streaming
parsers are _in general_ impossible, because there are pathological
cases like the one with parsing numbers. But this doesn't mean that
you cannot come up with a parser that is only useful _sometimes_.
And, in practice, languages like XML or JSON do well with streaming,
even though in general it's impossible.

I'm sorry if this comes as a surprise.  On one hand I don't want to
sound condescending, on the other hand, this is something that you'd
typically study in automata theory class.  Well, not exactly in the
very same words, but you should be able to figure this stuff out if
you had that class.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Greg Ewing via Python-list]

On 2/10/24 12:26 pm, avi.e.gr...@gmail.com wrote:

The real problem is how the JSON is set up. If you take umpteen data
structures and wrap them all in something like a list, then it may be a tad
hard to stream as you may not necessarily be examining the contents till the
list finishes gigabytes later.


Yes, if you want to process the items as they come in, you might
be better off sending a series of separate JSON strings, rather than
one JSON string containing a list.

Or, use a specialised JSON parser that processes each item of the
list as soon as it's finished parsing it, instead of collecting the
whole list first.

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Dan Sommers via Python-list]

On 2024-10-01 at 23:03:01 +0200,
Left Right  wrote:

> > If I recognize the first digit, then I *can* hand that over to an
> > external function to accumulate the digits that follow.
> 
> And what is that external function going to do with this information?
> The point is you didn't parse anything if you just sent the digit.
> You just delegated the parsing further. Parsing is only meaningful if
> you extracted some information, but your idea is, essentially "what if
> I do nothing?".

If the parser detects the first digit of a number, then the parser can
read digits one at a time (i.e., "streaming"), assimilate and accumulate
the value of the number being parsed, and successfully finish parsing
the number it reads a non-digit.  Whether the function that accumulates
the value during the process is internal or external isn't relevant; the
point is that it is possible to parse integers from most significant
digit to least significant digit under a streaming model (and if you're
sufficiently clever, you can even write partial results to external
storage and/or another transmission protocol, thus allowing for numbers
bigger (as measured by JSON or your internal representation) than your
RAM).

At most, the parser has to remember the non-digit character it read so
that it (the parser) can begin to parse whatever comes after the number.
Does that break your notion of "streaming"?

Why do I have to start with the least significant digit?

> > Under that constraint, I'm not sure I can parse anything.  How can I
> > parse a string (and hand it over to an external function) until I've
> > found the closing quote?
> 
> Nobody says that parsing a number is the only pathological case.  You,
> however, exaggerate by saying you cannot parse _anything_. You can
> parse booleans or null, for example.  There's no problem there.

My intent was only to repeat what you implied:  that any parser that
reads its input until it has parsed a value is not streaming.

So how much information can the parser keep before you consider it not
to be "streaming"?

[...]

> In principle, any language that has infinite words will have the same
> problem with streaming [...]

So what magic allows anyone to stream any JSON file over SCSI or IP?
Let alone some kind of "live stream" that by definition is indefinite,
even if it only lasts a few tenths of a second?

> [...] If you ever pondered h/w or low-level
> protocols s.a. SCSI or IP [...]

I spent a good deal of my career designing and implementing all manner
of communicaations protocols, from transmitting and receiving single
bits over a wire all the way up to what are now known as session and
presentation layers.  Some imposed maximum lengths in certain places;
some allowed for indefinite amounts of data to be transferred from one
end to the other without stopping, resetting, or overflowing.  And yet
somehow, the universe never collapsed.

If you believe that some implementation of fsync fails to meet a
specification, or fails to work correctly on files containign JSON, then
file a bug report.
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[AVI GROSS via Python-list]

This discussion has become less useful.

E can all agree that in Computer Science, real infinities are avoided, and
frankly, need not be taken seriously in any serious program.

You can store all kinds of infinities quite compactly as in a transcendental
number you can derive to as many decimal points as you like. Want 1/7 to a
thousand decimal places, no problem. You can be given a digit 1 and a digit
7 and asked to do a division to as many digits as you wish in a
deterministic manner. I can think of quite a few generators that could
easily supply the next digit, or just keep giving the next element from
142857 each time from a circular loop.

Sines, cosines, pi, e and so on, can often be calculated to arbitrary
precision by evaluating things like infinite Taylor Series as many times as
needed up to the precision of the data holding the number as you move along.

 Similar ideas allow generators to give you as many primes as you want, and
no more.

So, if you can store arbitrary python code as part of your JSON, you can
send quite a bit of somewhat compressed data.

The real problem is how the JSON is set up. If you take umpteen data
structures and wrap them all in something like a list, then it may be a tad
hard to stream as you may not necessarily be examining the contents till the
list finishes gigabytes later. But if, instead, you send lots of smaller
parts, such as perhaps sending each row of something like a data.frame
individually, the other side can recombine them incrementally to a larger
structure such as a data.frame and do some logic on it as it streams, such
as keeping only some columns and discarding the rest, or applying filters
that only keep rows you care about. And, of course, all rows could be
appended to one and perhaps more .CSV files as well so if you need multiple
passes on the data, it can now be processed locally in various modes,
including "streamed".

I think that for some purposes, it makes some sense to not stream anything
but results. I mean consider any database that allows a remote login and SQL
commands that only stream results. If I only want info on records about
company X between July 1 and September 15 of a particular year and only if
the amount paid remains zero or is less than the amount owed, ...


-Original Message-
From: Python-list  On
Behalf Of Greg Ewing via Python-list
Sent: Tuesday, October 1, 2024 5:48 PM
To: python-list@python.org
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data
(60 GB) from Kenna API

On 1/10/24 8:34 am, Left Right wrote:
> You probably forgot that it has to be _streaming_. Suppose you parse
> the first digit: can you hand this information over to an external
> function to process the parsed data? -- No! because you don't know the
> magnitude yet.

By that definition of "streaming", no parser can ever be streaming,
because there will be some constructs that must be read in their
entirety before a suitably-structured piece of output can be
emitted.

The context of this discussion about integers is the claim that
they *could* be parsed incrementally if they were written little
endian instead of big endian, but the same argument applies either
way.

-- 
Greg
-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Greg Ewing via Python-list]

On 2/10/24 10:03 am, Left Right wrote:

Consider also an interesting
consequence of SCSI not being able to have infinite words: this means,
besides other things that fsync() is nonsense! :) If you aren't
familiar with the concept: UNIX filesystem API suggests that it's
possible to destage arbitrary large file (or a chunk of file) to disk.
But SCSI is built of finite "words" and to describe an arbitrary large
file you'd need to list all the blocks that constitute the file!


I don't follow. What fsync() does is ensure that any data buffered
in the kernel relating to the file is sent to the storage device.
It can send as many blocks of data over SCSI as required to
achieve this. There's no requirement for it to be atomic at the
level of the interface between the kernel and the hardware.

Some devices do their own buffering in ways that are invisible to
the software, so fsync() can't guarantee that the data is actually
written to the storage medium. But that's a problem stemming from
the design of the hardware, not the design of the protocol for
communicating with the hardware.

> the only way to implement fsync() in compliance with the
> standard is to sync _everything_

Again I'm not sure what you mean here. It may be difficult for the
kernel to track down exactly what data is relevant to a particular file,
and so the kernel programmers take the easy way out and just implement
fsync() as sync(). But again that has nothing to do with the protocol.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Greg Ewing via Python-list]

On 1/10/24 8:34 am, Left Right wrote:

You probably forgot that it has to be _streaming_. Suppose you parse
the first digit: can you hand this information over to an external
function to process the parsed data? -- No! because you don't know the
magnitude yet.


By that definition of "streaming", no parser can ever be streaming,
because there will be some constructs that must be read in their
entirety before a suitably-structured piece of output can be
emitted.

The context of this discussion about integers is the claim that
they *could* be parsed incrementally if they were written little
endian instead of big endian, but the same argument applies either
way.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

> If I recognize the first digit, then I *can* hand that over to an
> external function to accumulate the digits that follow.

And what is that external function going to do with this information?
The point is you didn't parse anything if you just sent the digit.
You just delegated the parsing further. Parsing is only meaningful if
you extracted some information, but your idea is, essentially "what if
I do nothing?".

> Under that constraint, I'm not sure I can parse anything.  How can I
parse a string (and hand it over to an external function) until I've
found the closing quote?

Nobody says that parsing a number is the only pathological case.  You,
however, exaggerate by saying you cannot parse _anything_. You can
parse booleans or null, for example.  There's no problem there.

Again, I think you misunderstand what streaming is for. Let me remind:
it's for processing information as it comes, potentially,
indefinitely. This has far more important implications than what you
find in computer science. For example, some mathematicians use the
same argument to show that real numbers are either fiction or useless:
consider adding two real numbers (where real numbers are potentially
infinite strings of decimal digits after the period) -- there's no way
to prove that such an addition is possible because you would need an
infinite proof for that (because you need to start adding from the
least significant digit).

In principle, any language that has infinite words will have the same
problem with streaming. If you ever pondered h/w or low-level
protocols s.a. SCSI or IP, you'd see that they are specifically
designed in such a way as to never have infinite words (because they
must be amenable to streaming). Consider also an interesting
consequence of SCSI not being able to have infinite words: this means,
besides other things that fsync() is nonsense! :) If you aren't
familiar with the concept: UNIX filesystem API suggests that it's
possible to destage arbitrary large file (or a chunk of file) to disk.
But SCSI is built of finite "words" and to describe an arbitrary large
file you'd need to list all the blocks that constitute the file!  And
that's why fsync() and family are so hated by people who deal with
storage: the only way to implement fsync() in compliance with the
standard is to sync _everything_ (and it hurts!)

On Tue, Oct 1, 2024 at 5:49 PM Dan Sommers via Python-list
 wrote:
>
> On 2024-09-30 at 21:34:07 +0200,
> Regarding "Re: Help with Streaming and Chunk Processing for Large JSON Data 
> (60 GB) from Kenna API,"
> Left Right via Python-list  wrote:
>
> > > What am I missing?  Handwavingly, start with the first digit, and as
> > > long as the next character is a digit, multipliy the accumulated result
> > > by 10 (or the appropriate base) and add the next value.  Oh, and handle
> > > scientific notation as a special case, and perhaps fail spectacularly
> > > instead of recovering gracefully in certain edge cases.  And in the
> > > pathological case of a single number with 60 billion digits, run out of
> > > memory (and complain loudly to the person who claimed that the file
> > > contained a "dataset").  But why do I need to start with the least
> > > significant digit?
> >
> > You probably forgot that it has to be _streaming_. Suppose you parse
> > the first digit: can you hand this information over to an external
> > function to process the parsed data? -- No! because you don't know the
> > magnitude yet.  What about two digits? -- Same thing.  You cannot
> > leave the parser code until you know the magnitude (otherwise the
> > information is useless to the external code).
>
> If I recognize the first digit, then I *can* hand that over to an
> external function to accumulate the digits that follow.
>
> > So, even if you have enough memory and don't care about special cases
> > like scientific notation: yes, you will be able to parse it, but it
> > won't be a streaming parser.
>
> Under that constraint, I'm not sure I can parse anything.  How can I
> parse a string (and hand it over to an external function) until I've
> found the closing quote?
>
> How much state can a parser maintain (before it invokes an external
> function) and still be considered streaming?  I fear that we may be
> getting hung up on terminology rather than solving the problem at hand.
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Dan Sommers via Python-list]

On 2024-09-30 at 21:34:07 +0200,
Regarding "Re: Help with Streaming and Chunk Processing for Large JSON Data (60 
GB) from Kenna API,"
Left Right via Python-list  wrote:

> > What am I missing?  Handwavingly, start with the first digit, and as
> > long as the next character is a digit, multipliy the accumulated result
> > by 10 (or the appropriate base) and add the next value.  Oh, and handle
> > scientific notation as a special case, and perhaps fail spectacularly
> > instead of recovering gracefully in certain edge cases.  And in the
> > pathological case of a single number with 60 billion digits, run out of
> > memory (and complain loudly to the person who claimed that the file
> > contained a "dataset").  But why do I need to start with the least
> > significant digit?
> 
> You probably forgot that it has to be _streaming_. Suppose you parse
> the first digit: can you hand this information over to an external
> function to process the parsed data? -- No! because you don't know the
> magnitude yet.  What about two digits? -- Same thing.  You cannot
> leave the parser code until you know the magnitude (otherwise the
> information is useless to the external code).

If I recognize the first digit, then I *can* hand that over to an
external function to accumulate the digits that follow.

> So, even if you have enough memory and don't care about special cases
> like scientific notation: yes, you will be able to parse it, but it
> won't be a streaming parser.

Under that constraint, I'm not sure I can parse anything.  How can I
parse a string (and hand it over to an external function) until I've
found the closing quote?

How much state can a parser maintain (before it invokes an external
function) and still be considered streaming?  I fear that we may be
getting hung up on terminology rather than solving the problem at hand.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Dan Sommers via Python-list]

On 2024-09-30 at 18:48:02 -0700,
Keith Thompson via Python-list  wrote:

> 2qdxy4rzwzuui...@potatochowder.com writes:
> [...]
> > In Common Lisp, you can write integers as #nnR[digits], where nn is the
> > decimal representation of the base (possibly without a leading zero),
> > the # and the R are literal characters, and the digits are written in
> > the intended base.  So the input #16f is read as the integer 65535.
> 
> Typo: You meant #16R, not #16f.

Yep.  Sorry.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

> What am I missing?  Handwavingly, start with the first digit, and as
> long as the next character is a digit, multipliy the accumulated result
> by 10 (or the appropriate base) and add the next value.  Oh, and handle
> scientific notation as a special case, and perhaps fail spectacularly
> instead of recovering gracefully in certain edge cases.  And in the
> pathological case of a single number with 60 billion digits, run out of
> memory (and complain loudly to the person who claimed that the file
> contained a "dataset").  But why do I need to start with the least
> significant digit?

You probably forgot that it has to be _streaming_. Suppose you parse
the first digit: can you hand this information over to an external
function to process the parsed data? -- No! because you don't know the
magnitude yet.  What about two digits? -- Same thing.  You cannot
leave the parser code until you know the magnitude (otherwise the
information is useless to the external code).

So, even if you have enough memory and don't care about special cases
like scientific notation: yes, you will be able to parse it, but it
won't be a streaming parser.

On Mon, Sep 30, 2024 at 9:30 PM Left Right  wrote:
>
> > Streaming won't work because the file is gzipped.  You have to receive
> > the whole thing before you can unzip it. Once unzipped it will be even
> > larger, and all in memory.
>
> GZip is specifically designed to be streamed.  So, that's not a
> problem (in principle), but you would need to have a streaming GZip
> parser, quick search in PyPI revealed this package:
> https://pypi.org/project/gzip-stream/ .
>
> On Mon, Sep 30, 2024 at 6:20 PM Thomas Passin via Python-list
>  wrote:
> >
> > On 9/30/2024 11:30 AM, Barry via Python-list wrote:
> > >
> > >
> > >> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
> > >>  wrote:
> > >>
> > >>
> > >> import polars as pl
> > >> pl.read_json("file.json")
> > >>
> > >>
> > >
> > > This is not going to work unless the computer has a lot more the 60GiB of 
> > > RAM.
> > >
> > > As later suggested a streaming parser is required.
> >
> > Streaming won't work because the file is gzipped.  You have to receive
> > the whole thing before you can unzip it. Once unzipped it will be even
> > larger, and all in memory.
> > --
> > https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Keith Thompson via Python-list]

2qdxy4rzwzuui...@potatochowder.com writes:
[...]
> In Common Lisp, you can write integers as #nnR[digits], where nn is the
> decimal representation of the base (possibly without a leading zero),
> the # and the R are literal characters, and the digits are written in
> the intended base.  So the input #16f is read as the integer 65535.

Typo: You meant #16R, not #16f.

-- 
Keith Thompson (The_Other_Keith) keith.s.thompso...@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Dan Sommers via Python-list]

On 2024-10-01 at 09:09:07 +1000,
Chris Angelico via Python-list  wrote:

> On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list
>  wrote:
> >
> > On 2024-09-30, Dan Sommers via Python-list  wrote:
> >
> > > In Common Lisp, integers can be written in any integer base from two
> > > to thirty six, inclusive.  So knowing the last digit doesn't tell
> > > you whether an integer is even or odd until you know the base
> > > anyway.
> >
> > I had to think about that for an embarassingly long time before it
> > clicked.
> 
> The only part I'm not clear on is what identifies the base. If you're
> going to write numbers little-endian, it's not that hard to also write
> them with a base indicator before the digits [...]

In Common Lisp, you can write integers as #nnR[digits], where nn is the
decimal representation of the base (possibly without a leading zero),
the # and the R are literal characters, and the digits are written in
the intended base.  So the input #16f is read as the integer 65535.

You can also set or bind the global variable *read-base* (yes, the
asterisks are part of the name) to an integer between 2 and 36, and then
anything that looks like an integer in that base is interpreted as such
(including literals in programs).  The literals I described above are
still handled correctly no matter the current value of *read-base*.  So
if the value of *read-base* is 16, then the input  is read as the
integer 65535 (as is the input #16r).

(Pedants may point our details I omitted.  I admit to omitting them.)

IIRC, certain [old 8080 and Z-80?] assemblers used to put the base
indicator at the end.  So 10 meant, well, 10, but 10H meant 16 and 10b
meant 2 (IDK; the capital H and the lower case b both look right to me).

I don't recall numbers written from least significant digit to most
significant digit (big and little endian *storage*, yes, but not the
digits when presented to or read from a human).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Chris Angelico via Python-list]

On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list
 wrote:
>
> On 2024-09-30, Dan Sommers via Python-list  wrote:
>
> > In Common Lisp, integers can be written in any integer base from two
> > to thirty six, inclusive.  So knowing the last digit doesn't tell
> > you whether an integer is even or odd until you know the base
> > anyway.
>
> I had to think about that for an embarassingly long time before it
> clicked.

The only part I'm not clear on is what identifies the base. If you're
going to write numbers little-endian, it's not that hard to also write
them with a base indicator before the digits. But, whatever. This is a
typical tangent and people are argumentative for no reason. I was just
trying to add some explanatory notes to why little-endian does make
more sense than big-endian.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Grant Edwards via Python-list]

On 2024-09-30, Dan Sommers via Python-list  wrote:

> In Common Lisp, integers can be written in any integer base from two
> to thirty six, inclusive.  So knowing the last digit doesn't tell
> you whether an integer is even or odd until you know the base
> anyway.

I had to think about that for an embarassingly long time before it
clicked.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Dan Sommers via Python-list]

On 2024-10-01 at 04:46:35 +1000,
Chris Angelico via Python-list  wrote:

> On Tue, 1 Oct 2024 at 04:30, Dan Sommers via Python-list
>  wrote:
> >
> > But why do I need to start with the least
> > significant digit?
> 
> If you start from the most significant, you don't know anything about
> the number until you finish parsing it. There's almost nothing you can
> say about a number given that it starts with a particular sequence
> (since you don't know how MANY digits there are). However, if you know
> the LAST digits, you can make certain statements about it (trivial
> examples being whether it's odd or even).

But that wasn't the question.  Sure, under certain circumstances and for
specific use cases and/or requirements, there might be arguments to read
potential numbers as strings and possibly not have to parse them
completely before accepting or rejecting them.

And if I start with the least significant digit and the number happens
to be written in scientific notation and/or has a decimal point, then I
can't tell whether it's odd or even until I further process the whole
thing anyway.

> It's not very, well, significant. But there's something to it. And it
> extends nicely to p-adic numbers, which can have an infinite number of
> nonzero digits to the left of the decimal:
> 
> https://en.wikipedia.org/wiki/P-adic_number

In Common Lisp, integers can be written in any integer base from two to
thirty six, inclusive.  So knowing the last digit doesn't tell you
whether an integer is even or odd until you know the base anyway.

Curiously, we agree:  if you move the goal posts arbitrarily, then
some algorithms that parse JSON numbers will fail.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Thomas Passin via Python-list]

On 9/30/2024 11:30 AM, Barry via Python-list wrote:




On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
 wrote:


import polars as pl
pl.read_json("file.json")




This is not going to work unless the computer has a lot more the 60GiB of RAM.

As later suggested a streaming parser is required.


There is also the json-stream library, on PyPi at

https://pypi.org/project/json-stream/


--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

> Streaming won't work because the file is gzipped.  You have to receive
> the whole thing before you can unzip it. Once unzipped it will be even
> larger, and all in memory.

GZip is specifically designed to be streamed.  So, that's not a
problem (in principle), but you would need to have a streaming GZip
parser, quick search in PyPI revealed this package:
https://pypi.org/project/gzip-stream/ .

On Mon, Sep 30, 2024 at 6:20 PM Thomas Passin via Python-list
 wrote:
>
> On 9/30/2024 11:30 AM, Barry via Python-list wrote:
> >
> >
> >> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
> >>  wrote:
> >>
> >>
> >> import polars as pl
> >> pl.read_json("file.json")
> >>
> >>
> >
> > This is not going to work unless the computer has a lot more the 60GiB of 
> > RAM.
> >
> > As later suggested a streaming parser is required.
>
> Streaming won't work because the file is gzipped.  You have to receive
> the whole thing before you can unzip it. Once unzipped it will be even
> larger, and all in memory.
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Thomas Passin via Python-list]

On 9/30/2024 1:00 PM, Chris Angelico via Python-list wrote:

On Tue, 1 Oct 2024 at 02:20, Thomas Passin via Python-list
 wrote:


On 9/30/2024 11:30 AM, Barry via Python-list wrote:




On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
 wrote:


import polars as pl
pl.read_json("file.json")




This is not going to work unless the computer has a lot more the 60GiB of RAM.

As later suggested a streaming parser is required.


Streaming won't work because the file is gzipped.  You have to receive
the whole thing before you can unzip it. Once unzipped it will be even
larger, and all in memory.


Streaming gzip is perfectly possible. You may be thinking of PKZip
which has its EOCD at the end of the file (although it may still be
possible to stream-decompress if you work at it).

ChrisA


You're right, that's what I was thinking of.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Chris Angelico via Python-list]

On Tue, 1 Oct 2024 at 04:30, Dan Sommers via Python-list
 wrote:
>
> But why do I need to start with the least
> significant digit?

If you start from the most significant, you don't know anything about
the number until you finish parsing it. There's almost nothing you can
say about a number given that it starts with a particular sequence
(since you don't know how MANY digits there are). However, if you know
the LAST digits, you can make certain statements about it (trivial
examples being whether it's odd or even).

It's not very, well, significant. But there's something to it. And it
extends nicely to p-adic numbers, which can have an infinite number of
nonzero digits to the left of the decimal:

https://en.wikipedia.org/wiki/P-adic_number

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Grant Edwards via Python-list]

On 2024-09-30, Dan Sommers via Python-list  wrote:
> On 2024-09-30 at 11:44:50 -0400,
> Grant Edwards via Python-list  wrote:
>
>> On 2024-09-30, Left Right via Python-list  wrote:
>> > [...]
>> > Imagine a pathological case of this shape: 1... <60GB of digits>. This
>> > is still a valid JSON (it doesn't have any limits on how many digits a
>> > number can have). And you cannot parse this number in a streaming way
>> > because in order to do that, you need to start with the least
>> > significant digit.
>> 
>> Which is how arabic numbers were originally parsed, but when
>> westerners adopted them from a R->L written language, thet didn't
>> flip them around to match the L->R written language into which they
>> were being adopted.
>
> Interesting.
>
>> So now long numbers can't be parsed as a stream in software. They
>> should have anticipated this problem back in the 13th century and
>> flipped the numbers around.
>
> What am I missing?  Handwavingly, start with the first digit, and as
> long as the next character is a digit, multipliy the accumulated
> result by 10 (or the appropriate base) and add the next value.
> [...]  But why do I need to start with the least significant digit?

Excellent question.  That's actully a pretty standard way to parse
numeric literals. I accepted the claim at face value that in JSON
there is something that requires parsing numeric literals from the
least significant end -- but I can't think of why the usual algorithms
used by other languages' lexers for yonks wouldn't work for JSON.

--
Grant
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Dan Sommers via Python-list]

On 2024-09-30 at 11:44:50 -0400,
Grant Edwards via Python-list  wrote:

> On 2024-09-30, Left Right via Python-list  wrote:
> > Whether and to what degree you can stream JSON depends on JSON
> > structure. In general, however, JSON cannot be streamed (but commonly
> > it can be).
> >
> > Imagine a pathological case of this shape: 1... <60GB of digits>. This
> > is still a valid JSON (it doesn't have any limits on how many digits a
> > number can have). And you cannot parse this number in a streaming way
> > because in order to do that, you need to start with the least
> > significant digit.
> 
> Which is how arabic numbers were originally parsed, but when
> westerners adopted them from a R->L written language, thet didn't flip
> them around to match the L->R written language into which they were
> being adopted.

Interesting.

> So now long numbers can't be parsed as a stream in software. They
> should have anticipated this problem back in the 13th century and
> flipped the numbers around.

What am I missing?  Handwavingly, start with the first digit, and as
long as the next character is a digit, multipliy the accumulated result
by 10 (or the appropriate base) and add the next value.  Oh, and handle
scientific notation as a special case, and perhaps fail spectacularly
instead of recovering gracefully in certain edge cases.  And in the
pathological case of a single number with 60 billion digits, run out of
memory (and complain loudly to the person who claimed that the file
contained a "dataset").  But why do I need to start with the least
significant digit?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Chris Angelico via Python-list]

On Tue, 1 Oct 2024 at 02:20, Thomas Passin via Python-list
 wrote:
>
> On 9/30/2024 11:30 AM, Barry via Python-list wrote:
> >
> >
> >> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
> >>  wrote:
> >>
> >>
> >> import polars as pl
> >> pl.read_json("file.json")
> >>
> >>
> >
> > This is not going to work unless the computer has a lot more the 60GiB of 
> > RAM.
> >
> > As later suggested a streaming parser is required.
>
> Streaming won't work because the file is gzipped.  You have to receive
> the whole thing before you can unzip it. Once unzipped it will be even
> larger, and all in memory.

Streaming gzip is perfectly possible. You may be thinking of PKZip
which has its EOCD at the end of the file (although it may still be
possible to stream-decompress if you work at it).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Thomas Passin via Python-list]

On 9/30/2024 11:30 AM, Barry via Python-list wrote:




On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
 wrote:


import polars as pl
pl.read_json("file.json")




This is not going to work unless the computer has a lot more the 60GiB of RAM.

As later suggested a streaming parser is required.


Streaming won't work because the file is gzipped.  You have to receive 
the whole thing before you can unzip it. Once unzipped it will be even 
larger, and all in memory.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Grant Edwards via Python-list]

On 2024-09-30, Left Right via Python-list  wrote:
> Whether and to what degree you can stream JSON depends on JSON
> structure. In general, however, JSON cannot be streamed (but commonly
> it can be).
>
> Imagine a pathological case of this shape: 1... <60GB of digits>. This
> is still a valid JSON (it doesn't have any limits on how many digits a
> number can have). And you cannot parse this number in a streaming way
> because in order to do that, you need to start with the least
> significant digit.

Which is how arabic numbers were originally parsed, but when
westerners adopted them from a R->L written language, thet didn't flip
them around to match the L->R written language into which they were
being adopted.

So now long numbers can't be parsed as a stream in software. They
should have anticipated this problem back in the 13th century and
flipped the numbers around.




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Barry via Python-list]

> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list 
>  wrote:
> 
> 
> import polars as pl
> pl.read_json("file.json")
> 
> 

This is not going to work unless the computer has a lot more the 60GiB of RAM.

As later suggested a streaming parser is required.

Barry


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Left Right via Python-list]

Whether and to what degree you can stream JSON depends on JSON
structure. In general, however, JSON cannot be streamed (but commonly
it can be).

Imagine a pathological case of this shape: 1... <60GB of digits>. This
is still a valid JSON (it doesn't have any limits on how many digits a
number can have). And you cannot parse this number in a streaming way
because in order to do that, you need to start with the least
significant digit.

Typically, however, JSON can be parsed incrementally. The format is
conceptually very simple to write a parser for. There are plenty of
parsers that do that, for example, this one:
https://pypi.org/project/json-stream/ . But, I'd encourage you to do
it yourself.  It's fun, and the resulting parser should end up less
than some 50 LoC.  Also, it allows you to closer incorporate your
desired output into your parser.

On Mon, Sep 30, 2024 at 8:44 AM Asif Ali Hirekumbi via Python-list
 wrote:
>
> Thanks Abdur Rahmaan.
> I will give it a try !
>
> Thanks
> Asif
>
> On Mon, Sep 30, 2024 at 11:19 AM Abdur-Rahmaan Janhangeer <
> arj.pyt...@gmail.com> wrote:
>
> > Idk if you tried Polars, but it seems to work well with JSON data
> >
> > import polars as pl
> > pl.read_json("file.json")
> >
> > Kind Regards,
> >
> > Abdur-Rahmaan Janhangeer
> > about <https://compileralchemy.github.io/> | blog
> > <https://www.pythonkitchen.com>
> > github <https://github.com/Abdur-RahmaanJ>
> > Mauritius
> >
> >
> > On Mon, Sep 30, 2024 at 8:00 AM Asif Ali Hirekumbi via Python-list <
> > python-list@python.org> wrote:
> >
> >> Dear Python Experts,
> >>
> >> I am working with the Kenna Application's API to retrieve vulnerability
> >> data. The API endpoint provides a single, massive JSON file in gzip
> >> format,
> >> approximately 60 GB in size. Handling such a large dataset in one go is
> >> proving to be quite challenging, especially in terms of memory management.
> >>
> >> I am looking for guidance on how to efficiently stream this data and
> >> process it in chunks using Python. Specifically, I am wondering if there’s
> >> a way to use the requests library or any other libraries that would allow
> >> us to pull data from the API endpoint in a memory-efficient manner.
> >>
> >> Here are the relevant API endpoints from Kenna:
> >>
> >>- Kenna API Documentation
> >><https://apidocs.kennasecurity.com/reference/welcome>
> >>- Kenna Vulnerabilities Export
> >><https://apidocs.kennasecurity.com/reference/retrieve-data-export>
> >>
> >> If anyone has experience with similar use cases or can offer any advice,
> >> it
> >> would be greatly appreciated.
> >>
> >> Thank you in advance for your help!
> >>
> >> Best regards
> >> Asif Ali
> >> --
> >> https://mail.python.org/mailman/listinfo/python-list
> >>
> >
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Asif Ali Hirekumbi via Python-list]

Thanks Abdur Rahmaan.
I will give it a try !

Thanks
Asif

On Mon, Sep 30, 2024 at 11:19 AM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> Idk if you tried Polars, but it seems to work well with JSON data
>
> import polars as pl
> pl.read_json("file.json")
>
> Kind Regards,
>
> Abdur-Rahmaan Janhangeer
> about <https://compileralchemy.github.io/> | blog
> <https://www.pythonkitchen.com>
> github <https://github.com/Abdur-RahmaanJ>
> Mauritius
>
>
> On Mon, Sep 30, 2024 at 8:00 AM Asif Ali Hirekumbi via Python-list <
> python-list@python.org> wrote:
>
>> Dear Python Experts,
>>
>> I am working with the Kenna Application's API to retrieve vulnerability
>> data. The API endpoint provides a single, massive JSON file in gzip
>> format,
>> approximately 60 GB in size. Handling such a large dataset in one go is
>> proving to be quite challenging, especially in terms of memory management.
>>
>> I am looking for guidance on how to efficiently stream this data and
>> process it in chunks using Python. Specifically, I am wondering if there’s
>> a way to use the requests library or any other libraries that would allow
>> us to pull data from the API endpoint in a memory-efficient manner.
>>
>> Here are the relevant API endpoints from Kenna:
>>
>>- Kenna API Documentation
>><https://apidocs.kennasecurity.com/reference/welcome>
>>- Kenna Vulnerabilities Export
>><https://apidocs.kennasecurity.com/reference/retrieve-data-export>
>>
>> If anyone has experience with similar use cases or can offer any advice,
>> it
>> would be greatly appreciated.
>>
>> Thank you in advance for your help!
>>
>> Best regards
>> Asif Ali
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Abdur-Rahmaan Janhangeer via Python-list]

Idk if you tried Polars, but it seems to work well with JSON data

import polars as pl
pl.read_json("file.json")

Kind Regards,

Abdur-Rahmaan Janhangeer
about <https://compileralchemy.github.io/> | blog
<https://www.pythonkitchen.com>
github <https://github.com/Abdur-RahmaanJ>
Mauritius


On Mon, Sep 30, 2024 at 8:00 AM Asif Ali Hirekumbi via Python-list <
python-list@python.org> wrote:

> Dear Python Experts,
>
> I am working with the Kenna Application's API to retrieve vulnerability
> data. The API endpoint provides a single, massive JSON file in gzip format,
> approximately 60 GB in size. Handling such a large dataset in one go is
> proving to be quite challenging, especially in terms of memory management.
>
> I am looking for guidance on how to efficiently stream this data and
> process it in chunks using Python. Specifically, I am wondering if there’s
> a way to use the requests library or any other libraries that would allow
> us to pull data from the API endpoint in a memory-efficient manner.
>
> Here are the relevant API endpoints from Kenna:
>
>- Kenna API Documentation
><https://apidocs.kennasecurity.com/reference/welcome>
>- Kenna Vulnerabilities Export
><https://apidocs.kennasecurity.com/reference/retrieve-data-export>
>
> If anyone has experience with similar use cases or can offer any advice, it
> would be greatly appreciated.
>
> Thank you in advance for your help!
>
> Best regards
> Asif Ali
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

[Asif Ali Hirekumbi via Python-list]

Dear Python Experts,

I am working with the Kenna Application's API to retrieve vulnerability
data. The API endpoint provides a single, massive JSON file in gzip format,
approximately 60 GB in size. Handling such a large dataset in one go is
proving to be quite challenging, especially in terms of memory management.

I am looking for guidance on how to efficiently stream this data and
process it in chunks using Python. Specifically, I am wondering if there’s
a way to use the requests library or any other libraries that would allow
us to pull data from the API endpoint in a memory-efficient manner.

Here are the relevant API endpoints from Kenna:

   - Kenna API Documentation
   <https://apidocs.kennasecurity.com/reference/welcome>
   - Kenna Vulnerabilities Export
   <https://apidocs.kennasecurity.com/reference/retrieve-data-export>

If anyone has experience with similar use cases or can offer any advice, it
would be greatly appreciated.

Thank you in advance for your help!

Best regards
Asif Ali
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Tue, Aug 6, 2024 at 10:53 AM Bill Deegan 
wrote:

> I’m not looking through all the packages you have installed
>
> (Ctrl-F is your friend - - - )


> What version of python is installed on your system?
>
> There are actually 3 versions installed - - - which is why the whole list
was posted.
Your question was really not clear given the context.

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Bill Deegan via Python-list]

I’m not looking through all the packages you have installed

What version of python is installed on your system?


On Tue, Aug 6, 2024 at 5:24 AM o1bigtenor  wrote:

>
>
> On Mon, Aug 5, 2024 at 10:36 PM Bill Deegan 
> wrote:
>
>> why reply to me instead of to the list?
>> It's generally considered bad form to do so.
>>
>
> Got it - - - except this list wants only reply all the next one wants only
> reply and
> keeping straight which is which isn't always happening. I did apologize
> and asked
> you to advise in the next one whether you even wanted to be included
> (which you haven't
> responded to).
>
>>
>>
>> Do you have any python 3 installed on your system?
>> Or python 2.7?
>> If not, can you install such via system package?
>>
>
>
>  # dpkg -l | grep python*
> ii  2to3   3.11.2-1
>   all  2to3 binary using python3
> ii  idle-python3.113.11.2-6+deb12u2
>   all  IDE for Python (v3.11) using Tkinter
> ii  ipython3   8.5.0-4
>  all  Enhanced interactive Python 3 shell
> ii  libboost-python1.74.0  1.74.0+ds1-21
>  amd64Boost.Python Library
> ii  libpython3-all-dev:amd64   3.11.2-1+b1
>  amd64package depending on all supported Python 3
> development packages
> ii  libpython3-dev:amd64   3.11.2-1+b1
>  amd64header files and a static library for Python
> (default)
> ii  libpython3-stdlib:amd643.11.2-1+b1
>  amd64interactive high-level object-oriented language
> (default python3 version)
> rc  libpython3.10-minimal:amd643.10.9-1
>   amd64Minimal subset of the Python language (version
> 3.10)
> ii  libpython3.11:amd643.11.2-6+deb12u2
>   amd64Shared Python runtime library (version 3.11)
> ii  libpython3.11-dev:amd643.11.2-6+deb12u2
>   amd64Header files and a static library for Python
> (v3.11)
> ii  libpython3.11-minimal:amd643.11.2-6+deb12u2
>   amd64Minimal subset of the Python language (version
> 3.11)
> ii  libpython3.11-stdlib:amd64 3.11.2-6+deb12u2
>   amd64Interactive high-level object-oriented language
> (standard library, version 3.11)
> ii  libpython3.11-testsuite3.11.2-6+deb12u2
>   all  Testsuite for the Python standard library (v3.11)
> ii  libpython3.9-minimal:amd64 3.9.13-1
>   amd64Minimal subset of the Python language (version
> 3.9)
> ii  libpython3.9-stdlib:amd64  3.9.13-1
>   amd64Interactive high-level object-oriented language
> (standard library, version 3.9)
> ii  python-apsw-doc3.40.0.0-2
>   all  documentation for python-apsw
> ii  python-apt-common  2.6.0
>  all  Python interface to libapt-pkg (locales)
> ii  python-apt-common-devuan   2.5.3devuan2
>   all  Templates for aptitude
> ii  python-attr-doc22.2.0-1
>   all  documentation for the attrs Python library
> ii  python-babel-localedata2.10.3-1
>   all  tools for internationalizing Python applications
> - locale data files
> ii  python-cryptography-doc38.0.4-3
>   all  Python library exposing cryptographic recipes
> and primitives (documentation)
> ii  python-cycler-doc  0.11.0-1
>   all  composable kwarg iterator (documentation)
> ii  python-gmpy2-common2.1.2-2
>  all  common files for python3-gmpy2
> ii  python-ipython-doc 8.5.0-4
>  all  Enhanced interactive Python shell (documentation)
> ii  python-markdown-doc3.4.1-2
>  all  text-to-HTML conversion library/tool
> (documentation)
> ii  python-matplotlib-data 3.6.3-1
>  all  Python based plotting system (data package)
> ii  python-matplotlib-doc  3.5.2-4
>  all  Python based plotting system (documentation
> package)
> ii  python-mpmath-doc  1.2.1-2
>  all  library for arbitrary-precision floating-point
> arithmetic - Documentation
> ii  python-numpy-doc   1:1.23.5-2
>   all  NumPy documentation
> ii  python-pexpect-doc 4.8.0-4
>  al

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Mon, Aug 5, 2024 at 10:36 PM Bill Deegan 
wrote:

> why reply to me instead of to the list?
> It's generally considered bad form to do so.
>

Got it - - - except this list wants only reply all the next one wants only
reply and
keeping straight which is which isn't always happening. I did apologize and
asked
you to advise in the next one whether you even wanted to be included (which
you haven't
responded to).

>
>
> Do you have any python 3 installed on your system?
> Or python 2.7?
> If not, can you install such via system package?
>


 # dpkg -l | grep python*
ii  2to3   3.11.2-1
all  2to3 binary using python3
ii  idle-python3.113.11.2-6+deb12u2
all  IDE for Python (v3.11) using Tkinter
ii  ipython3   8.5.0-4
 all  Enhanced interactive Python 3 shell
ii  libboost-python1.74.0  1.74.0+ds1-21
 amd64Boost.Python Library
ii  libpython3-all-dev:amd64   3.11.2-1+b1
 amd64package depending on all supported Python 3
development packages
ii  libpython3-dev:amd64   3.11.2-1+b1
 amd64header files and a static library for Python
(default)
ii  libpython3-stdlib:amd643.11.2-1+b1
 amd64interactive high-level object-oriented language
(default python3 version)
rc  libpython3.10-minimal:amd643.10.9-1
amd64Minimal subset of the Python language (version
3.10)
ii  libpython3.11:amd643.11.2-6+deb12u2
amd64Shared Python runtime library (version 3.11)
ii  libpython3.11-dev:amd643.11.2-6+deb12u2
amd64Header files and a static library for Python
(v3.11)
ii  libpython3.11-minimal:amd643.11.2-6+deb12u2
amd64Minimal subset of the Python language (version
3.11)
ii  libpython3.11-stdlib:amd64 3.11.2-6+deb12u2
amd64Interactive high-level object-oriented language
(standard library, version 3.11)
ii  libpython3.11-testsuite3.11.2-6+deb12u2
all  Testsuite for the Python standard library (v3.11)
ii  libpython3.9-minimal:amd64 3.9.13-1
amd64Minimal subset of the Python language (version 3.9)
ii  libpython3.9-stdlib:amd64  3.9.13-1
amd64Interactive high-level object-oriented language
(standard library, version 3.9)
ii  python-apsw-doc3.40.0.0-2
all  documentation for python-apsw
ii  python-apt-common  2.6.0
 all  Python interface to libapt-pkg (locales)
ii  python-apt-common-devuan   2.5.3devuan2
all  Templates for aptitude
ii  python-attr-doc22.2.0-1
all  documentation for the attrs Python library
ii  python-babel-localedata2.10.3-1
all  tools for internationalizing Python applications -
locale data files
ii  python-cryptography-doc38.0.4-3
all  Python library exposing cryptographic recipes and
primitives (documentation)
ii  python-cycler-doc  0.11.0-1
all  composable kwarg iterator (documentation)
ii  python-gmpy2-common2.1.2-2
 all  common files for python3-gmpy2
ii  python-ipython-doc 8.5.0-4
 all  Enhanced interactive Python shell (documentation)
ii  python-markdown-doc3.4.1-2
 all  text-to-HTML conversion library/tool
(documentation)
ii  python-matplotlib-data 3.6.3-1
 all  Python based plotting system (data package)
ii  python-matplotlib-doc  3.5.2-4
 all  Python based plotting system (documentation
package)
ii  python-mpmath-doc  1.2.1-2
 all  library for arbitrary-precision floating-point
arithmetic - Documentation
ii  python-numpy-doc   1:1.23.5-2
all  NumPy documentation
ii  python-pexpect-doc 4.8.0-4
 all  Python module for automating interactive
applications (documentation)
ii  python-pil-doc 9.4.0-1.1+deb12u1
 all  Examples for the Python Imaging Library
ii  python-pygments-doc2.14.0+dfsg-1
 all  documentation for the Pygments
ii  python-scipy-doc

Re: Help needed - - running into issues with python and its tools

[Bill Deegan via Python-list]

why reply to me instead of to the list?
It's generally considered bad form to do so.


Do you have any python 3 installed on your system?
Or python 2.7?
If not, can you install such via system package?

-Bill



On Mon, Aug 5, 2024 at 6:06 PM o1bigtenor  wrote:

>
>
> On Mon, Aug 5, 2024 at 5:28 PM Bill Deegan via Python-list <
> python-list@python.org> wrote:
>
>> Did Mats suggestion of:
>> python3.13 -m venv new_venv
>> $ new_venv/bin/python --version
>> Python 3.13.0b4
>> $ source new_venv/bin/activate
>>
>> Not work?
>> That should work on any system, with any system installl python.
>> It's not trying to modify the system installed python in anyway...
>>
>> If not, please paste the error output you're getting.
>>
>
>  # python3.13 -m venv new_venv
> -bash: python3.13: command not found
>
> $ python3.13 -m venv new_venv
> bash: python3.13: command not found
>
> There you have it - - - the first one run as superuser and the second as
> usr.
>
> Regards
>
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Bill Deegan via Python-list]

Did Mats suggestion of:
python3.13 -m venv new_venv
$ new_venv/bin/python --version
Python 3.13.0b4
$ source new_venv/bin/activate

Not work?
That should work on any system, with any system installl python.
It's not trying to modify the system installed python in anyway...

If not, please paste the error output you're getting.


On Mon, Aug 5, 2024 at 3:13 PM Mats Wichmann via Python-list <
python-list@python.org> wrote:

> On 8/5/24 15:17, o1bigtenor via Python-list wrote:
>
> >> That's something like
> >>
> >> pyenv install 3.12.4
> >>
> >
> > $ pyenv install 3.12.4
> > bash: pyenv: command not found
> >
>
> > pyenv is not a 'global' package
> >
> > there is a mountain of /root/.pyenv  files though
> > there is also quite a number of /root/.pyenv/plugins/pyenv-virtualenv/
> > files
> >
> > when looking in the /root/.pyenv files I can find options for all the
> older
> > version of python
> > but none for anything newer than what is on my system
> >
> > is there something else to install to achieve this 'version freedom' that
> > pyenv promises?
>
> It has to go somewhere your shell can find it.  Mine is a shell
> function, but it was set up so many years ago I don't remember details.
> It's presumably the pyenv installation instructions...
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Mats Wichmann via Python-list]

On 8/5/24 15:17, o1bigtenor via Python-list wrote:


That's something like

pyenv install 3.12.4



$ pyenv install 3.12.4
bash: pyenv: command not found




pyenv is not a 'global' package

there is a mountain of /root/.pyenv  files though
there is also quite a number of /root/.pyenv/plugins/pyenv-virtualenv/
files

when looking in the /root/.pyenv files I can find options for all the older
version of python
but none for anything newer than what is on my system

is there something else to install to achieve this 'version freedom' that
pyenv promises?


It has to go somewhere your shell can find it.  Mine is a shell 
function, but it was set up so many years ago I don't remember details. 
It's presumably the pyenv installation instructions...




--
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Mon, Aug 5, 2024 at 3:56 PM Mats Wichmann  wrote:

>
> > On Mon, Aug 5, 2024 at 8:51 AM Mats Wichmann  > > wrote:
> >
> > On 8/5/24 06:48, o1bigtenor via Python-list wrote:
> >  > On Sun, Aug 4, 2024 at 8:49 AM Mats Wichmann via Python-list <
> >  > python-list@python.org > wrote:
> >  >
> >  >> On 8/3/24 20:03, o1bigtenor via Python-list wrote:
> >  >>
> >  >>> My question was, is and will be (and the doc absolutely doesn't
> > cover it)
> >  >>> how do I install a different version in the venv so that python
> > 3.11.x on
> >  >>> the
> >  >>> system is not discombobulated by the python 3.12.x in the venv.
> >  >>> That python 3.12 would let me run the tools needed.
> >  >>> (Its the how to install the next version of python that I just
> > haven't
> >  >> been
> >  >>> able to find information on - - - and I would be looking for
> > information
> >  >>> on how to install on a *nix.)
> >  >>
> >  >> To get a different Python "in" the venv, you use the version you
> > want in
> >  >> the construction of the venv. For example:
> >  >>
> >  >>
> >  >> $ python3.13 -m venv new_venv
> >  >> $ new_venv/bin/python --version
> >  >> Python 3.13.0b4
> >  >> $ source new_venv/bin/activate
> >  >>
> >  >>
> >  >   "https://peps.python.org/pep-0668/
> >  PEP 668, which prevents pip from
> >  > interacting with the OS installed python. This change has been
> > done in red
> >  > hat and other distros too . . . "
> >  >
> >  > similarly your first command produces and error code for the same
> > reason.
> >  >
> >  > Sorry - - - not my policy - - -
> >
> > What? Yes, the *system* pip should have some restrictions, if it's a
> > system mainly managed by a package manager.
> >
> > Setting up a venv is the *expected* approach to such situations, and
> > creating one doesn't cause any problems. You end up with a pip in the
> > activated venv that's going to install to a different path (the one
> in
> > the venv), and will not be marked as externally managed, as the
> package
> > manager has no control over that path.
> >
> > That's the whole point.  What error are you getting?  The venv
> > module is
> > not the pip module so restrictions on the system pip have nothing to
> do
> > with it.
> >
> > set up pyenv
> > activated a venv
> > trying to install python3.12 into it
> >
> > 1. download of python3.12 (blahblahblahetc).deb will not install
> > 2. download of python3.12.tar.xz similarly will not install
> >
> > (venv2) memyself@devuanbigbox:~$ pip install
> > /home/memyself/Downloads/Python-3.12.4.tar.xz
> > Processing ./Downloads/Python-3.12.4.tar.xz
> > ERROR: file:///home/memyself/Downloads/Python-3.12.4.tar.xz does not
> > appear to be a Python project: neither 'setup.py' nor 'pyproject.toml'
> > found.
> >
> > seems that I need a different version (installable as it were) of
> > python3.12
> > or my approach is all wrong!
>
>
> you can't install Python "into" a venv.
>
> you use a version of Python as the base when *creating* a venv, the venv
> will use the same binary as the base python (symlinks if possible, as in
> the Linux case), but with different paths.
>
> Since you've already got pyenv, use it to build the version you want to
> use - I think you said there wasn't a deb for 3.12 in your distro?
>

correct


> That's something like
>
> pyenv install 3.12.4
>

$ pyenv install 3.12.4
bash: pyenv: command not found


>
> that will use the build recipe it has... and hopefully work.  Distro
> Pythons sometimes have some strange setups that are hard to reproduce.
> Pyenv knows how to build micropython, too, should it ever come to that.
>
> If you indeed found a deb for the right Python, use apt to install it,
> and then use *that* Python to create your venv.
>
> If you have the pyenv-virtualenv plugin, you can ask it to make the
> virtualenv for you, if pyenv built the Python
>
>
pyenv is not a 'global' package

there is a mountain of /root/.pyenv  files though
there is also quite a number of /root/.pyenv/plugins/pyenv-virtualenv/
files

when looking in the /root/.pyenv files I can find options for all the older
version of python
but none for anything newer than what is on my system

is there something else to install to achieve this 'version freedom' that
pyenv promises?

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Mon, Aug 5, 2024 at 3:55 PM Bill Deegan 
wrote:

> Your approach is wrong.
> You don't build python from source using pip.
>
> You don't install new versions of python into a venv either.
>
> Have you read the following?
> https://docs.micropython.org/en/latest/esp8266/tutorial/intro.html
>
> That seems to have instructions for what you want to do..
>

See item 1.4 - - - - that's where I'm at (and that's where the problems
are hidden at as well!)

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Mats Wichmann via Python-list]

On 8/5/24 14:39, o1bigtenor wrote:
Matt - if you would rather that you were not included in the address 
list - -

please advise.

On Mon, Aug 5, 2024 at 8:51 AM Mats Wichmann > wrote:


On 8/5/24 06:48, o1bigtenor via Python-list wrote:
 > On Sun, Aug 4, 2024 at 8:49 AM Mats Wichmann via Python-list <
 > python-list@python.org > wrote:
 >
 >> On 8/3/24 20:03, o1bigtenor via Python-list wrote:
 >>
 >>> My question was, is and will be (and the doc absolutely doesn't
cover it)
 >>> how do I install a different version in the venv so that python
3.11.x on
 >>> the
 >>> system is not discombobulated by the python 3.12.x in the venv.
 >>> That python 3.12 would let me run the tools needed.
 >>> (Its the how to install the next version of python that I just
haven't
 >> been
 >>> able to find information on - - - and I would be looking for
information
 >>> on how to install on a *nix.)
 >>
 >> To get a different Python "in" the venv, you use the version you
want in
 >> the construction of the venv. For example:
 >>
 >>
 >> $ python3.13 -m venv new_venv
 >> $ new_venv/bin/python --version
 >> Python 3.13.0b4
 >> $ source new_venv/bin/activate
 >>
 >>
 >   "https://peps.python.org/pep-0668/
 PEP 668, which prevents pip from
 > interacting with the OS installed python. This change has been
done in red
 > hat and other distros too . . . "
 >
 > similarly your first command produces and error code for the same
reason.
 >
 > Sorry - - - not my policy - - -

What? Yes, the *system* pip should have some restrictions, if it's a
system mainly managed by a package manager.

Setting up a venv is the *expected* approach to such situations, and
creating one doesn't cause any problems. You end up with a pip in the
activated venv that's going to install to a different path (the one in
the venv), and will not be marked as externally managed, as the package
manager has no control over that path.

That's the whole point.  What error are you getting?  The venv
module is
not the pip module so restrictions on the system pip have nothing to do
with it.

set up pyenv
activated a venv
trying to install python3.12 into it

1. download of python3.12 (blahblahblahetc).deb will not install
2. download of python3.12.tar.xz similarly will not install

(venv2) memyself@devuanbigbox:~$ pip install 
/home/memyself/Downloads/Python-3.12.4.tar.xz

Processing ./Downloads/Python-3.12.4.tar.xz
ERROR: file:///home/memyself/Downloads/Python-3.12.4.tar.xz does not 
appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' 
found.


seems that I need a different version (installable as it were) of 
python3.12

or my approach is all wrong!



you can't install Python "into" a venv.

you use a version of Python as the base when *creating* a venv, the venv 
will use the same binary as the base python (symlinks if possible, as in 
the Linux case), but with different paths.


Since you've already got pyenv, use it to build the version you want to 
use - I think you said there wasn't a deb for 3.12 in your distro? 
That's something like


pyenv install 3.12.4

that will use the build recipe it has... and hopefully work.  Distro 
Pythons sometimes have some strange setups that are hard to reproduce. 
Pyenv knows how to build micropython, too, should it ever come to that.


If you indeed found a deb for the right Python, use apt to install it, 
and then use *that* Python to create your venv.


If you have the pyenv-virtualenv plugin, you can ask it to make the 
virtualenv for you, if pyenv built the Python





--
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Bill Deegan via Python-list]

Your approach is wrong.
You don't build python from source using pip.

You don't install new versions of python into a venv either.

Have you read the following?
https://docs.micropython.org/en/latest/esp8266/tutorial/intro.html

That seems to have instructions for what you want to do..

-Bill



On Mon, Aug 5, 2024 at 1:41 PM o1bigtenor via Python-list <
python-list@python.org> wrote:

> Matt - if you would rather that you were not included in the address list -
> -
> please advise.
>
> On Mon, Aug 5, 2024 at 8:51 AM Mats Wichmann  wrote:
>
> > On 8/5/24 06:48, o1bigtenor via Python-list wrote:
> > > On Sun, Aug 4, 2024 at 8:49 AM Mats Wichmann via Python-list <
> > > python-list@python.org> wrote:
> > >
> > >> On 8/3/24 20:03, o1bigtenor via Python-list wrote:
> > >>
> > >>> My question was, is and will be (and the doc absolutely doesn't cover
> > it)
> > >>> how do I install a different version in the venv so that python
> 3.11.x
> > on
> > >>> the
> > >>> system is not discombobulated by the python 3.12.x in the venv.
> > >>> That python 3.12 would let me run the tools needed.
> > >>> (Its the how to install the next version of python that I just
> haven't
> > >> been
> > >>> able to find information on - - - and I would be looking for
> > information
> > >>> on how to install on a *nix.)
> > >>
> > >> To get a different Python "in" the venv, you use the version you want
> in
> > >> the construction of the venv. For example:
> > >>
> > >>
> > >> $ python3.13 -m venv new_venv
> > >> $ new_venv/bin/python --version
> > >> Python 3.13.0b4
> > >> $ source new_venv/bin/activate
> > >>
> > >>
> > >   "https://peps.python.org/pep-0668/ PEP 668, which prevents pip from
> > > interacting with the OS installed python. This change has been done in
> > red
> > > hat and other distros too . . . "
> > >
> > > similarly your first command produces and error code for the same
> reason.
> > >
> > > Sorry - - - not my policy - - -
> >
> > What? Yes, the *system* pip should have some restrictions, if it's a
> > system mainly managed by a package manager.
> >
> > Setting up a venv is the *expected* approach to such situations, and
> > creating one doesn't cause any problems. You end up with a pip in the
> > activated venv that's going to install to a different path (the one in
> > the venv), and will not be marked as externally managed, as the package
> > manager has no control over that path.
> >
> > That's the whole point.  What error are you getting?  The venv module is
> > not the pip module so restrictions on the system pip have nothing to do
> > with it.
> >
> > set up pyenv
> activated a venv
> trying to install python3.12 into it
>
> 1. download of python3.12 (blahblahblahetc).deb will not install
> 2. download of python3.12.tar.xz similarly will not install
>
> (venv2) memyself@devuanbigbox:~$ pip install
> /home/memyself/Downloads/Python-3.12.4.tar.xz
> Processing ./Downloads/Python-3.12.4.tar.xz
> ERROR: file:///home/memyself/Downloads/Python-3.12.4.tar.xz does not appear
> to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
>
> seems that I need a different version (installable as it were) of
> python3.12
> or my approach is all wrong!
>
> Please advise
>
> TIA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

Matt - if you would rather that you were not included in the address list -
-
please advise.

On Mon, Aug 5, 2024 at 8:51 AM Mats Wichmann  wrote:

> On 8/5/24 06:48, o1bigtenor via Python-list wrote:
> > On Sun, Aug 4, 2024 at 8:49 AM Mats Wichmann via Python-list <
> > python-list@python.org> wrote:
> >
> >> On 8/3/24 20:03, o1bigtenor via Python-list wrote:
> >>
> >>> My question was, is and will be (and the doc absolutely doesn't cover
> it)
> >>> how do I install a different version in the venv so that python 3.11.x
> on
> >>> the
> >>> system is not discombobulated by the python 3.12.x in the venv.
> >>> That python 3.12 would let me run the tools needed.
> >>> (Its the how to install the next version of python that I just haven't
> >> been
> >>> able to find information on - - - and I would be looking for
> information
> >>> on how to install on a *nix.)
> >>
> >> To get a different Python "in" the venv, you use the version you want in
> >> the construction of the venv. For example:
> >>
> >>
> >> $ python3.13 -m venv new_venv
> >> $ new_venv/bin/python --version
> >> Python 3.13.0b4
> >> $ source new_venv/bin/activate
> >>
> >>
> >   "https://peps.python.org/pep-0668/ PEP 668, which prevents pip from
> > interacting with the OS installed python. This change has been done in
> red
> > hat and other distros too . . . "
> >
> > similarly your first command produces and error code for the same reason.
> >
> > Sorry - - - not my policy - - -
>
> What? Yes, the *system* pip should have some restrictions, if it's a
> system mainly managed by a package manager.
>
> Setting up a venv is the *expected* approach to such situations, and
> creating one doesn't cause any problems. You end up with a pip in the
> activated venv that's going to install to a different path (the one in
> the venv), and will not be marked as externally managed, as the package
> manager has no control over that path.
>
> That's the whole point.  What error are you getting?  The venv module is
> not the pip module so restrictions on the system pip have nothing to do
> with it.
>
> set up pyenv
activated a venv
trying to install python3.12 into it

1. download of python3.12 (blahblahblahetc).deb will not install
2. download of python3.12.tar.xz similarly will not install

(venv2) memyself@devuanbigbox:~$ pip install
/home/memyself/Downloads/Python-3.12.4.tar.xz
Processing ./Downloads/Python-3.12.4.tar.xz
ERROR: file:///home/memyself/Downloads/Python-3.12.4.tar.xz does not appear
to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

seems that I need a different version (installable as it were) of
python3.12
or my approach is all wrong!

Please advise

TIA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Mats Wichmann via Python-list]

On 8/5/24 06:48, o1bigtenor via Python-list wrote:

On Sun, Aug 4, 2024 at 8:49 AM Mats Wichmann via Python-list <
python-list@python.org> wrote:


On 8/3/24 20:03, o1bigtenor via Python-list wrote:


My question was, is and will be (and the doc absolutely doesn't cover it)
how do I install a different version in the venv so that python 3.11.x on
the
system is not discombobulated by the python 3.12.x in the venv.
That python 3.12 would let me run the tools needed.
(Its the how to install the next version of python that I just haven't

been

able to find information on - - - and I would be looking for information
on how to install on a *nix.)


To get a different Python "in" the venv, you use the version you want in
the construction of the venv. For example:


$ python3.13 -m venv new_venv
$ new_venv/bin/python --version
Python 3.13.0b4
$ source new_venv/bin/activate



  "https://peps.python.org/pep-0668/ PEP 668, which prevents pip from
interacting with the OS installed python. This change has been done in red
hat and other distros too . . . "

similarly your first command produces and error code for the same reason.

Sorry - - - not my policy - - -


What? Yes, the *system* pip should have some restrictions, if it's a 
system mainly managed by a package manager.


Setting up a venv is the *expected* approach to such situations, and 
creating one doesn't cause any problems. You end up with a pip in the 
activated venv that's going to install to a different path (the one in 
the venv), and will not be marked as externally managed, as the package 
manager has no control over that path.


That's the whole point.  What error are you getting?  The venv module is 
not the pip module so restrictions on the system pip have nothing to do 
with it.




--
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Sun, Aug 4, 2024 at 8:49 AM Mats Wichmann via Python-list <
python-list@python.org> wrote:

> On 8/3/24 20:03, o1bigtenor via Python-list wrote:
>
> > My question was, is and will be (and the doc absolutely doesn't cover it)
> > how do I install a different version in the venv so that python 3.11.x on
> > the
> > system is not discombobulated by the python 3.12.x in the venv.
> > That python 3.12 would let me run the tools needed.
> > (Its the how to install the next version of python that I just haven't
> been
> > able to find information on - - - and I would be looking for information
> > on how to install on a *nix.)
>
> To get a different Python "in" the venv, you use the version you want in
> the construction of the venv. For example:
>
>
> $ python3.13 -m venv new_venv
> $ new_venv/bin/python --version
> Python 3.13.0b4
> $ source new_venv/bin/activate
>
>
 "https://peps.python.org/pep-0668/ PEP 668, which prevents pip from
interacting with the OS installed python. This change has been done in red
hat and other distros too . . . "

similarly your first command produces and error code for the same reason.

Sorry - - - not my policy - - -

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Sun, Aug 4, 2024 at 4:24 AM Peter J. Holzer via Python-list <
python-list@python.org> wrote:

> On 2024-08-03 15:17:11 -0500, o1bigtenor via Python-list wrote:
> > One of the tools I need to be able to use is esptools - -  well in the
> > devuan world you need to run that on either Devaun 3 or 5 - - - its just
> > not available on devuan 4.
>
> Couldn't you just upgrade to Devuan 5, then?
>

I like uptime and not measured in hours either so I'm presently not running
Devuan 5
or testing - - - although I used to. Just found that I didn't need the most
recent versions
of most of the tools I was using so find the stability (except for the
bloody browsers)
in stable version is to be appreciated.

>
>
> > Tried installing all the tools I need using downloads and .deb installs
> but
> > then I need to have python3.12 and that's also not part of Devuan4.
>
> It seems weird that something would work with the (presumably) older
> version of Python in Devuan 3 and the (presumably) newer version of
> Python in Devuan 5, but not with the version in Devuan 4.
>

Well - - - pstool was available in a version that worked in Devuan 3,
psytool is available in a version that works in Devuan 4 - - - beyond my
pay
grade as to why there isn't a version available for Devuan 4.

>
>
> > Not versed enough to set up a good venv (if that's possible) so that I
> > could work in that specific venv and have my cake (and get to eat it too
> > (grin!).
>
> You need to install Python first to create a venv. AFAIK there is no way
> to set up a venv first and then install Python into it.
>
> Does Devuan have a testing or unstable suite? You might be able to
> install a newer Python version from that. If not your best bet is to
> install Python from source.
>
>
First question answered earlier.
If I installed python 3.12 from source I would create a hung system for
myself.
(Been there and done that - - - got the T-shirt and it still stinks!)

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Mats Wichmann via Python-list]

On 8/3/24 20:03, o1bigtenor via Python-list wrote:


My question was, is and will be (and the doc absolutely doesn't cover it)
how do I install a different version in the venv so that python 3.11.x on
the
system is not discombobulated by the python 3.12.x in the venv.
That python 3.12 would let me run the tools needed.
(Its the how to install the next version of python that I just haven't been
able to find information on - - - and I would be looking for information
on how to install on a *nix.)


To get a different Python "in" the venv, you use the version you want in 
the construction of the venv. For example:



$ python3.13 -m venv new_venv
$ new_venv/bin/python --version
Python 3.13.0b4
$ source new_venv/bin/activate
...




--
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Peter J. Holzer via Python-list]

On 2024-08-03 15:17:11 -0500, o1bigtenor via Python-list wrote:
> One of the tools I need to be able to use is esptools - -  well in the
> devuan world you need to run that on either Devaun 3 or 5 - - - its just
> not available on devuan 4.

Couldn't you just upgrade to Devuan 5, then?


> Tried installing all the tools I need using downloads and .deb installs but
> then I need to have python3.12 and that's also not part of Devuan4.

It seems weird that something would work with the (presumably) older
version of Python in Devuan 3 and the (presumably) newer version of
Python in Devuan 5, but not with the version in Devuan 4.


> Not versed enough to set up a good venv (if that's possible) so that I
> could work in that specific venv and have my cake (and get to eat it too
> (grin!).

You need to install Python first to create a venv. AFAIK there is no way
to set up a venv first and then install Python into it.

Does Devuan have a testing or unstable suite? You might be able to
install a newer Python version from that. If not your best bet is to
install Python from source.

hp


-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Sat, Aug 3, 2024 at 7:11 PM dn via Python-list 
wrote:

> On 4/08/24 09:34, o1bigtenor via Python-list wrote:
> > On Sat, Aug 3, 2024 at 4:06 PM dn via Python-list <
> python-list@python.org>
> > wrote:
> >
> >> On 4/08/24 08:17, o1bigtenor via Python-list wrote:
> >>> Greetings
> >>>
> >>> Looking at ESP8266 and wanting to program it using micropython (really
> >>> don't want to have to learn C++ (not enough hours in the day as it
> >> is!!)).
> >>>
> >>> One of the tools I need to be able to use is esptools - -  well in the
> >>> devuan world you need to run that on either Devaun 3 or 5 - - - its
> just
> >>> not available on devuan 4.
> >>>
> >>> Tried installing all the tools I need using downloads and .deb installs
> >> but
> >>> then I need to have python3.12 and that's also not part of Devuan4.
> >>>
> >>> Not versed enough to set up a good venv (if that's possible) so that I
> >>> could work in that specific venv and have my cake (and get to eat it
> too
> >>> (grin!).
> >>>
> >>> Suggestions - - - ideas - - - please?
> >>
> >> Sorry if this offends, but this is a list of short-cuts and reasons why
> >> they don't work (immediately).
> >>
> >>
> >> Have just come from a discussion about 'how to start a project'. Amongst
> >> the questions to ask are: "what resources do we have (or can add) to
> >> achieve?".
> >>
> >
> > Fair question details interleaved - - -
> >
> >>
> >> In this case, if Python-skill is a "personnel-resource" (and C++ a
> >> "constraint"), will question the ESP over Raspberry Pi (say)?
> >>
> >
> > Well - - - RPi world technical specs is usable from -20 to 60 C (iirc on
> > the top number)
> > and for my project I absolutely need to have usability to at least -40 -
> -
> > could possibly do a bit less but -35 C is a hard requirement so the RPi
> > and Pico (which I would like to use) is out but ESP8266 runs in that -40
> to
> > 65 C
> > range.
>
> Interesting, but creates a mis-match of tools - battles for you to fight...
>

Not really - - - it takes a computer with psytool and about 5 or 6 other
python3.12
programs to load the microcontroller.
It is in writing the program for the microcontroller that micropython is
used not at
all in the loading.

>
>
> >> Why talking of Python 3.12 when the solution involves MicroPython?
> >>
> >
> > Because one uses psytool on one computer to transfer a program to the
> > MicroPython system.  In fact there are a set of tools that need
> Python3.12
> > to be able to do this and therefore the question.
> >
> >>
> >>
> >> Perhaps need to take a step back and look at 'options' - relate needs to
> >> resources, and evaluate the impact of each decision on later ones - as
> >> well as against your personal skills (modify objectives to limits, or
> >> accept that some learning/training will be necessary as pre-requisite to
> >> (being able to) attack the project).
> >>
> >> I have been investigating using a venv but am not finding clear
> directions
> > so
> > that I could set up Python3.12 inside (along with the other needed
> tools).
> > The
> > more I'm looking the less useful most of the information I'm finding is
> > becoming.
> > Therefore I thought I would go to the python gurus for information - - -
> > which I
> > have.
> >
> > So please - - - how do I set up a venv so that I can install and run
> python
> > 3.12
> > (and other needed programs related to 3.12) inside?
>
> If you mean venv itself, which "directions" have you reviewed?
> This one (https://python.land/virtual-environments/virtualenv) seems
> very straight-forward and shows "What's inside a venv?" to include
> python.exe. Given that venv is more-or-less the official/traditional
> solution, what are you doing differently - perhaps the question is
> lacking detail.
>
> Interesting - - - that's the doc I have been reading.

My question was, is and will be (and the doc absolutely doesn't cover it)
how do I install a different version in the venv so that python 3.11.x on
the
system is not discombobulated by the python 3.12.x in the venv.
That python 3.12 would let me run the tools needed.
(Its the how to install the next version of python that I just haven't been
able to find information on - - - and I would be looking for information
on how to install on a *nix.)


> Personally, I'm using Poetry (https://python-poetry.org) which seemed
> just as easy to pick-up; plus pyenv to maintain multiple versions of
> Python on one machine.
>
>
Will give python-poetry a look.

Have been looking at pyenv but that seems to be a whole rat's nest of other
stuff to install and its using a bunch of different tools to get there - -
- is it
necessary - - - yes or no (in the running of multiple python versions on
the
same machine).

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Sat, Aug 3, 2024 at 6:20 PM Cameron Simpson via Python-list <
python-list@python.org> wrote:

> On 03Aug2024 16:34, o1bigtenor  wrote:
> >So please - - - how do I set up a venv so that I can install and run
> >python
> >3.12
> >(and other needed programs related to 3.12) inside?
>
> Maybe this github comment will help with this:
>
> https://github.com/orgs/micropython/discussions/10255#discussioncomment-671
>

Not really.

A computer that has psytool + about 5 or 6 other python 3.12 tools is
needed to load
the microcontroller so I don't see how having venv in the microcontroller
would help the
loading of software onto the microcontroller.

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[dn via Python-list]

On 4/08/24 09:34, o1bigtenor via Python-list wrote:

On Sat, Aug 3, 2024 at 4:06 PM dn via Python-list 
wrote:


On 4/08/24 08:17, o1bigtenor via Python-list wrote:

Greetings

Looking at ESP8266 and wanting to program it using micropython (really
don't want to have to learn C++ (not enough hours in the day as it

is!!)).


One of the tools I need to be able to use is esptools - -  well in the
devuan world you need to run that on either Devaun 3 or 5 - - - its just
not available on devuan 4.

Tried installing all the tools I need using downloads and .deb installs

but

then I need to have python3.12 and that's also not part of Devuan4.

Not versed enough to set up a good venv (if that's possible) so that I
could work in that specific venv and have my cake (and get to eat it too
(grin!).

Suggestions - - - ideas - - - please?


Sorry if this offends, but this is a list of short-cuts and reasons why
they don't work (immediately).


Have just come from a discussion about 'how to start a project'. Amongst
the questions to ask are: "what resources do we have (or can add) to
achieve?".



Fair question details interleaved - - -



In this case, if Python-skill is a "personnel-resource" (and C++ a
"constraint"), will question the ESP over Raspberry Pi (say)?



Well - - - RPi world technical specs is usable from -20 to 60 C (iirc on
the top number)
and for my project I absolutely need to have usability to at least -40 - -
could possibly do a bit less but -35 C is a hard requirement so the RPi
and Pico (which I would like to use) is out but ESP8266 runs in that -40 to
65 C
range.


Interesting, but creates a mis-match of tools - battles for you to fight...



Why talking of Python 3.12 when the solution involves MicroPython?



Because one uses psytool on one computer to transfer a program to the
MicroPython system.  In fact there are a set of tools that need Python3.12
to be able to do this and therefore the question.




Perhaps need to take a step back and look at 'options' - relate needs to
resources, and evaluate the impact of each decision on later ones - as
well as against your personal skills (modify objectives to limits, or
accept that some learning/training will be necessary as pre-requisite to
(being able to) attack the project).

I have been investigating using a venv but am not finding clear directions

so
that I could set up Python3.12 inside (along with the other needed tools).
The
more I'm looking the less useful most of the information I'm finding is
becoming.
Therefore I thought I would go to the python gurus for information - - -
which I
have.

So please - - - how do I set up a venv so that I can install and run python
3.12
(and other needed programs related to 3.12) inside?


If you mean venv itself, which "directions" have you reviewed?
This one (https://python.land/virtual-environments/virtualenv) seems 
very straight-forward and shows "What's inside a venv?" to include 
python.exe. Given that venv is more-or-less the official/traditional 
solution, what are you doing differently - perhaps the question is 
lacking detail.

(see also @Cameron's take)

Personally, I'm using Poetry (https://python-poetry.org) which seemed 
just as easy to pick-up; plus pyenv to maintain multiple versions of 
Python on one machine.


--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[Cameron Simpson via Python-list]

On 03Aug2024 16:34, o1bigtenor  wrote:
So please - - - how do I set up a venv so that I can install and run 
python

3.12
(and other needed programs related to 3.12) inside?


Maybe this github comment will help with this:
https://github.com/orgs/micropython/discussions/10255#discussioncomment-671
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

On Sat, Aug 3, 2024 at 4:06 PM dn via Python-list 
wrote:

> On 4/08/24 08:17, o1bigtenor via Python-list wrote:
> > Greetings
> >
> > Looking at ESP8266 and wanting to program it using micropython (really
> > don't want to have to learn C++ (not enough hours in the day as it
> is!!)).
> >
> > One of the tools I need to be able to use is esptools - -  well in the
> > devuan world you need to run that on either Devaun 3 or 5 - - - its just
> > not available on devuan 4.
> >
> > Tried installing all the tools I need using downloads and .deb installs
> but
> > then I need to have python3.12 and that's also not part of Devuan4.
> >
> > Not versed enough to set up a good venv (if that's possible) so that I
> > could work in that specific venv and have my cake (and get to eat it too
> > (grin!).
> >
> > Suggestions - - - ideas - - - please?
>
> Sorry if this offends, but this is a list of short-cuts and reasons why
> they don't work (immediately).
>
>
> Have just come from a discussion about 'how to start a project'. Amongst
> the questions to ask are: "what resources do we have (or can add) to
> achieve?".
>

Fair question details interleaved - - -

>
> In this case, if Python-skill is a "personnel-resource" (and C++ a
> "constraint"), will question the ESP over Raspberry Pi (say)?
>

Well - - - RPi world technical specs is usable from -20 to 60 C (iirc on
the top number)
and for my project I absolutely need to have usability to at least -40 - -
could possibly do a bit less but -35 C is a hard requirement so the RPi
and Pico (which I would like to use) is out but ESP8266 runs in that -40 to
65 C
range.

>
> Why talking of Python 3.12 when the solution involves MicroPython?
>

Because one uses psytool on one computer to transfer a program to the
MicroPython system.  In fact there are a set of tools that need Python3.12
to be able to do this and therefore the question.

>
>
> Perhaps need to take a step back and look at 'options' - relate needs to
> resources, and evaluate the impact of each decision on later ones - as
> well as against your personal skills (modify objectives to limits, or
> accept that some learning/training will be necessary as pre-requisite to
> (being able to) attack the project).
>
> I have been investigating using a venv but am not finding clear directions
so
that I could set up Python3.12 inside (along with the other needed tools).
The
more I'm looking the less useful most of the information I'm finding is
becoming.
Therefore I thought I would go to the python gurus for information - - -
which I
have.

So please - - - how do I set up a venv so that I can install and run python
3.12
(and other needed programs related to 3.12) inside?

TIA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help needed - - running into issues with python and its tools

[dn via Python-list]

On 4/08/24 08:17, o1bigtenor via Python-list wrote:

Greetings

Looking at ESP8266 and wanting to program it using micropython (really
don't want to have to learn C++ (not enough hours in the day as it is!!)).

One of the tools I need to be able to use is esptools - -  well in the
devuan world you need to run that on either Devaun 3 or 5 - - - its just
not available on devuan 4.

Tried installing all the tools I need using downloads and .deb installs but
then I need to have python3.12 and that's also not part of Devuan4.

Not versed enough to set up a good venv (if that's possible) so that I
could work in that specific venv and have my cake (and get to eat it too
(grin!).

Suggestions - - - ideas - - - please?


Sorry if this offends, but this is a list of short-cuts and reasons why 
they don't work (immediately).



Have just come from a discussion about 'how to start a project'. Amongst 
the questions to ask are: "what resources do we have (or can add) to 
achieve?".


In this case, if Python-skill is a "personnel-resource" (and C++ a 
"constraint"), will question the ESP over Raspberry Pi (say)?


Why talking of Python 3.12 when the solution involves MicroPython?


Perhaps need to take a step back and look at 'options' - relate needs to 
resources, and evaluate the impact of each decision on later ones - as 
well as against your personal skills (modify objectives to limits, or 
accept that some learning/training will be necessary as pre-requisite to 
(being able to) attack the project).


--
Regards =dn

--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list

Help needed - - running into issues with python and its tools

[o1bigtenor via Python-list]

Greetings

Looking at ESP8266 and wanting to program it using micropython (really
don't want to have to learn C++ (not enough hours in the day as it is!!)).

One of the tools I need to be able to use is esptools - -  well in the
devuan world you need to run that on either Devaun 3 or 5 - - - its just
not available on devuan 4.

Tried installing all the tools I need using downloads and .deb installs but
then I need to have python3.12 and that's also not part of Devuan4.

Not versed enough to set up a good venv (if that's possible) so that I
could work in that specific venv and have my cake (and get to eat it too
(grin!).

Suggestions - - - ideas - - - please?

TIA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help

[Thomas Passin via Python-list]

On 5/12/2024 7:56 PM, Enrder via Python-list wrote:

good tader

I need help to install the bcml, and is that after installing the python
and I go to the command prompt and put ''pip install bcml'' to install it
tells me "pip" is not recognized as an internal or external command,
program or executable batch file.

I have tried everything, uninstalling and installing, restoring it,
activating and deactivating all the checkboxes that appear in the options
section, I searched the internet and the same thing keeps happening to me.



If you can help me, I would appreciate it very much.


Launch pip using the following command.  If you don't run python using 
the name "python" then use the name you usually use (e.g., "py", 
"python3", etc.) -


python -m pip install bcml

This command causes Python to load and run its own copy of pip.  So the 
fact that the operating system can't find it doesn't matter.  You should 
always run pip this way, especially if you end up with several different 
versions of Python on the same computer.


--
https://mail.python.org/mailman/listinfo/python-list

help

[Enrder via Python-list]

   good tader

   I need help to install the bcml, and is that after installing the python
   and I go to the command prompt and put ''pip install bcml'' to install it
   tells me "pip" is not recognized as an internal or external command,
   program or executable batch file.

   I have tried everything, uninstalling and installing, restoring it,
   activating and deactivating all the checkboxes that appear in the options
   section, I searched the internet and the same thing keeps happening to me.



   If you can help me, I would appreciate it very much.



   Translated with DeepL.com (free version)

   Enviado desde [1]Correo para Windows



References

   Visible links
   1. https://go.microsoft.com/fwlink/?LinkId=550986
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help: pandas and 2d table

[jak via Python-list]

Stefan Ram ha scritto:

jak  wrote or quoted:

Stefan Ram ha scritto:

df = df.where( df == 'zz' ).stack().reset_index()
result ={ 'zz': list( zip( df.iloc[ :, 0 ], df.iloc[ :, 1 ]))}

Since I don't know Pandas, I will need a month at least to understand
these 2 lines of code. Thanks again.


   Here's a technique to better understand such code:

   Transform it into a program with small statements and small
   expressions with no more than one call per statement if possible.
   (After each litte change check that the output stays the same.)

import pandas as pd

# Warning! Will overwrite the file 'file_20240412201813_tmp_DML.csv'!
with open( 'file_20240412201813_tmp_DML.csv', 'w' )as out:
 print( '''obj,foo1,foo2,foo3,foo4,foo5,foo6
foo1,aa,ab,zz,ad,ae,af
foo2,ba,bb,bc,bd,zz,bf
foo3,ca,zz,cc,cd,ce,zz
foo4,da,db,dc,dd,de,df
foo5,ea,eb,ec,zz,ee,ef
foo6,fa,fb,fc,fd,fe,ff''', file=out )
# Note the "index_col=0" below, which is important here!
df = pd.read_csv( 'file_20240412201813_tmp_DML.csv', index_col=0 )

selection = df.where( df == 'zz' )
selection_stack = selection.stack()
df = selection_stack.reset_index()
df0 = df.iloc[ :, 0 ]
df1 = df.iloc[ :, 1 ]
z = zip( df0, df1 )
l = list( z )
result ={ 'zz': l }
print( result )

   I suggest to next insert print statements to print each intermediate
   value:

# Note the "index_col=0" below, which is important here!
df = pd.read_csv( 'file_20240412201813_tmp_DML.csv', index_col=0 )
print( 'df = \n', type( df ), ':\n"', df, '"\n' )

selection = df.where( df == 'zz' )
print( "result of where( df == 'zz' ) = \n", type( selection ), ':\n"',
   selection, '"\n' )

selection_stack = selection.stack()
print( 'result of stack() = \n', type( selection_stack ), ':\n"',
   selection_stack, '"\n' )

df = selection_stack.reset_index()
print( 'result of reset_index() = \n', type( df ), ':\n"', df, '"\n' )

df0 = df.iloc[ :, 0 ]
print( 'value of .iloc[ :, 0 ]= \n', type( df0 ), ':\n"', df0, '"\n' )

df1 = df.iloc[ :, 1 ]
print( 'value of .iloc[ :, 1 ] = \n', type( df1 ), ':\n"', df1, '"\n' )

z = zip( df0, df1 )
print( 'result of zip( df0, df1 )= \n', type( z ), ':\n"', z, '"\n' )

l = list( z )
print( 'result of list( z )= \n', type( l ), ':\n"', l, '"\n' )

result ={ 'zz': l }
print( "value of { 'zz': l }= \n", type( result ), ':\n"',
   result, '"\n' )

print( result )

   Now you can see what each single step does!

df =
   :
"  foo1 foo2 foo3 foo4 foo5 foo6
obj
foo1   aa   ab   zz   ad   ae   af
foo2   ba   bb   bc   bd   zz   bf
foo3   ca   zz   cc   cd   ce   zz
foo4   da   db   dc   dd   de   df
foo5   ea   eb   ec   zz   ee   ef
foo6   fa   fb   fc   fd   fe   ff "

result of where( df == 'zz' ) =
   :
"  foo1 foo2 foo3 foo4 foo5 foo6
obj
foo1  NaN  NaN   zz  NaN  NaN  NaN
foo2  NaN  NaN  NaN  NaN   zz  NaN
foo3  NaN   zz  NaN  NaN  NaN   zz
foo4  NaN  NaN  NaN  NaN  NaN  NaN
foo5  NaN  NaN  NaN   zz  NaN  NaN
foo6  NaN  NaN  NaN  NaN  NaN  NaN "

result of stack() =
   :
" obj
foo1  foo3zz
foo2  foo5zz
foo3  foo2zz
   foo6zz
foo5  foo4zz
dtype: object "

result of reset_index() =
   :
" obj level_1   0
0  foo1foo3  zz
1  foo2foo5  zz
2  foo3foo2  zz
3  foo3foo6  zz
4  foo5foo4  zz "

value of .iloc[ :, 0 ]=
   :
" 0foo1
1foo2
2foo3
3foo3
4foo5
Name: obj, dtype: object "

value of .iloc[ :, 1 ] =
   :
" 0foo3
1foo5
2foo2
3foo6
4foo4
Name: level_1, dtype: object "

result of zip( df0, df1 )=
   :
" "

result of list( z )=
   :
" [('foo1', 'foo3'), ('foo2', 'foo5'), ('foo3', 'foo2'), ('foo3', 'foo6'), ('foo5', 
'foo4')]"

value of { 'zz': l }=
   :
" {'zz': [('foo1', 'foo3'), ('foo2', 'foo5'), ('foo3', 'foo2'), ('foo3', 'foo6'), 
('foo5', 'foo4')]}"

{'zz': [('foo1', 'foo3'), ('foo2', 'foo5'), ('foo3', 'foo2'), ('foo3', 'foo6'), 
('foo5', 'foo4')]}

   The script reads a CSV file and stores the data in a Pandas
   DataFrame object named "df". The "index_col=0" parameter tells
   Pandas to use the first column as the index for the DataFrame,
   which is kinda like column headers.

   The "where" creates a new DataFrame selection that contains
   the same data as df, but with all values replaced by NaN (Not
   a Number) except for the values that are equal to 'zz'.

   "stack" returns a Series with a multi-level index created
   by pivoting the columns. Here it gives a Series with the
   row-col-addresses of a all the non-NaN values. The general
   meaning of "stack" might be the most complex operation of
   this script. It's explained in the pandas manual (see there).

   "reset_index" then just transforms this Series back into a
   DataFrame, and ".iloc[ :, 0 ]" and ".iloc[ :, 1 ]" are the
   first and second column, respectively, of that DataFrame. These
   then are zipped to get the desired form as a list of pairs.



And this is a technique very similar to reverse engineering. Thanks for
the explanation and examples. All this is really clear and I was able to
follow i

Re: help: pandas and 2d table

[jak via Python-list]

Stefan Ram ha scritto:

df = df.where( df == 'zz' ).stack().reset_index()
result ={ 'zz': list( zip( df.iloc[ :, 0 ], df.iloc[ :, 1 ]))}


Since I don't know Pandas, I will need a month at least to understand
these 2 lines of code. Thanks again.
--
https://mail.python.org/mailman/listinfo/python-list

Re: help: pandas and 2d table

[Tim Williams via Python-list]

On Sat, Apr 13, 2024 at 1:10 PM Mats Wichmann via Python-list <
python-list@python.org> wrote:

> On 4/13/24 07:00, jak via Python-list wrote:
>
> doesn't Pandas have a "where" method that can do this kind of thing? Or
> doesn't it match what you are looking for?  Pretty sure numpy does, but
> that's a lot to bring in if you don't need the rest of numpy.
>
> pandas.DataFrame.where — pandas 2.2.2 documentation (pydata.org)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: help: pandas and 2d table

[Mats Wichmann via Python-list]

On 4/13/24 07:00, jak via Python-list wrote:

Stefan Ram ha scritto:

jak  wrote or quoted:

Would you show me the path, please?


   I was not able to read xls here, so I used csv instead; Warning:
   the script will overwrite file "file_20240412201813_tmp_DML.csv"!

import pandas as pd

with open( 'file_20240412201813_tmp_DML.csv', 'w' )as out:
 print( '''obj,foo1,foo2,foo3,foo4,foo5,foo6
foo1,aa,ab,zz,ad,ae,af
foo2,ba,bb,bc,bd,zz,bf
foo3,ca,zz,cc,cd,ce,zz
foo4,da,db,dc,dd,de,df
foo5,ea,eb,ec,zz,ee,ef
foo6,fa,fb,fc,fd,fe,ff''', file=out )

df = pd.read_csv( 'file_20240412201813_tmp_DML.csv' )

result = {}

for rownum, row in df.iterrows():
 iterator = row.items()
 _, rowname = next( iterator )
 for colname, value in iterator:
 if value not in result: result[ value ]= []
 result[ value ].append( ( rowname, colname ))

print( result )



In reality what I wanted to achieve was this:

     what = 'zz'
     result = {what: []}

     for rownum, row in df.iterrows():
     iterator = row.items()
     _, rowname = next(iterator)
     for colname, value in iterator:
     if value == what:
     result[what] += [(rowname, colname)]
     print(result)

In any case, thank you again for pointing me in the right direction. I
had lost myself looking for a pandas method that would do this in a
single shot or almost.




doesn't Pandas have a "where" method that can do this kind of thing? Or 
doesn't it match what you are looking for?  Pretty sure numpy does, but 
that's a lot to bring in if you don't need the rest of numpy.



--
https://mail.python.org/mailman/listinfo/python-list

Re: help: pandas and 2d table

[jak via Python-list]

Stefan Ram ha scritto:

jak  wrote or quoted:

Would you show me the path, please?


   I was not able to read xls here, so I used csv instead; Warning:
   the script will overwrite file "file_20240412201813_tmp_DML.csv"!

import pandas as pd

with open( 'file_20240412201813_tmp_DML.csv', 'w' )as out:
 print( '''obj,foo1,foo2,foo3,foo4,foo5,foo6
foo1,aa,ab,zz,ad,ae,af
foo2,ba,bb,bc,bd,zz,bf
foo3,ca,zz,cc,cd,ce,zz
foo4,da,db,dc,dd,de,df
foo5,ea,eb,ec,zz,ee,ef
foo6,fa,fb,fc,fd,fe,ff''', file=out )

df = pd.read_csv( 'file_20240412201813_tmp_DML.csv' )

result = {}

for rownum, row in df.iterrows():
 iterator = row.items()
 _, rowname = next( iterator )
 for colname, value in iterator:
 if value not in result: result[ value ]= []
 result[ value ].append( ( rowname, colname ))

print( result )



In reality what I wanted to achieve was this:

what = 'zz'
result = {what: []}

for rownum, row in df.iterrows():
iterator = row.items()
_, rowname = next(iterator)
for colname, value in iterator:
if value == what:
result[what] += [(rowname, colname)]
print(result)

In any case, thank you again for pointing me in the right direction. I
had lost myself looking for a pandas method that would do this in a
single shot or almost.


--
https://mail.python.org/mailman/listinfo/python-list

help: pandas and 2d table

[jak via Python-list]

Hi everyone.
I state that I don't know anything about 'pandas' but I intuited that
it could do what I want. I get, through the "read_excel" method, a
table similar to this:

  obj| foo1 foo2 foo3 foo4 foo5 foo6
  ---
 foo1|   aa   ab   zz   ad   ae   af
 |
 foo2|   ba   bb   bc   bd   zz   bf
 |
 foo3|   ca   zz   cc   cd   ce   zz
 |
 foo4|   da   db   dc   dd   de   df
 |
 foo5|   ea   eb   ec   zz   ee   ef
 |
 foo6|   fa   fb   fc   fd   fe   ff


And I would like to get a result similar to this:

{
'zz':[('foo1','foo3'),
  ('foo2','foo5'),
  ('foo3','foo2'),
  ('foo3','foo6'),
  ('foo5','foo4')
 ]
}

Would you show me the path, please?
Thank you in advance.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Help Needed With a Python Gaming Module

[Thomas Passin via Python-list]

On 4/3/2024 3:06 PM, WordWeaver Evangelist via Python-list wrote:

Hello everyone! It has been a l-o-n-g time -- nine years in fact --since I last 
participated on this mailing list.


[snip]

3. You are very familiar with the Jython 2 environment, which I am told is 
based on Python 2 and NOT Python 3.


Yes, Jython 2 is currently more or less even with Python 2.7.

You are presumably writing or hosting this in a java environment, and 
you may not realize that you can call Jython code from a java class 
(calling java classes from Jython code is dead simple).  For example, I 
have some java servlet classes that invoke Jython classes and call their 
methods.  If that sounds useful for your project, I can let you know how 
to do it.


[more snips...]

--
https://mail.python.org/mailman/listinfo/python-list

Help Needed With a Python Gaming Module

[WordWeaver Evangelist via Python-list]

Hello everyone! It has been a l-o-n-g time -- nine years in fact --since I last 
participated on this mailing list.

At that time, I was trying to write a door/external for my PC-ANSI, 
Macintosh-based BBS. Thanks to some of the folks here, I got it done  
although I eventually lost the module a number of years later when I left 
BBSing behind, because I assumed it to be dead.

Anyway, just recently I put my BBS back online again for the fourth time since 
1993, and I am again endeavoring to write a new python-based external -- a game 
-- for my BBS.

Before I continue, let me inform you that I am 70 years old, I am NOT a 
programmer, and I do not know any programming languages. Yes, I am a newbie, a 
noob, a greenhorn. :) So, if anyone here is willing to help me, you are going 
to need a lot of patience . like ChatGPT :)

Following the coding example of another open-source external that was written 
for my BBS, I have made some significant progress, thanks in large part to 
spending hours and hours chatting with ChatGPT over the past two days.

However, due my lack of expertise, and even though ChatGPT had me try all sorts 
of functions and codes and methods, there is one very perplexing issue which 
has prevented me from further development on my game. Even ChatGPT couldn't 
figure it out.

But before I get into the meat of the matter and waste everyone's time here, I 
need to address several questions to all of you Python brainiacs. )

Is there anyone -- or any group of someones -- amongst this noble assemblage, 
who meets all of the following qualifications:

1. You are very familiar with Mac OS 9.2.2

2. You are either moderately or very familiar with Hermes II BBS software.

3. You are very familiar with the Jython 2 environment, which I am told is 
based on Python 2 and NOT Python 3.

If you meet all three of those qualifications, then you may possibly be able to 
help me to overcome the mysterious obstacle I am now facing. Once I see what 
kind of responses I get, I will be very happy to share my nagging problem with 
you.

Thank you for taking the time to read this. Here is to hoping that my coding 
genius is out there, ready and eager to come to my aid.

Thank you in advance.

Kind regads,

Bill Kochman

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Can you help me with this memoization simple example?

[AVI GROSS via Python-list]

I am not sure if it was made clear that there is a general rule in python for 
what is HASHABLE and lists are changeable while tuples are not so the latter 
can be hashed as a simple copy of a list, albeit the contents must also be 
immutable.

The memorize function uses a dictionary to store things and thus the things are 
hashed to decide how to store it in the inner representation of a dictionary 
and anything new that you want to look up in the dictionary has similar 
considerations as it is hashed to see where in the dictionary to look for it.

Of course, if you add enough overhead and the memorize function you make gets 
relatively few requests that are identical, it may not be worthwhile.

-Original Message-
From: Python-list  On 
Behalf Of MRAB via Python-list
Sent: Sunday, March 31, 2024 3:24 PM
To: python-list@python.org
Subject: Re: Can you help me with this memoization simple example?

On 2024-03-31 09:04, marc nicole wrote:
> Thanks for the first comment which I incorporated
>
> but when you say "You can't use a list as a key, but you can use a 
> tuple as a key,
> provided that the elements of the tuple are also immutable."
>
> does it mean  the result of sum of the array is not convenient to use 
> as key as I do?
> Which tuple I should use to refer to the underlying list value as you 
> suggest?
>
I was suggesting using `tuple` on the argument:

def memoize(f):
  cache = {}

  def g(*args):
  key = tuple(args[0]), args[1]

  if key not in cache:
  cache[key] = f(args[0], args[1])

  return cache[key]

  return g

> Anything else is good in my code ?
>
> Thanks
>
> Le dim. 31 mars 2024 à 01:44, MRAB via Python-list 
>  a écrit :
>
> On 2024-03-31 00:09, marc nicole via Python-list wrote:
> > I am creating a memoization example with a function that adds up
> / averages
> > the elements of an array and compares it with the cached ones to
> retrieve
> > them in case they are already stored.
> >
> > In addition, I want to store only if the result of the function
> differs
> > considerably (passes a threshold e.g. 50 below).
> >
> > I created an example using a decorator to do so, the results
> using the
> > decorator is slightly faster than without the memoization which
> is OK, but
> > is the logic of the decorator correct ? anybody can tell me ?
> >
> > My code is attached below:
> >
> >
> >
> > import time
> >
> >
> > def memoize(f):
> >  cache = {}
> >
> >  def g(*args):
> >  if args[1] == "avg":
> >  sum_key_arr = sum(list(args[0])) / len(list(args[0]))
>
> 'list' will iterate over args[0] to make a list, and 'sum' will
> iterate
> over that list.
>
> It would be simpler to just let 'sum' iterate over args[0].
>
> >  elif args[1] == "sum":
> >  sum_key_arr = sum(list(args[0]))
> >  if sum_key_arr not in cache:
> >  for (
> >  key,
> >  value,
> >  ) in (
> >  cache.items()
> >  ):  # key in dict cannot be an array so I use the
> sum of the
> > array as the key
>
> You can't use a list as a key, but you can use a tuple as a key,
> provided that the elements of the tuple are also immutable.
>
> >  if (
> >  abs(sum_key_arr - key) <= 50
> >  ):  # threshold is great here so that all
> values are
> > approximated!
> >  # print('approximated')
> >  return cache[key]
> >  else:
> >  # print('not approximated')
> >  cache[sum_key_arr] = f(args[0], args[1])
> >  return cache[sum_key_arr]
> >
> >  return g
> >
> >
> > @memoize
> > def aggregate(dict_list_arr, operation):
> >  if operation == "avg":
> >  return sum(list(dict_list_arr)) / len(list(dict_list_arr))
> >  if operation == "sum":
> >  return sum(list(dict_list_arr))
> >  return None
> >
> >
> > t = time.time()
> > for i in range(200, 15000):
> >  res = aggregate(list(range(i)), "avg")
> >
> > elapsed = time.time() - t
> > print(res)
> > print(elapsed)
>
>
> -- 
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Can you help me with this memoization simple example?

[MRAB via Python-list]

On 2024-03-31 09:04, marc nicole wrote:

Thanks for the first comment which I incorporated

but when you say "You can't use a list as a key, but you can use a 
tuple as a key,

provided that the elements of the tuple are also immutable."

does it mean  the result of sum of the array is not convenient to use 
as key as I do?
Which tuple I should use to refer to the underlying list value as you 
suggest?



I was suggesting using `tuple` on the argument:

def memoize(f):
 cache = {}

 def g(*args):
 key = tuple(args[0]), args[1]

 if key not in cache:
 cache[key] = f(args[0], args[1])

 return cache[key]

 return g


Anything else is good in my code ?

Thanks

Le dim. 31 mars 2024 à 01:44, MRAB via Python-list 
 a écrit :


On 2024-03-31 00:09, marc nicole via Python-list wrote:
> I am creating a memoization example with a function that adds up
/ averages
> the elements of an array and compares it with the cached ones to
retrieve
> them in case they are already stored.
>
> In addition, I want to store only if the result of the function
differs
> considerably (passes a threshold e.g. 50 below).
>
> I created an example using a decorator to do so, the results
using the
> decorator is slightly faster than without the memoization which
is OK, but
> is the logic of the decorator correct ? anybody can tell me ?
>
> My code is attached below:
>
>
>
> import time
>
>
> def memoize(f):
>      cache = {}
>
>      def g(*args):
>          if args[1] == "avg":
>              sum_key_arr = sum(list(args[0])) / len(list(args[0]))

'list' will iterate over args[0] to make a list, and 'sum' will
iterate
over that list.

It would be simpler to just let 'sum' iterate over args[0].

>          elif args[1] == "sum":
>              sum_key_arr = sum(list(args[0]))
>          if sum_key_arr not in cache:
>              for (
>                  key,
>                  value,
>              ) in (
>                  cache.items()
>              ):  # key in dict cannot be an array so I use the
sum of the
> array as the key

You can't use a list as a key, but you can use a tuple as a key,
provided that the elements of the tuple are also immutable.

>                  if (
>                      abs(sum_key_arr - key) <= 50
>                  ):  # threshold is great here so that all
values are
> approximated!
>                      # print('approximated')
>                      return cache[key]
>              else:
>                  # print('not approximated')
>                  cache[sum_key_arr] = f(args[0], args[1])
>          return cache[sum_key_arr]
>
>      return g
>
>
> @memoize
> def aggregate(dict_list_arr, operation):
>      if operation == "avg":
>          return sum(list(dict_list_arr)) / len(list(dict_list_arr))
>      if operation == "sum":
>          return sum(list(dict_list_arr))
>      return None
>
>
> t = time.time()
> for i in range(200, 15000):
>      res = aggregate(list(range(i)), "avg")
>
> elapsed = time.time() - t
> print(res)
> print(elapsed)


-- 
https://mail.python.org/mailman/listinfo/python-list



--
https://mail.python.org/mailman/listinfo/python-list

Re: Can you help me with this memoization simple example?

[marc nicole via Python-list]

Thanks for the first comment which I incorporated

but when you say "You can't use a list as a key, but you can use a tuple as
a key,
provided that the elements of the tuple are also immutable."

does it mean  the result of sum of the array is not convenient to use as
key as I do?
Which tuple I should use to refer to the underlying list value as you
suggest?

Anything else is good in my code ?

Thanks

Le dim. 31 mars 2024 à 01:44, MRAB via Python-list 
a écrit :

> On 2024-03-31 00:09, marc nicole via Python-list wrote:
> > I am creating a memoization example with a function that adds up /
> averages
> > the elements of an array and compares it with the cached ones to retrieve
> > them in case they are already stored.
> >
> > In addition, I want to store only if the result of the function differs
> > considerably (passes a threshold e.g. 50 below).
> >
> > I created an example using a decorator to do so, the results using the
> > decorator is slightly faster than without the memoization which is OK,
> but
> > is the logic of the decorator correct ? anybody can tell me ?
> >
> > My code is attached below:
> >
> >
> >
> > import time
> >
> >
> > def memoize(f):
> >  cache = {}
> >
> >  def g(*args):
> >  if args[1] == "avg":
> >  sum_key_arr = sum(list(args[0])) / len(list(args[0]))
>
> 'list' will iterate over args[0] to make a list, and 'sum' will iterate
> over that list.
>
> It would be simpler to just let 'sum' iterate over args[0].
>
> >  elif args[1] == "sum":
> >  sum_key_arr = sum(list(args[0]))
> >  if sum_key_arr not in cache:
> >  for (
> >  key,
> >  value,
> >  ) in (
> >  cache.items()
> >  ):  # key in dict cannot be an array so I use the sum of the
> > array as the key
>
> You can't use a list as a key, but you can use a tuple as a key,
> provided that the elements of the tuple are also immutable.
>
> >  if (
> >  abs(sum_key_arr - key) <= 50
> >  ):  # threshold is great here so that all values are
> > approximated!
> >  # print('approximated')
> >  return cache[key]
> >  else:
> >  # print('not approximated')
> >  cache[sum_key_arr] = f(args[0], args[1])
> >  return cache[sum_key_arr]
> >
> >  return g
> >
> >
> > @memoize
> > def aggregate(dict_list_arr, operation):
> >  if operation == "avg":
> >  return sum(list(dict_list_arr)) / len(list(dict_list_arr))
> >  if operation == "sum":
> >  return sum(list(dict_list_arr))
> >  return None
> >
> >
> > t = time.time()
> > for i in range(200, 15000):
> >  res = aggregate(list(range(i)), "avg")
> >
> > elapsed = time.time() - t
> > print(res)
> > print(elapsed)
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Can you help me with this memoization simple example?

[MRAB via Python-list]

On 2024-03-31 00:09, marc nicole via Python-list wrote:

I am creating a memoization example with a function that adds up / averages
the elements of an array and compares it with the cached ones to retrieve
them in case they are already stored.

In addition, I want to store only if the result of the function differs
considerably (passes a threshold e.g. 50 below).

I created an example using a decorator to do so, the results using the
decorator is slightly faster than without the memoization which is OK, but
is the logic of the decorator correct ? anybody can tell me ?

My code is attached below:



import time


def memoize(f):
 cache = {}

 def g(*args):
 if args[1] == "avg":
 sum_key_arr = sum(list(args[0])) / len(list(args[0]))


'list' will iterate over args[0] to make a list, and 'sum' will iterate 
over that list.


It would be simpler to just let 'sum' iterate over args[0].


 elif args[1] == "sum":
 sum_key_arr = sum(list(args[0]))
 if sum_key_arr not in cache:
 for (
 key,
 value,
 ) in (
 cache.items()
 ):  # key in dict cannot be an array so I use the sum of the
array as the key


You can't use a list as a key, but you can use a tuple as a key, 
provided that the elements of the tuple are also immutable.



 if (
 abs(sum_key_arr - key) <= 50
 ):  # threshold is great here so that all values are
approximated!
 # print('approximated')
 return cache[key]
 else:
 # print('not approximated')
 cache[sum_key_arr] = f(args[0], args[1])
 return cache[sum_key_arr]

 return g


@memoize
def aggregate(dict_list_arr, operation):
 if operation == "avg":
 return sum(list(dict_list_arr)) / len(list(dict_list_arr))
 if operation == "sum":
 return sum(list(dict_list_arr))
 return None


t = time.time()
for i in range(200, 15000):
 res = aggregate(list(range(i)), "avg")

elapsed = time.time() - t
print(res)
print(elapsed)



--
https://mail.python.org/mailman/listinfo/python-list

Can you help me with this memoization simple example?

[marc nicole via Python-list]

I am creating a memoization example with a function that adds up / averages
the elements of an array and compares it with the cached ones to retrieve
them in case they are already stored.

In addition, I want to store only if the result of the function differs
considerably (passes a threshold e.g. 50 below).

I created an example using a decorator to do so, the results using the
decorator is slightly faster than without the memoization which is OK, but
is the logic of the decorator correct ? anybody can tell me ?

My code is attached below:



import time


def memoize(f):
cache = {}

def g(*args):
if args[1] == "avg":
sum_key_arr = sum(list(args[0])) / len(list(args[0]))
elif args[1] == "sum":
sum_key_arr = sum(list(args[0]))
if sum_key_arr not in cache:
for (
key,
value,
) in (
cache.items()
):  # key in dict cannot be an array so I use the sum of the
array as the key
if (
abs(sum_key_arr - key) <= 50
):  # threshold is great here so that all values are
approximated!
# print('approximated')
return cache[key]
else:
# print('not approximated')
cache[sum_key_arr] = f(args[0], args[1])
return cache[sum_key_arr]

return g


@memoize
def aggregate(dict_list_arr, operation):
if operation == "avg":
return sum(list(dict_list_arr)) / len(list(dict_list_arr))
if operation == "sum":
return sum(list(dict_list_arr))
return None


t = time.time()
for i in range(200, 15000):
res = aggregate(list(range(i)), "avg")

elapsed = time.time() - t
print(res)
print(elapsed)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[Grant Edwards via Python-list]

On 2024-03-06, MRAB via Python-list  wrote:
> On 2024-03-06 01:44, Ethan Furman via Python-list wrote:
>> On 3/5/24 16:49, MRAB via Python-list wrote:
>>   > On 2024-03-06 00:24, Ethan Furman via Python-list wrote:
>>   >> On 3/5/24 16:06, Chano Fucks via Python-list wrote:
>>   >>
>>   >>> [image: image.png]
>>   >>
>>   >> The image is of MS-Windows with the python installation window of 
>> "Repair Successful".  Hopefully somebody better at
>>   >> explaining that problem can take it from here...
>>   >>
>>   > If the repair was successful, what's the problem?
>> 
>> I imagine the issue is trying get Python to run (as I recall, the python 
>> icon on the MS-Windows desktop is the
>> installer, not Python itself).
>
> There was an issue 3 years ago about renaming the installer for clarity:
>
> https://github.com/python/cpython/issues/87322

Yea, this problem comes up constantly (and has for many years).

People have suggested renaming the installer so it has "setup" or
"install" in the name.

People have suggested adding text to the installer splash screen to
explain that it's the installer and not python itself, that you
already have python installed, and if you want to _run_ python instead
of _install_ python, here's how.

People have suggested having the installer remove itself by default
when it's done installing.

People have suggested lots of solutions.

AFAICT, nobody has actually done anything.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[MRAB via Python-list]

On 2024-03-06 01:44, Ethan Furman via Python-list wrote:

On 3/5/24 16:49, MRAB via Python-list wrote:
  > On 2024-03-06 00:24, Ethan Furman via Python-list wrote:
  >> On 3/5/24 16:06, Chano Fucks via Python-list wrote:
  >>
  >>> [image: image.png]
  >>
  >> The image is of MS-Windows with the python installation window of "Repair 
Successful".  Hopefully somebody better at
  >> explaining that problem can take it from here...
  >>
  > If the repair was successful, what's the problem?

I imagine the issue is trying get Python to run (as I recall, the python icon 
on the MS-Windows desktop is the
installer, not Python itself).


There was an issue 3 years ago about renaming the installer for clarity:

https://github.com/python/cpython/issues/87322

--
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[Mats Wichmann via Python-list]

On 3/5/24 18:44, Ethan Furman via Python-list wrote:

On 3/5/24 16:49, MRAB via Python-list wrote:
 > On 2024-03-06 00:24, Ethan Furman via Python-list wrote:
 >> On 3/5/24 16:06, Chano Fucks via Python-list wrote:
 >>
 >>> [image: image.png]
 >>
 >> The image is of MS-Windows with the python installation window of 
"Repair Successful".  Hopefully somebody better at

 >> explaining that problem can take it from here...
 >>
 > If the repair was successful, what's the problem?

I imagine the issue is trying get Python to run (as I recall, the python 
icon on the MS-Windows desktop is the installer, not Python itself).


that's often it, yes - you keep getting the installer when you think you 
should get a snazzy Python window. Of course, you don't - Python is a 
command-line/terminal program, not a windows app.  Now if you tried to 
launch IDLE instead, you'd get at least a window.


But we're just guessing here. Perhaps Chano will come back with an 
updated question with some details.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[Ethan Furman via Python-list]

On 3/5/24 16:49, MRAB via Python-list wrote:
> On 2024-03-06 00:24, Ethan Furman via Python-list wrote:
>> On 3/5/24 16:06, Chano Fucks via Python-list wrote:
>>
>>> [image: image.png]
>>
>> The image is of MS-Windows with the python installation window of "Repair 
Successful".  Hopefully somebody better at
>> explaining that problem can take it from here...
>>
> If the repair was successful, what's the problem?

I imagine the issue is trying get Python to run (as I recall, the python icon on the MS-Windows desktop is the 
installer, not Python itself).

--
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[MRAB via Python-list]

On 2024-03-06 00:24, Ethan Furman via Python-list wrote:

On 3/5/24 16:06, Chano Fucks via Python-list wrote:


[image: image.png]


The image is of MS-Windows with the python installation window of "Repair 
Successful".  Hopefully somebody better at
explaining that problem can take it from here...


If the repair was successful, what's the problem?
--
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[MRAB via Python-list]

On 2024-03-06 00:06, Chano Fucks via Python-list wrote:

[image: image.png]


This list removes all images.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Can u help me?

[Ethan Furman via Python-list]

On 3/5/24 16:06, Chano Fucks via Python-list wrote:


[image: image.png]


The image is of MS-Windows with the python installation window of "Repair Successful".  Hopefully somebody better at 
explaining that problem can take it from here...


--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Can u help me?

[Chano Fucks via Python-list]

[image: image.png]
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help

[Mike Dewhirst via Python-list]

On 7/11/2023 9:02 am, Jason Friedman via Python-list wrote:

On Sun, Nov 5, 2023 at 1:23 PM office officce via Python-list <
python-list@python.org> wrote:


which python version is better to be used and how to make sure it works on
my window 10 because i downloaded it and it never worked so I uninstall to
do that again please can you give me the steps on how it will work perfectly


1. Download from https://python.org (not Microsoft) and always choose 
the 64-bit stable version


2. Choose the installation location as C:\Python311 (avoid the default)

4. Accept other recommended installation options especially to include 
Python on the path (if offered)


Guaranteed to work. Also, you will never have to uninstall. Install the 
next version in C:\Python312 etc


In due course, investigate virtual environments so you can work on 
projects simultaneously using different versions of Python or different 
versions of various Python libraries.


Good luck

Mike





If you are just starting out, the most recent version is 3.12 and is
probably your best choice.

When you say it never worked, can you describe in more detail what you did
and what error messages you encountered?

This mailing list does not accept screenshots.



--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Your
email software can handle signing.



OpenPGP_signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help

[Jason Friedman via Python-list]

On Sun, Nov 5, 2023 at 1:23 PM office officce via Python-list <
python-list@python.org> wrote:

> which python version is better to be used and how to make sure it works on
> my window 10 because i downloaded it and it never worked so I uninstall to
> do that again please can you give me the steps on how it will work perfectly
>
>
If you are just starting out, the most recent version is 3.12 and is
probably your best choice.

When you say it never worked, can you describe in more detail what you did
and what error messages you encountered?

This mailing list does not accept screenshots.
-- 
https://mail.python.org/mailman/listinfo/python-list

Help

[office officce via Python-list]

which python version is better to be used and how to make sure it works on my 
window 10 because i downloaded it and it never worked so I uninstall to do that 
again please can you give me the steps on how it will work perfectly


from
Kenny
-- 
https://mail.python.org/mailman/listinfo/python-list

Mtg ANN: Using computer vision AI to help protect the worlds rarest dolphin

[dn via Python-list]

Wed 16 Aug 2023, 1800~20:30 NZST (0600~0830 UTC, late-Tue in US)
Details and RSVP at https://www.meetup.com/nzpug-auckland/events/295091858/


Teaching a computer to see. How computer vision is helping to protect 
the world’s rarest dolphin and how you can train your own model.


Tane van der Boon will talk about the MAUI63 project, and then lead us 
through how to train and utilise a custom computer vision object 
detector. Essentially allowing our software to be able to see.


MAUI63 was given its name because when the venture began, there were 
only 63 Māui dolphins left in existence. Using drone technology and AI 
to collect and process visual data, the founders took action to collect 
data to help save the world’s rarest dolphin from extinction and 
influence future marine conservation efforts.


The MAUI63 team uses a customised drone to find and track the movement 
of dolphins, and enables individual dolphins to be identified through 
their fin markings. The information collected can then be used to inform 
better data driven decisions on how best to protect them.


Later, Tane will lead us through a smaller task to build our own image 
recognizer. (more details in due course...)



The Auckland Branch of the New Zealand Python Users' Group meet twice 
monthly. You're very welcome to attend online or in-person (as 
available). Meeting location-details or URL will be sent to those who RSVP.


We are a healthy mix of Python users. Students, academics, hobbyists, 
industry professionals, and many completely new to Python.


The "room" opens at 1800 (local time) with an opportunity to network 
with colleagues. Everything should be wrapped up by 2030.


We are always keen to hear suggestions for meeting-topics, and to meet 
with folk who'd like to present or lead - eg a brief lightning talk, a 
practical coaching-session, a full lecture... Help is available if 
you've never done such a thing before!


We follow the NZPUG Code of Conduct (https://python.nz/code-of-conduct) 
to create an inclusive and friendly environment.


We express thanks to, and encourage you to investigate, our sponsors: 
Catalyst Cloud, New Zealand Open Source Society, JetBrains, and IceHouse 
Ventures.


--
Regards =dn

--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Help on ImportError('Error: Reinit is forbidden')

[Barry]

 On 18 May 2023, at 13:56, Jason Qian  wrote:

 
 Hi Barry,
 void handleError(const char* msg)
 {
 ...
 PyErr_Fetch(&pyExcType, &pyExcValue, &pyExcTraceback);
 PyErr_NormalizeException(&pyExcType, &pyExcValue, &pyExcTraceback);

 PyObject* str_value = PyObject_Repr(pyExcValue);
 PyObject* pyExcValueStr = PyUnicode_AsEncodedString(str_value, "utf-8",
 "Error ~");
 const char *strErrValue = PyBytes_AS_STRING(pyExcValueStr);
 //where   strErrValue   = "ImportError('Error: Reinit is forbidden')"
 ...
 }
 What we imported is a Python file which import some pyd libraries.

   Please do not top post replies.
   Ok so assume the error is correct and hunt for the code that does the
   reimport.
   You may need to set break points in you C code to find tnw caller.
   Barry

 Thanks 
 Jason 
 On Thu, May 18, 2023 at 3:53 AM Barry <[1]ba...@barrys-emacs.org> wrote:

   > On 17 May 2023, at 20:35, Jason Qian via Python-list
   <[2]python-list@python.org> wrote:
   >
   >  Hi,
   >
   >   I Need some of your help.
   >
   > I have the following C code to import *Import python.*   It works
   99% of
   > the time, but sometimes  receives  "*ImportError('Error: Reinit is
   > forbidden')*". error.
   > **We run multiple instances of the app parallelly.
   >
   > *** Python version(3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51)
   [MSC
   > v.1914 64 bit (AMD64)]
   >
   > PyObject * importPythonModule(const char* pmodName)
   > {
   >    const char* errors = NULL;
   >     int nlen = strlen(pmodName);
   >     PyObject *pName = PyUnicode_DecodeUTF8(pmodName, nlen, errors);
   >     PyObject *pModule = *PyImport_Import*(pName);
   >     Py_DECREF(pName);
   >     if (pModule == NULL) {
   >     if (*PyErr_Occurred*()) {
   >            handleError("PyImport_Import()");
   >      }
   >   }
   > }
   > void handleError(const char* msg)
   > {
   >  ...
   >  "PyImport_Import() - ImportError('Error: Reinit is forbidden')"
   > }

   You do not seem to printing out msg, you have assumed it means reinit
   it seems.
   What does msg contain when it fails?

   Barry
   >
   >
   > Thanks
   > Jason
   > --
   > [3]https://mail.python.org/mailman/listinfo/python-list
   >

References

   Visible links
   1. mailto:ba...@barrys-emacs.org
   2. mailto:python-list@python.org
   3. https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help on ImportError('Error: Reinit is forbidden')

[Jason Qian via Python-list]

Hi Barry,

void handleError(const char* msg)
{
...
PyErr_Fetch(&pyExcType, &pyExcValue, &pyExcTraceback);
PyErr_NormalizeException(&pyExcType, &pyExcValue, &pyExcTraceback);

PyObject* str_value = PyObject_Repr(pyExcValue);
PyObject* pyExcValueStr = PyUnicode_AsEncodedString(str_value, "utf-8",
"Error ~");
const char **strErrValue* = PyBytes_AS_STRING(pyExcValueStr);

//where   *strErrValue*   = "ImportError('Error: Reinit is forbidden')"
...
}

What we imported is a Python file which import some pyd libraries.


Thanks
Jason


On Thu, May 18, 2023 at 3:53 AM Barry  wrote:

>
>
> > On 17 May 2023, at 20:35, Jason Qian via Python-list <
> python-list@python.org> wrote:
> >
> >  Hi,
> >
> >   I Need some of your help.
> >
> > I have the following C code to import *Import python.*   It works 99% of
> > the time, but sometimes  receives  "*ImportError('Error: Reinit is
> > forbidden')*". error.
> > **We run multiple instances of the app parallelly.
> >
> > *** Python version(3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC
> > v.1914 64 bit (AMD64)]
> >
> > PyObject * importPythonModule(const char* pmodName)
> > {
> >const char* errors = NULL;
> > int nlen = strlen(pmodName);
> > PyObject *pName = PyUnicode_DecodeUTF8(pmodName, nlen, errors);
> > PyObject *pModule = *PyImport_Import*(pName);
> > Py_DECREF(pName);
> > if (pModule == NULL) {
> > if (*PyErr_Occurred*()) {
> >handleError("PyImport_Import()");
> >  }
> >   }
> > }
> > void handleError(const char* msg)
> > {
> >  ...
> >  "PyImport_Import() - ImportError('Error: Reinit is forbidden')"
> > }
>
> You do not seem to printing out msg, you have assumed it means reinit it
> seems.
> What does msg contain when it fails?
>
> Barry
> >
> >
> > Thanks
> > Jason
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help on ImportError('Error: Reinit is forbidden')

[Barry]

> On 17 May 2023, at 20:35, Jason Qian via Python-list  
> wrote:
> 
>  Hi,
> 
>   I Need some of your help.
> 
> I have the following C code to import *Import python.*   It works 99% of
> the time, but sometimes  receives  "*ImportError('Error: Reinit is
> forbidden')*". error.
> **We run multiple instances of the app parallelly.
> 
> *** Python version(3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC
> v.1914 64 bit (AMD64)]
> 
> PyObject * importPythonModule(const char* pmodName)
> {
>const char* errors = NULL;
> int nlen = strlen(pmodName);
> PyObject *pName = PyUnicode_DecodeUTF8(pmodName, nlen, errors);
> PyObject *pModule = *PyImport_Import*(pName);
> Py_DECREF(pName);
> if (pModule == NULL) {
> if (*PyErr_Occurred*()) {
>handleError("PyImport_Import()");
>  }
>   }
> }
> void handleError(const char* msg)
> {
>  ...
>  "PyImport_Import() - ImportError('Error: Reinit is forbidden')"
> }

You do not seem to printing out msg, you have assumed it means reinit it seems.
What does msg contain when it fails?

Barry
> 
> 
> Thanks
> Jason
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Help on ImportError('Error: Reinit is forbidden')

[Jason Qian via Python-list]

 Hi,

   I Need some of your help.

 I have the following C code to import *Import python.*   It works 99% of
the time, but sometimes  receives  "*ImportError('Error: Reinit is
forbidden')*". error.
 **We run multiple instances of the app parallelly.

*** Python version(3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC
v.1914 64 bit (AMD64)]

PyObject * importPythonModule(const char* pmodName)
{
const char* errors = NULL;
 int nlen = strlen(pmodName);
 PyObject *pName = PyUnicode_DecodeUTF8(pmodName, nlen, errors);
 PyObject *pModule = *PyImport_Import*(pName);
 Py_DECREF(pName);
 if (pModule == NULL) {
 if (*PyErr_Occurred*()) {
handleError("PyImport_Import()");
  }
   }
}
void handleError(const char* msg)
{
  ...
  "PyImport_Import() - ImportError('Error: Reinit is forbidden')"
}


Thanks
Jason
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help on ctypes.POINTER for Python array

[Jason Qian via Python-list]

Awesome, thanks!

On Thu, May 11, 2023 at 1:47 PM Eryk Sun  wrote:

> On 5/11/23, Jason Qian via Python-list  wrote:
> >
> > in the Python, I have a array of string
> > var_array=["Opt1=DG","Opt1=DG2"]
> > I need to call c library and  pass var_array as parameter
> > In the   argtypes,  how do I set up ctypes.POINTER(???)  for var_array?
> >
> > func.argtypes=[ctypes.c_void_p,ctypes.c_int, ctypes.POINTER()]
> >
> > In the c code:
> > int  func (void* obj, int index,  char** opt)
>
> The argument type is ctypes.POINTER(ctypes.c_char_p), but that's not
> sufficient. It doesn't implement converting a list of str objects into
> an array of c_char_p pointers that reference byte strings. You could
> write a wrapper function that implements the conversion before calling
> func(), or you could set the argument type to a custom subclass of
> ctypes.POINTER(ctypes.c_char_p) that implements the conversion via the
> from_param() class method.
>
> https://docs.python.org/3/library/ctypes.html#ctypes._CData.from_param
>
> Here's an example of the latter.
>
> C library:
>
> #include 
>
> int
> func(void *obj, int index, char **opt)
> {
> int length;
> for (length=0; opt[length]; length++);
> if (index < 0 || index >= length) {
> return -1;
> }
> return printf("%s\n", opt[index]);
> }
>
>
> Python:
>
> import os
> import ctypes
>
> lib = ctypes.CDLL('./lib.so')
> BaseOptions = ctypes.POINTER(ctypes.c_char_p)
>
> class Options(BaseOptions):
> @classmethod
> def from_param(cls, param):
> if isinstance(param, list):
> new_param = (ctypes.c_char_p * (len(param) + 1))()
> for i, p in enumerate(param):
> new_param[i] = os.fsencode(p)
> param = new_param
> return BaseOptions.from_param(param)
>
> lib.func.argtypes = (ctypes.c_void_p, ctypes.c_int, Options)
>
>
> demo:
>
> >>> opts = ['Opt1=DG', 'Opt1=DG2']
> >>> lib.func(None, 0, opts)
> Opt1=DG
> 8
> >>> lib.func(None, 1, opts)
> Opt1=DG2
> 9
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help on ctypes.POINTER for Python array

[Eryk Sun]

On 5/11/23, Jason Qian via Python-list  wrote:
>
> in the Python, I have a array of string
> var_array=["Opt1=DG","Opt1=DG2"]
> I need to call c library and  pass var_array as parameter
> In the   argtypes,  how do I set up ctypes.POINTER(???)  for var_array?
>
> func.argtypes=[ctypes.c_void_p,ctypes.c_int, ctypes.POINTER()]
>
> In the c code:
> int  func (void* obj, int index,  char** opt)

The argument type is ctypes.POINTER(ctypes.c_char_p), but that's not
sufficient. It doesn't implement converting a list of str objects into
an array of c_char_p pointers that reference byte strings. You could
write a wrapper function that implements the conversion before calling
func(), or you could set the argument type to a custom subclass of
ctypes.POINTER(ctypes.c_char_p) that implements the conversion via the
from_param() class method.

https://docs.python.org/3/library/ctypes.html#ctypes._CData.from_param

Here's an example of the latter.

C library:

#include 

int
func(void *obj, int index, char **opt)
{
int length;
for (length=0; opt[length]; length++);
if (index < 0 || index >= length) {
return -1;
}
return printf("%s\n", opt[index]);
}


Python:

import os
import ctypes

lib = ctypes.CDLL('./lib.so')
BaseOptions = ctypes.POINTER(ctypes.c_char_p)

class Options(BaseOptions):
@classmethod
def from_param(cls, param):
if isinstance(param, list):
new_param = (ctypes.c_char_p * (len(param) + 1))()
for i, p in enumerate(param):
new_param[i] = os.fsencode(p)
param = new_param
return BaseOptions.from_param(param)

lib.func.argtypes = (ctypes.c_void_p, ctypes.c_int, Options)


demo:

>>> opts = ['Opt1=DG', 'Opt1=DG2']
>>> lib.func(None, 0, opts)
Opt1=DG
8
>>> lib.func(None, 1, opts)
Opt1=DG2
9
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help on ctypes.POINTER for Python array

[Jim Schwartz]

I’m not sure this is the shortest method, but you could set up two python 
scripts to do the same thing and convert them to c using cython. I wouldn’t be 
able to read the c scripts, but maybe you could. 

Maybe someone else has a more direct answer. 

Sent from my iPhone

> On May 11, 2023, at 10:00 AM, Jason Qian via Python-list 
>  wrote:
> 
> Hi,
> 
> Need some help,
> 
> in the Python, I have a array of string
> 
> var_array=["Opt1=DG","Opt1=DG2"]
> 
> I need to call c library and  pass var_array as parameter
> 
> In the   argtypes,  how do I set up ctypes.POINTER(???)  for var_array?
> 
> func.argtypes=[ctypes.c_void_p,ctypes.c_int, ctypes.POINTER()]
> 
> In the c code:
> 
> int  func (void* obj, int index,  char** opt)
> 
> Thanks
> Jason
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Help on ctypes.POINTER for Python array

[Jason Qian via Python-list]

Hi,

Need some help,

in the Python, I have a array of string

 var_array=["Opt1=DG","Opt1=DG2"]

I need to call c library and  pass var_array as parameter

In the   argtypes,  how do I set up ctypes.POINTER(???)  for var_array?

func.argtypes=[ctypes.c_void_p,ctypes.c_int, ctypes.POINTER()]

In the c code:

int  func (void* obj, int index,  char** opt)

Thanks
Jason
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Need help please

[Thomas Passin]

On 4/10/2023 9:59 AM, Jack Gilbert wrote:

I D/L 3.11.3, I can see it in CMD

running W10 64bit

I have IDL on my desktop,

HOW do I get 3.11.3 on my desktop?


If you mean "How can I create a shortcut to Python 3.11.3 on my desktop 
that opens an interactive Python session", here is one way:


1. Find where your Python 3.11.3 program has been installed.  On the 
command line in a console, type:


py -c "import sys; print(sys.executable)"

You will get a response like this:

C:\Users\tom\AppData\Local\Programs\Python\Python311\python.exe

NOTE:  If the "py" command is not on your computer or does not open 
Python 3.11, then open a python 3.11 session and type the same commands:


import sys
print(sys.executable)

2. Open the Windows file browser ("Windows Explorer") and navigate to 
that directory.  On my computer this is


C:\Users\tom\AppData\Local\Programs\Python\Python311

3. Press and hold both the CTRL and SHIFT keys down at the same time, 
and with the mouse drag the icon for "python.exe" to a blank space on 
the desktop.  This will not drag the program itself but will create a 
shortcut and drag that.


4. Test the new shortcut by double-clicking on it and seeing that a new 
console window opens with the Python interpreter running in it.


If you do not like the size, shape, or font of this new console, change 
them by clicking on the icon in the upper left, then selecting 
"Properties", and making changes in the dialog box that opens.  The new 
choices will be used whenever you use this shortcut again.


5. The new shortcut will probably be named "python.ex".  I suggest that 
you rename it to "Python 3.11".  This way you can create other python 
shortcuts without having their names conflict.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Need help please

[Sravan Kumar Chitikesi]

use where cmd to find out the path of the binary and create a shortcut to
that file on desktop

Regards,
*Sravan Chitikesi*
AWS Solutions Architect - Associate


On Mon, Apr 10, 2023 at 10:03 AM Jack Gilbert <00jhen...@gmail.com> wrote:

> I D/L 3.11.3, I can see it in CMD
>
> running W10 64bit
>
> I have IDL on my desktop,
>
> HOW do I get 3.11.3 on my desktop?
>
> Thanks
>
> Jack g
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Need help please

[Jack Gilbert]

I D/L 3.11.3, I can see it in CMD

running W10 64bit

I have IDL on my desktop,

HOW do I get 3.11.3 on my desktop?

Thanks

Jack g
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Numpy, Matplotlib crash Python 3.8 Windows 7, 32-bit - can you help ?

[Thomas Passin]

On 3/28/2023 1:50 PM, a a wrote:

On Tuesday, 28 March 2023 at 18:12:40 UTC+2, Thomas Passin wrote:

On 3/28/2023 8:47 AM, a a wrote:

Ok, I can export bookmarks to html file and open it in Firefox to get
a long list of clickable urls but icon of the bookmarked web page is missing.

When I open Bookmarks as right a side-bar I can view and identify an individual 
Boomarks by icon,
so I would like Firefox Library to export Bookmarks to html file, icons 
included 😉

Since accessing opened Tabs is my default use of history in Firefox and has 
worked fine for years
I paid no special interest to bookmark opened Tabs and assign labels to 
individual bookmark.

So, generally speaking, I am happy with 1,000+ opened Tabs in Firefox , not 
being sure if this number is for real or refers to every bookmark from the 
history + opened Tabs

But definitely I need a smarter solution and approach to manage 10,000+ opened 
Tabs in Firefox in a future 😉


I think you had better start using another name for this thread, if it
continues.

The HTML export file will contain the icons, but the HTML elements do
not provide for showing them.

I can't imagine how you can find anything among nor navigate through
1000 open tabs, let alone 10,000 in the future. I would think the memory
usage would be impossibly high. So I hope you are mostly using the
history and do not really have that many tabs open at once!



I am a plain guy, so if Firefox counted 1,000+ opened Tabs, I can be surprised, 
but have no idea how to check that number.

You are exactly right, icon URI and icon data come with saved opened Tabs,
a single example below.

So I am going to ask Firefox team to offer
export to html, modified to have :
icon, name of web page, url address
to appear in a single row (feature already supported by Firefox, when you open 
new Tab
and click: enter URL or search string - input field,
you get such list
List is limited in size for the reasons unknown to me, but feature works fine.


You should be aware that the HTML format for bookmarks is a standard 
developed back in the day by Netscape. It goes back to the early 1990s, 
I think. The FF folks will not be modifying it, since all browsers know 
how to generate it and consume it, and who knows how many software 
packages consume it.  No one can afford to have a change, even one 
that's supposed to be harmless, inadvertently break software that's 
worked for years.


They are going to need a lot of persuading.

Maybe there's something else they would be willing and able to do.  But 
you can expect that any proposed new feature will probably need to have 
some strong support.  Raymond Chen at Microsoft has written how each new 
feature proposal starts off with -100 points.  Only if the advantages 
get the score up above zero can the feature have any chance of getting 
adopted - and then it has to compete with other potential features that 
have their own scores.




--
So would prefer a horizontal list of opened Tabs
by htmlized, vertical list of the same opened Tabs,
featuring:
icon, name of web-site, URL address

Thank you for your excellent support


You're welcome.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Numpy, Matplotlib crash Python 3.8 Windows 7, 32-bit - can you help ?

[a a]

On Tuesday, 28 March 2023 at 18:12:40 UTC+2, Thomas Passin wrote:
> On 3/28/2023 8:47 AM, a a wrote: 
> > Ok, I can export bookmarks to html file and open it in Firefox to get 
> > a long list of clickable urls but icon of the bookmarked web page is 
> > missing. 
> > 
> > When I open Bookmarks as right a side-bar I can view and identify an 
> > individual Boomarks by icon,
> > so I would like Firefox Library to export Bookmarks to html file, icons 
> > included 😉
> > 
> > Since accessing opened Tabs is my default use of history in Firefox and has 
> > worked fine for years 
> > I paid no special interest to bookmark opened Tabs and assign labels to 
> > individual bookmark. 
> > 
> > So, generally speaking, I am happy with 1,000+ opened Tabs in Firefox , not 
> > being sure if this number is for real or refers to every bookmark from the 
> > history + opened Tabs 
> >
> > But definitely I need a smarter solution and approach to manage 10,000+ 
> > opened Tabs in Firefox in a future 😉 
> 
> I think you had better start using another name for this thread, if it 
> continues. 
> 
> The HTML export file will contain the icons, but the HTML elements do 
> not provide for showing them. 
> 
> I can't imagine how you can find anything among nor navigate through 
> 1000 open tabs, let alone 10,000 in the future. I would think the memory 
> usage would be impossibly high. So I hope you are mostly using the 
> history and do not really have that many tabs open at once!


I am a plain guy, so if Firefox counted 1,000+ opened Tabs, I can be surprised, 
but have no idea how to check that number.

You are exactly right, icon URI and icon data come with saved opened Tabs,
a single example below.

So I am going to ask Firefox team to offer
export to html, modified to have :
icon, name of web page, url address
to appear in a single row (feature already supported by Firefox, when you open 
new Tab
and click: enter URL or search string - input field,
you get such list
List is limited in size for the reasons unknown to me, but feature works fine.

--
So would prefer a horizontal list of opened Tabs
by htmlized, vertical list of the same opened Tabs,
featuring:
icon, name of web-site, URL address

Thank you for your excellent support

darius


"https://www.nasa.gov/feature/goddard/2017/nasas-sdo-watches-a-sunspot-turn-toward-earth";
 add_date="1499899506" last_modified="1499899507" 
icon_uri="https://www.nasa.gov/favicon.ico"; 
icon="data:image/png;base64,iVBORw0KGgoNSUhEUgAAADAwCAYAAABXAvmHAAAPkklEQVRoQ+1ZCVQUZ7b+qqo36AVommYXBARR45rnlpgEEZdoJokLmkWjyRjH5ZkzODEz5pk4mfdeEsf3krjEqHFiEk2c6KijJiqK2/h04r4EEQRBZFFkpxt6q6p3/8I2wDSIY86ZM+fMfw6H7q6//rrfXb97i8M/+eL+yeXHvwD8oy34wBaInf6ZrqbYFSlDTJQkOR4yFy5BDJRkqAQBdhlcJSdz1+lznhGGayUHplVzHCf/VMD/LgBLlsj8B8dWx4puaZQIjJZluS9JZKX/OiaYwHEwaAXUOTzNcnKcyMtcPcfjGs/hLypOtVtjMX9/e0u67UGAyEuW0HH3uUyjPk2QHI4ZJFq6JElxdDvf9gh2ql6jQoPzDgDaQOAIR/PjOPANHC8fU/PcmgCLNbNkS3rTfYoBedIkAZev9u80gNCRX+gbXLbnJFlcIElIIpF83qsReKgFDnanqPx3ky8F+KnQJViP7NI60Ne7i/DYeY7brtOq3q/JnP3D/YCQuySniQJiOgXAPHJVtMMpLxFl+Xmvm7R9GClYWcx1dGpynyY3IoP8UFzVCINOpXzOu9nQCoD3DJ7nc1Uq7j/GmEO2b9mSTl7Z8ZIjk4aLkH8haIU37wkgeMSqZLtLXiHJUmrLYyUmMROaTmA+HxGgQwP5fD39mf3VsLk8cIuyskXB1sKFfInHg69RqbAkxmL9JHtLusvXHlIeh5jkcZJHXAqe+0AoyVvbIQBjysdJHlFcT5p/xHsgE0ZFd0WRSxh0ajQ0unCrzgGrXg1/8vurlXaoVTxE5joEpNruYrJ3alGM2FQC/9a/Pd5jxeElKT8GEMM/ZIifVFL1MjzS2xR1h/hA7StcdratXQCWUWvCbQ73HyhQR7cU3o+Em/R4HIL0WpRVN0JLEcvTX3VtE07mVeKWzQmO/UY3qSkeXKKkVEuBfvO0DIB2IHEcX6cVuPn1h+Z+cfe5cT27SG7XIojydMoEV3gNP5krzM1tTgg+VsKY77RlTYXLSPvzWrkNCTCiTwQsgTpsO1ZIwslkSZCmNegZHYi4cCNyS+pw9lq1IuydpAMtAQk1aVFc07lkQzFxnVNrpzR6Zp5GQdKToiguJjM+TIq5wQv8DK44L8srl08AhpSV6W4R62VZMrQEQEkF04Z3w55TN8jf3aA0CL2fWnGZAgrQIL0Gw3qGwV+rwv7zpYr7UJZRjmBAO2EAMHsx6w11VRzaW7olXy2508kFA+j2Cl6tmssV525tKdPfAAh58rMwm82+Q5SlQW2Nw9zn+ZQEbD56DU0UpFYSmFKgklCLbjcq2zkK2b5dgzGgWzD2ny2j322Ki91rMcHZrliPDc/bcjCt7gfEuOvunIlbsqDOEG7kfN22iv/NyYbhH/+7y+35kCmt7UN15ApTUxPwzdFC1JIFmEWYVlmQet2F3UOUAnGhBqT1j8LRS+XIIbdqD4RX47GeBkywXcULDZfR3VlJD2+OfDq3GGpNBopytvmiIK0ARKZ+Hlztqd9NGWSwL40xRNNHdEPm2VLcqLLfray+9jIQLPc/PTgGhwnE5RYgmGhMcA0JmeSuVQSf0HAFia5q8LKERkEDjSyiQB2E/abEd+Zf2vR2exZsBcCYunqcy+X5hrK3X3tCTRgag1sUjH/JqVAyS0dLARHoh7GDu+DAuVIUVNjJrjxMkhsDnBWYaMvDaPs1RJOrMMHtvBZXtWa67kKxyoRFlmE4qwvbFd7VL71owwyHr2e1kkD/+KqP3JI4n21Ucvedq95NLLf3jTWTf1uwYX9ec2SyvS1ObguJEpVSM9L6ReLC8cvod+sqnrXnY5CjFEEelpVkOHg1zugicF4Xhn6Om8jRBOFd8yAUCQaWKG76qdUjaw7MutQhgLgRawLK3e49oiQNYfnbTyMoklEdgN0lKu7iTxTBSL+/ODIRn2Xm4XaDEyywWTbyFmaHRwSrQF7ftogO9Gssx4wwBwbH6mHdsxsaBwU8WaKeNP69XyS2m5Kh8bjwjP0qvjPEYZ2xF2ycyhsHkk4lzGhZF1oCuauwZsogZXkkKTwqyB/vzh6K+OggNNidWLzmBC4WVOGtaQ8jyKjD6ewy3Khuwp4zJXg0MQSLZgykdKpRqu+KLedw5EQB+ojVGGG/jlRUIHncEBgfHQiEhwEFlMGKSrFlxwV85Q5FiS4Ic1GAKdESqidORqZsxXufn0J5TePdGFMLwjL74bmvd2gBfcrKVI8HO2VI/ira+fmbaZg4qrtyz5/25+Ll/9qPjPS+SI4Nxq9XHcP4J0hTe3IRQ9kmc/l4hIU0l4yL3x5HbcZi9LaXI9BDmp71MvD2m6g8nY196/agYeBgxPeMwez3DyK0vBDzas6ix7Be6L30dXAWCzxkwZnvZOLLrDwI5AlsqXhuW3KSZsqZtbPcbUHctYAhdfUUl1vcCFmi5glY/0Yq0kd3Vw5popT589/uQ7TVgD4JIXjpvYMYOygaUnUd

Re: Numpy, Matplotlib crash Python 3.8 Windows 7, 32-bit - can you help ?

[Thomas Passin]

On 3/28/2023 8:47 AM, a a wrote:

Ok, I can export bookmarks to html file and open it in Firefox to get
a long list of clickable urls but icon of the bookmarked web page is missing.

When I open Bookmarks as right a side-bar I can view and identify an individual 
Boomarks by icon,
so I would like Firefox Library to export Bookmarks to html file, icons 
included 😉

Since accessing opened Tabs is my default use of history in Firefox and has 
worked fine for years
I paid no special interest to bookmark opened Tabs and assign labels to 
individual bookmark.

So, generally speaking, I am happy with 1,000+ opened Tabs in Firefox , not 
being sure if this number is for real or refers to every bookmark from the 
history + opened Tabs

But definitely I need a smarter solution and approach to manage 10,000+ opened 
Tabs in Firefox in a future 😉


I think you had better start using another name for this thread, if it 
continues.


The HTML export file will contain the icons, but the HTML elements do 
not provide for showing them.


I can't imagine how you can find anything among nor navigate through 
1000 open tabs, let alone 10,000 in the future. I would think the memory 
usage would be impossibly high.  So I hope you are mostly using the 
history and do not really have that many tabs open at once!


--
https://mail.python.org/mailman/listinfo/python-list

Re: Numpy, Matplotlib crash Python 3.8 Windows 7, 32-bit - can you help ?

[a a]

On Tuesday, 28 March 2023 at 06:33:44 UTC+2, Thomas Passin wrote:
> On 3/27/2023 8:37 PM, a a wrote: 
> >> To save the tabs, right click any one of them and select the "Select All 
> >> Tabs" item. They will all highlight. Right click on one of them and 
> >> select the "Bookmark Tabs" item. A dialog box will open with an entry 
> >> lone for the Name to use (like "Tabset1") and a location - a bookmark 
> >> folder - for them to go into. CAREFUL - if you just click "Save", you 
> >> may not be able to find them. Use the dropdown arrow to save them in 
> >> one of the top level folders, like "Bookmarks Toolbars". 
> > I can select All Opened Tabs (as from the given link) 
> > and get 1,000+ Opened Tabs ( I am afraid, this is s number of all saved 
> > bookmarks in the past) 
> > I go to menu, Bookmarks, Manage Boomarks and copy Tabs 
> > 
> > and 
> > https://www.textfixer.com/html/convert-url-to-html-link.php 
> > 
> > does the job, converting text urls into clickable web links 
> > 
> > I copy the result and past into Notepad++ to save file as html 
> > 
> > and what I get is web page of clickable Opened Tabs 
> > 
> > since icon and page name are lost
> I don't understand this. You don't really have 1000 tabs open at the 
> same time, do you? If you select all the open tabs - I think you wrote 
> that you only have 50 - then you can save them as bookmarks under a 
> folder name you choose. That folder will contain the 50 open links. I 
> tried it this evening, so I know that's how it works. (It happens that 
> I'm working on my own bookmark manager just now, so I've been messing 
> around with importing, exporting, and reading the bookmark files). 
> 
> Then you can export them and import the same bookmark file into another 
> browser on another computer. Whenever you want to reopen some of those 
> tabs, you would navigate to that part of the bookmarks and open the tabs 
> you want. 
> 
> Maybe you have something else in mind? Do you want to send the links of 
> the opened tab set to someone else, but not all your bookmarks? Please 
> explain more carefully what you want to do.

Ok, I can export bookmarks to html file and open it in Firefox to get
a long list of clickable urls but icon of the bookmarked web page is missing.

When I open Bookmarks as right a side-bar I can view and identify an individual 
Boomarks by icon,
so I would like Firefox Library to export Bookmarks to html file, icons 
included ;)

Since accessing opened Tabs is my default use of history in Firefox and has 
worked fine for years
I paid no special interest to bookmark opened Tabs and assign labels to 
individual bookmark.

So, generally speaking, I am happy with 1,000+ opened Tabs in Firefox , not 
being sure if this number is for real or refers to every bookmark from the 
history + opened Tabs

But definitely I need a smarter solution and approach to manage 10,000+ opened 
Tabs in Firefox in a future ;)

- I just build personal search engine resembling targets set by MyLifeBits 
Project by Microsoft in the past.

darius
-- 
https://mail.python.org/mailman/listinfo/python-list