Re: Encrypted web mail

[Uri Even-Chen]

Hi Dennis,



On Sat, Aug 15, 2015 at 7:35 PM, Dennis Lee Bieber 
wrote:

> On Sat, 15 Aug 2015 12:47:17 +0300, Uri Even-Chen 
> declaimed the following:
>
> >To Python, Django and Speedy Mail Software developers,
> >
> >Is it possible to make Speedy Mail encrypted? I want mail to be encrypted
> >on the server, and only the user will be able to read his/her mail. The
> >user's password will be encrypted on the server and nobody will be able to
>
> Most systems I know of don't store the password on the server in
> the
> first place. They store a one-way hash generated from the password
> (possibly using a randomly generated salt that is also saved with the hash
> -- that is, rather than just hash "password" into "hashstring", they hash
> "saltpassword" into "otherhash" and prepend the "salt" -> "saltotherhash".
> When user comes to connect later, they match the user name in the password
> database, extract the "salt" from "saltotherhash", attach it to the
> password given by the user, generate the hash, and see if it matches the
> rest of the saved hash). The hash value is only used for matching purposes,
> not for any subsequent processing -- it is not a cryptography key, nor is
> any cryptography key used to produce it.
>
>
Thanks for the feedback. Actually the passwords on my webmail in 2000 to
2005 were not encrypted, but I agree with you that passwords should be
always encrypted.



> >read the user's mail except the user himself. Is it possible? When  I had
>
> How do you intend to handle new inbound email? After all, the
> sender of
> the email sure won't know about any user encryption key -- that means you
> have to have the encryption key stored on your server associated with the
> recipient username, so that you can encrypt inbound email before putting it
> into the user's mailbox... Do you also intend to encrypt the header
> information or just the body of the message?
>
> A public key system MIGHT support that, in that the public key --
> used
> to "send to" the recipient is only used to encrypt the data, and can be
> stored on your server (in the same username/password account file). The
> private (decryption) key would have to be stored on the user's computer and
> never provided to your server machine -- and you'd need some way to send
> individual encrypted messages to the user where they are decrypted on their
> computer, NOT ON the server. You'd also need to be able to identify which
> messages are new, read, deleted -- if the mailbox is encrypted, this likely
> means each message is a file within the mailbox, since you can't do things
> like mark and compress an MBOX (all mail is in one file with a special
> header marking the start of a message) file without corrupting the
> encryption stream.
>
> If, at anytime, the decryption key is on the server, you have lost
> the
> security you claim to be striving for -- as any court ordered system could
> just patch in a packet sniffer and wait for your user to connect, capture
> the password, and capture the decryption key if it is sent to the server to
> retrieve mail (though they don't even need it at that point -- they could
> just capture the decrypted contents being sent to the user... TLS/SSL
> sessions may alleviate that problem, but it does mean having certificates
> to initiate the TLS session keys). If the packets are TLS encrypted, they
> can require one to patch into the server at the point where the contents
> are converted back to plain text and capture the traffic.
>
> Of course, this now means the user has to carry around a "keyring"
> that
> can be accessed by any computer used to read the email (since the
> decryption key can not be on the server, if they read email from an android
> tablet they need to have the key installed on the tablet; they also need it
> on their desktop if they use it to access the server; on their phone if it
> has a browser, etc.).
>
> A Javascript client is probably going to be rather slow at
> decrypting
> the emails -- but you may not be able to install a compiled Java encryption
> package on all the clients out there (you'd have to have something for iOS,
> something for Android, for Linux, Macintosh, and Windows -- though the
> latter three might be able to use the same core Java code).
>
> We've taken care of inbound email. What were your plans for
> outgoing
> email? You can't encrypt that as it is going to other systems and other
> users who are not part of your server and don't expect to see encrypted
> stuff. You could perhaps store the already sent messages using the same
> public key, but you can't do that with in-work drafts stored on the server
> prior to being sent (at least, not without requiring them to pass from the
> server to the client for decryption and then back to the server in
> plain-text for delivery -- deleting the draft copy and saving an encrypted
> sent copy)
>
> I think ProtonMail  is doing s

Re: How to model government organization hierarchies so that the list can expand and compress

[Denis McMahon]

On Thu, 13 Aug 2015 12:10:12 -0700, Alex Glaros wrote:

> What would the data model look like for this?

Define an organizational unit. Give it a unique name. Give it a list of 
superiors and a list of subordinates.

government = {

'president' : { 'superiors' : [], 'subordinates' : ['jcs', 'cia', 'fbi', 
'nsa', 'treasury', 'nasa' . ] },

'jcs' : { 'superiors': ['president', 'congress', 'senate', 'treasury'], 
'subordinates' : ['army', 'navy', 'air force', 'jsoc'] }, 

'navy' : { 'superiors' : ['jcs', 'nsa', 'cia'], 'subordinates' : 
['seals', 'marines', 'pacific fleet' ] }, 

}

The multiple parenting means that you need to use something as 
references. You can't represent the hierarchy as a simple tree, because 
in a simple tree a node only has one parent.

-- 
Denis McMahon, denismfmcma...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory control in Python

[Laura Creighton]

If the problem is that Python is using too much memory, then PyPy may
be able to help you.  PyPy is an alternative implementation of Python,
and by defaiult uses a minimark garbage collector.
https://pypy.readthedocs.org/en/release-2.4.x/garbage_collection.html

You will have to write your own bindings for the CPLEX C library, though,
using cffi. http://cffi.readthedocs.org/en/latest/overview.html  (since
the bindings you have assume the CPython ref counting gc).

But if your C program is itself using too much memory, then this probably
won't help.

Discuss this more on pypy-...@python.org or the #pypy channel on freenode.
People on pypy-dev would appreciate not getting libreoffice spreadsheet
attachments but just the figures as plain text.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory control in Python

[Mark Lawrence]

On 15/08/2015 18:28, Terry Reedy wrote:

On 8/15/2015 3:21 AM, dieter wrote:

Ping Liu  writes:

...
For small cases, Python works well. But if we consider longer time
period.
then it would fail due to the memory usage issues. We have tested
several
case studies to check the memory use for different time period,
including
1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
each day, as well as 4) 30 days with 24 hours each day. The first 3
cases
are feasible while the last case gives out the memory error.

When we are testing the forth case, the memory error comes out while
creating the inequality constraints. The problem size is 1) Aeq: 12 *
26,
Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284,
Aineq:
6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory
error.

The solver is CPLEX (academic). It turns out that the solver is taking a
lot of memory as you can see in the memory test report. for the first
three
cases, different memory usage is observed, and it grows up dramatically
with the increase of the time period. 1) solver memory usage: 25.6
MB, 2)
19.5 MB; 3) solver memory usage: 830.0742 MB.


Make sure that the solver is using numpy arrays.



I doubt that as CPLEX was first released in 1988.

A very quick bit of searching found this 
http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linprog.html 
which I believe is the equivalent of the Aeq and Aineq mentioned above. 
 Possibly a better option as must surely be using numpy, but as usual 
there's only one way to find out? :)


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: memory control in Python

[Terry Reedy]

On 8/15/2015 3:21 AM, dieter wrote:

Ping Liu  writes:

...
For small cases, Python works well. But if we consider longer time period.
then it would fail due to the memory usage issues. We have tested several
case studies to check the memory use for different time period, including
1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
are feasible while the last case gives out the memory error.

When we are testing the forth case, the memory error comes out while
creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.

The solver is CPLEX (academic). It turns out that the solver is taking a
lot of memory as you can see in the memory test report. for the first three
cases, different memory usage is observed, and it grows up dramatically
with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
19.5 MB; 3) solver memory usage: 830.0742 MB.


Make sure that the solver is using numpy arrays.


The C implementation of Python (called "CPython") does not use
memory compaction and places most of its objects on the heap.
Those implementations tend to suffer from memory fragmentation in
long running processes with large memory use.


This can be helped somewhat by reusing arrays instead of deleting and 
creating a new one.




You may consider "Jython" (a Python implementation in "Java") or
"IronPython" (a ".Net" based Python implementation) to get a
Python implementation with memory compaction.




--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Encrypted web mail

[Uri Even-Chen]

Hi Michael,

Thanks for the feedback. Actually the passwords on my webmail in 2000 to
2005 were not encrypted, but I agree with you that passwords should be
always encrypted.

Uri.

*Uri Even-Chen*   [image: photo] Phone: +972-54-3995700
Email: u...@speedy.net
Website: http://www.speedysoftware.com/uri/en/
  
    
> Speedypedia in Hebrew and English


On Sat, Aug 15, 2015 at 6:04 PM, Michael Torrie  wrote:

> On 08/15/2015 03:47 AM, Uri Even-Chen wrote:
> > To Python, Django and Speedy Mail Software developers,
> >
> > Is it possible to make Speedy Mail encrypted? I want mail to be encrypted
> > on the server, and only the user will be able to read his/her mail. The
> > user's password will be encrypted on the server and nobody will be able
> to
> > read the user's mail except the user himself. Is it possible? When  I had
> > Speedy Mail from 2000 to 2005 I received a court order by a court in Tel
> > Aviv once from two policemen to give a specific user's password to the
> > Israeli police, and the password and mail were not encrypted then. And I
> > was not allowed to tell anyone about this court order, except my lawyer.
> > But I refused to give the user's password to the police. So if I receive
> a
> > court order again, I don't want to be able to give the user's password or
> > mail to anyone, and I don't want it to be on the servers unless it's
> > encrypted. That's why I want it to be encrypted. If I receive a court
> order
> > again I want to be able to tell the police that the mail and password are
> > encrypted, and only the user knows the password and is able to read his
> > mail. Is it possible?
>
> Could you encrypt your email body with gpg (say with a subprocess) and
> then email that?
>
> As to the username and password, I've never worked on a system where
> passwords could even be extracted by a sysadmin.  I could tell you the
> salted hash but never the password.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Encrypted web mail

[Michael Torrie]

On 08/15/2015 03:47 AM, Uri Even-Chen wrote:
> To Python, Django and Speedy Mail Software developers,
> 
> Is it possible to make Speedy Mail encrypted? I want mail to be encrypted
> on the server, and only the user will be able to read his/her mail. The
> user's password will be encrypted on the server and nobody will be able to
> read the user's mail except the user himself. Is it possible? When  I had
> Speedy Mail from 2000 to 2005 I received a court order by a court in Tel
> Aviv once from two policemen to give a specific user's password to the
> Israeli police, and the password and mail were not encrypted then. And I
> was not allowed to tell anyone about this court order, except my lawyer.
> But I refused to give the user's password to the police. So if I receive a
> court order again, I don't want to be able to give the user's password or
> mail to anyone, and I don't want it to be on the servers unless it's
> encrypted. That's why I want it to be encrypted. If I receive a court order
> again I want to be able to tell the police that the mail and password are
> encrypted, and only the user knows the password and is able to read his
> mail. Is it possible?

Could you encrypt your email body with gpg (say with a subprocess) and
then email that?

As to the username and password, I've never worked on a system where
passwords could even be extracted by a sysadmin.  I could tell you the
salted hash but never the password.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Detecting if a library has native dependency

[Laura Creighton]

In a message of Sat, 15 Aug 2015 17:10:01 +0530, Harish Vishwanath writes:
>Hello
>
>Is there a reliable way to detect if a python library has native dependency
>or native code? For ex. can I programmatically determine that "lxml"
>package has native code and depends on the presence of libxml on the system?
>
>Regards,
>Harish

I'm using snakefood http://furius.ca/snakefood/ to make dependency graphs.
You can get it from PyPI.  It uses its own tool called sfood to make a
list of dependencies, but I am not sure how tweakable that is to get
the list of what you want.  Worth a look, at any rate.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Encrypted web mail

[Uri Even-Chen]

Thanks Pedro, actually I understand cryptography and I agree with you.
Sonebody replied to me off-list and wrote me about ProtonMail
, actually they are doing exactly what I wanted (I
think), so maybe I will try to cooperate with them and/or use their open
source source code. Anyway thanks for the feedback!

Uri.


*Uri Even-Chen*   [image: photo] Phone: +972-54-3995700
Email: u...@speedy.net
Website: http://www.speedysoftware.com/uri/en/
  
    
> Speedypedia in Hebrew and English


On Sat, Aug 15, 2015 at 3:32 PM, Pedro  wrote:

> Hello Uri,
>
> what you want is completely possible, but demands you to understand some
> concepts of cryptography. To make your server completely safe against court
> orders or anyone that have access to it, including yourself, you need to
> encrypt and decrypt the emails at client side. For this you may use some
> public-key cryptosystem and store all user's public keys at your server
> (they are not secret). Probably the best option for you is to use PGP. If
> the clients run in Javascript you can use OpenPGP.js for this. It is very
> easy to use and you won't need to know anything about how to implement a
> secure cipher.
>
>
>
> --
> Pedro Alves
>
> *Mestrando em Ciência da Computação*
> *Universidade Estadual de Campinas*
>
> 2015-08-15 6:47 GMT-03:00 Uri Even-Chen :
>
>> To Python, Django and Speedy Mail Software developers,
>>
>> Is it possible to make Speedy Mail encrypted? I want mail to be encrypted
>> on the server, and only the user will be able to read his/her mail. The
>> user's password will be encrypted on the server and nobody will be able to
>> read the user's mail except the user himself. Is it possible? When  I had
>> Speedy Mail from 2000 to 2005 I received a court order by a court in Tel
>> Aviv once from two policemen to give a specific user's password to the
>> Israeli police, and the password and mail were not encrypted then. And I
>> was not allowed to tell anyone about this court order, except my lawyer.
>> But I refused to give the user's password to the police. So if I receive a
>> court order again, I don't want to be able to give the user's password or
>> mail to anyone, and I don't want it to be on the servers unless it's
>> encrypted. That's why I want it to be encrypted. If I receive a court order
>> again I want to be able to tell the police that the mail and password are
>> encrypted, and only the user knows the password and is able to read his
>> mail. Is it possible?
>>
>> I believe a user's mail is something personal, like his thoughts. I don't
>> want the police to read my mail and it's similar to reading my thoughts.
>>
>> Thanks,
>> Uri.
>>
>> *Uri Even-Chen*   [image: photo] Phone: +972-54-3995700
>> Email: u...@speedy.net
>> Website: http://www.speedysoftware.com/uri/en/
>> 
>> 
>> 
>> 
>> > Speedypedia in Hebrew and English
>> 
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-us...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/CAMQ2MsHDNLRuth59FRAtXwMXY14LyMUibxHCFxZswdD7Jw4oyg%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-us...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CACW_pa3PGDg0b6zNQertR0NXHZPDd75oh1ZcVU12SXy9aN%3DmiQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Detecting if a library has native dependency

[Harish Vishwanath]

Hello

Is there a reliable way to detect if a python library has native dependency
or native code? For ex. can I programmatically determine that "lxml"
package has native code and depends on the presence of libxml on the system?

Regards,
Harish
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7: _PyLong_NumBits() Segfault

[Ned Batchelder]

On Saturday, August 15, 2015 at 2:36:44 AM UTC-4, Adam Meily wrote:
> I am working on a CPython library that serializes Python objects to disk in a 
> custom format. I'm using _PyLong_NumBits() to determine whether 
> PyLong_AsLong(), PyLong_AsUnsignedLong(), PyLong_AsLongLong(), or 
> PyLong_AsUnsignedLongLong() should be used.
> 
> 
> In Python 3.x, I'm able to determine how many bits are required to write an 
> int to disk without issue. However, in Python 2.7, _PyLong_NumBits() 
> segfaults when the type is a PyInt that has the value of 0x (maximum 
> unsigned 32-bit value). Additionally, _PyLong_NumBits() is not accurate when 
> the number is negative. My guess is that I can't use _PyLong_NumBits() on a 
> PyInt object. So, what is the correct way to determine the number of bits 
> required to represent a PyInt object in Python 2.7?
> 
> 
> I'm using Python 2.7.6, x86-64 on Ubuntu.

You are right that _PyLong_NumBits is only usable with long objects. PyInt
has PyInt_GetMax, which will tell you the maximum size for an int.  All ints
will fit in that number of bits, so you can use that size for any int.

Keep in mind that _PyLong_NumBits is not a documented and supported API, so
you'll have to be very careful using it.

Also, you might consider doing work in Python that doesn't absolutely have
to be done in C. It's much easier working in Python than in C.

--Ned.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mock object but also assert method calls?

[Thomas Lehmann via Python-list]

Am Freitag, 14. August 2015 04:53:56 UTC+2 schrieb Steven D'Aprano:
> On Fri, 14 Aug 2015 07:21 am, Ben Finney wrote:
> 
> >> If find following totally different to the normal API which
> >> is provided by the mock library:
> >>
> >> assert call().test2("hello") in mocked_objects.mock_calls
> > 
> > The 'assert' statement is a crude tool, which knows little about the
> > intent of your assertion.
> 
> I agree with Ben here. Despite the popularity of "nose" (I think it is
> nose?) which uses `assert` for testing, I think that is a gross misuse of
> the statement. It is okay to use assertions this way for quick and dirty ad
> hoc testing, say at the command line, but IMO totally inappropriate for
> anything more formal, like unit testing.
> 
> If for no other reason than the use of `assert` for testing makes it
> impossible to test your code when running with the Python -O (optimize)
> switch.
> 
> For more detail on the uses, and abuses, of `assert` see this:
> 
Of course you do NOT use assert in unit tests but I provided
just test code to show my problem. Of course I take nose and
hamcrest and code coverage ...

Here a complete example of my problem (with comments):


from mock import MagicMock, call, patch

class Bla:
def test1(self, value):
pass

mocked_object = MagicMock(Bla)
bla = Bla()
bla.test1("hello")
# empty, why?
print(mocked_object.mock_calls)
# does not work: mocked_object.test1.assert_called_with("hello")

with patch("__main__.Bla") as mocked_object:
bla = Bla()
bla.test1("hello")
# not empty!
print(mocked_object.mock_calls)
# does also not work: mocked_object.test1.assert_called_with("hello")
# but this does work:
assert call().test1("hello") in mocked_object.mock_calls


I don't wanna patch each individual method. Is there no other way?
-- 
https://mail.python.org/mailman/listinfo/python-list

Encrypted web mail

[Uri Even-Chen]

To Python, Django and Speedy Mail Software developers,

Is it possible to make Speedy Mail encrypted? I want mail to be encrypted
on the server, and only the user will be able to read his/her mail. The
user's password will be encrypted on the server and nobody will be able to
read the user's mail except the user himself. Is it possible? When  I had
Speedy Mail from 2000 to 2005 I received a court order by a court in Tel
Aviv once from two policemen to give a specific user's password to the
Israeli police, and the password and mail were not encrypted then. And I
was not allowed to tell anyone about this court order, except my lawyer.
But I refused to give the user's password to the police. So if I receive a
court order again, I don't want to be able to give the user's password or
mail to anyone, and I don't want it to be on the servers unless it's
encrypted. That's why I want it to be encrypted. If I receive a court order
again I want to be able to tell the police that the mail and password are
encrypted, and only the user knows the password and is able to read his
mail. Is it possible?

I believe a user's mail is something personal, like his thoughts. I don't
want the police to read my mail and it's similar to reading my thoughts.

Thanks,
Uri.

*Uri Even-Chen*   [image: photo] Phone: +972-54-3995700
Email: u...@speedy.net
Website: http://www.speedysoftware.com/uri/en/
  
    
> Speedypedia in Hebrew and English

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory control in Python

[dieter]

Ping Liu  writes:
> ...
> For small cases, Python works well. But if we consider longer time period.
> then it would fail due to the memory usage issues. We have tested several
> case studies to check the memory use for different time period, including
> 1) 2 hours in one day, 2) 24 hours in one day, 3) 20 days with 24 hours
> each day, as well as 4) 30 days with 24 hours each day. The first 3 cases
> are feasible while the last case gives out the memory error.
>
> When we are testing the forth case, the memory error comes out while
> creating the inequality constraints. The problem size is 1) Aeq: 12 * 26,
> Aineq: 30 * 26; 2) Aeq: 144*268, Aineq:316*268; 3) Aeq: 2880*5284, Aineq:
> 6244*5284; 4) Aeq: 4320 * 7924, Aineq is unknown due to the memory error.
>
> The solver is CPLEX (academic). It turns out that the solver is taking a
> lot of memory as you can see in the memory test report. for the first three
> cases, different memory usage is observed, and it grows up dramatically
> with the increase of the time period. 1) solver memory usage: 25.6 MB, 2)
> 19.5 MB; 3) solver memory usage: 830.0742 MB.

The C implementation of Python (called "CPython") does not use
memory compaction and places most of its objects on the heap.
Those implementations tend to suffer from memory fragmentation in
long running processes with large memory use.

You may consider "Jython" (a Python implementation in "Java") or
"IronPython" (a ".Net" based Python implementation) to get a
Python implementation with memory compaction.

-- 
https://mail.python.org/mailman/listinfo/python-list