Re: [Tutor] string to binary and back... Python 3

[eryksun]

On Fri, Jul 20, 2012 at 3:16 AM, Alan Gauld  wrote:
> On 20/07/12 06:48, wolfrage8...@gmail.com wrote:
>
> Using the format method it would look like:
>
> newhexdata = bytes("{0:x}".format(numdata), "ascii")

binascii.unhexlify needs an even number of hexadecimal digits (two per
byte). So you need to either modify the above string to prepend a '0'
if the length of newhexdata is odd, or calculate the size ahead of
time and format with zero padding:

nbytes = (numdata.bit_length() + 7) // 8
size = nbytes * 2
newhexdata = bytes('{0:0{1}x}'.format(numdata, size), 'ascii')

Though creating an intermediate string of hex digits seems the long
way around. In Python 3, I'd skip binascii and instead use
int.from_bytes and int.to_bytes:

>>> numdata = 0x0102
>>> nbytes = (numdata.bit_length() + 7) // 8
>>> numdata.to_bytes(nbytes, 'big')
b'\x01\x02'

That said, it still seems more reasonable and flexible to me to use a
generator expression to do the XOR byte by byte instead of turning the
message into a big integer. You could share a large file of random
data and send a starting position along with the encrypted text. Best
of luck.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Alan Gauld]

On 20/07/12 06:48, wolfrage8...@gmail.com wrote:


Thanks I will give this a try. Can you explian a little further for me
what exactly this:

newhexdata = bytes("%x" % numdata, "ascii")
line is doing? I don't quite understand the use of the "%x" % on numdata.


It is standard string formatting in pre Python 3 style.
Thus "%x" says I want a string containing a hex  number
and % numdata says use the value of hexdata to populate
the string.

Using the format method it would look like:

newhexdata = bytes("{0:x}".format(numdata), "ascii")

HTH
-
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Dave Angel]

You forgot to include the list.  So I'm forwarding it now, with new remarks.

On 07/20/2012 01:47 AM, wolfrage8...@gmail.com wrote:
> On Fri, Jul 20, 2012 at 12:33 AM, Dave Angel  wrote:
> 
>> On 07/19/2012 05:55 PM, Prasad, Ramit wrote:
 I am not sure how to answer that question because all files are binary,
 but the files that I will parse have an encoding that allows them to be
 read in a non-binary output. But my program will not use the in a
 non-binary way, that is why I plan to open them with the 'b' mode to
 open them as binary with no encoding assumed by python. I just not have
 tested this new technique that you gave me on a binary file yet as I was
 still implementing it for strings.
>>> As far as I know, even in binary mode, python will convert the
>>> binary data to read and write strings. So there is no reason
>>> this technique would not work for binary. Note, I was able to use
>>> the string representation of a PDF file to write another PDF file.
>>> So you do not need to worry about the conversion of binary to strings.
>>> All you need to do is convert the string to int, encrypt, decrypt,
>>> convert back to string, and write out again.
>>>
>>> Note Python3 being Unicode might change things a bit. Not sure if
>>> you will need to convert to bytes or some_string.decode('ascii').
>>
>> In Python 3, if you open the file  with "b"  (as Jordan has said), it
>> creates a bytes object.  No use of strings needed or wanted.  And no
>> assumptions of ascii, except for the output of the % operator on a hex
>> conversion.
>>
>>
>> myfile = open(filename, "b")
>> data = myfile.read(size)
>>
>> At that point, convert it to hex with:
>>hexdata = binascii.hexlify(data)
>> then convert that to an integer:
>>numdata = int(hexdata, 16)
>>
>> At that point, it's ready to xor with the one-time key, which had better
>> be the appropriate size to match the data length.
>>
>> newhexdata = bytes("%x" % numdata, "ascii")
>> newdata = binascii.unhexlify(newhexdata)
>>
>> If the file is bigger than the key, you have to get a new key. If the
>> keys are chosen with a range of 2**200, then you'd read and convert the
>> file 25 bytes at a time.
>>
> Thanks I will give this a try. Can you explian a little further for me what
> exactly this:
> newhexdata = bytes("%x" % numdata, "ascii")
> line is doing? I don't quite understand the use of the "%x" % on numdata.
> 

When you see an expression you don't understand, then try to decompose it.

"%x" % numdata is just one of many ways of converting an integer into a
hex string.  The %x is a format specifier, and says to use hex.
Actually, with Python 3, the format method is supposed to be preferred,
but I've been doing it the old way to long to learn format quickly.

bytes() is how we encode a string into bytes.  Since I know all the
characters created by my %x expression will be valid ASCII, I can safely
tell it to encode it using ASCII.

This isn't the only way to solve the original problem by a long shot,
but you did say the one-time pad was a numeric int.  If it had been a
byte string, the problem would have been much simpler, though probably
not as quick.

-- 

DaveA


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] string to binary and back... Python 3

[wolfrage8...@gmail.com]

  On Fri, Jul 20, 2012 at 12:33 AM, Dave Angel  wrote:

> On 07/19/2012 05:55 PM, Prasad, Ramit wrote:
> >> I am not sure how to answer that question because all files are binary,
> >> but the files that I will parse have an encoding that allows them to be
> >> read in a non-binary output. But my program will not use the in a
> >> non-binary way, that is why I plan to open them with the 'b' mode to
> >> open them as binary with no encoding assumed by python. I just not have
> >> tested this new technique that you gave me on a binary file yet as I was
> >> still implementing it for strings.
> > As far as I know, even in binary mode, python will convert the
> > binary data to read and write strings. So there is no reason
> > this technique would not work for binary. Note, I was able to use
> > the string representation of a PDF file to write another PDF file.
> > So you do not need to worry about the conversion of binary to strings.
> > All you need to do is convert the string to int, encrypt, decrypt,
> > convert back to string, and write out again.
> >
> > Note Python3 being Unicode might change things a bit. Not sure if
> > you will need to convert to bytes or some_string.decode('ascii').
>
> In Python 3, if you open the file  with "b"  (as Jordan has said), it
> creates a bytes object.  No use of strings needed or wanted.  And no
> assumptions of ascii, except for the output of the % operator on a hex
> conversion.
>
>
> myfile = open(filename, "b")
> data = myfile.read(size)
>
> At that point, convert it to hex with:
>hexdata = binascii.hexlify(data)
> then convert that to an integer:
>numdata = int(hexdata, 16)
>
> At that point, it's ready to xor with the one-time key, which had better
> be the appropriate size to match the data length.
>
> newhexdata = bytes("%x" % numdata, "ascii")
> newdata = binascii.unhexlify(newhexdata)
>
> If the file is bigger than the key, you have to get a new key. If the
> keys are chosen with a range of 2**200, then you'd read and convert the
> file 25 bytes at a time.
>
Thanks I will give this a try. Can you explian a little further for me what
exactly this:

newhexdata = bytes("%x" % numdata, "ascii")
line is doing? I don't quite understand the use of the "%x" % on numdata.

>
>
> --
>
> DaveA
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[eryksun]

On Thu, Jul 19, 2012 at 5:32 PM, Jordan  wrote:
>
> I am not sure how to answer that question because all files are binary,
> but the files that I will parse have an encoding that allows them to be
> read in a non-binary output. But my program will not use the in a
> non-binary way, that is why I plan to open them with the 'b' mode to
> open them as binary with no encoding assumed by python. I just not have
> tested this new technique that you gave me on a binary file yet as I was
> still implementing it for strings.

Reading from a file in binary mode returns a bytes object in Python 3.
Since iterating over bytes returns ints, you can cycle the key over
the plain text using zip and compute the XOR without having to convert
the entire message into a single big number in memory. Here's my
example from before, adapted for files:

>>> from itertools import cycle
>>> key = b'1234'
>>> kit = cycle(key)
>>> with open('temp.txt', 'rb') as f, open('cipher.txt', 'wb') as fo:
... fit = iter(lambda: f.read(512), b'')
... for text in fit:
... fo.write(bytes(x^y for x,y in zip(text, kit)))

Since the input file could be arbitrarily large and lack newlines, I'm
using "iter" to create a special iterator that reads 512-byte chunks.
The iterator stops when "read" returns an empty bytes object (i.e.
b''). You could use a while loop instead.

I assume here that the key is possibly shorter than the message (e.g.
encrypting 1 megabyte of text with a 128 byte key). If you're making a
one-time pad I think the key is the same length as the message. In
that case you wouldn't have to worry about cycling it. Anyway, I'm not
particularly interested in cryptography. I'm just trying to help with
the operations.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Dave Angel]

On 07/19/2012 05:55 PM, Prasad, Ramit wrote:
>> I am not sure how to answer that question because all files are binary,
>> but the files that I will parse have an encoding that allows them to be
>> read in a non-binary output. But my program will not use the in a
>> non-binary way, that is why I plan to open them with the 'b' mode to
>> open them as binary with no encoding assumed by python. I just not have
>> tested this new technique that you gave me on a binary file yet as I was
>> still implementing it for strings.
> As far as I know, even in binary mode, python will convert the 
> binary data to read and write strings. So there is no reason 
> this technique would not work for binary. Note, I was able to use
> the string representation of a PDF file to write another PDF file.
> So you do not need to worry about the conversion of binary to strings.
> All you need to do is convert the string to int, encrypt, decrypt, 
> convert back to string, and write out again.
>
> Note Python3 being Unicode might change things a bit. Not sure if
> you will need to convert to bytes or some_string.decode('ascii').

In Python 3, if you open the file  with "b"  (as Jordan has said), it
creates a bytes object.  No use of strings needed or wanted.  And no
assumptions of ascii, except for the output of the % operator on a hex
conversion.


myfile = open(filename, "b")
data = myfile.read(size)

At that point, convert it to hex with:
   hexdata = binascii.hexlify(data)
then convert that to an integer:
   numdata = int(hexdata, 16)

At that point, it's ready to xor with the one-time key, which had better
be the appropriate size to match the data length.

newhexdata = bytes("%x" % numdata, "ascii")
newdata = binascii.unhexlify(newhexdata)

If the file is bigger than the key, you have to get a new key. If the
keys are chosen with a range of 2**200, then you'd read and convert the
file 25 bytes at a time.



-- 

DaveA

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> I am not sure how to answer that question because all files are binary,
> but the files that I will parse have an encoding that allows them to be
> read in a non-binary output. But my program will not use the in a
> non-binary way, that is why I plan to open them with the 'b' mode to
> open them as binary with no encoding assumed by python. I just not have
> tested this new technique that you gave me on a binary file yet as I was
> still implementing it for strings.

As far as I know, even in binary mode, python will convert the 
binary data to read and write strings. So there is no reason 
this technique would not work for binary. Note, I was able to use
the string representation of a PDF file to write another PDF file.
So you do not need to worry about the conversion of binary to strings.
All you need to do is convert the string to int, encrypt, decrypt, 
convert back to string, and write out again.

Note Python3 being Unicode might change things a bit. Not sure if
you will need to convert to bytes or some_string.decode('ascii').

Now if you end up needing to handle non-ASCII data, then this exercise
gets more complicated. Not sure if a simple way to convert all characters
to a numerical point, but it should still be possible. If your data
is binary, then I do not think you will run into any issues.


Ramit


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 10:48 PM, Prasad, Ramit wrote:
 

 OK. I am using one time pads to XOR data, but the one time pads (keys)
 are very large numbers, converting them to binary increases their size
 exponentially, which allows me to get more XORing done out of a single
>>> You want to explain this impossibility of increasing size
>>> exponentially?  If you're wanting to waste memory, there are better
>>> ways.  But it's only 8 times as big to save a string of 1's and zeros as
>>> to save the large-int they represent.  And multiplying by 8 isn't an
>>> exponential function.
>>>
>> Yes if you wish to dissect my words the wrong word was chosen...
 key. I am XORing both files and strings so I need to have code that can
 do both even if that means two branches of code via an if/else perhaps
 with an isinstance(data, str).
 I do not need to actually see the binary form.

>>> Then don't use the binary form.  It doesn't make the computation any
>>> more "powerful" and it'll certainly slow it down.
>> The title of the question is string to binary and back.
>>> Are you trying to match some other program's algorithm, and thus have
>>> strange constraints on your data?  Or are you simply trying to make a
>>> secure way to encrypt binary files, using one-time pads?
>> I already answered this question...
> Yes, you stated that it had to work on string and files, but are the 
> files binary? DaveA and I are asking the questions because given
> what you are asking it just seems like you are not using the Right
> approach. I can touch my nose by touching my nose with my hand, 
> or asking the person next to me to pick up my hand and use it to
> touch my nose. Both work, one is just faster and easier to
> understand.
I am not sure how to answer that question because all files are binary,
but the files that I will parse have an encoding that allows them to be
read in a non-binary output. But my program will not use the in a
non-binary way, that is why I plan to open them with the 'b' mode to
open them as binary with no encoding assumed by python. I just not have
tested this new technique that you gave me on a binary file yet as I was
still implementing it for strings.
I may not be using the right appraoch that is why I am asking. I also
understand why the questions are needed, so you can understand my
intent, so that you can better help me. But since DaveA and I had a
misunderstanding over the missing indentation, for which I apologized
and explained that my email editor is stripping the spaces, he seems to
be badgering me.

"You want to explain this impossibility of increasing size
exponentially?  If you're wanting to waste memory, there are better
ways."

Now I would like to make it clear I very much so appreciate the help! So
again, Thank you.
>
>>> A one-time pad is the same size as the message, so you simply need to
>>> convert the message into a large-int, and xor them.
> Ramit
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of
> securities, accuracy and completeness of information, viruses,
> confidentiality, legal privilege, and legal entity disclaimers,
> available at http://www.jpmorgan.com/pages/disclosures/email.  
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> > bin_data = [ bin(ord(char)).split('b')[1].zfill(8) for char in data ]
> > bin_string = ''.join(bin_data)
> > bin_list = [ chr( int(char, 2) ) for char in bin_data ]
> Thank you exactly what I was looking for!
> >
> > I am not really sure what you are getting at with XOR and one time
> > padding, but it has been a while since I have done any encryption.
> And I have just started reading "Applied Cryptography", so I am putting
> some of what I learn into practice.
> >
> > I would think you could do all this by just converting everything
> > to int and then adding/replacing the pad in the list of ints.
> At first I was essentially doing just that, but when I first converted
> the large integers that are being used for the one time pad as the key
> to binary I saw how much larger it was, and then realized that was the
> bit length of the integer (technically Long). By doing that, I can get
> more out of the one time pad, but if you XOR binary against Ord, very
> few values will be changed because binary is only 1s and 0s as you know.
> To optimize the keys use, whether it wastes memory or not, I wanted to
> use binary on binary, this really comes into play with files, not so
> much the shorter strings.

How are you XOR-ing "binary" against something else? At a low level the data
is pretty similar so that they should be mostly interchangeable. It is
when you start abstracting the data that you have to convert between
abstractions.

Hold on, let me try a different angle. int, binary, and hex version of 
a number (lets say 65) are all just different representations of the same
number. The only thing that changes is the base.

65 in octal (base 10) is 65
65 in hex (base 16) is 41
65 in binary (base 2 ) is 101

But they are ALL the same number. 
>>> int( '65', 10 )
65
>>> int( '41', 16 )
65
>>> int( '101', 2 )
65


> But since you bring if up, how would you convert a file to a list of ints?

with open(filename, 'r' ) as f:
ints = [ ord( char ) for line in f for char in line ]

Now all you need to do is modify the list to include your padding.

> but when I first converted
> the large integers that are being used for the one time pad as the key
> to binary I saw how much larger it was, and then realized that was the
> bit length of the integer (technically Long). By doing that, I can get
> more out of the one time pad,

Large integers? Are you adding the integers for some reason? Extended 
ASCII only has ordinal values less than 256.

Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Dave Angel]

On 07/18/2012 05:07 PM, Jordan wrote:
> OK so I have been trying for a couple days now and I am throwing in the
> towel, Python 3 wins this one.

I should have paid more attention to this the first time.  Clearly you
don't want help.


-- 

DaveA

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> >> 
> >>
> >> OK. I am using one time pads to XOR data, but the one time pads (keys)
> >> are very large numbers, converting them to binary increases their size
> >> exponentially, which allows me to get more XORing done out of a single
> > You want to explain this impossibility of increasing size
> > exponentially?  If you're wanting to waste memory, there are better
> > ways.  But it's only 8 times as big to save a string of 1's and zeros as
> > to save the large-int they represent.  And multiplying by 8 isn't an
> > exponential function.
> >
> Yes if you wish to dissect my words the wrong word was chosen...
> >> key. I am XORing both files and strings so I need to have code that can
> >> do both even if that means two branches of code via an if/else perhaps
> >> with an isinstance(data, str).
> >> I do not need to actually see the binary form.
> >>
> > Then don't use the binary form.  It doesn't make the computation any
> > more "powerful" and it'll certainly slow it down.
> The title of the question is string to binary and back.
> >
> > Are you trying to match some other program's algorithm, and thus have
> > strange constraints on your data?  Or are you simply trying to make a
> > secure way to encrypt binary files, using one-time pads?
> I already answered this question...

Yes, you stated that it had to work on string and files, but are the 
files binary? DaveA and I are asking the questions because given
what you are asking it just seems like you are not using the Right
approach. I can touch my nose by touching my nose with my hand, 
or asking the person next to me to pick up my hand and use it to
touch my nose. Both work, one is just faster and easier to
understand.

> >
> > A one-time pad is the same size as the message, so you simply need to
> > convert the message into a large-int, and xor them.

Ramit
This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

Sorry, I am not sure why Thunderbird is stripping the spaces, may have
something to do with a plug-in that I have installed, I will have to
look into it.

On 07/19/2012 10:41 PM, Prasad, Ramit wrote:
> Sure, this makes perfect sense to me :) (adding indent)
>
> for char in data:
> bin_data += bin(ord(char)).split('b')[1].zfill(8)
> bin_list = [bin_data[x:x + 2] for x in range(0, len(bin_data), 2)]
>
> Why are you grabbing 2 binary digits? The only possibilities are 0,1,2,3
> and none are ASCII letters. You should be grabbing 8 at a time.
Right, sorry, first time working with binary and I was confused by a
previous attempt.
>
> bin_data = [ bin(ord(char)).split('b')[1].zfill(8) for char in data ]
> bin_string = ''.join(bin_data)
> bin_list = [ chr( int(char, 2) ) for char in bin_data ]
Thank you exactly what I was looking for!
>
> I am not really sure what you are getting at with XOR and one time
> padding, but it has been a while since I have done any encryption.
And I have just started reading "Applied Cryptography", so I am putting
some of what I learn into practice.
>
> I would think you could do all this by just converting everything
> to int and then adding/replacing the pad in the list of ints.
At first I was essentially doing just that, but when I first converted
the large integers that are being used for the one time pad as the key
to binary I saw how much larger it was, and then realized that was the
bit length of the integer (technically Long). By doing that, I can get
more out of the one time pad, but if you XOR binary against Ord, very
few values will be changed because binary is only 1s and 0s as you know.
To optimize the keys use, whether it wastes memory or not, I wanted to
use binary on binary, this really comes into play with files, not so
much the shorter strings.
But since you bring if up, how would you convert a file to a list of ints?
>
>
> Ramit
>
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of
> securities, accuracy and completeness of information, viruses,
> confidentiality, legal privilege, and legal entity disclaimers,
> available at http://www.jpmorgan.com/pages/disclosures/email.  
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 10:04 PM, eryksun wrote:
> On Thu, Jul 19, 2012 at 3:08 PM, Jordan  wrote:

> I'm not an expert with cryptography, but here's a simple XOR example:
 from itertools import cycle
 text = b'Mary had a little lamb.'
 key = b'1234'
 cypher = bytes(x^y for x,y in zip(text, cycle(key)))
 cypher
> b'|SAM\x11ZRP\x11S\x13XXFGXT\x12_U\\P\x1d'
 text2 = bytes(x^y for x,y in zip(cypher, cycle(key)))
 text2
> b'Mary had a little lamb.'
Hmm interesting, I am reading up on on itertools.cycle() and zip now.
Thanks always more than one way to solve a problem.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> > bin(integer).split('b')[1].zfill( multiple_of_eight )
> OK so using this: Hopefully my copy paste works this time.
> 
> bin_data = ''
> 
> for char in data:
> 
> bin_data += bin(ord(char)).split('b')[1].zfill(8)
> 
> print(bin_data)
> 
> bin_list = [bin_data[x:x + 2] for x in range(0, len(bin_data), 2)]
> 
> print(bin_list)
> 
> 
> 
> The paste looks good to me at this time.

Not to me, but I can probably figure out enough based on this.

> How do I get back to the string? If I use this:
> 
> data2 = []
> 
> for item in bin_list:
> 
> data2.append(int(item, 2))
> 
> print(data2)
> 
> 
> 
> The output is all too low of numbers for ord() to convert back to the
> correct string.
> >

Sure, this makes perfect sense to me :) (adding indent)

for char in data:
bin_data += bin(ord(char)).split('b')[1].zfill(8)
bin_list = [bin_data[x:x + 2] for x in range(0, len(bin_data), 2)]

Why are you grabbing 2 binary digits? The only possibilities are 0,1,2,3
and none are ASCII letters. You should be grabbing 8 at a time.

bin_data = [ bin(ord(char)).split('b')[1].zfill(8) for char in data ]
bin_string = ''.join(bin_data)
bin_list = [ chr( int(char, 2) ) for char in bin_data ]

I am not really sure what you are getting at with XOR and one time
padding, but it has been a while since I have done any encryption.

I would think you could do all this by just converting everything
to int and then adding/replacing the pad in the list of ints.


Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 09:53 PM, Dave Angel wrote:
> On 07/19/2012 03:19 PM, Jordan wrote:
>> 
>>
>> OK. I am using one time pads to XOR data, but the one time pads (keys)
>> are very large numbers, converting them to binary increases their size
>> exponentially, which allows me to get more XORing done out of a single
> You want to explain this impossibility of increasing size
> exponentially?  If you're wanting to waste memory, there are better
> ways.  But it's only 8 times as big to save a string of 1's and zeros as
> to save the large-int they represent.  And multiplying by 8 isn't an
> exponential function.
>
Yes if you wish to dissect my words the wrong word was chosen...
>> key. I am XORing both files and strings so I need to have code that can
>> do both even if that means two branches of code via an if/else perhaps
>> with an isinstance(data, str).
>> I do not need to actually see the binary form.
>>
> Then don't use the binary form.  It doesn't make the computation any
> more "powerful" and it'll certainly slow it down.
The title of the question is string to binary and back.
>
> Are you trying to match some other program's algorithm, and thus have
> strange constraints on your data?  Or are you simply trying to make a
> secure way to encrypt binary files, using one-time pads?
I already answered this question...
>
> A one-time pad is the same size as the message, so you simply need to
> convert the message into a large-int, and xor them.
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[eryksun]

On Thu, Jul 19, 2012 at 3:08 PM, Jordan  wrote:
>
> size = 8 * max(len(b3), len(b4))
> format(r, "0%db" % size)
>> '0011001100110011'
> Is this output the output for size rather than the two variables joined
> together?

Using "format" is useful if you need the string to be padded with
zeros for the most significant byte. I wouldn't think it's important
if you're just using the bitstring representation as a sanity check on
your algorithm. In that case you can more easily use "bin".

That said, len(b3) is the number of characters (bytes) in the bytes
object. Since b3 and b4 could be different lengths in general, I took
the max length to use for the zero padding. In this case both b3 and
b4 contain 4 bytes, so "size" is 32.

> OK. I am using one time pads to XOR data, but the one time pads (keys)
> are very large numbers, converting them to binary increases their size
> exponentially, which allows me to get more XORing done out of a single
> key. I am XORing both files and strings so I need to have code that can
> do both even if that means two branches of code via an if/else perhaps
> with an isinstance(data, str).

I'm not an expert with cryptography, but here's a simple XOR example:

>>> from itertools import cycle
>>> text = b'Mary had a little lamb.'
>>> key = b'1234'
>>> cypher = bytes(x^y for x,y in zip(text, cycle(key)))
>>> cypher
b'|SAM\x11ZRP\x11S\x13XXFGXT\x12_U\\P\x1d'
>>> text2 = bytes(x^y for x,y in zip(cypher, cycle(key)))
>>> text2
b'Mary had a little lamb.'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 09:23 PM, Prasad, Ramit wrote:
>> A question I have for the group before I respond is a option that I saw
>> that I had earlier was to ord() each element of a string and then bin()
>> that number. But since bin() produces a string I could not figure out
>> the correct way to attach two bin() outputs back together again due to
>> the leading 'b' and even if I use lstrip('b') I was not sure if that
>> would be correct?
> bin(integer).split('b')[1].zfill( multiple_of_eight )
OK so using this: Hopefully my copy paste works this time.

bin_data = ''

for char in data:

bin_data += bin(ord(char)).split('b')[1].zfill(8)

print(bin_data)

bin_list = [bin_data[x:x + 2] for x in range(0, len(bin_data), 2)]

print(bin_list)



The paste looks good to me at this time.
How do I get back to the string? If I use this:

data2 = []

for item in bin_list:

data2.append(int(item, 2))

print(data2)



The output is all too low of numbers for ord() to convert back to the
correct string.
>
>> My next hesitation is can the same or at least similar techniques be
>> applied to a file? I want to be able to work on both files and strings.
> Probably, but it depends on what you are trying to do and what
> data you are dealing with.
I just want to perform the same conversion on the file data, that is
down to binary and back to it's original state.
I was thinking I would just use the file in binary mode when I open it,
but I am not sure if that is true binary or if it is hex or something
else altogether. I think my confusion came from trying to do both files
and strings at the same time and failing back and forth.
>
> Ramit
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of
> securities, accuracy and completeness of information, viruses,
> confidentiality, legal privilege, and legal entity disclaimers,
> available at http://www.jpmorgan.com/pages/disclosures/email.  
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Dave Angel]

On 07/19/2012 03:19 PM, Jordan wrote:
> 
>
> OK. I am using one time pads to XOR data, but the one time pads (keys)
> are very large numbers, converting them to binary increases their size
> exponentially, which allows me to get more XORing done out of a single

You want to explain this impossibility of increasing size
exponentially?  If you're wanting to waste memory, there are better
ways.  But it's only 8 times as big to save a string of 1's and zeros as
to save the large-int they represent.  And multiplying by 8 isn't an
exponential function.

> key. I am XORing both files and strings so I need to have code that can
> do both even if that means two branches of code via an if/else perhaps
> with an isinstance(data, str).
> I do not need to actually see the binary form.
>

Then don't use the binary form.  It doesn't make the computation any
more "powerful" and it'll certainly slow it down.

Are you trying to match some other program's algorithm, and thus have
strange constraints on your data?  Or are you simply trying to make a
secure way to encrypt binary files, using one-time pads?

A one-time pad is the same size as the message, so you simply need to
convert the message into a large-int, and xor them.


-- 

DaveA

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> A question I have for the group before I respond is a option that I saw
> that I had earlier was to ord() each element of a string and then bin()
> that number. But since bin() produces a string I could not figure out
> the correct way to attach two bin() outputs back together again due to
> the leading 'b' and even if I use lstrip('b') I was not sure if that
> would be correct?

bin(integer).split('b')[1].zfill( multiple_of_eight )

> My next hesitation is can the same or at least similar techniques be
> applied to a file? I want to be able to work on both files and strings.

Probably, but it depends on what you are trying to do and what
data you are dealing with.

Ramit
This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 08:53 PM, Prasad, Ramit wrote:
>>> I think your basic problem is too much conversion because you do not
>>> understand the types. A string is represented by a series of bytes
>>> which are binary numbers. Do you understand the concept behind ASCII?
>>> Each letter has a numeric representation that are sequential. So the
>>> string 'abcd' is equivalent to a series of bytes 65,66,67,68. It is
>>> not equivalent to 65666768 or 65+66+67+68. So your first task is to
>>> convert each character to the numeric equivalent and store them in a
>>> list. Once you have them converted to a list of integers, you can
>>> create another list that is a list of characters.
>> Sorry for the long delay in getting back to you, I got called to the field.
>> Thank you, I agree I do feel like I am doing too much conversion. I do
>> understand the concept behind ASCII at least enough to know about ord()
>> although I did for get about chr() which is ord()'s reverse function. I
>> had tried to break them down to the ordinal value, but I really do want
>> to get the integer and the data down to binary, as it provides an
>> advantage for the overall program that I am writing. Thank you for your
>> time.
> Why not explain your usecase? Technically, everything is binary
> on a computer so the question is why do *you* need to see the
> binary form? Anyway, you can get the binary string by doing
> `bin(ord(character))` and reverse it by doing 
> `chr(int(binary_string,2))`. [1]
OK. I am using one time pads to XOR data, but the one time pads (keys)
are very large numbers, converting them to binary increases their size
exponentially, which allows me to get more XORing done out of a single
key. I am XORing both files and strings so I need to have code that can
do both even if that means two branches of code via an if/else perhaps
with an isinstance(data, str).
I do not need to actually see the binary form.
> If you are doing some kind of XOR (I think your first email mentioned 
> it) then you can XOR integers. Unless you are doing some kind of 
> display of binary output, you usually do not need to actually see 
> the binary string as most binary manipulation can be done via the 
> integer value.
Agreed. Although the visual does help for validation (seeing is believing).
>  [1] - Thanks to Steven.
Yes thank you Steven. I am working on the code now to see if I can make
the above work for me, if I need further help I will be back.
Thank you all again for your time.
>
>
>
>
> Ramit
>
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of
> securities, accuracy and completeness of information, viruses,
> confidentiality, legal privilege, and legal entity disclaimers,
> available at http://www.jpmorgan.com/pages/disclosures/email.  
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

A question I have for the group before I respond is a option that I saw
that I had earlier was to ord() each element of a string and then bin()
that number. But since bin() produces a string I could not figure out
the correct way to attach two bin() outputs back together again due to
the leading 'b' and even if I use lstrip('b') I was not sure if that
would be correct?
My next hesitation is can the same or at least similar techniques be
applied to a file? I want to be able to work on both files and strings.

On 07/19/2012 01:22 PM, eryksun wrote:
> On Thu, Jul 19, 2012 at 1:41 AM, wolfrage8...@gmail.com
>  wrote:
>> I was comparing them but I think I understand how to compare them well, now
>> I want to convert them both to binary so that I can XOR them together. Thank
>> you for your time and help Dave, now I need to reply to Ramit.
> A bytes object is a container of 8-bit numbers (i.e. range 0 to 255).
> If you index it, you'll get an int that supports the XOR operation:
>
 b1 = b'a'
 b2 = b'b'
 b1[0]
> 97
 b2[0]
> 98
 bin(b1[0])
> '0b111'
 bin(b2[0])
> '0b1100010'
 bin(b1[0] ^ b2[0])
> '0b11'
>
> You can use the int method  "from_bytes" to XOR two bitstrings stored
> as Python bytes:
>
 b3 = b''
 b4 = b''
 bin(int.from_bytes(b3, 'big') ^ int.from_bytes(b4, 'big'))
> '0b11001100110011'
>
> The computation is done between int objects, not strings. Creating a
> string using "bin" is just for presentation.
>
> P.S.:
>
> Instead of "bin" you can use the "format" command to have more
> control, such as for zero padding. The integer format code "b" is for
> a binary representation. Preceding it by a number starting with zero
> will pad with zeros to the given number of characters (e.g. 032 will
> prepend zeros to make the result at least 32 characters long):
The control sounds good and I may need that latter (To adjust things to
a fixed length), but for the purpose of XORing a message padding a key
with zeros would not be desirable if Eve was able to get her hands on
the source code.
 r = int.from_bytes(b3, 'big') ^ int.from_bytes(b4, 'big')
 format(r, "032b")
> '0011001100110011'
>
> Instead of hard coding the length (e.g. "032"), you can use the length
> of the input bitstrings to calculate the size of the result:
That sounds good.
 size = 8 * max(len(b3), len(b4))
 format(r, "0%db" % size)
> '0011001100110011'
Is this output the output for size rather than the two variables joined
together?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

My response is down lower, thank you Wayne.

On 07/19/2012 12:52 PM, Wayne Werner wrote:
> I'll preface my response by saying that I know/understand fairly
> little about
> it, but since I've recently been smacked by this same issue when
> converting
> stuff to Python3, I'll see if I can explain it in a way that makes sense.
>
> On Wed, 18 Jul 2012, Jordan wrote:
>
>> OK so I have been trying for a couple days now and I am throwing in the
>> towel, Python 3 wins this one.
>> I want to convert a string to binary and back again like in this
>> question: Stack Overflow: Convert Binary to ASCII and vice versa
>> (Python)
>> 
>>
>> But in Python 3 I consistently get  some sort of error relating to the
>> fact that nothing but bytes and bytearrays support the buffer interface
>> or I get an overflow error because something is too large to be
>> converted to bytes.
>> Please help me and then explian what I am not getting that is new in
>> Python 3. I would like to point out I realize that binary, hex, and
>> encodings are all a very complex subject and so I do not expect to
>> master it but I do hope that I can gain a deeper insight. Thank you all.
>
> The way I've read it - stop thinking about strings as if they are
> text. The
> biggest reason that all this has changed is because Python has grown
> up and
> entered the world where Unicode actually matters. To us poor shmucks
> in the
> English speaking countries of the world it's all very confusing
> becaust it's
> nothing we have to deal with. 26 letters is perfectly fine for us -
> and if we
> want uppercase we'll just throw another 26. Add a few dozen puncuation
> marks
> and 256 is a perfectly fine amount of characters.
>
> To make a slightly relevant side trip, when you were a kid did you
> ever send
> "secret" messages to a friend with a code like this?
>
> A = 1
> B = 2
> .
> .
> .
> Z = 26
>
> Well, that's basically what is going on when it comes to
> bytes/text/whatever.
> When you input some text, Python3 believes that whatever you wrote was
> encoded
> with Unicode. The nice thing for us 26-letter folks is that the ASCII
> alphabet
> we're so used to just so happens to map quite well to Unicode
> encodings - so
> 'A' in ASCII is the same number as 'A' in utf-8.
>
> Now, here's the part that I had to (and still need to) wrap my mind
> around - if
> the string is "just bytes" then it doesn't really matter what the
> string is
> supposed to represent. It could represent the LATIN-1 character set. Or
> UTF-8, -16, or some other weird encoding. And all the operations that are
> supposed to modify these strings of bytes (e.g. removing spaces,
> splitting on a
> certain "character", etc.) still work. Because if I have this string:
>
> 9 45 12 9 13 19 18 9 12 99 102
>
> and I tell you to split on the 9's, it doesn't matter if that's some
> weird
> ASCII character, or some equally weird UTF character, or something else
> entirely. And I don't have to worry about things getting munged up
> when I try
> to stick Unicode and ASCII values together - because they're converted
> to bytes
> first.
>
> So the question is, of course, if it's all bytes, then why does it
> look like
> text when I print it out? Well, that's because Python converts that
> byte stream
> to Unicode text when it's printed. Or ASCII, if you tell it to.
>
> But Python3 has converted all(?) of those functions that used to
> operate on
> text and made them operate on byte streams instead. Except for the
> ones that
> operate on text ;)
>
>
>
> Well, I hope that's of some use and isn't too much of a lie - like I
> said, I'm
> still trying to wrap my head around things and I've found that
> explaining (or
> trying to explain) to someone else is often the best way to work out
> the idea
> in your own head. If I've gone too far astray I'm sure the other
> helpful folks
> here will correct me :)
>
Thank you for the vary informative post, every bit helps. It has
certainly been a challenge for me with the new everything is bytes
scheme, especially how everything has to be converted to bytes prior to
going on a buffer.
> HTH,
> Wayne
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> > I think your basic problem is too much conversion because you do not
> > understand the types. A string is represented by a series of bytes
> > which are binary numbers. Do you understand the concept behind ASCII?
> > Each letter has a numeric representation that are sequential. So the
> > string 'abcd' is equivalent to a series of bytes 65,66,67,68. It is
> > not equivalent to 65666768 or 65+66+67+68. So your first task is to
> > convert each character to the numeric equivalent and store them in a
> > list. Once you have them converted to a list of integers, you can
> > create another list that is a list of characters.

> Sorry for the long delay in getting back to you, I got called to the field.
> Thank you, I agree I do feel like I am doing too much conversion. I do
> understand the concept behind ASCII at least enough to know about ord()
> although I did for get about chr() which is ord()'s reverse function. I
> had tried to break them down to the ordinal value, but I really do want
> to get the integer and the data down to binary, as it provides an
> advantage for the overall program that I am writing. Thank you for your
> time.

Why not explain your usecase? Technically, everything is binary
on a computer so the question is why do *you* need to see the
binary form? Anyway, you can get the binary string by doing
`bin(ord(character))` and reverse it by doing 
`chr(int(binary_string,2))`. [1]

If you are doing some kind of XOR (I think your first email mentioned 
it) then you can XOR integers. Unless you are doing some kind of 
display of binary output, you usually do not need to actually see 
the binary string as most binary manipulation can be done via the 
integer value.

 [1] - Thanks to Steven.




Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 12:46 PM, Dave Angel wrote:
> On 07/19/2012 01:41 AM, wolfrage8...@gmail.com wrote:
>> On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:

> That was just the first line that was not indented.  If I thought you
> had a one-line while loop, I certainly would have just indented it.  But
> I'm sure you have some unknown number of additional lines that were
> indented in your original.  Please post in text form.
I see now, I am sorry I did not know that Thunderbird had eliminated all
of my indentation. I had set it to plain text, but I guess it is not to
be trusted. I simply looked at the reply email and saw the line that you
pointed out and at that time figured Thunderbird had just word wrapped
that line on me. Would it be acceptable to add an attached Python file?

> Lots of details that have nothing to do with it.  For example, that
> whole thing about adding random digits together.  You could replace the
> whole thing with a simple assignment of a value that doesn't work for you.
OK I will, that was a test script and I was testing multiple things, I
did try to get rid of most of the cruft but I will attempt to do better
in the future. 
>> now I want to convert them both to binary so that I can XOR them together.
>> Thank you for your time and help Dave, now I need to reply to Ramit.
> Ah, so you don't actually want binary at all!!!   Why not state the real
> problem up front?  You can XOR two integers, without bothering to
> convert to a string of ones and zeroes.  Use the carat operator.
>
> print( 40 ^ 12)
>
> I suspect there's an equivalent for strings or byte-strings.  But if
> not, it's a simple loop.
Actually I do want binary, as it serves as an advantage for the overall
program that I am building.
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 08:14 AM, Mark Lawrence wrote:
> On 19/07/2012 06:41, wolfrage8...@gmail.com wrote:
>> On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:
>>
>

> Really?  Are you using a forked version of Python that doesn't need
> indentation after a while loop, or are you speaking with a forked
> tongue? :)  Strangely I believe the latter, so please take note of
> what Dave Angel has told you and post with the correct indentation.
>
http://www.101emailetiquettetips.com/
Number 101 is for you. Good day.
>>
>>>
>>> I'd also recommend you remove a lot of the irrelevant details
>>> there.  if
>>> you have a problem with hexlfy and/or unhexlify, then give a simple
>>> byte
>>> string that doesn't work for you, and somebody can probably identify
>>> why
>>> not.  And if you want people to run your code, include the imports
>>> as well.
>>>
>>> My problem is not specific to hexlify and unhexlify, my problem is
>>> trying
>> to convert from string to binary and back. That is why all of the
>> details,
>> to show I have tried on my own.
>> Sorry that I forgot to include sys and os for imports.
>>
>>
>>> As it is, you're apparently looping, comparing the byte memory size
>>> of a
>>> string (which is typically 4 bytes per character) with the number of
>>> significant bits in an unrelated number.
>>>
>>> I suspect what you want is something resembling (untested):
>>>
>>>  mybytes = bytes( "%x" % data, "ascii")
>>>  newdata = binascii.unexlify(mybytes)
>>>
>>> I was comparing them but I think I understand how to compare them well,
>> now I want to convert them both to binary so that I can XOR them
>> together.
>> Thank you for your time and help Dave, now I need to reply to Ramit.
>>
>>>
>>> -- 
>>> DaveA
>>>
>>
>>
>>
>> ___
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/mailman/listinfo/tutor
>>
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Jordan]

On 07/19/2012 12:15 AM, Prasad, Ramit wrote:

> I think your basic problem is too much conversion because you do not
> understand the types. A string is represented by a series of bytes
> which are binary numbers. Do you understand the concept behind ASCII?
> Each letter has a numeric representation that are sequential. So the
> string 'abcd' is equivalent to a series of bytes 65,66,67,68. It is
> not equivalent to 65666768 or 65+66+67+68. So your first task is to
> convert each character to the numeric equivalent and store them in a
> list. Once you have them converted to a list of integers, you can
> create another list that is a list of characters.
Sorry for the long delay in getting back to you, I got called to the field.
Thank you, I agree I do feel like I am doing too much conversion. I do
understand the concept behind ASCII at least enough to know about ord()
although I did for get about chr() which is ord()'s reverse function. I
had tried to break them down to the ordinal value, but I really do want
to get the integer and the data down to binary, as it provides an
advantage for the overall program that I am writing. Thank you for your
time.
>
> Look at the functions chr and ord here
> ( http://docs.python.org/py3k/library/functions.html )
>
> Ramit
>
>
> Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
> 712 Main Street | Houston, TX 77002
> work phone: 713 - 216 - 5423
>
> --
>
> This email is confidential and subject to important disclaimers and
> conditions including on offers for the purchase or sale of
> securities, accuracy and completeness of information, viruses,
> confidentiality, legal privilege, and legal entity disclaimers,
> available at http://www.jpmorgan.com/pages/disclosures/email.  
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[eryksun]

On Thu, Jul 19, 2012 at 1:41 AM, wolfrage8...@gmail.com
 wrote:
>
> I was comparing them but I think I understand how to compare them well, now
> I want to convert them both to binary so that I can XOR them together. Thank
> you for your time and help Dave, now I need to reply to Ramit.

A bytes object is a container of 8-bit numbers (i.e. range 0 to 255).
If you index it, you'll get an int that supports the XOR operation:

>>> b1 = b'a'
>>> b2 = b'b'
>>> b1[0]
97
>>> b2[0]
98
>>> bin(b1[0])
'0b111'
>>> bin(b2[0])
'0b1100010'
>>> bin(b1[0] ^ b2[0])
'0b11'

You can use the int method  "from_bytes" to XOR two bitstrings stored
as Python bytes:

>>> b3 = b''
>>> b4 = b''
>>> bin(int.from_bytes(b3, 'big') ^ int.from_bytes(b4, 'big'))
'0b11001100110011'

The computation is done between int objects, not strings. Creating a
string using "bin" is just for presentation.

P.S.:

Instead of "bin" you can use the "format" command to have more
control, such as for zero padding. The integer format code "b" is for
a binary representation. Preceding it by a number starting with zero
will pad with zeros to the given number of characters (e.g. 032 will
prepend zeros to make the result at least 32 characters long):

>>> r = int.from_bytes(b3, 'big') ^ int.from_bytes(b4, 'big')
>>> format(r, "032b")
'0011001100110011'

Instead of hard coding the length (e.g. "032"), you can use the length
of the input bitstrings to calculate the size of the result:

>>> size = 8 * max(len(b3), len(b4))
>>> format(r, "0%db" % size)
'0011001100110011'
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Wayne Werner]

I'll preface my response by saying that I know/understand fairly little about
it, but since I've recently been smacked by this same issue when converting
stuff to Python3, I'll see if I can explain it in a way that makes sense.

On Wed, 18 Jul 2012, Jordan wrote:


OK so I have been trying for a couple days now and I am throwing in the
towel, Python 3 wins this one.
I want to convert a string to binary and back again like in this
question: Stack Overflow: Convert Binary to ASCII and vice versa
(Python)

But in Python 3 I consistently get  some sort of error relating to the
fact that nothing but bytes and bytearrays support the buffer interface
or I get an overflow error because something is too large to be
converted to bytes.
Please help me and then explian what I am not getting that is new in
Python 3. I would like to point out I realize that binary, hex, and
encodings are all a very complex subject and so I do not expect to
master it but I do hope that I can gain a deeper insight. Thank you all.


The way I've read it - stop thinking about strings as if they are text. The
biggest reason that all this has changed is because Python has grown up and
entered the world where Unicode actually matters. To us poor shmucks in the
English speaking countries of the world it's all very confusing becaust it's
nothing we have to deal with. 26 letters is perfectly fine for us - and if we
want uppercase we'll just throw another 26. Add a few dozen puncuation marks
and 256 is a perfectly fine amount of characters.

To make a slightly relevant side trip, when you were a kid did you ever send
"secret" messages to a friend with a code like this?

A = 1
B = 2
.
.
.
Z = 26

Well, that's basically what is going on when it comes to bytes/text/whatever.
When you input some text, Python3 believes that whatever you wrote was encoded
with Unicode. The nice thing for us 26-letter folks is that the ASCII alphabet
we're so used to just so happens to map quite well to Unicode encodings - so
'A' in ASCII is the same number as 'A' in utf-8.

Now, here's the part that I had to (and still need to) wrap my mind around - if
the string is "just bytes" then it doesn't really matter what the string is
supposed to represent. It could represent the LATIN-1 character set. Or
UTF-8, -16, or some other weird encoding. And all the operations that are
supposed to modify these strings of bytes (e.g. removing spaces, splitting on a
certain "character", etc.) still work. Because if I have this string:

9 45 12 9 13 19 18 9 12 99 102

and I tell you to split on the 9's, it doesn't matter if that's some weird
ASCII character, or some equally weird UTF character, or something else
entirely. And I don't have to worry about things getting munged up when I try
to stick Unicode and ASCII values together - because they're converted to bytes
first.

So the question is, of course, if it's all bytes, then why does it look like
text when I print it out? Well, that's because Python converts that byte stream
to Unicode text when it's printed. Or ASCII, if you tell it to.

But Python3 has converted all(?) of those functions that used to operate on
text and made them operate on byte streams instead. Except for the ones that
operate on text ;)



Well, I hope that's of some use and isn't too much of a lie - like I said, I'm
still trying to wrap my head around things and I've found that explaining (or
trying to explain) to someone else is often the best way to work out the idea
in your own head. If I've gone too far astray I'm sure the other helpful folks
here will correct me :)

HTH,
Wayne
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Dave Angel]

On 07/19/2012 01:41 AM, wolfrage8...@gmail.com wrote:
> On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:
>
>> 
>> I don't get the same error you did.  I get:
>>
>>  File "jordan.py", line 13
>> test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
>> 'big')))
>>^
>>
> test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), \
> 'big')))
> # That was probably just do to the copy and paste.

That was just the first line that was not indented.  If I thought you
had a one-line while loop, I certainly would have just indented it.  But
I'm sure you have some unknown number of additional lines that were
indented in your original.  Please post in text form.

> 
>> I'd also recommend you remove a lot of the irrelevant details there.  if
>> you have a problem with hexlfy and/or unhexlify, then give a simple byte
>> string that doesn't work for you, and somebody can probably identify why
>> not.  And if you want people to run your code, include the imports as well.
>>
>> My problem is not specific to hexlify and unhexlify, my problem is trying
> to convert from string to binary and back. That is why all of the details,
> to show I have tried on my own.
> Sorry that I forgot to include sys and os for imports.

Lots of details that have nothing to do with it.  For example, that
whole thing about adding random digits together.  You could replace the
whole thing with a simple assignment of a value that doesn't work for you.

>
>> As it is, you're apparently looping, comparing the byte memory size of a
>> string (which is typically 4 bytes per character) with the number of
>> significant bits in an unrelated number.
>>
>> I suspect what you want is something resembling (untested):
>>
>> mybytes = bytes( "%x" % data, "ascii")
>> newdata = binascii.unexlify(mybytes)
>>
>> I was comparing them but I think I understand how to compare them well,
> now I want to convert them both to binary so that I can XOR them together.
> Thank you for your time and help Dave, now I need to reply to Ramit.

Ah, so you don't actually want binary at all!!!   Why not state the real
problem up front?  You can XOR two integers, without bothering to
convert to a string of ones and zeroes.  Use the carat operator.

print( 40 ^ 12)

I suspect there's an equivalent for strings or byte-strings.  But if
not, it's a simple loop.



-- 

DaveA

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Mark Lawrence]

On 19/07/2012 06:41, wolfrage8...@gmail.com wrote:

On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:


  On 07/18/2012 05:07 PM, Jordan wrote:

OK so I have been trying for a couple days now and I am throwing in the
towel, Python 3 wins this one.
I want to convert a string to binary and back again like in this
question: Stack Overflow: Convert Binary to ASCII and vice versa
(Python)
<

http://stackoverflow.com/questions/7396849/convert-binary-to-ascii-and-vice-versa-python


But in Python 3 I consistently get  some sort of error relating to the
fact that nothing but bytes and bytearrays support the buffer interface
or I get an overflow error because something is too large to be
converted to bytes.
Please help me and then explian what I am not getting that is new in
Python 3. I would like to point out I realize that binary, hex, and
encodings are all a very complex subject and so I do not expect to
master it but I do hope that I can gain a deeper insight. Thank you all.

test_script.py:
import binascii

test_int = 109

test_int = int(str(test_int) + '45670')
data = 'Testing XOR Again!'

while sys.getsizeof(data) > test_int.bit_length():

test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))

print('Bit Length: ' + str(test_int.bit_length()))

key = test_int # Yes I know this is an unnecessary step...

data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))

print(data)

data = int(data, 2)

print(data)

data = binascii.unhexlify('%x' % data)



I don't get the same error you did.  I get:

  File "jordan.py", line 13
 test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
'big')))
^


test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), \
 'big')))
# That was probably just do to the copy and paste.


IndentationError: expected an indented block


Please post it again, with correct indentation.  if you used tabs, then
expand them to spaces before pasting it into your test-mode mail editor.

I only use spaces and this program did not require any indentation until

it was pasted and the one line above became split across two line. Really
though that was a trivial error to correct.


Really?  Are you using a forked version of Python that doesn't need 
indentation after a while loop, or are you speaking with a forked 
tongue? :)  Strangely I believe the latter, so please take note of what 
Dave Angel has told you and post with the correct indentation.






I'd also recommend you remove a lot of the irrelevant details there.  if
you have a problem with hexlfy and/or unhexlify, then give a simple byte
string that doesn't work for you, and somebody can probably identify why
not.  And if you want people to run your code, include the imports as well.

My problem is not specific to hexlify and unhexlify, my problem is trying

to convert from string to binary and back. That is why all of the details,
to show I have tried on my own.
Sorry that I forgot to include sys and os for imports.



As it is, you're apparently looping, comparing the byte memory size of a
string (which is typically 4 bytes per character) with the number of
significant bits in an unrelated number.

I suspect what you want is something resembling (untested):

 mybytes = bytes( "%x" % data, "ascii")
 newdata = binascii.unexlify(mybytes)

I was comparing them but I think I understand how to compare them well,

now I want to convert them both to binary so that I can XOR them together.
Thank you for your time and help Dave, now I need to reply to Ramit.



--
DaveA





___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor




--
Cheers.

Mark Lawrence.



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[wolfrage8...@gmail.com]

On Thu, Jul 19, 2012 at 12:16 AM, Dave Angel  wrote:

>  On 07/18/2012 05:07 PM, Jordan wrote:
> > OK so I have been trying for a couple days now and I am throwing in the
> > towel, Python 3 wins this one.
> > I want to convert a string to binary and back again like in this
> > question: Stack Overflow: Convert Binary to ASCII and vice versa
> > (Python)
> > <
> http://stackoverflow.com/questions/7396849/convert-binary-to-ascii-and-vice-versa-python
> >
> > But in Python 3 I consistently get  some sort of error relating to the
> > fact that nothing but bytes and bytearrays support the buffer interface
> > or I get an overflow error because something is too large to be
> > converted to bytes.
> > Please help me and then explian what I am not getting that is new in
> > Python 3. I would like to point out I realize that binary, hex, and
> > encodings are all a very complex subject and so I do not expect to
> > master it but I do hope that I can gain a deeper insight. Thank you all.
> >
> > test_script.py:
> > import binascii
> >
> > test_int = 109
> >
> > test_int = int(str(test_int) + '45670')
> > data = 'Testing XOR Again!'
> >
> > while sys.getsizeof(data) > test_int.bit_length():
> >
> > test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))
> >
> > print('Bit Length: ' + str(test_int.bit_length()))
> >
> > key = test_int # Yes I know this is an unnecessary step...
> >
> > data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
> >
> > print(data)
> >
> > data = int(data, 2)
> >
> > print(data)
> >
> > data = binascii.unhexlify('%x' % data)
> >
>
> I don't get the same error you did.  I get:
>
>  File "jordan.py", line 13
> test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
> 'big')))
>^
>
test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), \
'big')))
# That was probably just do to the copy and paste.

> IndentationError: expected an indented block
>
>
> Please post it again, with correct indentation.  if you used tabs, then
> expand them to spaces before pasting it into your test-mode mail editor.
>
> I only use spaces and this program did not require any indentation until
it was pasted and the one line above became split across two line. Really
though that was a trivial error to correct.

>
> I'd also recommend you remove a lot of the irrelevant details there.  if
> you have a problem with hexlfy and/or unhexlify, then give a simple byte
> string that doesn't work for you, and somebody can probably identify why
> not.  And if you want people to run your code, include the imports as well.
>
> My problem is not specific to hexlify and unhexlify, my problem is trying
to convert from string to binary and back. That is why all of the details,
to show I have tried on my own.
Sorry that I forgot to include sys and os for imports.


> As it is, you're apparently looping, comparing the byte memory size of a
> string (which is typically 4 bytes per character) with the number of
> significant bits in an unrelated number.
>
> I suspect what you want is something resembling (untested):
>
> mybytes = bytes( "%x" % data, "ascii")
> newdata = binascii.unexlify(mybytes)
>
> I was comparing them but I think I understand how to compare them well,
now I want to convert them both to binary so that I can XOR them together.
Thank you for your time and help Dave, now I need to reply to Ramit.

>
> --
> DaveA
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Steven D'Aprano]

On Wed, Jul 18, 2012 at 10:22:43PM +, Prasad, Ramit wrote:

> I forgot to say, that once you have the integer equivalents, 
> you can then convert that easily to binary using bin.
> I used ast.literal_eval to convert from binary string 
> (as returned from bin) to number, but there might be better
> ways.

Do you mean something like this?


py> int('111001010100100', 2)
29348


-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> 
> I think your basic problem is too much conversion because you do not
> understand the types. A string is represented by a series of bytes
> which are binary numbers. Do you understand the concept behind ASCII?
> Each letter has a numeric representation that are sequential.
> So the string 'abcd' is equivalent to a series of bytes 65,66,67,68.
> It is not equivalent to 65666768 or 65+66+67+68. So your first
> task is to convert each character to the numeric equivalent and store
> them in a list. Once you have them converted to a list of integers,
> you can create another list that is a list of characters.
> 
> Look at the functions chr and ord here
> ( http://docs.python.org/py3k/library/functions.html )

I forgot to say, that once you have the integer equivalents, 
you can then convert that easily to binary using bin.
I used ast.literal_eval to convert from binary string 
(as returned from bin) to number, but there might be better
ways.

 
Ramit

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Dave Angel]

On 07/18/2012 05:07 PM, Jordan wrote:
> OK so I have been trying for a couple days now and I am throwing in the
> towel, Python 3 wins this one.
> I want to convert a string to binary and back again like in this
> question: Stack Overflow: Convert Binary to ASCII and vice versa
> (Python)
> 
> But in Python 3 I consistently get  some sort of error relating to the
> fact that nothing but bytes and bytearrays support the buffer interface
> or I get an overflow error because something is too large to be
> converted to bytes.
> Please help me and then explian what I am not getting that is new in
> Python 3. I would like to point out I realize that binary, hex, and
> encodings are all a very complex subject and so I do not expect to
> master it but I do hope that I can gain a deeper insight. Thank you all.
>
> test_script.py:
> import binascii
>
> test_int = 109
>
> test_int = int(str(test_int) + '45670')
> data = 'Testing XOR Again!'
>
> while sys.getsizeof(data) > test_int.bit_length():
>
> test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))
>
> print('Bit Length: ' + str(test_int.bit_length()))
>
> key = test_int # Yes I know this is an unnecessary step...
>
> data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
>
> print(data)
>
> data = int(data, 2)
>
> print(data)
>
> data = binascii.unhexlify('%x' % data)
>

I don't get the same error you did.  I get:

 File "jordan.py", line 13
test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1),
'big')))
   ^
IndentationError: expected an indented block


Please post it again, with correct indentation.  if you used tabs, then
expand them to spaces before pasting it into your test-mode mail editor.


I'd also recommend you remove a lot of the irrelevant details there.  if
you have a problem with hexlfy and/or unhexlify, then give a simple byte
string that doesn't work for you, and somebody can probably identify why
not.  And if you want people to run your code, include the imports as well.

As it is, you're apparently looping, comparing the byte memory size of a
string (which is typically 4 bytes per character) with the number of
significant bits in an unrelated number.

I suspect what you want is something resembling (untested):

mybytes = bytes( "%x" % data, "ascii")
newdata = binascii.unexlify(mybytes)


-- 
DaveA
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] string to binary and back... Python 3

[Prasad, Ramit]

> OK so I have been trying for a couple days now and I am throwing in the
> towel, Python 3 wins this one.
> I want to convert a string to binary and back again like in this
> question: Stack Overflow: Convert Binary to ASCII and vice versa
> (Python)
>  versa-python>
> But in Python 3 I consistently get  some sort of error relating to the
> fact that nothing but bytes and bytearrays support the buffer interface
> or I get an overflow error because something is too large to be
> converted to bytes.
> Please help me and then explian what I am not getting that is new in
> Python 3. I would like to point out I realize that binary, hex, and
> encodings are all a very complex subject and so I do not expect to
> master it but I do hope that I can gain a deeper insight. Thank you all.

[ snip program ]

> data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
> data = binascii.unhexlify('%x' % data)

[ more snipping ]

> wolfrage@lm12-laptop02 ~/Projects $ python3 test_script.py
> Bit Length: 134
> 0b1010100011001010111001101110100011010010110111001100111001001011100010100100010010101100111011101101001011011100011
> 7351954002991226380810260999848996570230305
> Traceback (most recent call last):
> File "test_script.py", line 24, in 
> data = binascii.unhexlify('%x' % data)
> TypeError: 'str' does not support the buffer interface

I think your basic problem is too much conversion because you do not
understand the types. A string is represented by a series of bytes
which are binary numbers. Do you understand the concept behind ASCII?
Each letter has a numeric representation that are sequential.
So the string 'abcd' is equivalent to a series of bytes 65,66,67,68.
It is not equivalent to 65666768 or 65+66+67+68. So your first
task is to convert each character to the numeric equivalent and store
them in a list. Once you have them converted to a list of integers,
you can create another list that is a list of characters.

Look at the functions chr and ord here
( http://docs.python.org/py3k/library/functions.html )

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] string to binary and back... Python 3

[Jordan]

OK so I have been trying for a couple days now and I am throwing in the
towel, Python 3 wins this one.
I want to convert a string to binary and back again like in this
question: Stack Overflow: Convert Binary to ASCII and vice versa
(Python)

But in Python 3 I consistently get  some sort of error relating to the
fact that nothing but bytes and bytearrays support the buffer interface
or I get an overflow error because something is too large to be
converted to bytes.
Please help me and then explian what I am not getting that is new in
Python 3. I would like to point out I realize that binary, hex, and
encodings are all a very complex subject and so I do not expect to
master it but I do hope that I can gain a deeper insight. Thank you all.

test_script.py:
import binascii

test_int = 109

test_int = int(str(test_int) + '45670')
data = 'Testing XOR Again!'

while sys.getsizeof(data) > test_int.bit_length():

test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))

print('Bit Length: ' + str(test_int.bit_length()))

key = test_int # Yes I know this is an unnecessary step...

data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))

print(data)

data = int(data, 2)

print(data)

data = binascii.unhexlify('%x' % data)


wolfrage@lm12-laptop02 ~/Projects $ python3 test_script.py
Bit Length: 134
0b1010100011001010111001101110100011010010110111001100111001001011100010100100010010101100111011101101001011011100011
7351954002991226380810260999848996570230305
Traceback (most recent call last):
File "test_script.py", line 24, in 
data = binascii.unhexlify('%x' % data)
TypeError: 'str' does not support the buffer interface



test_script2.py:
import binascii
test_int = 109
test_int = int(str(test_int) + '45670')
data = 'Testing XOR Again!'
while sys.getsizeof(data) > test_int.bit_length():
test_int = int(str(test_int) + str(int.from_bytes(os.urandom(1), 'big')))
print('Bit Length: ' + str(test_int.bit_length()))
key = test_int # Yes I know this is an unnecessary step...
data = bin(int(binascii.hexlify(bytes(data, 'UTF-8')), 16))
print(data)
data = int(data, 2)
print(data)
data = binascii.unhexlify(bytes(data, 'utf8'))



wolfrage@lm12-laptop02 ~/Projects $ python3 test_script2.py
Bit Length: 140
0b1010100011001010111001101110100011010010110111001100111001001011100010100100010010101100111011101101001011011100011
7351954002991226380810260999848996570230305
Traceback (most recent call last):
File "test_script.py", line 24, in 
data = binascii.unhexlify(bytes(data, 'utf8'))
OverflowError: cannot fit 'int' into an index-sized integer

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor