Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Bert JW Regeer]

That is because response.body_file is 
https://github.com/Pylons/webob/blob/99890b763dbd441ca1e5f14f8418efa2a8173ab0/src/webob/response.py#L1445
 

 which calls 
https://github.com/Pylons/webob/blob/99890b763dbd441ca1e5f14f8418efa2a8173ab0/src/webob/response.py#L671
 

 on write.

It does the conversion for you...

That's not the same as setting body_file directly on the response, which does 
NOT go through that same mechanism and instead assumes that the file is a file 
in binary mode.

> On Oct 19, 2020, at 10:23, Mike Orr  wrote:
> 
> On Sun, Oct 18, 2020 at 1:22 PM Bert JW Regeer  wrote:
>> 
>> Your body_file is not bytes, but str. You need to make sure that what you 
>> pass to body_file returns bytes.
> 
> I thought that but 'writer = csv.writer(response.body_file)' works for
> me, and my row items are 'str'. In Python 2 I had to convert unicode
> to boytes or I'd get an error, but Python 3 accepts 'str' values
> directly and the resulting file download is correct.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DurwVOtKjYweOyzG7YLA%2BQNb_hVG5zrrA2cjRoG%2BQmtB8Q%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/6BCDEE32-6614-4EA9-B0F2-7E176E54C81D%400x58.com.

Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Mike Orr]

On Sun, Oct 18, 2020 at 1:22 PM Bert JW Regeer  wrote:
>
> Your body_file is not bytes, but str. You need to make sure that what you 
> pass to body_file returns bytes.

I thought that but 'writer = csv.writer(response.body_file)' works for
me, and my row items are 'str'. In Python 2 I had to convert unicode
to boytes or I'd get an error, but Python 3 accepts 'str' values
directly and the resulting file download is correct.

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3DurwVOtKjYweOyzG7YLA%2BQNb_hVG5zrrA2cjRoG%2BQmtB8Q%40mail.gmail.com.

Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Jonathan Vanasco]

Thanks, everyone!

Now I've got both these debugtoolbar plugins updated PyPi and running on 
Github Actions with expanded test-suites:

* https://github.com/jvanasco/pyramid_debugtoolbar_api_performance
* https://github.com/jvanasco/pyramid_debugtoolbar_api_sqlalchemy


On Monday, October 19, 2020 at 11:59:05 AM UTC-4 Jonathan Vanasco wrote:

> Actually,  just using `body=csvfile.read()` works on Python2 & Python3 !
>
>
> On Monday, October 19, 2020 at 11:32:48 AM UTC-4 the...@luhn.com wrote:
>
>> Since you’re just dumping the entire file into BytesIO anyways, wouldn’t 
>> it be easier to do `body=csvfile.read().encode(“utf8”)` and skip BytesIO?
>>
>> On Oct 18, 2020, at 3:33 PM, Jonathan Vanasco  wrote:
>>
>> Thanks so much, Bert. That should have been obvious!
>>
>> As an interim solution, I'm just wrapping the inputs:
>>
>> if six.PY3:
>> csvfile = BytesIO(csvfile.read().encode("utf-8"))
>> csvfile.seek(0)
>>
>>
>>
>> On Sunday, October 18, 2020 at 4:22:16 PM UTC-4 Bert JW Regeer wrote:
>>
>>> Your body_file is not bytes, but str. You need to make sure that what 
>>> you pass to body_file returns bytes.
>>>
>>> On Oct 16, 2020, at 15:03, Jonathan Vanasco  wrote:
>>>
>>> I discovered an issue with our code when generating dynamic CSVs under 
>>> Python3.  I'm hoping someone can point me in the right direction.  I 
>>> couldn't find the appropriate migration/changelog information.  This works 
>>> fine under Python2.
>>>
>>> The generic way we create/serve the CSV and check it in tests are below.
>>>
>>> The two problems:
>>>
>>> 1. Under Python3, an exception is thrown under waitress, because of 
>>> attempted string+byte concatenation. (
>>> https://github.com/Pylons/waitress/blob/master/src/waitress/task.py#L316
>>> )
>>>
>>> >towrite += data + b"\r\n"
>>>
>>> > TypeError: can only concatenate str (not "bytes") to str
>>>
>>> 2. I picked this up in a unit test; i can't seem to access any sort of 
>>> body/text/content off the response.
>>>
>>> from pyramid.request import Request
>>> from pyramid import testing
>>>
>>> self.config = config = testing.setUp()
>>> app = self.config.make_wsgi_app()
>>>
>>> req = Request.blank(csv_link)
>>> req.remote_addr = "127.0.0.1"
>>> resp = req.get_response(app)
>>> self.assertEqual(resp.status_code, 200)
>>> self.assertTrue(resp.text.startswith("User CPU time,"))
>>>
>>>
>>>
>>> ---
>>>
>>>  def view_csv(request):
>>> csvfile = StringIO()
>>> csvwriter = csv.writer(csvfile)
>>> for row in ((1, "a"), (2, "b")):
>>> csvwriter.writerow(row)
>>> csvfile.seek(0)
>>> as_csv = Response(content_type="text/csv", body_file=csvfile, 
>>> status=200)
>>> as_csv.headers["Content-Disposition"] = str("attachment; 
>>> filename=example.csv")
>>> return as_csv
>>>
>>> ---
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "pylons-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to pylons-discus...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/pylons-discuss/7d3c278f-8d20-4456-b05c-33c8d2cb6a67n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>>
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discus...@googlegroups.com.
>>
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/e73aae83-0f7e-414c-a9c3-c20957793a0dn%40googlegroups.com
>>  
>> 
>> .
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/19714c15-c0db-44bf-b9d3-653d814725d1n%40googlegroups.com.

Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Jonathan Vanasco]

Actually,  just using `body=csvfile.read()` works on Python2 & Python3 !


On Monday, October 19, 2020 at 11:32:48 AM UTC-4 the...@luhn.com wrote:

> Since you’re just dumping the entire file into BytesIO anyways, wouldn’t 
> it be easier to do `body=csvfile.read().encode(“utf8”)` and skip BytesIO?
>
> On Oct 18, 2020, at 3:33 PM, Jonathan Vanasco  wrote:
>
> Thanks so much, Bert. That should have been obvious!
>
> As an interim solution, I'm just wrapping the inputs:
>
> if six.PY3:
> csvfile = BytesIO(csvfile.read().encode("utf-8"))
> csvfile.seek(0)
>
>
>
> On Sunday, October 18, 2020 at 4:22:16 PM UTC-4 Bert JW Regeer wrote:
>
>> Your body_file is not bytes, but str. You need to make sure that what you 
>> pass to body_file returns bytes.
>>
>> On Oct 16, 2020, at 15:03, Jonathan Vanasco  wrote:
>>
>> I discovered an issue with our code when generating dynamic CSVs under 
>> Python3.  I'm hoping someone can point me in the right direction.  I 
>> couldn't find the appropriate migration/changelog information.  This works 
>> fine under Python2.
>>
>> The generic way we create/serve the CSV and check it in tests are below.
>>
>> The two problems:
>>
>> 1. Under Python3, an exception is thrown under waitress, because of 
>> attempted string+byte concatenation. (
>> https://github.com/Pylons/waitress/blob/master/src/waitress/task.py#L316)
>>
>> >towrite += data + b"\r\n"
>>
>> > TypeError: can only concatenate str (not "bytes") to str
>>
>> 2. I picked this up in a unit test; i can't seem to access any sort of 
>> body/text/content off the response.
>>
>> from pyramid.request import Request
>> from pyramid import testing
>>
>> self.config = config = testing.setUp()
>> app = self.config.make_wsgi_app()
>>
>> req = Request.blank(csv_link)
>> req.remote_addr = "127.0.0.1"
>> resp = req.get_response(app)
>> self.assertEqual(resp.status_code, 200)
>> self.assertTrue(resp.text.startswith("User CPU time,"))
>>
>>
>>
>> ---
>>
>>  def view_csv(request):
>> csvfile = StringIO()
>> csvwriter = csv.writer(csvfile)
>> for row in ((1, "a"), (2, "b")):
>> csvwriter.writerow(row)
>> csvfile.seek(0)
>> as_csv = Response(content_type="text/csv", body_file=csvfile, 
>> status=200)
>> as_csv.headers["Content-Disposition"] = str("attachment; 
>> filename=example.csv")
>> return as_csv
>>
>> ---
>>
>>
>>
>>
>>
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discus...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/7d3c278f-8d20-4456-b05c-33c8d2cb6a67n%40googlegroups.com
>>  
>> 
>> .
>>
>>
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discus...@googlegroups.com.
>
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/e73aae83-0f7e-414c-a9c3-c20957793a0dn%40googlegroups.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/ce9e5409-eaac-4014-a98f-df5d1b808734n%40googlegroups.com.

Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Theron Luhn]

Since you’re just dumping the entire file into BytesIO anyways, wouldn’t it be 
easier to do `body=csvfile.read().encode(“utf8”)` and skip BytesIO?

> On Oct 18, 2020, at 3:33 PM, Jonathan Vanasco  wrote:
> 
> Thanks so much, Bert. That should have been obvious!
> 
> As an interim solution, I'm just wrapping the inputs:
> 
> if six.PY3:
> csvfile = BytesIO(csvfile.read().encode("utf-8"))
> csvfile.seek(0)
> 
> 
> 
> On Sunday, October 18, 2020 at 4:22:16 PM UTC-4 Bert JW Regeer wrote:
> Your body_file is not bytes, but str. You need to make sure that what you 
> pass to body_file returns bytes.
> 
> 
>> On Oct 16, 2020, at 15:03, Jonathan Vanasco > > wrote:
>> 
> 
>> I discovered an issue with our code when generating dynamic CSVs under 
>> Python3.  I'm hoping someone can point me in the right direction.  I 
>> couldn't find the appropriate migration/changelog information.  This works 
>> fine under Python2.
>> 
>> The generic way we create/serve the CSV and check it in tests are below.
>> 
>> The two problems:
>> 
>> 1. Under Python3, an exception is thrown under waitress, because of 
>> attempted string+byte concatenation. 
>> (https://github.com/Pylons/waitress/blob/master/src/waitress/task.py#L316 
>> )
>> 
>> >towrite += data + b"\r\n"
>> > TypeError: can only concatenate str (not "bytes") to str
>> 
>> 
>> 2. I picked this up in a unit test; i can't seem to access any sort of 
>> body/text/content off the response.
>> 
>> from pyramid.request import Request
>> from pyramid import testing
>> 
>> self.config = config = testing.setUp()
>> app = self.config.make_wsgi_app()
>> 
>> req = Request.blank(csv_link)
>> req.remote_addr = "127.0.0.1"
>> resp = req.get_response(app)
>> self.assertEqual(resp.status_code, 200)
>> self.assertTrue(resp.text.startswith("User CPU time,"))
>> 
>> 
>> 
>> ---
>> 
>>  def view_csv(request):
>> csvfile = StringIO()
>> csvwriter = csv.writer(csvfile)
>> for row in ((1, "a"), (2, "b")):
>> csvwriter.writerow(row)
>> csvfile.seek(0)
>> as_csv = Response(content_type="text/csv", body_file=csvfile, status=200)
>> as_csv.headers["Content-Disposition"] = str("attachment; 
>> filename=example.csv")
>> return as_csv
>> 
>> ---
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to pylons-discus...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/pylons-discuss/7d3c278f-8d20-4456-b05c-33c8d2cb6a67n%40googlegroups.com
>>  
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/e73aae83-0f7e-414c-a9c3-c20957793a0dn%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/1FB1489E-6227-47F7-9631-342F3D14B24C%40luhn.com.

Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Jonathan Vanasco]

Thanks so much, Bert. That should have been obvious!

As an interim solution, I'm just wrapping the inputs:

if six.PY3:
csvfile = BytesIO(csvfile.read().encode("utf-8"))
csvfile.seek(0)



On Sunday, October 18, 2020 at 4:22:16 PM UTC-4 Bert JW Regeer wrote:

> Your body_file is not bytes, but str. You need to make sure that what you 
> pass to body_file returns bytes.
>
> On Oct 16, 2020, at 15:03, Jonathan Vanasco  wrote:
>
> I discovered an issue with our code when generating dynamic CSVs under 
> Python3.  I'm hoping someone can point me in the right direction.  I 
> couldn't find the appropriate migration/changelog information.  This works 
> fine under Python2.
>
> The generic way we create/serve the CSV and check it in tests are below.
>
> The two problems:
>
> 1. Under Python3, an exception is thrown under waitress, because of 
> attempted string+byte concatenation. (
> https://github.com/Pylons/waitress/blob/master/src/waitress/task.py#L316)
>
> >towrite += data + b"\r\n"
>
> > TypeError: can only concatenate str (not "bytes") to str
>
> 2. I picked this up in a unit test; i can't seem to access any sort of 
> body/text/content off the response.
>
> from pyramid.request import Request
> from pyramid import testing
>
> self.config = config = testing.setUp()
> app = self.config.make_wsgi_app()
>
> req = Request.blank(csv_link)
> req.remote_addr = "127.0.0.1"
> resp = req.get_response(app)
> self.assertEqual(resp.status_code, 200)
> self.assertTrue(resp.text.startswith("User CPU time,"))
>
>
>
> ---
>
>  def view_csv(request):
> csvfile = StringIO()
> csvwriter = csv.writer(csvfile)
> for row in ((1, "a"), (2, "b")):
> csvwriter.writerow(row)
> csvfile.seek(0)
> as_csv = Response(content_type="text/csv", body_file=csvfile, 
> status=200)
> as_csv.headers["Content-Disposition"] = str("attachment; 
> filename=example.csv")
> return as_csv
>
> ---
>
>
>
>
>
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discus...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/7d3c278f-8d20-4456-b05c-33c8d2cb6a67n%40googlegroups.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/e73aae83-0f7e-414c-a9c3-c20957793a0dn%40googlegroups.com.

Re: [pylons-discuss] Issues migrating dynamic CSV to Python3

[Bert JW Regeer]

Your body_file is not bytes, but str. You need to make sure that what you pass 
to body_file returns bytes.

> On Oct 16, 2020, at 15:03, Jonathan Vanasco  wrote:
> 
> I discovered an issue with our code when generating dynamic CSVs under 
> Python3.  I'm hoping someone can point me in the right direction.  I couldn't 
> find the appropriate migration/changelog information.  This works fine under 
> Python2.
> 
> The generic way we create/serve the CSV and check it in tests are below.
> 
> The two problems:
> 
> 1. Under Python3, an exception is thrown under waitress, because of attempted 
> string+byte concatenation. 
> (https://github.com/Pylons/waitress/blob/master/src/waitress/task.py#L316)
> 
> >towrite += data + b"\r\n"
> > TypeError: can only concatenate str (not "bytes") to str
> 
> 
> 2. I picked this up in a unit test; i can't seem to access any sort of 
> body/text/content off the response.
> 
> from pyramid.request import Request
> from pyramid import testing
> 
> self.config = config = testing.setUp()
> app = self.config.make_wsgi_app()
> 
> req = Request.blank(csv_link)
> req.remote_addr = "127.0.0.1"
> resp = req.get_response(app)
> self.assertEqual(resp.status_code, 200)
> self.assertTrue(resp.text.startswith("User CPU time,"))
> 
> 
> 
> ---
> 
>  def view_csv(request):
> csvfile = StringIO()
> csvwriter = csv.writer(csvfile)
> for row in ((1, "a"), (2, "b")):
> csvwriter.writerow(row)
> csvfile.seek(0)
> as_csv = Response(content_type="text/csv", body_file=csvfile, status=200)
> as_csv.headers["Content-Disposition"] = str("attachment; 
> filename=example.csv")
> return as_csv
> 
> ---
> 
> 
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to pylons-discuss+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/pylons-discuss/7d3c278f-8d20-4456-b05c-33c8d2cb6a67n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to pylons-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/486D44C1-F653-4F86-9FED-DFDF5CD186D6%400x58.com.