Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?

2015-06-19 Thread Ryan (Cloud Platform Support) 
Glad that helped. As stated it's still being developed so bugs are still 
expected.

On Friday, June 19, 2015 at 12:26:45 AM UTC-4, Emlyn wrote:
>
> oh, update: 
>
> In the cloudstorage library, in common.py, I added this: 
>
>   @property 
>   def file_name(self): 
>   return self.filename 
>
> to the class GCSFileStat, and now GoogleCloudStorageLineInputReader 
> appears to be working. Woohoo! 
>
> On 19 June 2015 at 13:52, Emlyn  wrote: 
> > I'm trying this library here as pointed to by Ryan: 
> > https://github.com/rbruyere/appengine-mapreduce 
> > 
> > But I'm getting an odd mismatch between it and the Cloud Storage 
> > client library; it's looking for "file_name" on GCSFileStat objects, 
> > but in the library these only have "filename" 
> > 
> > I'm using version 1.9.21.0 of the cloud storage client library. Is 
> > there some other version I should be using? 
> > 
> > On 19 June 2015 at 10:57, Emlyn  wrote: 
> >> Thanks very much Ryan, 
> >> 
> >> I'll give it a shot. If I hit any walls or notice any issues, I'll let 
> you know. 
> >> 
> >> On 18 June 2015 at 23:21, Ryan (Cloud Platform Support) 
> >>  wrote: 
> >>> Salutations Emlun, 
> >>> 
> >>> There is work on LineInput Readers here. It is still Beta and has not 
> been 
> >>> accepted by the master thread yet so be careful using it. 
> >>> 
> >>> 
> >>> On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote: 
>  
>  This is a python appengine question, mapreduce 1.9.21 
>  
>  I have code writing lines to a blob in the local blobstore, then 
>  processing that using mapreduce BlobstoreLineInputReader. 
>  
>  Given that the files api is going away, I thought I'd retarget all my 
>  processing to cloud storage. 
>  
>  I would expect to find a class called 
> GoogleCloudStorageLineInputReader, 
>  but there isn't anything like that. 
>  
>  Is there something way I can use GoogleCloudStorageInputReader to 
> read 
>  lines? 
>  
>  Another possibility is using GoogleCloudStorageRecordInputReader, but 
> for 
>  that my input file needs to be in LevelDB format and I don't know how 
> to 
>  create that except with a 
> GoogleCloudStorageConsistentRecordOutputWriter, 
>  which I don't know how to use outside a mapreduce context. How might 
> I do 
>  that? 
>  
>  Or am I doing this all wrong, is there some other possibility I've 
> missed? 
>  
>  
>  
> >>> -- 
> >>> You received this message because you are subscribed to the Google 
> Groups 
> >>> "Google App Engine" group. 
> >>> To unsubscribe from this group and stop receiving emails from it, send 
> an 
> >>> email to google-appengine+unsubscr...@googlegroups.com. 
> >>> To post to this group, send email to google-appengine@googlegroups.com. 
>
> >>> Visit this group at http://groups.google.com/group/google-appengine. 
> >>> To view this discussion on the web visit 
> >>> 
> https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com.
>  
>
> >>> 
> >>> For more options, visit https://groups.google.com/d/optout. 
> >> 
> >> 
> >> 
> >> -- 
> >> Emlyn 
> >> 
> >> http://point7.wordpress.com - My blog 
> >> https://plus.google.com/u/0/100281903174934656260 - Google+ 
> > 
> > 
> > 
> > -- 
> > Emlyn 
> > 
> > http://point7.wordpress.com - My blog 
> > https://plus.google.com/u/0/100281903174934656260 - Google+ 
>
>
>
> -- 
> Emlyn 
>
> http://point7.wordpress.com - My blog 
> https://plus.google.com/u/0/100281903174934656260 - Google+ 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b77bb03c-1c7e-43ab-b1c2-fe748929c3dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?

2015-06-18 Thread Emlyn 
oh, update:

In the cloudstorage library, in common.py, I added this:

  @property
  def file_name(self):
  return self.filename

to the class GCSFileStat, and now GoogleCloudStorageLineInputReader
appears to be working. Woohoo!

On 19 June 2015 at 13:52, Emlyn  wrote:
> I'm trying this library here as pointed to by Ryan:
> https://github.com/rbruyere/appengine-mapreduce
>
> But I'm getting an odd mismatch between it and the Cloud Storage
> client library; it's looking for "file_name" on GCSFileStat objects,
> but in the library these only have "filename"
>
> I'm using version 1.9.21.0 of the cloud storage client library. Is
> there some other version I should be using?
>
> On 19 June 2015 at 10:57, Emlyn  wrote:
>> Thanks very much Ryan,
>>
>> I'll give it a shot. If I hit any walls or notice any issues, I'll let you 
>> know.
>>
>> On 18 June 2015 at 23:21, Ryan (Cloud Platform Support)
>>  wrote:
>>> Salutations Emlun,
>>>
>>> There is work on LineInput Readers here. It is still Beta and has not been
>>> accepted by the master thread yet so be careful using it.
>>>
>>>
>>> On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote:

 This is a python appengine question, mapreduce 1.9.21

 I have code writing lines to a blob in the local blobstore, then
 processing that using mapreduce BlobstoreLineInputReader.

 Given that the files api is going away, I thought I'd retarget all my
 processing to cloud storage.

 I would expect to find a class called GoogleCloudStorageLineInputReader,
 but there isn't anything like that.

 Is there something way I can use GoogleCloudStorageInputReader to read
 lines?

 Another possibility is using GoogleCloudStorageRecordInputReader, but for
 that my input file needs to be in LevelDB format and I don't know how to
 create that except with a GoogleCloudStorageConsistentRecordOutputWriter,
 which I don't know how to use outside a mapreduce context. How might I do
 that?

 Or am I doing this all wrong, is there some other possibility I've missed?



>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to google-appengine+unsubscr...@googlegroups.com.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Emlyn
>>
>> http://point7.wordpress.com - My blog
>> https://plus.google.com/u/0/100281903174934656260 - Google+
>
>
>
> --
> Emlyn
>
> http://point7.wordpress.com - My blog
> https://plus.google.com/u/0/100281903174934656260 - Google+



-- 
Emlyn

http://point7.wordpress.com - My blog
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPCnbwDQpqdiimFMVsD_h%3Dh6NJZfr31fgs7%2BF2ep6DFATg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?

2015-06-18 Thread Emlyn 
I'm trying this library here as pointed to by Ryan:
https://github.com/rbruyere/appengine-mapreduce

But I'm getting an odd mismatch between it and the Cloud Storage
client library; it's looking for "file_name" on GCSFileStat objects,
but in the library these only have "filename"

I'm using version 1.9.21.0 of the cloud storage client library. Is
there some other version I should be using?

On 19 June 2015 at 10:57, Emlyn  wrote:
> Thanks very much Ryan,
>
> I'll give it a shot. If I hit any walls or notice any issues, I'll let you 
> know.
>
> On 18 June 2015 at 23:21, Ryan (Cloud Platform Support)
>  wrote:
>> Salutations Emlun,
>>
>> There is work on LineInput Readers here. It is still Beta and has not been
>> accepted by the master thread yet so be careful using it.
>>
>>
>> On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote:
>>>
>>> This is a python appengine question, mapreduce 1.9.21
>>>
>>> I have code writing lines to a blob in the local blobstore, then
>>> processing that using mapreduce BlobstoreLineInputReader.
>>>
>>> Given that the files api is going away, I thought I'd retarget all my
>>> processing to cloud storage.
>>>
>>> I would expect to find a class called GoogleCloudStorageLineInputReader,
>>> but there isn't anything like that.
>>>
>>> Is there something way I can use GoogleCloudStorageInputReader to read
>>> lines?
>>>
>>> Another possibility is using GoogleCloudStorageRecordInputReader, but for
>>> that my input file needs to be in LevelDB format and I don't know how to
>>> create that except with a GoogleCloudStorageConsistentRecordOutputWriter,
>>> which I don't know how to use outside a mapreduce context. How might I do
>>> that?
>>>
>>> Or am I doing this all wrong, is there some other possibility I've missed?
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at http://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Emlyn
>
> http://point7.wordpress.com - My blog
> https://plus.google.com/u/0/100281903174934656260 - Google+



-- 
Emlyn

http://point7.wordpress.com - My blog
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPBJXKEMUsGM2xxrOg7vRkNk6-ONCiP0t%2BN%2BE6cRi7C1vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?

2015-06-18 Thread Emlyn 
Thanks very much Ryan,

I'll give it a shot. If I hit any walls or notice any issues, I'll let you know.

On 18 June 2015 at 23:21, Ryan (Cloud Platform Support)
 wrote:
> Salutations Emlun,
>
> There is work on LineInput Readers here. It is still Beta and has not been
> accepted by the master thread yet so be careful using it.
>
>
> On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote:
>>
>> This is a python appengine question, mapreduce 1.9.21
>>
>> I have code writing lines to a blob in the local blobstore, then
>> processing that using mapreduce BlobstoreLineInputReader.
>>
>> Given that the files api is going away, I thought I'd retarget all my
>> processing to cloud storage.
>>
>> I would expect to find a class called GoogleCloudStorageLineInputReader,
>> but there isn't anything like that.
>>
>> Is there something way I can use GoogleCloudStorageInputReader to read
>> lines?
>>
>> Another possibility is using GoogleCloudStorageRecordInputReader, but for
>> that my input file needs to be in LevelDB format and I don't know how to
>> create that except with a GoogleCloudStorageConsistentRecordOutputWriter,
>> which I don't know how to use outside a mapreduce context. How might I do
>> that?
>>
>> Or am I doing this all wrong, is there some other possibility I've missed?
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



-- 
Emlyn

http://point7.wordpress.com - My blog
https://plus.google.com/u/0/100281903174934656260 - Google+

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMp1VPBsUMJqCwUtCb8k_a8peniGeOBPbCiCDPaxngQHhi06qQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Replacing local blobstore with cloud storage, what is the replacement for BlobstoreLineInputReader?

2015-06-18 Thread Ryan (Cloud Platform Support) 
Salutations Emlun,

There is work on LineInput Readers here 
. It is still Beta and has 
not been accepted by the master thread yet so be careful using it.

On Thursday, June 18, 2015 at 3:44:26 AM UTC-4, Emlyn wrote:
>
> This is a python appengine question, mapreduce 1.9.21
>
> I have code writing lines to a blob in the local blobstore, then 
> processing that using mapreduce BlobstoreLineInputReader. 
>
> Given that the files api is going away, I thought I'd retarget all my 
> processing to cloud storage.
>
> I would expect to find a class called GoogleCloudStorageLineInputReader, 
> but there isn't anything like that. 
>
> Is there something way I can use GoogleCloudStorageInputReader to read 
> lines? 
>
> Another possibility is using GoogleCloudStorageRecordInputReader, but for 
> that my input file needs to be in LevelDB format and I don't know how to 
> create that except with a GoogleCloudStorageConsistentRecordOutputWriter, 
> which I don't know how to use outside a mapreduce context. How might I do 
> that?
>
> Or am I doing this all wrong, is there some other possibility I've missed?
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/aed62252-a74a-4170-b1c4-283bd9f62851%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.