Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
On 7 Nov 2014 23:48, PK p...@gae123.com wrote: wrt off topic As I have stated before, when I start working on a new project I always try to understand what is the developers' primary development environment and use the same or similar. This avoids too much bleeding especially for bleeding edge technologies. So I never used GAE on windows. My first apengine environment was on a windows notebook but the notebook was hosting a VM with the GAE SDK/editor etc. So it was really Linux. Having two windows systems on one machine is a pain so I started thinking what to do. Native development on bare metal Linux was not an option for a number of reasons but I had started seeing a lot of people with MacBooks. So, three years ago I made the decision and switched to a Mac. I liked the GUI, the multiple spaces---that allowed to visually contain different projects and personal work--etc. I also liked that there is UNIX underneath, I can always open a terminal window and ssh anywhere etc. At that time I figured out how to set up the SDK to run natively on the mac and have been living happily since then. I wrote about that here: http://www.gae123.com/articles/dpwf/#idp140641236359312 Using virtualenv was a key insight at the time. For the past week I spent a lot of time--definitely much more than I had allotted to-- to figure out how to run the SDK and my app engine dev environment on docker. My primary motivation has been that I want to upgrade to the latest MacOS 10.10 doing a clean install. However, I have been concerned that the native maces dev. environment will stop working and the last thing I want is to have to weather a production crisis with an unstable dev. environment. I think I have accomplished my goal but I will need one more week to make sure I have not missed anything. I will do a write-up when I feel comfortable with the new approach. PK http://www.gae123.com On November 6, 2014 at 3:41:35 PM, Kaan Soral (kaanso...@gmail.com) wrote: Off topic, if there is anyone around thinking of making the switch, the sqlite datastore seems to be compatible with mac os, the blobstore is compatible, the only issue seems to be this, for a direct switch macports py27-lxml + py27-pil + using the python2.7 of macports solves the dev_appserver requirements Generally speaking, I wouldn't suggest switching to mac for no reason, I switched because I wasn't able to easily move my win7 setup to a new hardware, and my laptop was glitching for years, mac os's time machine / easy restore features sounded great for future hardware issues So far I've been spending my time trying to select a new IDE / trying to solve font/svn issues - retina screens are great, however regular external lcd's render fonts really bad, so a 4K screen halved into retina resolution might be a must for those who are obsessed with IDE fonts, long story short, it's cost after cost after disappointment after cost ... but I'm hoping it's probably going to pay off in the near future -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Here is the final solution to the issue, I suggest the file_blob_storage
modification and renaming the existing files for everyone, it's extremely
clumsy to utilize special characters in file names (macos seems to handle
/ in the file name, don't know why / is used instead of : as windows,
the modification renames the file name, existing urls and references keep
working, yet a refactoring is needed for the files themselves)
*file_blob_storage.py modification*
def _FileForBlob(self, blob_key):
blob_key = self._BlobKey(blob_key)
return os.path.join(self._DirectoryForBlob(blob_key),
str(blob_key)[1:].replace(/,;).replace(:,;))
This ensures that files have ; instead of / or :
*To pull out the special files from the windows SDK*
1) Go to blob_image.py's serve_image function and pass/comment out
the datastore.Get(key) check (because we're going to ping files from the
serving url scheme, this ensures every file is served from that url scheme)
2) Modify file_blob_storage.py to copy the accessed files to a new location
(from the old SDK)
def _FileForBlob(self, blob_key):
blob_key = self._BlobKey(blob_key); handle=False
path=os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:])
if path.find(ncoded)!=-1: handle=True
if handle and not kaan_marked.get(blob_key):
logging.info(YEY %s%blob_key)
kaan_marked[blob_key]=True
blob_file = self._FileForBlob(blob_key)
input = _local_open(blob_file, 'rb')
output= _local_open(C:/GCS/+str(blob_key)[1:].replace(:,;)
,'wb')
try:
while True:
block = input.read(1 20)
if not block:
break
output.write(block)
finally:
output.close()
logging.info(path)
return os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:])
3) Ping each image, I decided to ping them by browsing the
__GsFileInfo__'s from the Datastore VIewer, couldn't access them easily
programatically
gg=document.getElementsByTagName(td); for(var i=0;igg.length;i++)
if(gg[i].innerHTML.indexOf(ncoded)!=-1) { var image =
document.createElement(img); image.src =
http://localhost/_ah/img/+gg[i].innerHTML.trim();
document.body.appendChild(image); }
Then put the images back to the old or new n folder
I think all windows SDK users should do this, I don't know how macos
handles / in the filenames, yet I would suggest everyone to fix the file
access manually
I've added this to my list of manual sdk modifications, currently I have 9
manual modifications, sad ... :)
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Well, it turns out the gcs files were never written to the blobstore directory, I'm hunting where they are, I have a hunch they are in the db.rdbms, hoping they are in a separate temp directory instead (my last blobstore update was from 05/2013) However on the mac dev_appserver, they are actually physical files such as ncoded_gs_file:Z2VvYmlyZGVsZW1lbnRzL2ZpbGVzL2Zha2UtaWU2VjNaVHhjWUg0TXpwUjNsUnoyQT09 in the n directory so the mac sdk is looking for them in the right place (mac sdk-wise) however they are not there Any ideas? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Hi Kaan, do you have the latest app engine GCS SDK? The docs say that the SDK is in googlecode but it seems it has moved in github here. This past week I switched my development environment from native mac (and native linux on another machine) to native docker. I need to write up about the experience but one of the obstacles was that tests and functionality around GCS stopped working. Then I updated my GCS client from the latest git repository and I am back in business on that front. I am also not sure from your write-up, are you trying to move some data already in app engine SDK during the transition? PK http://www.gae123.com On November 7, 2014 at 2:49:50 AM, Kaan Soral (kaanso...@gmail.com) wrote: Well, it turns out the gcs files were never written to the blobstore directory, I'm hunting where they are, I have a hunch they are in the db.rdbms, hoping they are in a separate temp directory instead (my last blobstore update was from 05/2013) However on the mac dev_appserver, they are actually physical files such as ncoded_gs_file:Z2VvYmlyZGVsZW1lbnRzL2ZpbGVzL2Zha2UtaWU2VjNaVHhjWUg0TXpwUjNsUnoyQT09 in the n directory so the mac sdk is looking for them in the right place (mac sdk-wise) however they are not there Any ideas? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Hi PK That is exactly what I'm trying to do, thanks for the reply The SDK/GCS serving part seems to be separate from the upload/library logic, If I'm not mistaken, so my issue is currently the serving part (offtopic, I seem to be manually modifying the cloudstorage library for the serving url part, don't remember why, I probably need to re-visit that logic when I restore old images because I can upload and verify new images, however their serving urls seem to be faulty for me, concatenating port and hostname without a : etc ... ) I've just uploaded a 2MB photo to my old setup, checking whether it's loaded as a blob into the .rdbms, however that wasn't the case, yet the photo is missing, I suspect I need to find the location of that photo, move all photos, and continue from there (my current theory) Yet I'm unable to find that path, I always had --storage_path / --blobstore_path's set -- yet the photo is missing I'm puzzled, it doesn't seem to be in the AppData / Temp folder either, although I couldn't verify it 100%, the old directories seem to be untouched TLDR, I'm missing my GCS images, can't find them, can't transport them to my new setup -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Sorry I cannot help there, I hope there is a way that is not too much trouble. I have never trusted compatibility of the SDK format on the disk, I assume it can break in a moment’s notice between releases even in the same platform. So I have an import/export tool that I use for data portability” purposes. I use it both with the SDK and production. I highly recommend you move to a similar approach. Furthermore, I really do not use development to store data I care about. I have a tool that generates data for testing purposes and once in a while I blow up all my SDK data and generate a new system with the tool. PK http://www.gae123.com On November 7, 2014 at 9:49:17 AM, Kaan Soral (kaanso...@gmail.com) wrote: Hi PK That is exactly what I'm trying to do, thanks for the reply The SDK/GCS serving part seems to be separate from the upload/library logic, If I'm not mistaken, so my issue is currently the serving part (offtopic, I seem to be manually modifying the cloudstorage library for the serving url part, don't remember why, I probably need to re-visit that logic when I restore old images because I can upload and verify new images, however their serving urls seem to be faulty for me, concatenating port and hostname without a : etc ... ) I've just uploaded a 2MB photo to my old setup, checking whether it's loaded as a blob into the .rdbms, however that wasn't the case, yet the photo is missing, I suspect I need to find the location of that photo, move all photos, and continue from there (my current theory) Yet I'm unable to find that path, I always had --storage_path / --blobstore_path's set -- yet the photo is missing I'm puzzled, it doesn't seem to be in the AppData / Temp folder either, although I couldn't verify it 100%, the old directories seem to be untouched TLDR, I'm missing my GCS images, can't find them, can't transport them to my new setup -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Well checking the blobstore's n folder now, I see that the 0byte/empty ncoded_gs_file updated when I uploaded the photo So I'm guessing there is a file access issue, the images are likely on another dimension :) I've checked the folder with cygwin too, no trace of the images, I'm guessing it's the / in the images names They seem to be named like ncoded_gs_file/Z2VvYmlyZGVsZW1lbnRzL2ZpbGVzL2Zha2UtenFTLWhEcHFUclhKZ1N2RFROLS1Ydz09 etc... Going to look for a way to look for them / access them now, haven't verified their existence yet -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
wrt “off topic” As I have stated before, when I start working on a new project I always try to understand what is the developers' primary development environment and use the same or similar. This avoids too much bleeding especially for bleeding edge technologies. So I never used GAE on windows. My first apengine environment was on a windows notebook but the notebook was hosting a VM with the GAE SDK/editor etc. So it was really Linux. Having two windows systems on one machine is a pain so I started thinking what to do. Native development on bare metal Linux was not an option for a number of reasons but I had started seeing a lot of people with MacBooks. So, three years ago I made the decision and switched to a Mac. I liked the GUI, the multiple spaces---that allowed to visually contain different projects and personal work—etc. I also liked that there is UNIX underneath, I can always open a terminal window and ssh anywhere etc. At that time I figured out how to set up the SDK to run natively on the mac and have been living happily since then. I wrote about that here: http://www.gae123.com/articles/dpwf/#idp140641236359312 Using virtualenv was a key insight at the time. For the past week I spent a lot of time—definitely much more than I had allotted to— to figure out how to run the SDK and my app engine dev environment on docker. My primary motivation has been that I want to upgrade to the latest MacOS 10.10 doing a clean install. However, I have been concerned that the native maces dev. environment will stop working and the last thing I want is to have to weather a production crisis with an unstable dev. environment. I think I have accomplished my goal but I will need one more week to make sure I have not missed anything. I will do a write-up when I feel comfortable with the new approach. PK http://www.gae123.com On November 6, 2014 at 3:41:35 PM, Kaan Soral (kaanso...@gmail.com) wrote: Off topic, if there is anyone around thinking of making the switch, the sqlite datastore seems to be compatible with mac os, the blobstore is compatible, the only issue seems to be this, for a direct switch macports py27-lxml + py27-pil + using the python2.7 of macports solves the dev_appserver requirements Generally speaking, I wouldn't suggest switching to mac for no reason, I switched because I wasn't able to easily move my win7 setup to a new hardware, and my laptop was glitching for years, mac os's time machine / easy restore features sounded great for future hardware issues So far I've been spending my time trying to select a new IDE / trying to solve font/svn issues - retina screens are great, however regular external lcd's render fonts really bad, so a 4K screen halved into retina resolution might be a must for those who are obsessed with IDE fonts, long story short, it's cost after cost after disappointment after cost ... but I'm hoping it's probably going to pay off in the near future -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Thanks anyway That is actually a great approach, I considered that a couple of times myself, I use the same approach with other platforms, but compared to them, GAE/SDK is extremely stable, the GCS issue is a shame, whoever implemented it should be ashamed, other than that, things seem pretty persistent across operating systems and versions I've been maintaining the SDK setup I'm currently working on for more than 2 years, so It kind of has a lot of sentimental value, a lot of memories, that's why I'm very keen on restoring it Worst case scenario, I might ping/upload each file manually/programmatically, from old to new, however I hope it doesn't come to that It seems like this issue affects all windows sdk users, so please pitch in with your experience -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Really looking forward to that article :) I guess you could also backup, upgrade and restore a mac, if things doesn't go your way with an upgrade, however downgrading the os itself would be a challenge I guess (haven't tried/researched the process myself, I hope the installation restores the os too, instead of restoring and upgrading, in a case like that, things would be really easy in that case) I was looking forward to having a firewall to play with, on unix/macos, however I really hate the pf firewall, there is also no simple articles online that accomplishes simple tasks with pf etc, so I'm guessing 10.10 update might be irritating for previous version users On Friday, November 7, 2014 8:18:54 PM UTC+2, PK wrote: wrt “off topic” As I have stated before, when I start working on a new project I always try to understand what is the developers' primary development environment and use the same or similar. This avoids too much bleeding especially for bleeding edge technologies. So I never used GAE on windows. My first apengine environment was on a windows notebook but the notebook was hosting a VM with the GAE SDK/editor etc. So it was really Linux. Having two windows systems on one machine is a pain so I started thinking what to do. Native development on bare metal Linux was not an option for a number of reasons but I had started seeing a lot of people with MacBooks. So, three years ago I made the decision and switched to a Mac. I liked the GUI, the multiple spaces---that allowed to visually contain different projects and personal work—etc. I also liked that there is UNIX underneath, I can always open a terminal window and ssh anywhere etc. At that time I figured out how to set up the SDK to run natively on the mac and have been living happily since then. I wrote about that here: http://www.gae123.com/articles/dpwf/#idp140641236359312 Using virtualenv was a key insight at the time. For the past week I spent a lot of time—definitely much more than I had allotted to— to figure out how to run the SDK and my app engine dev environment on docker. My primary motivation has been that I want to upgrade to the latest MacOS 10.10 doing a clean install. However, I have been concerned that the native maces dev. environment will stop working and the last thing I want is to have to weather a production crisis with an unstable dev. environment. I think I have accomplished my goal but I will need one more week to make sure I have not missed anything. I will do a write-up when I feel comfortable with the new approach. PK http://www.gae123.com On November 6, 2014 at 3:41:35 PM, Kaan Soral (kaan...@gmail.com javascript:) wrote: Off topic, if there is anyone around thinking of making the switch, the sqlite datastore seems to be compatible with mac os, the blobstore is compatible, the only issue seems to be this, for a direct switch macports py27-lxml + py27-pil + using the python2.7 of macports solves the dev_appserver requirements Generally speaking, I wouldn't suggest switching to mac for no reason, I switched because I wasn't able to easily move my win7 setup to a new hardware, and my laptop was glitching for years, mac os's time machine / easy restore features sounded great for future hardware issues So far I've been spending my time trying to select a new IDE / trying to solve font/svn issues - retina screens are great, however regular external lcd's render fonts really bad, so a 4K screen halved into retina resolution might be a must for those who are obsessed with IDE fonts, long story short, it's cost after cost after disappointment after cost ... but I'm hoping it's probably going to pay off in the near future -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com javascript:. To post to this group, send email to google-a...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Switched from Windows to Mac - blobstore issues - /n/ncoded_gs_file
Off topic, if there is anyone around thinking of making the switch, the sqlite datastore seems to be compatible with mac os, the blobstore is compatible, the only issue seems to be this, for a direct switch macports py27-lxml + py27-pil + using the python2.7 of macports solves the dev_appserver requirements Generally speaking, I wouldn't suggest switching to mac for no reason, I switched because I wasn't able to easily move my win7 setup to a new hardware, and my laptop was glitching for years, mac os's time machine / easy restore features sounded great for future hardware issues So far I've been spending my time trying to select a new IDE / trying to solve font/svn issues - retina screens are great, however regular external lcd's render fonts really bad, so a 4K screen halved into retina resolution might be a must for those who are obsessed with IDE fonts, long story short, it's cost after cost after disappointment after cost ... but I'm hoping it's probably going to pay off in the near future -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
