[google-appengine] Re: I appear to be corrupting my datastore
Hi,
Thanks for filing the issue.
This is related to storing a floatproperty in the SDK datastore, on Macs the
datastore file may get occasionally corrupted. Currently you will need to
clear the datastore on your local machine to fix this issue.
-Marzia
On Sat, Dec 13, 2008 at 7:52 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.
class 'struct.error': unpack requires a string argument of length 8
def checkScoreValue(self, value):
valid = False
while valid == False:
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
value = value + 0.001
else:
valid = True
return value
I've filed an issue, #922 -
http://code.google.com/p/googleappengine/issues/detail?id=922
On Dec 13, 10:01 am, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I'm trying to make sure a score field I set for articles on my site in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.
Here's what I'm doing.
I set up a new Score Property.
class ScoreProperty(db.FloatProperty):
def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value
def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
Then when I go to add a story, I use this try statement
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
What should happen is that when a put is attempted, a check is done to
see if a story with that score exists. If it does exist, add 0.001 and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
see a score of ###.0 and then the next would be something like ###.
043, so I'm not sure what happened to .001-.0042. Also, after running
the process the datastore appears to be corrupted as well.
I'm considering swapping to using a DateTimeProperty and keying off of
the miliseconds to see if that is handled better, but I'm confused as
to why this current method is creating problems.
One thing that might be an issue is the development is happening on an
eeepc, which is a single core processor and it's SSD isn't the fast
read/write storage device.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: I appear to be corrupting my datastore
This is actually happening on Linux, not sure if that matters or not.
Clearing the datastore is my only solution to getting it back up on
the SDK currently, but once I run the routine again, it's corrupted
again. I did try converting the entire property to a
db.DateTimeProperty and adding milliseconds, and had the same
corruption issues.
On Dec 15, 2:07 pm, Marzia Niccolai ma...@google.com wrote:
Hi,
Thanks for filing the issue.
This is related to storing a floatproperty in the SDK datastore, on Macs the
datastore file may get occasionally corrupted. Currently you will need to
clear the datastore on your local machine to fix this issue.
-Marzia
On Sat, Dec 13, 2008 at 7:52 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.
class 'struct.error': unpack requires a string argument of length 8
def checkScoreValue(self, value):
valid = False
while valid == False:
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
value = value + 0.001
else:
valid = True
return value
I've filed an issue, #922 -
http://code.google.com/p/googleappengine/issues/detail?id=922
On Dec 13, 10:01 am, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I'm trying to make sure a score field I set for articles on my site in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.
Here's what I'm doing.
I set up a new Score Property.
class ScoreProperty(db.FloatProperty):
def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value
def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
Then when I go to add a story, I use this try statement
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
What should happen is that when a put is attempted, a check is done to
see if a story with that score exists. If it does exist, add 0.001 and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
see a score of ###.0 and then the next would be something like ###.
043, so I'm not sure what happened to .001-.0042. Also, after running
the process the datastore appears to be corrupted as well.
I'm considering swapping to using a DateTimeProperty and keying off of
the miliseconds to see if that is handled better, but I'm confused as
to why this current method is creating problems.
One thing that might be an issue is the development is happening on an
eeepc, which is a single core processor and it's SSD isn't the fast
read/write storage device.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: I appear to be corrupting my datastore
Hi,
I just assumed it was Mac because it's an issue we've seen with Macs that
have 2.5.0 installed, and I'm assuming this is your Python installation?
Generally, float values in the datastore just don't work with Python 2.5.0.
Support for buffer objects in struct.unpack was not added until version
54695 (Apr 2007), this is the underlying cause of the error message.
-Marzia
On Mon, Dec 15, 2008 at 11:54 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
This is actually happening on Linux, not sure if that matters or not.
Clearing the datastore is my only solution to getting it back up on
the SDK currently, but once I run the routine again, it's corrupted
again. I did try converting the entire property to a
db.DateTimeProperty and adding milliseconds, and had the same
corruption issues.
On Dec 15, 2:07 pm, Marzia Niccolai ma...@google.com wrote:
Hi,
Thanks for filing the issue.
This is related to storing a floatproperty in the SDK datastore, on Macs
the
datastore file may get occasionally corrupted. Currently you will need to
clear the datastore on your local machine to fix this issue.
-Marzia
On Sat, Dec 13, 2008 at 7:52 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.
class 'struct.error': unpack requires a string argument of length 8
def checkScoreValue(self, value):
valid = False
while valid == False:
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
value = value + 0.001
else:
valid = True
return value
I've filed an issue, #922 -
http://code.google.com/p/googleappengine/issues/detail?id=922
On Dec 13, 10:01 am, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I'm trying to make sure a score field I set for articles on my site
in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't
view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.
Here's what I'm doing.
I set up a new Score Property.
class ScoreProperty(db.FloatProperty):
def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value
def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
Then when I go to add a story, I use this try statement
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
What should happen is that when a put is attempted, a check is done
to
see if a story with that score exists. If it does exist, add 0.001
and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
see a score of ###.0 and then the next would be something like ###.
043, so I'm not sure what happened to .001-.0042. Also, after running
the process the datastore appears to be corrupted as well.
I'm considering swapping to using a DateTimeProperty and keying off
of
the miliseconds to see if that is handled better, but I'm confused as
to why this current method is creating problems.
One thing that might be an issue is the development is happening on
an
eeepc, which is a single core processor and it's SSD isn't the fast
read/write storage device.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: I appear to be corrupting my datastore
Ok great. I'll have to check on the python version, it's whatever
xandros makes available (I haven't gotten rid of the default OS on my
eeepc because I'm lazy). If worse comes to worse I'll try compiling
the latest python by hand and see if that works. I should have this
done by the weekend. I'll post a comment to the ticket if this
resolves the issue. Thanks for the quick knowledgeable response as
always.
On Dec 15, 3:06 pm, Marzia Niccolai ma...@google.com wrote:
Hi,
I just assumed it was Mac because it's an issue we've seen with Macs that
have 2.5.0 installed, and I'm assuming this is your Python installation?
Generally, float values in the datastore just don't work with Python 2.5.0.
Support for buffer objects in struct.unpack was not added until version
54695 (Apr 2007), this is the underlying cause of the error message.
-Marzia
On Mon, Dec 15, 2008 at 11:54 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
This is actually happening on Linux, not sure if that matters or not.
Clearing the datastore is my only solution to getting it back up on
the SDK currently, but once I run the routine again, it's corrupted
again. I did try converting the entire property to a
db.DateTimeProperty and adding milliseconds, and had the same
corruption issues.
On Dec 15, 2:07 pm, Marzia Niccolai ma...@google.com wrote:
Hi,
Thanks for filing the issue.
This is related to storing a floatproperty in the SDK datastore, on Macs
the
datastore file may get occasionally corrupted. Currently you will need to
clear the datastore on your local machine to fix this issue.
-Marzia
On Sat, Dec 13, 2008 at 7:52 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.
class 'struct.error': unpack requires a string argument of length 8
def checkScoreValue(self, value):
valid = False
while valid == False:
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
value = value + 0.001
else:
valid = True
return value
I've filed an issue, #922 -
http://code.google.com/p/googleappengine/issues/detail?id=922
On Dec 13, 10:01 am, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I'm trying to make sure a score field I set for articles on my site
in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't
view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.
Here's what I'm doing.
I set up a new Score Property.
class ScoreProperty(db.FloatProperty):
def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value
def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
Then when I go to add a story, I use this try statement
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
What should happen is that when a put is attempted, a check is done
to
see if a story with that score exists. If it does exist, add 0.001
and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
see a score of ###.0 and then the next would be something like ###.
043, so I'm not sure what happened to .001-.0042. Also, after running
the process the datastore appears to be corrupted as well.
I'm considering swapping to using a DateTimeProperty and keying off
of
the miliseconds to see if that is handled better, but I'm confused as
to why this current method is creating problems.
One thing that might be an issue is the development is happening on
an
eeepc, which is a single core processor
[google-appengine] Re: I appear to be corrupting my datastore
compiling 2.5.2 from source appears to have fixed the database
corruption, though I'm still getting the http response too large error
when I try to bring up the stories now that they're scored as floating
points. If it turns out to be an appengine problem, I'll reply to this
thread, otherwise if it is just a problem with my code, I'll let this
topic die. Thanks again.
On Dec 15, 3:33 pm, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
Ok great. I'll have to check on the python version, it's whatever
xandros makes available (I haven't gotten rid of the default OS on my
eeepc because I'm lazy). If worse comes to worse I'll try compiling
the latest python by hand and see if that works. I should have this
done by the weekend. I'll post a comment to the ticket if this
resolves the issue. Thanks for the quick knowledgeable response as
always.
On Dec 15, 3:06 pm, Marzia Niccolai ma...@google.com wrote:
Hi,
I just assumed it was Mac because it's an issue we've seen with Macs that
have 2.5.0 installed, and I'm assuming this is your Python installation?
Generally, float values in the datastore just don't work with Python 2.5.0.
Support for buffer objects in struct.unpack was not added until version
54695 (Apr 2007), this is the underlying cause of the error message.
-Marzia
On Mon, Dec 15, 2008 at 11:54 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
This is actually happening on Linux, not sure if that matters or not.
Clearing the datastore is my only solution to getting it back up on
the SDK currently, but once I run the routine again, it's corrupted
again. I did try converting the entire property to a
db.DateTimeProperty and adding milliseconds, and had the same
corruption issues.
On Dec 15, 2:07 pm, Marzia Niccolai ma...@google.com wrote:
Hi,
Thanks for filing the issue.
This is related to storing a floatproperty in the SDK datastore, on Macs
the
datastore file may get occasionally corrupted. Currently you will need
to
clear the datastore on your local machine to fix this issue.
-Marzia
On Sat, Dec 13, 2008 at 7:52 AM, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.
class 'struct.error': unpack requires a string argument of length 8
def checkScoreValue(self, value):
valid = False
while valid == False:
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
value = value + 0.001
else:
valid = True
return value
I've filed an issue, #922 -
http://code.google.com/p/googleappengine/issues/detail?id=922
On Dec 13, 10:01 am, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I'm trying to make sure a score field I set for articles on my site
in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't
view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.
Here's what I'm doing.
I set up a new Score Property.
class ScoreProperty(db.FloatProperty):
def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value
def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
Then when I go to add a story, I use this try statement
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
What should happen is that when a put is attempted, a check is done
to
see if a story with that score exists. If it does exist, add 0.001
and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
[google-appengine] Re: I appear to be corrupting my datastore
I didn't have much luck with switching to a datetimeproperty using
milliseconds eithers. I also tried handling the value changing in the
check method directly, since it looked like it was possible there.
Same results, the datastore gets corrupted. The error I get when I try
to start the datastore after stopping it is.
class 'struct.error': unpack requires a string argument of length 8
def checkScoreValue(self, value):
valid = False
while valid == False:
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
value = value + 0.001
else:
valid = True
return value
I've filed an issue, #922 -
http://code.google.com/p/googleappengine/issues/detail?id=922
On Dec 13, 10:01 am, bowman.jos...@gmail.com
bowman.jos...@gmail.com wrote:
I'm trying to make sure a score field I set for articles on my site in
unique, however, I'm running into an issue where my method is
appearing to corrupt my datastore. After I input stories, I can't view
pages, getting a return size too large error, and when I stop and
start the SDK, it won't restart.
Here's what I'm doing.
I set up a new Score Property.
class ScoreProperty(db.FloatProperty):
def checkScoreValue(self, value):
query = db.GqlQuery('SELECT * FROM Story WHERE score = :1', value)
results = query.fetch(1)
if len(results) 0:
raise db.BadValueError(
'Property %s must be unique' % self.name)
return value
def __init__(self, verbose_name=None, name=None):
super(ScoreProperty, self).__init__(
verbose_name, name, required=False,
validator=self.checkScoreValue)
Then when I go to add a story, I use this try statement
story_added = False
while story_added == False:
try:
story.put()
story_added = True
except db.BadValueError:
story.score = story.score + 0.001
What should happen is that when a put is attempted, a check is done to
see if a story with that score exists. If it does exist, add 0.001 and
try to put again. After adding several stories in batch mode, I can
using the data view that appears to be working, but strangely. I'll
see a score of ###.0 and then the next would be something like ###.
043, so I'm not sure what happened to .001-.0042. Also, after running
the process the datastore appears to be corrupted as well.
I'm considering swapping to using a DateTimeProperty and keying off of
the miliseconds to see if that is handled better, but I'm confused as
to why this current method is creating problems.
One thing that might be an issue is the development is happening on an
eeepc, which is a single core processor and it's SSD isn't the fast
read/write storage device.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
