[google-appengine] Re: Data Modeling help for simple messaging
Overall, read/write costs in Google Datastore are completely deterministic so I do not believe implementing something I know will already be too expensive for me would be very efficient. The ideas I have come up with are quite expensive so I was looking for a possibly better way to implement these requirements. Yes, the users will need to read old messages which will need to be queried for in batches of 10 at a time. This is I think my biggest issue, is how to store these message for easy retrieval and modification(update hasRead attribute). On Monday, 28 December 2015 22:04:32 UTC-8, timh wrote: > > To be honest. > > I would try a niave implementation and deal with how you will > fetch/send/receive information and how you plan to implement your workflows. > Then profile the application and then start looking at how you might > optimise things. > > Doing things like "I then thought of creating an entity to store all the > sent/received messages in a string for a user inside of a single entity" > means you can't scale the number of messages without dealing with > overflows, and will people read old messages. If not then retrieving that > single growing entity may cost you more than the writes (in terms of time, > cache hit ratios etc)). Trying to optimise things before you have working > code is possibly not the best way to approach development. > > > T > > On Tuesday, December 29, 2015 at 12:29:48 PM UTC+8, Susan Lin wrote: >> >> I have a very simple requirement. I have an application where users log >> in, add/remove/view friends, and can send/receive (also view the message >> you sent and received) a message (message is more like an email, not a real >> time chat) from a friend. When they receive the message, the application >> needs to mark the message as 'read'. The messages must also come ordered by >> newest to oldest. >> >> My thoughts: >> >> KINDS: >> >> >> USER >> username (ID) >> password >> friends (will contain a max of 100 >> friends). >> >> MESSAGE >> msg_id (ID) >> from (indexed) >> to (indexed) >> timestamp (indexed) >> msg >> hasRead >> >> >> 1) User would login using their username and password >> 2) User could get friends by getting their USER entity based on their >> username >> 3) User could add/remove friends by getting the entity of user1 and user2 >> and either adding or removing friend via transaction to make sure they are >> consistent, >> 4) User could get all the message they have sent by using indexing the >> the 'from' attribute (limit of 10 message per request). The same could be >> done to view all the messages they have received by using the 'to' >> attribute. When the message has been seen for the first time I would go to >> the message (1 get) and update the entity (1 write + (4 writes x 3) = 13 >> writes to update the entity). >> >> My major concern - If a user gets 10 messages, this will require 10 get >> requests plus if all 10 messages are new I will need to update each entity >> and if all of them are new that is (10 x 14) 140 writes. Then if I get >> another 10 message for this user the same process and this could all add up >> very quickly. >> >> >> I then thought of creating an entity to store all the sent/received >> messages in a string for a user inside of a single entity: >> >> MESSAGE >> user_entitynum1 (ID) >> messages > msg2_touser8_read_timestamp, msg3_touser5_unread_timestamp, etc> >> >> This way I could store all the message (under 1mb) inside of this one >> entity, but I would have to keep track of which entity each user is at (if >> the user exceeds the 1mb limit I will have to create a new entity). This >> proved to also not be too efficient because if I send a message to perform >> 2 gets to see which message entity I am currently at and which message >> entity they are currently at. From there I must now use another 2 reads to >> get those entities and another 4 writes to update them both (considering I >> do not need to create another entity if it is full). >> >> >> I was looking for any ideas to accomplish what I need in a more efficient >> way. I have nothing implemented yet so I am open to ANY ideas. My main >> concern is for this to be efficient, >> >> Cheers! >> >> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/708c6dbb-70ac-45f4-9de6-0f2a6b3c2b41%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Data Modeling help for simple messaging
Overall, read/write costs in Google Datastore are completely deterministic. I don't believe it would be very efficient for me to implement something which I already know would be too expensive. I have just outlined some ideas which could meet the requirement, but they are too expensive for me. Yes, users need to be able to read old messages (in batches of 10 at a time). This I think is my biggest issue. How to efficiently model the messages to be able to be retrieved and modified (updated to the hasRead attribute) efficiently. On Monday, December 28, 2015 at 10:04:32 PM UTC-8, timh wrote: > > To be honest. > > I would try a niave implementation and deal with how you will > fetch/send/receive information and how you plan to implement your workflows. > Then profile the application and then start looking at how you might > optimise things. > > Doing things like "I then thought of creating an entity to store all the > sent/received messages in a string for a user inside of a single entity" > means you can't scale the number of messages without dealing with > overflows, and will people read old messages. If not then retrieving that > single growing entity may cost you more than the writes (in terms of time, > cache hit ratios etc)). Trying to optimise things before you have working > code is possibly not the best way to approach development. > > > T > > On Tuesday, December 29, 2015 at 12:29:48 PM UTC+8, Susan Lin wrote: >> >> I have a very simple requirement. I have an application where users log >> in, add/remove/view friends, and can send/receive (also view the message >> you sent and received) a message (message is more like an email, not a real >> time chat) from a friend. When they receive the message, the application >> needs to mark the message as 'read'. The messages must also come ordered by >> newest to oldest. >> >> My thoughts: >> >> KINDS: >> >> >> USER >> username (ID) >> password >> friends (will contain a max of 100 >> friends). >> >> MESSAGE >> msg_id (ID) >> from (indexed) >> to (indexed) >> timestamp (indexed) >> msg >> hasRead >> >> >> 1) User would login using their username and password >> 2) User could get friends by getting their USER entity based on their >> username >> 3) User could add/remove friends by getting the entity of user1 and user2 >> and either adding or removing friend via transaction to make sure they are >> consistent, >> 4) User could get all the message they have sent by using indexing the >> the 'from' attribute (limit of 10 message per request). The same could be >> done to view all the messages they have received by using the 'to' >> attribute. When the message has been seen for the first time I would go to >> the message (1 get) and update the entity (1 write + (4 writes x 3) = 13 >> writes to update the entity). >> >> My major concern - If a user gets 10 messages, this will require 10 get >> requests plus if all 10 messages are new I will need to update each entity >> and if all of them are new that is (10 x 14) 140 writes. Then if I get >> another 10 message for this user the same process and this could all add up >> very quickly. >> >> >> I then thought of creating an entity to store all the sent/received >> messages in a string for a user inside of a single entity: >> >> MESSAGE >> user_entitynum1 (ID) >> messages > msg2_touser8_read_timestamp, msg3_touser5_unread_timestamp, etc> >> >> This way I could store all the message (under 1mb) inside of this one >> entity, but I would have to keep track of which entity each user is at (if >> the user exceeds the 1mb limit I will have to create a new entity). This >> proved to also not be too efficient because if I send a message to perform >> 2 gets to see which message entity I am currently at and which message >> entity they are currently at. From there I must now use another 2 reads to >> get those entities and another 4 writes to update them both (considering I >> do not need to create another entity if it is full). >> >> >> I was looking for any ideas to accomplish what I need in a more efficient >> way. I have nothing implemented yet so I am open to ANY ideas. My main >> concern is for this to be efficient, >> >> Cheers! >> >> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7edf29f7-f8b6-4911-98ac-6410d66c8915%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: Data Modeling help for simple messaging
>From your description "When the message has been seen for the first time I would go to the message (1 get) and update the entity (1 write + (4 writes x 3) = 13 writes to update the entity)." I am having trouble working out where you get the "(4 writes x 3)" when updating a single message as "hasRead" T On Wednesday, December 30, 2015 at 1:31:31 AM UTC+8, Susan Lin wrote: > > Overall, read/write costs in Google Datastore are completely > deterministic. I don't believe it would be very efficient for me to > implement something which I already know would be too expensive. I have > just outlined some ideas which could meet the requirement, but they are too > expensive for me. > > Yes, users need to be able to read old messages (in batches of 10 at a > time). This I think is my biggest issue. How to efficiently model the > messages to be able to be retrieved and modified (updated to the hasRead > attribute) efficiently. > > On Monday, December 28, 2015 at 10:04:32 PM UTC-8, timh wrote: >> >> To be honest. >> >> I would try a niave implementation and deal with how you will >> fetch/send/receive information and how you plan to implement your workflows. >> Then profile the application and then start looking at how you might >> optimise things. >> >> Doing things like "I then thought of creating an entity to store all the >> sent/received messages in a string for a user inside of a single entity" >> means you can't scale the number of messages without dealing with >> overflows, and will people read old messages. If not then retrieving that >> single growing entity may cost you more than the writes (in terms of time, >> cache hit ratios etc)). Trying to optimise things before you have working >> code is possibly not the best way to approach development. >> >> >> T >> >> On Tuesday, December 29, 2015 at 12:29:48 PM UTC+8, Susan Lin wrote: >>> >>> I have a very simple requirement. I have an application where users log >>> in, add/remove/view friends, and can send/receive (also view the message >>> you sent and received) a message (message is more like an email, not a real >>> time chat) from a friend. When they receive the message, the application >>> needs to mark the message as 'read'. The messages must also come ordered by >>> newest to oldest. >>> >>> My thoughts: >>> >>> KINDS: >>> >>> >>> USER >>> username (ID) >>> password >>> friends (will contain a max of 100 >>> friends). >>> >>> MESSAGE >>> msg_id (ID) >>> from (indexed) >>> to (indexed) >>> timestamp (indexed) >>> msg >>> hasRead >>> >>> >>> 1) User would login using their username and password >>> 2) User could get friends by getting their USER entity based on their >>> username >>> 3) User could add/remove friends by getting the entity of user1 and >>> user2 and either adding or removing friend via transaction to make sure >>> they are consistent, >>> 4) User could get all the message they have sent by using indexing the >>> the 'from' attribute (limit of 10 message per request). The same could be >>> done to view all the messages they have received by using the 'to' >>> attribute. When the message has been seen for the first time I would go to >>> the message (1 get) and update the entity (1 write + (4 writes x 3) = 13 >>> writes to update the entity). >>> >>> My major concern - If a user gets 10 messages, this will require 10 get >>> requests plus if all 10 messages are new I will need to update each entity >>> and if all of them are new that is (10 x 14) 140 writes. Then if I get >>> another 10 message for this user the same process and this could all add up >>> very quickly. >>> >>> >>> I then thought of creating an entity to store all the sent/received >>> messages in a string for a user inside of a single entity: >>> >>> MESSAGE >>> user_entitynum1 (ID) >>> messages >> msg2_touser8_read_timestamp, msg3_touser5_unread_timestamp, etc> >>> >>> This way I could store all the message (under 1mb) inside of this one >>> entity, but I would have to keep track of which entity each user is at (if >>> the user exceeds the 1mb limit I will have to create a new entity). This >>> proved to also not be too efficient because if I send a message to perform >>> 2 gets to see which message entity I am currently at and which message >>> entity they are currently at. From there I must now use another 2 reads to >>> get those entities and another 4 writes to update them both (considering I >>> do not need to create another entity if it is full). >>> >>> >>> I was looking for any ideas to accomplish what I need in a more >>> efficient way. I have nothing implemented yet so I am open to ANY ideas. >>> My main concern is for this to be efficient, >>> >>> Cheers! >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com.
[google-appengine] Re: Indexed fields misrepresented in the new console for nested structs?
It definitely seems, from a quick run-through of the doc and your schema,
that the inner entities' flattened fields should be "noindex" here. Are you
able to run queries which would hit the index, and do you see results?
On Sunday, December 27, 2015 at 10:15:02 AM UTC-5, Rohan Chandiramani wrote:
>
> I have the following struct in my Go code:
>
> Channel struct {
> Name string
> `datastore:"-" json:"name,omitempty" goon:"id"`
> UserId int
> `datastore:",noindex" json:"userid,omitempty"`
> SubscriptionSubscription
> `datastore:",noindex" json:"subscription,omitempty"`
> PersonalInformation []PersonalInformation`datastore:",noindex"
> json:"personalInformation,omitempty"`
> }
>
> PersonalInformation struct {
> FirstNamestring `datastore:",noindex" json:"firstName,omitempty"`
> LastName string `datastore:",noindex" json:"lastName,omitempty"`
> StreetAdress string `datastore:",noindex" json:"streetAdress,omitempty"`
> Province string `datastore:",noindex" json:"province,omitempty"`
> PostalCode string `datastore:",noindex" json:"postalCode,omitempty"`
> CountryCode string `datastore:",noindex" json:"countryCode,omitempty"`
> }
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> When I look at the new console, all values from PersonalInformation are
> marked indexed and greyed out.
> It seems to only happen with Arrays, because the nested type Subscription
> isn't an array and is correctly marked as non indexed.
>
> https://cloud.google.com/appengine/docs/go/datastore/reference?hl=en
> If an outer struct is tagged "noindex" then all of its implicit flattened
> fields are effectively "noindex".
>
> So is the console incorrectly marking these nested array values as indexed.
> or
> Is the datastore itself marking these as indexed correctly and I can't
> find this specific behaviour in the documentation?
>
> -Not related to this problem-
> Thank you for adding the 'Refresh' button!
> Thank you for showing what entities derive from if they have an ancestor!
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/be5e5cfa-b72d-40d6-8de2-40d36ec00b7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: datastore timeouts?
yes, we're seeing many timeouts of the same sort... across multiple
libraries. we've submitted a ticket
Timeout: The datastore operation timed out, or the data was temporarily
unavailable
DeadlineExceededError: The API call datastore_v3.Delete() took too long to
respond and was cancelled
BadRequestError: The referenced transaction has expired or is no longer
valid
DeadlineExceededError: The API call datastore_v3.RunQuery() took too long
to respond and was cancelled
DeadlineExceededError: The API call search.ListDocuments() took too long to
respond and was cancelled
DeadlineExceededError(The API call datastore_v3.AllocateIds() took too long
to respond and was cancelled
On Monday, December 28, 2015 at 6:37:42 PM UTC-5, Adam Sah wrote:
>
> anyone else seeing increased datastore timeouts? I've confirmed that
> traffic levels are normal i.e. not getting DOS'd...
>
> E 2015-12-28 15:19:09.728 error:Timeout('The datastore operation timed
> out, or the data was temporarily unavailable.',)
> ...
> E 2015-12-28 15:08:31.447 error:TransactionFailedError('too much
> contention on these datastore entities. please try again.',)
>
> adam
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/67167e70-e98d-451d-816d-f71af986c77b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: datastore timeouts?
weird - the datastore timeouts went away, but then came back this morning.
nobody else is seeing this?
adam
On Tuesday, December 29, 2015 at 10:37:42 AM UTC+11, Adam Sah wrote:
>
> anyone else seeing increased datastore timeouts? I've confirmed that
> traffic levels are normal i.e. not getting DOS'd...
>
> E 2015-12-28 15:19:09.728 error:Timeout('The datastore operation timed
> out, or the data was temporarily unavailable.',)
> ...
> E 2015-12-28 15:08:31.447 error:TransactionFailedError('too much
> contention on these datastore entities. please try again.',)
>
> adam
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/78a6c2d8-f338-4311-be00-87feb94d394f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[google-appengine] Re: I do not have sufficient permissions to view Error log?
Is it possible that you're attempting to view the logs section of a project for which you aren't haven't been added as an administrator with VIEW privileges? Did you create the project? On Tuesday, December 29, 2015 at 6:49:46 AM UTC-5, Restive CloudDev wrote: > > > When I navigate to the errors page ( > https://console.developers.google.com/errors), I get an error saying that > "I do not have sufficient permission to view this page". > > I have billing enabled, and I haven't logged in to the same browser with > multiple accounts, just one. > > What could be causing this? > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ab0c9292-1ddc-4b9e-a677-871c3b1d6321%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] App Engine Java log messages missing
Hello list, After finally getting my "Hello World" web app working in the App Engine development server I've uploaded it to the cloud and immediately struck problems. The main problem is that I'm not seeing my java.util.logging.Logger log messages (from either my Java or Lisp code) in the App Engine console. I've set ".level=FINER" in my logging.properties file but I don't see any of the messages I see using the development server, apart from a single INFO-level message saying "Startup completed in -0.001 seconds.", sent to stdout. The other message I do see is an exception from my code, which suggests that either the servlet's init method hasn't been called or it couldn't read the context-param values from the web.xml file (via GenericServlet.getInitParameter()). I wondered whether I should've used system-properties in my appengine-web.xml file instead of context-param elements in web.xml; I'm now checking both, to no apparent effect. But I could debug this if I had my log messages. In the same log message as the exception there is also the text "javax.servlet.ServletContext log: unavailable" which must be a clue, but Google doesn't throw up any obvious leads for it. I have added a dirt-simple "Hello World" servlet to my application that calls log.severe() and log.info() to try and get any log messages to appear in the Google Cloud Platform Console Log Viewer (or the legacy AppEngine Log Viewer). On running this I don't get the "log: unavailable" message but I don't get any log messages either. I can't help the feeling I'm missing something obvious; as I said, this all works fine in the development server (I'm using Java 7 on OS X 10.9). Where do people go to see their Java log messages in the Google Cloud Platform Console? Cheers, John :^P -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/1ab8b286-40d1-45fa-8635-a15bbc30f7e3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[google-appengine] I do not have sufficient permissions to view Error log?
When I navigate to the errors page (https://console.developers.google.com/errors), I get an error saying that "I do not have sufficient permission to view this page". I have billing enabled, and I haven't logged in to the same browser with multiple accounts, just one. What could be causing this? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9947923a-3d55-4ada-ae06-c0f459291bd2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
