[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Aaron Boodman]

On Wed, Oct 28, 2009 at 11:39 AM, Tim Steele  wrote:
> The update{foo} update{blech} case is most likely a different kind of
> failure, though, and I was thinking we could limit that with a generic cap
> on just the "number of updates in a period of time". From the data we have
> seen so far, most common is 'update{foo}' in a loop type behavior. But if
> the generic cap gets us far enough along, then I agree simpler is better.

Cool. I hope and suspect we can get far enough with the simple approach.

> We have this. The problem is that the user doesn't even realize the
> extension is spamming the server. Our server knows the client is producing a
> lot of traffic, that's all. So what happens is we limit the client and the
> user is left bewildered and helpless because a rogue extension is eating
> away his quota.  What I just landed was a way to correlate how many of our
> sync commits originate from extensions, but we need to find a way to solve
> the problem once we learn from the data.  The reason I suggested this, is
> because it dawned on me that this problem affects any extension author
> trying to send updates from Chrome to their servers. If extension Bob and
> Eve are installed, Eve can bring down Bob's service because of a silly bug.
>  I was proposing we just add some layer of protection in here, because we
> can, to help our extensions developers out (and Chrome Sync in the process).

Thanks for explaining. Makes sense to me.

- a

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Tim Steele]

>
>
>
> Ugh!
>
> I don't think there is going to be a way to make it impossible to
> write poorly written extensions. Perhaps sync should have a way to
> detect lots of mutations from an extension and then disable either the
> extension or itself at some point with a suitable warning. It should
> certainly be possible to track number of mutations from an extension
> and to know which extension is the result of the mutations.


Yes, I just added ExtensionsActivityMonitor to get this kind of data.  Our
quickfix idea is precisely to do this and kill either the extension or
Chrome Sync if we detect that things get out of hand.  But this feels like a
problem that could be nipped at a lower level than Chrome Sync. If we have
this kind of data available, we could make it available to the developers by
reporting quota-limited extensions so they can realize that their extension
is acting wrongly. Maybe they already know and/or won't do anything about
it.  Either way, we will likely need to disable that extension *anyway *if
it is bringing down *our* service. So if we have to solve this problem for
"us", I don't see a great reason not to offer the help to "them" (e.g if
Chrome Sync isn't installed) by reporting the data back and choking the
traffic at its origin, instead of falling back and relying on our sync
servers to differentiate traffic from different parts of the client (which
would have to be done at the point when our actual Chrome sync protocol
messages are being processed, which means all the server infrastructure
treats these requests as "valid" until quite late in the chain eating up
resources along the way).

Granted, I'm not an extensions developer, but I have a hard time believing
this wouldn't be a useful and friendly feature to offer.

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Tim Steele]

On Tue, Oct 27, 2009 at 11:00 PM, Aaron Boodman  wrote:

> On Tue, Oct 27, 2009 at 8:55 PM, Tim Steele  wrote:
> > I can take a stab at more formal heuristics for bookmarks, at least.  We
> > will have a better idea of actual limiting parameters for bookmarks (as
> in
> > how many operations in a certain time frame is reasonable) once the
> > ExtensionsActivityMonitor I just landed percolates and we can aggregate
> > across representative sample data it produces.
>
> A couple thoughts:
>
> a) I think it is overly clever to hash the changes to the bookmarks
> and count per unique-params. This can be easily or accidentally
> defeated by just doing something like update({title:"foo"}),
> update({url:"blech"}), over and over, anyway. Instead, at least for
> bookmarks, I think a simple per-item-count is very reasonable. It
> doesn't make sense to me to update the same bookmark more than a few
> times per minute. An easy heuristic could be that updating the same
> bookmark more than twice a minute sustained over 10 minutes. For
> creates it's a bit tricker. In that case, maybe the best we can do is
> the same combination of properties.
>

The update{foo} update{blech} case is most likely a different kind of
failure, though, and I was thinking we could limit that with a generic cap
on just the "number of updates in a period of time". From the data we have
seen so far, most common is 'update{foo}' in a loop type behavior. But if
the generic cap gets us far enough along, then I agree simpler is better.

>
> b) What is wrong with solving this by rate limiting the communication
> between the Chrome sync client and the Chrome sync server? It seems
> like that would be more foolproof; if the client tries to send too
> many updates for any reason, they are rate limited.
>

We have this. The problem is that the user doesn't even realize the
extension is spamming the server. Our server knows the client is producing a
lot of traffic, that's all. So what happens is we limit the client and the
user is left bewildered and helpless because a rogue extension is eating
away his quota.  What I just landed was a way to correlate how many of our
sync commits originate from extensions, but we need to find a way to solve
the problem once we learn from the data.  The reason I suggested this, is
because it dawned on me that this problem affects any extension author
trying to send updates from Chrome to their servers. If extension Bob and
Eve are installed, Eve can bring down Bob's service because of a silly bug.
 I was proposing we just add some layer of protection in here, because we
can, to help our extensions developers out (and Chrome Sync in the process).

>
> - a
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Scott Violet]

On Wed, Oct 28, 2009 at 10:42 AM, Nick Carter  wrote:
> On Wed, Oct 28, 2009 at 8:46 AM, Scott Violet  wrote:
>>
>> I'm confused by the diagram. In step 5, why does F' get added to the
>> model. Are you saying the 'extension cloud' service always creates a
>> new bookmark, without verifying if the model already has a matching
>> entry?
>
> This is indeed the behavior we're seeing with one extension -- "delete all
> old bookmarks, write a whole new copy".

Ugh!

I don't think there is going to be a way to make it impossible to
write poorly written extensions. Perhaps sync should have a way to
detect lots of mutations from an extension and then disable either the
extension or itself at some point with a suitable warning. It should
certainly be possible to track number of mutations from an extension
and to know which extension is the result of the mutations.

  -Scott


>  - nick
>
>>
>>  -Scott
>>
>> On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele  wrote:
>> > [re-sending from correct email account]
>> > Hi,
>> > I wrote up a document that discusses some interesting unintentional
>> > relationships that can exist between independent extensions, and how
>> > this
>> > general problem also currently affects the browser sync engine.  This
>> > issue
>> > was discovered from trying to explain the primary symptom of unusually
>> > high
>> > syncing traffic generated by Chrome clients.  Please find it here:
>> > A Tale of Two (or more) Sync Engines
>> > You should read that before continuing!
>> > This led to me thinking about what we do long term, short term, or
>> > basically
>> > before Chrome Sync and extensions are running in parallel in a beta
>> > channel
>> > environment. You'll see a bit of this at the end of the first document,
>> > but
>> > after posing the problem as an extensions problem I ended up at a random
>> > idea that I think makes at least a little sense, though I admit I was
>> > having
>> > fun thinking and writing about it so maybe I missed some major
>> > roadblocks
>> > along the way.  There are downsides, mainly revolving around the added
>> > hand-holding we would impose on extensions.  Please read! Hoping for
>> > comments and feedback. Extensions API "quotaserver"
>> > In addition to that, Colin and Todd (cc'ed) brought up some sync
>> > specific
>> > ideas they have (I mention it a bit at the end of the first doc).  We'll
>> > try
>> > to get a separate thread going about this soon!
>> > Thanks!
>> > Tim
>> > >
>> >
>>
>> >>
>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Aaron Boodman]

On Wed, Oct 28, 2009 at 10:42 AM, Nick Carter  wrote:
> On Wed, Oct 28, 2009 at 8:46 AM, Scott Violet  wrote:
>>
>> I'm confused by the diagram. In step 5, why does F' get added to the
>> model. Are you saying the 'extension cloud' service always creates a
>> new bookmark, without verifying if the model already has a matching
>> entry?
>
> This is indeed the behavior we're seeing with one extension -- "delete all
> old bookmarks, write a whole new copy".

One thing I forgot I wanted to point out about that extension in
particular. In Tim's proposal he says:

"...in Chrome Sync, we try really hard to not get into feedback loops
with ourselves, and not to make unnecessary / no-op changes, but we
have seen at least one extension (GBX) periodically delete all
bookmarks and then recreate all bookmarks with no changes to the nodes
in question. That seems almost positively unnecessary.  If we detect
that they have removed the same tree (from the same root position) 10
times in one day, this does really seem fishy and enough to trigger
limiting.  But at that point the onus is on extension developers, not
us."

It sounds like Tim agrees here, but I just want to reiterate: we
should not be the arbiters of quality or taste wrt extensions. There
will be buggy and crappy extensions. We should try and design things
such that you have to go out of your way to write crappy things, but
at the end of the day we want this to be an open environment where
people surprise us -- both in good and bad ways :).

If we need to protect our own systems that is fine, but other than
that we shouldn't prevent things in the extensions system, even if
they don't seem to make much sense.

-a

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Nick Carter]

On Wed, Oct 28, 2009 at 8:46 AM, Scott Violet  wrote:

>
> I'm confused by the diagram. In step 5, why does F' get added to the
> model. Are you saying the 'extension cloud' service always creates a
> new bookmark, without verifying if the model already has a matching
> entry?


This is indeed the behavior we're seeing with one
extension -- "delete
all old bookmarks, write a whole new copy".

 - nick


>
>  -Scott
>
> On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele  wrote:
> > [re-sending from correct email account]
> > Hi,
> > I wrote up a document that discusses some interesting unintentional
> > relationships that can exist between independent extensions, and how this
> > general problem also currently affects the browser sync engine.  This
> issue
> > was discovered from trying to explain the primary symptom of unusually
> high
> > syncing traffic generated by Chrome clients.  Please find it here:
> > A Tale of Two (or more) Sync Engines
> > You should read that before continuing!
> > This led to me thinking about what we do long term, short term, or
> basically
> > before Chrome Sync and extensions are running in parallel in a beta
> channel
> > environment. You'll see a bit of this at the end of the first document,
> but
> > after posing the problem as an extensions problem I ended up at a random
> > idea that I think makes at least a little sense, though I admit I was
> having
> > fun thinking and writing about it so maybe I missed some major roadblocks
> > along the way.  There are downsides, mainly revolving around the added
> > hand-holding we would impose on extensions.  Please read! Hoping for
> > comments and feedback. Extensions API "quotaserver"
> > In addition to that, Colin and Todd (cc'ed) brought up some sync specific
> > ideas they have (I mention it a bit at the end of the first doc).  We'll
> try
> > to get a separate thread going about this soon!
> > Thanks!
> > Tim
> > >
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Scott Violet]

I'm confused by the diagram. In step 5, why does F' get added to the
model. Are you saying the 'extension cloud' service always creates a
new bookmark, without verifying if the model already has a matching
entry?

  -Scott

On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele  wrote:
> [re-sending from correct email account]
> Hi,
> I wrote up a document that discusses some interesting unintentional
> relationships that can exist between independent extensions, and how this
> general problem also currently affects the browser sync engine.  This issue
> was discovered from trying to explain the primary symptom of unusually high
> syncing traffic generated by Chrome clients.  Please find it here:
> A Tale of Two (or more) Sync Engines
> You should read that before continuing!
> This led to me thinking about what we do long term, short term, or basically
> before Chrome Sync and extensions are running in parallel in a beta channel
> environment. You'll see a bit of this at the end of the first document, but
> after posing the problem as an extensions problem I ended up at a random
> idea that I think makes at least a little sense, though I admit I was having
> fun thinking and writing about it so maybe I missed some major roadblocks
> along the way.  There are downsides, mainly revolving around the added
> hand-holding we would impose on extensions.  Please read! Hoping for
> comments and feedback. Extensions API "quotaserver"
> In addition to that, Colin and Todd (cc'ed) brought up some sync specific
> ideas they have (I mention it a bit at the end of the first doc).  We'll try
> to get a separate thread going about this soon!
> Thanks!
> Tim
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Aaron Boodman]

On Tue, Oct 27, 2009 at 8:55 PM, Tim Steele  wrote:
> I can take a stab at more formal heuristics for bookmarks, at least.  We
> will have a better idea of actual limiting parameters for bookmarks (as in
> how many operations in a certain time frame is reasonable) once the
> ExtensionsActivityMonitor I just landed percolates and we can aggregate
> across representative sample data it produces.

A couple thoughts:

a) I think it is overly clever to hash the changes to the bookmarks
and count per unique-params. This can be easily or accidentally
defeated by just doing something like update({title:"foo"}),
update({url:"blech"}), over and over, anyway. Instead, at least for
bookmarks, I think a simple per-item-count is very reasonable. It
doesn't make sense to me to update the same bookmark more than a few
times per minute. An easy heuristic could be that updating the same
bookmark more than twice a minute sustained over 10 minutes. For
creates it's a bit tricker. In that case, maybe the best we can do is
the same combination of properties.

b) What is wrong with solving this by rate limiting the communication
between the Chrome sync client and the Chrome sync server? It seems
like that would be more foolproof; if the client tries to send too
many updates for any reason, they are rate limited.

- a

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Tim Steele]

I can take a stab at more formal heuristics for bookmarks, at least.  We
will have a better idea of actual limiting parameters for bookmarks (as in
how many operations in a certain time frame is reasonable) once the
ExtensionsActivityMonitor I just landed percolates and we can aggregate
across representative sample data it produces.

I guess the docs were too long for more folks to read through :)

For those who have read, what are thoughts on the proposal to maintain a
counter per-function, and per-item-id in some cases? So for detecting the
situation

bookmarks.update(123, "title:Foo")
bookmarks.update(123, "title:Foo")
bookmarks.update(123, "title:Foo")
... 17 times for example

we would have a map somewhere with one entry for each bookmarks function,
and the 'update' entry would look like

( "bookmarks.update", { (123, hash, 20), ...} )

maintaining a history of length *at most* 1 for at most every bookmark.  If
an 'update' comes in for '123' that tries to update it in some *other* way,
the hash would be different so we'd reset the count and allow the operation.
 We could also either store a timestamp or use a timer task to
discard records that were 'old enough'.

For bookmarks.create, store (hash, count), and limit quota to a
count of 10 per hour (mostly arbitrary numbers right now).
Same for bookmarks.remove.  For starters, out of simplicity I wouldn't try
and associate removals with adds or do any cross-referencing to detect
things like 'you just added and removed the same item 100 times in a
minute', but hope that 'you just added the same item 100 times in a minute'
+ 'you just removed the same item 100 times in a minute' signals will help a
little on their own, independently from each other.

Do these seem like they would eat up too much memory?

As for requesting such a quota limitation be turned on, what are thoughts on
my proposal to request it in the manifest/permissions block? Is that doable?
We would need a way for browser/sync to toggle it too, but that's likely
just an implementation detail.

On Wed, Oct 21, 2009 at 5:55 PM, Aaron Boodman  wrote:

> Have not read all of it yet, but am in favor in general of having some
> sort of quota for extension api calls to protect from meltdown.
>
> We have a very convenient chokepoint to implement this in our system,
> so it would just be a matter of a more detailed design of the
> heuristics to use for each API type.
>
> - a
>
> On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele  wrote:
> > [re-sending from correct email account]
> > Hi,
> > I wrote up a document that discusses some interesting unintentional
> > relationships that can exist between independent extensions, and how this
> > general problem also currently affects the browser sync engine.  This
> issue
> > was discovered from trying to explain the primary symptom of unusually
> high
> > syncing traffic generated by Chrome clients.  Please find it here:
> > A Tale of Two (or more) Sync Engines
> > You should read that before continuing!
> > This led to me thinking about what we do long term, short term, or
> basically
> > before Chrome Sync and extensions are running in parallel in a beta
> channel
> > environment. You'll see a bit of this at the end of the first document,
> but
> > after posing the problem as an extensions problem I ended up at a random
> > idea that I think makes at least a little sense, though I admit I was
> having
> > fun thinking and writing about it so maybe I missed some major roadblocks
> > along the way.  There are downsides, mainly revolving around the added
> > hand-holding we would impose on extensions.  Please read! Hoping for
> > comments and feedback. Extensions API "quotaserver"
> > In addition to that, Colin and Todd (cc'ed) brought up some sync specific
> > ideas they have (I mention it a bit at the end of the first doc).  We'll
> try
> > to get a separate thread going about this soon!
> > Thanks!
> > Tim
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: A tale of two (or more) syncing extensions, and a proposal

[Aaron Boodman]

Have not read all of it yet, but am in favor in general of having some
sort of quota for extension api calls to protect from meltdown.

We have a very convenient chokepoint to implement this in our system,
so it would just be a matter of a more detailed design of the
heuristics to use for each API type.

- a

On Wed, Oct 21, 2009 at 5:08 PM, Tim Steele  wrote:
> [re-sending from correct email account]
> Hi,
> I wrote up a document that discusses some interesting unintentional
> relationships that can exist between independent extensions, and how this
> general problem also currently affects the browser sync engine.  This issue
> was discovered from trying to explain the primary symptom of unusually high
> syncing traffic generated by Chrome clients.  Please find it here:
> A Tale of Two (or more) Sync Engines
> You should read that before continuing!
> This led to me thinking about what we do long term, short term, or basically
> before Chrome Sync and extensions are running in parallel in a beta channel
> environment. You'll see a bit of this at the end of the first document, but
> after posing the problem as an extensions problem I ended up at a random
> idea that I think makes at least a little sense, though I admit I was having
> fun thinking and writing about it so maybe I missed some major roadblocks
> along the way.  There are downsides, mainly revolving around the added
> hand-holding we would impose on extensions.  Please read! Hoping for
> comments and feedback. Extensions API "quotaserver"
> In addition to that, Colin and Todd (cc'ed) brought up some sync specific
> ideas they have (I mention it a bit at the end of the first doc).  We'll try
> to get a separate thread going about this soon!
> Thanks!
> Tim
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---