Re: Background Jobs For Mission Critical Info

[Carson Gross]

Richard & Keenan,

Thanks for the feedback.  This is going to involve lots of relatively small 
jobs, and Resque looks like it might be a better option to get the db out 
of the request cycle as much as possible.

For now I think I'll extract the logic out into a logical work unit that's 
both DJ and Resque friendly, and then play around with both of 
them/get familiar w/ the internals and see how it goes.  I'm still torn on 
whether the added complexity is worth it, or if I should just pitch the 
idea and simply pay for more workers if life gets busy.

Thanks,
Carson

On Thursday, September 27, 2012 9:05:57 AM UTC-7, Keenan wrote:
>
> Hi Carson,
>
> You may also want to look at https://github.com/ryandotsmith/queue_classic
> Ryan is also at Heroku. And designed this with the heroku infrastructure 
> in mind.
>
> I have not used queue classic, but have had good experience with DJ and 
> Resque. (not a big fan of backgroundrb)
>
> --Keenan
>
>  On Thursday, September 27, 2012 at 11:55 AM, Richard Schneeman wrote:
>
>  The maintainer of Resque (Terence Lee <http://twitter.com/hone02>) works 
> for Heroku. Failed Resque jobs can be retried via the built in web 
> interface. I've never had a bad experience with Delayed Job, but if you are 
> using it heavily it can impact your database performance. 
>
> -- 
> Richard Schneeman
> http://heroku.com
> @schneems <http://twitter.com/schneems>
>
> On Wednesday, September 26, 2012 at 9:21 PM, Carson Gross wrote:
>
> Hi,
>
> I've got a bit of processing that is not time critical but that is mission 
> critical, and I'm considering running as a Delayed Job in order to minimize 
> request times.  My experience with Delayed Jobs on bamboo was that there 
> were times when the queue just hung, and I had to delete the jobs to get 
> things running again.  I won't be able to just drop these jobs on the floor 
> like I could on our bamboo app, so I was wondering if anyone can comment on 
> the reliability of DJ's on the cedar stack, if they've had problems w/ 
> things hanging, etc.
>
> Thanks for any comments or opinions,
> Carson
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Heroku" group.
>  
> To unsubscribe from this group, send email to
> heroku+un...@googlegroups.com 
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en_US?hl=en
>  
>
>  -- 
> You received this message because you are subscribed to the Google
> Groups "Heroku" group.
>  
> To unsubscribe from this group, send email to
> heroku+un...@googlegroups.com 
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en_US?hl=en
>  
>  
>  

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Background Jobs For Mission Critical Info

[Carson Gross]

Hi,

I've got a bit of processing that is not time critical but that is mission 
critical, and I'm considering running as a Delayed Job in order to minimize 
request times.  My experience with Delayed Jobs on bamboo was that there 
were times when the queue just hung, and I had to delete the jobs to get 
things running again.  I won't be able to just drop these jobs on the floor 
like I could on our bamboo app, so I was wondering if anyone can comment on 
the reliability of DJ's on the cedar stack, if they've had problems w/ 
things hanging, etc.

Thanks for any comments or opinions,
Carson

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Re: Multiple apps/Same DB

[Carson Gross]

Makes sense, I can hack that.

I think a potentially good solution would be to allow sub-domain isolation 
of web workers for an app.  It's a pretty common pattern to split out your 
app along sub-domains (e.g. api.foo.com vs app.foo.com) which have 
different access patterns/performance requirements and need perf isolation 
from one another.  Of course, the devil is in the details and maybe that 
ends up being a terrible idea to implement, but it's exactly what I'd like 
right now.  :)

Thanks,
Carson

On Friday, August 24, 2012 4:55:09 PM UTC-7, Peter van Hardenberg wrote:
>
> We do kinda-sorta support it, but it currently has a limitation in the 
> form that if we ever need to update your DATABASE_URL, we can't tell that 
> you're connecting from two apps and one of them (the non-owning app) will 
> go unupdated. We've got a feature in alpha right now which will resolve 
> this, but I don't believe it's ready for external users yet.
>
> In conclusion: go ahead, but it's grody, and we're hoping it will be 
> better soon.
>
> -p
>
> On Fri, Aug 24, 2012 at 3:27 PM, Carson Gross 
> > wrote:
>
>> Hey Everyone,
>>
>> I'm going to be running two heroku apps with the same backing database (I 
>> want request isolation between the API and the web application so that if I 
>> screw the web application up from a perf perspective it won't back up the 
>> API.)
>>
>> It seems pretty simple to do this:
>>
>>   
>> http://stackoverflow.com/questions/5981508/share-database-between-2-apps-in-heroku
>>
>> and it seems like Heroku kinda-sorta considers it a supported 
>> configuration.
>>
>> Anyone have comments on and/or experience with it?
>>
>> Thanks,
>> Carson
>>  
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Heroku" group.
>>  
>> To unsubscribe from this group, send email to
>> heroku+un...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/heroku?hl=en_US?hl=en
>>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Multiple apps/Same DB

[Carson Gross]

Hey Everyone,

I'm going to be running two heroku apps with the same backing database (I 
want request isolation between the API and the web application so that if I 
screw the web application up from a perf perspective it won't back up the 
API.)

It seems pretty simple to do this:

  
http://stackoverflow.com/questions/5981508/share-database-between-2-apps-in-heroku

and it seems like Heroku kinda-sorta considers it a supported configuration.

Anyone have comments on and/or experience with it?

Thanks,
Carson

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Re: Multiple differently-sized memcache caches

[Carson Gross]

Hey Richard,

I think the way I'm going to solve it is to split the functionality across 
two heroku app instances.  I want request isolation between this sub-system 
(which should have lots of very fast, small requests) and the main website 
(which may have longer running requests until I chase down all the perf 
issues) anyway, so they can both rely on the normal rails cache.

Looks like sharing a db between two apps is an exotic-but-supported 
configuration for you guys now, right?

Cheers,
Carson

On Saturday, August 18, 2012 7:52:53 PM UTC-7, richard schneeman wrote:
>
> Honestly i've never seen this done before, so I can't be much help. Have 
> you tried asking on StackOverflow? It's an interesting idea, I would like 
> to hear how to do it, as well as if it was worth the time/effort in the 
> long run. If someone else on the list has tried and has suggestions, please 
> let us know. 
>
>
> Depending on your situation you might consider putting at TTL (expires in) 
> on your large data, so that it will expire by itself rather than relying on 
> memcache to reclaim memory, this might decrease the amount of items you 
> care about falling out of cache. 
>
> -- 
> Richard Schneeman
> http://heroku.com
> @schneems <http://twitter.com/schneems>
>
> On Saturday, August 18, 2012 at 11:52 AM, Carson Gross wrote:
>
> Hi,
>
> Googling around didn't turn up an answer I was 100% confident in, so I'm 
> hoping someone can help me out.
>
> I'd like to set up a few different caches in my app with different sizes. 
>  In particular I've got one stream of data that is fairly large and I want 
> to speculatively cache, but that becomes useless very quickly and so I want 
> to discard it quickly if it is not used.  I don't want this data polluting 
> the main Rails.cache.
>
> From googline, it looks like creating multiple 
> ActiveSupport::Cache::MemoryStore's 
> is what I want to do, but I want to make sure I do it in a way that uses 
> Dalli and the heroku memcache plugin properly, and ensure that I split and 
> isolate the memcaches from one another properly.
>
> Anyone done something like this before?
>
> Thanks,
> Carson 
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Heroku" group.
>  
> To unsubscribe from this group, send email to
> heroku+un...@googlegroups.com 
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en_US?hl=en
>  
>  
>  

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Multiple differently-sized memcache caches

[Carson Gross]

Hi,

Googling around didn't turn up an answer I was 100% confident in, so I'm 
hoping someone can help me out.

I'd like to set up a few different caches in my app with different sizes. 
 In particular I've got one stream of data that is fairly large and I want 
to speculatively cache, but that becomes useless very quickly and so I want 
to discard it quickly if it is not used.  I don't want this data polluting 
the main Rails.cache.

>From googline, it looks like creating multiple 
>ActiveSupport::Cache::MemoryStore's 
is what I want to do, but I want to make sure I do it in a way that uses 
Dalli and the heroku memcache plugin properly, and ensure that I split and 
isolate the memcaches from one another properly.

Anyone done something like this before?

Thanks,
Carson 

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Adding pessimistic locking, need advice

[Carson Gross]

Hi All,

I apologize in advance if this isn't on topic, but I've had success with 
general rails questions here before, so here goes:

We are going to add some pessimistic locking to a DelayedJob that we are 
running.  The code appears easy enough (this is rails2):

  User.transation do
user = User.find( id, :lock => true )

# some tragically complicated code

  end

The code that will be run within the transaction is non-trivial.  I'm going 
to try to shave it down to the minimal amount but, unfortunately, we need 
to do a read early on to establish some data for a series of calculations, 
so it isn't possible to make the code totally trivial.

Here are my questions:


   - Will the transaction (and row lock) unroll if an exception occurs (I 
   have to assume so, given the API)
   - If all hell breaks loose and we manage to lock a row and not unlock it 
   (e.g. an infinite loop in the DelayedJob body) would my only option to be 
   to restart the database?  What's the command to do that on heroku?
   - Is there any other way I could leave the row locked beyond an infinite 
   loop in the code within the transaction?  My hope is that the transaction 
   API guarantees the lock is released once you exit the block
   - What questions *should* I be asking? 

Thanks for any help and advice,
Carson

-- 
You received this message because you are subscribed to the Google
Groups "Heroku" group.

To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en_US?hl=en

Re: Rails parameter question...

[Carson Gross]

Thanks guys, this is very promising.  I think I've almost got a
middleware app going.  I've got the following class:

class CustomHandler

  def initialize(app)
puts "Initialized..."
@app = app
  end

  def call(env)
if env["PATH_INFO"] =~ /\/bad_path\//
  [201, {"Content-Type" => "text/html"}, "Ingored"]
end
@app.call(env)
  end

end

And I add the middleware like so in environment.rb:

  config.middleware.insert_before(ActionController::ParamsParser,
CustomHandler )

However, at runtime I get this complaint:

  Status: 500 Internal Server Error
  undefined method `call' for CustomHandler:Class

So, obviously it is trying to invoke 'call' on the class, rather than
an instance of the class.  What's confusing me is that I had to add
that constructor that takes the app to the class, or it failed during
the insert_before(), so I assumed that rack was reflectively creating
an instance.

The code above, structurally, is identical to the ParamsParser code: a
single-arg constructor and a non-static call method, so I must be
doing something wrong in the insert_before code.  Do I need to pass an
instance, rather than the class in?  If so, how can I get at the app?

Thanks for all your help everyone,
Carson

On Mar 17, 5:09 pm, Dan Croak  wrote:
> Peeking into the Rails 2.3.8, params parsing is handled by
> ActionContoller::ParamsParser, which is located in
> actionpack/lib/action_controller/params_parser.rb:
>
> https://github.com/rails/rails/blob/240f4e944cd90fca138aba84674560439...
>
> ParamsParser is a Rack app which is autoloaded inside the
> ActionController module:
>
> https://github.com/rails/rails/blob/240f4e944cd90fca138aba84674560439...
>
> So, I think you want a Rack app as Jeff suggested:
>
>  class CustomXmlHandler
>     def self.call(env)
>       if env["PATH_INFO"] =~ /^\/big-xml-requests/
>         DatabaseObject.create :xml => request.raw_post
>         [201, {"Content-Type" => "text/html"}, "XML saved"]
>       else
>         [404, {"Content-Type" => "text/html"}, "Not Found"]
>       end
>     end
>   end
>
> Then, insert it as middleware before the built-in Rails params parser
> in config/environment.rb:
>
> configuration.middleware.insert_before ActionContoller::ParamsParser,
> CustomXmlHandler
>
> On Sat, Mar 17, 2012 at 7:26 PM, Jeff Schmitz
>
>
>
>
>
>
>
>  wrote:
> > Can you do this part as a rack app?
>
> > Jeff
>
> > On Mar 17, 2012, at 4:32 PM, Carson  Gross  wrote:
>
> >> According to my tests, yes, the parameter parsing happens before the
> >> before_filters are run.
>
> >> Thanks,
> >> Carson
>
> >> On Mar 17, 11:15 am, Jeff Schmitz  wrote:
> >>> Does Rails parse the body prior to any before_filters run?
>
> >>> Jeff
>
> >>> On Mar 17, 2012, at 12:57 PM, Carson  Gross  wrote:
>
> >>>> Hey Guys,
>
> >>>> This is a rails question, so please ignore if that doesn't interest
> >>>> you.
>
> >>>> Does anyone know if there is a way to make rails *not* parse the body
> >>>> of an XML post to a given controller method?  I've been googling for
> >>>> an hour now and can't figure it out.  Basically we've got some huge
> >>>> XML docs coming in and we'd like to just jam them into the database as
> >>>> text and then process them in the background using nokogiri.
>
> >>>> Unfortunately, rails is still parsing them using REXML before they
> >>>> ever hit our controller which I just realized after a week of pulling
> >>>> my hair out: the perf issue doesn't show up in New Relic because all
> >>>> the processing time is *before* the request.
>
> >>>> We could switch the body to something like CSV, but I'm hoping there
> >>>> is a way to tell rails to just not bother parsing the body, so we can
> >>>> just grab it with:
>
> >>>>  request.body.read
>
> >>>> This is rails 2.3.8.
>
> >>>> Thanks,
> >>>> Carson
>
> >>>> --
> >>>> You received this message because you are subscribed to the Google 
> >>>> Groups "Heroku" group.
> >>>> To post to this group, send email to heroku@googlegroups.com.
> >>>> To unsubscribe from this group, send email to 
> >>>> heroku+unsubscr...@googlegroups.com.
> >>>> For more options, visit this group 
&g

Re: Rails parameter question...

[Carson Gross]

According to my tests, yes, the parameter parsing happens before the
before_filters are run.

Thanks,
Carson

On Mar 17, 11:15 am, Jeff Schmitz  wrote:
> Does Rails parse the body prior to any before_filters run?
>
> Jeff
>
> On Mar 17, 2012, at 12:57 PM, Carson  Gross  wrote:
>
>
>
>
>
>
>
> > Hey Guys,
>
> > This is a rails question, so please ignore if that doesn't interest
> > you.
>
> > Does anyone know if there is a way to make rails *not* parse the body
> > of an XML post to a given controller method?  I've been googling for
> > an hour now and can't figure it out.  Basically we've got some huge
> > XML docs coming in and we'd like to just jam them into the database as
> > text and then process them in the background using nokogiri.
>
> > Unfortunately, rails is still parsing them using REXML before they
> > ever hit our controller which I just realized after a week of pulling
> > my hair out: the perf issue doesn't show up in New Relic because all
> > the processing time is *before* the request.
>
> > We could switch the body to something like CSV, but I'm hoping there
> > is a way to tell rails to just not bother parsing the body, so we can
> > just grab it with:
>
> >  request.body.read
>
> > This is rails 2.3.8.
>
> > Thanks,
> > Carson
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > heroku+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Rails parameter question...

[Carson Gross]

Hey Guys,

This is a rails question, so please ignore if that doesn't interest
you.

Does anyone know if there is a way to make rails *not* parse the body
of an XML post to a given controller method?  I've been googling for
an hour now and can't figure it out.  Basically we've got some huge
XML docs coming in and we'd like to just jam them into the database as
text and then process them in the background using nokogiri.

Unfortunately, rails is still parsing them using REXML before they
ever hit our controller which I just realized after a week of pulling
my hair out: the perf issue doesn't show up in New Relic because all
the processing time is *before* the request.

We could switch the body to something like CSV, but I'm hoping there
is a way to tell rails to just not bother parsing the body, so we can
just grab it with:

  request.body.read

This is rails 2.3.8.

Thanks,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Sanity check: pg follow -> alter column

[Carson Gross]

Keenan,

Thanks, that makes a lot of sense.  This is a leaf table in our model,
so nothing points to it except the UI (where you can update stuff
from), so I think we could do something like this:

* As you suggest, create a new column of type bigint with an auto-
increment/sequence set up via a migration
* Do batch updates to the table, setting values from this new
sequence, as you suggest, until we hit rows that already have values
for it
* Do the rename

Thanks for the suggestion: postgres not giving me any feedback on
progress was making me sick to my stomach and this is a nice,
incremental, bail-out-able way to unfsck our db.  Beers on me next
time you are in Sacramento, California.

Cheers,
Carson

On Mar 14, 4:12 am, Keenan Brock  wrote:
> Hi Carson,
>
> Adding a column is quick
> Renaming a column is quick too.
> Updating… not so much.
>
> I would
>
> 1. create/run a migration that adds the column.
> do not add any indexes on the column yet
>
> 2. update the columns:
> start_id = connection.select_value "select min(id) from big_table where 
> new_col is null" #or 0 if you set not null/default to 0
> end_id = connection.select_value "select max(id) from big_table"
>
> BATCH_SIZE=1
> while start_id < end_id
>   connection.execute "update big_table set new_col = old_col where id >= 
> #{start_id} and id < #{start_id + BATCH_SIZE}"
>   start_id += BATCH_SIZE
> end
>
> 3. drop the old column and rename the new column to the name of the old 
> column.
> add indexes/foreign keys
> this migration will require a rails restart
>
> Play around with the batch size.
>
> There will be a sweet spot. doubling the number of rows will only increase 
> the time by a few seconds. Then all of a sudden it will take longer.
>
> Also, while you are playing with batch size, keep increasing the start_id. so 
> you are getting the work done while you are playing/experimenting.
>
> Time the execution to give you an idea of just how long it will take to 
> update every row.
> —Keenan
>
>
>
>
>
>
>
> On Wednesday, March 14, 2012 at 4:57 AM, Peter van Hardenberg wrote:
> > Hi Carson,
>
> > Unfortunately, followers are exact replicas of the master database and do 
> > not support having changes made to them, so unfortunately your proposed 
> > plan won't work.
>
> > That said, if the on-disk storage type is compatible, it should be 
> > basically instantaneous. Try it on a follower and see if it happens 
> > instantly. Otherwise, I recommend you create the new column and migrate the 
> > data with a worker process gradually. Gigantic transactions can cause pain 
> > for running databases due to a large number of non-visible rows polluting 
> > things like indices. Someone else may have more specific advice for your 
> > case.
>
> > Let us know what you discover,
> > -pvh
>
> > On Tue, Mar 13, 2012 at 9:11 PM, Carson Gross  > (mailto:carsongr...@gmail.com)> wrote:
> > > Hi Guys,
>
> > > So, we've got ourselves into a bit of a pickle, and have an integer
> > > column in a (100 million plus rows) table that really needs to be
> > > bigint.  Not this year, but certainly before we die.  I'd like to make
> > > the change sooner rather than later, and I *think* that heroku is
> > > gonna save our bacon here if the following technique could work:
>
> > > * Create a database that follows our current db
> > > * In the follower run 'ALTER TABLE my_table ALTER COLUMN id TYPE
> > > bigint'
> > > * Wait a long time
> > > * Wait for the follower to catch up  (I assume that given that we are
> > > widening the datatype the follower will work again after the upgrade)
> > > * Move the app over to the new db.
>
> > > Does this sound plausible?
>
> > > Thanks,
> > > Carson
>
> > > --
> > > You received this message because you are subscribed to the Google Groups 
> > > "Heroku" group.
> > > To post to this group, send email to heroku@googlegroups.com 
> > > (mailto:heroku@googlegroups.com).
> > > To unsubscribe from this group, send email to 
> > > heroku+unsubscr...@googlegroups.com 
> > > (mailto:heroku%2bunsubscr...@googlegroups.com).
> > > For more options, visit this group 
> > > athttp://groups.google.com/group/heroku?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com 
> > (mailto:heroku@googlegroups.com).
> > To unsubscribe from this group, send email to 
> > heroku+unsubscr...@googlegroups.com 
> > (mailto:heroku+unsubscr...@googlegroups.com).
> > For more options, visit this group 
> > athttp://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Sanity check: pg follow -> alter column

[Carson Gross]

Hi Guys,

So, we've got ourselves into a bit of a pickle, and have an integer
column in a (100 million plus rows) table that really needs to be
bigint.  Not this year, but certainly before we die.  I'd like to make
the change sooner rather than later, and I *think* that heroku is
gonna save our bacon here if the following technique could work:

* Create a database that follows our current db
* In the follower run 'ALTER TABLE my_table ALTER COLUMN id TYPE
bigint'
* Wait a long time
* Wait for the follower to catch up  (I assume that given that we are
widening the datatype the follower will work again after the upgrade)
* Move the app over to the new db.

Does this sound plausible?

Thanks,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Database CPU

[Carson Gross]

Will do, thanks Peter.

On Feb 22, 5:51 pm, Peter van Hardenberg  wrote:
> On Tue, Feb 21, 2012 at 12:51 PM, Carson  Gross  wrote:
>
> > Is there a way to get this information?
>
> Not today -- we have some customers who look at more database specific
> information, but CPU load is a lousy metric for how your database is
> performing under most circumstances and we haven't figured out
> something better to provide.
>
> We'd love to get more feedback from you about what you're looking for.
> Drop us an email to matt at heroku.com, I know he's working on this
> problem.
>
> -p

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Database CPU

[Carson Gross]

Hi,

We are using a Ronin database right now, and I'd like to get an idea
how much room we have to run on the hardware.  New Relic does a great
job of showing us wall time spent on queries and what queries to
optimize for or eliminate, but it doesn't give us an idea of what the
CPU load on our database machine is.

Is there a way to get this information?

Thanks,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Newb question on static assets

[Carson Gross]

Hi Oren,

Thanks for the links.  After reading through them, my best theory for
the reason Chrome is aggressively reloading some resources I'd prefer
that it didn't is because of the relatively recent Last-Modified
headers coming from Varnish.  Is there a way to tie the Last-Modified
header back to when we last deployed?

Thanks,
Carson

On Jan 26, 10:22 pm, Oren Teich  wrote:
> There are different types of caches - client and server side.
> Check out this presentation to learn more about what's going on:
>
> http://tomayko.com/writings/railsconf-caching-talkhttp://www.slideshare.net/rtomayko/https-bestkept-secret-cachinghttp://tomayko.com/writings/things-caches-dohttp://tomayko.com/writings/rack-cache-announce
>
> Oren
>
> On Thu, Jan 26, 2012 at 9:53 PM, Carson  Gross  wrote:
>
>
>
>
>
>
>
> > And one last note: due to the way some of these resources are included
> > (e.g. the stylesheet_link_tag method in rails) they end up with a
> > query string.
>
> > Reading online, it appears that resources with a query string are not
> > supposed to be cached, but that the major browsers ignore that.
>
> > Anyone know for sure?
>
> > Thanks,
> > Carson
>
> > On Jan 26, 9:41 pm, Carson  Gross  wrote:
> >> OK, now that I have my head around what is going on, it appears that
> >> the static stuff is being cached and served up via Varnish.
>
> >> It appears that my browser is not respecting the varish response
> >> though.  The response has the following header:
>
> >>   Cache-Control:public, max-age=43200
>
> >> and yet another request is made when I hit cmd-r.  When I click
> >> around, I still see requests for this resource, with the 304 response
> >> code.
>
> >> I'm using Chrome.
>
> >> UPDATE:
>
> >> OK, digging a bit more, I found this page:
>
> >>  http://code.google.com/speed/page-speed/docs/caching.html
>
> >> In particular, I note this bit:
>
> >> "Set the Last-Modified date to the last time the resource was changed.
> >> If the Last-Modified date is sufficiently far enough in the past,
> >> chances are the browser won't refetch it."
>
> >> And, looking more closely at the response headers, I see this:
>
> >> Cache-Control:public, max-age=43200
> >> Connection:keep-alive
> >> Date:Fri, 27 Jan 2012 05:31:46 GMT
> >> Last-Modified:Thu, 26 Jan 2012 17:14:44 GMT
> >> Server:nginx
> >> Via:1.1 varnish
> >> X-Varnish:800861986
>
> >> And note that the Last-Modified date is today, even though we haven't
> >> deployed a new version of our app for a while, which I believe may be
> >> why Chrome is issuing requests for the resources even though it isn't
> >> changing.
>
> >> When I resubmit, I see a different Last-Modified header for the same
> >> resource.
>
> >> So my guess here is that the Last-Modified header is generated by the
> >> change date of the file on a particular instance of Varnish, and that
> >> various instances are coming up and going down, causing the Last
> >> Modified date to be relatively recent, causing Chrome to submit more
> >> requests to the server than I'd like.
>
> >> Does that sound at all plausible?
>
> >> Thanks,
> >> Carson
>
> >> On Jan 26, 5:56 pm, Carson  Gross  wrote:
>
> >> > Looking at it a bit more (as well as some *yikes* requests to our
> >> > server) I see that we are getting a ton of 304s, which are, no doubt,
> >> > tying up our dynos.
>
> >> > So, this may be more of a rails question than a heroku question, but
> >> > how can I set expires headers on certain static directories?
>
> >> > Yes, I am a complete and utter newb at this.
>
> >> > Cheers,
> >> > Carson
>
> >> > On Jan 26, 5:00 pm, Carson  Gross  wrote:
>
> >> > > Heya,
>
> >> > > Sorry, digging through the docs I couldn't really get an answer on
> >> > > this that I understand:
>
> >> > > So, with static assets, deployed on the bamboo-ree-1.8.7 stack, are
> >> > > static assets served by web dynos?  As our application has grown over
> >> > > time, we've evolved to issue a horrific number of requests for static
> >> > > content (css, javascript, etc.) and I'm trying to understand just how
> >> > > bad that is for our performance.  Seems like we'd be tying up web
> >> > > dy

Re: Newb question on static assets

[Carson Gross]

And one last note: due to the way some of these resources are included
(e.g. the stylesheet_link_tag method in rails) they end up with a
query string.

Reading online, it appears that resources with a query string are not
supposed to be cached, but that the major browsers ignore that.

Anyone know for sure?

Thanks,
Carson

On Jan 26, 9:41 pm, Carson  Gross  wrote:
> OK, now that I have my head around what is going on, it appears that
> the static stuff is being cached and served up via Varnish.
>
> It appears that my browser is not respecting the varish response
> though.  The response has the following header:
>
>   Cache-Control:public, max-age=43200
>
> and yet another request is made when I hit cmd-r.  When I click
> around, I still see requests for this resource, with the 304 response
> code.
>
> I'm using Chrome.
>
> UPDATE:
>
> OK, digging a bit more, I found this page:
>
>  http://code.google.com/speed/page-speed/docs/caching.html
>
> In particular, I note this bit:
>
> "Set the Last-Modified date to the last time the resource was changed.
> If the Last-Modified date is sufficiently far enough in the past,
> chances are the browser won't refetch it."
>
> And, looking more closely at the response headers, I see this:
>
> Cache-Control:public, max-age=43200
> Connection:keep-alive
> Date:Fri, 27 Jan 2012 05:31:46 GMT
> Last-Modified:Thu, 26 Jan 2012 17:14:44 GMT
> Server:nginx
> Via:1.1 varnish
> X-Varnish:800861986
>
> And note that the Last-Modified date is today, even though we haven't
> deployed a new version of our app for a while, which I believe may be
> why Chrome is issuing requests for the resources even though it isn't
> changing.
>
> When I resubmit, I see a different Last-Modified header for the same
> resource.
>
> So my guess here is that the Last-Modified header is generated by the
> change date of the file on a particular instance of Varnish, and that
> various instances are coming up and going down, causing the Last
> Modified date to be relatively recent, causing Chrome to submit more
> requests to the server than I'd like.
>
> Does that sound at all plausible?
>
> Thanks,
> Carson
>
> On Jan 26, 5:56 pm, Carson  Gross  wrote:
>
>
>
>
>
>
>
> > Looking at it a bit more (as well as some *yikes* requests to our
> > server) I see that we are getting a ton of 304s, which are, no doubt,
> > tying up our dynos.
>
> > So, this may be more of a rails question than a heroku question, but
> > how can I set expires headers on certain static directories?
>
> > Yes, I am a complete and utter newb at this.
>
> > Cheers,
> > Carson
>
> > On Jan 26, 5:00 pm, Carson  Gross  wrote:
>
> > > Heya,
>
> > > Sorry, digging through the docs I couldn't really get an answer on
> > > this that I understand:
>
> > > So, with static assets, deployed on the bamboo-ree-1.8.7 stack, are
> > > static assets served by web dynos?  As our application has grown over
> > > time, we've evolved to issue a horrific number of requests for static
> > > content (css, javascript, etc.) and I'm trying to understand just how
> > > bad that is for our performance.  Seems like we'd be tying up web
> > > dynos, but I want to make sure.
>
> > > Secondly, given that I don't feel like rewriting all that code right
> > > now, is there a way to map certain paths to a heroku app to another
> > > server, so we aren't hammering our dynos so badly?
>
> > > Thanks,
> > > Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Newb question on static assets

[Carson Gross]

OK, now that I have my head around what is going on, it appears that
the static stuff is being cached and served up via Varnish.

It appears that my browser is not respecting the varish response
though.  The response has the following header:

  Cache-Control:public, max-age=43200

and yet another request is made when I hit cmd-r.  When I click
around, I still see requests for this resource, with the 304 response
code.

I'm using Chrome.

UPDATE:

OK, digging a bit more, I found this page:

  http://code.google.com/speed/page-speed/docs/caching.html

In particular, I note this bit:

"Set the Last-Modified date to the last time the resource was changed.
If the Last-Modified date is sufficiently far enough in the past,
chances are the browser won't refetch it."

And, looking more closely at the response headers, I see this:

Cache-Control:public, max-age=43200
Connection:keep-alive
Date:Fri, 27 Jan 2012 05:31:46 GMT
Last-Modified:Thu, 26 Jan 2012 17:14:44 GMT
Server:nginx
Via:1.1 varnish
X-Varnish:800861986

And note that the Last-Modified date is today, even though we haven't
deployed a new version of our app for a while, which I believe may be
why Chrome is issuing requests for the resources even though it isn't
changing.

When I resubmit, I see a different Last-Modified header for the same
resource.

So my guess here is that the Last-Modified header is generated by the
change date of the file on a particular instance of Varnish, and that
various instances are coming up and going down, causing the Last
Modified date to be relatively recent, causing Chrome to submit more
requests to the server than I'd like.

Does that sound at all plausible?

Thanks,
Carson

On Jan 26, 5:56 pm, Carson  Gross  wrote:
> Looking at it a bit more (as well as some *yikes* requests to our
> server) I see that we are getting a ton of 304s, which are, no doubt,
> tying up our dynos.
>
> So, this may be more of a rails question than a heroku question, but
> how can I set expires headers on certain static directories?
>
> Yes, I am a complete and utter newb at this.
>
> Cheers,
> Carson
>
> On Jan 26, 5:00 pm, Carson  Gross  wrote:
>
>
>
>
>
>
>
> > Heya,
>
> > Sorry, digging through the docs I couldn't really get an answer on
> > this that I understand:
>
> > So, with static assets, deployed on the bamboo-ree-1.8.7 stack, are
> > static assets served by web dynos?  As our application has grown over
> > time, we've evolved to issue a horrific number of requests for static
> > content (css, javascript, etc.) and I'm trying to understand just how
> > bad that is for our performance.  Seems like we'd be tying up web
> > dynos, but I want to make sure.
>
> > Secondly, given that I don't feel like rewriting all that code right
> > now, is there a way to map certain paths to a heroku app to another
> > server, so we aren't hammering our dynos so badly?
>
> > Thanks,
> > Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Newb question on static assets

[Carson Gross]

Looking at it a bit more (as well as some *yikes* requests to our
server) I see that we are getting a ton of 304s, which are, no doubt,
tying up our dynos.

So, this may be more of a rails question than a heroku question, but
how can I set expires headers on certain static directories?

Yes, I am a complete and utter newb at this.

Cheers,
Carson

On Jan 26, 5:00 pm, Carson  Gross  wrote:
> Heya,
>
> Sorry, digging through the docs I couldn't really get an answer on
> this that I understand:
>
> So, with static assets, deployed on the bamboo-ree-1.8.7 stack, are
> static assets served by web dynos?  As our application has grown over
> time, we've evolved to issue a horrific number of requests for static
> content (css, javascript, etc.) and I'm trying to understand just how
> bad that is for our performance.  Seems like we'd be tying up web
> dynos, but I want to make sure.
>
> Secondly, given that I don't feel like rewriting all that code right
> now, is there a way to map certain paths to a heroku app to another
> server, so we aren't hammering our dynos so badly?
>
> Thanks,
> Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Newb question on static assets

[Carson Gross]

Heya,

Sorry, digging through the docs I couldn't really get an answer on
this that I understand:

So, with static assets, deployed on the bamboo-ree-1.8.7 stack, are
static assets served by web dynos?  As our application has grown over
time, we've evolved to issue a horrific number of requests for static
content (css, javascript, etc.) and I'm trying to understand just how
bad that is for our performance.  Seems like we'd be tying up web
dynos, but I want to make sure.

Secondly, given that I don't feel like rewriting all that code right
now, is there a way to map certain paths to a heroku app to another
server, so we aren't hammering our dynos so badly?

Thanks,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Jar Files

[Carson Gross]

I'll say this again: I think that Heroku is making a mistake by mixing
the build and the deploy steps in java applications.  A lot of
applications are not built using maven, that don't have easily
resolvable dependencies, or that might rely on system-specific
resources.  Additionally, a lot of java shops aren't as comfortable
shipping source code out to third parties as the scripting language
communities are.

I feel we should be able to deploy wars/jars locally and deploy them
directly.  It is admittedly different than the rails approach, but it
fits the java development process much more naturally.

Cheers,
Carson

On Sep 21, 7:00 pm, TomXH  wrote:
> I am trying to port a java application to Heroku.  The application has
> dependencies on many jars.  Several of these were available in the
> Maven repository so no problem.   However, I have a couple that are
> not available on public repositories.  I added these jars to my local
> maven repository using "mvn install:install".  With that, I am
> able to build and run my project locally.  However, when I push to
> Heroku, I get an error indicating "artifacts cannot be resolved".
>
> What is the best way to make these jar files available to my
> application on Heroku?  I have to think this is a common problem and
> there must be a simple answer.
>
> Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: Java apps

[Carson Gross]

Yeah, it seems like the natural way to do Java deployments.  Doing a
build in a remote environment (and dealing with all the uncertainty
that comes along with it) every time you do a deploy doesn't seem to
me to add much value, particularly if it locks you in to a certain
build system.

Heroku guys, any thoughts?

I don't want to come across as negative on this: I'm very excited that
you are getting into this business and desperately want to get Gosu/
Ronin applications deployed in Heroku.

Cheers,
Carson

On Sep 7, 7:05 am, John McCaffrey  wrote:
> Also, isn't that how things are deployed on google app engine? (just push up
> a war file)
>
>  It would be nice if the deployment process was similar, so that users of
> other hosting systems could easily give heroku a try, and see how awesome it
> is.
>
> You may also want to checkout CloudFoundry <http://cloudfoundry.com/faq> and
> keep an eye on how they do things. (I just pushed out a rails app to their
> beta product, and ran some performance tests. They are still adding
> features, but my app is running fine, so I'll definitely keep them on my
> radar)
>
> -John
>
>
>
>
>
> On Wed, Sep 7, 2011 at 12:21 AM, Carson Gross  wrote:
> > Guys,
>
> > Just saw your announcement on supporting Java applications, and I'm
> > very excited about it.  I've been working on a rails app deployed on
> > heroku and always wondered if someone would make a heroku for java
> > and, well, here we are.  I also work on Gosu (a JVM language) and
> > Ronin (a Gosu-based web app framework) and I would love to get them
> > deployed on Heroku
>
> > I'd like to ask you to consider a change to your deployment model,
> > however.  I realize that ruby apps deploy with their code in place and
> > so it makes a lot of sense to manage things as a git push as there
> > isn't really a compilation phase.  However, java has a much more
> > diverse set of build tools since it is a compiled language: you've got
> > ant, gant, maven, buildr, aardvark (the one I use, Gosu based) and so
> > on.  As such, forcing us to use Maven (which has a mixed reputation in
> > the java world) is a tough pill to swallow.
>
> > So what I'd like to ask you to consider is accepting simple war files,
> > built locally and pushed out to heroku.  This takes the build and
> > verification phase out of a deployment and provides a simple,
> > standardized way for us java devs to deploy to heroku.  It also opens
> > up the platform to other languages and frameworks: almost all of them
> > provide some way to bundle up a war for deployment.
>
> > Thanks again for all your great work,
> > Carson
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.
>
> --
> Thanks,
> -John

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Java apps

[Carson Gross]

Guys,

Just saw your announcement on supporting Java applications, and I'm
very excited about it.  I've been working on a rails app deployed on
heroku and always wondered if someone would make a heroku for java
and, well, here we are.  I also work on Gosu (a JVM language) and
Ronin (a Gosu-based web app framework) and I would love to get them
deployed on Heroku

I'd like to ask you to consider a change to your deployment model,
however.  I realize that ruby apps deploy with their code in place and
so it makes a lot of sense to manage things as a git push as there
isn't really a compilation phase.  However, java has a much more
diverse set of build tools since it is a compiled language: you've got
ant, gant, maven, buildr, aardvark (the one I use, Gosu based) and so
on.  As such, forcing us to use Maven (which has a mixed reputation in
the java world) is a tough pill to swallow.

So what I'd like to ask you to consider is accepting simple war files,
built locally and pushed out to heroku.  This takes the build and
verification phase out of a deployment and provides a simple,
standardized way for us java devs to deploy to heroku.  It also opens
up the platform to other languages and frameworks: almost all of them
provide some way to bundle up a war for deployment.

Thanks again for all your great work,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: request.remote_ip

[Carson Gross]

OK, circling back on this:

  request.remote_ip

Does *not* work consistently in our heroku application, even on non-
https requests.

Any solutions or suggestions for this from the heroku guys?  I guess I
can use clicky's API, but that would be a lot of work...

Cheers,
Carson

On Mar 15, 9:39 am, marcel  wrote:
> I ran into this same problem playing with Rack::Bug. I couldn't access
> my site and determined it was because my client IP was not in the
> whitelist I'd configured.
>
> I read the Rails docs and even the source and it seemed like
> request.remote_ipshould just work... but it clearly did not. I logged
> all the request object's properties and saw my client IP in there, but
> for whatever reason request.remote_ipalways returned Heroku's load
> balancer's IP and not my real client IP.
>
> In the end I side-stepped the problem and just setup a password for
> access.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: request.remote_ip

[Carson Gross]

Hmmm.  Looking at the docs here:

  
http://docs.huihoo.com/api/ruby-on-rails/classes/ActionController/AbstractRequest.html#M000235

It looks like request.remote_ip is actually smart-ish and tries to
resolve the original IP.

I don't see an 'ip' method there (I've got to admit, I find the
standard ruby docs impenetrable.)

Maybe the heroku load balancers do something funny with requests?
Seems unlikely, but possible.

Heroku guys?

Cheers,
Carson

On Mar 14, 2:46 pm, kowsik  wrote:
> Try request.ip which uses the X-Forwarded-For header automatically.
>
> K.
> ---http://blitz.iohttp://twitter.com/pcapr
>
>
>
> On Mon, Mar 14, 2011 at 2:32 PM, Travis Reeder  wrote:
> > Check out the headers, you may find X-Forwarded-For or X-Forwarded-Proto
> > which will be the IP you want.
>
> > On Mon, Mar 14, 2011 at 1:42 PM, Carson Gross  wrote:
>
> >> Hello,
>
> >> I'm trying to get at the remote ip address for requests, and I'm using
> >> the standard
>
> >>  request.remote_ip
>
> >> method, but it appears that this is returning the same few IP's for
> >> all our users.  My guess is that this is the heroku load balancer at
> >> work, but that's just a shot in the dark.
>
> >> Any suggestions or pointers?
>
> >> Cheers,
> >> Carson
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Heroku" group.
> >> To post to this group, send email to heroku@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> heroku+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/heroku?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: request.remote_ip

[Carson Gross]

Travis,

Thanks.  Any heroku guys care to chime in and confirm?

Cheers,
Carson

On Mar 14, 2:32 pm, Travis Reeder  wrote:
> Check out the headers, you may find X-Forwarded-For or X-Forwarded-Proto
> which will be the IP you want.
>
>
>
> On Mon, Mar 14, 2011 at 1:42 PM, Carson Gross  wrote:
> > Hello,
>
> > I'm trying to get at the remote ip address for requests, and I'm using
> > the standard
>
> >  request.remote_ip
>
> > method, but it appears that this is returning the same few IP's for
> > all our users.  My guess is that this is the heroku load balancer at
> > work, but that's just a shot in the dark.
>
> > Any suggestions or pointers?
>
> > Cheers,
> > Carson
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

request.remote_ip

[Carson Gross]

Hello,

I'm trying to get at the remote ip address for requests, and I'm using
the standard

  request.remote_ip

method, but it appears that this is returning the same few IP's for
all our users.  My guess is that this is the heroku load balancer at
work, but that's just a shot in the dark.

Any suggestions or pointers?

Cheers,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: postgres tuning

[Carson Gross]

John

It's hard to simulate production loads of the particular queries I'm
concerned with, so I was hoping that we could tweak stuff and see.

We've got out app pretty tuned up with indexes and all the rest, at
this point I'm just looking for any last bits of low hanging fruit,
er, laying around.  (We are running a ronin database, so kicking that
up is another obvious lever to pull.)

The two params I mentioned seem like the potentially big wins,
effective_cache_size because it *should* if I understand boost the
memory that postgres will dedicate to queries across the board (don't
know how it interacts w/ work_mem) and checkpoint segments because
we've got one table that is appended to quite frequently and in large
volumes.  Unfortunately, without trying it out in production, it will
be hard to know what the benefits are.

I'll let you know if we get anywhere.

Cheers,
Carson

On Jan 13, 2:05 pm, John McCaffrey  wrote:
> I've been working on postgres tuning for an app recently, and I was
> wondering if you were able to change these configs locally and see a
> meaningful improvement.
>
> I've seen improvements of large queries with sorts, by increasing work_mem
> (which can be done within your session), but other settings always seem to
> take a good deal of fiddling with, in order to get them to make a
> difference, so I'm wondering if you've already proven that this change will
> help or not.
>
> (it might be bold of me to assume that you'd pull down your prod data from
> heroku, and fire up a local postgres to tune your queries against)
>
> Here's a good list of postgres tuning options that you can do 
> nowhttp://robots.thoughtbot.com/post/2638538135/postgresql-performance-c...
>
> I'd be interested in any of the things you've already done to improve your
> performance on heroku, The more we share the better things are for all of
> us.
>
> -John
>
> On Tue, Jan 11, 2011 at 11:48 AM, Carson Gross wrote:
>
>
>
>
>
> > Hey Guys,
>
> > Any further thoughts on this?
>
> > Thanks,
> > Carson
>
> > On Jan 7, 1:35 pm, Carson  Gross  wrote:
> > > Sure.  We've got a big, ugly database that we are constantly slamming
> > > data into (nearly constant appends on one table, with occasionally
> > > purges.)  We'd like to increase the checkpoint segments to see if that
> > > boosts write performance.
>
> > > On the cache side, according to this website:
>
> > >  http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
>
> > > "Setting effective_cache_size to 1/2 of total memory would be a normal
> > > conservative setting, and 3/4 of memory is a more aggressive but still
> > > reasonable amount."
>
> > > Again, as near as I can tell, the default in Heroku is to set it to
> > > 200MB.  We are running a ronin database, which should have 1.7 Gigs of
> > > RAM, giving 850MB to 1.3 GB of cache as the ideal, at least according
> > > to that wiki entry.
>
> > > Our goal is to extract maximum performance with minimum effort and bug
> > > creation and, therefore, tweaking some postgres parameters would be an
> > > ideal way to do so.  Let me know if you need any more details on our
> > > use case.
>
> > > Thanks!
> > > Carson
>
> > > On Jan 7, 12:40 pm, Matthew Soldo  wrote:
>
> > > > Hi Carson,
>
> > > > We don't support tuning the dedicated database.
>
> > > > I'd love to hear more about your requirements and needs around this.
> > It's
> > > > possible that this could be supported in the future.
>
> > > > Matt
>
> > > > On Fri, Jan 7, 2011 at 10:50 AM, Carson Gross 
> > wrote:
> > > > > Is it possible to tune a dedicated postgres database?  In particular,
> > > > > we'd like to change the effective cache size and checkpoint segments
> > > > > setting from the defaults (appear to be ~200MB and 40, respectively.)
>
> > > > > Cheers,
> > > > > Carson
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > Groups
> > > > > "Heroku" group.
> > > > > To post to this group, send email to heroku@googlegroups.com.
> > > > > To unsubscribe from this group, send email to
> > > > > heroku+unsubscr...@googlegroups.com
> >  > com>
>
> > > > > .
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/heroku?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.
>
> --
> -John

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: postgres tuning

[Carson Gross]

Yeah.

Sadly, we are all very much stuck in the YesSQL mindset (our data is
relational) and would prefer to stick with that horse as long as is
possible.  Our performance is good right now and appears to scale to
the loads we'd like to see, but since perf is typically a non-linear
sort of thing regardless of what solution you use, I want to buy us as
much performance with as few radical, bug-introducing changes as
possible.

Tweaking two postgres parameters seems like an ideal match to that
conservatism and, OK, I'll admit it, laziness.

Cheers,
Carson

On Jan 11, 10:43 am, chris  wrote:
> Your use case (high volume, low value writes) sounds like it would be a
> perfect match for one of heroku's nosql addons... mongo or couch in
> particular.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: postgres tuning

[Carson Gross]

Hey Guys,

Any further thoughts on this?

Thanks,
Carson

On Jan 7, 1:35 pm, Carson  Gross  wrote:
> Sure.  We've got a big, ugly database that we are constantly slamming
> data into (nearly constant appends on one table, with occasionally
> purges.)  We'd like to increase the checkpoint segments to see if that
> boosts write performance.
>
> On the cache side, according to this website:
>
>  http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
>
> "Setting effective_cache_size to 1/2 of total memory would be a normal
> conservative setting, and 3/4 of memory is a more aggressive but still
> reasonable amount."
>
> Again, as near as I can tell, the default in Heroku is to set it to
> 200MB.  We are running a ronin database, which should have 1.7 Gigs of
> RAM, giving 850MB to 1.3 GB of cache as the ideal, at least according
> to that wiki entry.
>
> Our goal is to extract maximum performance with minimum effort and bug
> creation and, therefore, tweaking some postgres parameters would be an
> ideal way to do so.  Let me know if you need any more details on our
> use case.
>
> Thanks!
> Carson
>
> On Jan 7, 12:40 pm, Matthew Soldo  wrote:
>
>
>
> > Hi Carson,
>
> > We don't support tuning the dedicated database.
>
> > I'd love to hear more about your requirements and needs around this. It's
> > possible that this could be supported in the future.
>
> > Matt
>
> > On Fri, Jan 7, 2011 at 10:50 AM, Carson Gross  wrote:
> > > Is it possible to tune a dedicated postgres database?  In particular,
> > > we'd like to change the effective cache size and checkpoint segments
> > > setting from the defaults (appear to be ~200MB and 40, respectively.)
>
> > > Cheers,
> > > Carson
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Heroku" group.
> > > To post to this group, send email to her...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > heroku+unsubscr...@googlegroups.com
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: postgres tuning

[Carson Gross]

Sure.  We've got a big, ugly database that we are constantly slamming
data into (nearly constant appends on one table, with occasionally
purges.)  We'd like to increase the checkpoint segments to see if that
boosts write performance.

On the cache side, according to this website:

  http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

"Setting effective_cache_size to 1/2 of total memory would be a normal
conservative setting, and 3/4 of memory is a more aggressive but still
reasonable amount."

Again, as near as I can tell, the default in Heroku is to set it to
200MB.  We are running a ronin database, which should have 1.7 Gigs of
RAM, giving 850MB to 1.3 GB of cache as the ideal, at least according
to that wiki entry.

Our goal is to extract maximum performance with minimum effort and bug
creation and, therefore, tweaking some postgres parameters would be an
ideal way to do so.  Let me know if you need any more details on our
use case.

Thanks!
Carson

On Jan 7, 12:40 pm, Matthew Soldo  wrote:
> Hi Carson,
>
> We don't support tuning the dedicated database.
>
> I'd love to hear more about your requirements and needs around this. It's
> possible that this could be supported in the future.
>
> Matt
>
>
>
> On Fri, Jan 7, 2011 at 10:50 AM, Carson Gross  wrote:
> > Is it possible to tune a dedicated postgres database?  In particular,
> > we'd like to change the effective cache size and checkpoint segments
> > setting from the defaults (appear to be ~200MB and 40, respectively.)
>
> > Cheers,
> > Carson
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to her...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

postgres tuning

[Carson Gross]

Is it possible to tune a dedicated postgres database?  In particular,
we'd like to change the effective cache size and checkpoint segments
setting from the defaults (appear to be ~200MB and 40, respectively.)

Cheers,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: memcached servers

[Carson Gross]

To the top: is there a way to find out how well our dalli-based
memcache is working?

Thanks,
Carson

On Dec 21, 10:09 pm, Carson  Gross  wrote:
> To the top:  any good way to look at the utilization and  of our dalli
> memcache clien?  The code in the heroku docs doesn't work with it, and
> I'd like to know if the cache is working and, assuming so, how well it
> is working, and if it would be worth it to buy more memcache...
>
> Thanks,
> Carson
>
> On Dec 20, 11:15 pm, Carson  Gross  wrote:
>
>
>
> > Thanks guys.  I *think* we've got things installed correctly now.
>
> > Is there an easy way to see the stats for our memcache?  I tried what
> > the docs say, and got this:
>
> > >> mc = Memcached.new
>
> > NameError: uninitialized constant Memcached
> >         /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
> > active_support/dependencies.rb:443:in `load_missing_constant'
> >         /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
> > active_support/dependencies.rb:80:in
> > `const_missing_not_from_s3_library'
> >         /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/
> > extensions.rb:206:in `const_missing'
> >         /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
> > active_support/dependencies.rb:92:in `const_missing'
>
> > Is there some web UI somewhere I can see?
>
> > Thanks again for everyones help,
> > Carson
>
> > On Dec 20, 11:12 am, chris  wrote:
>
> > > FYI, dalli will detect that ENV variable automatically -- you shouldn't 
> > > need
> > > to specify the server(s) at all. Here's our dalli configuration (rails3 
> > > app,
> > > yours may vary):
>
> > >       config.cache_store = :dalli_store, { :namespace => 'll', :expires_in
> > > => 30.minutes, :compress => true, :compress_threshold => 64*1024 }

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: memcached servers

[Carson Gross]

To the top:  any good way to look at the utilization and  of our dalli
memcache clien?  The code in the heroku docs doesn't work with it, and
I'd like to know if the cache is working and, assuming so, how well it
is working, and if it would be worth it to buy more memcache...

Thanks,
Carson

On Dec 20, 11:15 pm, Carson  Gross  wrote:
> Thanks guys.  I *think* we've got things installed correctly now.
>
> Is there an easy way to see the stats for our memcache?  I tried what
> the docs say, and got this:
>
> >> mc = Memcached.new
>
> NameError: uninitialized constant Memcached
>         /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
> active_support/dependencies.rb:443:in `load_missing_constant'
>         /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
> active_support/dependencies.rb:80:in
> `const_missing_not_from_s3_library'
>         /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/
> extensions.rb:206:in `const_missing'
>         /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
> active_support/dependencies.rb:92:in `const_missing'
>
> Is there some web UI somewhere I can see?
>
> Thanks again for everyones help,
> Carson
>
> On Dec 20, 11:12 am, chris  wrote:
>
>
>
> > FYI, dalli will detect that ENV variable automatically -- you shouldn't need
> > to specify the server(s) at all. Here's our dalli configuration (rails3 app,
> > yours may vary):
>
> >       config.cache_store = :dalli_store, { :namespace => 'll', :expires_in
> > => 30.minutes, :compress => true, :compress_threshold => 64*1024 }

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: memcached servers

[Carson Gross]

Thanks guys.  I *think* we've got things installed correctly now.

Is there an easy way to see the stats for our memcache?  I tried what
the docs say, and got this:

>> mc = Memcached.new
NameError: uninitialized constant Memcached
/usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
active_support/dependencies.rb:443:in `load_missing_constant'
/usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
active_support/dependencies.rb:80:in
`const_missing_not_from_s3_library'
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/
extensions.rb:206:in `const_missing'
/usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/
active_support/dependencies.rb:92:in `const_missing'

Is there some web UI somewhere I can see?

Thanks again for everyones help,
Carson

On Dec 20, 11:12 am, chris  wrote:
> FYI, dalli will detect that ENV variable automatically -- you shouldn't need
> to specify the server(s) at all. Here's our dalli configuration (rails3 app,
> yours may vary):
>
>       config.cache_store = :dalli_store, { :namespace => 'll', :expires_in
> => 30.minutes, :compress => true, :compress_threshold => 64*1024 }

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: memcached servers

[Carson Gross]

Awesome, very informative.

I see in my ENV hash the following entry:

  MEMCACHE_SERVER : mc2.ec2.northscale.net

Is that the memcache server I should be using?

I see I can get the app URL there as well, as you said, so I can use
that if memcache is running at that URL.

Thanks!

Cheers,
Carson

On Dec 18, 11:38 am, Jason Morrison  wrote:
> Hi Carson,
>
> Heroku exposes addon parameters as config vars, which are populated as ENV
> vars available to your app:
>
> https://gist.github.com/597532192c16b13f4a2d
>
> Similarly, you can access ENV['APP_NAME'] to fetch the app name, and
> ENV['URL'] for the app base URL.  Pop open `heroku console` and inspect the
> ENV hash - a bunch of neat stuff in there.  You can set the RAILS_ENV with
> config vars, too:http://docs.heroku.com/config-vars#rackenv-railsenv-merbenv
>
> Hope this helps,
> -Jason
>
>
>
>
>
>
>
>
>
> On Sat, Dec 18, 2010 at 9:59 AM, Carson Gross  wrote:
> > Memcached/rube/rails/heroku newb.  Apologies in advance.
>
> > The docs here:http://docs.heroku.com/memcachegive this for the
> > memcached config:
>
> >  # Session cache
> >  ActionController::Base.session = {
> >    :namespace   => 'sessions',
> >    :expire_after => 20.minutes.to_i,
> >    :memcache_server => ['server-1:11211', 'server-2:11211'],
> >    :key         => ...,
> >    :secret      => ...
> >  }
>
> >  require 'action_controller/session/dalli_store'
> >  ActionController::Base.session_store = :dalli_store
>
> > First question: is 'server-1' our app url?  (
>
> > Second question: is there a way to get the name of the current app?
> > We run a few versions of our app (staging, production, and, obviously
> > locally on our own machines) and I obviously don't want them all going
> > to our production memcache instance.  What's the orthodox way to
> > parameterized the server url, and then is there an orthodox way to
> > derive that from the environment?
>
> > Thanks,
> > Carson
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to her...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.
>
> --
> Jason Morrison
> thoughtbot.com
> 585-216-5657
> @jayunit

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

memcached servers

[Carson Gross]

Memcached/rube/rails/heroku newb.  Apologies in advance.

The docs here: http://docs.heroku.com/memcache give this for the
memcached config:

  # Session cache
  ActionController::Base.session = {
:namespace   => 'sessions',
:expire_after => 20.minutes.to_i,
:memcache_server => ['server-1:11211', 'server-2:11211'],
:key => ...,
:secret  => ...
  }

  require 'action_controller/session/dalli_store'
  ActionController::Base.session_store = :dalli_store

First question: is 'server-1' our app url?  (

Second question: is there a way to get the name of the current app?
We run a few versions of our app (staging, production, and, obviously
locally on our own machines) and I obviously don't want them all going
to our production memcache instance.  What's the orthodox way to
parameterized the server url, and then is there an orthodox way to
derive that from the environment?

Thanks,
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

Re: User striping

[Carson Gross]

I see.  So if we were going to do this internally in multiple heroku
db's we might use a technique like this:

  http://programmerassist.com/article/302

correct?

And if we were going to do it externally, with redirects, we would
just need to create multiple heroku applications, and write our own
redirect system.

Thanks!,
Carson

On Dec 1, 2:16 pm, Peter van Hardenberg  wrote:
> Hi Carson,
>
> that's an interesting scaling model, but it doesn't map well onto the way
> Heroku is architected. The Heroku model runs many distributed processes
> throughout our single shared cloud. You don't have any "nodes" in a
> traditional sense, but each web server dyno and background worker is
> independent and stateless so they can be added or removed on demand and
> scaled in a moment.
>
> As to the database scaling, you have several options. We don't have an ideal
> horizontal scaling setup for independent, equally sized databases at the
> moment, mainly because for boring technical reasons you can't add more than
> one database with the same plan to a given app, but there are pretty trivial
> workarounds for that.
>
> -p
>
>
>
> On Wed, Dec 1, 2010 at 5:10 AM, Carson Gross  wrote:
> > Hi Guys,
>
> > The datamodel in our little heroku application is set up so there are
> > no data dependencies between users.  We would like our scaling story
> > to be "as we add users, we add nodes" where the nodes can be totally
> > independent (separate web server, separate db) from one another.  (We
> > don't want to just scale up the heroku specifications on a single node
> > because there is one table with insert/update contention between
> > users, and our data sizes are pretty big.)
>
> > Are there any tools for doing this sort of thing?  We can implement it
> > ourselves, of course, but would prefer to color within the lines if
> > something already exists out there.
>
> > Thanks!
> > Carson
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to her...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/heroku?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.

User striping

[Carson Gross]

Hi Guys,

The datamodel in our little heroku application is set up so there are
no data dependencies between users.  We would like our scaling story
to be "as we add users, we add nodes" where the nodes can be totally
independent (separate web server, separate db) from one another.  (We
don't want to just scale up the heroku specifications on a single node
because there is one table with insert/update contention between
users, and our data sizes are pretty big.)

Are there any tools for doing this sort of thing?  We can implement it
ourselves, of course, but would prefer to color within the lines if
something already exists out there.

Thanks!
Carson

-- 
You received this message because you are subscribed to the Google Groups 
"Heroku" group.
To post to this group, send email to her...@googlegroups.com.
To unsubscribe from this group, send email to 
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/heroku?hl=en.