Re: Sharding issues
Yes, that seemed to fix it.
Thanks!!
J
On Thursday, January 12, 2017 at 4:16:17 PM UTC-5, Jeremy Evans wrote:
>
> On Thursday, January 12, 2017 at 12:52:17 PM UTC-8, Jay Danielian wrote:
>>
>> Thanks for the response, you are a lifesaver! Yes, removing the plugin
>> seems to work.
>>
>> I can confirm with tracing it through it seems like the issue is in the
>> prepared_statements plugin. The code dataset.opts.fetch(:server, :default)
>> seems to miss the supplied :server key (or set_server key) as dataset.opts
>> is missing the supplied value, so I think :default is always returned.
>>
>> # Return a prepared statement that can be used to insert a row
>> using the given columns
>> # and return that column values for the row created.
>> def prepared_insert_select(cols)
>> if dataset.supports_insert_select?
>> cached_prepared_statement(:insert_select,
>> prepared_columns(cols)){prepare_explicit_statement(naked.clone(:server=>dataset.opts.fetch(:server,
>>
>> :default)), :insert_select, prepared_statement_key_hash(cols))}
>> end
>> end
>>
>> It *seems* like these methods that do the preparation
>> _insert_raw, _insert_select_raw, _refresh_get would work if inside those
>> methods there was something like
>>
>> model.dataset.opts.merge!(ds.opts)
>>
>> before calling the model.send(:prepared_blah...) would work as I could
>> see the ds.opts variable having the :server key set but the model.dataset
>> did not (which is what is being used to pick the server).
>>
>> Thanks for the great support!
>>
>
> I've fixed the issues in the prepared_statements plugin (and related
> plugins), could you please try the master branch and see if that fixes the
> issues for you?
>
> Thanks,
> Jeremy
>
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sequel-talk+unsubscr...@googlegroups.com.
To post to this group, send email to sequel-talk@googlegroups.com.
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
Re: Sharding issues
On Thursday, January 12, 2017 at 12:52:17 PM UTC-8, Jay Danielian wrote:
>
> Thanks for the response, you are a lifesaver! Yes, removing the plugin
> seems to work.
>
> I can confirm with tracing it through it seems like the issue is in the
> prepared_statements plugin. The code dataset.opts.fetch(:server, :default)
> seems to miss the supplied :server key (or set_server key) as dataset.opts
> is missing the supplied value, so I think :default is always returned.
>
> # Return a prepared statement that can be used to insert a row
> using the given columns
> # and return that column values for the row created.
> def prepared_insert_select(cols)
> if dataset.supports_insert_select?
> cached_prepared_statement(:insert_select,
> prepared_columns(cols)){prepare_explicit_statement(naked.clone(:server=>dataset.opts.fetch(:server,
>
> :default)), :insert_select, prepared_statement_key_hash(cols))}
> end
> end
>
> It *seems* like these methods that do the preparation
> _insert_raw, _insert_select_raw, _refresh_get would work if inside those
> methods there was something like
>
> model.dataset.opts.merge!(ds.opts)
>
> before calling the model.send(:prepared_blah...) would work as I could see
> the ds.opts variable having the :server key set but the model.dataset did
> not (which is what is being used to pick the server).
>
> Thanks for the great support!
>
I've fixed the issues in the prepared_statements plugin (and related
plugins), could you please try the master branch and see if that fixes the
issues for you?
Thanks,
Jeremy
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sequel-talk+unsubscr...@googlegroups.com.
To post to this group, send email to sequel-talk@googlegroups.com.
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
Re: Sharding issues
Thanks for the response, you are a lifesaver! Yes, removing the plugin
seems to work.
I can confirm with tracing it through it seems like the issue is in the
prepared_statements plugin. The code dataset.opts.fetch(:server, :default)
seems to miss the supplied :server key (or set_server key) as dataset.opts
is missing the supplied value, so I think :default is always returned.
# Return a prepared statement that can be used to insert a row
using the given columns
# and return that column values for the row created.
def prepared_insert_select(cols)
if dataset.supports_insert_select?
cached_prepared_statement(:insert_select,
prepared_columns(cols)){prepare_explicit_statement(naked.clone(:server=>dataset.opts.fetch(:server,
:default)), :insert_select, prepared_statement_key_hash(cols))}
end
end
It *seems* like these methods that do the preparation
_insert_raw, _insert_select_raw, _refresh_get would work if inside those
methods there was something like
model.dataset.opts.merge!(ds.opts)
before calling the model.send(:prepared_blah...) would work as I could see
the ds.opts variable having the :server key set but the model.dataset did
not (which is what is being used to pick the server).
Thanks for the great support!
J
On Thursday, January 12, 2017 at 3:20:52 PM UTC-5, Jeremy Evans wrote:
>
> On Thursday, January 12, 2017 at 11:53:50 AM UTC-8, Jay Danielian wrote:
>>
>> After adding DB.extension :server_logging to get further details about
>> what is executing where I see a potential issue (question is still why).
>>
>> Here is what I see in the logs. Looks interesting, I have bolded the
>> relevant where it looks like it is starting on node_2, then does the actual
>> execution on the :default node.
>>
>> I, [2017-01-12T14:47:26.377432 #3176] INFO -- : (0.000112s) (conn:
>> 70118877855900, *server: node_2*) BEGIN
>> I, [2017-01-12T14:47:26.382943 #3176] INFO -- : (0.001143s) (conn:
>> 70118870257360, *server: default*) PREPARE smpsp_1 AS INSERT INTO
>> "contacts" ("shard_id", "owner_id", "first_name", "last_name", "job_title")
>> VALUES ($1, $2, $3, $4, $5) RETURNING "id", "shard_id", "owner_id",
>> "unified_id", "external_id", "source_id", "created_on", "updated_on",
>> "prefix", "first_name", "last_name", "middle_name", "suffix",
>> "company_name", "job_title", "home_email_1", "home_email_2",
>> "home_email_3", "work_email_1", "work_email_2", "work_email_3",
>> "work_phone", "work_phone_direct", "work_fax", "home_fax",
>> "mobile_phone_1", "mobile_phone_2", "home_phone_1", "home_phone_2",
>> "home_street_1", "home_city_1", "home_state_1", "home_postal_code_1",
>> "home_country_1", "home_street_2", "home_city_2", "home_state_2",
>> "home_postal_code_2", "home_country_2", "work_street_1", "work_city_1",
>> "work_state_1", "work_postal_code_1", "work_country_1", "work_street_2",
>> "work_city_2", "work_state_2", "work_postal_code_2", "work_country_2",
>> "is_deleted", "lock_version", "properties", "notes"
>> I, [2017-01-12T14:47:26.392660 #3176] INFO -- : (0.009518s) (conn:
>> 70118870257360,* server: default*) EXECUTE smpsp_1 (INSERT INTO
>> "contacts" ("shard_id", "owner_id", "first_name", "last_name", "job_title")
>> VALUES ($1, $2, $3, $4, $5) RETURNING "id", "shard_id", "owner_id",
>> "unified_id", "external_id", "source_id", "created_on", "updated_on",
>> "prefix", "first_name", "last_name", "middle_name", "suffix",
>> "company_name", "job_title", "home_email_1", "home_email_2",
>> "home_email_3", "work_email_1", "work_email_2", "work_email_3",
>> "work_phone", "work_phone_direct", "work_fax", "home_fax",
>> "mobile_phone_1", "mobile_phone_2", "home_phone_1", "home_phone_2",
>> "home_street_1", "home_city_1", "home_state_1", "home_postal_code_1",
>> "home_country_1", "home_street_2", "home_city_2", "home_state_2",
>> "home_postal_code_2", "home_country_2", "work_street_1", "work_city_1",
>> "work_state_1", "work_postal_code_1", "work_country_1", "work_street_2",
>> "work_city_2", "work_state_2", "work_postal_code_2", "work_country_2",
>> "is_deleted", "lock_version", "properties", "notes"); [6992,
>> "9775d295-b58f-4452-81a9-b49672b0fb9e", "Frank", "Jones", "Intern"]
>> I, [2017-01-12T14:47:26.395028 #3176] INFO -- : (0.000212s) (conn:
>> 70118877855900, *server: node_2*) COMMIT
>>
>
> Looks like this is an issue with the prepared_statements plugin. There
> are a couple of immediate options:
>
> 1) Don't use the prepared statements plugin. This should have no effect
> other than possibly performance.
>
> 2) Use the server_block extension:
>
> DB.extension :server_block
> DB.with_server(:node_2) do
> contact.save
> end
>
> I'll look into the issue with the prepared_statements plugin and see if it
> can be fixed in the next version.
>
> Thanks,
> Jeremy
>
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from t
Re: Sharding issues
On Thursday, January 12, 2017 at 11:53:50 AM UTC-8, Jay Danielian wrote:
>
> After adding DB.extension :server_logging to get further details about
> what is executing where I see a potential issue (question is still why).
>
> Here is what I see in the logs. Looks interesting, I have bolded the
> relevant where it looks like it is starting on node_2, then does the actual
> execution on the :default node.
>
> I, [2017-01-12T14:47:26.377432 #3176] INFO -- : (0.000112s) (conn:
> 70118877855900, *server: node_2*) BEGIN
> I, [2017-01-12T14:47:26.382943 #3176] INFO -- : (0.001143s) (conn:
> 70118870257360, *server: default*) PREPARE smpsp_1 AS INSERT INTO
> "contacts" ("shard_id", "owner_id", "first_name", "last_name", "job_title")
> VALUES ($1, $2, $3, $4, $5) RETURNING "id", "shard_id", "owner_id",
> "unified_id", "external_id", "source_id", "created_on", "updated_on",
> "prefix", "first_name", "last_name", "middle_name", "suffix",
> "company_name", "job_title", "home_email_1", "home_email_2",
> "home_email_3", "work_email_1", "work_email_2", "work_email_3",
> "work_phone", "work_phone_direct", "work_fax", "home_fax",
> "mobile_phone_1", "mobile_phone_2", "home_phone_1", "home_phone_2",
> "home_street_1", "home_city_1", "home_state_1", "home_postal_code_1",
> "home_country_1", "home_street_2", "home_city_2", "home_state_2",
> "home_postal_code_2", "home_country_2", "work_street_1", "work_city_1",
> "work_state_1", "work_postal_code_1", "work_country_1", "work_street_2",
> "work_city_2", "work_state_2", "work_postal_code_2", "work_country_2",
> "is_deleted", "lock_version", "properties", "notes"
> I, [2017-01-12T14:47:26.392660 #3176] INFO -- : (0.009518s) (conn:
> 70118870257360,* server: default*) EXECUTE smpsp_1 (INSERT INTO
> "contacts" ("shard_id", "owner_id", "first_name", "last_name", "job_title")
> VALUES ($1, $2, $3, $4, $5) RETURNING "id", "shard_id", "owner_id",
> "unified_id", "external_id", "source_id", "created_on", "updated_on",
> "prefix", "first_name", "last_name", "middle_name", "suffix",
> "company_name", "job_title", "home_email_1", "home_email_2",
> "home_email_3", "work_email_1", "work_email_2", "work_email_3",
> "work_phone", "work_phone_direct", "work_fax", "home_fax",
> "mobile_phone_1", "mobile_phone_2", "home_phone_1", "home_phone_2",
> "home_street_1", "home_city_1", "home_state_1", "home_postal_code_1",
> "home_country_1", "home_street_2", "home_city_2", "home_state_2",
> "home_postal_code_2", "home_country_2", "work_street_1", "work_city_1",
> "work_state_1", "work_postal_code_1", "work_country_1", "work_street_2",
> "work_city_2", "work_state_2", "work_postal_code_2", "work_country_2",
> "is_deleted", "lock_version", "properties", "notes"); [6992,
> "9775d295-b58f-4452-81a9-b49672b0fb9e", "Frank", "Jones", "Intern"]
> I, [2017-01-12T14:47:26.395028 #3176] INFO -- : (0.000212s) (conn:
> 70118877855900, *server: node_2*) COMMIT
>
Looks like this is an issue with the prepared_statements plugin. There are
a couple of immediate options:
1) Don't use the prepared statements plugin. This should have no effect
other than possibly performance.
2) Use the server_block extension:
DB.extension :server_block
DB.with_server(:node_2) do
contact.save
end
I'll look into the issue with the prepared_statements plugin and see if it
can be fixed in the next version.
Thanks,
Jeremy
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sequel-talk+unsubscr...@googlegroups.com.
To post to this group, send email to sequel-talk@googlegroups.com.
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
Re: Sharding issues
After adding DB.extension :server_logging to get further details about what
is executing where I see a potential issue (question is still why).
Here is what I see in the logs. Looks interesting, I have bolded the
relevant where it looks like it is starting on node_2, then does the actual
execution on the :default node.
I, [2017-01-12T14:47:26.377432 #3176] INFO -- : (0.000112s) (conn:
70118877855900, *server: node_2*) BEGIN
I, [2017-01-12T14:47:26.382943 #3176] INFO -- : (0.001143s) (conn:
70118870257360, *server: default*) PREPARE smpsp_1 AS INSERT INTO
"contacts" ("shard_id", "owner_id", "first_name", "last_name", "job_title")
VALUES ($1, $2, $3, $4, $5) RETURNING "id", "shard_id", "owner_id",
"unified_id", "external_id", "source_id", "created_on", "updated_on",
"prefix", "first_name", "last_name", "middle_name", "suffix",
"company_name", "job_title", "home_email_1", "home_email_2",
"home_email_3", "work_email_1", "work_email_2", "work_email_3",
"work_phone", "work_phone_direct", "work_fax", "home_fax",
"mobile_phone_1", "mobile_phone_2", "home_phone_1", "home_phone_2",
"home_street_1", "home_city_1", "home_state_1", "home_postal_code_1",
"home_country_1", "home_street_2", "home_city_2", "home_state_2",
"home_postal_code_2", "home_country_2", "work_street_1", "work_city_1",
"work_state_1", "work_postal_code_1", "work_country_1", "work_street_2",
"work_city_2", "work_state_2", "work_postal_code_2", "work_country_2",
"is_deleted", "lock_version", "properties", "notes"
I, [2017-01-12T14:47:26.392660 #3176] INFO -- : (0.009518s) (conn:
70118870257360,* server: default*) EXECUTE smpsp_1 (INSERT INTO "contacts"
("shard_id", "owner_id", "first_name", "last_name", "job_title") VALUES
($1, $2, $3, $4, $5) RETURNING "id", "shard_id", "owner_id", "unified_id",
"external_id", "source_id", "created_on", "updated_on", "prefix",
"first_name", "last_name", "middle_name", "suffix", "company_name",
"job_title", "home_email_1", "home_email_2", "home_email_3",
"work_email_1", "work_email_2", "work_email_3", "work_phone",
"work_phone_direct", "work_fax", "home_fax", "mobile_phone_1",
"mobile_phone_2", "home_phone_1", "home_phone_2", "home_street_1",
"home_city_1", "home_state_1", "home_postal_code_1", "home_country_1",
"home_street_2", "home_city_2", "home_state_2", "home_postal_code_2",
"home_country_2", "work_street_1", "work_city_1", "work_state_1",
"work_postal_code_1", "work_country_1", "work_street_2", "work_city_2",
"work_state_2", "work_postal_code_2", "work_country_2", "is_deleted",
"lock_version", "properties", "notes"); [6992,
"9775d295-b58f-4452-81a9-b49672b0fb9e", "Frank", "Jones", "Intern"]
I, [2017-01-12T14:47:26.395028 #3176] INFO -- : (0.000212s) (conn:
70118877855900, *server: node_2*) COMMIT
Thanks!
J
On Thursday, January 12, 2017 at 2:23:42 PM UTC-5, Jay Danielian wrote:
>
> Quick update - I have tried to simplify things further, yet it still is
> not working. I realized from before that I only need the symbol key that
> has the different sharding config attached to it. But even using the
> :servers_hash option to tell me when my shard :server is not found, it
> still does not work. I don't get an error when passing in the symbol
> :node_2, so it seems to think its ok - yet the contact is still saved into
> contact-db-1.
>
> DB = Sequel.connect(:adapter => 'postgres', :host => 'localhost',:user =>
> 'contact_user', :password => 'password', :database => 'contact-db-1',
> :servers => {:node_1 => { :database =>
> 'contact-db-1'}, :node_2 => { :database => 'contact-db-2' }, :read_only
> =>{}}, :servers_hash=>Hash.new{|h,v| raise Exception.new("Unknown
> server: #{v}")},
> :loggers => [Logger.new($stdout)])
>
>
> neither contact.save(:server => :node_2).save
>
> or contact.set_server(:node_2).save seems to save the record into
> contact-db-2 database.
>
> Thanks!
>
> J
>
>
>
> On Thursday, January 12, 2017 at 11:26:26 AM UTC-5, Jay Danielian wrote:
>>
>> I am new to Sequel, so I am likely doing something wrong - but would
>> appreciate some guidance. I am trying to test out sharding locally by
>> having two postgres databases locally running.
>>
>> I start them as such: (I am able to connect to both using psql, so they
>> are running properly.)
>>
>> pg_ctl -D ~/work/postgres-db/contact-db-1 -l logfile start
>>
>> pg_ctl -D ~/work/postgres-db/contact-db-2 -l log file start
>>
>>
>>
>> I have connection information as follows:
>>
>> NODE_LIST = {}
>>
>> DEFAULT_DB_CONNECTION =
>> 'postgres://contact_user:pwd@localhost/contact-db-1'
>>
>> NODE_LIST[:node_1] = {:host => 'localhost',:user => 'contact_user',
>> :password => 'pwd', :database => 'contact-db-1'}
>>
>> NODE_LIST[:node_2] = {:host => 'localhost',:user => 'contact_user',
>> :password => 'pwd', :database => 'contact-db-2'}
>>
>>
>> DB = Sequel.connect(DEFAULT_DB_CONNECTION, :servers => NODE_LIST,
>> :loggers => [Logger.new($stdout)])
>>
>>
>> I have
Re: Sharding issues
Quick update - I have tried to simplify things further, yet it still is not
working. I realized from before that I only need the symbol key that has
the different sharding config attached to it. But even using the
:servers_hash option to tell me when my shard :server is not found, it
still does not work. I don't get an error when passing in the symbol
:node_2, so it seems to think its ok - yet the contact is still saved into
contact-db-1.
DB = Sequel.connect(:adapter => 'postgres', :host => 'localhost',:user =>
'contact_user', :password => 'password', :database => 'contact-db-1',
:servers => {:node_1 => { :database => 'contact-db-1'},
:node_2 => { :database => 'contact-db-2' }, :read_only =>{}},
:servers_hash=>Hash.new{|h,v| raise Exception.new("Unknown server: #{v}")},
:loggers => [Logger.new($stdout)])
neither contact.save(:server => :node_2).save
or contact.set_server(:node_2).save seems to save the record into
contact-db-2 database.
Thanks!
J
On Thursday, January 12, 2017 at 11:26:26 AM UTC-5, Jay Danielian wrote:
>
> I am new to Sequel, so I am likely doing something wrong - but would
> appreciate some guidance. I am trying to test out sharding locally by
> having two postgres databases locally running.
>
> I start them as such: (I am able to connect to both using psql, so they
> are running properly.)
>
> pg_ctl -D ~/work/postgres-db/contact-db-1 -l logfile start
>
> pg_ctl -D ~/work/postgres-db/contact-db-2 -l log file start
>
>
>
> I have connection information as follows:
>
> NODE_LIST = {}
>
> DEFAULT_DB_CONNECTION =
> 'postgres://contact_user:pwd@localhost/contact-db-1'
>
> NODE_LIST[:node_1] = {:host => 'localhost',:user => 'contact_user',
> :password => 'pwd', :database => 'contact-db-1'}
>
> NODE_LIST[:node_2] = {:host => 'localhost',:user => 'contact_user',
> :password => 'pwd', :database => 'contact-db-2'}
>
>
> DB = Sequel.connect(DEFAULT_DB_CONNECTION, :servers => NODE_LIST, :loggers
> => [Logger.new($stdout)])
>
>
> I have code in a spec (just trying to test this out) like this:
>
>
>contact = Contact.new do |c|
>
> c.owner_id = owner_id
>
> c.first_name = 'Frank'
>
> c.last_name = 'Jones'
>
> c.job_title = 'Intern'
>
>end
>
>
>contact.set_server(NODE_LIST[:node_2]).save #initial save
>
>
> Yet, I never see this record get committed to the contact-db-2 database.
> All my inserts go into the contact-db-1 (which is the default). I can see
> all my inserts when querying them via psql, so the basic functionality
> seems to be working fine. Any pointers as to what I am doing wrong? Not
> sure if it is an issue with having two identical nodes / hosts setup and
> only varying the database name? Or I am just overlooking something even
> more obvious :)
>
>
> Thanks!!
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sequel-talk+unsubscr...@googlegroups.com.
To post to this group, send email to sequel-talk@googlegroups.com.
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
Sharding issues
I am new to Sequel, so I am likely doing something wrong - but would
appreciate some guidance. I am trying to test out sharding locally by
having two postgres databases locally running.
I start them as such: (I am able to connect to both using psql, so they are
running properly.)
pg_ctl -D ~/work/postgres-db/contact-db-1 -l logfile start
pg_ctl -D ~/work/postgres-db/contact-db-2 -l log file start
I have connection information as follows:
NODE_LIST = {}
DEFAULT_DB_CONNECTION = 'postgres://contact_user:pwd@localhost/contact-db-1'
NODE_LIST[:node_1] = {:host => 'localhost',:user => 'contact_user',
:password => 'pwd', :database => 'contact-db-1'}
NODE_LIST[:node_2] = {:host => 'localhost',:user => 'contact_user',
:password => 'pwd', :database => 'contact-db-2'}
DB = Sequel.connect(DEFAULT_DB_CONNECTION, :servers => NODE_LIST, :loggers
=> [Logger.new($stdout)])
I have code in a spec (just trying to test this out) like this:
contact = Contact.new do |c|
c.owner_id = owner_id
c.first_name = 'Frank'
c.last_name = 'Jones'
c.job_title = 'Intern'
end
contact.set_server(NODE_LIST[:node_2]).save #initial save
Yet, I never see this record get committed to the contact-db-2 database.
All my inserts go into the contact-db-1 (which is the default). I can see
all my inserts when querying them via psql, so the basic functionality
seems to be working fine. Any pointers as to what I am doing wrong? Not
sure if it is an issue with having two identical nodes / hosts setup and
only varying the database name? Or I am just overlooking something even
more obvious :)
Thanks!!
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sequel-talk+unsubscr...@googlegroups.com.
To post to this group, send email to sequel-talk@googlegroups.com.
Visit this group at https://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
