Jira (PDB-3357) PuppetDB fails to update catalogs when rich data types are enabled

[Charlie Sharpsteen (JIRA)]

Title: Message Title

Charlie Sharpsteen created an issue   PuppetDB / PDB-3357 PuppetDB fails to update catalogs when rich data types are enabled Issue Type: Bug Assignee: Unassigned Created: 2017/03/15 12:37 PM Priority: Normal Reporter: Charlie Sharpsteen When a rich data type is added to a resource, the resulting catalog contains resources with ext_parameters keys. PuppetDB tries to work these new keys into an UPDATE operation on the catalog_resources table, which fails during statement preparation with a 'No hstore extension installed' error. No errors occur if the resource is initially stored with sensitive data. The root cause for this behavior is the same as PDB-3322, which concerns the Sensitive data type. Reproduction Case Install PE 2017.1.0 on CentOS 7. Add a file resource to the default node in /etc/puppetlabs/code/environments/production/manifests/site.pp that uses the binary_file function to manage content with binary data: file{'/tmp/rich_data_test': ensure => present, content => binary_file('/bin/true'), } Run puppet agent -t to compile a catalog and enter the resource into PuppetDB. Enable rich data and reload Puppet Server: puppet config set --section main rich_data true systemctl reload pe-puppetserver Run puppet agent -t. Outcome The agent run completes successfully, but /var/log/puppetlabs/puppetdb/puppetdb.log shows an error during catalog storage: 2017-03-15 19:33:12,583 ERROR [p.p.command] [8] [replace catalog] Retrying after attempt 0 for pe-201710-master.puppetdebug.vlan, due to: org.postgresql.util.PSQLException: No hstore extension installed. org.postgresql.util.PSQLException: No hstore extension installed. at org.postgresql.jdbc.PgPreparedStatement.setMap(PgPreparedStatement.java:553) at org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1036) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setObject(HikariProxyPreparedStatement.java) at clojure.java.jdbc$eval21518$fn__21519.invokePrim(jdbc.clj:341) at clojure.java.jdbc$eval21518$fn__21519.invoke(jdbc.clj) at clojure.java.jdbc$eval21497$fn__21498$G__21488__21507.invoke(jdbc.clj:328) at clojure.java.jdbc$set_parameters$fn__21594.invoke(jdbc.clj:478) at clojure.core$map_indexed$mapi__7050$fn__7051.invoke(core.clj:7024) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:521) at clojure.core$seq__4357.invokeStatic(core.clj:137) at clojure.core$dorun.invokeStatic(core.clj:3024) at clojure.core$dorun.invoke(core.clj:3024) at clojure.java.jdbc$set_parameters.invokeStatic(jdbc.clj:477) at clojure.java.jdbc$set_parameters.invoke(jdbc.clj:474) at clojure.java.jdbc$db_do_execute_prepared_statement.invokeStatic(jdbc.clj:760) at clojure.java.jdbc$db_do_execute_prepared_statement.invoke(jdbc.clj:748) at clojure.java.jdbc$db_do_prepared.invokeStatic(jdbc.clj:786) at clojure.java.jdbc$db_do_prepared.doInvoke(jdbc.clj:770) at clojure.lang.RestFn.invoke(RestFn.java:464) at clojure.java.jdbc$execute_BANG_$execute_helper__21696.invoke(jdbc.clj:891) at clojure.java.jdbc$execute_BANG_.invokeStatic(jdbc.clj:894) at clojure.java.jdbc$execute_BANG_.doInvoke(jdbc.clj:875) at clojure.lang.RestFn.invoke(RestFn.java:464) at clojure.java.jdbc$update_BANG_.invokeStatic(jdbc.clj:1077) at clojure.java.jdbc$update_BANG_.doInvoke(jdbc.clj:1066) at clojure.lang.RestFn.invoke(RestFn.java:470) at clojure.lang.AFn.applyToHelper(AFn.java:165) at clojure.lang.RestFn.applyTo(RestFn.java:132) at clojure.core$apply.invokeStatic(core.clj:654) at clojure.core$apply.doInvoke(core.clj:641) at clojure.lang.RestFn.invoke(RestFn.java:533) at puppetlabs.puppetdb.jdbc$update_BANG_.invokeStatic(jdbc.clj:72) at puppetlabs.puppetdb.jdbc$update_BANG_.doInvoke(jdbc.clj:66) at clojure.lang.RestFn.invoke(RestFn.java:445) at puppetlabs.puppetdb.scf.storage$eval28865$update_catalog_resources_BANG___28870$fn__28871$fn__28872.invoke(storage.clj:583) at puppetlabs.puppetdb.utils$eval6351$diff_fn__6356$fn__6357.invoke(utils.clj:67) at puppetlabs.puppetdb.utils$eval6351$diff_fn__6356.invoke(utils.clj:56) at puppetlabs.puppetdb.scf.storage$eval28905$add_resources_BANG___28910$fn__28911$fn__28912.invoke(storage.clj:603) at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620) at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568) at clojure.lang.RestFn.invoke(RestFn.java:425) at puppetlabs.puppetdb.scf.storage$eval28905$add_resources_BANG___28910$fn__28911.invoke(storage.clj:601) at puppetlabs.puppetdb.scf.storage$eval28905$add_resources_BANG___28910.invoke(storage.clj:594) at puppetlabs.puppetdb.scf.storage$eval29106$update_catalog_associations_BANG___29111$fn__29115$fn__29117.invoke(storage.clj:713) at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) at com.codahale.metrics.Timer.time(Timer.java:101) at puppetlabs.puppetdb.scf.storage$eval29106$update_catalog_associations_BANG___29111$fn__29115.invoke(storage.clj:712) at puppetlabs.puppetdb.scf.storage$eval29106$update_catalog_associations_BANG___29111.invoke(storage.clj:707) at puppetlabs.puppetdb.scf.storage$eval29144$replace_existing_catalog__29149$fn__29150$fn__29151.invoke(storage.clj:731) at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) at com.codahale.metrics.Timer.time(Timer.java:101) at puppetlabs.puppetdb.scf.storage$eval29144$replace_existing_catalog__29149$fn__29150.invoke(storage.clj:729) at puppetlabs.puppetdb.scf.storage$eval29144$replace_existing_catalog__29149.invoke(storage.clj:717) at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214$fn__29220$fn__29222$fn__29223.invoke(storage.clj:773) at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620) at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568) at clojure.lang.RestFn.invoke(RestFn.java:425) at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214$fn__29220$fn__29222.invoke(storage.clj:753) at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) at com.codahale.metrics.Timer.time(Timer.java:101) at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214$fn__29220.invoke(storage.clj:752) at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214.invoke(storage.clj:745) at puppetlabs.puppetdb.command$replace_catalog_STAR_$fn__32369.invoke(command.clj:279) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__22193$fn__22194.invoke(jdbc.clj:308) at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:595) at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568) at clojure.lang.RestFn.invoke(RestFn.java:464) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__22193.invoke(jdbc.clj:307) at puppetlabs.puppetdb.jdbc$eval22167$retry_sql_STAR___22172$fn__22173$fn__22174.invoke(jdbc.clj:285) at puppetlabs.puppetdb.jdbc$eval22167$retry_sql_STAR___22172$fn__22173.invoke(jdbc.clj:284) at puppetlabs.puppetdb.jdbc$eval22167$retry_sql_STAR___22172.invoke(jdbc.clj:275) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invokeStatic(jdbc.clj:305) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:300) at puppetlabs.puppetdb.command$replace_catalog_STAR_.invokeStatic(command.clj:277) at puppetlabs.puppetdb.command$replace_catalog_STAR_.invoke(command.clj:274) at puppetlabs.puppetdb.command$replace_catalog.invokeStatic(command.clj:288) at puppetlabs.puppetdb.command$replace_catalog.invoke(command.clj:283) at puppetlabs.puppetdb.command$process_command_BANG_.invokeStatic(command.clj:395) at puppetlabs.puppetdb.command$process_command_BANG_.invoke(command.clj:389) at puppetlabs.puppetdb.command$process_command_and_respond_BANG_$fn__32525.invoke(command.clj:448) at puppetlabs.puppetdb.command$call_with_quick_retry$fn__32518.invoke(command.clj:432) at puppetlabs.puppetdb.command$call_with_quick_retry.invokeStatic(command.clj:431) at puppetlabs.puppetdb.command$call_with_quick_retry.invoke(command.clj:429) at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invokeStatic(command.clj:446) at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invoke(command.clj:444) at puppetlabs.puppetdb.command$process_cmdref$fn__32534.invoke(command.clj:508) at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357$fn__30358.invoke(metrics.clj:14) at puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) at com.codahale.metrics.Timer.time(Timer.java:101) at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357.invoke(metrics.clj:14) at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357$fn__30358.invoke(metrics.clj:14) at puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) at com.codahale.metrics.Timer.time(Timer.java:101) at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357.invoke(metrics.clj:14) at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_.invokeStatic(metrics.clj:17) at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:6) at puppetlabs.puppetdb.command$process_cmdref.invokeStatic(command.clj:504) at puppetlabs.puppetdb.command$process_cmdref.invoke(command.clj:490) at puppetlabs.puppetdb.command$message_handler$fn__32542.invoke(command.clj:554) at puppetlabs.puppetdb.threadpool$dochan$fn__32301$fn__32302.invoke(threadpool.clj:117) at puppetlabs.puppetdb.threadpool$call_on_threadpool$fn__32296.invoke(threadpool.clj:95) at clojure.lang.AFn.run(AFn.java:22) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Expected Outcome PuppetDB accepts catalogs updates when rich data has been enabled. Add Comment   This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.