Hi Wyatt,
Thank you for the help!
>From the DLO entries I could get the exact command payload and found a
malformed structured fact. It had a nested dictionary, where most entries
were formed "foo" => "bar", except for one that looked like
"\"foo"=>"bar\"".
After fixing this, the node managed to successfully send its facts to
PuppetDB.
Cheers,
Akos
On Monday, March 14, 2016 at 4:54:26 PM UTC+1, Wyatt Alt wrote:
>
> On 03/14/2016 06:34 AM, Akos Hencz wrote:
>
> Hello,
>
> We have a few nodes that fail to submit facts to PuppetDB, and the replace
> facts commands end up in the DLO. The exception is:
> clojure.lang.ExceptionInfo: Input to insert-facts-pv-pairs! does not match
> schema.
> I suspect it is a problem with one of the facts, but I cannot determine
> which one. Looking at the output of 'facter -p' on any of the nodes seem to
> be fine.
>
> I would like to ask for some advice on how to investigate the cause of
> this problem. :)
>
> We run Puppet 3.8.4, PuppetDB 2.3.4, and PuppetServer 1.1.3. You can see
> the full exception text below.
>
> Cheers,
> Akos
>
>
> clojure.lang.ExceptionInfo: Input to insert-facts-pv-pairs! does not match
> schema: [nil (named (not (some (check % a-clojure.lang.LazySeq) schemas))
> pairs)] {:error [nil (named (not (some (check % a-clojure.lang.LazySeq)
> schemas)) pairs)], :value [44289 ([17 954] [90 58] [100 207518370] [147
> 137] [72 86] [96 274005961] [49 207518365] [31 57830745] [152 1307] [48
> 928] [104 295874761] [81 21735] [167 302087295] [162 167699] [68 25] [1
> 826] [57 23] [119 291018089] [12 302087311] [7 912] [139 193] [134 539]
> [205 28] [137 1299] [34 49] [18 200] [4 7644696] [14 7] [35 59021633] [1526
> 302087296] [58 73] [120 830] [163 65] [178 1293] [80 870] [1527 302087294]
> [95 274005961] [40 157758900] [77 302087312] [116 101387066] [111 112] [41
> 57830745] [269911 302087303] [269910 193] [269912 57830745] [269909
> 57830745] [269908 57830745] [269914 57830745] [269913 302087292] [195 65]
> [1248 207518367] [2472 800] [2487 136] [2489 765] [2491 488] [2467
> 207518365] [2478 830] [2485 800] [2475 207518370] [2496 793] [2501
> 207518365] [2486 765] [2466 488] [93 302087302] [126 136] [23 105] [15
> 50105231] [115 110] [136 299] [121 800] [129 57830729] [56 232889320]
> [56434 8543] [276 939] [70 75] [106 302087301] [98 229] [110 57830742] [45
> 67] [24 883] [107 904] [124 182796497] [99 89] [47985 302087314] [47983
> 81057077] [47982 223071] [47989 294247] [47988 492094] [47991 82005831]
> [47987 228539865] [47986 81057081] [47990 136333789] [47984 137976183]
> [49100 563] [49098 82071940] [49093 136059519] [49095 108941256] [49116
> 82065378] [49114 82063765] [49118 82063767] [49117 82063764] [49121
> 82063916] [49120 16973] [49115 63121] [49119 88397486] [49150 82064260] [66
> 302087301] [5 302087302] [63 31] [37 287849745] [51 25] [28 302087305] [73
> 3927] [485 28] [59 125] [13 302087310] [91 780] [54 94] [127 299] [11 765]
> [1528 488] [156 65] [215 28] [239 302087304] [21 295871169] [38 302087315]
> [25 882] [114 17] [101 57830743] [86 57830733] [39 302087308] [297092
> 195213941] [131 302087293] [130 802] [22 923] [1246 302087313] [2494
> 302087300] [2462 68508392] [2473 57830743] [2457 3927] [67 1326] [133
> 302087305] [71 136778] [65 207518365] [2 169693143] [103 167718] [19 763]
> [1529 302087289] [2461 28] [2453 302087302] [2500 200] [2458 21735] [2493
> 302087302] [2477 295871174] [2464 200] [2492 539] [2459 8224991] [2476
> 167718] [2483 38] [2495 302087291] [2451 136778] [2460 91] [2488 17] [2454
> 67] [2469 112] [2463 302087305] [2470 302087302] [141 1317] [88 302087297]
> [52 929] [1250 8224986] [53 302087300] [123 46] [75 302087307] [135
> 57830744] [2246 302087298] [2471 302087309] [2455 539] [2452 302087302]
> [2465 188476291] [2484 188476291] [nil 302087288] [2497 1326] [2590
> 302087290] [2498 302087311] [33 25] [83 302087305] [79 87] [89 38] [188
> 140] [69 169693143] [8 302087306] [122 13] [140 1283] [84 94] [1249
> 295871170] [2479 295871169] [2525 295871169] [2474 59021633] [2468
> 59021633] [164 65] [92 295871169] [456928 36] [6 1608] [248 302087299] [16
> 57830733] [1015 104292798] [174 236] [128 302087302] [20 68508392] [102
> 765] [113 793] [125 80] [26 765] [118 101387069] [9 296645235] [30 290]
> [108 800] [76 302087302] [183 44875635] [158 296658966] [1530 8224991] [278
> 938] [29 295871174] [10 91] [105 302087291] [27 302087302] [142 181]
> [263862 302087287] [50 67] [196 167697] [1531 268074168] [2482 266778936]
> [2490 57830732] [2499 116471] [2481 929] [109 928] [43 2780] [146 65] [1532
> 31] [97 59021633] [36 67] [254 28] [3 1671] [87 200] [62 802] [181 912]
> [191 1299] [74 904] [60 57830733] [44 302087297] [94 296930720] [85
> 59021638])], :schema [#schema.core.One{:schema Int, :optional? false, :name
> factset-id} #schema.core.One{:schema (either [[(one Int "path-id") (one Int
> "value-id")]] #{[(one Int "path-id") (one Int "value-id")]}), :optional?
> false, :name pairs}], :type :schema.core/error}
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To view this discussion on the web visit
> <https://groups.google.com/d/msgid/puppet-users/046f28c4-867c-40ab-86c1-e09bef5d9a80%40googlegroups.com?utm_medium=email&utm_source=footer>
> https://groups.google.com/d/msgid/puppet-users/046f28c4-867c-40ab-86c1-e09bef5d9a80%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
> Hey Akos, thanks for the report. Please take a look at
> https://tickets.puppetlabs.com/browse/PDB-2034 and see if it matches your
> scenario.
>
> We've seen this bug before in relation to a fact generated by certain
> versions of the puppet-community rundeck module. The initial fix was here:
> https://github.com/voxpupuli/puppet-rundeck/pull/138 but I think that the
> module has been refactored since.
>
> The bug on the PuppetDB side was related to failed processing of facts
> that started with an escaped quote but did not end with one. To investigate
> it in PuppetDB, you might try looking in
> /var/lib/puppetdb/mq/discarded/replace-facts to see if any failed commands
> have made it through. If there are messages in there, you'll be able to
> determine the certname for the host that originated the facts, and then
> search either the facts in the message or the facter output on that box for
> fact names starting with an escaped quote.
>
> If there are no messages getting discarded tracking it down is a bit
> trickier, but I'd be happy to help with that.
>
> Wyatt
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/acd231a1-768d-405b-91b5-932d42de4928%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
