New question #109514 on Graphite: https://answers.launchpad.net/graphite/+question/109514
I'm running 0.9.6. Not all data points that I send to carbon are getting into the whisper data files. I suspected this because I saw holes in the graphs, so I devised a test by siphoning off about 1 1/2 hours of data from my data stream, and ran a second client against the siphoned file (it has ~52000 lines in it). carbon.conf is the default example file storage-schemas.conf: [opsview_stg] priority = 100 pattern = ^staging\.* retentions = 300:25920,900:70080 [opsview_bugtest] priority = 99 pattern = ^bugtest\.* retentions = 300:25920,900:70080 I ran my client against the data in the test file, and it seemed to complete fine, but not every data point was stored in whisper. From my client's log, for example, the following lines were sent (many more lines were sent of course, but this is the grepped output to show the problem): bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271896572 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271896872 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271897172 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271897472 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271897772 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898072 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898372 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898672 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271898972 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271899272 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271899572 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271899872 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271900172 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271900472 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271900772 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271901072 bugtest.aibpdap1_mycompany_com.ssh_check_linux_disk_-opt.available 512 1271901367 Yet when I look at the data in the whisper data file I see this: whisper-fetch.py --from=1271896200 --until=1271901600 /opt/graphite/storage/whisper/bugtest/aibpdap1_mycompany_com/ssh_check_linux_disk_-opt/available.wsp 1271896500 None 1271896800 None 1271897100 None 1271897400 None 1271897700 None 1271898000 512.000000 1271898300 None 1271898600 None 1271898900 None 1271899200 512.000000 1271899500 None 1271899800 None 1271900100 None 1271900400 512.000000 1271900700 None 1271901000 None 1271901300 512.000000 1271901600 None I suspect there is some weird buffer overrun going on, but I don't know what. To test this, I restricted the output to just data from the machines aibpdap1 and aibpdap2, added a third config to storage-schemas.conf [opsview_liltest] priority = 99 pattern = ^liltest\.* retentions = 300:25920,900:70080 Result, the data got stored OK. whisper-fetch.py --from=1271896200 --until=1271901600 /opt/graphite/storage/whisper/liltest/aibpdap1_mycompany_com/ssh_check_linux_disk_-opt/available.wsp 1271896500 512.000000 1271896800 512.000000 1271897100 512.000000 1271897400 512.000000 1271897700 512.000000 1271898000 512.000000 1271898300 512.000000 1271898600 512.000000 1271898900 512.000000 1271899200 512.000000 1271899500 512.000000 1271899800 512.000000 1271900100 512.000000 1271900400 512.000000 1271900700 512.000000 1271901000 512.000000 1271901300 512.000000 1271901600 None My client is configured to send data to the carbon-cache daemon in groups of about 400 data lines. I've tried reducing this to no avail, and tried sleeping between sends, also to no avail. **** The following may be a related problem **** I can't look at the logs because the only log file is console.log. I see in the code where there should be others, but they don't appear. Any help gratefully accepted. Thanks, Steve Keller -- You received this question notification because you are a member of graphite-dev, which is an answer contact for Graphite. _______________________________________________ Mailing list: https://launchpad.net/~graphite-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~graphite-dev More help : https://help.launchpad.net/ListHelp
