bd808 added a comment.
In T206636#4946372, @Smalyshev wrote:
Not sure if virtualization that we do now allows such kind of fixed resource allocations (probably also I/O resources need to be taken care of?)
We have a few special instances that we do this for today. It is something that we are
Smalyshev added a comment.
Yes, judging from our preliminary test, if we get uncontested use of the server or even a certain chunk of it maybe (not sure if possible?) it would be enough. Note that an interesting scenario that we want to test in foreseeable future involves cluster setup, so we'd
bd808 added a comment.
The next step here is to determine the hardware requirements and then find out if there is current fiscal year budget to cover procuring the required hardware, or if we can allocate a dedicated server from the existing cloudvirt pool and backfill that capacity.
The
Smalyshev added a comment.
In the meantime, can I delete t206636-3?
Yes.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana,
Andrew added a comment.
So it sounds like you will need dedicated hardware to make this work (if the original test with the t206636 VM was adequate.) The analytics team is experimenting with this model currently so we'll see how it goes. You would need to figure out about
Smalyshev added a comment.
Tried with incoming stream, the machine can't keep up - by now it's 5 hours behind, and processing about half the necessary requests.
I think the conclusion is mostly clear - this VM as such is not suitable for any performance testing or any workloads that are close to
Smalyshev added a comment.
Updater seems to be able to get about 4-5 updates per second, which is abut 2x slower than production. Summarily, it looks like this setup may be fit for functionality testing, but decidedly unfit for any performance testing, as it is 2-3x slower than production.
Final
Smalyshev added a comment.
Loading finished, overall took 8 days and 9 hours, or 201 hours, or 3x compared to production. Launching updater next to see the updates speed.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL
Smalyshev added a comment.
Looks like this VM is substantially slower - I started data load at Nov 9, now is end of Nov 13, and it's only 75% done, which means it'll take about a week to load all data, which is significantly slower than in production (it was done in 63 hours).
However, the real
Smalyshev added a comment.
Data loading test launched for t206636-3 and should take several days to complete. Ideally, it should not be interrupted. Please ping me if VM needs to be rebooted, etc. - it's possible to recover but to have accurate record of timings I need to know when it
Smalyshev added a comment.
I've build a new VM t206636-3 that should have the same specs as your original test VM. Is it OK if I delete -2 now?
Yes, sure. I'll use t206636-3.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL
Andrew added a comment.
In T206636#4690379, @Smalyshev wrote:
I've created a new VM, t206636-2.wikidata-query.eqiad.wmflabs
I see that t206636-2 is listed as having 4 VPU and 24G RAM. That sounds too small for what I'd need - is this accurate?
That's my mistake -- I've build a new VM t206636-3
Gehel added a comment.
In T206636#4690384, @Smalyshev wrote:
@Andrew Also looks like there is some puppet issue there:
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error
Smalyshev added a comment.
@Andrew Also looks like there is some puppet issue there:
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call,
Smalyshev added a comment.
I've created a new VM, t206636-2.wikidata-query.eqiad.wmflabs
I see that t206636-2 is listed as having 4 VPU and 24G RAM. That sounds too small for what I'd need - is this accurate?TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL
Andrew added a comment.
I've created a new VM, t206636-2.wikidata-query.eqiad.wmflabs. This is in the older region, on a host that is not super busy but is supporting quite a few other VMs. If your tests look good there too then we're probably in good shape and can avoid needing special hardware
Smalyshev added a comment.
Run a second set of tests on a similar VM that shares a host with other active cloud instances (to see if we can meet your needs as a standard user), or
Sure. This would take a bit more time though - we'll need to load a new dump (probably several days) and run updater
Andrew added a comment.
Thanks, Stas. There are two ways I think we can go forward with this:
Run a second set of tests on a similar VM that shares a host with other active cloud instances (to see if we can meet your needs as a standard user), or
Start talking about buying you a private virt
Smalyshev added a comment.
I did a short evaluation on provided VM and it looks like it behaves well enough to run Blazegraph tests on it. So if we get something like that, we could probably do tests there. The t206636 can be shut down now, if necessary.TASK
Andrew added a comment.
note to self, I can merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/468377/ after Stas releases this VM (or at least stops caring about resource contention)TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL
Smalyshev added a comment.
I definitely see role::wdqs::labs as an available class in Horizon, in the 'Puppet Configuration' tab.
I found it, I was looking at the wrong region. Dumps are ok now too, so I think I've got enough to start testing.TASK
Andrew added a comment.
@Smalyshev, dumps should be available on that host now.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc,
Andrew added a comment.
Sorry! Since you mentioned having tested this on a VM in the past I assumed you had a plan in mind to set something up for testing.
All of the storage for that instance is hosted on SSDs. I can partition the extra space for you if you want -- you would just want to add
Smalyshev added a comment.
@Andrew I logged in there and I see this set of disks:
Filesystem Size Used Avail Use% Mounted on
udev65G 0 65G 0% /dev
tmpfs 13G 19M 13G 1% /run
/dev/vda3
Smalyshev added a comment.
@Andrew thanks, I'll start testing on it tomorrow.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc,
25 matches
Mail list logo
