Hey Shaunak, You might want to run a set of tests across the parameter space of all the relevant variables:
Vary the total number of tests: N = number of tests total Vary the proportion of N taken up by each type of test: N_a = number of tests of type a N_b = number of tests of type b . . . N_e = number of tests of type e . . . Vary the parameters of the tests themselves: a_x = critical parameter x involved in running a test of type a a_y = critical parameter y involved in running a test of type a . . . e_x = critical parameter x involved in running a test of type e . . . You would script a series of runs of test suites defined by their position in "test suite parameter space" (N, N_a, N_b, . . . , N_e, . . . , a_x, a_y, . . . , e_x, . . .), programmatically logging whether the test suite completed or crashed, how long it took before it crashed if so, and you could possibly observe correlations between crashing and certain kinds of tests and/or rule out which variables seem to have no impact on whether it crashes or not. * * * Beyond that, though, it seems you'd need to either go source-code diving or get the assistance of someone who actually works on the codebase of the go SDK and aetest. You might want to file an issue in the github issue tracker for aetest <https://github.com/golang/appengine> as soon as you can get more info from a more rigorous test of test suite parameters as described above. Cheers, Nick Cloud Platform Community Support On Wednesday, May 10, 2017 at 1:07:39 PM UTC-4, Shaunak Godbole wrote: > > Hello, we are using aetest to write some integration tests as part of our > project. The integration tests spin up an instance of a dev_appserver.py > in a *BeforeSuite* function. We use APIs to guarantee that the suite > executes before we start the test: > > func BeforeSuite() { > params := aetest.InstanceParams{StrongConsistency: true} > inst := aetest.NewInstance(params) > req := aetest.NewRequest("Get", "/", nil) > ctx := appengine.NewContext(req) > } > > We have a similar *AfterSuite* that tears down the instance. > > func AfterSuite() { > inst.Close() > } > > Since we want the tests to be completely hermetic, we delete all the > entities that we had created during the tests, usually in an *AfterEach* > method. > > func AfterEach() { > deleteAllEntitiesFromDatastore("ENTITY_1") > deleteAllEntitiesFromDatastore("ENTITY_2") > deleteAllEntitiesFromDatastore("ENTITY_3") > ... > } > > And, the we have a lot of test cases that simply run against the created > instance and use the context acquired through *BeforeSuite*. We have a > good mixture of Gets, Puts, Entity Queries, Filter Queries etc. Since a > few days, every-time we run our tests, the tests hang at some point during > the execution. This point changes as we add and remove some of the tests, > but the tests always hang if the number of suites are above a certain > threshold. We usually see errors like: > > <string>: Call error 11: Deadline exceeded (timeout) > > > or like: > > Expected success, but got an error: > <*url.Error | 0xc8210570b0>: { > Op: "Post", > URL: "http://localhost:59072";, > Err: {s: "EOF"}, > } > Post http://localhost:59072: EOF > > > While the tests are hung, if we try to access the datastore through the ip > produced in the tests, the request just hangs similar to how it hangs > during the tests. This makes us suspicious that there is some kind of a > deadlock that is getting introduced during our execution, but we are not > able to figure out how to debug it. > > Also, the test suites have a timeout after which the tests simply crash. > At this point, the AfterSuite cannot execute so the test-server remains > active in the background, and when we try to access datastore through the > admin, we can see all the data that was added as part of the test. > > For completeness sake, we also use gomocks to abstract out some external > services as part of the tests. > > Any help on debugging this would be very much appreciated. > > Thanks, > Shaunak > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/96fd7d98-d4c9-4631-ae45-0b7318863818%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
