I was just reminded of another problem: adaptTo (especially within service code) makes it hard to write unit tests. It's hard to mock.
An adaptTo-only example is the sling ResourceCollectionManager that you can only retrieve via resourceResolver.adaptTo(), but there is no corresponding service. Should I create an issue? Cheers, Alex