I left a few comments, but overall this sounds like a good plan to me - thanks for the writeup!
On Tue, Aug 16, 2022 at 9:36 AM Anand Inguva via dev <dev@beam.apache.org> wrote: > Hi, > > I created a doc > <https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit#>[1] > which outlines the plan for the RunInference API[2] benchmark/performance > tests. I would appreciate feedback on the following, > > - Models used for the benchmark tests. > - Metrics calculated as part of the benchmark tests. > > > If you have any inputs or any suggestions on additional metrics/models > that would be helpful for the Beam ML community as part of the benchmark > tests, please let us know. > > [1] > https://docs.google.com/document/d/1xmh9D_904H-6X19Mi0-tDACwCCMvP4_MFA9QT0TOym8/edit# > [2] > > https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269 > <https://github.com/apache/beam/blob/67cb87ecc2d01b88f8620ed6821bcf71376d9849/sdks/python/apache_beam/ml/inference/base.py#L269> > > > Thanks, > Anand >
