jbewing commented on PR #15410: URL: https://github.com/apache/iceberg/pull/15410#issuecomment-4332072248
Sorry to keep you waiting here @pvary. > Is there a way to add tests to the arrow package? I really hate that we change something here and only test things through Spark. Especially troubling considering the size of this refactor. I'm sure that you could—although that decision predates my working in here by a large margin. I'm also inclined to say that while things could be bolstered in the arrow package directly, I think that the integration tests that we currently have lined up that live in the spark modules (some could be added for flink as well) are likely more valuable just given that they more closely represent paths that real users will take when using these modules. There are still quite a few layers of abstraction built on top of the actual raw parquet file readers in terms of mapping the types to parquet types, etc... All to say, I understand your concern about not having really lightweight tests here, however, I think that the coverage that we do have is more valuable than the presence of lightweight tests in the arrow module. This sounds like a wonderful idea for a follow up that someone could take on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
