gaogaotiantian commented on PR #57122: URL: https://github.com/apache/spark/pull/57122#issuecomment-4986389565
I have a few questions about this PR: 1. Is it intentional to include the test scripts in `MANIFEST.in`? Why do we need to package these files for pyspark? 2. Why do we choose to have a million test fixture files which are invalid Python, rather than a file with test cases which can be a valid Python? The test results are aggregated and compared in a single test file, but the fixtures are different files - that does not quite make sense to me. We had to do some tricks like injecting `spark` to global namespace - that's a bit unusual. I think we can achieve the exact same thing if we just put everything in a single file and make every test fixture a method. 3. Do we expect spark end users to use this framework? Is it officially supported as a spark feature? `pyspark/testing` is user-interfacing and having `df_golden.py` there means this is usable by users and we need to maintain the utility (and keep backwards compatibility, which could be some effort). 4. Is the golden file format matching the Scala SQL or it's something new we design for Python side? 5. What is the ultimate goal for this framework? Do we expect more tests written in this format? I don't think this is very conventional in Python world. Is there some specific benefit from it? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
