Hi all, I happen to come up with another idea about pandas redesign. Thanks Reynold, Bryan, Xiangrui, Takuya and Tim for offline discussions and helping me to write this proposal.
Please take a look and let me know what you guys think. - https://docs.google.com/document/d/1-kV0FS_LF2zvaRh_GhkV32Uqksm_Sq8SvnBBmRyxm30/edit?usp=sharing - https://issues.apache.org/jira/browse/SPARK-28264 I know it's a holiday season but please have some time to take a look so we can make it on time before code freeze (31st Jan).
