Hi Sean, Thank you for replying in detail Pls see inline: On Thu, May 28, 2026 at 8:58 AM Sean Owen <[email protected]> wrote:
> Asif, I'm not familiar with your specific changes, but, I think you are > misreading how a big OSS project works, and not helping anyone including > yourself. > > Thousands of people have proposed changes to Spark in its 12 years; there > are tens of thousands of JIRAs and pull requests. From experience, most are > noise. > Yours may or may not be. But, you will not be nearly the first person that > said "hey my change is important and made sense to me! why wasn't it > accepted?" > The explanation is simply not going to be because a 'cartel' got > together and decided it would be, what, easier to go steal, modify, > duplicate your work, in public, and alienate you, instead of just pressing > 'merge' on your work? > The far more likely explanation is that nobody has even looked at it, not > in any detail. > Pls notice, that this email chain has not started , as to why the my change was not taken.. It started because exact same change with some mods was taken, while the original was still open. At some point I was keen to get changes in the project, but I have myself closed those PRs. > > There are no paid staff here. There are no rewards for contributing, or > merging a PR. There is nobody that is obligated to review your proposals, > either. > I'm sure some great PRs have been overlooked as a result, but that's just > life. The game here is to play nice and make clear succinct changes that > stand a chance of getting attention. People do it all the time. > I am not sure about the altruistic aspect. I wonder how many committers would there be working for companies / or self employed, who have no direct interest in apache spark. > Committers, certainly, have far more influence in the project, by design. > Committers get that status by contributing a lot. Often people that > contribute a lot are people for whom this is part of a day job. Those > people have a set of things they want to work on and not others. They have > credibility to get attention for reviews and power to merge. None of that > is nefarious. It's how OSS projects just work. > People get committer bits regularly. We just added like 3 of them? > Yes. I understand that. And I also understand the "process" of becoming committer . and I also agree nothing nefarious there. > > I bet you aren't the first to feel this way at some point, but I dont' > recall any message like yours claiming a conspiracy to steal credit, no, > because it just does not map to how an OSS project like this works. > You want your code reviewed, you want appreciation here, but 99% of what > people know about you so far is this angry message alleging a conspiracy. > I am not looking for appreciation. Its the question of transfer of credit and ethics. I am not alleging conspiracy. I am just stating how it works... > > I'd drop this line and focus on working constructively with people whose > interests clearly overlap with yours, like Peter, instead of writing to the > whole list about cartels. > > > > On Thu, May 28, 2026 at 10:33 AM Asif Shahid <[email protected]> > wrote: > >> Hi Peter, >> Pls see inline for comments/ replies >> >> On Thu, May 28, 2026 at 6:11 AM Peter Toth <[email protected]> wrote: >> >>> Hey Asif, >>> >>> Are you referring to https://github.com/apache/spark/pull/49154/changes >>> vs. https://github.com/apache/spark/pull/55644/changes? Those are >>> definitely solving the same issue but I can assure you I wouldn't take any >>> code from your PR without consulting with you first. >>> >> Yes Indeed Peter, I am referring to those. >> As for the fix, itself, is not indicative of any thing as its a one >> liner, test has uncanny resemblance. >> >> >>> As far as I remember, I opened SPARK-56694 / >>> https://github.com/apache/spark/pull/55644 because I ran into that >>> minor bug during the implementation of >>> https://github.com/apache/spark/pull/55298. >>> >> >> >>> Sorry, I didn't check whether a ticket or PR already existed. >>> >> >> The below I am addressing to the whole cartel.: >> I have experienced this before, as recent as couple of months back ( >> https://issues.apache.org/jira/browse/SPARK-54386) >> I have experienced, my personal effort ( going into weeks) to debug, >> reproduce issue reliably , being hijacked by members, without even >> discussing the fix proposed, ( by opening new PRs). ( If interested, I can >> provide details of the PRs / issues I am talking about) >> I have seen a perfectly valid PR being nixed , by following comment which >> essentially said >> " my code of making the cache lookup more effective , would result in >> greater chances of stale cache being picked, which already spark suffers >> from." >> Now the PR was related to collapsing the projects in analysis phase, and >> side effect was cache pick up being more sensitive. >> So this is such a frivolous reason to nix the PR , because "staleness" is >> an underlying existing issue which had nothing to do with my PR. And its >> more amusing , that if a DB is giving even one wrong result in millions, >> that makes all the results a suspect in any case. It does not matter at >> what frequency this occurs. To me the real reason was code complexity ( & >> more likely the loss of control of the code to the outsider). >> >> The reason I call this open source community as cartel, is because, I >> have seen the way it works pretty closely and have experienced it in the >> email exchanges which happen on this group. >> For the same PR , same issue, if advertently or inadvertently , other >> person ( especially a member) gets his changes pushed, by the virtue of his >> standing/position and the "for profit" company the person works, how would >> you give the credit to the original person who discovered the issue first / >> provided the fix? >> Why are issues filed by some immediately worked upon by members ( some of >> whom claim to be working full time on spark) ? Is it because certain >> companies / groups ( for profit companies, mind you ) exert undue >> control, or the petty newbee has to be in the good books of members ( with >> the hope that at some point they will also reach that position of power ?) >> >> Given the AI advent and such occurrences, how will you give due credit >> to the original creators and how do you plan to prevent some member for >> taking up idea of any old open PR ( which for reasons of complexity and non >> technical reasons) , polishing it up and pushing it as their own? >> >> I am also curious , am I the only one who is troubled by all this, or >> there are others who have experienced it? >> >> Regards >> Asif >> >> >>> If you have further improvements please feel free to open a PR. >>> >>> Best, >>> Peter >>> >>> On Thu, May 28, 2026 at 8:20 AM Asif Shahid <[email protected]> >>> wrote: >>> >>>> Hi, >>>> I had filed a bug >>>> https://issues.apache.org/jira/browse/SPARK-45866 >>>> >>>> I had also opened a PR for the same. >>>> >>>> Now I see that the ticket I filed is still open, but the issue has >>>> been fixed using a new ticket >>>> https://issues.apache.org/jira/browse/SPARK-56694 >>>> >>>> and on top of that the bug test and ofcourse the fix ( which in any >>>> case would be same) has been taken from my PR for >>>> https://github.com/apache/spark/pull/49154/changes#diff-137d880ff73623bf7a452bb84f9c3dbbb27ba929e7f5e070c6bff68cfc8ec71f >>>> >>>> To me this is clear unethical conduct of cartel member, unless I am >>>> missing some valid reason. >>>> >>>> And the irony is that the fix is still incomplete, as I just found and >>>> filed a new ticket >>>> https://issues.apache.org/jira/browse/SPARK-57126 >>>> >>>> I know that atleast some cartel members are insecure and think of OSS >>>> as their fiefdom, but this sort of behaviour , I never expected. >>>> Regards >>>> Asif >>>> >>>
