hi Jacob — that's great to hear. We're standing by to help you out with this.
On Fri, Jun 25, 2021 at 4:57 PM Jacob Quinn <quinn.jac...@gmail.com> wrote: > > Hi Kou, > > Sorry for the slow response here, but it's been great to see how the new > Rust process has shaken out and I think it working well. I'd like to move > forward with transferring the JuliaData/Arrow.jl repository to > apache/arrow-julia and following a similar process to Rust in terms of > development/release. I can start on a Julia-specific proposal with > specifics. > > Thanks for all the help! > > -Jacob > > On Sun, Apr 25, 2021 at 11:56 PM Sutou Kouhei <k...@clear-code.com> wrote: > > > Hi, > > > > I think that we can say the Rust migration is complete once > > we merge https://github.com/apache/arrow/pull/10096. But > > it's a good time to think about the Julia migration. > > > > Jacob, here is the Rust's new development process: > > > > > > https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit# > > > > (It seems that an anonymous user deleted a part of it > > accidentally.) > > > > Do you want to use the same development process as the > > Rust's one? Do you have any item you want to change? > > > > > > Thanks, > > -- > > kou > > > > In <cak7z5t96rapgltnasg6v+hxd7qqreuhxa5xzt7wcdv_gelh...@mail.gmail.com> > > "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04 > > -0700, > > Micah Kornfield <emkornfi...@gmail.com> wrote: > > > > > It seems the Rust migration is now complete. Do we want to wait to iron > > > out the other potential issues? > > > > > > I think the outstanding ones might be: > > > 1. Issue management > > > 2. Integration testing > > > > > > -Micah > > > > > > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei <k...@clear-code.com> > > wrote: > > > > > >> Hi Jacob, > > >> > > >> OK. Here is my plan: > > >> > > >> 1. We wait for the Rust's move to complete > > >> 2. We use a process similar to the Rust's move > > >> > > >> > > >> Thanks, > > >> -- > > >> kou > > >> > > >> In <cakyxbqrt6yahesuqg8beuh6u58smc95jtgejej2kuy0zrgy...@mail.gmail.com> > > >> "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021 > > 08:37:41 > > >> -0600, > > >> Jacob Quinn <quinn.jac...@gmail.com> wrote: > > >> > > >> > Thank you kou! I appreciate the help. I'm happy to do whatever is > > >> required > > >> > to facilitate the moving/donating process from JuliaData/Arrow.jl to > > >> > apache/arrow-julia. > > >> > > > >> > -Jacob > > >> > > > >> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei <k...@clear-code.com> > > wrote: > > >> > > > >> >> Hi Jacob, > > >> >> > > >> >> I, a PMC member, talked to Kenta Murata, a commiter and a > > >> >> Julia user, about this. > > >> >> > > >> >> We support that you and Julia folks work on > > >> >> arrow/arrow-julia until we have enough PMC members from > > >> >> Julia folks. For example, we'll help IP clearance process to > > >> >> import the latest JuliaData/Arrow.js changes to apache/ and > > >> >> we'll start voting on Julia package release. > > >> >> > > >> >> > > >> >> Thanks, > > >> >> -- > > >> >> kou > > >> >> > > >> >> In <CAKyXBQo2=+NfS=hX6nEDPmMno-bSuXMR+= > > d-heskgn2mm57...@mail.gmail.com> > > >> >> "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021 > > >> 23:06:27 > > >> >> -0600, > > >> >> Jacob Quinn <quinn.jac...@gmail.com> wrote: > > >> >> > > >> >> > Micah/Wes, > > >> >> > > > >> >> > Yes, I've been following the rust proposal thread with great > > >> interest. I > > >> >> do > > >> >> > think that provides a great path forward: transferring the > > >> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve > > the > > >> >> > "package history" technical challenges that in part led to the > > current > > >> >> > setup and concerns. I think being able to utilize github issues > > would > > >> >> also > > >> >> > be great; as I've mentioned elsewhere, it's much more > > >> >> traditional/expected > > >> >> > in the Julia ecosystem. > > >> >> > > > >> >> > I think the package could retain an independent versioning scheme. > > The > > >> >> >> additional process would be voting on release candidates. If the > > >> Julia > > >> >> >> folks want to try again and move development to a new, > > Julia-specific > > >> >> >> apache/* repository and apply the ASF governance to the project, > > the > > >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In > > some > > >> >> >> code donations / IP clearance, the contributors for the donated > > code > > >> >> >> become committers as part of the transaction. > > >> >> >> > > >> >> > > > >> >> > These all sound great and would greatly facilitate a better > > >> integration > > >> >> > under ASF governance. These points definitely resolve my main > > >> concerns. > > >> >> > > > >> >> > As I commented on the rust thread, I'm mostly interested in the > > >> future of > > >> >> > integration testing for rust/julia if they are split out into > > separate > > >> >> > repos. In the current Julia implementation, we have all the code to > > >> read > > >> >> > arrow json, and I just hand-generated the integration test data and > > >> >> > committed them in the repo itself, but it doesn't interface with > > other > > >> >> > languages (just reads arrow json, produces arrow file, reads arrow > > >> file, > > >> >> > compares w/ original arrow json). I'm happy to help work on the > > >> details > > >> >> of > > >> >> > what that looks like and pilot some solutions. I think with a solid > > >> >> > inter-repo integration testing framework, we can keep a strong sync > > >> >> between > > >> >> > projects. > > >> >> > > > >> >> > -Jacob > > >> >> > > > >> >> > > > >> >> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney <wesmck...@gmail.com> > > >> >> wrote: > > >> >> > > > >> >> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield < > > >> emkornfi...@gmail.com> > > >> >> >> wrote: > > >> >> >> > > > >> >> >> > > > > >> >> >> > > Ok, I've had a chance to discuss with a few other Julia > > >> developers > > >> >> and > > >> >> >> > > review various options. I think it's best to drop the Julia > > code > > >> >> from > > >> >> >> the > > >> >> >> > > physical apache/arrow repo. The extra overhead on development, > > >> >> release > > >> >> >> > > process, and user issue reporting and PR contributing are too > > >> much > > >> >> in > > >> >> >> > > addition to the technical challenges that we never resolved > > >> >> involving > > >> >> >> > > including the past Arrow.jl release version git trees in the > > >> >> >> apache/arrow > > >> >> >> > > repo. > > >> >> >> > > > >> >> >> > > > >> >> >> > Hi Jacob, > > >> >> >> > It seems you are on the new thread discussing a proposal for > > >> changing > > >> >> >> > Rust's development model. Would the proposal [1] address most > > of > > >> >> these > > >> >> >> > concerns if Julia was set up in the same way? > > >> >> >> > > > >> >> >> > It seems in the short term the stickiest point would be > > committer > > >> >> access > > >> >> >> > to the new repos, and I suppose the release mechanics still > > might > > >> be > > >> >> >> > challenging? > > >> >> >> > > >> >> >> I think the package could retain an independent versioning scheme. > > >> The > > >> >> >> additional process would be voting on release candidates. If the > > >> Julia > > >> >> >> folks want to try again and move development to a new, > > Julia-specific > > >> >> >> apache/* repository and apply the ASF governance to the project, > > the > > >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In > > some > > >> >> >> code donations / IP clearance, the contributors for the donated > > code > > >> >> >> become committers as part of the transaction. > > >> >> >> > > >> >> >> > > > >> >> >> > Thanks, > > >> >> >> > Micah > > >> >> >> > > > >> >> >> > [1] > > >> >> >> > > > >> >> >> > > >> >> > > >> > > https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit > > >> >> >> > > > >> >> >> > On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney < > > wesmck...@gmail.com> > > >> >> wrote: > > >> >> >> > > > >> >> >> > > I went back and read the mailing list discussions from > > September > > >> >> about > > >> >> >> > > the donation and I would say there was not a clear enough > > >> statement > > >> >> >> > > from us about what the donation and IP clearance meant as far > > as > > >> the > > >> >> >> > > future of the Julia codebase. This is partly our fault — we > > have > > >> >> taken > > >> >> >> > > in 9 other code donations over the last 5 years, and in all > > cases > > >> >> the > > >> >> >> > > developers understood that they were to move their process to > > the > > >> >> >> > > Arrow repositories and communications channels. > > >> >> >> > > > > >> >> >> > > It did not occur to me at all that the code that you were > > >> putting in > > >> >> >> > > the Arrow repository would get treated like a read-only fork > > that > > >> >> you > > >> >> >> > > update periodically. If I had realized that, we wouldn't be in > > >> this > > >> >> >> > > situation. > > >> >> >> > > > > >> >> >> > > As a reminder about what Arrow and the ASF are all about: > > >> Community > > >> >> >> > > over Code. We think that building a collaborative, open > > community > > >> >> that > > >> >> >> > > works and plans together in public, makes decisions based on > > >> >> consensus > > >> >> >> > > with clear meritocratic ("doers decide") governance is the > > best > > >> way > > >> >> to > > >> >> >> > > build this project. The concerns that you have around the > > timing > > >> and > > >> >> >> > > frequency of releases for the Julia codebase are in my mind > > easy > > >> to > > >> >> >> > > resolve, and if you had indicated that having a customized > > >> process > > >> >> for > > >> >> >> > > Julia releases was a condition for your joining the community > > >> >> >> > > wholeheartedly, we would have been happy to help. I think that > > >> the > > >> >> >> > > benefits of common CI/CD infrastructure and opportunities to > > >> build > > >> >> >> > > deeper integrations between the Julia implementation and the > > >> other > > >> >> >> > > implementations (imagine... Julia kernels running in > > DataFusion?) > > >> >> >> > > would outweigh the sense of "loss of control" from developing > > >> >> within a > > >> >> >> > > larger project. > > >> >> >> > > > > >> >> >> > > On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn < > > >> quinn.jac...@gmail.com > > >> >> > > > >> >> >> > > wrote: > > >> >> >> > > > > > >> >> >> > > > Responses inline below: > > >> >> >> > > > > > >> >> >> > > > On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão < > > >> >> >> > > > jorgecarlei...@gmail.com> wrote: > > >> >> >> > > > > > >> >> >> > > > > Hi, > > >> >> >> > > > > > > >> >> >> > > > > > you all did not attempt to work in the community for any > > >> >> >> meaningful > > >> >> >> > > > > amount of time and > > >> >> >> > > > > are choosing not to try based on the perception that it > > will > > >> >> create > > >> >> >> > > > > unacceptable overhead for you > > >> >> >> > > > > > > >> >> >> > > > > It is not self-evident to me that Julia's community was > > >> >> >> sufficiently > > >> >> >> > > > > informed about what they > > >> >> >> > > > > had to give in in terms of process and release management > > >> when > > >> >> >> merging > > >> >> >> > > / > > >> >> >> > > > > donating. > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > Yes, it was pretty unclear what the process was if we needed > > >> to do > > >> >> >> any > > >> >> >> > > kind > > >> >> >> > > > of patch release. I know that has been sorted out better > > >> recently, > > >> >> >> but > > >> >> >> > > back > > >> >> >> > > > in November, it didn't really seem like an option (i.e. > > >> >> independent > > >> >> >> > > > language patch releases). > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> > > > > IMO this is a plausible explanation as to why the donation > > >> was > > >> >> >> made and > > >> >> >> > > > > then later abandoned. > > >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > I'll just note that the "abandonment" can only be a > > perception > > >> >> from > > >> >> >> the > > >> >> >> > > > apache/arrow side of things, but as I mentioned above, I > > also > > >> >> tried > > >> >> >> to > > >> >> >> > > > clearly state in the julia/Arrow/README that the development > > >> >> process > > >> >> >> > > would > > >> >> >> > > > continue with the JuliaData/Arrow.jl repo as the main "dev" > > >> >> branch, > > >> >> >> with > > >> >> >> > > > changes being upstreamed to the apache/arrow repo, which was > > >> >> followed > > >> >> >> > > > through, having an upstream of commits right before the > > 3.0.0 > > >> >> >> release, > > >> >> >> > > and > > >> >> >> > > > I was planning on doing the same soon for the 4.0.0 release. > > >> That > > >> >> is > > >> >> >> to > > >> >> >> > > > say, the Julia implementation has continued progressing > > forward > > >> >> quite > > >> >> >> > > > rapidly, IMO, but I can see that perhaps apache/arrow repo > > >> members > > >> >> >> may > > >> >> >> > > have > > >> >> >> > > > viewed it as "abandoned". > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> > > > > I do not fully understand why the pain points Jacob > > mentioned > > >> >> were > > >> >> >> not > > >> >> >> > > > > brought up to the mailing list sooner, though. > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > To be honest and frank, I didn't have pain points with the > > >> >> >> development > > >> >> >> > > > process I outlined when the code was donated and as stated > > in > > >> the > > >> >> >> README. > > >> >> >> > > > That was the process that made the donation possible and I > > >> >> imagined > > >> >> >> would > > >> >> >> > > > work well going forward, and has, until this thread started > > >> and it > > >> >> >> was > > >> >> >> > > > pointed out that this process isn't viable. The pain points > > >> were > > >> >> >> > > discussed > > >> >> >> > > > with the initial code donation, but in my mind were resolved > > >> with > > >> >> the > > >> >> >> > > > development process that was decided upon. > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> > > > > This made us unable to potentially take corrective > > measures. > > >> I > > >> >> >> think > > >> >> >> > > that > > >> >> >> > > > > this is why everyone was taken a bit by surprise with > > this. > > >> >> >> > > > > > > >> >> >> > > > > Best, > > >> >> >> > > > > Jorge > > >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney < > > >> >> wesmck...@gmail.com> > > >> >> >> > > wrote: > > >> >> >> > > > > > > >> >> >> > > > > > hi Jacob — sorry to hear that. It's a bummer that you > > all > > >> did > > >> >> not > > >> >> >> > > > > > attempt to work in the community for any meaningful > > amount > > >> of > > >> >> >> time > > >> >> >> > > and > > >> >> >> > > > > > are choosing not to try based on the perception that it > > >> will > > >> >> >> create > > >> >> >> > > > > > unacceptable overhead for you. I believe the benefits > > would > > >> >> >> outweigh > > >> >> >> > > > > > the costs, but I suppose we will have to agree to > > disagree. > > >> >> >> > > > > > > > >> >> >> > > > > > Can you prepare a pull request to do the requisite > > >> repository > > >> >> >> > > surgery? > > >> >> >> > > > > > I hope the development goes well in the future and look > > >> >> forward > > >> >> >> to > > >> >> >> > > > > > seeing folks from the Julia ecosystem engaged here on > > >> growing > > >> >> the > > >> >> >> > > > > > Arrow ecosystem. > > >> >> >> > > > > > > > >> >> >> > > > > > Thanks, > > >> >> >> > > > > > Wes > > >> >> >> > > > > > > > >> >> >> > > > > > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn < > > >> >> >> quinn.jac...@gmail.com> > > >> >> >> > > > > wrote: > > >> >> >> > > > > > > > > >> >> >> > > > > > > Ok, I've had a chance to discuss with a few other > > Julia > > >> >> >> developers > > >> >> >> > > and > > >> >> >> > > > > > > review various options. I think it's best to drop the > > >> Julia > > >> >> >> code > > >> >> >> > > from > > >> >> >> > > > > the > > >> >> >> > > > > > > physical apache/arrow repo. The extra overhead on > > >> >> development, > > >> >> >> > > release > > >> >> >> > > > > > > process, and user issue reporting and PR contributing > > are > > >> >> too > > >> >> >> much > > >> >> >> > > in > > >> >> >> > > > > > > addition to the technical challenges that we never > > >> resolved > > >> >> >> > > involving > > >> >> >> > > > > > > including the past Arrow.jl release version git trees > > in > > >> the > > >> >> >> > > > > apache/arrow > > >> >> >> > > > > > > repo. > > >> >> >> > > > > > > > > >> >> >> > > > > > > We're still very much committed to working on the > > Julia > > >> >> >> > > implementation > > >> >> >> > > > > > and > > >> >> >> > > > > > > participating in the broader arrow community. I've > > >> enjoyed > > >> >> >> > > following > > >> >> >> > > > > the > > >> >> >> > > > > > > user/dev mailing lists and will continue to do so. We > > >> >> monitor > > >> >> >> > > format > > >> >> >> > > > > > > proposals and try to implement new functionality as > > >> quickly > > >> >> as > > >> >> >> > > > > possible. > > >> >> >> > > > > > We > > >> >> >> > > > > > > got the initial arrow flight proto code generated just > > >> last > > >> >> >> night > > >> >> >> > > in > > >> >> >> > > > > > fact. > > >> >> >> > > > > > > I'd still like to explore official integration with > > the > > >> >> archery > > >> >> >> > > test > > >> >> >> > > > > > suite > > >> >> >> > > > > > > to solidify the Julia implementation with integration > > >> >> tests; I > > >> >> >> > > think > > >> >> >> > > > > that > > >> >> >> > > > > > > would be very valuable for long-term confidence in the > > >> >> >> > > cross-language > > >> >> >> > > > > > > support of the Julia implementation. > > >> >> >> > > > > > > > > >> >> >> > > > > > > We realize one of the main implications will probably > > be > > >> >> >> dropping > > >> >> >> > > Julia > > >> >> >> > > > > > > from the list of "official implementations". We're > > >> >> encouraged > > >> >> >> by > > >> >> >> > > the > > >> >> >> > > > > many > > >> >> >> > > > > > > users who have already started using the Julia > > >> >> implementation > > >> >> >> and > > >> >> >> > > will > > >> >> >> > > > > > > strive to maintain a high rate of issue responsiveness > > >> and > > >> >> >> feature > > >> >> >> > > > > > > development to maintain project confidence. If > > there's a > > >> >> >> > > possibility of > > >> >> >> > > > > > > being included somewhere as an "unofficial" or > > >> >> "semi-official" > > >> >> >> > > > > > > implementation, we'd love to still be bundled with the > > >> >> broader > > >> >> >> > > arrow > > >> >> >> > > > > > > project somehow, like, for example, showing how Julia > > >> >> >> integrates > > >> >> >> > > with > > >> >> >> > > > > the > > >> >> >> > > > > > > archery test suite, once the work there is done. > > >> >> >> > > > > > > > > >> >> >> > > > > > > Best, > > >> >> >> > > > > > > > > >> >> >> > > > > > > -Jacob > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney < > > >> >> >> wesmck...@gmail.com> > > >> >> >> > > > > > wrote: > > >> >> >> > > > > > > > > >> >> >> > > > > > > > Also, on the issue that there are no Julia-focused > > PMC > > >> >> >> members — > > >> >> >> > > note > > >> >> >> > > > > > > > that I helped the JavaScript folks make their own > > >> >> independent > > >> >> >> > > > > releases > > >> >> >> > > > > > > > for quite a while: called the votes (e.g. [1]), > > helped > > >> get > > >> >> >> > > people to > > >> >> >> > > > > > > > verify and vote on the releases. After a time, it > > was > > >> >> >> decided to > > >> >> >> > > stop > > >> >> >> > > > > > > > releasing independently because there wasn't enough > > >> >> >> development > > >> >> >> > > > > > > > activity to justify it. > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > [1]: > > >> >> >> > > > > https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney < > > >> >> >> > > wesmck...@gmail.com> > > >> >> >> > > > > > wrote: > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > hi Jacob, > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn < > > >> >> >> > > > > quinn.jac...@gmail.com> > > >> >> >> > > > > > > > wrote: > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > I can comment as the primary apache arrow > > liaison > > >> for > > >> >> the > > >> >> >> > > > > Arrow.jl > > >> >> >> > > > > > > > > > repository and original code donator. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > I apologize for the "surprise", but I commented > > a > > >> few > > >> >> >> times > > >> >> >> > > in > > >> >> >> > > > > > various > > >> >> >> > > > > > > > > > places and put a snippet in the README > > >> >> >> > > > > > > > > > < > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > >> >> >> > > >> >> > > >> > > https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > about > > >> >> >> > > > > > > > > > the approach I wanted to take w/ the Julia > > >> >> >> implementation in > > >> >> >> > > > > terms > > >> >> >> > > > > > of > > >> >> >> > > > > > > > > > keeping the JuliaData/Arrow.jl repository as a > > "dev > > >> >> >> branch" > > >> >> >> > > of > > >> >> >> > > > > > sorts > > >> >> >> > > > > > > > of the > > >> >> >> > > > > > > > > > apache/arrow code, upstreaming changes > > >> periodically. > > >> >> >> There's > > >> >> >> > > > > even a > > >> >> >> > > > > > > > script > > >> >> >> > > > > > > > > > < > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > >> >> >> > > >> >> > > >> > > https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > I wrote to mostly automate this upstreaming. I > > >> realize > > >> >> >> now > > >> >> >> > > that I > > >> >> >> > > > > > > > didn't > > >> >> >> > > > > > > > > > consider the "Arrow PMC" position on this kind > > of > > >> >> setup > > >> >> >> or > > >> >> >> > > seek > > >> >> >> > > > > to > > >> >> >> > > > > > > > affirm > > >> >> >> > > > > > > > > > that it would be ok to approach things like > > this. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > The reality is that Julia users are very > > engrained > > >> to > > >> >> >> expect > > >> >> >> > > > > Julia > > >> >> >> > > > > > > > packages > > >> >> >> > > > > > > > > > to live in a single stand-alone github repo, > > where > > >> >> issues > > >> >> >> > > can be > > >> >> >> > > > > > > > opened, > > >> >> >> > > > > > > > > > and pull requests are welcome. It was hard and > > >> still > > >> >> is > > >> >> >> hard > > >> >> >> > > to > > >> >> >> > > > > > imagine > > >> >> >> > > > > > > > > > "turning that off", since I believe we would > > lose a > > >> >> lot > > >> >> >> of > > >> >> >> > > > > > valuable bug > > >> >> >> > > > > > > > > > reports and first-time contributions. This isn't > > >> >> >> necessarily > > >> >> >> > > any > > >> >> >> > > > > > fault > > >> >> >> > > > > > > > of > > >> >> >> > > > > > > > > > how the bug report/contribution process is > > handled > > >> for > > >> >> >> the > > >> >> >> > > arrow > > >> >> >> > > > > > > > project > > >> >> >> > > > > > > > > > overall, though I'm also aware that there's a > > >> desire > > >> >> to > > >> >> >> make > > >> >> >> > > it > > >> >> >> > > > > > easier > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > < > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > >> >> >> > > >> >> > > >> > > https://lists.apache.org/x/thread.html/r8817dfba08ef8daa210956db69d513fd27b7a751d28fb8f27e39cc7e@%3Cdev.arrow.apache.org%3E > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > and > > >> >> >> > > > > > > > > > it currently requires more and different effort > > >> than > > >> >> >> Julia > > >> >> >> > > users > > >> >> >> > > > > > are > > >> >> >> > > > > > > > used > > >> >> >> > > > > > > > > > to. I think it's more from how open, welcoming, > > and > > >> >> how > > >> >> >> > > strong > > >> >> >> > > > > the > > >> >> >> > > > > > > > culture > > >> >> >> > > > > > > > > > is in Julia around encouraging community > > >> contributions > > >> >> >> and > > >> >> >> > > the > > >> >> >> > > > > > tight > > >> >> >> > > > > > > > > > integration with github and its open-source > > project > > >> >> >> > > management > > >> >> >> > > > > > tools. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > Well, we are on track to having 1000 different > > people > > >> >> >> > > contribute to > > >> >> >> > > > > > > > > the project and have over 12,000 issues, so I > > don't > > >> >> think > > >> >> >> > > there is > > >> >> >> > > > > > > > > evidence that we are failing to attract new > > >> >> contributors or > > >> >> >> > > that > > >> >> >> > > > > > > > > feature requests / bugs aren't being reported. The > > >> way > > >> >> >> that we > > >> >> >> > > work > > >> >> >> > > > > > is > > >> >> >> > > > > > > > > _different_, so adapting to the Apache process > > will > > >> >> require > > >> >> >> > > change. > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > Additionally, I was and still am concerned about > > >> the > > >> >> >> overall > > >> >> >> > > > > > release > > >> >> >> > > > > > > > > > process of the apache/arrow project. I know > > there > > >> have > > >> >> >> been > > >> >> >> > > > > efforts > > >> >> >> > > > > > > > there > > >> >> >> > > > > > > > > > as well to make it easier for individual > > languages > > >> to > > >> >> >> > > release on > > >> >> >> > > > > > their > > >> >> >> > > > > > > > own > > >> >> >> > > > > > > > > > cadence, but just anecdotally, the > > >> JuliaData/Arrow.jl > > >> >> has > > >> >> >> > > > > > > > had/needed/wanted > > >> >> >> > > > > > > > > > 10 patch and minor releases since the original > > code > > >> >> >> donation, > > >> >> >> > > > > > whereas > > >> >> >> > > > > > > > the > > >> >> >> > > > > > > > > > apache/arrow project has had one (3.0.0). This > > >> leads > > >> >> to > > >> >> >> some > > >> >> >> > > of > > >> >> >> > > > > the > > >> >> >> > > > > > > > > > concerns I have with restricting development to > > >> just > > >> >> the > > >> >> >> > > > > > apache/arrow > > >> >> >> > > > > > > > > > repository: how exactly does the release process > > >> work > > >> >> for > > >> >> >> > > > > > individual > > >> >> >> > > > > > > > > > languages who may desire independent releases > > apart > > >> >> from > > >> >> >> the > > >> >> >> > > > > > quarterly > > >> >> >> > > > > > > > > > overall project releases? I think from the Rust > > >> >> thread I > > >> >> >> > > remember > > >> >> >> > > > > > that > > >> >> >> > > > > > > > you > > >> >> >> > > > > > > > > > just need a group of language contributors to > > all > > >> >> agree, > > >> >> >> but > > >> >> >> > > what > > >> >> >> > > > > > if > > >> >> >> > > > > > > > I'm > > >> >> >> > > > > > > > > > the only "active" Julia contributor? It's also > > >> unclear > > >> >> >> what > > >> >> >> > > the > > >> >> >> > > > > > > > > > expectations are for actual development: with > > the > > >> >> >> original > > >> >> >> > > code > > >> >> >> > > > > > > > donation > > >> >> >> > > > > > > > > > PRs, I know Neal "reviewed" the PRs, but perhaps > > >> >> missed > > >> >> >> the > > >> >> >> > > > > details > > >> >> >> > > > > > > > around > > >> >> >> > > > > > > > > > how I proposed development continue going > > forward. > > >> Is > > >> >> it > > >> >> >> > > required > > >> >> >> > > > > > to > > >> >> >> > > > > > > > have a > > >> >> >> > > > > > > > > > certain number of reviews before merging? On the > > >> Julia > > >> >> >> side, > > >> >> >> > > I > > >> >> >> > > > > can > > >> >> >> > > > > > try > > >> >> >> > > > > > > > to > > >> >> >> > > > > > > > > > encourage/push for those who have contributed to > > >> the > > >> >> >> > > > > > JuliaData/Arrow.jl > > >> >> >> > > > > > > > > > repository to help review PRs to apache/arrow, > > but > > >> I > > >> >> also > > >> >> >> > > can't > > >> >> >> > > > > > > > guarantee > > >> >> >> > > > > > > > > > we would always have someone to review. It just > > >> feels > > >> >> >> pretty > > >> >> >> > > > > > awkward > > >> >> >> > > > > > > > if I > > >> >> >> > > > > > > > > > keep needing to ping non-Julia people to > > "review" a > > >> >> PR to > > >> >> >> > > merge > > >> >> >> > > > > it. > > >> >> >> > > > > > > > Perhaps > > >> >> >> > > > > > > > > > this is just a problem of the overall Julia > > >> >> >> implementation > > >> >> >> > > > > > "smallness" > > >> >> >> > > > > > > > in > > >> >> >> > > > > > > > > > terms of contributors, but I'm not sure on the > > best > > >> >> >> answer > > >> >> >> > > here. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > Several things here: > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > * If you want to do separate Julia releases, you > > are > > >> >> free > > >> >> >> to do > > >> >> >> > > > > that, > > >> >> >> > > > > > > > > but you have to follow the process (voting on the > > >> >> mailing > > >> >> >> list, > > >> >> >> > > > > > > > > publishing GPG-signed source artifacts) > > >> >> >> > > > > > > > > * If you had been working "in the community" since > > >> >> >> November, > > >> >> >> > > you > > >> >> >> > > > > > would > > >> >> >> > > > > > > > > probably already be a committer, so there is a > > >> >> >> bootstrapping > > >> >> >> > > here > > >> >> >> > > > > > that > > >> >> >> > > > > > > > > has failed to take place. In the meantime, we are > > >> more > > >> >> than > > >> >> >> > > happy > > >> >> >> > > > > to > > >> >> >> > > > > > > > > help you "earn your wings" (as a committer) as > > >> quickly > > >> >> as > > >> >> >> > > possible. > > >> >> >> > > > > > > > > But from my perspective, I see a code donation and > > >> two > > >> >> >> other > > >> >> >> > > > > commits, > > >> >> >> > > > > > > > > which isn't enough to make a case for > > committership. > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > So in short, I'm not sure on the best path > > >> forward. I > > >> >> >> think > > >> >> >> > > > > > strictly > > >> >> >> > > > > > > > > > restricting development to the apache/arrow > > >> physical > > >> >> >> > > repository > > >> >> >> > > > > > would > > >> >> >> > > > > > > > > > actively hurt the progress of the Julia > > >> >> implementation, > > >> >> >> > > whereas > > >> >> >> > > > > it > > >> >> >> > > > > > > > *has* > > >> >> >> > > > > > > > > > been progressing with increasing momentum since > > >> first > > >> >> >> > > released. > > >> >> >> > > > > > There > > >> >> >> > > > > > > > are > > >> >> >> > > > > > > > > > posts on the Julia discourse forum, in the Julia > > >> slack > > >> >> >> and > > >> >> >> > > zulip > > >> >> >> > > > > > > > > > communities, and quite a few issues/PRs being > > >> opened > > >> >> at > > >> >> >> the > > >> >> >> > > > > > > > > > JuliaData/Arrow.jl repository. There have been > > >> several > > >> >> >> calls > > >> >> >> > > for > > >> >> >> > > > > > arrow > > >> >> >> > > > > > > > > > flight support, with a member from Julia > > Computing > > >> >> >> actually > > >> >> >> > > close > > >> >> >> > > > > > to > > >> >> >> > > > > > > > > > releasing a gRPC client > > >> >> >> > > > > > > > > > < > > https://github.com/JuliaComputing/gRPCClient.jl> > > >> >> >> > > specifically > > >> >> >> > > > > > > > > > to help with flight support. But in terms of > > actual > > >> >> >> > > committers, > > >> >> >> > > > > > it's > > >> >> >> > > > > > > > been > > >> >> >> > > > > > > > > > primarily just myself, with a few minor > > >> contributions > > >> >> by > > >> >> >> > > others. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > I guess the big question that comes to mind is > > what > > >> >> are > > >> >> >> the > > >> >> >> > > hard > > >> >> >> > > > > > > > > > requirements to be considered an "official > > >> >> >> implementation"? > > >> >> >> > > Does > > >> >> >> > > > > > the > > >> >> >> > > > > > > > code > > >> >> >> > > > > > > > > > *have* to live in the same physical repo? Or if > > it > > >> >> >> passed the > > >> >> >> > > > > > series of > > >> >> >> > > > > > > > > > archery integration tests, would that be > > enough? I > > >> >> >> apologize > > >> >> >> > > for > > >> >> >> > > > > my > > >> >> >> > > > > > > > > > naivete/inexperience on all things "apache", > > but I > > >> >> >> imagine > > >> >> >> > > that's > > >> >> >> > > > > > a big > > >> >> >> > > > > > > > > > part of it: having official development/releases > > >> >> through > > >> >> >> the > > >> >> >> > > > > > > > apache/arrow > > >> >> >> > > > > > > > > > community, though again I'm not exactly sure on > > the > > >> >> >> formal > > >> >> >> > > > > > processes > > >> >> >> > > > > > > > here? > > >> >> >> > > > > > > > > > I would like to keep Julia as an official > > >> >> implementation, > > >> >> >> > > but I'm > > >> >> >> > > > > > also > > >> >> >> > > > > > > > > > mostly carrying the maintainership alone at the > > >> moment > > >> >> >> and > > >> >> >> > > want > > >> >> >> > > > > to > > >> >> >> > > > > > be > > >> >> >> > > > > > > > > > realistic with the future of the project. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > The critical matter is whether the > > >> >> development/maintenance > > >> >> >> > > work is > > >> >> >> > > > > > > > > conducted by the "Arrow community" in accordance > > with > > >> >> the > > >> >> >> > > Apache > > >> >> >> > > > > Way, > > >> >> >> > > > > > > > > which is to say individuals collaborating with > > each > > >> >> other > > >> >> >> on > > >> >> >> > > Apache > > >> >> >> > > > > > > > > channels (for communication and development) and > > >> >> avoiding > > >> >> >> the > > >> >> >> > > bad > > >> >> >> > > > > > > > > patterns you see sometimes in other communities > > (e.g. > > >> >> >> > > inconsistent > > >> >> >> > > > > > > > > openness). > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > It's fine — really, no pressure — if you want to > > be > > >> >> >> > > independent and > > >> >> >> > > > > > do > > >> >> >> > > > > > > > > things your own way, you just have to be clear > > that > > >> you > > >> >> are > > >> >> >> > > > > > > > > independent and not operating as part of the > > Apache > > >> >> Arrow > > >> >> >> > > > > community. > > >> >> >> > > > > > > > > You can't have it both ways, though. No hard > > feelings > > >> >> >> whatever > > >> >> >> > > you > > >> >> >> > > > > > > > > decide, but the current "dump code over the wall > > >> >> >> occasionally" > > >> >> >> > > > > > > > > approach but work on independent channels is not > > >> >> >> compatible. > > >> >> >> > > > > Building > > >> >> >> > > > > > > > > healthy open source communities is hard, but this > > way > > >> >> has > > >> >> >> been > > >> >> >> > > > > shown > > >> >> >> > > > > > > > > to work well, which is why I've spent the last 6 > > >> years > > >> >> >> working > > >> >> >> > > hard > > >> >> >> > > > > > to > > >> >> >> > > > > > > > > bring people together to build this project and > > >> >> ecosystem! > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > If you want to maintain a test harness here to > > >> verify an > > >> >> >> > > > > independent > > >> >> >> > > > > > > > > Julia implementation, that's fine, too. I'm > > >> disappointed > > >> >> >> that > > >> >> >> > > > > things > > >> >> >> > > > > > > > > failed to bootstrap after the code donation, so I > > >> want > > >> >> to > > >> >> >> see > > >> >> >> > > if we > > >> >> >> > > > > > > > > can course correct quickly or if not decide to go > > our > > >> >> >> separate > > >> >> >> > > > > ways. > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > Thanks, > > >> >> >> > > > > > > > > Wes > > >> >> >> > > > > > > > > > > >> >> >> > > > > > > > > > I'm open to discussion and ideas on the best way > > >> >> forward. > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > -Jacob > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > On Tue, Mar 30, 2021 at 2:03 PM Wes McKinney < > > >> >> >> > > > > wesmck...@gmail.com> > > >> >> >> > > > > > > > wrote: > > >> >> >> > > > > > > > > > > > >> >> >> > > > > > > > > > > hi folks, > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > I was very surprised today to learn that the > > >> Julia > > >> >> >> Arrow > > >> >> >> > > > > > > > > > > implementation has continued operating more or > > >> less > > >> >> >> like an > > >> >> >> > > > > > > > > > > independent open source project since the code > > >> >> donation > > >> >> >> > > last > > >> >> >> > > > > > > > November: > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> https://github.com/JuliaData/Arrow.jl/commits/main > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > There may have been a misunderstanding about > > what > > >> >> was > > >> >> >> > > expected > > >> >> >> > > > > to > > >> >> >> > > > > > > > > > > occur after the code donation, but it's > > >> problematic > > >> >> >> for a > > >> >> >> > > bunch > > >> >> >> > > > > > of > > >> >> >> > > > > > > > > > > reasons (IP lineage / governance / community > > >> >> >> development) > > >> >> >> > > to > > >> >> >> > > > > have > > >> >> >> > > > > > > > work > > >> >> >> > > > > > > > > > > happening on the implementation "outside the > > >> >> >> community". > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > In any case, what is done is done, so the > > Arrow > > >> >> PMC's > > >> >> >> > > position > > >> >> >> > > > > on > > >> >> >> > > > > > > > this > > >> >> >> > > > > > > > > > > would be roughly to regard the work as a hard > > >> fork > > >> >> of > > >> >> >> > > what's in > > >> >> >> > > > > > > > Apache > > >> >> >> > > > > > > > > > > Arrow, which given its development activity is > > >> more > > >> >> or > > >> >> >> less > > >> >> >> > > > > > inactive > > >> >> >> > > > > > > > > > > [1]. (I had actually thought the project was > > >> simply > > >> >> >> > > inactive > > >> >> >> > > > > > after > > >> >> >> > > > > > > > the > > >> >> >> > > > > > > > > > > code donation) > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > The critical question now is, is there > > interest > > >> from > > >> >> >> Julia > > >> >> >> > > > > > developers > > >> >> >> > > > > > > > > > > in working "in the community", which is to > > say: > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > * Having development discussions on ASF > > channels > > >> >> >> (mailing > > >> >> >> > > list, > > >> >> >> > > > > > > > > > > GitHub, JIRA), planning and communicating in > > the > > >> >> open > > >> >> >> > > > > > > > > > > * Doing all development in ASF GitHub > > >> repositories > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > The answer to the question may be "no" (which > > is > > >> >> okay), > > >> >> >> > > but if > > >> >> >> > > > > > that's > > >> >> >> > > > > > > > > > > the case, I don't think we should be giving > > the > > >> >> >> impression > > >> >> >> > > that > > >> >> >> > > > > > we > > >> >> >> > > > > > > > > > > have an official Julia implementation that is > > >> >> >> developed and > > >> >> >> > > > > > > > maintained > > >> >> >> > > > > > > > > > > by the community (and so my argument would be > > >> >> >> > > unfortunately to > > >> >> >> > > > > > drop > > >> >> >> > > > > > > > > > > the donated code from the project). > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > If the answer is "yes", there needs to be a > > hard > > >> >> >> > > commitment to > > >> >> >> > > > > > move > > >> >> >> > > > > > > > > > > development to Apache channels and not look > > >> back. We > > >> >> >> would > > >> >> >> > > also > > >> >> >> > > > > > need > > >> >> >> > > > > > > > > > > to figure out what to do to document and > > >> synchronize > > >> >> >> the > > >> >> >> > > new IP > > >> >> >> > > > > > > > that's > > >> >> >> > > > > > > > > > > been created since the code donation. > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > Thanks, > > >> >> >> > > > > > > > > > > Wes > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > [1]: > > >> >> >> > > > > > > https://github.com/apache/arrow/commits/master/julia/Arrow > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > >> >> >> > > >> >> > > >> > >