quinnj opened a new issue #284:
URL: https://github.com/apache/arrow-julia/issues/284
Hey all,
I'm opening this issue to facilitate some discussion around the current
state of and future direction of the Julia arrow implementation (that lives in
this repo). As the original primary code author and main initiator of the
transfer of the repo from the JuliaData github repo to the apache organization,
I feel somewhat responsible to give an accounting and help craft a plan around
the future.
Before transferring the repo, there were a couple key things I, and other
Julia contributors, failed to realize/fully consider:
* No existing Arrow.jl committers would maintain any write permissions
post-transfer until a roughly defined "amount of time" had passed showing
consistent contributions to be officially nominated as an Arrow PMC member
* How much administrative overhead would be required by the [official
Apache release policies](https://www.apache.org/legal/release-policy.html)
The reality of the Julia arrow implementation is as follows:
* None of the key contributors, especially me, work on this code in any
kind of "full time" or even very regular cadence. I personally am the primary
maintainer of some dozen "data" packages in the Julia ecosystem and tend to
work in short "blitzes", working through as many open issues/enhancements as
possible for a few weeks to push towards a bigger release, while fixing minor
or urgent bugs in between blitzes, in addition to reviewing other contributions
as they come in. This doesn't ideally fit, in my understanding, the traditional
PMC member profile, as it may be an entire year in between major "blitzes" all
dependent on my own personal time/energy and demands of other projects.
* This also impacts the 2nd point above where the overhead of Apache
release policies has a heavier impact on the small amount of contributor effort
we're already trying to muster.
In short, I think:
* The Julia implementation might just be too small to realistically be an
official Apache project due to the overhead of required apache policies
* In addition, the current Apache release policies seem more geared to
languages (C++, java) that don't have existing infrastructure/processes around
package release/maintenance and for projects with more contributor resources
* The Julia General registry, builtin Pkg.jl package manager, and
ecosystem culture have evolved a pretty high-quality "package maintenance and
deployment" infrastructure/process that means the Julia implementation just
doesn't benefit as much from the rigorous Apache policies compared to other
languages/projects (I can expound on specifics here if people are interested,
but in short, many specific Apache release policies are already kind of
"builtin" to the standard Julia package release process).
* Lastly, it's been pretty demoralizing while trying to make an effort to
"officially" join the broader arrow community, to then be personally "reset" to
the status of a completely new contributor. The key bottleneck in open source
is _always_ finding enough contributor effort, and yet this "contributor reset"
policy seems to actively impede existing contributors.
All of that said, I'd like to tentatively propose moving (or just forking)
the Julia implementation back to the JuliaData GitHub organization. I think
that would "unblock" the implementation from the current lack of contributor
ability and releases, while hopefully allowing the project to grow and gain new
contributor resources.
I recognize this may lead to being classified an "unofficial" implementation
with regards to the arrow community, but personally, it won't change my desire
to actively follow/contribute to arrow-dev mailing list discussions, join the
biweekly community call, or contribute in other ways to the broader arrow
community. I'm also still interested in improving the level of integration
testing between the Julia implementation and other languages (via archery) and
in general, making the Julia implementation as high-quality as possible and
adding to the overall value of the arrow project as a whole. I don't propose
this break from any negative feelings toward the arrow community or PMC members
(they've only been helpful throughout this process), but more from a realistic
consideration of resource constraints, policy applicability, and what I believe
will _currently_ foster the best context for the Julia implementation to
grow/succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
