Re: Status of Arrow Julia implementation?

[Wes McKinney]

hi Jacob — that's great to hear. We're standing by to help you out with this.

On Fri, Jun 25, 2021 at 4:57 PM Jacob Quinn  wrote:
>
> Hi Kou,
>
> Sorry for the slow response here, but it's been great to see how the new
> Rust process has shaken out and I think it working well. I'd like to move
> forward with transferring the JuliaData/Arrow.jl repository to
> apache/arrow-julia and following a similar process to Rust in terms of
> development/release. I can start on a Julia-specific proposal with
> specifics.
>
> Thanks for all the help!
>
> -Jacob
>
> On Sun, Apr 25, 2021 at 11:56 PM Sutou Kouhei  wrote:
>
> > Hi,
> >
> > I think that we can say the Rust migration is complete once
> > we merge https://github.com/apache/arrow/pull/10096. But
> > it's a good time to think about the Julia migration.
> >
> > Jacob, here is the Rust's new development process:
> >
> >
> > https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#
> >
> > (It seems that an anonymous user deleted a part of it
> > accidentally.)
> >
> > Do you want to use the same development process as the
> > Rust's one? Do you have any item you want to change?
> >
> >
> > Thanks,
> > --
> > kou
> >
> > In 
> >   "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04
> > -0700,
> >   Micah Kornfield  wrote:
> >
> > > It seems the Rust migration is now complete.  Do we want to wait to iron
> > > out the other potential issues?
> > >
> > > I think the outstanding ones might be:
> > > 1.  Issue management
> > > 2.  Integration testing
> > >
> > > -Micah
> > >
> > > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei 
> > wrote:
> > >
> > >> Hi Jacob,
> > >>
> > >> OK. Here is my plan:
> > >>
> > >>   1. We wait for the Rust's move to complete
> > >>   2. We use a process similar to the Rust's move
> > >>
> > >>
> > >> Thanks,
> > >> --
> > >> kou
> > >>
> > >> In 
> > >>   "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021
> > 08:37:41
> > >> -0600,
> > >>   Jacob Quinn  wrote:
> > >>
> > >> > Thank you kou! I appreciate the help. I'm happy to do whatever is
> > >> required
> > >> > to facilitate the moving/donating process from JuliaData/Arrow.jl to
> > >> > apache/arrow-julia.
> > >> >
> > >> > -Jacob
> > >> >
> > >> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei 
> > wrote:
> > >> >
> > >> >> Hi Jacob,
> > >> >>
> > >> >> I, a PMC member, talked to Kenta Murata, a commiter and a
> > >> >> Julia user, about this.
> > >> >>
> > >> >> We support that you and Julia folks work on
> > >> >> arrow/arrow-julia until we have enough PMC members from
> > >> >> Julia folks. For example, we'll help IP clearance process to
> > >> >> import the latest JuliaData/Arrow.js changes to apache/ and
> > >> >> we'll start voting on Julia package release.
> > >> >>
> > >> >>
> > >> >> Thanks,
> > >> >> --
> > >> >> kou
> > >> >>
> > >> >> In  > d-heskgn2mm57...@mail.gmail.com>
> > >> >>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021
> > >> 23:06:27
> > >> >> -0600,
> > >> >>   Jacob Quinn  wrote:
> > >> >>
> > >> >> > Micah/Wes,
> > >> >> >
> > >> >> > Yes, I've been following the rust proposal thread with great
> > >> interest. I
> > >> >> do
> > >> >> > think that provides a great path forward: transferring the
> > >> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve
> > the
> > >> >> > "package history" technical challenges that in part led to the
> > current
> > >> >> > setup and concerns. I think being able to utilize github issues
> > would
> > >> >> also
> > >> >> > be great; as I've mention

Re: Status of Arrow Julia implementation?

[Jacob Quinn]

Hi Kou,

Sorry for the slow response here, but it's been great to see how the new
Rust process has shaken out and I think it working well. I'd like to move
forward with transferring the JuliaData/Arrow.jl repository to
apache/arrow-julia and following a similar process to Rust in terms of
development/release. I can start on a Julia-specific proposal with
specifics.

Thanks for all the help!

-Jacob

On Sun, Apr 25, 2021 at 11:56 PM Sutou Kouhei  wrote:

> Hi,
>
> I think that we can say the Rust migration is complete once
> we merge https://github.com/apache/arrow/pull/10096. But
> it's a good time to think about the Julia migration.
>
> Jacob, here is the Rust's new development process:
>
>
> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#
>
> (It seems that an anonymous user deleted a part of it
> accidentally.)
>
> Do you want to use the same development process as the
> Rust's one? Do you have any item you want to change?
>
>
> Thanks,
> --
> kou
>
> In 
>   "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04
> -0700,
>   Micah Kornfield  wrote:
>
> > It seems the Rust migration is now complete.  Do we want to wait to iron
> > out the other potential issues?
> >
> > I think the outstanding ones might be:
> > 1.  Issue management
> > 2.  Integration testing
> >
> > -Micah
> >
> > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei 
> wrote:
> >
> >> Hi Jacob,
> >>
> >> OK. Here is my plan:
> >>
> >>   1. We wait for the Rust's move to complete
> >>   2. We use a process similar to the Rust's move
> >>
> >>
> >> Thanks,
> >> --
> >> kou
> >>
> >> In 
> >>   "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021
> 08:37:41
> >> -0600,
> >>   Jacob Quinn  wrote:
> >>
> >> > Thank you kou! I appreciate the help. I'm happy to do whatever is
> >> required
> >> > to facilitate the moving/donating process from JuliaData/Arrow.jl to
> >> > apache/arrow-julia.
> >> >
> >> > -Jacob
> >> >
> >> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei 
> wrote:
> >> >
> >> >> Hi Jacob,
> >> >>
> >> >> I, a PMC member, talked to Kenta Murata, a commiter and a
> >> >> Julia user, about this.
> >> >>
> >> >> We support that you and Julia folks work on
> >> >> arrow/arrow-julia until we have enough PMC members from
> >> >> Julia folks. For example, we'll help IP clearance process to
> >> >> import the latest JuliaData/Arrow.js changes to apache/ and
> >> >> we'll start voting on Julia package release.
> >> >>
> >> >>
> >> >> Thanks,
> >> >> --
> >> >> kou
> >> >>
> >> >> In  d-heskgn2mm57...@mail.gmail.com>
> >> >>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021
> >> 23:06:27
> >> >> -0600,
> >> >>   Jacob Quinn  wrote:
> >> >>
> >> >> > Micah/Wes,
> >> >> >
> >> >> > Yes, I've been following the rust proposal thread with great
> >> interest. I
> >> >> do
> >> >> > think that provides a great path forward: transferring the
> >> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve
> the
> >> >> > "package history" technical challenges that in part led to the
> current
> >> >> > setup and concerns. I think being able to utilize github issues
> would
> >> >> also
> >> >> > be great; as I've mentioned elsewhere, it's much more
> >> >> traditional/expected
> >> >> > in the Julia ecosystem.
> >> >> >
> >> >> > I think the package could retain an independent versioning scheme.
> The
> >> >> >> additional process would be voting on release candidates. If the
> >> Julia
> >> >> >> folks want to try again and move development to a new,
> Julia-specific
> >> >> >> apache/* repository and apply the ASF governance to the project,
> the
> >> >> >> Arrow PMC could probably fast-track making Jacob a committer. In
> some
> >> >> >> 

Re: Status of Arrow Julia implementation?

[Sutou Kouhei]

Hi,

I think that we can say the Rust migration is complete once
we merge https://github.com/apache/arrow/pull/10096. But
it's a good time to think about the Julia migration.

Jacob, here is the Rust's new development process:

  
https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#

(It seems that an anonymous user deleted a part of it
accidentally.)

Do you want to use the same development process as the
Rust's one? Do you have any item you want to change?


Thanks,
--
kou

In 
  "Re: Status of Arrow Julia implementation?" on Sun, 25 Apr 2021 13:34:04 
-0700,
  Micah Kornfield  wrote:

> It seems the Rust migration is now complete.  Do we want to wait to iron
> out the other potential issues?
> 
> I think the outstanding ones might be:
> 1.  Issue management
> 2.  Integration testing
> 
> -Micah
> 
> On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei  wrote:
> 
>> Hi Jacob,
>>
>> OK. Here is my plan:
>>
>>   1. We wait for the Rust's move to complete
>>   2. We use a process similar to the Rust's move
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In 
>>   "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021 08:37:41
>> -0600,
>>   Jacob Quinn  wrote:
>>
>> > Thank you kou! I appreciate the help. I'm happy to do whatever is
>> required
>> > to facilitate the moving/donating process from JuliaData/Arrow.jl to
>> > apache/arrow-julia.
>> >
>> > -Jacob
>> >
>> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei  wrote:
>> >
>> >> Hi Jacob,
>> >>
>> >> I, a PMC member, talked to Kenta Murata, a commiter and a
>> >> Julia user, about this.
>> >>
>> >> We support that you and Julia folks work on
>> >> arrow/arrow-julia until we have enough PMC members from
>> >> Julia folks. For example, we'll help IP clearance process to
>> >> import the latest JuliaData/Arrow.js changes to apache/ and
>> >> we'll start voting on Julia package release.
>> >>
>> >>
>> >> Thanks,
>> >> --
>> >> kou
>> >>
>> >> In 
>> >>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021
>> 23:06:27
>> >> -0600,
>> >>   Jacob Quinn  wrote:
>> >>
>> >> > Micah/Wes,
>> >> >
>> >> > Yes, I've been following the rust proposal thread with great
>> interest. I
>> >> do
>> >> > think that provides a great path forward: transferring the
>> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the
>> >> > "package history" technical challenges that in part led to the current
>> >> > setup and concerns. I think being able to utilize github issues would
>> >> also
>> >> > be great; as I've mentioned elsewhere, it's much more
>> >> traditional/expected
>> >> > in the Julia ecosystem.
>> >> >
>> >> > I think the package could retain an independent versioning scheme. The
>> >> >> additional process would be voting on release candidates. If the
>> Julia
>> >> >> folks want to try again and move development to a new, Julia-specific
>> >> >> apache/* repository and apply the ASF governance to the project, the
>> >> >> Arrow PMC could probably fast-track making Jacob a committer. In some
>> >> >> code donations / IP clearance, the contributors for the donated code
>> >> >> become committers as part of the transaction.
>> >> >>
>> >> >
>> >> > These all sound great and would greatly facilitate a better
>> integration
>> >> > under ASF governance. These points definitely resolve my main
>> concerns.
>> >> >
>> >> > As I commented on the rust thread, I'm mostly interested in the
>> future of
>> >> > integration testing for rust/julia if they are split out into separate
>> >> > repos. In the current Julia implementation, we have all the code to
>> read
>> >> > arrow json, and I just hand-generated the integration test data and
>> >> > committed them in the repo itself, but it doesn't interface with other
>> >> > languages (just reads arrow json, produces arrow file, reads arrow
>> file,
>> >> > compares w/ origina

Re: Status of Arrow Julia implementation?

[Micah Kornfield]

It seems the Rust migration is now complete.  Do we want to wait to iron
out the other potential issues?

I think the outstanding ones might be:
1.  Issue management
2.  Integration testing

-Micah

On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei  wrote:

> Hi Jacob,
>
> OK. Here is my plan:
>
>   1. We wait for the Rust's move to complete
>   2. We use a process similar to the Rust's move
>
>
> Thanks,
> --
> kou
>
> In 
>   "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021 08:37:41
> -0600,
>   Jacob Quinn  wrote:
>
> > Thank you kou! I appreciate the help. I'm happy to do whatever is
> required
> > to facilitate the moving/donating process from JuliaData/Arrow.jl to
> > apache/arrow-julia.
> >
> > -Jacob
> >
> > On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei  wrote:
> >
> >> Hi Jacob,
> >>
> >> I, a PMC member, talked to Kenta Murata, a commiter and a
> >> Julia user, about this.
> >>
> >> We support that you and Julia folks work on
> >> arrow/arrow-julia until we have enough PMC members from
> >> Julia folks. For example, we'll help IP clearance process to
> >> import the latest JuliaData/Arrow.js changes to apache/ and
> >> we'll start voting on Julia package release.
> >>
> >>
> >> Thanks,
> >> --
> >> kou
> >>
> >> In 
> >>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021
> 23:06:27
> >> -0600,
> >>   Jacob Quinn  wrote:
> >>
> >> > Micah/Wes,
> >> >
> >> > Yes, I've been following the rust proposal thread with great
> interest. I
> >> do
> >> > think that provides a great path forward: transferring the
> >> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the
> >> > "package history" technical challenges that in part led to the current
> >> > setup and concerns. I think being able to utilize github issues would
> >> also
> >> > be great; as I've mentioned elsewhere, it's much more
> >> traditional/expected
> >> > in the Julia ecosystem.
> >> >
> >> > I think the package could retain an independent versioning scheme. The
> >> >> additional process would be voting on release candidates. If the
> Julia
> >> >> folks want to try again and move development to a new, Julia-specific
> >> >> apache/* repository and apply the ASF governance to the project, the
> >> >> Arrow PMC could probably fast-track making Jacob a committer. In some
> >> >> code donations / IP clearance, the contributors for the donated code
> >> >> become committers as part of the transaction.
> >> >>
> >> >
> >> > These all sound great and would greatly facilitate a better
> integration
> >> > under ASF governance. These points definitely resolve my main
> concerns.
> >> >
> >> > As I commented on the rust thread, I'm mostly interested in the
> future of
> >> > integration testing for rust/julia if they are split out into separate
> >> > repos. In the current Julia implementation, we have all the code to
> read
> >> > arrow json, and I just hand-generated the integration test data and
> >> > committed them in the repo itself, but it doesn't interface with other
> >> > languages (just reads arrow json, produces arrow file, reads arrow
> file,
> >> > compares w/ original arrow json). I'm happy to help work on the
> details
> >> of
> >> > what that looks like and pilot some solutions. I think with a solid
> >> > inter-repo integration testing framework, we can keep a strong sync
> >> between
> >> > projects.
> >> >
> >> > -Jacob
> >> >
> >> >
> >> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney 
> >> wrote:
> >> >
> >> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield <
> emkornfi...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > >
> >> >> > > Ok, I've had a chance to discuss with a few other Julia
> developers
> >> and
> >> >> > > review various options. I think it's best to drop the Julia code
> >> from
> >> >> the
> >> >> > > physical apache/arrow repo. The extra overhead on developmen

Re: Status of Arrow Julia implementation?

[Sutou Kouhei]

Hi Jacob,

OK. Here is my plan:

  1. We wait for the Rust's move to complete
  2. We use a process similar to the Rust's move


Thanks,
--
kou

In 
  "Re: Status of Arrow Julia implementation?" on Wed, 14 Apr 2021 08:37:41 
-0600,
  Jacob Quinn  wrote:

> Thank you kou! I appreciate the help. I'm happy to do whatever is required
> to facilitate the moving/donating process from JuliaData/Arrow.jl to
> apache/arrow-julia.
> 
> -Jacob
> 
> On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei  wrote:
> 
>> Hi Jacob,
>>
>> I, a PMC member, talked to Kenta Murata, a commiter and a
>> Julia user, about this.
>>
>> We support that you and Julia folks work on
>> arrow/arrow-julia until we have enough PMC members from
>> Julia folks. For example, we'll help IP clearance process to
>> import the latest JuliaData/Arrow.js changes to apache/ and
>> we'll start voting on Julia package release.
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In 
>>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021 23:06:27
>> -0600,
>>   Jacob Quinn  wrote:
>>
>> > Micah/Wes,
>> >
>> > Yes, I've been following the rust proposal thread with great interest. I
>> do
>> > think that provides a great path forward: transferring the
>> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the
>> > "package history" technical challenges that in part led to the current
>> > setup and concerns. I think being able to utilize github issues would
>> also
>> > be great; as I've mentioned elsewhere, it's much more
>> traditional/expected
>> > in the Julia ecosystem.
>> >
>> > I think the package could retain an independent versioning scheme. The
>> >> additional process would be voting on release candidates. If the Julia
>> >> folks want to try again and move development to a new, Julia-specific
>> >> apache/* repository and apply the ASF governance to the project, the
>> >> Arrow PMC could probably fast-track making Jacob a committer. In some
>> >> code donations / IP clearance, the contributors for the donated code
>> >> become committers as part of the transaction.
>> >>
>> >
>> > These all sound great and would greatly facilitate a better integration
>> > under ASF governance. These points definitely resolve my main concerns.
>> >
>> > As I commented on the rust thread, I'm mostly interested in the future of
>> > integration testing for rust/julia if they are split out into separate
>> > repos. In the current Julia implementation, we have all the code to read
>> > arrow json, and I just hand-generated the integration test data and
>> > committed them in the repo itself, but it doesn't interface with other
>> > languages (just reads arrow json, produces arrow file, reads arrow file,
>> > compares w/ original arrow json). I'm happy to help work on the details
>> of
>> > what that looks like and pilot some solutions. I think with a solid
>> > inter-repo integration testing framework, we can keep a strong sync
>> between
>> > projects.
>> >
>> > -Jacob
>> >
>> >
>> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney 
>> wrote:
>> >
>> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield 
>> >> wrote:
>> >> >
>> >> > >
>> >> > > Ok, I've had a chance to discuss with a few other Julia developers
>> and
>> >> > > review various options. I think it's best to drop the Julia code
>> from
>> >> the
>> >> > > physical apache/arrow repo. The extra overhead on development,
>> release
>> >> > > process, and user issue reporting and PR contributing are too much
>> in
>> >> > > addition to the technical challenges that we never resolved
>> involving
>> >> > > including the past Arrow.jl release version git trees in the
>> >> apache/arrow
>> >> > > repo.
>> >> >
>> >> >
>> >> > Hi Jacob,
>> >> > It seems you are on the new thread discussing a proposal for changing
>> >> > Rust's development model.   Would the proposal [1] address most of
>> these
>> >> > concerns if Julia was set up in the same way?
>> >> >
>> >> >  It seems in the short term the stickiest point would 

Re: Status of Arrow Julia implementation?

[Jacob Quinn]

Thank you kou! I appreciate the help. I'm happy to do whatever is required
to facilitate the moving/donating process from JuliaData/Arrow.jl to
apache/arrow-julia.

-Jacob

On Mon, Apr 12, 2021 at 7:53 PM Sutou Kouhei  wrote:

> Hi Jacob,
>
> I, a PMC member, talked to Kenta Murata, a commiter and a
> Julia user, about this.
>
> We support that you and Julia folks work on
> arrow/arrow-julia until we have enough PMC members from
> Julia folks. For example, we'll help IP clearance process to
> import the latest JuliaData/Arrow.js changes to apache/ and
> we'll start voting on Julia package release.
>
>
> Thanks,
> --
> kou
>
> In 
>   "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021 23:06:27
> -0600,
>   Jacob Quinn  wrote:
>
> > Micah/Wes,
> >
> > Yes, I've been following the rust proposal thread with great interest. I
> do
> > think that provides a great path forward: transferring the
> > JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the
> > "package history" technical challenges that in part led to the current
> > setup and concerns. I think being able to utilize github issues would
> also
> > be great; as I've mentioned elsewhere, it's much more
> traditional/expected
> > in the Julia ecosystem.
> >
> > I think the package could retain an independent versioning scheme. The
> >> additional process would be voting on release candidates. If the Julia
> >> folks want to try again and move development to a new, Julia-specific
> >> apache/* repository and apply the ASF governance to the project, the
> >> Arrow PMC could probably fast-track making Jacob a committer. In some
> >> code donations / IP clearance, the contributors for the donated code
> >> become committers as part of the transaction.
> >>
> >
> > These all sound great and would greatly facilitate a better integration
> > under ASF governance. These points definitely resolve my main concerns.
> >
> > As I commented on the rust thread, I'm mostly interested in the future of
> > integration testing for rust/julia if they are split out into separate
> > repos. In the current Julia implementation, we have all the code to read
> > arrow json, and I just hand-generated the integration test data and
> > committed them in the repo itself, but it doesn't interface with other
> > languages (just reads arrow json, produces arrow file, reads arrow file,
> > compares w/ original arrow json). I'm happy to help work on the details
> of
> > what that looks like and pilot some solutions. I think with a solid
> > inter-repo integration testing framework, we can keep a strong sync
> between
> > projects.
> >
> > -Jacob
> >
> >
> > On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney 
> wrote:
> >
> >> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield 
> >> wrote:
> >> >
> >> > >
> >> > > Ok, I've had a chance to discuss with a few other Julia developers
> and
> >> > > review various options. I think it's best to drop the Julia code
> from
> >> the
> >> > > physical apache/arrow repo. The extra overhead on development,
> release
> >> > > process, and user issue reporting and PR contributing are too much
> in
> >> > > addition to the technical challenges that we never resolved
> involving
> >> > > including the past Arrow.jl release version git trees in the
> >> apache/arrow
> >> > > repo.
> >> >
> >> >
> >> > Hi Jacob,
> >> > It seems you are on the new thread discussing a proposal for changing
> >> > Rust's development model.   Would the proposal [1] address most of
> these
> >> > concerns if Julia was set up in the same way?
> >> >
> >> >  It seems in the short term the stickiest point would be committer
> access
> >> > to the new repos, and I suppose the release mechanics still might be
> >> > challenging?
> >>
> >> I think the package could retain an independent versioning scheme. The
> >> additional process would be voting on release candidates. If the Julia
> >> folks want to try again and move development to a new, Julia-specific
> >> apache/* repository and apply the ASF governance to the project, the
> >> Arrow PMC could probably fast-track making Jacob a committer. In some
> >> code donations / IP clearance, the contributors for the donated code
> &g

Re: Status of Arrow Julia implementation?

[Sutou Kouhei]

Hi Jacob,

I, a PMC member, talked to Kenta Murata, a commiter and a
Julia user, about this.

We support that you and Julia folks work on
arrow/arrow-julia until we have enough PMC members from
Julia folks. For example, we'll help IP clearance process to
import the latest JuliaData/Arrow.js changes to apache/ and
we'll start voting on Julia package release.


Thanks,
--
kou

In 
  "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021 23:06:27 
-0600,
  Jacob Quinn  wrote:

> Micah/Wes,
> 
> Yes, I've been following the rust proposal thread with great interest. I do
> think that provides a great path forward: transferring the
> JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the
> "package history" technical challenges that in part led to the current
> setup and concerns. I think being able to utilize github issues would also
> be great; as I've mentioned elsewhere, it's much more traditional/expected
> in the Julia ecosystem.
> 
> I think the package could retain an independent versioning scheme. The
>> additional process would be voting on release candidates. If the Julia
>> folks want to try again and move development to a new, Julia-specific
>> apache/* repository and apply the ASF governance to the project, the
>> Arrow PMC could probably fast-track making Jacob a committer. In some
>> code donations / IP clearance, the contributors for the donated code
>> become committers as part of the transaction.
>>
> 
> These all sound great and would greatly facilitate a better integration
> under ASF governance. These points definitely resolve my main concerns.
> 
> As I commented on the rust thread, I'm mostly interested in the future of
> integration testing for rust/julia if they are split out into separate
> repos. In the current Julia implementation, we have all the code to read
> arrow json, and I just hand-generated the integration test data and
> committed them in the repo itself, but it doesn't interface with other
> languages (just reads arrow json, produces arrow file, reads arrow file,
> compares w/ original arrow json). I'm happy to help work on the details of
> what that looks like and pilot some solutions. I think with a solid
> inter-repo integration testing framework, we can keep a strong sync between
> projects.
> 
> -Jacob
> 
> 
> On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney  wrote:
> 
>> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield 
>> wrote:
>> >
>> > >
>> > > Ok, I've had a chance to discuss with a few other Julia developers and
>> > > review various options. I think it's best to drop the Julia code from
>> the
>> > > physical apache/arrow repo. The extra overhead on development, release
>> > > process, and user issue reporting and PR contributing are too much in
>> > > addition to the technical challenges that we never resolved involving
>> > > including the past Arrow.jl release version git trees in the
>> apache/arrow
>> > > repo.
>> >
>> >
>> > Hi Jacob,
>> > It seems you are on the new thread discussing a proposal for changing
>> > Rust's development model.   Would the proposal [1] address most of these
>> > concerns if Julia was set up in the same way?
>> >
>> >  It seems in the short term the stickiest point would be committer access
>> > to the new repos, and I suppose the release mechanics still might be
>> > challenging?
>>
>> I think the package could retain an independent versioning scheme. The
>> additional process would be voting on release candidates. If the Julia
>> folks want to try again and move development to a new, Julia-specific
>> apache/* repository and apply the ASF governance to the project, the
>> Arrow PMC could probably fast-track making Jacob a committer. In some
>> code donations / IP clearance, the contributors for the donated code
>> become committers as part of the transaction.
>>
>> >
>> > Thanks,
>> > Micah
>> >
>> > [1]
>> >
>> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit
>> >
>> > On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney  wrote:
>> >
>> > > I went back and read the mailing list discussions from September about
>> > > the donation and I would say there was not a clear enough statement
>> > > from us about what the donation and IP clearance meant as far as the
>> > > future of the Julia codebase. This is partly our fault — we have taken
>> > > in 9 other code

Re: Status of Arrow Julia implementation?

[Jacob Quinn]

Micah/Wes,

Yes, I've been following the rust proposal thread with great interest. I do
think that provides a great path forward: transferring the
JuliaData/Arrow.jl repo to apache/arrow-julia would help to solve the
"package history" technical challenges that in part led to the current
setup and concerns. I think being able to utilize github issues would also
be great; as I've mentioned elsewhere, it's much more traditional/expected
in the Julia ecosystem.

I think the package could retain an independent versioning scheme. The
> additional process would be voting on release candidates. If the Julia
> folks want to try again and move development to a new, Julia-specific
> apache/* repository and apply the ASF governance to the project, the
> Arrow PMC could probably fast-track making Jacob a committer. In some
> code donations / IP clearance, the contributors for the donated code
> become committers as part of the transaction.
>

These all sound great and would greatly facilitate a better integration
under ASF governance. These points definitely resolve my main concerns.

As I commented on the rust thread, I'm mostly interested in the future of
integration testing for rust/julia if they are split out into separate
repos. In the current Julia implementation, we have all the code to read
arrow json, and I just hand-generated the integration test data and
committed them in the repo itself, but it doesn't interface with other
languages (just reads arrow json, produces arrow file, reads arrow file,
compares w/ original arrow json). I'm happy to help work on the details of
what that looks like and pilot some solutions. I think with a solid
inter-repo integration testing framework, we can keep a strong sync between
projects.

-Jacob


On Sun, Apr 11, 2021 at 5:08 PM Wes McKinney  wrote:

> On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield 
> wrote:
> >
> > >
> > > Ok, I've had a chance to discuss with a few other Julia developers and
> > > review various options. I think it's best to drop the Julia code from
> the
> > > physical apache/arrow repo. The extra overhead on development, release
> > > process, and user issue reporting and PR contributing are too much in
> > > addition to the technical challenges that we never resolved involving
> > > including the past Arrow.jl release version git trees in the
> apache/arrow
> > > repo.
> >
> >
> > Hi Jacob,
> > It seems you are on the new thread discussing a proposal for changing
> > Rust's development model.   Would the proposal [1] address most of these
> > concerns if Julia was set up in the same way?
> >
> >  It seems in the short term the stickiest point would be committer access
> > to the new repos, and I suppose the release mechanics still might be
> > challenging?
>
> I think the package could retain an independent versioning scheme. The
> additional process would be voting on release candidates. If the Julia
> folks want to try again and move development to a new, Julia-specific
> apache/* repository and apply the ASF governance to the project, the
> Arrow PMC could probably fast-track making Jacob a committer. In some
> code donations / IP clearance, the contributors for the donated code
> become committers as part of the transaction.
>
> >
> > Thanks,
> > Micah
> >
> > [1]
> >
> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit
> >
> > On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney  wrote:
> >
> > > I went back and read the mailing list discussions from September about
> > > the donation and I would say there was not a clear enough statement
> > > from us about what the donation and IP clearance meant as far as the
> > > future of the Julia codebase. This is partly our fault — we have taken
> > > in 9 other code donations over the last 5 years, and in all cases the
> > > developers understood that they were to move their process to the
> > > Arrow repositories and communications channels.
> > >
> > > It did not occur to me at all that the code that you were putting in
> > > the Arrow repository would get treated like a read-only fork that you
> > > update periodically. If I had realized that, we wouldn't be in this
> > > situation.
> > >
> > > As a reminder about what Arrow and the ASF are all about: Community
> > > over Code. We think that building a collaborative, open community that
> > > works and plans together in public, makes decisions based on consensus
> > > with clear meritocratic ("doers decide") governance is the best way to
> > > build this project. The concerns that you have around the timing and
> > > frequency of releases for the Julia codebase are in my mind easy to
> > > resolve, and if you had indicated that having a customized process for
> > > Julia releases was a condition for your joining the community
> > > wholeheartedly, we would have been happy to help. I think that the
> > > benefits of common CI/CD infrastructure and opportunities to build
> > > deeper integrations between the Julia implementation 

Re: Status of Arrow Julia implementation?

[Wes McKinney]

On Sat, Apr 10, 2021 at 4:07 PM Micah Kornfield  wrote:
>
> >
> > Ok, I've had a chance to discuss with a few other Julia developers and
> > review various options. I think it's best to drop the Julia code from the
> > physical apache/arrow repo. The extra overhead on development, release
> > process, and user issue reporting and PR contributing are too much in
> > addition to the technical challenges that we never resolved involving
> > including the past Arrow.jl release version git trees in the apache/arrow
> > repo.
>
>
> Hi Jacob,
> It seems you are on the new thread discussing a proposal for changing
> Rust's development model.   Would the proposal [1] address most of these
> concerns if Julia was set up in the same way?
>
>  It seems in the short term the stickiest point would be committer access
> to the new repos, and I suppose the release mechanics still might be
> challenging?

I think the package could retain an independent versioning scheme. The
additional process would be voting on release candidates. If the Julia
folks want to try again and move development to a new, Julia-specific
apache/* repository and apply the ASF governance to the project, the
Arrow PMC could probably fast-track making Jacob a committer. In some
code donations / IP clearance, the contributors for the donated code
become committers as part of the transaction.

>
> Thanks,
> Micah
>
> [1]
> https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit
>
> On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney  wrote:
>
> > I went back and read the mailing list discussions from September about
> > the donation and I would say there was not a clear enough statement
> > from us about what the donation and IP clearance meant as far as the
> > future of the Julia codebase. This is partly our fault — we have taken
> > in 9 other code donations over the last 5 years, and in all cases the
> > developers understood that they were to move their process to the
> > Arrow repositories and communications channels.
> >
> > It did not occur to me at all that the code that you were putting in
> > the Arrow repository would get treated like a read-only fork that you
> > update periodically. If I had realized that, we wouldn't be in this
> > situation.
> >
> > As a reminder about what Arrow and the ASF are all about: Community
> > over Code. We think that building a collaborative, open community that
> > works and plans together in public, makes decisions based on consensus
> > with clear meritocratic ("doers decide") governance is the best way to
> > build this project. The concerns that you have around the timing and
> > frequency of releases for the Julia codebase are in my mind easy to
> > resolve, and if you had indicated that having a customized process for
> > Julia releases was a condition for your joining the community
> > wholeheartedly, we would have been happy to help. I think that the
> > benefits of common CI/CD infrastructure and opportunities to build
> > deeper integrations between the Julia implementation and the other
> > implementations (imagine... Julia kernels running in DataFusion?)
> > would outweigh the sense of "loss of control" from developing within a
> > larger project.
> >
> > On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn 
> > wrote:
> > >
> > > Responses inline below:
> > >
> > > On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão <
> > > jorgecarlei...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > > you all did not attempt to work in the community for any meaningful
> > > > amount of time and
> > > > are choosing not to try based on the perception that it will create
> > > > unacceptable overhead for you
> > > >
> > > > It is not self-evident to me that Julia's community was sufficiently
> > > > informed about what they
> > > > had to give in in terms of process and release management when merging
> > /
> > > > donating.
> > > >
> > >
> > > Yes, it was pretty unclear what the process was if we needed to do any
> > kind
> > > of patch release. I know that has been sorted out better recently, but
> > back
> > > in November, it didn't really seem like an option (i.e. independent
> > > language patch releases).
> > >
> > >
> > > > IMO this is a plausible explanation as to why the donation was made and
> > > > then later abandoned.
> > > >
> > > >
> > > I'll just note that the "abandonment" can only be a perception from the
> > > apache/arrow side of things, but as I mentioned above, I also tried to
> > > clearly state in the julia/Arrow/README that the development process
> > would
> > > continue with the JuliaData/Arrow.jl repo as the main "dev" branch, with
> > > changes being upstreamed to the apache/arrow repo, which was followed
> > > through, having an upstream of commits right before the 3.0.0 release,
> > and
> > > I was planning on doing the same soon for the 4.0.0 release. That is to
> > > say, the Julia implementation has continued progressing forward quite
> > > rapidly, IMO, but I ca

Re: Status of Arrow Julia implementation?

[Micah Kornfield]

>
> Ok, I've had a chance to discuss with a few other Julia developers and
> review various options. I think it's best to drop the Julia code from the
> physical apache/arrow repo. The extra overhead on development, release
> process, and user issue reporting and PR contributing are too much in
> addition to the technical challenges that we never resolved involving
> including the past Arrow.jl release version git trees in the apache/arrow
> repo.


Hi Jacob,
It seems you are on the new thread discussing a proposal for changing
Rust's development model.   Would the proposal [1] address most of these
concerns if Julia was set up in the same way?

 It seems in the short term the stickiest point would be committer access
to the new repos, and I suppose the release mechanics still might be
challenging?

Thanks,
Micah

[1]
https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit

On Wed, Apr 7, 2021 at 4:17 AM Wes McKinney  wrote:

> I went back and read the mailing list discussions from September about
> the donation and I would say there was not a clear enough statement
> from us about what the donation and IP clearance meant as far as the
> future of the Julia codebase. This is partly our fault — we have taken
> in 9 other code donations over the last 5 years, and in all cases the
> developers understood that they were to move their process to the
> Arrow repositories and communications channels.
>
> It did not occur to me at all that the code that you were putting in
> the Arrow repository would get treated like a read-only fork that you
> update periodically. If I had realized that, we wouldn't be in this
> situation.
>
> As a reminder about what Arrow and the ASF are all about: Community
> over Code. We think that building a collaborative, open community that
> works and plans together in public, makes decisions based on consensus
> with clear meritocratic ("doers decide") governance is the best way to
> build this project. The concerns that you have around the timing and
> frequency of releases for the Julia codebase are in my mind easy to
> resolve, and if you had indicated that having a customized process for
> Julia releases was a condition for your joining the community
> wholeheartedly, we would have been happy to help. I think that the
> benefits of common CI/CD infrastructure and opportunities to build
> deeper integrations between the Julia implementation and the other
> implementations (imagine... Julia kernels running in DataFusion?)
> would outweigh the sense of "loss of control" from developing within a
> larger project.
>
> On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn 
> wrote:
> >
> > Responses inline below:
> >
> > On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão <
> > jorgecarlei...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > > you all did not attempt to work in the community for any meaningful
> > > amount of time and
> > > are choosing not to try based on the perception that it will create
> > > unacceptable overhead for you
> > >
> > > It is not self-evident to me that Julia's community was sufficiently
> > > informed about what they
> > > had to give in in terms of process and release management when merging
> /
> > > donating.
> > >
> >
> > Yes, it was pretty unclear what the process was if we needed to do any
> kind
> > of patch release. I know that has been sorted out better recently, but
> back
> > in November, it didn't really seem like an option (i.e. independent
> > language patch releases).
> >
> >
> > > IMO this is a plausible explanation as to why the donation was made and
> > > then later abandoned.
> > >
> > >
> > I'll just note that the "abandonment" can only be a perception from the
> > apache/arrow side of things, but as I mentioned above, I also tried to
> > clearly state in the julia/Arrow/README that the development process
> would
> > continue with the JuliaData/Arrow.jl repo as the main "dev" branch, with
> > changes being upstreamed to the apache/arrow repo, which was followed
> > through, having an upstream of commits right before the 3.0.0 release,
> and
> > I was planning on doing the same soon for the 4.0.0 release. That is to
> > say, the Julia implementation has continued progressing forward quite
> > rapidly, IMO, but I can see that perhaps apache/arrow repo members may
> have
> > viewed it as "abandoned".
> >
> >
> > > I do not fully understand why the pain points Jacob mentioned were not
> > > brought up to the mailing list sooner, though.
> > >
> >
> > To be honest and frank, I didn't have pain points with the development
> > process I outlined when the code was donated and as stated in the README.
> > That was the process that made the donation possible and I imagined would
> > work well going forward, and has, until this thread started and it was
> > pointed out that this process isn't viable. The pain points were
> discussed
> > with the initial code donation, but in my mind were resolved with the
> > development process 

Re: Status of Arrow Julia implementation?

[Wes McKinney]

I went back and read the mailing list discussions from September about
the donation and I would say there was not a clear enough statement
from us about what the donation and IP clearance meant as far as the
future of the Julia codebase. This is partly our fault — we have taken
in 9 other code donations over the last 5 years, and in all cases the
developers understood that they were to move their process to the
Arrow repositories and communications channels.

It did not occur to me at all that the code that you were putting in
the Arrow repository would get treated like a read-only fork that you
update periodically. If I had realized that, we wouldn't be in this
situation.

As a reminder about what Arrow and the ASF are all about: Community
over Code. We think that building a collaborative, open community that
works and plans together in public, makes decisions based on consensus
with clear meritocratic ("doers decide") governance is the best way to
build this project. The concerns that you have around the timing and
frequency of releases for the Julia codebase are in my mind easy to
resolve, and if you had indicated that having a customized process for
Julia releases was a condition for your joining the community
wholeheartedly, we would have been happy to help. I think that the
benefits of common CI/CD infrastructure and opportunities to build
deeper integrations between the Julia implementation and the other
implementations (imagine... Julia kernels running in DataFusion?)
would outweigh the sense of "loss of control" from developing within a
larger project.

On Wed, Apr 7, 2021 at 12:16 AM Jacob Quinn  wrote:
>
> Responses inline below:
>
> On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão <
> jorgecarlei...@gmail.com> wrote:
>
> > Hi,
> >
> > > you all did not attempt to work in the community for any meaningful
> > amount of time and
> > are choosing not to try based on the perception that it will create
> > unacceptable overhead for you
> >
> > It is not self-evident to me that Julia's community was sufficiently
> > informed about what they
> > had to give in in terms of process and release management when merging /
> > donating.
> >
>
> Yes, it was pretty unclear what the process was if we needed to do any kind
> of patch release. I know that has been sorted out better recently, but back
> in November, it didn't really seem like an option (i.e. independent
> language patch releases).
>
>
> > IMO this is a plausible explanation as to why the donation was made and
> > then later abandoned.
> >
> >
> I'll just note that the "abandonment" can only be a perception from the
> apache/arrow side of things, but as I mentioned above, I also tried to
> clearly state in the julia/Arrow/README that the development process would
> continue with the JuliaData/Arrow.jl repo as the main "dev" branch, with
> changes being upstreamed to the apache/arrow repo, which was followed
> through, having an upstream of commits right before the 3.0.0 release, and
> I was planning on doing the same soon for the 4.0.0 release. That is to
> say, the Julia implementation has continued progressing forward quite
> rapidly, IMO, but I can see that perhaps apache/arrow repo members may have
> viewed it as "abandoned".
>
>
> > I do not fully understand why the pain points Jacob mentioned were not
> > brought up to the mailing list sooner, though.
> >
>
> To be honest and frank, I didn't have pain points with the development
> process I outlined when the code was donated and as stated in the README.
> That was the process that made the donation possible and I imagined would
> work well going forward, and has, until this thread started and it was
> pointed out that this process isn't viable. The pain points were discussed
> with the initial code donation, but in my mind were resolved with the
> development process that was decided upon.
>
>
> > This made us unable to potentially take corrective measures. I think that
> > this is why everyone was taken a bit by surprise with this.
> >
> > Best,
> > Jorge
> >
> >
> > On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney  wrote:
> >
> > > hi Jacob — sorry to hear that. It's a bummer that you all did not
> > > attempt to work in the community for any meaningful amount of time and
> > > are choosing not to try based on the perception that it will create
> > > unacceptable overhead for you. I believe the benefits would outweigh
> > > the costs, but I suppose we will have to agree to disagree.
> > >
> > > Can you prepare a pull request to do the requisite repository surgery?
> > > I hope the development goes well in the future and look forward to
> > > seeing folks from the Julia ecosystem engaged here on growing the
> > > Arrow ecosystem.
> > >
> > > Thanks,
> > > Wes
> > >
> > > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn 
> > wrote:
> > > >
> > > > Ok, I've had a chance to discuss with a few other Julia developers and
> > > > review various options. I think it's best to drop the Julia code from
> > t

Re: Status of Arrow Julia implementation?

[Jacob Quinn]

Responses inline below:

On Tue, Apr 6, 2021 at 9:46 PM Jorge Cardoso Leitão <
jorgecarlei...@gmail.com> wrote:

> Hi,
>
> > you all did not attempt to work in the community for any meaningful
> amount of time and
> are choosing not to try based on the perception that it will create
> unacceptable overhead for you
>
> It is not self-evident to me that Julia's community was sufficiently
> informed about what they
> had to give in in terms of process and release management when merging /
> donating.
>

Yes, it was pretty unclear what the process was if we needed to do any kind
of patch release. I know that has been sorted out better recently, but back
in November, it didn't really seem like an option (i.e. independent
language patch releases).


> IMO this is a plausible explanation as to why the donation was made and
> then later abandoned.
>
>
I'll just note that the "abandonment" can only be a perception from the
apache/arrow side of things, but as I mentioned above, I also tried to
clearly state in the julia/Arrow/README that the development process would
continue with the JuliaData/Arrow.jl repo as the main "dev" branch, with
changes being upstreamed to the apache/arrow repo, which was followed
through, having an upstream of commits right before the 3.0.0 release, and
I was planning on doing the same soon for the 4.0.0 release. That is to
say, the Julia implementation has continued progressing forward quite
rapidly, IMO, but I can see that perhaps apache/arrow repo members may have
viewed it as "abandoned".


> I do not fully understand why the pain points Jacob mentioned were not
> brought up to the mailing list sooner, though.
>

To be honest and frank, I didn't have pain points with the development
process I outlined when the code was donated and as stated in the README.
That was the process that made the donation possible and I imagined would
work well going forward, and has, until this thread started and it was
pointed out that this process isn't viable. The pain points were discussed
with the initial code donation, but in my mind were resolved with the
development process that was decided upon.


> This made us unable to potentially take corrective measures. I think that
> this is why everyone was taken a bit by surprise with this.
>
> Best,
> Jorge
>
>
> On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney  wrote:
>
> > hi Jacob — sorry to hear that. It's a bummer that you all did not
> > attempt to work in the community for any meaningful amount of time and
> > are choosing not to try based on the perception that it will create
> > unacceptable overhead for you. I believe the benefits would outweigh
> > the costs, but I suppose we will have to agree to disagree.
> >
> > Can you prepare a pull request to do the requisite repository surgery?
> > I hope the development goes well in the future and look forward to
> > seeing folks from the Julia ecosystem engaged here on growing the
> > Arrow ecosystem.
> >
> > Thanks,
> > Wes
> >
> > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn 
> wrote:
> > >
> > > Ok, I've had a chance to discuss with a few other Julia developers and
> > > review various options. I think it's best to drop the Julia code from
> the
> > > physical apache/arrow repo. The extra overhead on development, release
> > > process, and user issue reporting and PR contributing are too much in
> > > addition to the technical challenges that we never resolved involving
> > > including the past Arrow.jl release version git trees in the
> apache/arrow
> > > repo.
> > >
> > > We're still very much committed to working on the Julia implementation
> > and
> > > participating in the broader arrow community. I've enjoyed following
> the
> > > user/dev mailing lists and will continue to do so. We monitor format
> > > proposals and try to implement new functionality as quickly as
> possible.
> > We
> > > got the initial arrow flight proto code generated just last night in
> > fact.
> > > I'd still like to explore official integration with the archery test
> > suite
> > > to solidify the Julia implementation with integration tests; I think
> that
> > > would be very valuable for long-term confidence in the cross-language
> > > support of the Julia implementation.
> > >
> > > We realize one of the main implications will probably be dropping Julia
> > > from the list of "official implementations". We're encouraged by the
> many
> > > users who have already started using the Julia implementation and will
> > > strive to maintain a high rate of issue responsiveness and feature
> > > development to maintain project confidence. If there's a possibility of
> > > being included somewhere as an "unofficial" or "semi-official"
> > > implementation, we'd love to still be bundled with the broader arrow
> > > project somehow, like, for example, showing how Julia integrates with
> the
> > > archery test suite, once the work there is done.
> > >
> > > Best,
> > >
> > > -Jacob
> > >
> > >
> > >
> > > On Tue, Mar 30, 2021 at 4:10 

Re: Status of Arrow Julia implementation?

[Jorge Cardoso Leitão]

Hi,

> you all did not attempt to work in the community for any meaningful
amount of time and
are choosing not to try based on the perception that it will create
unacceptable overhead for you

It is not self-evident to me that Julia's community was sufficiently
informed about what they
had to give in in terms of process and release management when merging /
donating.
IMO this is a plausible explanation as to why the donation was made and
then later abandoned.

I do not fully understand why the pain points Jacob mentioned were not
brought up to the mailing list sooner, though.
This made us unable to potentially take corrective measures. I think that
this is why everyone was taken a bit by surprise with this.

Best,
Jorge


On Fri, Apr 2, 2021 at 10:18 PM Wes McKinney  wrote:

> hi Jacob — sorry to hear that. It's a bummer that you all did not
> attempt to work in the community for any meaningful amount of time and
> are choosing not to try based on the perception that it will create
> unacceptable overhead for you. I believe the benefits would outweigh
> the costs, but I suppose we will have to agree to disagree.
>
> Can you prepare a pull request to do the requisite repository surgery?
> I hope the development goes well in the future and look forward to
> seeing folks from the Julia ecosystem engaged here on growing the
> Arrow ecosystem.
>
> Thanks,
> Wes
>
> On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn  wrote:
> >
> > Ok, I've had a chance to discuss with a few other Julia developers and
> > review various options. I think it's best to drop the Julia code from the
> > physical apache/arrow repo. The extra overhead on development, release
> > process, and user issue reporting and PR contributing are too much in
> > addition to the technical challenges that we never resolved involving
> > including the past Arrow.jl release version git trees in the apache/arrow
> > repo.
> >
> > We're still very much committed to working on the Julia implementation
> and
> > participating in the broader arrow community. I've enjoyed following the
> > user/dev mailing lists and will continue to do so. We monitor format
> > proposals and try to implement new functionality as quickly as possible.
> We
> > got the initial arrow flight proto code generated just last night in
> fact.
> > I'd still like to explore official integration with the archery test
> suite
> > to solidify the Julia implementation with integration tests; I think that
> > would be very valuable for long-term confidence in the cross-language
> > support of the Julia implementation.
> >
> > We realize one of the main implications will probably be dropping Julia
> > from the list of "official implementations". We're encouraged by the many
> > users who have already started using the Julia implementation and will
> > strive to maintain a high rate of issue responsiveness and feature
> > development to maintain project confidence. If there's a possibility of
> > being included somewhere as an "unofficial" or "semi-official"
> > implementation, we'd love to still be bundled with the broader arrow
> > project somehow, like, for example, showing how Julia integrates with the
> > archery test suite, once the work there is done.
> >
> > Best,
> >
> > -Jacob
> >
> >
> >
> > On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney 
> wrote:
> >
> > > Also, on the issue that there are no Julia-focused PMC members — note
> > > that I helped the JavaScript folks make their own independent releases
> > > for quite a while: called the votes (e.g. [1]), helped get people to
> > > verify and vote on the releases. After a time, it was decided to stop
> > > releasing independently because there wasn't enough development
> > > activity to justify it.
> > >
> > > [1]: https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html
> > >
> > > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney 
> wrote:
> > > >
> > > > hi Jacob,
> > > >
> > > > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn 
> > > wrote:
> > > > >
> > > > > I can comment as the primary apache arrow liaison for the Arrow.jl
> > > > > repository and original code donator.
> > > > >
> > > > > I apologize for the "surprise", but I commented a few times in
> various
> > > > > places and put a snippet in the README
> > > > > <
> > >
> https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository
> > > >
> > > > > about
> > > > > the approach I wanted to take w/ the Julia implementation in terms
> of
> > > > > keeping the JuliaData/Arrow.jl repository as a "dev branch" of
> sorts
> > > of the
> > > > > apache/arrow code, upstreaming changes periodically. There's even a
> > > script
> > > > > <
> > >
> https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl
> > > >
> > > > > I wrote to mostly automate this upstreaming. I realize now that I
> > > didn't
> > > > > consider the "Arrow PMC" position on this kind of setup or seek to
> > > affirm
> > > > > that it 

Re: Status of Arrow Julia implementation?

[Antoine Pitrou]

Hi Jacob,

Le 02/04/2021 à 22:03, Jacob Quinn a écrit :


We realize one of the main implications will probably be dropping Julia
from the list of "official implementations". We're encouraged by the many
users who have already started using the Julia implementation and will
strive to maintain a high rate of issue responsiveness and feature
development to maintain project confidence. If there's a possibility of
being included somewhere as an "unofficial" or "semi-official"
implementation, we'd love to still be bundled with the broader arrow
project somehow, like, for example, showing how Julia integrates with the
archery test suite, once the work there is done.


It seems reasonable to me to mention the Julia implementation somewhere 
in the docs, for example by keeping a mention in the existing status page:

https://arrow.apache.org/docs/status.html

Regards

Antoine.

Re: Status of Arrow Julia implementation?

[Wes McKinney]

hi Jacob — sorry to hear that. It's a bummer that you all did not
attempt to work in the community for any meaningful amount of time and
are choosing not to try based on the perception that it will create
unacceptable overhead for you. I believe the benefits would outweigh
the costs, but I suppose we will have to agree to disagree.

Can you prepare a pull request to do the requisite repository surgery?
I hope the development goes well in the future and look forward to
seeing folks from the Julia ecosystem engaged here on growing the
Arrow ecosystem.

Thanks,
Wes

On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn  wrote:
>
> Ok, I've had a chance to discuss with a few other Julia developers and
> review various options. I think it's best to drop the Julia code from the
> physical apache/arrow repo. The extra overhead on development, release
> process, and user issue reporting and PR contributing are too much in
> addition to the technical challenges that we never resolved involving
> including the past Arrow.jl release version git trees in the apache/arrow
> repo.
>
> We're still very much committed to working on the Julia implementation and
> participating in the broader arrow community. I've enjoyed following the
> user/dev mailing lists and will continue to do so. We monitor format
> proposals and try to implement new functionality as quickly as possible. We
> got the initial arrow flight proto code generated just last night in fact.
> I'd still like to explore official integration with the archery test suite
> to solidify the Julia implementation with integration tests; I think that
> would be very valuable for long-term confidence in the cross-language
> support of the Julia implementation.
>
> We realize one of the main implications will probably be dropping Julia
> from the list of "official implementations". We're encouraged by the many
> users who have already started using the Julia implementation and will
> strive to maintain a high rate of issue responsiveness and feature
> development to maintain project confidence. If there's a possibility of
> being included somewhere as an "unofficial" or "semi-official"
> implementation, we'd love to still be bundled with the broader arrow
> project somehow, like, for example, showing how Julia integrates with the
> archery test suite, once the work there is done.
>
> Best,
>
> -Jacob
>
>
>
> On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney  wrote:
>
> > Also, on the issue that there are no Julia-focused PMC members — note
> > that I helped the JavaScript folks make their own independent releases
> > for quite a while: called the votes (e.g. [1]), helped get people to
> > verify and vote on the releases. After a time, it was decided to stop
> > releasing independently because there wasn't enough development
> > activity to justify it.
> >
> > [1]: https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html
> >
> > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney  wrote:
> > >
> > > hi Jacob,
> > >
> > > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn 
> > wrote:
> > > >
> > > > I can comment as the primary apache arrow liaison for the Arrow.jl
> > > > repository and original code donator.
> > > >
> > > > I apologize for the "surprise", but I commented a few times in various
> > > > places and put a snippet in the README
> > > > <
> > https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository
> > >
> > > > about
> > > > the approach I wanted to take w/ the Julia implementation in terms of
> > > > keeping the JuliaData/Arrow.jl repository as a "dev branch" of sorts
> > of the
> > > > apache/arrow code, upstreaming changes periodically. There's even a
> > script
> > > > <
> > https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl
> > >
> > > > I wrote to mostly automate this upstreaming. I realize now that I
> > didn't
> > > > consider the "Arrow PMC" position on this kind of setup or seek to
> > affirm
> > > > that it would be ok to approach things like this.
> > > >
> > > > The reality is that Julia users are very engrained to expect Julia
> > packages
> > > > to live in a single stand-alone github repo, where issues can be
> > opened,
> > > > and pull requests are welcome. It was hard and still is hard to imagine
> > > > "turning that off", since I believe we would lose a lot of valuable bug
> > > > reports and first-time contributions. This isn't necessarily any fault
> > of
> > > > how the bug report/contribution process is handled for the arrow
> > project
> > > > overall, though I'm also aware that there's a desire to make it easier
> > > >
> > > >
> > > <
> > https://lists.apache.org/x/thread.html/r8817dfba08ef8daa210956db69d513fd27b7a751d28fb8f27e39cc7e@%3Cdev.arrow.apache.org%3E
> > >
> > > > and
> > > > it currently requires more and different effort than Julia users are
> > used
> > > > to. I think it's more from how open, welcoming, and how strong the
> > culture
> > > > i

Re: Status of Arrow Julia implementation?

[Jacob Quinn]

Ok, I've had a chance to discuss with a few other Julia developers and
review various options. I think it's best to drop the Julia code from the
physical apache/arrow repo. The extra overhead on development, release
process, and user issue reporting and PR contributing are too much in
addition to the technical challenges that we never resolved involving
including the past Arrow.jl release version git trees in the apache/arrow
repo.

We're still very much committed to working on the Julia implementation and
participating in the broader arrow community. I've enjoyed following the
user/dev mailing lists and will continue to do so. We monitor format
proposals and try to implement new functionality as quickly as possible. We
got the initial arrow flight proto code generated just last night in fact.
I'd still like to explore official integration with the archery test suite
to solidify the Julia implementation with integration tests; I think that
would be very valuable for long-term confidence in the cross-language
support of the Julia implementation.

We realize one of the main implications will probably be dropping Julia
from the list of "official implementations". We're encouraged by the many
users who have already started using the Julia implementation and will
strive to maintain a high rate of issue responsiveness and feature
development to maintain project confidence. If there's a possibility of
being included somewhere as an "unofficial" or "semi-official"
implementation, we'd love to still be bundled with the broader arrow
project somehow, like, for example, showing how Julia integrates with the
archery test suite, once the work there is done.

Best,

-Jacob



On Tue, Mar 30, 2021 at 4:10 PM Wes McKinney  wrote:

> Also, on the issue that there are no Julia-focused PMC members — note
> that I helped the JavaScript folks make their own independent releases
> for quite a while: called the votes (e.g. [1]), helped get people to
> verify and vote on the releases. After a time, it was decided to stop
> releasing independently because there wasn't enough development
> activity to justify it.
>
> [1]: https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html
>
> On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney  wrote:
> >
> > hi Jacob,
> >
> > On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn 
> wrote:
> > >
> > > I can comment as the primary apache arrow liaison for the Arrow.jl
> > > repository and original code donator.
> > >
> > > I apologize for the "surprise", but I commented a few times in various
> > > places and put a snippet in the README
> > > <
> https://github.com/apache/arrow/tree/master/julia/Arrow#difference-between-this-code-and-the-juliadataarrowjl-repository
> >
> > > about
> > > the approach I wanted to take w/ the Julia implementation in terms of
> > > keeping the JuliaData/Arrow.jl repository as a "dev branch" of sorts
> of the
> > > apache/arrow code, upstreaming changes periodically. There's even a
> script
> > > <
> https://github.com/JuliaData/Arrow.jl/blob/main/scripts/update_apache_arrow_code.jl
> >
> > > I wrote to mostly automate this upstreaming. I realize now that I
> didn't
> > > consider the "Arrow PMC" position on this kind of setup or seek to
> affirm
> > > that it would be ok to approach things like this.
> > >
> > > The reality is that Julia users are very engrained to expect Julia
> packages
> > > to live in a single stand-alone github repo, where issues can be
> opened,
> > > and pull requests are welcome. It was hard and still is hard to imagine
> > > "turning that off", since I believe we would lose a lot of valuable bug
> > > reports and first-time contributions. This isn't necessarily any fault
> of
> > > how the bug report/contribution process is handled for the arrow
> project
> > > overall, though I'm also aware that there's a desire to make it easier
> > >
> > >
> > <
> https://lists.apache.org/x/thread.html/r8817dfba08ef8daa210956db69d513fd27b7a751d28fb8f27e39cc7e@%3Cdev.arrow.apache.org%3E
> >
> > > and
> > > it currently requires more and different effort than Julia users are
> used
> > > to. I think it's more from how open, welcoming, and how strong the
> culture
> > > is in Julia around encouraging community contributions and the tight
> > > integration with github and its open-source project management tools.
> > >
> >
> > Well, we are on track to having 1000 different people contribute to
> > the project and have over 12,000 issues, so I don't think there is
> > evidence that we are failing to attract new contributors or that
> > feature requests / bugs aren't being reported. The way that we work is
> > _different_, so adapting to the Apache process will require change.
> >
> > > Additionally, I was and still am concerned about the overall release
> > > process of the apache/arrow project. I know there have been efforts
> there
> > > as well to make it easier for individual languages to release on their
> own
> > > cadence, but just anecdotally, the JuliaData/A

Re: Status of Arrow Julia implementation?

[Wes McKinney]

Also, on the issue that there are no Julia-focused PMC members — note
that I helped the JavaScript folks make their own independent releases
for quite a while: called the votes (e.g. [1]), helped get people to
verify and vote on the releases. After a time, it was decided to stop
releasing independently because there wasn't enough development
activity to justify it.

[1]: https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html

On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney  wrote:
>
> hi Jacob,
>
> On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn  wrote:
> >
> > I can comment as the primary apache arrow liaison for the Arrow.jl
> > repository and original code donator.
> >
> > I apologize for the "surprise", but I commented a few times in various
> > places and put a snippet in the README
> > 
> > about
> > the approach I wanted to take w/ the Julia implementation in terms of
> > keeping the JuliaData/Arrow.jl repository as a "dev branch" of sorts of the
> > apache/arrow code, upstreaming changes periodically. There's even a script
> > 
> > I wrote to mostly automate this upstreaming. I realize now that I didn't
> > consider the "Arrow PMC" position on this kind of setup or seek to affirm
> > that it would be ok to approach things like this.
> >
> > The reality is that Julia users are very engrained to expect Julia packages
> > to live in a single stand-alone github repo, where issues can be opened,
> > and pull requests are welcome. It was hard and still is hard to imagine
> > "turning that off", since I believe we would lose a lot of valuable bug
> > reports and first-time contributions. This isn't necessarily any fault of
> > how the bug report/contribution process is handled for the arrow project
> > overall, though I'm also aware that there's a desire to make it easier
> >
> >
> 
> > and
> > it currently requires more and different effort than Julia users are used
> > to. I think it's more from how open, welcoming, and how strong the culture
> > is in Julia around encouraging community contributions and the tight
> > integration with github and its open-source project management tools.
> >
>
> Well, we are on track to having 1000 different people contribute to
> the project and have over 12,000 issues, so I don't think there is
> evidence that we are failing to attract new contributors or that
> feature requests / bugs aren't being reported. The way that we work is
> _different_, so adapting to the Apache process will require change.
>
> > Additionally, I was and still am concerned about the overall release
> > process of the apache/arrow project. I know there have been efforts there
> > as well to make it easier for individual languages to release on their own
> > cadence, but just anecdotally, the JuliaData/Arrow.jl has had/needed/wanted
> > 10 patch and minor releases since the original code donation, whereas the
> > apache/arrow project has had one (3.0.0). This leads to some of the
> > concerns I have with restricting development to just the apache/arrow
> > repository: how exactly does the release process work for individual
> > languages who may desire independent releases apart from the quarterly
> > overall project releases? I think from the Rust thread I remember that you
> > just need a group of language contributors to all agree, but what if I'm
> > the only "active" Julia contributor? It's also unclear what the
> > expectations are for actual development: with the original code donation
> > PRs, I know Neal "reviewed" the PRs, but perhaps missed the details around
> > how I proposed development continue going forward. Is it required to have a
> > certain number of reviews before merging? On the Julia side, I can try to
> > encourage/push for those who have contributed to the JuliaData/Arrow.jl
> > repository to help review PRs to apache/arrow, but I also can't guarantee
> > we would always have someone to review. It just feels pretty awkward if I
> > keep needing to ping non-Julia people to "review" a PR to merge it. Perhaps
> > this is just a problem of the overall Julia implementation "smallness" in
> > terms of contributors, but I'm not sure on the best answer here.
> >
>
> Several things here:
>
> * If you want to do separate Julia releases, you are free to do that,
> but you have to follow the process (voting on the mailing list,
> publishing GPG-signed source artifacts)
> * If you had been working "in the community" since November, you would
> probably already be a committer, so there is a bootstrapping here that
> has failed to take place. In the meantime, we are more than happy to
> help you "earn your wings" (as a committer) as quickly as possible.
> Bu

Re: Status of Arrow Julia implementation?

[Wes McKinney]

hi Jacob,

On Tue, Mar 30, 2021 at 4:18 PM Jacob Quinn  wrote:
>
> I can comment as the primary apache arrow liaison for the Arrow.jl
> repository and original code donator.
>
> I apologize for the "surprise", but I commented a few times in various
> places and put a snippet in the README
> 
> about
> the approach I wanted to take w/ the Julia implementation in terms of
> keeping the JuliaData/Arrow.jl repository as a "dev branch" of sorts of the
> apache/arrow code, upstreaming changes periodically. There's even a script
> 
> I wrote to mostly automate this upstreaming. I realize now that I didn't
> consider the "Arrow PMC" position on this kind of setup or seek to affirm
> that it would be ok to approach things like this.
>
> The reality is that Julia users are very engrained to expect Julia packages
> to live in a single stand-alone github repo, where issues can be opened,
> and pull requests are welcome. It was hard and still is hard to imagine
> "turning that off", since I believe we would lose a lot of valuable bug
> reports and first-time contributions. This isn't necessarily any fault of
> how the bug report/contribution process is handled for the arrow project
> overall, though I'm also aware that there's a desire to make it easier
>
>

> and
> it currently requires more and different effort than Julia users are used
> to. I think it's more from how open, welcoming, and how strong the culture
> is in Julia around encouraging community contributions and the tight
> integration with github and its open-source project management tools.
>

Well, we are on track to having 1000 different people contribute to
the project and have over 12,000 issues, so I don't think there is
evidence that we are failing to attract new contributors or that
feature requests / bugs aren't being reported. The way that we work is
_different_, so adapting to the Apache process will require change.

> Additionally, I was and still am concerned about the overall release
> process of the apache/arrow project. I know there have been efforts there
> as well to make it easier for individual languages to release on their own
> cadence, but just anecdotally, the JuliaData/Arrow.jl has had/needed/wanted
> 10 patch and minor releases since the original code donation, whereas the
> apache/arrow project has had one (3.0.0). This leads to some of the
> concerns I have with restricting development to just the apache/arrow
> repository: how exactly does the release process work for individual
> languages who may desire independent releases apart from the quarterly
> overall project releases? I think from the Rust thread I remember that you
> just need a group of language contributors to all agree, but what if I'm
> the only "active" Julia contributor? It's also unclear what the
> expectations are for actual development: with the original code donation
> PRs, I know Neal "reviewed" the PRs, but perhaps missed the details around
> how I proposed development continue going forward. Is it required to have a
> certain number of reviews before merging? On the Julia side, I can try to
> encourage/push for those who have contributed to the JuliaData/Arrow.jl
> repository to help review PRs to apache/arrow, but I also can't guarantee
> we would always have someone to review. It just feels pretty awkward if I
> keep needing to ping non-Julia people to "review" a PR to merge it. Perhaps
> this is just a problem of the overall Julia implementation "smallness" in
> terms of contributors, but I'm not sure on the best answer here.
>

Several things here:

* If you want to do separate Julia releases, you are free to do that,
but you have to follow the process (voting on the mailing list,
publishing GPG-signed source artifacts)
* If you had been working "in the community" since November, you would
probably already be a committer, so there is a bootstrapping here that
has failed to take place. In the meantime, we are more than happy to
help you "earn your wings" (as a committer) as quickly as possible.
But from my perspective, I see a code donation and two other commits,
which isn't enough to make a case for committership.

> So in short, I'm not sure on the best path forward. I think strictly
> restricting development to the apache/arrow physical repository would
> actively hurt the progress of the Julia implementation, whereas it *has*
> been progressing with increasing momentum since first released. There are
> posts on the Julia discourse forum, in the Julia slack and zulip
> communities, and quite a few issues/PRs being opened at the
> JuliaData/Arrow.jl repository. There have been several calls for arrow
> flight support, with 

Re: Status of Arrow Julia implementation?

[Jacob Quinn]

I can comment as the primary apache arrow liaison for the Arrow.jl
repository and original code donator.

I apologize for the "surprise", but I commented a few times in various
places and put a snippet in the README

about
the approach I wanted to take w/ the Julia implementation in terms of
keeping the JuliaData/Arrow.jl repository as a "dev branch" of sorts of the
apache/arrow code, upstreaming changes periodically. There's even a script

I wrote to mostly automate this upstreaming. I realize now that I didn't
consider the "Arrow PMC" position on this kind of setup or seek to affirm
that it would be ok to approach things like this.

The reality is that Julia users are very engrained to expect Julia packages
to live in a single stand-alone github repo, where issues can be opened,
and pull requests are welcome. It was hard and still is hard to imagine
"turning that off", since I believe we would lose a lot of valuable bug
reports and first-time contributions. This isn't necessarily any fault of
how the bug report/contribution process is handled for the arrow project
overall, though I'm also aware that there's a desire to make it easier

and
it currently requires more and different effort than Julia users are used
to. I think it's more from how open, welcoming, and how strong the culture
is in Julia around encouraging community contributions and the tight
integration with github and its open-source project management tools.

Additionally, I was and still am concerned about the overall release
process of the apache/arrow project. I know there have been efforts there
as well to make it easier for individual languages to release on their own
cadence, but just anecdotally, the JuliaData/Arrow.jl has had/needed/wanted
10 patch and minor releases since the original code donation, whereas the
apache/arrow project has had one (3.0.0). This leads to some of the
concerns I have with restricting development to just the apache/arrow
repository: how exactly does the release process work for individual
languages who may desire independent releases apart from the quarterly
overall project releases? I think from the Rust thread I remember that you
just need a group of language contributors to all agree, but what if I'm
the only "active" Julia contributor? It's also unclear what the
expectations are for actual development: with the original code donation
PRs, I know Neal "reviewed" the PRs, but perhaps missed the details around
how I proposed development continue going forward. Is it required to have a
certain number of reviews before merging? On the Julia side, I can try to
encourage/push for those who have contributed to the JuliaData/Arrow.jl
repository to help review PRs to apache/arrow, but I also can't guarantee
we would always have someone to review. It just feels pretty awkward if I
keep needing to ping non-Julia people to "review" a PR to merge it. Perhaps
this is just a problem of the overall Julia implementation "smallness" in
terms of contributors, but I'm not sure on the best answer here.

So in short, I'm not sure on the best path forward. I think strictly
restricting development to the apache/arrow physical repository would
actively hurt the progress of the Julia implementation, whereas it *has*
been progressing with increasing momentum since first released. There are
posts on the Julia discourse forum, in the Julia slack and zulip
communities, and quite a few issues/PRs being opened at the
JuliaData/Arrow.jl repository. There have been several calls for arrow
flight support, with a member from Julia Computing actually close to
releasing a gRPC client
 specifically
to help with flight support. But in terms of actual committers, it's been
primarily just myself, with a few minor contributions by others.

I guess the big question that comes to mind is what are the hard
requirements to be considered an "official implementation"? Does the code
*have* to live in the same physical repo? Or if it passed the series of
archery integration tests, would that be enough? I apologize for my
naivete/inexperience on all things "apache", but I imagine that's a big
part of it: having official development/releases through the apache/arrow
community, though again I'm not exactly sure on the formal processes here?
I would like to keep Julia as an official implementation, but I'm also
mostly carrying the maintainership alone at the moment and want to be
realistic with the future of the project.

I'm open to discussion and ideas on the best way forward.

-Jacob

On Tue, Mar 30, 2021 at 2:03 PM Wes McKinney  wrote:

> hi folks,
>
> I was very surprised today t

Status of Arrow Julia implementation?

[Wes McKinney]

hi folks,

I was very surprised today to learn that the Julia Arrow
implementation has continued operating more or less like an
independent open source project since the code donation last November:

https://github.com/JuliaData/Arrow.jl/commits/main

There may have been a misunderstanding about what was expected to
occur after the code donation, but it's problematic for a bunch of
reasons (IP lineage / governance / community development) to have work
happening on the implementation "outside the community".

In any case, what is done is done, so the Arrow PMC's position on this
would be roughly to regard the work as a hard fork of what's in Apache
Arrow, which given its development activity is more or less inactive
[1]. (I had actually thought the project was simply inactive after the
code donation)

The critical question now is, is there interest from Julia developers
in working "in the community", which is to say:

* Having development discussions on ASF channels (mailing list,
GitHub, JIRA), planning and communicating in the open
* Doing all development in ASF GitHub repositories

The answer to the question may be "no" (which is okay), but if that's
the case, I don't think we should be giving the impression that we
have an official Julia implementation that is developed and maintained
by the community (and so my argument would be unfortunately to drop
the donated code from the project).

If the answer is "yes", there needs to be a hard commitment to move
development to Apache channels and not look back. We would also need
to figure out what to do to document and synchronize the new IP that's
been created since the code donation.

Thanks,
Wes

[1]: https://github.com/apache/arrow/commits/master/julia/Arrow