Re: [Java] Source control of generated flatbuffers code

2021-03-23 Thread Micah Kornfield 
>
> I have a concern, though. Four other languages (Java would be five) check
> in the generated flatbuffers code, and it appears (based on a quick scan of
> Git logs) that this is done manually. Is there a danger that the binary
> format could change, but some language might get forgotten, and thus be
> working with the old format?

The format changes relatively slowly and any changes at this point should
be backwards compatible.



> Or is there enough interop testing that the problem would get caught right
> away?

In most cases I would expect integration tests to catch these types of
error.

On Tue, Mar 23, 2021 at 4:26 PM bobtins  wrote:

> I'm happy to check in the generated Java source. I would also update the
> Java build info to reflect this change and document how to regenerate the
> source as needed.
>
> I have a concern, though. Four other languages (Java would be five) check
> in the generated flatbuffers code, and it appears (based on a quick scan of
> Git logs) that this is done manually. Is there a danger that the binary
> format could change, but some language might get forgotten, and thus be
> working with the old format? Or is there enough interop testing that the
> problem would get caught right away?
>
> I'm new to the project and don't know how big of an issue this is in
> practice. Thanks for any enlightenment.
>
> On 2021/03/23 07:39:16, Micah Kornfield  wrote:
> > I think checking in the java files is fine and probably better then
> relying
> > on a third party package.  We should make sure there are instructions on
> > how to regenerate them along with the PR
> >
> > On Monday, March 22, 2021, Antoine Pitrou  wrote:
> >
> > >
> > > Le 22/03/2021 à 20:17, bobtins a écrit :
> > >
> > >> TL;DR: The Java implementation doesn't have generated flatbuffers code
> > >> under source control, and the code generation depends on an
> > >> unofficially-maintained Maven artifact. Other language
> implementations do
> > >> check in the generated code; would it make sense for this to be done
> for
> > >> Java as well?
> > >>
> > >> I'm currently focusing on Java development; I started building on
> Windows
> > >> and got a failure under java/format, because I couldn't download the
> > >> flatbuffers compiler (flatc) to generate Java source.
> > >> The artifact for the flatc binary is provided "unofficially" (not by
> the
> > >> flatbuffers project), and there was no Windows version, so I had to
> jump
> > >> through hoops to build it and proceed.
> > >>
> > >
> > > While this does not answer the more general question of checking in the
> > > generated Flatbuffers code (which sounds like a good idea, but I'm not
> a
> > > Java developer), note that you could workaround this by installing the
> > > Conda-provided flatbuffers package:
> > >
> > >   $ conda install flatbuffers
> > >
> > > which should get you the `flatc` compiler, even on Windows.
> > > (see https://docs.conda.io/projects/conda/en/latest/ for installing
> conda)
> > >
> > > You may also try other package managers such as Chocolatey:
> > >
> > >   https://chocolatey.org/packages/flatc
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> >
>

Re: [Java] Source control of generated flatbuffers code

2021-03-23 Thread bobtins 
I'm happy to check in the generated Java source. I would also update the Java 
build info to reflect this change and document how to regenerate the source as 
needed.

I have a concern, though. Four other languages (Java would be five) check in 
the generated flatbuffers code, and it appears (based on a quick scan of Git 
logs) that this is done manually. Is there a danger that the binary format 
could change, but some language might get forgotten, and thus be working with 
the old format? Or is there enough interop testing that the problem would get 
caught right away?

I'm new to the project and don't know how big of an issue this is in practice. 
Thanks for any enlightenment.

On 2021/03/23 07:39:16, Micah Kornfield  wrote: 
> I think checking in the java files is fine and probably better then relying
> on a third party package.  We should make sure there are instructions on
> how to regenerate them along with the PR
> 
> On Monday, March 22, 2021, Antoine Pitrou  wrote:
> 
> >
> > Le 22/03/2021 à 20:17, bobtins a écrit :
> >
> >> TL;DR: The Java implementation doesn't have generated flatbuffers code
> >> under source control, and the code generation depends on an
> >> unofficially-maintained Maven artifact. Other language implementations do
> >> check in the generated code; would it make sense for this to be done for
> >> Java as well?
> >>
> >> I'm currently focusing on Java development; I started building on Windows
> >> and got a failure under java/format, because I couldn't download the
> >> flatbuffers compiler (flatc) to generate Java source.
> >> The artifact for the flatc binary is provided "unofficially" (not by the
> >> flatbuffers project), and there was no Windows version, so I had to jump
> >> through hoops to build it and proceed.
> >>
> >
> > While this does not answer the more general question of checking in the
> > generated Flatbuffers code (which sounds like a good idea, but I'm not a
> > Java developer), note that you could workaround this by installing the
> > Conda-provided flatbuffers package:
> >
> >   $ conda install flatbuffers
> >
> > which should get you the `flatc` compiler, even on Windows.
> > (see https://docs.conda.io/projects/conda/en/latest/ for installing conda)
> >
> > You may also try other package managers such as Chocolatey:
> >
> >   https://chocolatey.org/packages/flatc
> >
> > Regards
> >
> > Antoine.
> >
>

Re: [VOTE] Accept donation of Rust Ballista project

2021-03-23 Thread Rok Mihevc 
+1 (non-binding)

On Tue, Mar 23, 2021 at 4:40 PM Eric Burden  wrote:

> +1
>
> On Tue, Mar 23, 2021 at 7:18 AM Francois Saint-Jacques <
> fsaintjacq...@gmail.com> wrote:
>
> > +1
> >
> > On Mon, Mar 22, 2021 at 8:33 AM Andrew Lamb 
> wrote:
> > >
> > > +1
> > >
> > > On Sun, Mar 21, 2021 at 7:08 PM paddy horan 
> > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > >
> > > > 
> > > > From: Sutou Kouhei 
> > > > Sent: Sunday, March 21, 2021 4:34:43 PM
> > > > To: dev@arrow.apache.org 
> > > > Subject: Re: [VOTE] Accept donation of Rust Ballista project
> > > >
> > > > +1 (binding)
> > > >
> > > > In  g9unbudmdyg7wlornhehz99sgtkmo...@mail.gmail.com
> > >
> > > >   "[VOTE] Accept donation of Rust Ballista project" on Sun, 21 Mar
> 2021
> > > > 09:56:32 -0600,
> > > >   Andy Grove  wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > On behalf of the Ballista community, I would like to propose that
> we
> > > > donate
> > > > > Ballista to the Apache Arrow project.
> > > > >
> > > > > Ballista is a distributed scheduler based on Arrow standards
> (memory
> > > > > format, IPC, Flight) and supports distributed query execution with
> > the
> > > > > DataFusion query engine.
> > > > >
> > > > > The community has had an opportunity to discuss this [1] and there
> > do not
> > > > > seem to be objections to this.
> > > > >
> > > > > The code donation in the form of a pull request:
> > > > >
> > > > >
> > > >
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Farrow%2Fpull%2F9723data=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=MjSjUnA1L%2BV3QRRiK%2FjwoBFMAYZ61cpmwCbZ5WqyBm8%3Dreserved=0
> > > > >
> > > > > This vote is to determine if the Arrow PMC is in favor of accepting
> > this
> > > > > donation. If the vote passes, the PMC and the authors of the code
> > will
> > > > work
> > > > > together to complete the ASF IP Clearance process (
> > > > >
> > > >
> >
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fincubator.apache.org%2Fip-clearance%2Fdata=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=y6AsPjwyysggZI%2BmDeBkU%2Fu%2B8RGYbRY5PYv9D2uoKac%3Dreserved=0
> > )
> > > > and import this Rust codebase
> > > > > implementation into Apache Arrow.
> > > > >
> > > > > [ ] +1 : Accept contribution of Ballista [ ] 0 : No opinion [ ] -1
> :
> > > > Reject
> > > > > contribution because...
> > > > >
> > > > > Here is my vote: +1
> > > > >
> > > > > The vote will be open for at least 72 hours.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Andy.
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fx%2Fthread.html%2Fr09556898c9c94259c00e35c04ea051040931bbe9ce577cba60c148c8%40%253Cdev.arrow.apache.org%253Edata=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=SGfaUnM0ZcDKUoF4qa0hmCSZsQ1fhDdXuka1a5c337s%3Dreserved=0
> > > >
> >
>

Re: [VOTE] Accept donation of Rust Ballista project

2021-03-23 Thread Eric Burden 
+1

On Tue, Mar 23, 2021 at 7:18 AM Francois Saint-Jacques <
fsaintjacq...@gmail.com> wrote:

> +1
>
> On Mon, Mar 22, 2021 at 8:33 AM Andrew Lamb  wrote:
> >
> > +1
> >
> > On Sun, Mar 21, 2021 at 7:08 PM paddy horan 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > >
> > > 
> > > From: Sutou Kouhei 
> > > Sent: Sunday, March 21, 2021 4:34:43 PM
> > > To: dev@arrow.apache.org 
> > > Subject: Re: [VOTE] Accept donation of Rust Ballista project
> > >
> > > +1 (binding)
> > >
> > > In  >
> > >   "[VOTE] Accept donation of Rust Ballista project" on Sun, 21 Mar 2021
> > > 09:56:32 -0600,
> > >   Andy Grove  wrote:
> > >
> > > > Dear all,
> > > >
> > > > On behalf of the Ballista community, I would like to propose that we
> > > donate
> > > > Ballista to the Apache Arrow project.
> > > >
> > > > Ballista is a distributed scheduler based on Arrow standards (memory
> > > > format, IPC, Flight) and supports distributed query execution with
> the
> > > > DataFusion query engine.
> > > >
> > > > The community has had an opportunity to discuss this [1] and there
> do not
> > > > seem to be objections to this.
> > > >
> > > > The code donation in the form of a pull request:
> > > >
> > > >
> > >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Farrow%2Fpull%2F9723data=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=MjSjUnA1L%2BV3QRRiK%2FjwoBFMAYZ61cpmwCbZ5WqyBm8%3Dreserved=0
> > > >
> > > > This vote is to determine if the Arrow PMC is in favor of accepting
> this
> > > > donation. If the vote passes, the PMC and the authors of the code
> will
> > > work
> > > > together to complete the ASF IP Clearance process (
> > > >
> > >
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fincubator.apache.org%2Fip-clearance%2Fdata=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=y6AsPjwyysggZI%2BmDeBkU%2Fu%2B8RGYbRY5PYv9D2uoKac%3Dreserved=0
> )
> > > and import this Rust codebase
> > > > implementation into Apache Arrow.
> > > >
> > > > [ ] +1 : Accept contribution of Ballista [ ] 0 : No opinion [ ] -1 :
> > > Reject
> > > > contribution because...
> > > >
> > > > Here is my vote: +1
> > > >
> > > > The vote will be open for at least 72 hours.
> > > >
> > > > Thanks,
> > > >
> > > > Andy.
> > > >
> > > > [1]
> > > >
> > >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fx%2Fthread.html%2Fr09556898c9c94259c00e35c04ea051040931bbe9ce577cba60c148c8%40%253Cdev.arrow.apache.org%253Edata=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=SGfaUnM0ZcDKUoF4qa0hmCSZsQ1fhDdXuka1a5c337s%3Dreserved=0
> > >
>

Re: [VOTE] Accept donation of Rust Ballista project

2021-03-23 Thread Francois Saint-Jacques 
+1

On Mon, Mar 22, 2021 at 8:33 AM Andrew Lamb  wrote:
>
> +1
>
> On Sun, Mar 21, 2021 at 7:08 PM paddy horan  wrote:
>
> > +1 (non-binding)
> >
> >
> > 
> > From: Sutou Kouhei 
> > Sent: Sunday, March 21, 2021 4:34:43 PM
> > To: dev@arrow.apache.org 
> > Subject: Re: [VOTE] Accept donation of Rust Ballista project
> >
> > +1 (binding)
> >
> > In 
> >   "[VOTE] Accept donation of Rust Ballista project" on Sun, 21 Mar 2021
> > 09:56:32 -0600,
> >   Andy Grove  wrote:
> >
> > > Dear all,
> > >
> > > On behalf of the Ballista community, I would like to propose that we
> > donate
> > > Ballista to the Apache Arrow project.
> > >
> > > Ballista is a distributed scheduler based on Arrow standards (memory
> > > format, IPC, Flight) and supports distributed query execution with the
> > > DataFusion query engine.
> > >
> > > The community has had an opportunity to discuss this [1] and there do not
> > > seem to be objections to this.
> > >
> > > The code donation in the form of a pull request:
> > >
> > >
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Farrow%2Fpull%2F9723data=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=MjSjUnA1L%2BV3QRRiK%2FjwoBFMAYZ61cpmwCbZ5WqyBm8%3Dreserved=0
> > >
> > > This vote is to determine if the Arrow PMC is in favor of accepting this
> > > donation. If the vote passes, the PMC and the authors of the code will
> > work
> > > together to complete the ASF IP Clearance process (
> > >
> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fincubator.apache.org%2Fip-clearance%2Fdata=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=y6AsPjwyysggZI%2BmDeBkU%2Fu%2B8RGYbRY5PYv9D2uoKac%3Dreserved=0)
> > and import this Rust codebase
> > > implementation into Apache Arrow.
> > >
> > > [ ] +1 : Accept contribution of Ballista [ ] 0 : No opinion [ ] -1 :
> > Reject
> > > contribution because...
> > >
> > > Here is my vote: +1
> > >
> > > The vote will be open for at least 72 hours.
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > > [1]
> > >
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fx%2Fthread.html%2Fr09556898c9c94259c00e35c04ea051040931bbe9ce577cba60c148c8%40%253Cdev.arrow.apache.org%253Edata=04%7C01%7C%7C4a5c92ba10ac41a6679c08d8eca8ceaa%7C84df9e7fe9f640afb435%7C1%7C0%7C637519557060004893%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=SGfaUnM0ZcDKUoF4qa0hmCSZsQ1fhDdXuka1a5c337s%3Dreserved=0
> >

[NIGHTLY] Arrow Build Report for Job nightly-2021-03-23-0

2021-03-23 Thread Crossbow 


Arrow Build Report for Job nightly-2021-03-23-0

All tasks: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0

Failed Tasks:
- conda-linux-gcc-py37-aarch64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-drone-conda-linux-gcc-py37-aarch64
- conda-linux-gcc-py39-aarch64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-drone-conda-linux-gcc-py39-aarch64
- test-conda-python-3.7-pandas-master:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-conda-python-3.7-pandas-master
- test-conda-python-3.7-turbodbc-latest:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-conda-python-3.7-turbodbc-latest
- test-conda-python-3.7-turbodbc-master:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-conda-python-3.7-turbodbc-master
- test-conda-python-3.8-jpype:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-conda-python-3.8-jpype
- test-conda-python-3.8-pandas-nightly:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-conda-python-3.8-pandas-nightly
- test-ubuntu-16.04-cpp:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-ubuntu-16.04-cpp
- test-ubuntu-18.04-docs:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-test-ubuntu-18.04-docs
- test-ubuntu-18.04-r-sanitizer:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-test-ubuntu-18.04-r-sanitizer
- wheel-osx-high-sierra-cp36m:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-high-sierra-cp36m
- wheel-osx-high-sierra-cp37m:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-high-sierra-cp37m
- wheel-osx-high-sierra-cp38:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-high-sierra-cp38
- wheel-osx-high-sierra-cp39:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-high-sierra-cp39
- wheel-osx-mavericks-cp36m:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-mavericks-cp36m
- wheel-osx-mavericks-cp37m:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-mavericks-cp37m
- wheel-osx-mavericks-cp38:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-mavericks-cp38
- wheel-osx-mavericks-cp39:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-wheel-osx-mavericks-cp39

Pending Tasks:
- test-conda-python-3.7:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-test-conda-python-3.7
- test-r-rstudio-r-base-3.6-opensuse15:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-test-r-rstudio-r-base-3.6-opensuse15

Succeeded Tasks:
- centos-7-amd64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-centos-7-amd64
- centos-8-aarch64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-travis-centos-8-aarch64
- centos-8-amd64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-github-centos-8-amd64
- conda-clean:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-conda-clean
- conda-linux-gcc-py36-aarch64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-drone-conda-linux-gcc-py36-aarch64
- conda-linux-gcc-py36-cpu-r36:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-conda-linux-gcc-py36-cpu-r36
- conda-linux-gcc-py36-cuda:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-conda-linux-gcc-py36-cuda
- conda-linux-gcc-py37-cpu-r40:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-conda-linux-gcc-py37-cpu-r40
- conda-linux-gcc-py37-cuda:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-conda-linux-gcc-py37-cuda
- conda-linux-gcc-py38-aarch64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-drone-conda-linux-gcc-py38-aarch64
- conda-linux-gcc-py38-cpu:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-23-0-azure-conda-linux-gcc-py38-cpu
- conda-linux-gcc-py38-cuda:
  URL:

Re: Wrap Unwarp Scalars in Cython API

2021-03-23 Thread Antoine Pitrou 



Hi Vibhatha,

The APIs exist and are declared (in Cython) as:

cdef public object pyarrow_wrap_scalar(const shared_ptr[CScalar]& sp_scalar)
cdef public shared_ptr[CScalar] pyarrow_unwrap_scalar(object scalar)

However, it appears that we forgot to document them.

Regards

Antoine.



Le 23/03/2021 à 04:34, Vibhatha Abeykoon a écrit :

The use case is to pass a Scalar created in Python to a kernel written in
C++ backend which supports arrow data types.
To support this I need to unwrap the Pyarrow Scalar to a C++ arrow Scalar.

With Regards,
Vibhatha Abeykoon


On Mon, Mar 22, 2021 at 11:15 PM Benjamin Kietzman 
wrote:


I'm not sure what kind of unwrapping you are looking for, would
pyarrow.scalar and Scalar.as_py address your use case? For example,
pa.scalar(128) will wrap that integer into a Scalar

On Mon, Mar 22, 2021, 11:15 Vibhatha Abeykoon  wrote:


Hello,

Is there a way to wrap and unwrap Scalars using the Cython API?

I am following the docs:
https://arrow.apache.org/docs/python/extending.html
But I couldn't find an option. Not sure if I am following the correct

docs.


With Regards,
Vibhatha Abeykoon,
PhD Candidate | Research Assistant,
Digital Science Center,
Luddy School of Informatics Computing and Engineering,
Indiana University Bloomington,
Cell : +1-812-955-1394
Web: https://www.vibhatha.org

Re: [Java] Source control of generated flatbuffers code

2021-03-23 Thread Micah Kornfield 
I think checking in the java files is fine and probably better then relying
on a third party package.  We should make sure there are instructions on
how to regenerate them along with the PR

On Monday, March 22, 2021, Antoine Pitrou  wrote:

>
> Le 22/03/2021 à 20:17, bobtins a écrit :
>
>> TL;DR: The Java implementation doesn't have generated flatbuffers code
>> under source control, and the code generation depends on an
>> unofficially-maintained Maven artifact. Other language implementations do
>> check in the generated code; would it make sense for this to be done for
>> Java as well?
>>
>> I'm currently focusing on Java development; I started building on Windows
>> and got a failure under java/format, because I couldn't download the
>> flatbuffers compiler (flatc) to generate Java source.
>> The artifact for the flatc binary is provided "unofficially" (not by the
>> flatbuffers project), and there was no Windows version, so I had to jump
>> through hoops to build it and proceed.
>>
>
> While this does not answer the more general question of checking in the
> generated Flatbuffers code (which sounds like a good idea, but I'm not a
> Java developer), note that you could workaround this by installing the
> Conda-provided flatbuffers package:
>
>   $ conda install flatbuffers
>
> which should get you the `flatc` compiler, even on Windows.
> (see https://docs.conda.io/projects/conda/en/latest/ for installing conda)
>
> You may also try other package managers such as Chocolatey:
>
>   https://chocolatey.org/packages/flatc
>
> Regards
>
> Antoine.
>