Re: Add support for Decimal64

If the timeline is not tight, I can help with Java side implementation. IIRC, we already have have 16 byte and 32 byte 2's complement based decimal vector implementations in Java based off BigDecimal. Is this similar work for 4 and 8 byte implementations? I will have to refresh my memory of code b

Re: [ANNOUNCE] New Arrow PMC chair: Wes McKinney

Congratulations, Wes On Fri, Oct 23, 2020, 4:40 PM Neal Richardson wrote: > Congratulations, Wes! > > On Fri, Oct 23, 2020 at 4:35 PM Jacques Nadeau wrote: > > > I am pleased to announce that we have a new PMC chair and VP as per our > > newly started tradition of rotating the chair once a year

Re: Help with Java PR backlog

I can take a look as well. On Thu, Jun 11, 2020, 7:18 PM Fan Liya wrote: > I would like to help with the review. > I will spend some time on it late today. > > Best, > Liya Fan > > > On Fri, Jun 12, 2020 at 9:56 AM Wes McKinney wrote: > > > hi folks, > > > > There's a number of Java PRs that se

Re: [ANNOUNCE] New Arrow committers: Ji Liu and Liya Fan

invited to be Arrow committers and they have both > > accepted. > > > > Welcome, and thank you for your contributions! > > > -- *Best Regards,* *SIDDHARTH TEOTIA* *2008C6PS540G* *BITS PILANI- GOA CAMPUS* *+91 87911 75932*

Re: [Java] PR Reviewers

cent Java PRs (I think there are a > few dealing directly trying to separate ArrowBuf from Netty which I believe > builds off work you contributed in the past. Those might be a good place > to start). > > On Sun, Feb 2, 2020 at 9:20 PM siddharth teotia > wrote: > >> Hi A

Re: [Java] PR Reviewers

Hi All, I can help review Java PRs. Thanks Sidd On Sun, Feb 2, 2020, 8:37 PM Micah Kornfield wrote: > OK, I think I've triaged the open Java PRs. Lets see how it goes. > > On Mon, Jan 27, 2020 at 11:13 PM Micah Kornfield > wrote: > > > Somewhat related, but are there any thoughts about grow

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory and setting io.netty.tryReflectionSetAccessible to true for java builds

> > On Thu, May 2, 2019 at 8:43 PM Jacques Nadeau wrote: > > > I'm onboard with this change. > > > > On Fri, Apr 26, 2019 at 2:14 AM Siddharth Teotia > > wrote: > > > > > As part of working on this patch < > > > https://github.com/a

Re: [DISCUSS][JAVA]Support Fast/Unsafe Vector APIs for Arrow

Looks like there are 2 PRs for this work -- https://github.com/apache/arrow/pull/4186 this PR adds new getUnsafe type APIs to ArrowBuf that don't do checkIndex() before calling PlatformDependent.get(memory address). So the access will go through vector.get() -> buffer.get() -> PlatformDependent.get

Re: ARROW-3191: Status update: Making ArrowBuf work with arbitrary memory

ilures now. I hope people have looked at my previous email for the change I had made to get the jdk >= 9 builds passing. On Thu, Apr 25, 2019 at 3:13 PM Siddharth Teotia wrote: > As part of working on this patch > <https://github.com/apache/arrow/pull/4151>, I ran into a problem w

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory and setting io.netty.tryReflectionSetAccessible to true for java builds

this to the maintainers of io.netty.util.internal.ReflectionUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thanks. On Thu, Apr 18, 2019 at 3:39 PM Siddharth Teotia wro

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory

changes). As of now, the java modules build fine but I have to fix test failures. That is in progress. On Wed, Apr 17, 2019 at 6:41 AM Jacques Nadeau wrote: > Are there any other general comments here? If not, let's get this done and > merged. > > On Mon, Apr 15, 2019, 4:19 PM

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory

ncept of the reader and writer index need to be > on ArrowBuf? It seems like something that could be added as an additional > decoration/wrapper when needed instead of being part of the core structure. > > On Sat, Apr 13, 2019 at 11:26 AM Siddharth Teotia > wrote: > > > Hi A

ARROW-3191: Making ArrowBuf work with arbitrary memory

Hi All, I have put a PR with WIP changes. All the major set of changes have been done to decouple the usage of ArrowBuf and reference management. The ArrowBuf interface is much simpler and clean now. I believe there would be several folks in the community interested in these changes so please fee

Re: [VOTE] Proposed change to Arrow Flight protocol: endpoint URIs

+1 (binding) On Tue, Apr 9, 2019 at 9:53 PM Kouhei Sutou wrote: > +1 (binding) > > In > "[VOTE] Proposed change to Arrow Flight protocol: endpoint URIs" on Mon, > 8 Apr 2019 20:36:26 +0200, > Antoine Pitrou wrote: > > > > > Hello, > > > > David Li has proposed to make the following change

Re: Java allocate buffer code

tion, this code should release > (combinedSizeWith2ThePowerSize - combinedSize) = 4064 bytes. But, this is > not happening. > > let me know if this example helps. Do we have some other channel to talk? > > Thanks. > Hitesh. > > > > > > > On Thursday, March 28, 2

Re: Java allocate buffer code

ink about it? > Thanks.Hitesh. >On Wednesday, March 27, 2019, 12:23:47 PM PDT, Siddharth Teotia < > siddha...@dremio.com> wrote: > > Hi Hitesh, > > The code you referenced allocates data and validity buffers for a fixed > width vector. It first determines the appropriate

Re: Java allocate buffer code

Hi Hitesh, The code you referenced allocates data and validity buffers for a fixed width vector. It first determines the appropriate buffer size for a given value count and then allocates a compound buffer. The compound buffer is then sliced to get data and validity buffers and finally compound bu

Re: Arrow development sync call today 12pm Eastern / 17:00 UTC

Notes: Attendees: Sidd Wes Ravindra Arvind Shyam Bryan Francois Bryan: 1. Switching over to Java time from Joda time. At Dremio we need to assess the impact of these changes. Bryan will put a WIP PR soon. There has been a discussion about this on mailing list 2. The Gandiva microbenchmark test fa

[jira] [Created] (ARROW-3194) Fix setValueCount in spitAndTransfer for variable width vectors

Siddharth Teotia created ARROW-3194: --- Summary: Fix setValueCount in spitAndTransfer for variable width vectors Key: ARROW-3194 URL: https://issues.apache.org/jira/browse/ARROW-3194 Project: Apache

Arrow Sync

I have a clash this morning so won't be able to join the call.

Re: [VOTE] Accept donation of Gandiva to Apache Arrow

+1 On Thu, Aug 16, 2018 at 9:57 AM, Julian Hyde wrote: > +1 > On Thu, Aug 16, 2018 at 8:56 AM Wes McKinney wrote: > > > > Dear all, > > > > The developers of Gandiva, an LLVM-based vectorized expression > > evaluation engine for Arrow columnar memory, are proposing to donate > > the project to

Re: [JAVA] SIMD vectorized fill of ArrowBuf from Java primitive type array?

nts to copying of data buffer. Further down in the file, you will see BitCopier to copy validity bits. On Mon, Jul 23, 2018 at 5:19 PM, Siddharth Teotia wrote: > Eric, you can take a look here > <https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/sabo

Re: [JAVA] SIMD vectorized fill of ArrowBuf from Java primitive type array?

Eric, you can take a look here how we try to optimize the copy (validity and data) in/out of vectors. We try to start with word-wise copy (64 column values and thus 64 valid

Re: [DISCUSS] Developing a standard memory layout for in-memory records / "row-oriented" data

I am wondering if this can be considered as an opportunity to implement support in Arrow for building high performance in-memory row stores for low latency and high throughput key based queries. In other words, we can design the in-memory record format keeping efficient RDMA reads as one of the goa

Re: JDBC Adapter PR - 1759

:laur...@dremio.com] > > Sent: Wednesday, May 30, 2018 5:38 AM > > To: dev@arrow.apache.org > > Subject: Re: JDBC Adapter PR - 1759 > > > > Same here. > > > > On Tue, May 29, 2018 at 9:59 AM, Siddharth Teotia > > wrote: > > > >> Hi Atu

Re: Arrow sync at 12:00 US/Eastern today

I have a conflict so won't be able to join. On Wed, Jun 13, 2018, 5:46 AM Wes McKinney wrote: > As usual we will be meeting at https://meet.google.com/vtm-teks-phx >

Re: JDBC Adapter PR - 1759

Hi Atul, I will take a look today. Thanks, Sidd On Tue, May 29, 2018 at 2:45 AM, Atul Dambalkar wrote: > Hi Sid, Laurent, Uwe, > > Any idea when can someone take a look at the PR https://github.com/apache/ > arrow/pull/1759/. > > Laurent had given bunch of comments earlier and now we have take

Re: Is there list writer in Java?

Hi Teddy, Yes UnionListWriter currently doesn't support writing decimals into list vector. Basically we are missing APIs like UnionListWriter.decimal() which will return a DecimalWriter(we already have this) and the latter can be used to write decimals in list. I'd suggest you to go ahead and file

Re: Correct way to set NULL values in VarCharVector (Java API)?

Another option is to use the set() API that allows you to indicate whether the value is NULL or not using an isSet parameter (0 for NULL, 1 otherwise). This is similar to holder based APIs where you need to indicate in holder.isSet whether value is NULL or not. https://github.com/apache/arrow/blob

Re: What do people think about a one day get together?

+1. I would love to attend. On Tue, Apr 3, 2018 at 4:18 PM, Kevin Moore wrote: > Sounds great. Quilt Data may be able to sponsor some of the refreshment > costs. > > > Kevin Moore > CEO, Quilt Data, Inc. > ke...@quiltdata.io | LinkedIn > (415) 497-

Re: Arrow sync tomorrow: 12:00 US/Eastern, please review packaging thread

Got it: https://meet.google.com/vtm-teks-phx On Wed, Apr 4, 2018 at 8:48 AM, Siddharth Teotia wrote: > Can someone please send me the link to gcal? For some reason it has > vanished from my calendar. > > On Wed, Apr 4, 2018 at 7:49 AM, Li Jin wrote: > >> Sorry I have a co

Re: Arrow sync tomorrow: 12:00 US/Eastern, please review packaging thread

Can someone please send me the link to gcal? For some reason it has vanished from my calendar. On Wed, Apr 4, 2018 at 7:49 AM, Li Jin wrote: > Sorry I have a conflict today so won't be able to join. > > Li > > On Wed, Apr 4, 2018 at 1:53 AM, Bhaskar Mookerji > wrote: > > > Can someone attending

Re: Trouble Updating Java artifacts

u need to first setup up Maven to know your Apache credentials: > http://www.apache.org/dev/publishing-maven-artifacts.html#dev-env > > > > I have taken care of the upload, please verify that the artifacts are > all up. > > > > Uwe > > > > On Wed, Mar 21,

Trouble Updating Java artifacts

Hi All, I think the steps mentioned in RM doc for updating java artifacts are incomplete. I am getting the following error: Failed to deploy artifacts: Could not transfer artifact org.apache.arrow:arrow-java-root:pom:0.9.0 from/to apache.releases.https ( https://repository.apache.org/service/loca

Re: Arrow Sync Call Started

1 hour > from now. In case there's some confusion, we may need to reschedule > and make sure we're all agreed on what time zone we're pinned to > > On Wed, Mar 21, 2018 at 12:02 PM, Siddharth Teotia > wrote: > > https://www.google.com/url?q=https%3A%2F%2Fmeet.google.com% > 2Fvtm-teks-phx >

Arrow Sync Call Started

https://www.google.com/url?q=https%3A%2F%2Fmeet.google.com%2Fvtm-teks-phx

[jira] [Created] (ARROW-2329) [Website]: 0.9.0 release update

Siddharth Teotia created ARROW-2329: --- Summary: [Website]: 0.9.0 release update Key: ARROW-2329 URL: https://issues.apache.org/jira/browse/ARROW-2329 Project: Apache Arrow Issue Type: Task

Re: [RESULT] [VOTE] Release Apache Arrow 0.9.0 (RC2)

launch. > > > > > > On Mon, Mar 19, 2018 at 12:51 PM Li Jin wrote: > > > > > >> +1 > > >> > > >> Ran Java Tests on Mac OS. > > >> > > >> On Mon, Mar 19, 2018 at 11:20 AM, Siddharth Teotia < > > siddha...@d

Re: [VOTE] Release Apache Arrow 0.9.0 (RC2)

+1 Verified RC on Mac OS. Things look good. Thanks Sidd On Mar 19, 2018 12:31 AM, "Kouhei Sutou" wrote: +1 (binding), tested on Debian GNU/Linux sid with * GCC 7.3.0 * OpenJDK 9.0.4 * Ruby 2.5.0p0 * NodeJS 8.9.3 -- kou In "Re: [VOTE] Release Apache Arrow 0.9.0 (RC2)" on Mon, 19

[jira] [Created] (ARROW-2294) Fix splitAndTransfer for variable width vector

Siddharth Teotia created ARROW-2294: --- Summary: Fix splitAndTransfer for variable width vector Key: ARROW-2294 URL: https://issues.apache.org/jira/browse/ARROW-2294 Project: Apache Arrow

Re: Working towards getting 0.9.0 release candidate up next week

is on Monday. > > - Wes > > On Thu, Mar 8, 2018 at 2:25 PM, Siddharth Teotia > wrote: > > All, > > > > I plan to get RC out over the weekend or early Monday. Is that fine with > > everybody? > > > > We have 6 items in progress -- > > https://

Re: Working towards getting 0.9.0 release candidate up next week

can make an RC by Friday but if not we can defer to Monday/Tuesday > the following week (3/12 or 13). I will trim as much as possible out > of the current backlog to get things down to the essential > > - Wes > > On Sun, Feb 25, 2018 at 11:58 AM, Siddharth Teotia > wrote: &g

Arrow sync call today - March 7

I will be at Strata conference today and won't be able to join the call. Thanks Sidd

Re: Working towards getting 0.9.0 release candidate up next week

Sounds good. Thanks Sidd On Feb 24, 2018 6:24 PM, "Wes McKinney" wrote: Hi Sidd, I think we have too many bugs to make an RC this coming week. I suggest we defer to the following week. Thanks Wes On Feb 24, 2018 7:09 PM, "Siddharth Teotia" wrote: Hi All, We current

Working towards getting 0.9.0 release candidate up next week

Hi All, We currently have 10 issues in progress and PRs are available for 8 of them. In interest of getting a release candidate next week, I would request people to review PRs as soon as they can to help make progress and close out as many JIRAs as we can. There are 32 issues in TODO list and 25

Re: Allocating additional memory to the Java Vector objects

nking of doing explicitly in the code). Although it uses "reAlloc" > which allocates double the current size, for me it works - as I plan to > start with moderate initial capacity for the vectors. > > -Atul > > -----Original Message- > From: Siddharth Teotia

Re: Allocating additional memory to the Java Vector objects

Hi Atul, Currently there is no way for doing this. The only exposed method of expanding the vector buffer is reAlloc() and it allocates a new buffer of double the original capacity and copies the old contents into the new buffer. Thanks, Sidd On Fri, Feb 23, 2018 at 12:06 PM, Atul Dambalkar wro

[jira] [Created] (ARROW-2199) Follow up fixes for ARROW-2019. Ensure density driven capacity is never less than 1 and propagate density throughout the vector tree

Siddharth Teotia created ARROW-2199: --- Summary: Follow up fixes for ARROW-2019. Ensure density driven capacity is never less than 1 and propagate density throughout the vector tree Key: ARROW-2199 URL: https

[jira] [Created] (ARROW-2019) Control the memory allocated for inner vector in LIST

Siddharth Teotia created ARROW-2019: --- Summary: Control the memory allocated for inner vector in LIST Key: ARROW-2019 URL: https://issues.apache.org/jira/browse/ARROW-2019 Project: Apache Arrow

Re: Arrow-Parquet converters in Java

Hi Li, We do have support for Parquet <-> Arrow reader/writer in Dremio OSS. Please take a look here: https://github.com/dremio/dremio-oss/tree/master/sabot/kernel/src/main/java/com/dremio/exec/store/parquet https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/ex

[jira] [Created] (ARROW-2001) Add getInitReservation() to BufferAllocator interface similar to getLimit(), getHeadRoom() APIs

Siddharth Teotia created ARROW-2001: --- Summary: Add getInitReservation() to BufferAllocator interface similar to getLimit(), getHeadRoom() APIs Key: ARROW-2001 URL: https://issues.apache.org/jira/browse/ARROW

Re: [DRAFT] Apache Arrow board report

very healthy with a growing developer and user community. > > > > > > > > ## PMC changes: > > > > > > > > - Currently 20 PMC members. > > > > - No new PMC members added in the last 3 months > > > > - Last PMC addition was Kouhe

[jira] [Created] (ARROW-1946) Add APIs to decimal vector for writing big endian data

Siddharth Teotia created ARROW-1946: --- Summary: Add APIs to decimal vector for writing big endian data Key: ARROW-1946 URL: https://issues.apache.org/jira/browse/ARROW-1946 Project: Apache Arrow

[jira] [Created] (ARROW-1943) Handle setInitialCapacity() for deeply nested lists of lists

Siddharth Teotia created ARROW-1943: --- Summary: Handle setInitialCapacity() for deeply nested lists of lists Key: ARROW-1943 URL: https://issues.apache.org/jira/browse/ARROW-1943 Project: Apache

[jira] [Created] (ARROW-1939) Correct links in release 0.8 blog post

Siddharth Teotia created ARROW-1939: --- Summary: Correct links in release 0.8 blog post Key: ARROW-1939 URL: https://issues.apache.org/jira/browse/ARROW-1939 Project: Apache Arrow Issue Type

Re: arrow read/write examples in Java

>From Arrow 0.8, the second step "Grab the corresponding mutator and accessor objects by calls to getMutator(), getAccessor()" is not needed. In fact, it is not even there. On Tue, Dec 19, 2017 at 10:01 AM, Siddharth Teotia wrote: > Hi Animesh, > > Firstly I would like to

Re: arrow read/write examples in Java

Hi Animesh, Firstly I would like to suggest switching over to Arrow 0.8 release asap since you are writing JAVA programs and the API usage has changed drastically. The new APIs are much simpler with good javadocs and detailed internal comments. If you are writing stop-gap implementation then it i

Re: Confirming Release Owners

Conda is done -- I updated arrow-cpp-feedstock and Uwe took care of parquet-cpp and pyarrow. On Sun, Dec 17, 2017 at 4:14 PM, Jacques Nadeau wrote: > Wes: Post to Dist, Upload Maven artifacts, Send Announce > Jacques: Update website/docs > Sidd (with Help from Uwe): Update conda > Bryan: Update

[jira] [Created] (ARROW-1922) Blog post on recent improvements/changes in JAVA Vectors

Siddharth Teotia created ARROW-1922: --- Summary: Blog post on recent improvements/changes in JAVA Vectors Key: ARROW-1922 URL: https://issues.apache.org/jira/browse/ARROW-1922 Project: Apache Arrow

[jira] [Created] (ARROW-1876) Transfer validity vector buffer data word at a time (currently we do byte at a time)

Siddharth Teotia created ARROW-1876: --- Summary: Transfer validity vector buffer data word at a time (currently we do byte at a time) Key: ARROW-1876 URL: https://issues.apache.org/jira/browse/ARROW-1876

General Suggestions and Request regarding recent JAVA changes.

Folks, Over the last couple of weeks, we have had several changes (both merged and in the pipeline) as follow-up work after ARROW-1463 was merged. I feel that refactoring suggestions are being proposed on-the-fly while the developer is already in progress with the code changes and it's too late t

[jira] [Created] (ARROW-1826) [JAVA] Avoid branching at cell level (copyFrom)

Siddharth Teotia created ARROW-1826: --- Summary: [JAVA] Avoid branching at cell level (copyFrom) Key: ARROW-1826 URL: https://issues.apache.org/jira/browse/ARROW-1826 Project: Apache Arrow

[jira] [Created] (ARROW-1813) Enforce checkstyle failure in JAVA build and fix all checkstyle

Siddharth Teotia created ARROW-1813: --- Summary: Enforce checkstyle failure in JAVA build and fix all checkstyle Key: ARROW-1813 URL: https://issues.apache.org/jira/browse/ARROW-1813 Project: Apache

Re: Update on ARROW-1463 - Request for merging java-vector-refactor into master

e further Java changes easier. There are still left items > > on https://issues.apache.org/jira/browse/ARROW-1463, we can work on > those > > after merging to master. > > > > > > On Mon, Nov 13, 2017 at 5:31 PM, Siddharth Teotia > > wrote: > > > >

Update on ARROW-1463 - Request for merging java-vector-refactor into master

Functional and Performance testing has been completed with Dremio. We have seen overall improvement in TPCH numbers. We had about 8000 regression tests and 12000 unit tests. I would like to start the process of merging java-vector-refactor branch into master. The branch has 2 patches with 95% of c

[jira] [Created] (ARROW-1807) [JAVA] Reduce Heap Usage (Phase 3): consolidate buffers

Siddharth Teotia created ARROW-1807: --- Summary: [JAVA] Reduce Heap Usage (Phase 3): consolidate buffers Key: ARROW-1807 URL: https://issues.apache.org/jira/browse/ARROW-1807 Project: Apache Arrow

Re: [DISCUSS] readerIndex/writerIndex in Java vector refactor

code and come cross > readerIndex/writerIndex on ArrowBuf. This issue has been brought up by > Siddharth > Teotia earlier but I might have missed the discussion so what to clarify. > > My understanding is that the current implementation in java refactor branch > ignore reader/writ

Re: Arrow sync today

There was no meeting today. On Wed, Nov 1, 2017 at 10:14 AM, Li Jin wrote: > I wasn't able to join the chat room so not sure what's going on. Did we > have the meeting? > On Wed, Nov 1, 2017 at 12:55 PM Bryan Cutler wrote: > > > Sorry, I won't be able to make today's call either. I have been w

Update on ARROW-1463 (Java Vector Refactoring)

Just FYI: I spent the last week making the necessary code changes in Dremio to move to new java vector code. This week I am running tests and debugging failures (plenty at the moment). Trying to complete the work asap. Thanks, Siddharth

Re: Arrow sync today

I have joined the meeting here -- https://meet.google.com/vtm-teks-phx I don't see anybody. Can someone please send out the correct link? On Wed, Nov 1, 2017 at 8:43 AM, Wes McKinney wrote: > I am not able to attend today’s Arrow sync. Others are free to meet and > relay notes to the mailing lis

Re: Update on ARROW-1463, related subtasks and plan for testing and merging

t to > pass? > >> What do you think is the easiest? > >> > >> Reader/Writer: > >> Yes we can address this later. > >> > >> Apologies if I seem to add more work for merging https://github.com/ > >> apache/arrow/pull/1164, that's not my int

Re: Update on ARROW-1463, related subtasks and plan for testing and merging

st? > > Reader/Writer: > Yes we can address this later. > > Apologies if I seem to add more work for merging https://github.com/ > apache/arrow/pull/1164, that's not my intention, I think the PR looks > good - > just want to bring up some major design decisions so people can commen

Re: Update on ARROW-1463, related subtasks and plan for testing and merging

w vectors (zero-copy, as the underlying buffers should be > transferred directly) > > Jacques also has a few comments, I don't know if they have been addressed. > > For other comments, I think we can add TODO and do it later. I think we can > merge this PR if we address (1) (2)

Re: Update on ARROW-1463, related subtasks and plan for testing and merging

g has been done. On the testing front, as far as the correctness is concerned, I have two failures in TestArrowFile and TestValueVector. I have added some more tests too. On Thu, Oct 12, 2017 at 2:18 PM, Siddharth Teotia wrote: > Yes, that is the intention. Good that we all are on the

Re: Update on ARROW-1463, related subtasks and plan for testing and merging

ecause it's really just a > migration > > path to help Dremio to migrate to the new vectors. I don't think other > > users, i.e., Spark will use the Legacy vector class. Bryan and I will > just > > migrate Spark to new vectors directly because Spark's

Re: Update on ARROW-1463, related subtasks and plan for testing and merging

sts: > > > > Bryan and I can help with integration test with Spark. I think the target > > timeline for Spark 2.3 release is some time in mid Nov (Bryan please > > correct me if I am wrong). > > > > I will take a look at the PR today. > > > > > > > > >

Update on ARROW-1463, related subtasks and plan for testing and merging

Hi All, I wanted to update everyone on state of this mini-project: - Requirements document and initial design proposal were sent out to the community for review and we have received some good feedback. All required docs are attached with corresponding JIRAs. - The initial prototype

[jira] [Created] (ARROW-1655) Add Scale and Precision to ValueVectorTypes.tdd for Decimals

Siddharth Teotia created ARROW-1655: --- Summary: Add Scale and Precision to ValueVectorTypes.tdd for Decimals Key: ARROW-1655 URL: https://issues.apache.org/jira/browse/ARROW-1655 Project: Apache

Re: Question regarding scope of Arrow

I think it's a good idea to have SIMD support inbuilt in Arrow libraries. Simple analytic operations like SUM, MIN, MAX, COUNT, AVG, FILTER (especially for fixed width values and dictionary encoded columns) can be made substantially faster by providing APIs that internally use SIMD (probably throug

Re: Arrow sync call tomorrow 4 October @ 16:00 UTC

I am out for doctor's appointment. I may have to miss it this time or I might join a bit late. On Oct 4, 2017 7:16 AM, "Wes McKinney" wrote: Heimir has offered to create a Hangout that should accommodate up to 25 participants: https://plus.google.com/hangouts/_/mojotech.com/array-sync?hceid= aG

Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.

Anyway, I will create a PR in some time for the WIP prototype. I think once people eyeball the code there, we may have a consensus. On Tue, Oct 3, 2017 at 3:53 PM, Siddharth Teotia wrote: > Li, > > This is exactly what I was referring to in my previous email. I think if >

Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.

Sorry I couldn't provide too much useful feedback without digging into the > template, but this is any general feeling about these templates - too many > "if" to types like "Interval" "Decimal" "Timestamp" > > > > On Tue, Oct 3, 2017 at 3:59

Re: ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.

create a PR from this prototype code once it in reasonable shape for review but was wondering if people have any opinion. Thanks, Sidd On Tue, Oct 3, 2017 at 3:16 AM, Siddharth Teotia wrote: > Hi All, > > You should have received an invitation to edit the following document. > Please

Re: [ANNOUNCE] New Arrow committers: Phillip Cloud and Bryan Cutler

Congrats Philip and Bryan :) On Tue, Oct 3, 2017 at 11:57 AM, Holden Karau wrote: > Congrats to the both of you :) Really excited to see the Areow project > continue to grow :) > > On Tue, Oct 3, 2017 at 10:24 AM Julian Hyde wrote: > > > Congratulations and welcome, Philip and Bryan! > > > > >

ARROW-1463: SubTask ARROW-1472: Design updated Value Vector hierarchy.

Hi All, You should have received an invitation to edit the following document. Please feel free to add comments or additional content. https://docs.google.com/document/d/1rl0PK5OnbQAnFUrhd4bQPtP0u7930sBKKaiyggOY7t4/edit Let me know if the document is not editable. Thanks, Siddharth

Re: ARROW-1463: Subtask ARROW-1471 Requirements for Value Vector Updates

PM, Li Jin wrote: > Siddharth, > > The requirement doc looks good to me. Thanks for putting this together. > > Li > > On Sun, Oct 1, 2017 at 1:20 PM, Siddharth Teotia > wrote: > > > Hi All, > > > > I am assuming that there are no more requirements

Re: ARROW-1463: Subtask ARROW-1471 Requirements for Value Vector Updates

te I'll try adding it again > > https://docs.google.com/document/d/1ysZ76zritBDwkeQz3C6- > vhQwD32jEXd1kUF4T936G1U/edit > > On Thu, Sep 21, 2017 at 2:18 PM, Siddharth Teotia > wrote: > > > Hi All, > > > > You should have received an invitation to edit the foll

[jira] [Created] (ARROW-1621) Reduce Heap Usage per Vector

Siddharth Teotia created ARROW-1621: --- Summary: Reduce Heap Usage per Vector Key: ARROW-1621 URL: https://issues.apache.org/jira/browse/ARROW-1621 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-1618) See if the heap usage in vectors can be reduced.

Siddharth Teotia created ARROW-1618: --- Summary: See if the heap usage in vectors can be reduced. Key: ARROW-1618 URL: https://issues.apache.org/jira/browse/ARROW-1618 Project: Apache Arrow

ARROW-1463: Subtask ARROW-1471 Requirements for Value Vector Updates

Hi All, You should have received an invitation to edit the following document. Please feel free to add comments or additional content. https://docs.google.com/document/d/1ysZ76zritBDwkeQz3C6- vhQwD32jEXd1kUF4T936G1U/edit?usp=sharing Thanks, Siddharth

[jira] [Created] (ARROW-1553) Implement setInitialCapacity for MapWriter and pass on this capacity during lazy creation of child vectors

Siddharth Teotia created ARROW-1553: --- Summary: Implement setInitialCapacity for MapWriter and pass on this capacity during lazy creation of child vectors Key: ARROW-1553 URL: https://issues.apache.org/jira

[jira] [Created] (ARROW-1547) Fix 8x memory over-allocation in BitVector

Siddharth Teotia created ARROW-1547: --- Summary: Fix 8x memory over-allocation in BitVector Key: ARROW-1547 URL: https://issues.apache.org/jira/browse/ARROW-1547 Project: Apache Arrow Issue

[jira] [Created] (ARROW-1533) As part of buffer transfer (transferTo function), we should transfer the state for realloc

Siddharth Teotia created ARROW-1533: --- Summary: As part of buffer transfer (transferTo function), we should transfer the state for realloc Key: ARROW-1533 URL: https://issues.apache.org/jira/browse/ARROW-1533

Re: Travis CI problems

ted sometime in the last 3 days. > > > > If Laurent or someone with Java background could investigate the cause > > and either fix or disable entry in the Travis build matrix, that would > > be great. > > > > Thanks > > > > On Thu, Sep 7, 2017 at 7:18 PM,

Travis CI problems

Is anyone else seeing the following failures in Travis CI build? I am seeing these problems for PR https://github.com/apache/arrow/pull/1052 I looked at the raw log and nothing seems to indicate problems w.r.t code changes. travis_time:end:082db3a8:start=1504813701624387520,finish=150481370162812

ARROW-1463 subtask assignments - https://issues.apache.org/jira/browse/ARROW-1463

Hi All, I am wondering if anyone is interested in working on sub-tasks for ARROW-1463. Please feel free to grab the child JIRAs. Thanks, Siddharth

[jira] [Created] (ARROW-1478) clear should release the buffer only if the buffer is not NULL

SIDDHARTH TEOTIA created ARROW-1478: --- Summary: clear should release the buffer only if the buffer is not NULL Key: ARROW-1478 URL: https://issues.apache.org/jira/browse/ARROW-1478 Project: Apache

[jira] [Created] (ARROW-1467) Fix reset() and allocateNew() in Nullable Value Vectors template

SIDDHARTH TEOTIA created ARROW-1467: --- Summary: Fix reset() and allocateNew() in Nullable Value Vectors template Key: ARROW-1467 URL: https://issues.apache.org/jira/browse/ARROW-1467 Project: Apache

[jira] [Created] (ARROW-1444) BitVector.splitAndTransfer copies last byte incorrectly

SIDDHARTH TEOTIA created ARROW-1444: --- Summary: BitVector.splitAndTransfer copies last byte incorrectly Key: ARROW-1444 URL: https://issues.apache.org/jira/browse/ARROW-1444 Project: Apache Arrow

[jira] [Created] (ARROW-1373) Implement getBuffer() methods at the ValueVector interface

SIDDHARTH TEOTIA created ARROW-1373: --- Summary: Implement getBuffer() methods at the ValueVector interface Key: ARROW-1373 URL: https://issues.apache.org/jira/browse/ARROW-1373 Project: Apache Arrow

  1   2   >