Thanks a lot Wes, I am not sure how to proceed from here:
1. how do we generate the html from the xml? I.e. https://incubator.apache.org/ip-clearance/arrow-rust-ballista.html 2. how do I trigger the the process to start? can I just email the incubator with the proposal? Best, Jorge On Mon, Jul 5, 2021 at 10:38 AM Wes McKinney <wesmck...@gmail.com> wrote: > Great, thanks for the update and pushing this forward. Let us know if > you need help with anything. > > On Sun, Jul 4, 2021 at 8:26 PM Jorge Cardoso Leitão > <jorgecarlei...@gmail.com> wrote: > > > > Hi, > > > > Wes and Neils, > > > > Thank you for your feedback and offer. I have created the two .xml > reports: > > > > > http://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/arrow-rust-experimental-arrow.xml > > > http://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/arrow-rust-experimental-parquet.xml > > > > I based them on the report for Ballista. I also requested, on the PRs > > [1,2], clarification wrt to every contributors' contributions to each. > > > > Best, > > Jorge > > > > [1] https://github.com/apache/arrow-experimental-rs-arrow2/pull/1 > > [2] https://github.com/apache/arrow-experimental-rs-parquet2/pull/1 > > > > > > > > On Mon, Jun 7, 2021 at 11:55 PM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > On Sun, Jun 6, 2021 at 1:47 AM Jorge Cardoso Leitão > > > <jorgecarlei...@gmail.com> wrote: > > > > > > > > Hi, > > > > > > > > Thanks a lot for your feedback. I agree with all the arguments put > > > forward, > > > > including Andrew's point about the large change. > > > > > > > > I tried a gradual 4 months ago, but it was really difficult and I > gave > > > up. > > > > I estimate that the work involved is half the work of writing > parquet2 > > > and > > > > arrow2 in the first place. The internal dependency on ArrayData (the > main > > > > culprit of the unsafe) on arrow-rs is so prevalent that all core > > > components > > > > need to be re-written from scratch (IPC, FFI, IO, array/transform/*, > > > > compute, SIMD). I personally do not have the motivation to do it, > though. > > > > > > > > Jed, the public API changes are small for end users. A typical > migration > > > is > > > > [1]. I agree that we can further reduce the change-set by keeping > legacy > > > > interfaces available. > > > > > > > > Andy, on my machine, the current benchmarks on query 1 yield: > > > > > > > > type, master (ms), PR [2] for arrow2+parquet2 (ms) > > > > memory (-m): 332.9, 239.6 > > > > load (the initial time in -m with --format parquet): 5286.0, 3043.0 > > > > parquet format: 1316.1, 930.7 > > > > tbl format: 5297.3, 5383.1 > > > > > > > > i.e. I am observing some improvements. Queries with joins are still > > > slower. > > > > The pruning of parquet groups and pages based on stats are not yet > > > there; I > > > > am working on them. > > > > > > > > I agree that this should go through IP clearance. I will start this > > > > process. My thinking would be to create two empty repos on apache/*, > and > > > > create 2 PRs from the main branches of each of my repos to those > repos, > > > and > > > > only merge them once IP is cleared. Would that be a reasonable > process, > > > Wes? > > > > > > This sounds plenty fine to me — I'm happy to assist with the IP > > > clearance process having done it several times in the past. I don't > > > have an opinion about the names, but having experimental- in the name > > > sounds in line with the previous discussion we had about this. > > > > > > > Names: arrow-experimental-rs2 and arrow-experimental-rs-parquet2, or? > > > > > > > > Best, > > > > Jorge > > > > > > > > [1] > > > > > > > > https://github.com/apache/arrow-datafusion/pull/68/files#diff-2ec0d66fd16c73ff72a23d40186944591e040507c731228ad70b4e168e2a4660 > > > > [2] https://github.com/apache/arrow-datafusion/pull/68 > > > > > > > > > > > > On Fri, May 28, 2021 at 5:22 AM Josh Taylor <joshuatayl...@gmail.com > > > > > wrote: > > > > > > > > > I played around with it, for my use case I really like the new way > of > > > > > writing CSVs, it's much more obvious. I love the > `read_stream_metadata` > > > > > function as well. > > > > > > > > > > I'm seeing a very slight speed (~8ms) improvement on my end, but I > > > read a > > > > > bunch of files in a directory and spit out a CSV, the bottleneck > is the > > > > > parsing of lots of files, but it's pretty quick per file. > > > > > > > > > > old: > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_0 > > > 120224 > > > > > bytes took 1ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_1 > > > 123144 > > > > > bytes took 1ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_10 > > > > > 17127928 bytes took 159ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_11 > > > > > 17127144 bytes took 160ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_12 > > > > > 17130352 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_13 > > > > > 17128544 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_14 > > > > > 17128664 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_15 > > > > > 17128328 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_16 > > > > > 17129288 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_17 > > > > > 17131056 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_18 > > > > > 17130344 bytes took 158ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_19 > > > > > 17128432 bytes took 160ms > > > > > > > > > > new: > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_0 > > > 120224 > > > > > bytes took 1ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_1 > > > 123144 > > > > > bytes took 1ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_10 > > > > > 17127928 bytes took 157ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_11 > > > > > 17127144 bytes took 152ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_12 > > > > > 17130352 bytes took 154ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_13 > > > > > 17128544 bytes took 153ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_14 > > > > > 17128664 bytes took 154ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_15 > > > > > 17128328 bytes took 153ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_16 > > > > > 17129288 bytes took 152ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_17 > > > > > 17131056 bytes took 153ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_18 > > > > > 17130344 bytes took 155ms > > > > > /home/josh/staging/019c4715-3200-48fa-0000-4105000cd71e/data_0_0_19 > > > > > 17128432 bytes took 153ms > > > > > > > > > > I'm going to chunk the dirs to speed up the reads and throw it > into a > > > par > > > > > iter. > > > > > > > > > > On Fri, 28 May 2021 at 09:09, Josh Taylor <joshuatayl...@gmail.com > > > > > wrote: > > > > > > > > > > > Hi! > > > > > > > > > > > > I've been using arrow/arrow-rs for a while now, my use case is to > > > parse > > > > > > Arrow streaming files and convert them into CSV. > > > > > > > > > > > > Rust has been an absolute fantastic tool for this, the > performance is > > > > > > outstanding and I have had no issues using it for my use case. > > > > > > > > > > > > I would be happy to test out the branch and let you know what the > > > > > > performance is like, as I was going to improve the current > > > implementation > > > > > > that i have for the CSV writer, as it takes a while for bigger > > > datasets > > > > > > (multi-GB). > > > > > > > > > > > > Josh > > > > > > > > > > > > > > > > > > On Thu, 27 May 2021 at 22:49, Jed Brown <j...@jedbrown.org> > wrote: > > > > > > > > > > > >> Andy Grove <andygrov...@gmail.com> writes: > > > > > >> > > > > > > >> > Looking at this purely from the DataFusion/Ballista point of > view, > > > > > what > > > > > >> I > > > > > >> > would be interested in would be having a branch of DF that > uses > > > arrow2 > > > > > >> and > > > > > >> > once that branch has all tests passing and can run queries > with > > > > > >> performance > > > > > >> > that is at least as good as the original arrow crate, then cut > > > over. > > > > > >> > > > > > > >> > However, for developers using the arrow APIs directly, I don't > > > see an > > > > > >> easy > > > > > >> > path. We either try and gradually PR the changes in (which > seems > > > > > really > > > > > >> > hard given that there are significant changes to APIs and > internal > > > > > data > > > > > >> > structures) or we port some portion of the existing tests > over to > > > > > arrow2 > > > > > >> > and then make that the official crate once all test pass. > > > > > >> > > > > > >> How feasible would it be to make a legacy module in arrow2 that > > > would > > > > > >> enable (some large subset of) existing arrow users to try arrow2 > > > after > > > > > >> adjusting their use statements? (That is, implement the > > > public-facing > > > > > >> legacy interfaces in terms of arrow2's new, safe interface.) > This > > > would > > > > > >> make it easier to test with DataFusion/Ballista and external > users > > > of > > > > > the > > > > > >> current arrow crate, then cut over and let those packages update > > > > > >> incrementally from legacy to modern arrow2. > > > > > >> > > > > > >> I think it would be okay to tolerate some performance > degradation > > > when > > > > > >> working through these legacy interfaces,so long as there was > > > confidence > > > > > >> that modernizing the callers would recover the performance (as > tests > > > > > have > > > > > >> been showing). > > > > > >> > > > > > > > > > > > > > > >
