David, You bring up a great point. This hits on short term and long term goals.
Version 1.0 I’ve purposely not included a DAG, because the DAG Planner trivially translates each operation (basically each line in the example) in to one operator. Additionally, every operator has one input port for each DataFrame involved and one output port for each DataFrame involved. As a result, almost every operator has one input and one output (except for joining two data frames, and similar cases). Version 2.0 Once we have functioning operations, we can also optimize our DAG to reduce redundant work and increase efficiency. In a second iteration, we’d like to evaluate creating a DAG planner which can act as a query optimizer. This would allow users to write the same code, but then the DAG Planner would seek out the most optimized DAG (similar to Flink and SystemML). Thanks, Brendan Herger Data Scientist, Member of Technical Staff Data Innovation Lab @ Capital One 415.582.7457 (cell) On 3/23/16, 4:12 PM, "David Yan" <[email protected]> wrote: >Hi Brendan, > >Can you draw the result DAG for the DataFrame pseudo-code in the document? >I'd assume there will be a file input operator, a CSV parser operator, a >filter operator (for price != null and datetime > 1992/1/24), an enricher >(for normalized price and price max 30 days), a custom non-DataFrame >operator, and an output operator to H2O and an output operator to csv, with >both of the operators connecting to the output port of the custom >non-DataFrame operator? > >Thanks, > >David > >On Tue, Mar 22, 2016 at 4:49 PM, Herger, Brendan < >[email protected]> wrote: > >> Hey all, >> >> I and the Data Innovation Lab team at Capital One (including Ilya Ganelin, >> Brennon York, Dongming Liang and Rikin Shah) would like to propose the >> DataFrame for Apache Apex. >> >> We feel that a DataFrame object would solve Apex’s need for a standardized >> internal data format (schema aware object containing data), and a high >> level API for common data operations (integrated with the data store). >> >> This standardized data storage and data operations allow for rapid >> development and iteration on data pipelines, by allowing users to leverage >> common code components. It also allows for more standardized workflows, >> built on common components with regularized syntax. >> >> For further details, please see the design document located at: >> >> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing >> >> Thanks, >> Brendan Herger >> Data Scientist, Member of Technical Staff >> Data Innovation Lab @ Capital One >> >> ________________________________________________________ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
