Hi Brendan, Can you draw the result DAG for the DataFrame pseudo-code in the document? I'd assume there will be a file input operator, a CSV parser operator, a filter operator (for price != null and datetime > 1992/1/24), an enricher (for normalized price and price max 30 days), a custom non-DataFrame operator, and an output operator to H2O and an output operator to csv, with both of the operators connecting to the output port of the custom non-DataFrame operator?
Thanks, David On Tue, Mar 22, 2016 at 4:49 PM, Herger, Brendan < [email protected]> wrote: > Hey all, > > I and the Data Innovation Lab team at Capital One (including Ilya Ganelin, > Brennon York, Dongming Liang and Rikin Shah) would like to propose the > DataFrame for Apache Apex. > > We feel that a DataFrame object would solve Apex’s need for a standardized > internal data format (schema aware object containing data), and a high > level API for common data operations (integrated with the data store). > > This standardized data storage and data operations allow for rapid > development and iteration on data pipelines, by allowing users to leverage > common code components. It also allows for more standardized workflows, > built on common components with regularized syntax. > > For further details, please see the design document located at: > > https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing > > Thanks, > Brendan Herger > Data Scientist, Member of Technical Staff > Data Innovation Lab @ Capital One > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
