GitHub user bamaer added a comment to the discussion: Using Hop to transform data and write to another database
@harrisward created a [sample repository](https://github.com/harrisward/apache-hop-sakila-data-warehouse) that loads data from the MySQL Sakila database to a small data warehouse, you'll find some great examples there. To aggregate: Group By transform. This requires sorted data. For small data sets that fit entirely in memory, you can use the Memory Group By transform. For the deleted users: either exclude those users from your input query or with a Filter Rows transform. Your questions: - A [best practice](https://hop.apache.org/manual/latest/best-practices/index.html) is to split your work in smaller manageable pipelines instead of fewer large/huge pipelines. - SQL: Table Input should get you a (very) long way - Scripting: Script, Javascript transforms. Scripting is very much possible, but try to use it as a last resort - NewUserThatDay: something like Table/File Input -> (optional) Filter to exclude deleted -> Group By ("that day" would be your key, number of rows as type, user_id as subject). GitHub link: https://github.com/apache/hop/discussions/6308#discussioncomment-15427632 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
