[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-674720876 I have refactored some of the code to use the new API (in 1.25.0), to create and parameterized innodb planner rules. Please refer to https://issues.apache.org/jira/browse/CALCITE-3923. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-674738512 @XuQianJin-Stars I have addressed the comments from Julian (discussion in JIRA), and made the latest code compatible with 1.25.0, the binary files are not a concern anymore. Are there any other works to be done for this PR? I 'd very much like to push forward the work. Many thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-636909734 @XuQianJin-Stars many thanks 😃 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-638970225 @XuQianJin-Stars I have rebased master, did some refinement and updated `innodb_adapter.md`, for how the adapter works, please refer to `innodb_adapter.md`, I think this doc explains most of the design and use cases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-641320784 @XuQianJin-Stars No hurry, take your time, thanks very much! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-651476622 @zinking Thanks for reviewing! Could you give me some testing SQL examples and maybe explain the meaning behind this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996:
URL: https://github.com/apache/calcite/pull/1996#issuecomment-651833186
In MySQL 5.6, `COMPACT` is the default row format. After MySQL 5.7 (include
8.0), `DYNAMIC` is the default row format. The two are the most popular row
formats.
For `COMPRESSED`, it is not supported yet. Users who cares about storage
size rather than CPU load might choose this format. But IMHO, most MySQL users
do not specify row format when creating table.
For `FIXED` row format, it is rarely used. Refer to
https://dev.mysql.com/doc/refman/5.7/en/create-table.html
```
ROW_FORMAT=FIXED is not supported. If ROW_FORMAT=FIXED is specified while
innodb_strict_mode is disabled, InnoDB issues a warning and assumes
ROW_FORMAT=DYNAMIC. If ROW_FORMAT=FIXED is specified while innodb_strict_mode
is enabled, which is the default, InnoDB returns an error.
```
For `REDUNDANT` row format, it is an very old format before MySQL 5.1.
For `extra`, there is no such row format. Valid row formats are {DEFAULT |
DYNAMIC | FIXED | COMPRESSED | REDUNDANT | COMPACT}
To conclude, the adapter supports `COMPACT` and `DYNAMIC` format which are
most commonly used nowadays. I can add explanations in `Limitation` section.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [calcite] neoremind commented on pull request #1996: [CALCITE-4034] Implement a MySQL InnoDB adapter
neoremind commented on pull request #1996: URL: https://github.com/apache/calcite/pull/1996#issuecomment-652330139 @XuQianJin-Stars I have addressed the comments above. For the question: *What is the production usage scenario of this MySQL InnoDB Java Reader?* Calcite's InnoDB adapter allows user to query the InnoDB data based on ibd files directly as illustrated below. `innodb-java-reader` provides some basic storage APIs, for example, `queryByPrimaryKey`, `queryAll` and `queryBySk`, etc. With this adapter, user is able to query through SQL, and the internal embedded execution engine can work pretty well under most cases. ``` SQL Query | | / \ -- -- | | v v MySQL Server +---+ +--+ | SQL/Parser/Optimizer..| |Calcite innodb adapter| +---+ +--+ +---+++ | InnoDB Storage Engine || innodb-java-reader | +---+++ -File System ++ +-+ | .ibd files | ... | |InnoDB Data files ++ +-+ ``` The by-pass querying capability can benefit the following scenarios: 1. Query by offloading mysql-server process, even without mysql-server running. Backup MySQL without disturbing mysql-server process, this is a good alternative to dump data besides mysql-dump and `mysql -e` command. 2. InnoDB data file can be distributed to other storage. Based on this adapter, users can transform as what ever they wanted, for example, transforming data to columnar format (ORC, Parquet) for OLAP engines; dumping data to heterogeneous NewSQL systems, so that Data Lake analytics can leverage this. Although real-time is hard to achieve because we only get a snapshot of the table, it is still useful in Lambda architecture. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
