gnehil opened a new pull request, #239:
URL: https://github.com/apache/doris-spark-connector/pull/239
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
In the past, all codes were in the same module. In order to be compatible
with different Spark versions, many features available in higher versions were
abandoned, which brought inconvenience to use.
So refactoring is imperative.
After reconstruction, it is divided into the following modules:
```bash
├── spark-doris-connector-base
├── spark-doris-connector-dist
├── spark-doris-connector-it
├── spark-doris-connector-shims
├── spark-doris-connector-spark-2
└── spark-doris-connector-spark-3
```
- **spark-doris-connector-base**
This module contains the basic implementation of interaction with Doris,
such as reading, writing and other API access.
In this module, there are several important abstractions, DorisReader,
DorisWriter and DorisCommitter
- **DorisReader**
This interface provides hashNext and next methods to determine whether
the reading is finished and to obtain the next piece of data. Currently, two
reading methods based on Thrift and Arrow Flight SQL are implemented.
- **DorisWriter**
This interface mainly provides the load and stop methods for writing a
piece of data and stopping an import task.
- **DorisCommitter**
This interface mainly provides commit and abort methods for committing
and rolling back transactions.
Considering the consistency of writing and transaction processing, the
write processor based on stream load or copy into implements both DorisWriter
and DorisCommitter in the same class.
- **spark-doris-connector-dist**
This module is used to package.
- **spark-doris-connector-it**
This module is for end-to-end testing.
- **spark-doris-connector-shims**
This module is a shim layer used to implement functions of different spark
versions.
Currently, the version split is only based on spark 3.
```bash
├── spark-doris-connector-shims_spark-3.0
├── spark-doris-connector-shims_spark-3.1
├── spark-doris-connector-shims_spark-3.2
├── spark-doris-connector-shims_spark-3.3
├── spark-doris-connector-shims_spark-3.4
├── spark-doris-connector-shims_spark-3.5
└── spark-doris-connector-shims_spark-3-base
```
For the future higher spark 4 version, you can also add modules here to
implement the function of minor version differences.
- **spark-doris-connector-spark-2**
This module is implemented based on spark 2.
- **spark-doris-connector-spark-3**
This module is implemented based on spark 3. It relies on different shim
layers for different minor versions, which provides better compatibility.
In addition, a new feature has been added: converting and reading Bitmap
type data through Arrow Flight SQL
**Example**
```scala
val df = spark.read
.format("doris")
.option("doris.fenodes", "192.168.0.1:8030")
.option("doris.read.arrow-flight-sql.port", "11110")
.option("doris.table.identifier", "test.t_bitmap_read")
.option("user", "root")
.option("password", "")
.option("doris.read.mode", "arrow")
.option("doris.read.bitmap-to-string", "true")
.load
```
**Result**
```sql
+------+----------------------+
| k1 | v1 |
+------+----------------------+
| 1 | 243 |
| 2 | 1,2,3,4,5,434543 |
| 3 | 287667876573 |
+------+----------------------+
```
Convert a bitmap into a comma-delimited string containing all the set BIT
bits. If the input is null, null will be returned.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[[email protected]](mailto:[email protected]) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
