FANNG1 commented on code in PR #13929:
URL: https://github.com/apache/gravitino/pull/13929#discussion_r4476157076
##########
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java:
##########
@@ -287,6 +306,73 @@ public void testSelectFromEmptyTableViaSpark() {
Assertions.assertTrue(rows.isEmpty(), "Expected no rows for a newly
created empty table");
}
+ @Test
+ public void testSparkReadsTableCreatedThroughGravitino() {
+ // Creating through Gravitino's own API is the only path that reaches
+ // LanceTableOperations#createDataset, which leaves the Lance file format
version at
+ // lance-core's
+ // default whenever the table has no blob column - 2.2 as of lance-core
12.0.0, where earlier
+ // releases defaulted lower. Spark's CREATE TABLE goes through `declare`
instead, where
+ // Gravitino
+ // writes no dataset at all and the bundle writes it at its own version.
So this is the one
+ // combination that puts a Gravitino-written dataset in front of the
bundle's Lance, which in
+ // this matrix goes back to lance-core 2.0.0.
+ String schemaName = createSchema("spark_gravitino_created");
+ String tableName = newTableName("orders");
+
+ catalog
+ .asTableCatalog()
+ .createTable(
+ NameIdentifier.of(schemaName, tableName),
+ new Column[] {
+ Column.of("id", Types.IntegerType.get()), Column.of("score",
Types.FloatType.get())
+ },
+ "created through the Gravitino API",
+ // Deliberately no LANCE_TABLE_DECLARED: that is the `declare`
path, where Gravitino
+ // writes no dataset and the point of this test is lost.
+ ImmutableMap.of("format", "lance"),
+ Transforms.EMPTY_TRANSFORM,
+ null,
+ null);
+
+ // First the manifest on its own: a freshly created dataset holds no data
files, so this only
+ // proves the bundle can open a manifest written by lance-core 12.0.0.
+ Assertions.assertTrue(
+ sql("SELECT id, score FROM %s.%s", schemaName, tableName).isEmpty(),
+ "Expected no rows in a freshly created Lance dataset");
+
+ // Then the data files. Gravitino's API never writes rows, so seed them
here with the same
+ // lance-core the server runs - 12.0.0, at file format version 2.2 - and
read them back
+ // through the bundle. Without this the SELECT above would be the only
read in the test and
+ // the matrix would never decode a byte written by the new writer.
+ Table createdTable =
+ catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName,
tableName));
+
appendRowsWithServerLance(createdTable.properties().get(Table.PROPERTY_LOCATION));
+
+ List<Row> seededRows = sql("SELECT id, score FROM %s.%s ORDER BY id",
schemaName, tableName);
+ Assertions.assertEquals(2, seededRows.size());
+ assertRow(seededRows.get(0), 1, 1.5f);
+ assertRow(seededRows.get(1), 2, 2.5f);
+
+ // Writes are a different matter. A bundle whose Lance predates file
format version 2.2 has
+ // its commit rejected with
+ // "added files with version 2.0. However, the data storage version is
2.2."
+ // Only lance-spark 0.2.0 (lance-core 2.0.0) is affected; 0.4.0
(lance-core 4.0.0) and later
+ // write fine. This is a behaviour change from Gravitino's previous
lance-core 6.0.0, whose
+ // default was below 2.2, and it is tracked as a separate issue rather
than worked around here.
Review Comment:
No issue filed, because there is no regression — measurements in my
top-level comment. Short version: `lance-spark` 0.2.0 writes file format
version 2.0 whatever the dataset says, so it was already refused when
Gravitino's datasets sat at 2.1. Pinning them back to 2.1 would not restore the
write; only pinning 2.0 would, and that costs `lance.blob` for everyone.
The comment is rewritten accordingly, and the test is split:
`testSparkReadsTableCreatedThroughGravitino` keeps the reads and runs for every
bundle, `testSparkWritesToGravitinoCreatedTable` holds the `INSERT` and is the
only one skipped. The old assumption skipped the whole method, so a 0.2.0 run
reported a reason-less `<skipped/>` and hid two passing cross-version read
assertions — they now show as a 6.3s pass. `PRE_BLOB_V2_BUNDLE_VERSIONS` is
renamed `BUNDLES_IGNORING_DATASET_FORMAT_VERSION`, since nothing about it is
blob related.
##########
docs/lance-rest-integration.md:
##########
@@ -53,6 +55,14 @@ The following table outlines the tested compatibility
between Gravitino versions
raising `ImportError: cannot import name
'LanceNamespaceStorageOptionsProvider' from 'lance'`. You can use
`lance-ray` 0.2.0 with Gravitino 1.3.0 by pinning `pylance` to 3.x or 4.x.
+- **Gravitino 2.0.0 embeds `lance-core` 12.0.0 and `lance-namespace-core`
0.11.1**, the pair
Review Comment:
Added to the `:::note` block. No issue link, and qualified for every release
rather than this one, since the limit is not new:
> **`lance-spark` 0.2.0 cannot write to a table created through the
Gravitino table API**, on 2.0.0 and on every earlier release alike. It passes
no data storage version to Lance, so it writes at its own `lance-core` 2.0.0
default of file format version 2.0 while such a dataset is pinned at the
server's `lance-core` default — 2.2 on 2.0.0, 2.1 on 1.3.0 — and Lance refuses
the commit. Reads are unaffected, and so are tables a connector created itself
through `POST /lance/v1/table/{id}/declare`, which the connector writes at its
own version. `lance-spark` 0.4.0 and later read the dataset's version before
writing and have no such limit.
##########
docs/lance-rest-integration.md:
##########
@@ -24,6 +24,7 @@ The following table outlines the tested compatibility between
Gravitino versions
| ------------------------------ | ------------------------------ |
--------------------------------------------- |
| 1.1.1 - 1.2.1 | 0.0.10 - 0.0.15 | 0.0.6 -
0.0.8 |
| 1.3.0 | 0.2.0, 0.4.0, 0.5.1 | 0.3.0 -
0.4.2 (0.2.0 conditionally supported) |
+| 2.0.0 | 0.2.0, 0.4.0, 0.5.1, 0.6.1, 0.7.1, 0.8.0 |
0.2.0 + `lance-namespace` 0.4.5; 0.3.0 and 0.4.2 + 0.7.5; 0.5.0 + 0.11.1 |
Review Comment:
Done — all three columns padded to the widest cell.
##########
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java:
##########
@@ -287,6 +306,73 @@ public void testSelectFromEmptyTableViaSpark() {
Assertions.assertTrue(rows.isEmpty(), "Expected no rows for a newly
created empty table");
}
+ @Test
+ public void testSparkReadsTableCreatedThroughGravitino() {
+ // Creating through Gravitino's own API is the only path that reaches
+ // LanceTableOperations#createDataset, which leaves the Lance file format
version at
+ // lance-core's
+ // default whenever the table has no blob column - 2.2 as of lance-core
12.0.0, where earlier
+ // releases defaulted lower. Spark's CREATE TABLE goes through `declare`
instead, where
+ // Gravitino
+ // writes no dataset at all and the bundle writes it at its own version.
So this is the one
Review Comment:
Fixed by moving it: the shared setup is now a
`createTableThroughGravitinoApi` helper and this text became its javadoc, so
the dangling lines are gone. It also drops the claim that this is the only path
reaching `createDataset` — the REST `create` endpoint reaches it too; what is
unique is that it is the only *Spark-visible* one, since Spark's `CREATE TABLE`
goes through `declare`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]