FANNG1 commented on code in PR #13929:
URL: https://github.com/apache/gravitino/pull/13929#discussion_r4476157076


##########
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java:
##########
@@ -287,6 +306,73 @@ public void testSelectFromEmptyTableViaSpark() {
     Assertions.assertTrue(rows.isEmpty(), "Expected no rows for a newly 
created empty table");
   }
 
+  @Test
+  public void testSparkReadsTableCreatedThroughGravitino() {
+    // Creating through Gravitino's own API is the only path that reaches
+    // LanceTableOperations#createDataset, which leaves the Lance file format 
version at
+    // lance-core's
+    // default whenever the table has no blob column - 2.2 as of lance-core 
12.0.0, where earlier
+    // releases defaulted lower. Spark's CREATE TABLE goes through `declare` 
instead, where
+    // Gravitino
+    // writes no dataset at all and the bundle writes it at its own version. 
So this is the one
+    // combination that puts a Gravitino-written dataset in front of the 
bundle's Lance, which in
+    // this matrix goes back to lance-core 2.0.0.
+    String schemaName = createSchema("spark_gravitino_created");
+    String tableName = newTableName("orders");
+
+    catalog
+        .asTableCatalog()
+        .createTable(
+            NameIdentifier.of(schemaName, tableName),
+            new Column[] {
+              Column.of("id", Types.IntegerType.get()), Column.of("score", 
Types.FloatType.get())
+            },
+            "created through the Gravitino API",
+            // Deliberately no LANCE_TABLE_DECLARED: that is the `declare` 
path, where Gravitino
+            // writes no dataset and the point of this test is lost.
+            ImmutableMap.of("format", "lance"),
+            Transforms.EMPTY_TRANSFORM,
+            null,
+            null);
+
+    // First the manifest on its own: a freshly created dataset holds no data 
files, so this only
+    // proves the bundle can open a manifest written by lance-core 12.0.0.
+    Assertions.assertTrue(
+        sql("SELECT id, score FROM %s.%s", schemaName, tableName).isEmpty(),
+        "Expected no rows in a freshly created Lance dataset");
+
+    // Then the data files. Gravitino's API never writes rows, so seed them 
here with the same
+    // lance-core the server runs - 12.0.0, at file format version 2.2 - and 
read them back
+    // through the bundle. Without this the SELECT above would be the only 
read in the test and
+    // the matrix would never decode a byte written by the new writer.
+    Table createdTable =
+        catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName, 
tableName));
+    
appendRowsWithServerLance(createdTable.properties().get(Table.PROPERTY_LOCATION));
+
+    List<Row> seededRows = sql("SELECT id, score FROM %s.%s ORDER BY id", 
schemaName, tableName);
+    Assertions.assertEquals(2, seededRows.size());
+    assertRow(seededRows.get(0), 1, 1.5f);
+    assertRow(seededRows.get(1), 2, 2.5f);
+
+    // Writes are a different matter. A bundle whose Lance predates file 
format version 2.2 has
+    // its commit rejected with
+    // "added files with version 2.0. However, the data storage version is 
2.2."
+    // Only lance-spark 0.2.0 (lance-core 2.0.0) is affected; 0.4.0 
(lance-core 4.0.0) and later
+    // write fine. This is a behaviour change from Gravitino's previous 
lance-core 6.0.0, whose
+    // default was below 2.2, and it is tracked as a separate issue rather 
than worked around here.

Review Comment:
   No issue filed, because there is no regression — measurements in my 
top-level comment. Short version: `lance-spark` 0.2.0 writes file format 
version 2.0 whatever the dataset says, so it was already refused when 
Gravitino's datasets sat at 2.1. Pinning them back to 2.1 would not restore the 
write; only pinning 2.0 would, and that costs `lance.blob` for everyone.
   
   The comment is rewritten accordingly, and the test is split: 
`testSparkReadsTableCreatedThroughGravitino` keeps the reads and runs for every 
bundle, `testSparkWritesToGravitinoCreatedTable` holds the `INSERT` and is the 
only one skipped. The old assumption skipped the whole method, so a 0.2.0 run 
reported a reason-less `<skipped/>` and hid two passing cross-version read 
assertions — they now show as a 6.3s pass. `PRE_BLOB_V2_BUNDLE_VERSIONS` is 
renamed `BUNDLES_IGNORING_DATASET_FORMAT_VERSION`, since nothing about it is 
blob related.
   



##########
docs/lance-rest-integration.md:
##########
@@ -53,6 +55,14 @@ The following table outlines the tested compatibility 
between Gravitino versions
   raising `ImportError: cannot import name
   'LanceNamespaceStorageOptionsProvider' from 'lance'`. You can use
   `lance-ray` 0.2.0 with Gravitino 1.3.0 by pinning `pylance` to 3.x or 4.x.
+- **Gravitino 2.0.0 embeds `lance-core` 12.0.0 and `lance-namespace-core` 
0.11.1**, the pair

Review Comment:
   Added to the `:::note` block. No issue link, and qualified for every release 
rather than this one, since the limit is not new:
   
   > **`lance-spark` 0.2.0 cannot write to a table created through the 
Gravitino table API**, on 2.0.0 and on every earlier release alike. It passes 
no data storage version to Lance, so it writes at its own `lance-core` 2.0.0 
default of file format version 2.0 while such a dataset is pinned at the 
server's `lance-core` default — 2.2 on 2.0.0, 2.1 on 1.3.0 — and Lance refuses 
the commit. Reads are unaffected, and so are tables a connector created itself 
through `POST /lance/v1/table/{id}/declare`, which the connector writes at its 
own version. `lance-spark` 0.4.0 and later read the dataset's version before 
writing and have no such limit.
   



##########
docs/lance-rest-integration.md:
##########
@@ -24,6 +24,7 @@ The following table outlines the tested compatibility between 
Gravitino versions
 | ------------------------------ | ------------------------------ | 
--------------------------------------------- |
 | 1.1.1 - 1.2.1                  | 0.0.10 - 0.0.15                | 0.0.6 - 
0.0.8                                 |
 | 1.3.0                          | 0.2.0, 0.4.0, 0.5.1            | 0.3.0 - 
0.4.2 (0.2.0 conditionally supported) |
+| 2.0.0                          | 0.2.0, 0.4.0, 0.5.1, 0.6.1, 0.7.1, 0.8.0 | 
0.2.0 + `lance-namespace` 0.4.5; 0.3.0 and 0.4.2 + 0.7.5; 0.5.0 + 0.11.1 |

Review Comment:
   Done — all three columns padded to the widest cell.
   



##########
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java:
##########
@@ -287,6 +306,73 @@ public void testSelectFromEmptyTableViaSpark() {
     Assertions.assertTrue(rows.isEmpty(), "Expected no rows for a newly 
created empty table");
   }
 
+  @Test
+  public void testSparkReadsTableCreatedThroughGravitino() {
+    // Creating through Gravitino's own API is the only path that reaches
+    // LanceTableOperations#createDataset, which leaves the Lance file format 
version at
+    // lance-core's
+    // default whenever the table has no blob column - 2.2 as of lance-core 
12.0.0, where earlier
+    // releases defaulted lower. Spark's CREATE TABLE goes through `declare` 
instead, where
+    // Gravitino
+    // writes no dataset at all and the bundle writes it at its own version. 
So this is the one

Review Comment:
   Fixed by moving it: the shared setup is now a 
`createTableThroughGravitinoApi` helper and this text became its javadoc, so 
the dangling lines are gone. It also drops the claim that this is the only path 
reaching `createDataset` — the REST `create` endpoint reaches it too; what is 
unique is that it is the only *Spark-visible* one, since Spark's `CREATE TABLE` 
goes through `declare`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to