unknowntpo commented on PR #56:
URL:
https://github.com/apache/gravitino-playground/pull/56#issuecomment-2450919920
@jerqi @danhuawang
I found a wired thing and still can't solve it:
in `init/jupyter/gravitino-spark-trino-example.ipynb`, we create table with
`spark.sql.warehouse.dir` config
```
spark = SparkSession.builder \
.appName("PySpark SQL Example") \
.config("spark.plugins",
"org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin") \
.config("spark.jars",
f"{spark_home}/jars/iceberg-spark-runtime-3.4_2.12-1.5.2.jar,{spark_home}/jars/gravitino-spark-connector-runtime-3.4_2.12-0.6.0-incubating.jar")
\
.config("spark.sql.gravitino.uri", f"http://{gravitino_host_ip}:8090";) \
.config("spark.sql.gravitino.metalake", "metalake_demo") \
.config("spark.sql.gravitino.enableIcebergSupport", "true") \
.config("spark.sql.catalog.catalog_rest",
"org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.catalog_rest.type", "rest") \
.config("spark.sql.catalog.catalog_rest.uri",
f"http://{gravitino_host_ip}:9001/iceberg/";) \
.config("spark.locality.wait.node", "0") \
.config("spark.sql.warehouse.dir",
f"hdfs://{hive_host_ip}:9000/user/hive/warehouse") \
.enableHiveSupport() \
.getOrCreate()
```
But if we describe this table, the domain name will be changed to pod name:
`hive-c6544769c-gnzsl`,
which means that if we execute this sql in next step,
```
spark.sql("INSERT OVERWRITE TABLE employees
PARTITION(department='Engineering') VALUES (1, 'John Doe', 30), (2, 'Jane
Smith', 28);")
spark.sql("INSERT OVERWRITE TABLE employees
PARTITION(department='Marketing') VALUES (3, 'Mike Brown', 32);")
spark.sql("SELECT * from employees").show()
```
we will get this error:
```
---------------------------------------------------------------------------
IllegalArgumentException Traceback (most recent call last)
Cell In[4], line 1
----> 1 spark.sql("INSERT OVERWRITE TABLE employees
PARTITION(department='Engineering') VALUES (1, 'John Doe', 30), (2, 'Jane
Smith', 28);")
2 spark.sql("INSERT OVERWRITE TABLE employees
PARTITION(department='Marketing') VALUES (3, 'Mike Brown', 32);")
3 spark.sql("SELECT * from employees").show()
File /usr/local/spark/python/pyspark/sql/session.py:1440, in
SparkSession.sql(self, sqlQuery, args, **kwargs)
1438 try:
1439 litArgs = {k: _to_java_column(lit(v)) for k, v in (args or
{}).items()}
-> 1440 return DataFrame(self._jsparkSession.sql(sqlQuery, litArgs),
self)
1441 finally:
1442 if len(kwargs) > 0:
File
/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1322, in
JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +\
1317 self.command_header +\
1318 args_command +\
1319 proto.END_COMMAND_PART
1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
1323 answer, self.gateway_client, self.target_id, self.name)
1325 for temp_arg in temp_args:
1326 if hasattr(temp_arg, "_detach"):
File /usr/local/spark/python/pyspark/errors/exceptions/captured.py:175, in
capture_sql_exception.<locals>.deco(*a, **kw)
171 converted = convert_exception(e.java_exception)
172 if not isinstance(converted, UnknownException):
173 # Hide where the exception came from that shows a non-Pythonic
174 # JVM exception message.
--> 175 raise converted from None
176 else:
177 raise
IllegalArgumentException: java.net.UnknownHostException: hive-c6544769c-gnzsl
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
