[
https://issues.apache.org/jira/browse/SPARK-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008027#comment-15008027
]
Brody Messmer commented on SPARK-5918:
--------------------------------------
Where this becomes an issue is when you utilize an ODBC driver to connect to
Spark's thrift server. The ODBC driver is forced to set some sort of length
restriction which is a blanket restriction across all string/varchar columns.
If the ODBC driver describes the string/varchar column as something large, the
ODBC app will attempt to allocate memory based on that size, and easily become
a memory hog resulting in poor performance and/or OOM errors. If the ODBC
driver describes the varchar column too small, it will truncate data, which is
bad news as well.
It seems like the "string" type could continue to be used in scenarios where a
high tolerance is needed for schemas of other systems, while a fixed varchar
implementation would allow for a significantly improved experience from all
ODBC enabled applications.
> Spark Thrift server reports metadata for VARCHAR column as STRING in result
> set schema
> --------------------------------------------------------------------------------------
>
> Key: SPARK-5918
> URL: https://issues.apache.org/jira/browse/SPARK-5918
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.1, 1.2.0
> Reporter: Holman Lan
> Assignee: Cheng Lian
>
> This is reproducible using the open source JDBC driver by executing a query
> that will return a VARCHAR column then retrieving the result set metadata.
> The type name returned by the JDBC driver is VARCHAR which is expected but
> reports the column type as string[12] and precision/column length as
> 2147483647 (which is what the JDBC driver would return for STRING column)
> even though we created a VARCHAR column with max length of 1000.
> Further investigation indicates the GetResultSetMetadata Thrift client API
> call returns the incorrect metadata.
> We have confirmed this behaviour in versions 1.1.1 and 1.2.0. We have not
> yet tested this against 1.2.1 but will do so and report our findings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
