[
https://issues.apache.org/jira/browse/SPARK-32632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro resolved SPARK-32632.
--------------------------------------
Resolution: Invalid
> Bad partitioning in spark jdbc method with parameter lowerBound and upperBound
> ------------------------------------------------------------------------------
>
> Key: SPARK-32632
> URL: https://issues.apache.org/jira/browse/SPARK-32632
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Liu Dinghua
> Priority: Major
>
> When I use the jdbc methed
> {code:java}
> def jdbc( url: String, table: String, columnName: String, lowerBound: Long,
> upperBound: Long, numPartitions: Int, connectionProperties: Properties)
> {code}
>
> I am confused by the partitions generated by this method, for rows of the
> first partition aren't limited by the lowerBound and the ones of the last
> partition are not limited by the upperBound.
>
> For example, I use the method as follow:
>
> {code:java}
> val data = spark.read.jdbc(url, table, "id", 2, 5, 3,buildProperties())
> .selectExpr("id","appkey","funnel_name")
> data.show(100, false)
> {code}
>
> The result partitions info is :
> 20/08/05 16:58:59 INFO JDBCRelation: Number of partitions: 3, WHERE clauses
> of these partitions: `id` < 3 or `id` is null, `id` >= 3 AND `id` < 4, `id`
> >= 4
> The returned data is:
> ||id|| appkey||funnel_name||
> |0|yanshi|test001|
> |1|yanshi|test002|
> |2|yanshi|test003|
> |3|xingkong|test_funnel|
> |4|xingkong|test_funnel2|
> |5|xingkong|test_funnel3|
> |6|donews|test_funnel4|
> |7|donews|test_funnel|
> |8|donews|test_funnel2|
> |9|dami|test_funnel3|
> |13|dami|test_funnel4|
> |15|xiaoai|test_funnel6|
>
> Normally, the clause of the first partition should be " `id` >=2 and `id` < 3
> " because the lowerBound is 2, and the clause of the last partition should
> be " `id` >= 4 and `id` < 5 ", but the facts are not.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
