[
https://issues.apache.org/jira/browse/SPARK-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15071347#comment-15071347
]
Xiu (Joe) Guo commented on SPARK-12521:
---------------------------------------
In 1.5.2 {code}sqlContext.load(){code} is deprecated, but I can still reproduce
with:{code}sqlContext.read.jdbc(){code}
I don't think it is the size of your numbers. I can reproduce with small
integers given as lowerBound/upperBound with my setup. Can you maybe try adding
"L" at the end of your number to verify that it still gives wrong results?
I think the problem is the lowerBound and upperBound are not honored here,
Spark just retrieves every row instead of 1001 rows bounded in your case.
> DataFrame Partitions in java does not work
> ------------------------------------------
>
> Key: SPARK-12521
> URL: https://issues.apache.org/jira/browse/SPARK-12521
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.5.2
> Reporter: Sergey Podolsky
>
> Hello,
> Partition does not work in Java interface of the DataFrame:
> {code}
> SQLContext sqlContext = new SQLContext(sc);
> Map<String, String> options = new HashMap<>();
> options.put("driver", ORACLE_DRIVER);
> options.put("url", ORACLE_CONNECTION_URL);
> options.put("dbtable",
> "(SELECT * FROM JOBS WHERE ROWNUM < 10000) tt");
> options.put("lowerBound", "2704225000");
> options.put("upperBound", "2704226000");
> options.put("partitionColumn", "ID");
> options.put("numPartitions", "10");
> DataFrame jdbcDF = sqlContext.load("jdbc", options);
> List<Row> jobsRows = jdbcDF.collectAsList();
> System.out.println(jobsRows.size());
> {code}
> gives 9999 while expected 1000. Is it because of big decimal of boundaries or
> partitioins does not work at all in Java?
> Thanks.
> Sergey
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
