Github user pwendell commented on a diff in the pull request:
https://github.com/apache/incubator-spark/pull/638#discussion_r9980751
--- Diff:
core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala ---
@@ -330,4 +335,74 @@ class PairRDDFunctionsSuite extends FunSuite with
SharedSparkContext {
(1, ArrayBuffer(1)),
(2, ArrayBuffer(1))))
}
+
+ test("saveNewAPIHadoopFile should call setConf if format is
configurable") {
+ val pairs = sc.parallelize(Array((new Integer(1), new Integer(1))))
+ val conf = new Configuration()
+
+ //No error, non-configurable formats still work
+ pairs.saveAsNewAPIHadoopFile[FakeFormat]("ignored")
+
+ //Configurable intercepts get configured
+ //ConfigTestFormat throws an exception if we try to write to it
+ //when setConf hasn't been thrown first.
+ //Assertion is in ConfigTestFormat.getRecordWriter
+ pairs.saveAsNewAPIHadoopFile[ConfigTestFormat]("ignored")
+ }
+}
+
+// These classes are fakes for testing
+// "saveNewAPIHadoopFile should call setConf if format is configurable".
+// Unfortunately, they have to be top level classes, and not defined in
+// the test method, because otherwise Scala won't generate no-args
constructors
+// and the test will therefore throw InstantiationException when
saveAsNewAPIHadoopFile
+// tries to instantiate them with Class.newInstance.
+class FakeWriter extends RecordWriter[Integer,Integer] {
--- End diff --
`Integer, Integer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
[email protected] or file a JIRA ticket with INFRA.
---
