[ https://issues.apache.org/jira/browse/SPARK-22019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167734#comment-16167734 ]
Jen-Ming Chung edited comment on SPARK-22019 at 9/15/17 11:29 AM: ------------------------------------------------------------------ The alternative is giving the explicit schema instead inferring that you don't need to change your pojo class in above test case. {code} StructType schema = new StructType() .add("id", IntegerType) .add("str", StringType); Dataset<SampleData> df = spark.read().schema(schema).json(stringdataset).as( org.apache.spark.sql.Encoders.bean(SampleData.class)); {code} was (Author: jmchung): The alternative is giving the explicit schema instead inferring, means you don't need to change your pojo class. {code} StructType schema = new StructType() .add("id", IntegerType) .add("str", StringType); Dataset<SampleData> df = spark.read().schema(schema).json(stringdataset).as( org.apache.spark.sql.Encoders.bean(SampleData.class)); {code} > JavaBean int type property > --------------------------- > > Key: SPARK-22019 > URL: https://issues.apache.org/jira/browse/SPARK-22019 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: taiho choi > > when the type of SampleData's id is int, following code generates errors. > when long, it's ok. > > {code:java} > @Test > public void testDataSet2() { > ArrayList<String> arr= new ArrayList(); > arr.add("{\"str\": \"everyone\", \"id\": 1}"); > arr.add("{\"str\": \"Hello\", \"id\": 1}"); > //1.read array and change to string dataset. > JavaRDD<String> data = sc.parallelize(arr); > Dataset<String> stringdataset = sqc.createDataset(data.rdd(), > Encoders.STRING()); > stringdataset.show(); //PASS > //2. convert string dataset to sampledata dataset > Dataset<SampleData> df = > sqc.read().json(stringdataset).as(Encoders.bean(SampleData.class)); > df.show();//PASS > df.printSchema();//PASS > Dataset<SampleDataFlat> fad = df.flatMap(SampleDataFlat::flatMap, > Encoders.bean(SampleDataFlat.class)); > fad.show(); //ERROR > fad.printSchema(); > } > public static class SampleData implements Serializable { > public String getStr() { > return str; > } > public void setStr(String str) { > this.str = str; > } > public int getId() { > return id; > } > public void setId(int id) { > this.id = id; > } > String str; > int id; > } > public static class SampleDataFlat { > String str; > public String getStr() { > return str; > } > public void setStr(String str) { > this.str = str; > } > public SampleDataFlat(String str, long id) { > this.str = str; > } > public static Iterator<SampleDataFlat> flatMap(SampleData data) { > ArrayList<SampleDataFlat> arr = new ArrayList<>(); > arr.add(new SampleDataFlat(data.getStr(), data.getId())); > arr.add(new SampleDataFlat(data.getStr(), data.getId()+1)); > arr.add(new SampleDataFlat(data.getStr(), data.getId()+2)); > return arr.iterator(); > } > } > {code} > ==Error message== > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 38, Column 16: failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 38, Column 16: No applicable constructor/method found for actual parameters > "long"; candidates are: "public void SparkUnitTest$SampleData.setId(int)" > /* 024 */ public java.lang.Object apply(java.lang.Object _i) { > /* 025 */ InternalRow i = (InternalRow) _i; > /* 026 */ > /* 027 */ final SparkUnitTest$SampleData value1 = false ? null : new > SparkUnitTest$SampleData(); > /* 028 */ this.javaBean = value1; > /* 029 */ if (!false) { > /* 030 */ > /* 031 */ > /* 032 */ boolean isNull3 = i.isNullAt(0); > /* 033 */ long value3 = isNull3 ? -1L : (i.getLong(0)); > /* 034 */ > /* 035 */ if (isNull3) { > /* 036 */ throw new NullPointerException(((java.lang.String) > references[0])); > /* 037 */ } > /* 038 */ javaBean.setId(value3); -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org