[jira] [Reopened] (SPARK-32486) Issue with deserialization and persist api in latest spark java versions
[ https://issues.apache.org/jira/browse/SPARK-32486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Kumar reopened SPARK-32486: -- This issue is happening in all higher version, please fix it > Issue with deserialization and persist api in latest spark java versions > > > Key: SPARK-32486 > URL: https://issues.apache.org/jira/browse/SPARK-32486 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.4.4, 2.4.5, 2.4.6, 3.0.0 > Environment: It's happening on all the os and java8 >Reporter: Dinesh Kumar >Priority: Blocker > > Hey Team, We have class level object instantiations in one of our Classes. > When we want to persist that data into the Dataset of this class Type it's > not persisting the null values instead it's taking class level precedence. > i.e. It's showing as new object. > Eg: > _Test.class has below class level attributes:_ > _private Test1 testNumber = new Test1();_ > _private Test2 testNumber2;_ > > String inputLocation = "src/test/resources/pipeline/test.parquet"; > Dataset ds = this.session.read().parquet(inputLocation); > ds.printSchema(); > ds.foreach(input->{ > System.out.println(input); // When we verified it's showing testNumber, > testNumber2 as null > }); > Dataset inputDataSet = ds.as(Encoders.bean(Test.class)); > inputDataSet.foreach(input->{ > System.out.println(input); // When we verified it's showing testNumber as > new Test1(), testNumber2 as null > }); > > > This is the same issue with dataset.persist() call aswell. It is happening > with all 2.4.4 and higher versions. Can you please fix it? > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32486) Issue with deserialization and persist api in latest spark java versions
[ https://issues.apache.org/jira/browse/SPARK-32486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167614#comment-17167614 ] Dinesh Kumar commented on SPARK-32486: -- Hey, Its happening in all the higher versions. Can you please reopen? > Issue with deserialization and persist api in latest spark java versions > > > Key: SPARK-32486 > URL: https://issues.apache.org/jira/browse/SPARK-32486 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.4.4, 2.4.5, 2.4.6, 3.0.0 > Environment: It's happening on all the os and java8 >Reporter: Dinesh Kumar >Priority: Blocker > > Hey Team, We have class level object instantiations in one of our Classes. > When we want to persist that data into the Dataset of this class Type it's > not persisting the null values instead it's taking class level precedence. > i.e. It's showing as new object. > Eg: > _Test.class has below class level attributes:_ > _private Test1 testNumber = new Test1();_ > _private Test2 testNumber2;_ > > String inputLocation = "src/test/resources/pipeline/test.parquet"; > Dataset ds = this.session.read().parquet(inputLocation); > ds.printSchema(); > ds.foreach(input->{ > System.out.println(input); // When we verified it's showing testNumber, > testNumber2 as null > }); > Dataset inputDataSet = ds.as(Encoders.bean(Test.class)); > inputDataSet.foreach(input->{ > System.out.println(input); // When we verified it's showing testNumber as > new Test1(), testNumber2 as null > }); > > > This is the same issue with dataset.persist() call aswell. It is happening > with all 2.4.4 and higher versions. Can you please fix it? > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-32486) Issue with deserialization and persist api in latest spark java versions
Dinesh Kumar created SPARK-32486: Summary: Issue with deserialization and persist api in latest spark java versions Key: SPARK-32486 URL: https://issues.apache.org/jira/browse/SPARK-32486 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 3.0.0, 2.4.6, 2.4.5, 2.4.4 Environment: It's happening on all the os and java8 Reporter: Dinesh Kumar Fix For: 2.3.3 Hey Team, We have class level object instantiations in one of our Classes. When we want to persist that data into the Dataset of this class Type it's not persisting the null values instead it's taking class level precedence. i.e. It's showing as new object. Eg: _Test.class has below class level attributes:_ _private Test1 testNumber = new Test1();_ _private Test2 testNumber2;_ String inputLocation = "src/test/resources/pipeline/test.parquet"; Dataset ds = this.session.read().parquet(inputLocation); ds.printSchema(); ds.foreach(input->{ System.out.println(input); // When we verified it's showing testNumber, testNumber2 as null }); Dataset inputDataSet = ds.as(Encoders.bean(Test.class)); inputDataSet.foreach(input->{ System.out.println(input); // When we verified it's showing testNumber as new Test1(), testNumber2 as null }); This is the same issue with dataset.persist() call aswell. It is happening with all 2.4.4 and higher versions. Can you please fix it? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org