Re: Change nullable property in Dataset schema

ork for my purpose. It actually does nothing. Kazuaki Ishizaki From: Jacek Laskowski <ja...@japila.pl> To: Kazuaki Ishizaki/Japan/IBM@IBMJP Cc: user <user@spark.apache.org> Date: 2016/08/15 04:56 Subject: Re: Change nullable property in Dataset schema On Wed,

Re: Change nullable property in Dataset schema

<ko...@tresata.com> To: Kazuaki Ishizaki/Japan/IBM@IBMJP Cc: "user@spark.apache.org" <user@spark.apache.org> Date: 2016/08/16 04:35 Subject: Re: Change nullable property in Dataset schema why do you want the array to have nullable = false? what is the b

Re: Change nullable property in Dataset schema

ow to change nullable property in Dataframe schema, I > found the following approaches. > http://stackoverflow.com/questions/33193958/change- > nullable-property-of-column-in-spark-dataframe > https://github.com/apache/spark/pull/13873(Not merged yet) > > However, I cannot find how

Re: Change nullable property in Dataset schema

On Wed, Aug 10, 2016 at 12:04 AM, Kazuaki Ishizaki wrote: > import testImplicits._ > test("test") { > val ds1 = sparkContext.parallelize(Seq(Array(1, 1), Array(2, 2), > Array(3, 3)), 1).toDS You should just Seq(...).toDS > val ds2 = ds1.map(e => e) Why are you

Re: Change nullable property in Dataset schema

azuaki Ishizaki From: Kazuaki Ishizaki/Japan/IBM@IBMJP To: user@spark.apache.org Date: 2016/08/03 23:46 Subject: Change nullable property in Dataset schema Dear all, Would it be possible to let me know how to change nullable property in Dataset? When I looked for how to change nullabl

Change nullable property in Dataset schema

https://github.com/apache/spark/pull/13873 (Not merged yet) However, I cannot find how to change nullable property in Dataset schema. Even when I wrote the following program, nullable property for "value: array" in ds2.schema is not changed. If my understanding is correct, current Spar