Hello there,
I am trying to understand how and when does DataFrame (or Dataset) sets
nullable = true vs false on a schema.
Here is my observation from a sample code I tried...
scala> spark.createDataset(Seq((1, "a", 2.0d), (2, "b", 2.0d), (3, "c",
2.0d))).toDF("col1", "col2", "col3").withColumn
Nullable is just a hint to the optimizer that its impossible for there to
be a null value in this column, so that it can avoid generating code for
null-checks. When in doubt, we set nullable=true since it is always safer
to check.
Why in particular are you trying to change the nullability of the
Hello Michael,
Thank you for looking into this query. In my case there seem to be an issue
when I union a parquet file read from disk versus another dataframe that I
construct in-memory. The only difference I see is the containsNull = true.
In fact, I do not see any errors with union on the simple
What is the issue you see when unioning?
On Wed, Oct 19, 2016 at 6:39 PM, Muthu Jayakumar wrote:
> Hello Michael,
>
> Thank you for looking into this query. In my case there seem to be an
> issue when I union a parquet file read from disk versus another dataframe
> that I construct in-memory. Th
Sorry for the late response. Here is what I am seeing...
Schema from parquet file.
d1.printSchema()
root
|-- task_id: string (nullable = true)
|-- task_name: string (nullable = true)
|-- some_histogram: struct (nullable = true)
||-- values: array (nullable = true)
|||-- element
Hi Muthu,
What is the version of Spark are you using? This seems to be a bug in
the analysis phase.
Cheng
On 10/21/16 12:50 PM, Muthu Jayakumar wrote:
Sorry for the late response. Here is what I am seeing...
Schema from parquet file.
d1.printSchema()
root
|-- task_id: string (nullable =
Yea, confirmed. While analyzing unions, we treat StructTypes with
different field nullabilities as incompatible types and throws this error.
Opened https://issues.apache.org/jira/browse/SPARK-18058 to track this
issue. Thanks for reporting!
Cheng
On 10/21/16 3:15 PM, Cheng Lian wrote:
Hi
Thanks Cheng Lian for opening the JIRA. I found this with Spark 2.0.0.
Thanks,
Muthu
On Fri, Oct 21, 2016 at 3:30 PM, Cheng Lian wrote:
> Yea, confirmed. While analyzing unions, we treat StructTypes with
> different field nullabilities as incompatible types and throws this error.
>
> Opened htt
This rather innocent looking optimization flag nullable has caused a lot of
bugs... Makes me wonder if we are better off without it
On Oct 21, 2016 8:37 PM, "Muthu Jayakumar" wrote:
> Thanks Cheng Lian for opening the JIRA. I found this with Spark 2.0.0.
>
> Thanks,
> Muthu
>
> On Fri, Oct 21, 2
On Fri, Oct 21, 2016 at 8:40 PM, Koert Kuipers wrote:
> This rather innocent looking optimization flag nullable has caused a lot
> of bugs... Makes me wonder if we are better off without it
>
Yes... my most regretted design decision :(
Please give thoughts here: https://issues.apache.org/jira/b
Dear all,
In the latest version of spark there's a feature called : automatic
partition discovery and Schema migration for parquet. As far as I know,
this gives the ability to split the DataFrame into several parquet files,
and by just loading the parent directory one can get the global schema of
apply the schema on top the existing Dataframe ?. Schema
may be just doing the columns reordering in the most of the cases.
i have tried this "
DataFrame dfNew = hc.createDataFrame(df.rdd(), ((StructType)
DataType.fromJson(schema)));
"
But this will map the columns based on index a
Hi Spark users and developers,
I have a dataframe with the following schema (Spark 1.5.1):
StructType(StructField(type,StringType,true),
StructField(timestamp,LongType,false))
After I save the dataframe in parquet and read it back, I get the following
schema:
StructType(StructField(timestamp,Lo
I forgot to mention that the imageId field is a custom scala object. Do I
need to implement some special method to make it works (equal, hashCode ) ?
On Tue, Apr 14, 2015 at 5:00 PM, Jaonary Rabarisoa
wrote:
> Dear all,
>
> In the latest version of spark there's a feature called : automatic
> pa
ply
> different schema for that dataframe. Column names will be same, but Data or
> Schema may contain some extra columns.
>
> Is there any way i can apply the schema on top the existing Dataframe ?.
> Schema may be just doing the columns reordering in the most of the cases.
>
> i
For compatibility reasons, we always write data out as nullable in
parquet. Given that that bit is only an optimization that we don't
actually make much use of, I'm curious why you are worried that its
changing to true?
On Tue, Oct 20, 2015 at 8:24 AM, Jerry Lam wrote:
> Hi Spark users and deve
Let me share my 2 cents.
First, this is not documented in the official document. Maybe we should do
it? http://spark.apache.org/docs/latest/sql-programming-guide.html
Second, nullability is a significant concept in the database people. It is
part of schema. Extra codes are needed for evaluating i
>
> First, this is not documented in the official document. Maybe we should do
> it? http://spark.apache.org/docs/latest/sql-programming-guide.html
>
Pull requests welcome.
> Second, nullability is a significant concept in the database people. It is
> part of schema. Extra codes are needed for e
Sure. Will try to do a pull request this week.
Schema evolution is always painful for database people. IMO, NULL is a bad
design in the original system R. It introduces a lot of problems during the
system migration and data integration.
Let me find a possible scenario: RDBMS is used as an ODS. Sp
t; Date: 10/20/2015 01:18 PM
> Subject: Re: Spark SQL: Preserving Dataframe Schema
>
> Sure. Will try to do a pull request this week.
>
> Schema evolution is always painful for database people. IMO, NULL is
> a bad design in the original system R. It introduces a lot of
>
Hey,
I'm working on this use case that involves converting DStreams to
Dataframes after some transformations. I've simplified my code into the
following snippet so as to reproduce the error. Also, I've mentioned below
my environment settings.
*Environment:*
Spark Version: 2.2.0
Java: 1.8
Executi
Hi all,
If you follow the example of schema merging in the spark documentation
http://spark.apache.org/docs/latest/sql-programming-guide.html#schema-merging
you obtain the following results when you want to load the result data :
single triple double
1 3 null
2 6 null
4 1
Schema merging is not the feature you are looking for. It is designed when
you are adding new records (that are not associated with old records),
which may or may not have new or missing columns.
In your case it looks like you have two datasets that you want to load
separately and join on a key.
Hi,
I am trying to create new Cassandra table by inferring schema from JSON:
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md
I am not able to get createCassandraTable function on Dataframe:
import com.datastax.spark.connector._
df.createCassandraTable(
Hello,
Anyone has any comment or ideas regarding:
https://stackoverflow.com/questions/71610435/how-to-overwrite-pyspark-dataframe-schema-without-data-scan
please?
Cheers - Rafal
25 matches
Mail list logo