niuge01 commented on a change in pull request #3515: [CARBONDATA-3623]: Fixed
global sort compaction failure on timestamp column
URL: https://github.com/apache/carbondata/pull/3515#discussion_r364135260
##########
File path:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -443,23 +438,18 @@ object DataLoadProcessBuilderOnSpark {
.asScala
.map(_.getColName)
.toArray
+
+ /**
+ *
[[org.apache.spark.sql.catalyst.expressions.objects.ValidateExternalType]]
validates the
+ * datatype of column data and corresponding datatype in schema provided
to create dataframe.
+ * Since carbonScanRDD gives Long data for timestamp column and
corresponding column datatype in
+ * schema is Timestamp, this validation fails if we use createDataFrame
API which takes rdd as
+ * input. Hence, We need to give the List[Row] compatible with the schema
datatypes. So using
+ * the createDataFrame API which takes List[Row] and schema as input.
+ */
val schema = SparkTypeConverter.createSparkSchema(carbonTable, columns)
- val rdd: RDD[InternalRow] = new CarbonScanRDD[CarbonRow](
- sparkSession,
- columnProjection = new CarbonProjection(columns),
- null,
- carbonTable.getAbsoluteTableIdentifier,
- carbonTable.getTableInfo.serialize,
- carbonTable.getTableInfo,
- new CarbonInputMetrics,
- null,
- classOf[SparkDataTypeConverterImpl],
- classOf[CarbonRowReadSupport],
- splits.asJava)
- .map { row =>
- new GenericInternalRow(row.getData.asInstanceOf[Array[Any]])
- }
- SparkSQLUtil.execute(rdd, schema, sparkSession)
+ val listOfRows =
sparkSession.sqlContext.table(carbonTable.getTableName).collect().toList.asJava
Review comment:
If use List[Row] to instead of RDD, i think it need use more memory to cache
all rows, all right?
May increase OOM risk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
