Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Hi, Have you raised a Jira for this? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Hi Gene, It's a Spark 2.0 issue. I switch to Spark 1.6.1 it's ok now. Thanks. On Thursday, July 28, 2016 at 4:25:48 PM UTC+7, Chanh Le wrote: > > Hi everyone, > > I have problem when I create a external table in Spark Thrift Server (STS) > and query the data. > > Scenario: > *Spark 2.0* >

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

ok I am using Spark 1.6.1 and Hive 2 but don't seem to be the issue. The errors look very similar I guess someone from Hive can answer this issue! Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Hi Mich some thing different between your log > On Jul 30, 2016, at 6:58 PM, Mich Talebzadeh > wrote: > > parquet-mr version 1.6.0 > org.apache.parquet.VersionParser$VersionParseException: Could not parse > created_by: parquet-mr version 1.6.0 using format: (.+)

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Yes. Something is wrong even when I query table in Hive with correct data it throws error about corrupt stats before showing the result of 1 row hive> select * from abc limit 1; Jul 30, 2016 12:52:14 PM WARNING: org.apache.parquet.CorruptStatistics: Ignoring statistics because created_by could

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

I received this log when recent debug. Is that related to PARQUET-251 But I checked Spark current using parquet 1.8.1 means it already fixed. 16/07/30 18:32:11 INFO SparkExecuteStatementOperation: Running query 'select * from topic18' with 72649e37-3ef4-4acd-8d01-4a28e79a1f9a 16/07/30 18:32:11

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Actually Hive SQL is a superset of Spark SQL. Data type may not be an issue. If I create the table after DataFrame creation as explicitly a Hive parquet table through Spark, Hive sees it and you can see it in Spark thrift server with data in it (basically you are using Hive Thrift server under

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

I agree with you. Maybe some change on data type in Spark that Hive still not support or not competitive so that why It shows NULL. > On Jul 30, 2016, at 5:47 PM, Mich Talebzadeh > wrote: > > I think it is still a Hive problem because Spark thrift server is

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

I think it is still a Hive problem because Spark thrift server is basically a Hive thrift server. An ACID test would be to log in to Hive CLI or Hive thrift server (you are actually using Hive thrift server on port 1 when using Spark thrift server) and see whether you see data When you use

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Hi Mich, Thanks for supporting. Here some of my thoughts. > BTW can you log in to thrift server and do select * from limit 10 > > Do you see the rows? Yes I can see the row but all the fields value NULL. > Works OK for me You just test the number of row. In my case I check and it shows 117

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

BTW can you log in to thrift server and do select * from limit 10 Do you see the rows? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Spark Thrift Server (Spark 2.0) show table has value with NULL in all fields

Works OK for me scala> val df = sqlContext.read.format("com.databricks.spark.csv").option("inferSchema", "true").option("header", "false").load("hdfs://rhes564:9000/data/stg/accounts/ll/18740868") df: org.apache.spark.sql.DataFrame = [C0: string, C1: string, C2: string, C3: string, C4: string,