Re: How to specify default value for StructField?

You can try the below code. val df = spark.read.format("orc").load("/user/hos/orc_files_test_together") df.select(“f1”,”f2”).show 在 2017/2/14

Re: why does spark web UI keeps changing its port?

Config your spark master web ui you can set env SPARK_MASTER_WEBUI_PORT= You can running cmd netstat –nao|grep 4040 to check 4040 is in using ——— I am not sure why Spark web UI keeps changing its port every time I restart a cluster? how can I make it run always on one port? I did make

Re: Writing Spark SQL output in Local and HDFS path

not member of org.apache.spark.sql.DataFrameWriter. Regards Prasad On Thu, Jan 19, 2017 at 4:35 PM, smartzjp <zjp_j...@163.com> wrote: Beacause the reduce number will be not one, so it will out put a fold on the HDFS, You can use “result.write.csv(foldPath)”. -- Hi, Can anyone please

Re: Writing Spark SQL output in Local and HDFS path

Beacause the reduce number will be not one, so it will out put a fold on the HDFS, You can use “result.write.csv(foldPath)”. -- Hi, Can anyone please let us know how to write the output of the Spark SQL in Local and HDFS path using Scala code. Code :- scala> val result =

Re: how the sparksession initialization, set currentDatabase value?

I think if you want to run spark sql on CLI this configuration will be ok, but if you want to run with distributed query engine, start the JDBC/ODBC server and set the hive address info. You can reference this description for more detail.

Re: Spark 2.0.2, KyroSerializer, double[] is not registered.

You can have a try the following code. ObjectArraySerializer serializer = new ObjectArraySerializer(kryo, Double[].class); kryo.register(Double[].class, serializer); --- Hi, all. I enable kyro in spark with spark-defaults.conf: spark.serializer