Re: Thrift2 Server on Kubernetes?

Hi Meikel, If you want to run Spark Thrift Server on Kubernetes, take a look at my blog post: https://itnext.io/hive-on-spark-in-kubernetes-115c8e9fa5c1 Cheers, - Kidong Lee. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Hive on Spark in Kubernetes.

Hi all, I have recently written a blog about hive on spark in kubernetes environment: - https://itnext.io/hive-on-spark-in-kubernetes-115c8e9fa5c1 In this blog, you can find how to run hive on kubernetes using spark thrift server compatible with hive server2. Cheers, - Kidong. -- Sent from:

Re: UnknownHostException is thrown when spark job whose jar files will be uploaded to s3 object storage via https is submitted to kubernetes

Sorry, I have missed setting path style access of s3 property to submit. If I have added --conf spark.hadoop.fs.s3a.path.style.access=true in the spark submit, it works fine! - Kidong. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

UnknownHostException is thrown when spark job whose jar files will be uploaded to s3 object storage via https is submitted to kubernetes

MinIO S3 Object Storage, spark submit cannot find my S3 endpoint with the WRONG HOST NAME. Let's see my spark job submit: export MASTER=k8s://https://10.0.4.5:6443; export NAMESPACE=ai-developer; export ENDPOINT=https://mykidong-tenant.minio.cloudchef-labs.com; spark-submit \ --master $MASTER

Re: spark-shell, how it works internally

I have found a source how to compile spark codes and dynamically load them into distributed executors in spark repl: https://ardoris.wordpress.com/2014/03/30/how-spark-does-class-loading/ If you run spark repl, you can find the spark configuration like this :

spark-shell, how it works internally

Hi, I have a plan to program such functions like in spark-shell. When spark-shell is run for yarn, it seems that spark-shell application is submitted to yarn with yarn cluster client mode. I am curious when the input codes in scala are typed in spark-shell, how the input codes in scala are

Re: KafkaUtils and specifying a specific partition

If you want to use another kafka receiver instead of current spark kafka receiver, You can see this: https://github.com/mykidong/spark-kafka-simple-consumer-receiver/blob/master/src/main/java/spark/streaming/receiver/kafka/KafkaReceiverUtils.java You can handle to get just the stream from

Re: Any sample code for Kafka consumer

In java, you can see this example: https://github.com/mykidong/spark-kafka-simple-consumer-receiver - Kidong. -- Original Message -- From: icecreamlc [via Apache Spark User List] ml-node+s1001560n21746...@n3.nabble.com To: mykidong mykid...@gmail.com Sent: 2015-02-21 오전 11:16:37 Subject

Re: Integrerate Spark Streaming and Kafka, but get bad symbolic reference error

Maybe, you can use alternative kafka receiver which I wrote: https://github.com/mykidong/spark-kafka-simple-consumer-receiver - Kidong. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Integrerate-Spark-Streaming-and-Kafka-but-get-bad-symbolic-reference

Re: How to replay consuming messages from kafka using spark streaming?

Hi, I have written spark streaming kafka receiver using kafka simple consumer api: https://github.com/mykidong/spark-kafka-simple-consumer-receiver This kafka receiver can be used as alternative to the current spark streaming kafka receiver which is just written in high level kafka consumer api

Re: Low Level Kafka Consumer for Spark

Hi Dibyendu, I am using kafka 0.8.1.1 and spark 1.2.0. After modifying these version of your pom, I have rebuilt your codes. But I have not got any messages from ssc.receiverStream(new KafkaReceiver(_props, i)). I have found, in your codes, all the messages are retrieved correctly, but

How to replay consuming messages from kafka using spark streaming?

Hi, My Spark Streaming Job is doing like kafka etl to HDFS. For instance, every 10 min. my streaming job is retrieving messages from kafka, and save them as avro files onto hdfs. My question is, if worker fails to write avro to hdfs, sometimes, I want to replay consuming messages from the last

Re: JdbcRDD

I had also same problem to use JdbcRDD in java. For me, I have written a class in scala to get JdbcRDD, and I call this instance from java. for instance, JdbcRDDWrapper.scala like this: ... import java.sql._ import org.apache.spark.SparkContext import org.apache.spark.rdd.JdbcRDD import

How to read just specified columns from parquet file using SparkSQL.

Hi, I am new to SparkSQL. I want to read the specified columns from the parquet, not all the columns defined in the parquet file. For instance, the schema of the parquet file would look like this: { type: record, name: ElectricPowerUsage, namespace: jcascalog.parquet.example, fields: [