Re: How to reissue a delegated token after max lifetime passes for a spark streaming application on a Kerberized cluster

The submit command: spark-submit \ --master yarn \ --deploy-mode cluster \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf -Dlog4j.configuration=xxx -Djava.util.Arrays.useLegacyMergeSort=true" \ --conf

[Spark-ml]Error in training ML models: Missing an output location for shuffle xxx

Hi Spark Comminuty, I was using XGBoost-spark to train a machine learning model. The dataset was not large (around 1G). And I used the following command to submit my application: ''' ./bin/spark-submit --master yarn --deploy-mode client --num-executors 50 --executor-cores 2 --executor-memory 3g

Re: Re: Can an UDF return a custom class other than case class?

Perhaps use of generic StructType may work in your situation of being language agnostic? case-classes are backed by implicits to provide type conversions into columnar. My 2 cents. Thanks, Mutu On Mon, Jan 7, 2019 at 4:13 AM yeikel valdes wrote: > > > Forwarded Message

Re:Parquet file number of columns

Not according to Parquet dev group https://groups.google.com/forum/m/#!topic/parquet-dev/jj7TWPIUlYI On Mon, 07 Jan 2019 05:11:51 -0800 gourav.sengu...@gmail.com wrote Hi, Is there any limit to the number of columns that we can have in Parquet file format?  Thanks and Regards,

Parquet file number of columns

Hi, Is there any limit to the number of columns that we can have in Parquet file format? Thanks and Regards, Gourav Sengupta

RE: RE: Re: Spark Kinesis Connector SSL issue

Hi, The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via: Worker worker = new

Re: Re:Writing RDDs to HDFS is empty

Ideally...we would like to copy paste and try in our end. A screenshot is not enough. If you have private information just remove and create a minimum example we can use to replicate the issue. I'd say similar to this : https://stackoverflow.com/help/mcve On Mon, 07 Jan 2019 04:15:16

RE: Re: Spark Kinesis Connector SSL issue

Any chance you can share a minimum example to replicate the issue? On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbang...@discover.com wrote Hi Valdes,   Thank you for your response, to answer to your question. yes I can   @ben : correct me if I am wrong.   Cheers, Shashi  

RE: Re: Spark Kinesis Connector SSL issue

Hi Valdes, Thank you for your response, to answer to your question. yes I can @ben : correct me if I am wrong. Cheers, Shashi Shashikant Bangera | DevOps Engineer Payment Services DevOps Engineering Email: shashikantbang...@discover.com Group email:

Re: Re:Writing RDDs to HDFS is empty

Sorry,the code is too long,it is simple to say look at the photo i define a arrayBuffer ,there are "1 2", '' 2 3" ," 4 5" in it ,I want to save in hdfs ,so i make it to RDD, sc.

Re: Spark Kinesis Connector SSL issue

Can you call this service with regular code(No Spark)? On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbang...@discover.com wrote Hi team, please help , we are kind of blocked here. Cheers, Shashi -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Fwd:Re: Can an UDF return a custom class other than case class?

Forwarded Message >From : em...@yeikel.com To : kfehl...@gmail.com Date : Mon, 07 Jan 2019 04:11:22 -0800 Subject : Re: Can an UDF return a custom class other than case class? In this case I am just curious because I'd like to know if it is possible. At the same

Re:Writing RDDs to HDFS is empty

Please share a minimum amount of code to try reproduce the issue... On Mon, 07 Jan 2019 00:46:42 -0800 fyyleej...@163.com wrote Hi all, In my experiment program,I used spark Graphx, when running on the Idea in windows,the result is right, but when runing on the linux distributed

Re: Spark Kinesis Connector SSL issue

Hi team, please help , we are kind of blocked here. Cheers, Shashi -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Writing RDDs to HDFS is empty

Hi all, In my experiment program,I used spark Graphx, when running on the Idea in windows,the result is right, but when runing on the linux distributed cluster,the result in hdfs is empty, why?how to solve?