Hi All;
I have use case where some of the jars on HDFS, these jars I want to include
in my driver class path
if I pass with --jars it works fine, but if I pass using
spark.driver.extraClassPath it is failed
spark-sql --master yarn --jars hdfs://hacluster/tmp/testjar/* //Jars are
loaded to the
As per the documentation
http://spark.apache.org/docs/2.3.2/structured-streaming-programming-guide.html#stream-stream-joins
, only append mode is supported
*As of Spark 2.3, you can use joins only when the query is in Append output
mode. Other output modes are not yet supported.*
But as per the
Now I ve added same aggregation query as below but still it is didn't filter
val lines_stream = spark.readStream.
format("kafka").
option("kafka.bootstrap.servers", "vm3:21005,vm2:21005").
option("subscribe", "s1").
load().
withColumn("tokens", split('value, ",")).
I am trying to test the water mark concept in structured streaming using the
below program
import java.sql.Timestamp
import org.apache.spark.sql.functions.{col, expr}
import org.apache.spark.sql.streaming.Trigger
val lines_stream = spark.readStream.
format("kafka").
Hi All,
Recently I started migrating the code from kafka08 to kafka010.
in 08 *topics * argument takes care of consuming number of partitions for
each topic.
def createStream(
ssc: StreamingContext,
zkQuorum: String,
groupId: String,
topics: Map[String, Int],
let's say I create table using below command
create table csvTable . using CSV options (path
"/user/data/customer.csv");
Crate Table command executes successfully irrespective of the presence of
file(/user/data/customer.csv)
If I try to insert some rows into this table it fails with below
I want to attempt *SPARK-23545* bug,so I have some questions regarding the
design,
I am analyzing the communications between App Master->Driver and
Executor->Driver and found that only Executors send HeartBeat to Driver.
As per design Executor sends HearBeat to Driver for every
In case of client mode App Master and Driver are in different JVM process,the
port opened by the Driver is vulnerable for flooding attacks as it is not
closing the IDLE connections.
I am thinking to fix this issue using below mechanism
1.Expose configuration to close the IDLE connections as
SparkSubmit will open the port to communicate with the APP Master and
executors.
This port is not closing the IDLE connections,so it is vulnerable for DOS
attack,I did telnet IP port and this connection is not closed.
In order to fix this I tried to Handle in the *userEventTriggered * of
