Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r239670323
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -631,6 +631,33 @@ object SQLConf {
.intConf
Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r239113033
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -631,6 +631,33 @@ object SQLConf {
.intConf
Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r239109280
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/SQLStreamingSink.scala
---
@@ -0,0 +1,115 @@
+/*
+ * Licensed
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/21306
@mccheah you mean the tables user created do not distinguish between stream
and batch, but only when they are actually read from
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/21306
> @stczwd, thanks for taking a look at this. What are the differences
between batch and stream DDL that you think will come up?
1. Source needs to be defined for stream table
2. Str
Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r237721103
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/StreamTableDDLCommandSuite.scala
---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/21306
@rdblue Have you considered about stream table API? It may have some
differences between batch table ddl and stream table ddl
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
I hive send an email to Ryan Blue.
> > > Can you send a mail to Ryan blue for adding this SPIP topic in
tomorrow meeting. Meeting will be conducted tomorrow 05:00 pm PST. If you
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
> Can you send a mail to Ryan blue for adding this SPIP topic in tomorrow
meeting. Meeting will be conducted tomorrow 05:00 pm PST. If you confirm then
we can also attend the meeting.
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
>
I have removed the 'stream' keyword.
> There is a DatasourceV2 com
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
@sujithjay
Please refer
[SPARK-24630](https://issues.apache.org/jira/browse/SPARK-24630) for more
details
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
@cloud-fan @zsxwing @tdas @xuanyuanking
This patch has been submitted for a long time. Do you have any questions?
Can this patch be merged
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
> Currently Dataframe API support "writeStream.start()" api to run
streaming in background, so that query can be executed on that sink, also
multiple stream to stream processing can hap
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
@tdas @zsxwing @cloud-fan
Hi, any other questions block this patch for merge in?
---
-
To unsubscribe, e-mail: reviews
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
@WangTaoTheTonic @cloud-fan @xuanyuanking
I have removed the stream keyword. Table API is supoorted in SQLStreaming
now
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
> ql and normal sql? how could users define watermark with SQL?
Yes, the 'stream' keyword is the only difference from normal sql.
We can use configuration to define waterm
Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r226853809
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -631,6 +631,33 @@ object SQLConf {
.intConf
Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r226853804
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/StreamTableDDLCommandSuite.scala
---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache
Github user stczwd commented on a diff in the pull request:
https://github.com/apache/spark/pull/22575#discussion_r226853724
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -63,7 +63,9 @@ private[kafka010] class
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
cc @xuanyuanking
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
> How should we do if we wanna join two kafka stream and sink the result to
another stream?
`insert into kafka_sql_out select stream t1.value from (select cast(value
as string), timest
Github user stczwd commented on the issue:
https://github.com/apache/spark/pull/22575
@WangTaoTheTonic
Adding 'stream' keyword has two purposes:
- **Mark the entire sql query as a stream query and generate the
SQLStreaming plan tree.**
- **Mark the table type
GitHub user stczwd opened a pull request:
https://github.com/apache/spark/pull/22575
[SPARK-24630][SS][WIP] Support SQLStreaming in Spark
## What changes were proposed in this pull request?
This patch propose new support of SQLStreaming in Spark, Please refer
[SPARK-24630
23 matches
Mail list logo
