[jira] [Commented] (SPARK-25985) Verify the SPARK-24613 Cache with UDF could not be matched with subsequent dependent caches
[ https://issues.apache.org/jira/browse/SPARK-25985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109201#comment-17109201 ] Nick Afshartous commented on SPARK-25985: - [~smilegator] Can you please comment if this task is still relevant. If so I'd like to look into it if you could please elaborate on "works well" in the description. > Verify the SPARK-24613 Cache with UDF could not be matched with subsequent > dependent caches > --- > > Key: SPARK-25985 > URL: https://issues.apache.org/jira/browse/SPARK-25985 > Project: Spark > Issue Type: Sub-task > Components: SQL, Tests >Affects Versions: 3.0.0 >Reporter: Xiao Li >Priority: Major > Labels: starter > > Verify whether recacheByCondition works well when the cache data is with UDF. > This is a follow-up of https://github.com/apache/spark/pull/21602 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104661#comment-17104661 ] Nick Afshartous commented on SPARK-27249: - [~enrush] Hi Everett, can you please chime in on the thread in the PR. There's a question about whether or not the need is covered by existing API's. > Developers API for Transformers beyond UnaryTransformer > --- > > Key: SPARK-27249 > URL: https://issues.apache.org/jira/browse/SPARK-27249 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.1.0 >Reporter: Everett Rush >Priority: Minor > Labels: starter > Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png > > Original Estimate: 96h > Remaining Estimate: 96h > > It would be nice to have a developers' API for dataset transformations that > need more than one column from a row (ie UnaryTransformer inputs one column > and outputs one column) or that contain objects too expensive to initialize > repeatedly in a UDF such as a database connection. > > Design: > Abstract class PartitionTransformer extends Transformer and defines the > partition transformation function as Iterator[Row] => Iterator[Row] > NB: This parallels the UnaryTransformer createTransformFunc method > > When developers subclass this transformer, they can provide their own schema > for the output Row in which case the PartitionTransformer creates a row > encoder and executes the transformation. Alternatively the developer can set > output Datatype and output col name. Then the PartitionTransformer class will > create a new schema, a row encoder, and execute the transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[
https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086029#comment-17086029
]
Nick Afshartous edited comment on SPARK-27249 at 4/17/20, 7:54 PM:
---
[~enrush] Hi Everett,
Using the {{Iterator}} approach would seem to deviate from the existing
{{Transformer}} contract. More specifically, the {{transform}} function should
output a new {{DataFrame}} object. What I would propose is adding a new
function {{Transformer.compose}} to allow the composition of {{Transformers}}.
{code}
def compose(other: Transformer): Transformer = {
new Transformer {
override def transform(dataset: Dataset[_]): DataFrame = {
other.transform(this.transform(dataset));
}
...
{code}
Then one could {{compose}} {{Transformers}} which effectively would enable
multi-column transformations.
{code}
val dataFrame = ...
val transformers = List(transformer1, transformer2, transformer3)
val multiColumnTransformer = transformers.reduce((x, y) => x.compose(y))
multiColumnTransformer.transform(dataFrame)
{code}
I'd be happy to submit a PR if this meets your requirments.
was (Author: nafshartous):
[~enrush] Hi Everett,
Using the {{Iterator}} approach would seem to deviate from the existing
{{Transformer}} contract. More specifically, the {{transform}} function should
output a new {{DataFrame}} object. What I would propose is adding a new
function {{Transformer.compose}} to allow the composition of {{Transformers}}.
{code}
def compose(other: Transformer): Transformer = {
new Transformer {
override def transform(dataset: Dataset[_]): DataFrame = {
other.transform(this.transform(dataset));
}
...
{code}
Then one could {{compose}} {{Transformers}} which effectively would enable
multi-column transformations.
{code}
val dataFrame = ...
val transformers = List(transformer1, transformer2, transformer3)
val multiColumnTransformer = transformers.reduce((x, y) => x.compose(y))
multiColumnTransformer.transform(dataFrame)
{code}
I'd be happy to submit a PR if this meets your requirments.
> Developers API for Transformers beyond UnaryTransformer
> ---
>
> Key: SPARK-27249
> URL: https://issues.apache.org/jira/browse/SPARK-27249
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Affects Versions: 3.1.0
>Reporter: Everett Rush
>Priority: Minor
> Labels: starter
> Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> It would be nice to have a developers' API for dataset transformations that
> need more than one column from a row (ie UnaryTransformer inputs one column
> and outputs one column) or that contain objects too expensive to initialize
> repeatedly in a UDF such as a database connection.
>
> Design:
> Abstract class PartitionTransformer extends Transformer and defines the
> partition transformation function as Iterator[Row] => Iterator[Row]
> NB: This parallels the UnaryTransformer createTransformFunc method
>
> When developers subclass this transformer, they can provide their own schema
> for the output Row in which case the PartitionTransformer creates a row
> encoder and executes the transformation. Alternatively the developer can set
> output Datatype and output col name. Then the PartitionTransformer class will
> create a new schema, a row encoder, and execute the transformation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[
https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086029#comment-17086029
]
Nick Afshartous commented on SPARK-27249:
-
[~enrush] Hi Everett,
Using the {{Iterator}} approach would seem to deviate from the existing
{{Transformer}} contract. More specifically, the {{transform}} function should
output a new {{DataFrame}} object. What I would propose is adding a new
function {{Transformer.compose}} to allow the composition of {{Transformers}}.
{code}
def compose(other: Transformer): Transformer = {
new Transformer {
override def transform(dataset: Dataset[_]): DataFrame = {
other.transform(this.transform(dataset));
}
...
{code}
Then one could {{compose}} {{Transformers}} which effectively would enable
multi-column transformations.
{code}
val dataFrame = ...
val transformers = List(transformer1, transformer2, transformer3)
val multiColumnTransformer = transformers.reduce((x, y) => x.compose(y))
multiColumnTransformer.transform(dataFrame)
{code}
I'd be happy to submit a PR if this meets your requirments.
> Developers API for Transformers beyond UnaryTransformer
> ---
>
> Key: SPARK-27249
> URL: https://issues.apache.org/jira/browse/SPARK-27249
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Affects Versions: 3.1.0
>Reporter: Everett Rush
>Priority: Minor
> Labels: starter
> Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> It would be nice to have a developers' API for dataset transformations that
> need more than one column from a row (ie UnaryTransformer inputs one column
> and outputs one column) or that contain objects too expensive to initialize
> repeatedly in a UDF such as a database connection.
>
> Design:
> Abstract class PartitionTransformer extends Transformer and defines the
> partition transformation function as Iterator[Row] => Iterator[Row]
> NB: This parallels the UnaryTransformer createTransformFunc method
>
> When developers subclass this transformer, they can provide their own schema
> for the output Row in which case the PartitionTransformer creates a row
> encoder and executes the transformation. Alternatively the developer can set
> output Datatype and output col name. Then the PartitionTransformer class will
> create a new schema, a row encoder, and execute the transformation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078751#comment-17078751 ] Nick Afshartous commented on SPARK-27249: - [~enrush] Hi Everett, checking back on my question in the last comment. > Developers API for Transformers beyond UnaryTransformer > --- > > Key: SPARK-27249 > URL: https://issues.apache.org/jira/browse/SPARK-27249 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.1.0 >Reporter: Everett Rush >Priority: Minor > Labels: starter > Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png > > Original Estimate: 96h > Remaining Estimate: 96h > > It would be nice to have a developers' API for dataset transformations that > need more than one column from a row (ie UnaryTransformer inputs one column > and outputs one column) or that contain objects too expensive to initialize > repeatedly in a UDF such as a database connection. > > Design: > Abstract class PartitionTransformer extends Transformer and defines the > partition transformation function as Iterator[Row] => Iterator[Row] > NB: This parallels the UnaryTransformer createTransformFunc method > > When developers subclass this transformer, they can provide their own schema > for the output Row in which case the PartitionTransformer creates a row > encoder and executes the transformation. Alternatively the developer can set > output Datatype and output col name. Then the PartitionTransformer class will > create a new schema, a row encoder, and execute the transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[
https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018329#comment-17018329
]
Nick Afshartous edited comment on SPARK-27249 at 1/17/20 9:29 PM:
--
[~enrush] Hi Everett,
The {{Dataset}} API has an experimental function {{mapPartitions}} for
transforming {{Datasets}}. Does this satisfy your requirements ?
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset
was (Author: nafshartous):
[~enrush] Hi Everett,
The {{Dataset}} API has an experimental function {{mapPartitions}} for
transforming {{Dataset}}s. Does this satisfy your requirements ?
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset
> Developers API for Transformers beyond UnaryTransformer
> ---
>
> Key: SPARK-27249
> URL: https://issues.apache.org/jira/browse/SPARK-27249
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Affects Versions: 3.0.0
>Reporter: Everett Rush
>Priority: Minor
> Labels: starter
> Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> It would be nice to have a developers' API for dataset transformations that
> need more than one column from a row (ie UnaryTransformer inputs one column
> and outputs one column) or that contain objects too expensive to initialize
> repeatedly in a UDF such as a database connection.
>
> Design:
> Abstract class PartitionTransformer extends Transformer and defines the
> partition transformation function as Iterator[Row] => Iterator[Row]
> NB: This parallels the UnaryTransformer createTransformFunc method
>
> When developers subclass this transformer, they can provide their own schema
> for the output Row in which case the PartitionTransformer creates a row
> encoder and executes the transformation. Alternatively the developer can set
> output Datatype and output col name. Then the PartitionTransformer class will
> create a new schema, a row encoder, and execute the transformation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[
https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018329#comment-17018329
]
Nick Afshartous commented on SPARK-27249:
-
[~enrush] Hi Everett,
The {{Dataset}} API has an experimental function {{mapPartitions}} for
transforming {{Dataset}} . Does this satisfy your requirements ?
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset
> Developers API for Transformers beyond UnaryTransformer
> ---
>
> Key: SPARK-27249
> URL: https://issues.apache.org/jira/browse/SPARK-27249
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Affects Versions: 3.0.0
>Reporter: Everett Rush
>Priority: Minor
> Labels: starter
> Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> It would be nice to have a developers' API for dataset transformations that
> need more than one column from a row (ie UnaryTransformer inputs one column
> and outputs one column) or that contain objects too expensive to initialize
> repeatedly in a UDF such as a database connection.
>
> Design:
> Abstract class PartitionTransformer extends Transformer and defines the
> partition transformation function as Iterator[Row] => Iterator[Row]
> NB: This parallels the UnaryTransformer createTransformFunc method
>
> When developers subclass this transformer, they can provide their own schema
> for the output Row in which case the PartitionTransformer creates a row
> encoder and executes the transformation. Alternatively the developer can set
> output Datatype and output col name. Then the PartitionTransformer class will
> create a new schema, a row encoder, and execute the transformation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[
https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018329#comment-17018329
]
Nick Afshartous edited comment on SPARK-27249 at 1/17/20 9:28 PM:
--
[~enrush] Hi Everett,
The {{Dataset}} API has an experimental function {{mapPartitions}} for
transforming {{Dataset}}s. Does this satisfy your requirements ?
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset
was (Author: nafshartous):
[~enrush] Hi Everett,
The {{Dataset}} API has an experimental function {{mapPartitions}} for
transforming {{Dataset}} . Does this satisfy your requirements ?
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset
> Developers API for Transformers beyond UnaryTransformer
> ---
>
> Key: SPARK-27249
> URL: https://issues.apache.org/jira/browse/SPARK-27249
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Affects Versions: 3.0.0
>Reporter: Everett Rush
>Priority: Minor
> Labels: starter
> Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> It would be nice to have a developers' API for dataset transformations that
> need more than one column from a row (ie UnaryTransformer inputs one column
> and outputs one column) or that contain objects too expensive to initialize
> repeatedly in a UDF such as a database connection.
>
> Design:
> Abstract class PartitionTransformer extends Transformer and defines the
> partition transformation function as Iterator[Row] => Iterator[Row]
> NB: This parallels the UnaryTransformer createTransformFunc method
>
> When developers subclass this transformer, they can provide their own schema
> for the output Row in which case the PartitionTransformer creates a row
> encoder and executes the transformation. Alternatively the developer can set
> output Datatype and output col name. Then the PartitionTransformer class will
> create a new schema, a row encoder, and execute the transformation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Afshartous updated SPARK-27249: Attachment: Screen Shot 2020-01-17 at 4.20.57 PM.png > Developers API for Transformers beyond UnaryTransformer > --- > > Key: SPARK-27249 > URL: https://issues.apache.org/jira/browse/SPARK-27249 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.0.0 >Reporter: Everett Rush >Priority: Minor > Labels: starter > Attachments: Screen Shot 2020-01-17 at 4.20.57 PM.png > > Original Estimate: 96h > Remaining Estimate: 96h > > It would be nice to have a developers' API for dataset transformations that > need more than one column from a row (ie UnaryTransformer inputs one column > and outputs one column) or that contain objects too expensive to initialize > repeatedly in a UDF such as a database connection. > > Design: > Abstract class PartitionTransformer extends Transformer and defines the > partition transformation function as Iterator[Row] => Iterator[Row] > NB: This parallels the UnaryTransformer createTransformFunc method > > When developers subclass this transformer, they can provide their own schema > for the output Row in which case the PartitionTransformer creates a row > encoder and executes the transformation. Alternatively the developer can set > output Datatype and output col name. Then the PartitionTransformer class will > create a new schema, a row encoder, and execute the transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018105#comment-17018105 ] Nick Afshartous commented on SPARK-27249: - Thanks Everett, and can someone with permission assign this ticket to me. > Developers API for Transformers beyond UnaryTransformer > --- > > Key: SPARK-27249 > URL: https://issues.apache.org/jira/browse/SPARK-27249 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.0.0 >Reporter: Everett Rush >Priority: Minor > Labels: starter > Original Estimate: 96h > Remaining Estimate: 96h > > It would be nice to have a developers' API for dataset transformations that > need more than one column from a row (ie UnaryTransformer inputs one column > and outputs one column) or that contain objects too expensive to initialize > repeatedly in a UDF such as a database connection. > > Design: > Abstract class PartitionTransformer extends Transformer and defines the > partition transformation function as Iterator[Row] => Iterator[Row] > NB: This parallels the UnaryTransformer createTransformFunc method > > When developers subclass this transformer, they can provide their own schema > for the output Row in which case the PartitionTransformer creates a row > encoder and executes the transformation. Alternatively the developer can set > output Datatype and output col name. Then the PartitionTransformer class will > create a new schema, a row encoder, and execute the transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[
https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010756#comment-17010756
]
Nick Afshartous edited comment on SPARK-27249 at 1/9/20 7:49 PM:
-
I could try and look into this. Could someone validate that this feature is
still needed ?
[~enrush] It would also be helpful if you could provide a code example
illustrating how the {{PartitionTransformer}} would be used.
was (Author: nafshartous):
I could try and look into this. Could someone validate that this feature is
still needed ?
> Developers API for Transformers beyond UnaryTransformer
> ---
>
> Key: SPARK-27249
> URL: https://issues.apache.org/jira/browse/SPARK-27249
> Project: Spark
> Issue Type: New Feature
> Components: ML
>Affects Versions: 3.0.0
>Reporter: Everett Rush
>Priority: Minor
> Labels: starter
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> It would be nice to have a developers' API for dataset transformations that
> need more than one column from a row (ie UnaryTransformer inputs one column
> and outputs one column) or that contain objects too expensive to initialize
> repeatedly in a UDF such as a database connection.
>
> Design:
> Abstract class PartitionTransformer extends Transformer and defines the
> partition transformation function as Iterator[Row] => Iterator[Row]
> NB: This parallels the UnaryTransformer createTransformFunc method
>
> When developers subclass this transformer, they can provide their own schema
> for the output Row in which case the PartitionTransformer creates a row
> encoder and executes the transformation. Alternatively the developer can set
> output Datatype and output col name. Then the PartitionTransformer class will
> create a new schema, a row encoder, and execute the transformation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010756#comment-17010756 ] Nick Afshartous commented on SPARK-27249: - I could try and look into this. Could someone validate that this feature is still needed ? > Developers API for Transformers beyond UnaryTransformer > --- > > Key: SPARK-27249 > URL: https://issues.apache.org/jira/browse/SPARK-27249 > Project: Spark > Issue Type: New Feature > Components: ML >Affects Versions: 3.0.0 >Reporter: Everett Rush >Priority: Minor > Labels: starter > Original Estimate: 96h > Remaining Estimate: 96h > > It would be nice to have a developers' API for dataset transformations that > need more than one column from a row (ie UnaryTransformer inputs one column > and outputs one column) or that contain objects too expensive to initialize > repeatedly in a UDF such as a database connection. > > Design: > Abstract class PartitionTransformer extends Transformer and defines the > partition transformation function as Iterator[Row] => Iterator[Row] > NB: This parallels the UnaryTransformer createTransformFunc method > > When developers subclass this transformer, they can provide their own schema > for the output Row in which case the PartitionTransformer creates a row > encoder and executes the transformation. Alternatively the developer can set > output Datatype and output col name. Then the PartitionTransformer class will > create a new schema, a row encoder, and execute the transformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410334#comment-16410334
]
Nick Afshartous commented on SPARK-19767:
-
Running the full doc build using {{jekyll build}} I get the error
{code:java}
OSError: [Errno 13] Permission denied: '/usr/lib64/python2.7/site-packages/yaml'
{code}
Can someone advise.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: DStreams
>Affects Versions: 2.1.0
>Reporter: Nick Afshartous
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403702#comment-16403702
]
Nick Afshartous commented on SPARK-19767:
-
Sounds good [~c...@koeninger.org]. I'll close the current PR and submit a new
one.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: DStreams
>Affects Versions: 2.1.0
>Reporter: Nick Afshartous
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900216#comment-15900216
]
Nick Afshartous commented on SPARK-19767:
-
Missed that one, thanks.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick Afshartous
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900184#comment-15900184
]
Nick Afshartous commented on SPARK-19767:
-
Yes, I completed the steps in the Prerequisites section of
https://github.com/apache/spark/blob/master/docs/README.md
and got the same error about unknown tag {{include_example}} on two different
computers (Linux and OSX).
Looks like {{include_example}} is local in
{{./docs/_plugins/include_example.rb}}, so maybe this is some kind of path
issue where its not finding the local file ?
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick Afshartous
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897965#comment-15897965
]
Nick Afshartous commented on SPARK-19767:
-
I believe all the dependencies are installed and got this error about a missing
tag {{'include_example'}}
{code}
SKIP_API=1 jekyll build
Configuration file: none
Source: /home/nafshartous/Projects/spark
Destination: /home/nafshartous/Projects/spark/_site
Incremental build: disabled. Enable with --incremental
Generating...
Build Warning: Layout 'global' requested in docs/api.md does not exist.
Build Warning: Layout 'global' requested in docs/building-spark.md does
not exist.
Build Warning: Layout 'global' requested in docs/cluster-overview.md does
not exist.
Build Warning: Layout 'global' requested in docs/configuration.md does not
exist.
Build Warning: Layout 'global' requested in docs/contributing-to-spark.md
does not exist.
Build Warning: Layout 'global' requested in docs/ec2-scripts.md does not
exist.
Liquid Exception: Liquid syntax error (line 581): Unknown tag
'include_example' in docs/graphx-programming-guide.md
jekyll 3.4.0 | Error: Liquid syntax error (line 581): Unknown tag
'include_example'
{code}
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick Afshartous
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888572#comment-15888572
]
Nick Afshartous edited comment on SPARK-19767 at 3/1/17 10:15 PM:
--
Yes the code examples on the Integration Page are current. The issue with the
linked API pages looks like more than incompleteness because the package name
{code}org.apache.spark.streaming.kafka{code} should be
{code}org.apache.spark.streaming.kafka10{code}.
I'd be happy to help. Tried to build the doc running "jekyll build" from the
docs dir and got the error below. Is this target broken or my env ?
{code}
[info] Note: Custom tags that could override future standard tags: @todo,
@note, @tparam, @constructor, @groupname, @example, @group. To avoid potential
overrides, use at least one period character (.) in custom tag names.
[info] Note: Custom tags that were not seen: @todo, @tparam, @constructor,
@groupname, @group
[info] 1 error
[info] 100 warnings
[error] (spark/javaunidoc:doc) javadoc returned nonzero exit code
[error] Total time: 198 s, completed Feb 28, 2017 11:56:20 AM
jekyll 3.4.0 | Error: Unidoc generation failed
{code}
was (Author: nafshartous):
Yes the code examples on the Integration Page are current. The issue with the
linked API pages looks like more than incompleteness because the package names
name {code}org.apache.spark.streaming.kafka{code} should be
{code}org.apache.spark.streaming.kafka10{code}.
I'd be happy to help. Tried to build the doc running "jekyll build" from the
docs dir and got the error below. Is this target broken or my env ?
{code}
[info] Note: Custom tags that could override future standard tags: @todo,
@note, @tparam, @constructor, @groupname, @example, @group. To avoid potential
overrides, use at least one period character (.) in custom tag names.
[info] Note: Custom tags that were not seen: @todo, @tparam, @constructor,
@groupname, @group
[info] 1 error
[info] 100 warnings
[error] (spark/javaunidoc:doc) javadoc returned nonzero exit code
[error] Total time: 198 s, completed Feb 28, 2017 11:56:20 AM
jekyll 3.4.0 | Error: Unidoc generation failed
{code}
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick Afshartous
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
