[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster
[
https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16555403#comment-16555403
]
nick commented on SPARK-21063:
--
[~paulstaab]
It does work when both registering the dialect and using fetchsize, but how can
we get all the data (don't set "fetchsize") ?
> Spark return an empty result from remote hadoop cluster
> ---
>
> Key: SPARK-21063
> URL: https://issues.apache.org/jira/browse/SPARK-21063
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Peter Bykov
>Priority: Major
>
> Spark returning empty result from when querying remote hadoop cluster.
> All firewall settings removed.
> Querying using JDBC working properly using hive-jdbc driver from version 1.1.1
> Code snippet is:
> {code:java}
> val spark = SparkSession.builder
> .appName("RemoteSparkTest")
> .master("local")
> .getOrCreate()
> val df = spark.read
> .option("url", "jdbc:hive2://remote.hive.local:1/default")
> .option("user", "user")
> .option("password", "pass")
> .option("dbtable", "test_table")
> .option("driver", "org.apache.hive.jdbc.HiveDriver")
> .format("jdbc")
> .load()
>
> df.show()
> {code}
> Result:
> {noformat}
> +---+
> |test_table.test_col|
> +---+
> +---+
> {noformat}
> All manipulations like:
> {code:java}
> df.select(*).show()
> {code}
> returns empty result too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890200#comment-15890200
]
Nick commented on SPARK-19767:
--
Perhaps in the interim there could be a note on the 0.10 Integration page about
the missing API doc and how to build it. That would save people from fishing
around for doc that's not there.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888932#comment-15888932
]
Nick edited comment on SPARK-19767 at 2/28/17 9:58 PM:
---
{{SKIP_API=1 jekyll build}} yields the error
{code}
Dependency Error: Yikes! It looks like you don't have
/home/nafshartous/Projects/spark/docs/_plugins/include_example.rb or one of its
dependencies installed. In order to use Jekyll as currently configured, you'll
need to install this gem. The full error message from Ruby is: 'cannot load
such file -- pygments' If you run into trouble, you can find helpful resources
at https://jekyllrb.com/help/!
jekyll 3.4.0 | Error:
/home/nafshartous/Projects/spark/docs/_plugins/include_example.rb
{code}
The file {{_plugins/include_example.rb}} is present.
was (Author: nafshartous):
`SKIP_API=1 jekyll build` yields the error
{code}
Dependency Error: Yikes! It looks like you don't have
/home/nafshartous/Projects/spark/docs/_plugins/include_example.rb or one of its
dependencies installed. In order to use Jekyll as currently configured, you'll
need to install this gem. The full error message from Ruby is: 'cannot load
such file -- pygments' If you run into trouble, you can find helpful resources
at https://jekyllrb.com/help/!
jekyll 3.4.0 | Error:
/home/nafshartous/Projects/spark/docs/_plugins/include_example.rb
{code}
The file `_plugins/include_example.rb` is present.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888932#comment-15888932
]
Nick commented on SPARK-19767:
--
`SKIP_API=1 jekyll build` yields the error
{code}
Dependency Error: Yikes! It looks like you don't have
/home/nafshartous/Projects/spark/docs/_plugins/include_example.rb or one of its
dependencies installed. In order to use Jekyll as currently configured, you'll
need to install this gem. The full error message from Ruby is: 'cannot load
such file -- pygments' If you run into trouble, you can find helpful resources
at https://jekyllrb.com/help/!
jekyll 3.4.0 | Error:
/home/nafshartous/Projects/spark/docs/_plugins/include_example.rb
{code}
The file `_plugins/include_example.rb` is present.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888572#comment-15888572
]
Nick commented on SPARK-19767:
--
Yes the code examples on the Integration Page are current. The issue with the
linked API pages looks like more than incompleteness because the package names
name {code}org.apache.spark.streaming.kafka{code} should be
{code}org.apache.spark.streaming.kafka10{code}.
I'd be happy to help. Tried to build the doc running "jekyll build" from the
docs dir and got the error below. Is this target broken or my env ?
{code}
[info] Note: Custom tags that could override future standard tags: @todo,
@note, @tparam, @constructor, @groupname, @example, @group. To avoid potential
overrides, use at least one period character (.) in custom tag names.
[info] Note: Custom tags that were not seen: @todo, @tparam, @constructor,
@groupname, @group
[info] 1 error
[info] 100 warnings
[error] (spark/javaunidoc:doc) javadoc returned nonzero exit code
[error] Total time: 198 s, completed Feb 28, 2017 11:56:20 AM
jekyll 3.4.0 | Error: Unidoc generation failed
{code}
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888205#comment-15888205
]
Nick commented on SPARK-19767:
--
I'm looking for a code example showing how to use the Java API to start
streaming from specific offsets. Thanks for any pointers to code or doc.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick updated SPARK-19767:
-
Description:
The API docs linked from the Spark Kafka 0.10 Integration page are not current.
For instance, on the page
https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
the code examples show the new API (i.e. class ConsumerStrategies). However,
following the links
API Docs --> (Scala | Java)
lead to API pages that do not have class ConsumerStrategies) . The API doc
package names also have {code}streaming.kafka{code} as opposed to
{code}streaming.kafka10{code}
as in the code examples on streaming-kafka-0-10-integration.html.
was:
The API docs linked from the Spark Kafka 0.10 Integration page are not current.
For instance, on the page
https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
the code examples show the new API (i.e. class ConsumerStrategies). However,
following the links
API Docs --> (Scala | Java)
lead to API pages that do not have class ConsumerStrategies) . The API doc
package names also have {code}streaming.kafka{code} as opposed to
{code}streaming.kafka10{code}.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}
> as in the code examples on streaming-kafka-0-10-integration.html.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
[
https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick updated SPARK-19767:
-
Description:
The API docs linked from the Spark Kafka 0.10 Integration page are not current.
For instance, on the page
https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
the code examples show the new API (i.e. class ConsumerStrategies). However,
following the links
API Docs --> (Scala | Java)
lead to API pages that do not have class ConsumerStrategies) . The API doc
package names also have {code}streaming.kafka{code} as opposed to
{code}streaming.kafka10{code}.
was:
The API docs linked from the Spark Kafka 0.10 Integration page are not current.
For instance, on the page
https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
the code examples show the new API (i.e. class ConsumerStrategies). However,
following the links
API Docs --> (Scala | Java)
leads to API pages that do not have class ConsumerStrategies) . The API doc
package names also have {code}streaming.kafka{code} as opposed to
{code}streaming.kafka10{code}.
> API Doc pages for Streaming with Kafka 0.10 not current
> ---
>
> Key: SPARK-19767
> URL: https://issues.apache.org/jira/browse/SPARK-19767
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
>Affects Versions: 2.1.0
>Reporter: Nick
>Priority: Minor
>
> The API docs linked from the Spark Kafka 0.10 Integration page are not
> current. For instance, on the page
>https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> the code examples show the new API (i.e. class ConsumerStrategies). However,
> following the links
> API Docs --> (Scala | Java)
> lead to API pages that do not have class ConsumerStrategies) . The API doc
> package names also have {code}streaming.kafka{code} as opposed to
> {code}streaming.kafka10{code}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current
Nick created SPARK-19767:
Summary: API Doc pages for Streaming with Kafka 0.10 not current
Key: SPARK-19767
URL: https://issues.apache.org/jira/browse/SPARK-19767
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Nick
Priority: Minor
The API docs linked from the Spark Kafka 0.10 Integration page are not current.
For instance, on the page
https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
the code examples show the new API (i.e. class ConsumerStrategies). However,
following the links
API Docs --> (Scala | Java)
leads to API pages that do not have class ConsumerStrategies) . The API doc
package names also have {code}streaming.kafka{code} as opposed to
{code}streaming.kafka10{code}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-6500) Scala code example in README.md does not compile
[
https://issues.apache.org/jira/browse/SPARK-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick closed SPARK-6500.
---
> Scala code example in README.md does not compile
>
>
> Key: SPARK-6500
> URL: https://issues.apache.org/jira/browse/SPARK-6500
> Project: Spark
> Issue Type: Bug
>Reporter: Nick
>Priority: Trivial
>
> I just downloaded and installed Spark 1.3.
> Inside README.md there is this example
>
> {code}
> And run the following command, which should also return 1000:
> >>> sc.parallelize(range(1000)).count()
> {code}
> which does not compile
> {code}
> :22: error: not found: value range
> {code}
> This example does work
> {code}
> sc.parallelize(1 to 1000).count()
> {code}
> I'd be happy to create a pull request if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-6500) Scala code example in README.md does not compile
[
https://issues.apache.org/jira/browse/SPARK-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick updated SPARK-6500:
Description:
I just downloaded and installed Spark 1.3.
Inside README.md there is this example
{code}
And run the following command, which should also return 1000:
>>> sc.parallelize(range(1000)).count()
{code}
which does not compile
{code}
:22: error: not found: value range
{code}
This example does work
{code}
sc.parallelize(1 to 1000).count()
{code}
I'd be happy to create a pull request if necessary.
was:
I just downloaded and installed Spark 1.3.
Inside README.md there is this example
{code}
And run the following command, which should also return 1000:
>>> sc.parallelize(range(1000)).count()
{code}
which does not compile
{code}
:22: error: not found: value range
{code}
This example does work
{code}
sc.parallelize(1 to 1000).count()
{code}
> Scala code example in README.md does not compile
>
>
> Key: SPARK-6500
> URL: https://issues.apache.org/jira/browse/SPARK-6500
> Project: Spark
> Issue Type: Bug
>Reporter: Nick
>Priority: Trivial
>
> I just downloaded and installed Spark 1.3.
> Inside README.md there is this example
>
> {code}
> And run the following command, which should also return 1000:
> >>> sc.parallelize(range(1000)).count()
> {code}
> which does not compile
> {code}
> :22: error: not found: value range
> {code}
> This example does work
> {code}
> sc.parallelize(1 to 1000).count()
> {code}
> I'd be happy to create a pull request if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6500) Scala code example in README.md does not compile
Nick created SPARK-6500:
---
Summary: Scala code example in README.md does not compile
Key: SPARK-6500
URL: https://issues.apache.org/jira/browse/SPARK-6500
Project: Spark
Issue Type: Bug
Reporter: Nick
Priority: Trivial
I just downloaded and installed Spark 1.3.
Inside README.md there is this example
{code}
And run the following command, which should also return 1000:
>>> sc.parallelize(range(1000)).count()
{code}
which does not compile
{code}
:22: error: not found: value range
{code}
This example does work
{code}
sc.parallelize(1 to 1000).count()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
