Mailing lists matching spark.apache.org

commits spark.apache.org
dev spark.apache.org
issues spark.apache.org
reviews spark.apache.org
user spark.apache.org


[GitHub] [spark] Ngone51 edited a comment on pull request #28072: [SPARK-31314][CORE] Revert SPARK-29285 to fix shuffle regression caused by creating temporary file eagerly

2021-08-05 Thread GitBox
`spark.yarn.executor.launch.excludeOnFailure.enabled`) to ban such problematic nodes. You can check the configuration [here](https://spark.apache.org/docs/latest/configuration.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] sarutak commented on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala

2021-10-30 Thread GitBox
is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] sarutak edited a comment on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala

2021-11-01 Thread GitBox
layout is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] sarutak edited a comment on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala

2021-11-02 Thread GitBox
layout is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] gaborgsomogyi commented on issue #23348: [SPARK-25857][core] Add developer documentation regarding delegation tokens.

2019-01-03 Thread GitBox
about considering to put this in somewhere like https://spark.apache.org/docs/latest/running-on-yarn.html after refining it? We have Kerberos chapter https://spark.apache.org/docs/latest/running-on-yarn.html#kerberos` This is an auto

Re: [PR] [SPARK-47912][SQL] Infer serde class from format classes [spark]

2024-04-22 Thread via GitHub
ing process. This doesn't seem to be relevant to the current PR, you can get more help from https://spark.apache.org/community.html. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] pan3793 commented on pull request #42493: [SPARK-44811][BUILD] Upgrade Guava to 32+

2023-08-15 Thread via GitHub
oved in Spark 4?](https://www.mail-archive.com/dev@spark.apache.org/msg30708.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: rev

Re: [PR] [SPARK-46111][DOCS][PYTHON] Add copyright to the PySpark official documentation. [spark]

2023-11-26 Thread via GitHub
/licenses/LICENSE-2.0";>Apache License, Version 2.0. Review Comment: FYI: I follow this copyright format from [Apache Spark official web page](https://spark.apache.org/). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-46351][SQL] Require an error class in `AnalysisException` [spark]

2023-12-11 Thread via GitHub
block" ] }, + "_LEGACY_ERROR_TEMP_3050" : { +"message" : [ + "Cannot modify the value of a static config: " +] Review Comment: Can we use this? - https://spark.apache.org/docs/latest/sql-error-conditions.html#cannot_modify_config -- This is

[GitHub] [spark] zhengruifeng commented on pull request #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics

2022-06-20 Thread GitBox
zhengruifeng commented on PR #36926: URL: https://github.com/apache/spark/pull/36926#issuecomment-1161256434 since it is only exposed in the `.mllib` side, and it is not listed in the [official documents ](https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html#ranking

[GitHub] [spark] dongjoon-hyun commented on pull request #36069: [SPARK-38767][SQL] Support `ignoreCorruptFiles` and `ignoreMissingFiles` in Data Source options

2022-07-20 Thread GitBox
dongjoon-hyun commented on PR #36069: URL: https://github.com/apache/spark/pull/36069#issuecomment-1190997582 FYI, Apache Spark 3.4 code freeze was March 15th, 2022. This patch simply arrived one month later after code freeze deadline. - https://spark.apache.org/versioning

Re: [PR] [SPARK-45273][CORE][UI] Support for set the access host in http header [spark]

2023-10-09 Thread via GitHub
srowen commented on PR #43169: URL: https://github.com/apache/spark/pull/43169#issuecomment-1752893775 How does this arise? Use priv...@spark.apache.org if needed. I am not clear what attack you have in mind or whether it can affect spark, so, no this would not be useful unless there'

Re: Spark Language / Data Base Question

2015-06-25 Thread pandees waran
ng else ? > > > Please help me > > > Thanks > > > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >

Fwd: use S3-Compatible Storage with spark

2015-07-19 Thread Schmirr Wurst
amazon, is there >> a way I can specify the host somewhere ? >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > -

Re: Save DataFrame to HBase

2016-04-24 Thread Benjamin Kim
t;> wrote: > >> Has anyone found an easy way to save a DataFrame into HBase? >> >> Thanks, >> Ben >> >> >> ----- >> To unsubscribe, e-mail: user-unsubscr...@spar

Re: Save DataFrame to HBase

2016-04-27 Thread Benjamin Kim
to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> > >

Re: broadcast variable not picked up

2016-05-16 Thread Davies Liu
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > ----- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Allowing parallelism in spark local mode

2016-02-12 Thread Silvio Fiorito
You’ll want to setup the FAIR scheduler as described here: https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application From: yael aharon mailto:yael.aharo...@gmail.com>> Date: Friday, February 12, 2016 at 2:00 PM To: "user@spark.apache.org&

Re: How to delete a record from parquet files using dataframes

2016-02-24 Thread Cheng Lian
. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional

Re: Dataframe constructor

2015-11-23 Thread Fengdong Yu
> > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-constructor-tp25455.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > ---

Re: Where is Redgate's HDFS explorer?

2015-08-29 Thread Dino Fancellu
>> http://apache-spark-user-list.1001560.n3.nabble.com/Where-is-Redgate-s-HDFS-explorer-tp24431.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> ----

Re: RDD String foreach println

2015-02-24 Thread Sean Owen
as why 1) behaves differently on job submit and shell? > > Best, > Patcharee > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > Fo

Re: Effects of persist(XYZ_2)

2015-02-25 Thread Marius Soutier
ding network transfers? Assuming I’m doing joins or other >> shuffle operations. >> >> Thanks >> >> >> ----- >> To unsubscribe, e-mail: user-u

Re: Upgrade to Spark 1.2.1 using Guava

2015-02-27 Thread Pat Ferrel
rk. -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional

Re: multiple sparkcontexts and streamingcontexts

2015-03-02 Thread Sean Owen
contexts-tp21876.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >

RE: Is SQLContext thread-safe?

2015-03-02 Thread Haopu Wang
text to do Select operations in different threads at the same time? Thank you very much! - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.

Re: [ANNOUNCE] Announcing Spark 1.3!

2015-03-13 Thread Kushal Datta
ot on-list). > > Thanks to everyone who helped work on this release! > > [1] http://spark.apache.org/releases/spark-release-1-3-0.html > [2] http://spark.apache.org/downloads.html > > - > To unsubscribe, e-ma

Re: Spark Job History Server

2015-03-20 Thread Zhan Zhang
r > > What class is really needed? How to fix it? > > Br, > Patcharee > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >

Re: How to merge two dataframes with same schema

2015-04-22 Thread Peter Rudenko
mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To

Re: Mapping with extra arguments

2014-08-20 Thread zzl
in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Mapping-with-extra-arguments-tp12541.html > Sent from the Apache Spark User List mailing list archive at Nabble.com > (http://Nabble.com). > > -

Re: Spark Screencast doesn't show in Chrome on OS X

2014-08-25 Thread Michael Hausenblas
> https://spark.apache.org/screencasts/1-first-steps-with-spark.html > > The embedded YouTube video shows up in Safari on OS X but not in Chrome. I’m using Chrome 36.0.1985.143 on MacOS 10.9.4 and it it works like a charm for me. Cheers, Michael -- Michael Hausenbla

Re: nested rdd operation

2014-09-10 Thread Sean Owen
ter exception , > can somebody explain why i cannot have a nested rdd operation ? > > --pavlos > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apa

RE: Cassandra connector

2014-09-10 Thread Wade Wilkins
pache-spark-user-list.1001560.n3.nabble.com/Cassandra-connector-tp13896p13897.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For addit

Re: Spark on Raspberry Pi?

2014-09-11 Thread Chanwit Kaewkasi
> Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For a

Re: Accuracy hit in classification with Spark

2014-09-15 Thread Xiangrui Meng
html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >

Re: Stable spark streaming app

2014-09-17 Thread Soumitra Kumar
er of nodes, events per second, broad stream processing workflow, config highlights etc? Thanks, Tim - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-ma

Re: K-means faster on Mahout then on Spark

2014-09-25 Thread Xiangrui Meng
List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsu

Re: SparkContext.stop() ?

2014-10-31 Thread Marcelo Vanzin
ntext-stop-tp17826.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For addit

Re: Matrix multiplication in spark

2014-11-05 Thread Xiangrui Meng
2562p18164.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional co

Re: saveAsParquetFile throwing exception

2014-11-14 Thread Cheng Lian
8929p18935.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.

Re: spark-sql broken

2014-11-22 Thread Cheng Lian
r-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Unable to generate assembly jar which includes jdbc-thrift server

2014-11-26 Thread Cheng Lian
st mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org --

Re: Elastic allocation(spark.dynamicAllocation.enabled) results in task never being executed.

2015-01-04 Thread Tsuyoshi Ozawa
ults-in-task-never-being-executed-tp18969p20957.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional comma

Re: acquire and give back resources dynamically

2014-08-16 Thread fireflyc
http://spark.apache.org/docs/latest/running-on-yarn.html Spark just a Yarn application > 在 2014年8月14日,11:12,牛兆捷 写道: > > Dear all: > > Does spark can acquire resources from and give back resources to > YARN dynamically ? > > >

[jira] [Created] (SPARK-32191) Migration Guide

2020-07-06 Thread Hyukjin Kwon (Jira)
: Documentation, PySpark Affects Versions: 3.1.0 Reporter: Hyukjin Kwon Port http://spark.apache.org/docs/latest/pyspark-migration-guide.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-09 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20188 Actually in R setCheckpointDir method is not attached to the SparkContext; I’d leave it as “not set” or “not set in the session” https://spark.apache.org/docs/latest/api/R

[GitHub] spark issue #21207: SPARK-24136: Fix MemoryStreamDataReader.next to skip sle...

2018-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21207 @arunmahadevan, not a big deal but mind if I ask to fix the PR title to `[SPARK-24136][SS] blabla`? It's actually encouraged in the guide - https://spark.apache.org/contributing

[GitHub] spark issue #21767: SPARK-24804 There are duplicate words in the test title ...

2018-07-18 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21767 yeah, please avoid PRs that are this trivial, it's just not worth the overhead. But I merged it this time. Also please read https://spark.apache.org/contributing

[GitHub] spark issue #21828: Update regression.py

2018-07-22 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21828 @woodthom2, if you have some plans to update this PR quite soon, please see https://spark.apache.org/contributing.html and proceed. Otherwise, I would suggest to leave this closed so that

[GitHub] spark issue #19714: [SPARK-22489][SQL] Shouldn't change broadcast join build...

2017-11-30 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19714 LGTM Thanks! Merged to master. Could you submit a follow-up PR to document the behavior changes in migration section of Spark SQL? https://spark.apache.org/docs/latest/sql

[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...

2017-09-26 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19343 @squito Thank you! Instead of changing the source codes, could we just update the document https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#hive-tables ? This might be

Re: Supported Hadoop versions for Spark 3.3

2022-08-15 Thread pengyh
adoop” package from https://spark.apache.org/downloads.html? <https://spark.apache.org/downloads.html?> - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: udf StructField to JSON String

2016-03-11 Thread Tristan Nixon
Have you looked at DataFrame.write.json( path )? https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriter > On Mar 11, 2016, at 7:15 AM, Caires Vinicius wrote: > > I have one DataFrame with nested StructField and I want to convert to JSON > S

Re: I want to unsubscribe

2016-04-05 Thread Jakob Odersky
to unsubscribe, send an email to user-unsubscr...@spark.apache.org On Tue, Apr 5, 2016 at 4:50 PM, Ranjana Rajendran wrote: > I get to see the threads in the public mailing list. I don;t want so many > messages in my inbox. I want to unsub

RE: unsubscribe

2014-03-11 Thread Kapil Malik
Ohh ! I thought you're unsubscribing :) Kapil Malik | kma...@adobe.com | 33430 / 8800836581 -Original Message- From: Matei Zaharia [mailto:matei.zaha...@gmail.com] Sent: 12 March 2014 00:51 To: user@spark.apache.org Subject: Re: unsubscribe To unsubscribe from this list, p

Re: Sample date_trunc error for webpage (https://spark.apache.org/docs/2.3.0/api/sql/#date_trunc )

2019-07-07 Thread Sean Owen
gt;> >> On Sun, Jul 7, 2019, 2:22 PM Chris Lambertus wrote: >> >>> Spark, >>> >>> We received this message. I have not ACKd it. >>> >>> -Chris >>> INFRA >>> >>> >>> Begin forwarded message: >&

Re: May we merge into branch-1.3 at this point?

2015-03-13 Thread Mridul Muralidharan
: > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=4aaf48d46d13129f0f9bdafd771dd80fe568a7dc > > Open season on 1.3 branch then... > > On Fri, Mar 13, 2015 at 4:20 PM, Nicholas Chammas > > wrote: > > Looks like the release is out: > > http://spa

Re: [VOTE] Release Apache Spark 1.5.1 (RC1)

2015-09-28 Thread Sean Owen
am wrote: > Hi Spark Developers, > > The Spark 1.5.1 documentation is already publicly accessible > (https://spark.apache.org/docs/latest/index.html) but the release is not. Is > it intentional? > > Best Regards, > > Jerry > > On Mon, Sep 28, 2015 at 9:21

Re: [ANNOUNCE] Announcing Spark 1.5.1

2015-10-11 Thread Sean Owen
t; >> Spark 1.5.1 is a maintenance release containing stability fixes. This >> release is based on the branch-1.5 maintenance branch of Spark. We >> *strongly recommend* all 1.5.0 users to upgrade to this release. >> >> The full list of bug fixes is here: http://s.apache.org/spa

Updating docs for running on Mesos

2014-05-15 Thread Andrew Ash
es - (same as current) Running Alongside Hadoop - (trim this down) Does that work for people? Thanks! Andrew PS Basically all the same: http://spark.apache.org/docs/0.6.0/running-on-mesos.html http://spark.apache.org/docs/0.6.2/running-on-mesos.html http://spark.apache.org/docs/0.7.3/

Re: Spark spilling location

2014-09-18 Thread Patrick Wendell
nce, ~/temp > instead of /tmp. Will this be the location where all data will be spilled > to? I assume it is, based on the description of spark.local.dir at > https://spark.apache.org/docs/latest/configuration.html: > "Directory to use for "scratch" space in Spark, incl

Re: Question about SparkSQL and Hive-on-Spark

2014-09-23 Thread Will Benton
gt; From: "Yi Tian" > To: dev@spark.apache.org > Sent: Tuesday, September 23, 2014 2:47:17 AM > Subject: Question about SparkSQL and Hive-on-Spark > > Hi all, > > I have some questions about the SparkSQL and Hive-on-Spark > > Will SparkSQL support all the

Fwd: [ANNOUNCE] Announcing Apache Spark 2.1.0

2016-12-29 Thread MrAsanjar .
OUNCE] Announcing Apache Spark 2.1.0 To: "d...@spark.apache.org" , user < u...@spark.apache.org> Hi all, Apache Spark 2.1.0 is the second release of Spark 2.x line. This release makes significant strides in the production readiness of Structured Streaming, with added support for event t

[jira] [Updated] (SPARK-18279) ML programming guide should have R examples

2016-11-09 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18279: -- Description: http://spark.apache.org/docs/latest/ml-classification-regression.html for

[jira] [Resolved] (SPARK-25082) Documentation for Spark Function expm1 is incomplete

2018-08-15 Thread Hyukjin Kwon (JIRA)
For: 2.4.0 > > > The documentation for the function expm1 that takes in a string > public static  > [Column|https://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/sql/Column.html] >  expm1(String columnName) > ([https://spark.apache.org/docs/2.3.1/api/java/org/apache/spar

[jira] [Commented] (SPARK-19445) Please remove tylerchap...@yahoo-inc.com subscription from u...@spark.apache.org

2017-02-03 Thread Sean Owen (JIRA)
east I am not sure who the admin is. It is a question for dev@ not JIRA > Please remove tylerchap...@yahoo-inc.com subscription from > u...@spark.apache.org > > > Key: SPARK-19445 >

[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example

2018-10-21 Thread Dongjoon Hyun (JIRA)
iven input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/de

[jira] [Commented] (SPARK-25795) Fix CSV SparkR SQL Example

2018-10-22 Thread Apache Spark (JIRA)
k/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/sql-programming-guide.html#manually-specifying-options > - > http://spark.apache.org/docs/2.3.2/sql-programming-guide.html#manually-specifying-options > - >

[jira] [Commented] (SPARK-25795) Fix CSV SparkR SQL Example

2018-10-22 Thread Apache Spark (JIRA)
k/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/sql-programming-guide.html#manually-specifying-options > - > http://spark.apache.org/docs/2.3.2/sql-programming-guide.html#manually-specifying-options > - >

[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example

2018-10-22 Thread Dongjoon Hyun (JIRA)
ot resolve '`name`' > given input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R > - >

[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example

2018-10-28 Thread Xiao Li (JIRA)
sql.AnalysisException: cannot resolve '`name`' > given input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R &

[jira] [Updated] (SPARK-17794) 2.0.1 not in maven central repo?

2016-10-05 Thread Harrison Kiang (JIRA)
Put the following into pom.xml as shown here: https://spark.apache.org/downloads.html {code:java} org.apache.spark spark-core_2.11 2.0.1 {code} Version 2.0.1 does not seem to exist in the Central Repository: https://repo1.maven.org/maven2/org/apache/spark/spark

[jira] [Comment Edited] (SPARK-3044) Create RSS feed for Spark News

2014-08-26 Thread Nicholas Chammas (JIRA)
0 PM: -- Hi Michael, I don't know if the site itself is open-source. We might need someone from Databricks to update it. [~pwendell], [~rxin], or [~matei] - Is it possible for contributors to contribute to the [main Spark site|http://spark.apache.org/]? was (Author: nchammas): Hi Mi

[jira] [Created] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

2020-06-24 Thread Micah Kornfield (Jira)
wants the operators pushed down [2]. Is the documentation for SupportsReportStatistics referring to something other than [2] or should it be updated?   [[1]https://spark.apache.org/docs/2.4.6/api/java/org/apache/spark/sql/sources/v2/reader/SupportsReportStatistics.html|https://spark.apache.org/

[jira] [Created] (SPARK-29830) PySpark.context.Sparkcontext.binaryfiles improved memory with buffer

2019-11-10 Thread Jira
) differs from the Scala implementation (see pyspark here: [https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/context.html#SparkContext.binaryFiles).   |https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/context.html#SparkContext.binaryFiles] In Scala, Spark returns a

[jira] [Commented] (SPARK-27972) Move SQL migration guide to the top level

2019-06-09 Thread Josh Rosen (JIRA)
hese guides from the spark.apache.org site's release notes pages! I've talked to a few folks who were unaware that these guides existed and linking them more prominently from the release page could really help with discoverability. > Move SQL migration guide

[jira] [Created] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome

2019-06-17 Thread Andrew Leverentz (JIRA)
following get redirected to a generic "root" package page: [https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Dataset.html] [https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset] Here's the URL that I get : [https://spark.apache.

[jira] [Updated] (SPARK-32186) User Guide - Debugging

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules

[jira] [Updated] (SPARK-32186) User Guide - Debugging

2020-09-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules

[jira] [Updated] (SPARK-7096) Java example for Streaming on site uses map instead of mapToPair

2015-04-23 Thread Edward Sargisson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Sargisson updated SPARK-7096: Description: https://spark.apache.org/docs/latest/streaming-programming-guide.html Here

[jira] [Commented] (SPARK-7255) spark.streaming.kafka.maxRetries not documented

2015-04-29 Thread Sean Owen (JIRA)
question, first. Some stuff is undocumented on purpose as it's experimental or not really to be changed by most users. I don't know about this one. It only exists for the direct, Kafka-based stream. I thought the config docs (which are at http://spark.apache.org/docs/latest/confi

[jira] [Comment Edited] (SPARK-12694) The detailed rest API documentation for each field is missing

2017-03-09 Thread Ehsun Behravesh (JIRA)
1 AM: - Does this still need someone to work on? I checked it at http://spark.apache.org/docs/latest/monitoring.html it seems to be fixed. Last commit is commit 4a4c3dc9ca10e52f7981b225ec44e97247986905 Author: jerryshao Date: Fri Jan 6 10:07:54 2017 -0600 was (Author: ehsun7b): Does

[jira] [Commented] (SPARK-20199) GradientBoostedTreesModel doesn't have Column Sampling Rate Paramenter

2017-04-03 Thread sudipto pal (JIRA)
fication can’t be done 5. Offset: present in H2O 6. Choice of distributions do not include Gamma, Tweedie, Poisson 7. Generates classes, not probabilities (they said later version will take care of this) http://spark.apache.org/docs/latest/api/scala/in

[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...

2014-09-11 Thread andrewor14
, including a programming -guide, on the project webpage at <http://spark.apache.org/documentation.html>. +guide, on the [project web page](http://spark.apache.org/documentation.html). This README file only contains basic setup instructions. ## Building Spark -Sp

[GitHub] [spark] dongjoon-hyun commented on pull request #34195: [SPARK-36939][PYTHON][DOCS] Add orphan migration page into list in PySpark documentation

2023-01-03 Thread GitBox
guide/pyspark_3.2_to_3.3' ``` In short, although SPARK-36939 landed at 3.2.1. `pyspark_3.2_to_3.3` doesn't show up in Apache Spark 3.2.1 documentation becuase it doesn't exist in branch-3.2. Let me revert this from branch-3.2. - https://spark.apache.org/docs/3.3.0/api/python/migrat

[GitHub] [spark] itholic commented on a diff in pull request #39820: [SPARK-42249][SQL] Refining html link for documentation in error messages.

2023-02-03 Thread via GitHub
t found an encoder of the type to Spark SQL internal representation. Consider to change the input type to one of supported at https://spark.apache.org/docs/latest/sql-ref-datatypes.html."; + "Not found an encoder of the type to Spark SQL internal representation. Consider to change th

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33435: [SPARK-31907][DOCS][SQL] Adding location of SQL API documentation

2021-07-20 Thread GitBox
documentation. + * at SQL API documentation (https://spark.apache.org/docs/latest/api/sql/index.html). Review comment: the problem is that the link became stale. For example, in Spark 3.0.0 release documentation, https://spark.apache.org/docs/latest/api/sql/index.html will point out the latest

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33435: [SPARK-31907][DOCS][SQL] Adding location of SQL API documentation

2021-07-20 Thread GitBox
documentation. + * at SQL API documentation (https://spark.apache.org/docs/latest/api/sql/index.html). Review comment: the problem is that the link becomes stale. For example, in Spark 3.0.0 release documentation, https://spark.apache.org/docs/latest/api/sql/index.html will point out the latest

[GitHub] [spark] dhruvildave opened a new pull request #33107: [SPARK-35909] Fix broken Python Links in docs/sql-getting-started.md

2021-06-27 Thread GitBox
dhruvildave opened a new pull request #33107: URL: https://github.com/apache/spark/pull/33107 ### What changes were proposed in this pull request? The hyperlinks in Python code blocks in [Spark SQL Guide - Getting Started](https://spark.apache.org/docs/latest/sql-getting

[GitHub] [spark] risinga commented on a change in pull request #34359: [SPARK-36986][SQL] Improving external schema management flexibility on DataSet and StructType

2021-12-01 Thread GitBox
], schema: StructType): DataFrame = { Review comment: @HyukjinKwon I've sent an email to d...@spark.apache.org, as well as dev-subscr...@spark.apache.org and I've got no response. I can find the subject on the mailing lists http://mail-archives.apache.org/mod_mbox/spark-dev/2

[GitHub] [spark] panbingkun commented on pull request #42425: [SPARK-44729][PYTHON][DOCS] Add canonical links to the PySpark docs page

2023-08-13 Thread via GitHub
panbingkun commented on PR #42425: URL: https://github.com/apache/spark/pull/42425#issuecomment-1676556063 > Here is an example of a documentation page for a specific version: https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html >

[PR] [SPARK-46464][DOC] Fix the scroll issue of tables when overflow [spark]

2023-12-20 Thread via GitHub
yaooqinn opened a new pull request, #44423: URL: https://github.com/apache/spark/pull/44423 ### What changes were proposed in this pull request? https://spark.apache.org/docs/3.4.1/running-on-kubernetes.html#spark-properties https://spark.apache.org/docs/latest

RE: Spark Version 3.0.1 Gui Display Query

2021-03-04 Thread Ranju Jain
Hi Attila, I checked the Section < https://spark.apache.org/docs/latest/monitoring.html#web-interfaces> and Web UI Page What document is saying that if I want to view information only for the duration of the application, then I do not need to generate the event logs and do not need

Re: number of partitions in join: Spark documentation misleading!

2015-06-16 Thread Davies Liu
yKey and join, the largest > number of partitions in a parent RDD." > https://spark.apache.org/docs/latest/configuration.html > > And the Partitioner.scala comments (line 51) state that: > "Unless spark.default.parallelism is set, the number of partitions will be > the same a

Re: Configuring Ports for Network Security

2016-03-02 Thread Jörn Franke
ote: > > Hi > > We want to use spark in a secure cluster with iptables enabled. > For this, we need a specific list of ports used by spark so that we can > whitelist them. > > From what I could learn from - > http://spark.apache.org/docs/latest/security.html#configur

Re: StateSpec raises error "missing arguments for method"

2016-03-27 Thread Cyril Scetbon
arguments for method mappingFunction; > follow this method with `_' if you want to treat it as a partially applied > function > val spec = StateSpec.function(mappingFunction) > ^ > I followed the current documentation

Re: Stop Spark Streaming Jobs

2016-08-02 Thread Pradeep
msung.com] > Sent: Wednesday, August 03, 2016 11:24 AM > To: 'Pradeep'; 'user@spark.apache.org' > Subject: RE: Stop Spark Streaming Jobs > > Hi. Paradeep > > > Did you mean, how to kill the job? > If yes, you should kill the driver and follow next.

RE: SparkR Error in sparkR.init(master=“local”) in RStudio

2015-10-06 Thread Sun, Rui
ny hint in the console output -Original Message- From: akhandeshi [mailto:ami.khande...@gmail.com] Sent: Tuesday, October 6, 2015 8:21 PM To: user@spark.apache.org Subject: Re: SparkR Error in sparkR.init(master=“local”) in RStudio I couldn't get this working... I have have JAVA_H

RE: SparkR Supported Types - Please add "bigint"

2015-07-23 Thread Sun, Rui
these two type name representations. -Original Message- From: Exie [mailto:tfind...@prodevelop.com.au] Sent: Friday, July 24, 2015 1:35 PM To: user@spark.apache.org Subject: Re: SparkR Supported Types - Please add "bigint" Interestingly, after more digging, df.printSchema() in raw

Re: how to integrate Apache Kafka with spark ?

2016-12-28 Thread Tushar Adeshara
Please see below links depending on version of Spark 2.x http://spark.apache.org/docs/latest/streaming-kafka-integration.html Spark Streaming + Kafka Integration Guide - Spark 2.0.2 ...<http://spark.apache.org/docs/latest/streaming-kafka-integration.html> spark.apache.org Spark Str

<    12   13   14   15   16   17   18   19   20   21   >