Mailing lists matching spark.apache.org

commits spark.apache.org
dev spark.apache.org
issues spark.apache.org
reviews spark.apache.org
user spark.apache.org

[GitHub] [spark] Ngone51 edited a comment on pull request #28072: [SPARK-31314][CORE] Revert SPARK-29285 to fix shuffle regression caused by creating temporary file eagerly

`spark.yarn.executor.launch.excludeOnFailure.enabled`) to ban such problematic nodes. You can check the configuration [here](https://spark.apache.org/docs/latest/configuration.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] sarutak commented on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala

is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] sarutak edited a comment on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala

layout is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] sarutak edited a comment on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala

layout is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] gaborgsomogyi commented on issue #23348: [SPARK-25857][core] Add developer documentation regarding delegation tokens.

about considering to put this in somewhere like https://spark.apache.org/docs/latest/running-on-yarn.html after refining it? We have Kerberos chapter https://spark.apache.org/docs/latest/running-on-yarn.html#kerberos` This is an auto

Re: [PR] [SPARK-47912][SQL] Infer serde class from format classes [spark]

ing process. This doesn't seem to be relevant to the current PR, you can get more help from https://spark.apache.org/community.html. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] pan3793 commented on pull request #42493: [SPARK-44811][BUILD] Upgrade Guava to 32+

oved in Spark 4?](https://www.mail-archive.com/dev@spark.apache.org/msg30708.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: rev

Re: [PR] [SPARK-46111][DOCS][PYTHON] Add copyright to the PySpark official documentation. [spark]

/licenses/LICENSE-2.0";>Apache License, Version 2.0. Review Comment: FYI: I follow this copyright format from [Apache Spark official web page](https://spark.apache.org/). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-46351][SQL] Require an error class in `AnalysisException` [spark]

block" ] }, + "_LEGACY_ERROR_TEMP_3050" : { +"message" : [ + "Cannot modify the value of a static config: " +] Review Comment: Can we use this? - https://spark.apache.org/docs/latest/sql-error-conditions.html#cannot_modify_config -- This is

[GitHub] [spark] zhengruifeng commented on pull request #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics

zhengruifeng commented on PR #36926: URL: https://github.com/apache/spark/pull/36926#issuecomment-1161256434 since it is only exposed in the `.mllib` side, and it is not listed in the [official documents ](https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html#ranking

[GitHub] [spark] dongjoon-hyun commented on pull request #36069: [SPARK-38767][SQL] Support `ignoreCorruptFiles` and `ignoreMissingFiles` in Data Source options

dongjoon-hyun commented on PR #36069: URL: https://github.com/apache/spark/pull/36069#issuecomment-1190997582 FYI, Apache Spark 3.4 code freeze was March 15th, 2022. This patch simply arrived one month later after code freeze deadline. - https://spark.apache.org/versioning

Re: [PR] [SPARK-45273][CORE][UI] Support for set the access host in http header [spark]

srowen commented on PR #43169: URL: https://github.com/apache/spark/pull/43169#issuecomment-1752893775 How does this arise? Use priv...@spark.apache.org if needed. I am not clear what attack you have in mind or whether it can affect spark, so, no this would not be useful unless there'

Re: Spark Language / Data Base Question

ng else ? > > > Please help me > > > Thanks > > > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >

Fwd: use S3-Compatible Storage with spark

amazon, is there >> a way I can specify the host somewhere ? >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > -

Re: Save DataFrame to HBase

t;> wrote: > >> Has anyone found an easy way to save a DataFrame into HBase? >> >> Thanks, >> Ben >> >> >> ----- >> To unsubscribe, e-mail: user-unsubscr...@spar

Re: Save DataFrame to HBase

to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> > >

Re: broadcast variable not picked up

> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > ----- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Allowing parallelism in spark local mode

You’ll want to setup the FAIR scheduler as described here: https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application From: yael aharon mailto:yael.aharo...@gmail.com>> Date: Friday, February 12, 2016 at 2:00 PM To: "user@spark.apache.org&

Re: How to delete a record from parquet files using dataframes

. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional

Re: Dataframe constructor

> > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-constructor-tp25455.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > ---

Re: Where is Redgate's HDFS explorer?

>> http://apache-spark-user-list.1001560.n3.nabble.com/Where-is-Redgate-s-HDFS-explorer-tp24431.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> ----

Re: RDD String foreach println

as why 1) behaves differently on job submit and shell? > > Best, > Patcharee > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > Fo

Re: Effects of persist(XYZ_2)

ding network transfers? Assuming I’m doing joins or other >> shuffle operations. >> >> Thanks >> >> >> ----- >> To unsubscribe, e-mail: user-u

Re: Upgrade to Spark 1.2.1 using Guava

rk. -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional

Re: multiple sparkcontexts and streamingcontexts

contexts-tp21876.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >

RE: Is SQLContext thread-safe?

text to do Select operations in different threads at the same time? Thank you very much! - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.

Re: [ANNOUNCE] Announcing Spark 1.3!

ot on-list). > > Thanks to everyone who helped work on this release! > > [1] http://spark.apache.org/releases/spark-release-1-3-0.html > [2] http://spark.apache.org/downloads.html > > - > To unsubscribe, e-ma

Re: Spark Job History Server

r > > What class is really needed? How to fix it? > > Br, > Patcharee > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >

Re: How to merge two dataframes with same schema

mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To

Re: Mapping with extra arguments

in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Mapping-with-extra-arguments-tp12541.html > Sent from the Apache Spark User List mailing list archive at Nabble.com > (http://Nabble.com). > > -

Re: Spark Screencast doesn't show in Chrome on OS X

> https://spark.apache.org/screencasts/1-first-steps-with-spark.html > > The embedded YouTube video shows up in Safari on OS X but not in Chrome. I’m using Chrome 36.0.1985.143 on MacOS 10.9.4 and it it works like a charm for me. Cheers, Michael -- Michael Hausenbla

Re: nested rdd operation

ter exception , > can somebody explain why i cannot have a nested rdd operation ? > > --pavlos > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apa

RE: Cassandra connector

pache-spark-user-list.1001560.n3.nabble.com/Cassandra-connector-tp13896p13897.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For addit

Re: Spark on Raspberry Pi?

> Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For a

Re: Accuracy hit in classification with Spark

html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >

Re: Stable spark streaming app

er of nodes, events per second, broad stream processing workflow, config highlights etc? Thanks, Tim - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-ma

Re: K-means faster on Mahout then on Spark

List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsu

Re: SparkContext.stop() ?

ntext-stop-tp17826.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For addit

Re: Matrix multiplication in spark

2562p18164.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional co

Re: saveAsParquetFile throwing exception

8929p18935.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.

Re: spark-sql broken

r-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Unable to generate assembly jar which includes jdbc-thrift server

st mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org --

Re: Elastic allocation(spark.dynamicAllocation.enabled) results in task never being executed.

ults-in-task-never-being-executed-tp18969p20957.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional comma

Re: acquire and give back resources dynamically

http://spark.apache.org/docs/latest/running-on-yarn.html Spark just a Yarn application > 在 2014年8月14日，11:12，牛兆捷 写道： > > Dear all: > > Does spark can acquire resources from and give back resources to > YARN dynamically ? > > >

[jira] [Created] (SPARK-32191) Migration Guide

: Documentation, PySpark Affects Versions: 3.1.0 Reporter: Hyukjin Kwon Port http://spark.apache.org/docs/latest/pyspark-migration-guide.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20188 Actually in R setCheckpointDir method is not attached to the SparkContext; I’d leave it as “not set” or “not set in the session” https://spark.apache.org/docs/latest/api/R

[GitHub] spark issue #21207: SPARK-24136: Fix MemoryStreamDataReader.next to skip sle...

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21207 @arunmahadevan, not a big deal but mind if I ask to fix the PR title to `[SPARK-24136][SS] blabla`? It's actually encouraged in the guide - https://spark.apache.org/contributing

[GitHub] spark issue #21767: SPARK-24804 There are duplicate words in the test title ...

Github user srowen commented on the issue: https://github.com/apache/spark/pull/21767 yeah, please avoid PRs that are this trivial, it's just not worth the overhead. But I merged it this time. Also please read https://spark.apache.org/contributing

[GitHub] spark issue #21828: Update regression.py

Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21828 @woodthom2, if you have some plans to update this PR quite soon, please see https://spark.apache.org/contributing.html and proceed. Otherwise, I would suggest to leave this closed so that

[GitHub] spark issue #19714: [SPARK-22489][SQL] Shouldn't change broadcast join build...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19714 LGTM Thanks! Merged to master. Could you submit a follow-up PR to document the behavior changes in migration section of Spark SQL? https://spark.apache.org/docs/latest/sql

[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19343 @squito Thank you! Instead of changing the source codes, could we just update the document https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#hive-tables ? This might be

Re: Supported Hadoop versions for Spark 3.3

adoop” package from https://spark.apache.org/downloads.html? <https://spark.apache.org/downloads.html?> - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: udf StructField to JSON String

Have you looked at DataFrame.write.json( path )? https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriter > On Mar 11, 2016, at 7:15 AM, Caires Vinicius wrote: > > I have one DataFrame with nested StructField and I want to convert to JSON > S

Re: I want to unsubscribe

to unsubscribe, send an email to user-unsubscr...@spark.apache.org On Tue, Apr 5, 2016 at 4:50 PM, Ranjana Rajendran wrote: > I get to see the threads in the public mailing list. I don;t want so many > messages in my inbox. I want to unsub

RE: unsubscribe

Ohh ! I thought you're unsubscribing :) Kapil Malik | kma...@adobe.com | 33430 / 8800836581 -Original Message- From: Matei Zaharia [mailto:matei.zaha...@gmail.com] Sent: 12 March 2014 00:51 To: user@spark.apache.org Subject: Re: unsubscribe To unsubscribe from this list, p

Re: Sample date_trunc error for webpage （https://spark.apache.org/docs/2.3.0/api/sql/#date_trunc ）

gt;> >> On Sun, Jul 7, 2019, 2:22 PM Chris Lambertus wrote: >> >>> Spark, >>> >>> We received this message. I have not ACKd it. >>> >>> -Chris >>> INFRA >>> >>> >>> Begin forwarded message: >&

Re: May we merge into branch-1.3 at this point?

: > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=4aaf48d46d13129f0f9bdafd771dd80fe568a7dc > > Open season on 1.3 branch then... > > On Fri, Mar 13, 2015 at 4:20 PM, Nicholas Chammas > > wrote: > > Looks like the release is out: > > http://spa

Re: [VOTE] Release Apache Spark 1.5.1 (RC1)

am wrote: > Hi Spark Developers, > > The Spark 1.5.1 documentation is already publicly accessible > (https://spark.apache.org/docs/latest/index.html) but the release is not. Is > it intentional? > > Best Regards, > > Jerry > > On Mon, Sep 28, 2015 at 9:21

Re: [ANNOUNCE] Announcing Spark 1.5.1

t; >> Spark 1.5.1 is a maintenance release containing stability fixes. This >> release is based on the branch-1.5 maintenance branch of Spark. We >> *strongly recommend* all 1.5.0 users to upgrade to this release. >> >> The full list of bug fixes is here: http://s.apache.org/spa

Updating docs for running on Mesos

es - (same as current) Running Alongside Hadoop - (trim this down) Does that work for people? Thanks! Andrew PS Basically all the same: http://spark.apache.org/docs/0.6.0/running-on-mesos.html http://spark.apache.org/docs/0.6.2/running-on-mesos.html http://spark.apache.org/docs/0.7.3/

Re: Spark spilling location

nce, ~/temp > instead of /tmp. Will this be the location where all data will be spilled > to? I assume it is, based on the description of spark.local.dir at > https://spark.apache.org/docs/latest/configuration.html: > "Directory to use for "scratch" space in Spark, incl

Re: Question about SparkSQL and Hive-on-Spark

gt; From: "Yi Tian" > To: dev@spark.apache.org > Sent: Tuesday, September 23, 2014 2:47:17 AM > Subject: Question about SparkSQL and Hive-on-Spark > > Hi all, > > I have some questions about the SparkSQL and Hive-on-Spark > > Will SparkSQL support all the

Fwd: [ANNOUNCE] Announcing Apache Spark 2.1.0

OUNCE] Announcing Apache Spark 2.1.0 To: "d...@spark.apache.org" , user < u...@spark.apache.org> Hi all, Apache Spark 2.1.0 is the second release of Spark 2.x line. This release makes significant strides in the production readiness of Structured Streaming, with added support for event t

[jira] [Updated] (SPARK-18279) ML programming guide should have R examples

[ https://issues.apache.org/jira/browse/SPARK-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18279: -- Description: http://spark.apache.org/docs/latest/ml-classification-regression.html for

[jira] [Resolved] (SPARK-25082) Documentation for Spark Function expm1 is incomplete

For: 2.4.0 > > > The documentation for the function expm1 that takes in a string > public static  > [Column|https://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/sql/Column.html] >  expm1(String columnName) > ([https://spark.apache.org/docs/2.3.1/api/java/org/apache/spar

[jira] [Commented] (SPARK-19445) Please remove tylerchap...@yahoo-inc.com subscription from u...@spark.apache.org

east I am not sure who the admin is. It is a question for dev@ not JIRA > Please remove tylerchap...@yahoo-inc.com subscription from > u...@spark.apache.org > > > Key: SPARK-19445 >

[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example

iven input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/de

[jira] [Commented] (SPARK-25795) Fix CSV SparkR SQL Example

k/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/sql-programming-guide.html#manually-specifying-options > - > http://spark.apache.org/docs/2.3.2/sql-programming-guide.html#manually-specifying-options > - >

[jira] [Commented] (SPARK-25795) Fix CSV SparkR SQL Example

k/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/sql-programming-guide.html#manually-specifying-options > - > http://spark.apache.org/docs/2.3.2/sql-programming-guide.html#manually-specifying-options > - >

[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example

ot resolve '`name`' > given input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R > - >

[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example

sql.AnalysisException: cannot resolve '`name`' > given input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R &

[jira] [Updated] (SPARK-17794) 2.0.1 not in maven central repo?

Put the following into pom.xml as shown here: https://spark.apache.org/downloads.html {code:java} org.apache.spark spark-core_2.11 2.0.1 {code} Version 2.0.1 does not seem to exist in the Central Repository: https://repo1.maven.org/maven2/org/apache/spark/spark

[jira] [Comment Edited] (SPARK-3044) Create RSS feed for Spark News

0 PM: -- Hi Michael, I don't know if the site itself is open-source. We might need someone from Databricks to update it. [~pwendell], [~rxin], or [~matei] - Is it possible for contributors to contribute to the [main Spark site|http://spark.apache.org/]? was (Author: nchammas): Hi Mi

[jira] [Created] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?

wants the operators pushed down [2]. Is the documentation for SupportsReportStatistics referring to something other than [2] or should it be updated?   [[1]https://spark.apache.org/docs/2.4.6/api/java/org/apache/spark/sql/sources/v2/reader/SupportsReportStatistics.html|https://spark.apache.org/

[jira] [Created] (SPARK-29830) PySpark.context.Sparkcontext.binaryfiles improved memory with buffer

) differs from the Scala implementation (see pyspark here: [https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/context.html#SparkContext.binaryFiles).   |https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/context.html#SparkContext.binaryFiles] In Scala, Spark returns a

[jira] [Commented] (SPARK-27972) Move SQL migration guide to the top level

hese guides from the spark.apache.org site's release notes pages! I've talked to a few folks who were unaware that these guides existed and linking them more prominently from the release page could really help with discoverability. > Move SQL migration guide

[jira] [Created] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome

following get redirected to a generic "root" package page: [https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Dataset.html] [https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset] Here's the URL that I get : [https://spark.apache.

[jira] [Updated] (SPARK-32186) User Guide - Debugging

[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules

[jira] [Updated] (SPARK-32186) User Guide - Debugging

[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules

[jira] [Updated] (SPARK-7096) Java example for Streaming on site uses map instead of mapToPair

[ https://issues.apache.org/jira/browse/SPARK-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Sargisson updated SPARK-7096: Description: https://spark.apache.org/docs/latest/streaming-programming-guide.html Here

[jira] [Commented] (SPARK-7255) spark.streaming.kafka.maxRetries not documented

question, first. Some stuff is undocumented on purpose as it's experimental or not really to be changed by most users. I don't know about this one. It only exists for the direct, Kafka-based stream. I thought the config docs (which are at http://spark.apache.org/docs/latest/confi

[jira] [Comment Edited] (SPARK-12694) The detailed rest API documentation for each field is missing

1 AM: - Does this still need someone to work on? I checked it at http://spark.apache.org/docs/latest/monitoring.html it seems to be fixed. Last commit is commit 4a4c3dc9ca10e52f7981b225ec44e97247986905 Author: jerryshao Date: Fri Jan 6 10:07:54 2017 -0600 was (Author: ehsun7b): Does

[jira] [Commented] (SPARK-20199) GradientBoostedTreesModel doesn't have Column Sampling Rate Paramenter

fication can’t be done 5. Offset: present in H2O 6. Choice of distributions do not include Gamma, Tweedie, Poisson 7. Generates classes, not probabilities (they said later version will take care of this) http://spark.apache.org/docs/latest/api/scala/in

[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...

, including a programming -guide, on the project webpage at <http://spark.apache.org/documentation.html>. +guide, on the [project web page](http://spark.apache.org/documentation.html). This README file only contains basic setup instructions. ## Building Spark -Sp

[GitHub] [spark] dongjoon-hyun commented on pull request #34195: [SPARK-36939][PYTHON][DOCS] Add orphan migration page into list in PySpark documentation

guide/pyspark_3.2_to_3.3' ``` In short, although SPARK-36939 landed at 3.2.1. `pyspark_3.2_to_3.3` doesn't show up in Apache Spark 3.2.1 documentation becuase it doesn't exist in branch-3.2. Let me revert this from branch-3.2. - https://spark.apache.org/docs/3.3.0/api/python/migrat

[GitHub] [spark] itholic commented on a diff in pull request #39820: [SPARK-42249][SQL] Refining html link for documentation in error messages.

t found an encoder of the type to Spark SQL internal representation. Consider to change the input type to one of supported at https://spark.apache.org/docs/latest/sql-ref-datatypes.html."; + "Not found an encoder of the type to Spark SQL internal representation. Consider to change th

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33435: [SPARK-31907][DOCS][SQL] Adding location of SQL API documentation

documentation. + * at SQL API documentation (https://spark.apache.org/docs/latest/api/sql/index.html). Review comment: the problem is that the link became stale. For example, in Spark 3.0.0 release documentation, https://spark.apache.org/docs/latest/api/sql/index.html will point out the latest

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33435: [SPARK-31907][DOCS][SQL] Adding location of SQL API documentation

documentation. + * at SQL API documentation (https://spark.apache.org/docs/latest/api/sql/index.html). Review comment: the problem is that the link becomes stale. For example, in Spark 3.0.0 release documentation, https://spark.apache.org/docs/latest/api/sql/index.html will point out the latest

[GitHub] [spark] dhruvildave opened a new pull request #33107: [SPARK-35909] Fix broken Python Links in docs/sql-getting-started.md

dhruvildave opened a new pull request #33107: URL: https://github.com/apache/spark/pull/33107 ### What changes were proposed in this pull request? The hyperlinks in Python code blocks in [Spark SQL Guide - Getting Started](https://spark.apache.org/docs/latest/sql-getting

[GitHub] [spark] risinga commented on a change in pull request #34359: [SPARK-36986][SQL] Improving external schema management flexibility on DataSet and StructType

], schema: StructType): DataFrame = { Review comment: @HyukjinKwon I've sent an email to d...@spark.apache.org, as well as dev-subscr...@spark.apache.org and I've got no response. I can find the subject on the mailing lists http://mail-archives.apache.org/mod_mbox/spark-dev/2

[GitHub] [spark] panbingkun commented on pull request #42425: [SPARK-44729][PYTHON][DOCS] Add canonical links to the PySpark docs page

panbingkun commented on PR #42425: URL: https://github.com/apache/spark/pull/42425#issuecomment-1676556063 > Here is an example of a documentation page for a specific version: https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html >

[PR] [SPARK-46464][DOC] Fix the scroll issue of tables when overflow [spark]

yaooqinn opened a new pull request, #44423: URL: https://github.com/apache/spark/pull/44423 ### What changes were proposed in this pull request? https://spark.apache.org/docs/3.4.1/running-on-kubernetes.html#spark-properties https://spark.apache.org/docs/latest

RE: Spark Version 3.0.1 Gui Display Query

Hi Attila, I checked the Section < https://spark.apache.org/docs/latest/monitoring.html#web-interfaces> and Web UI Page What document is saying that if I want to view information only for the duration of the application, then I do not need to generate the event logs and do not need

Re: number of partitions in join: Spark documentation misleading!

yKey and join, the largest > number of partitions in a parent RDD." > https://spark.apache.org/docs/latest/configuration.html > > And the Partitioner.scala comments (line 51) state that: > "Unless spark.default.parallelism is set, the number of partitions will be > the same a

Re: Configuring Ports for Network Security

ote: > > Hi > > We want to use spark in a secure cluster with iptables enabled. > For this, we need a specific list of ports used by spark so that we can > whitelist them. > > From what I could learn from - > http://spark.apache.org/docs/latest/security.html#configur

Re: StateSpec raises error "missing arguments for method"

arguments for method mappingFunction; > follow this method with `_' if you want to treat it as a partially applied > function > val spec = StateSpec.function(mappingFunction) > ^ > I followed the current documentation

Re: Stop Spark Streaming Jobs

msung.com] > Sent: Wednesday, August 03, 2016 11:24 AM > To: 'Pradeep'; 'user@spark.apache.org' > Subject: RE: Stop Spark Streaming Jobs > > Hi. Paradeep > > > Did you mean, how to kill the job? > If yes, you should kill the driver and follow next.

RE: SparkR Error in sparkR.init(master=“local”) in RStudio

ny hint in the console output -Original Message- From: akhandeshi [mailto:ami.khande...@gmail.com] Sent: Tuesday, October 6, 2015 8:21 PM To: user@spark.apache.org Subject: Re: SparkR Error in sparkR.init(master=“local”) in RStudio I couldn't get this working... I have have JAVA_H

RE: SparkR Supported Types - Please add "bigint"

these two type name representations. -Original Message- From: Exie [mailto:tfind...@prodevelop.com.au] Sent: Friday, July 24, 2015 1:35 PM To: user@spark.apache.org Subject: Re: SparkR Supported Types - Please add "bigint" Interestingly, after more digging, df.printSchema() in raw

Re: how to integrate Apache Kafka with spark ?

Please see below links depending on version of Spark 2.x http://spark.apache.org/docs/latest/streaming-kafka-integration.html Spark Streaming + Kafka Integration Guide - Spark 2.0.2 ...<http://spark.apache.org/docs/latest/streaming-kafka-integration.html> spark.apache.org Spark Str

<    12   13   14   15   16   17   18   19   20   21   >