Mailing lists matching spark.apache.org
commits spark.apache.orgdev spark.apache.org
issues spark.apache.org
reviews spark.apache.org
user spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on pull request #28072: [SPARK-31314][CORE] Revert SPARK-29285 to fix shuffle regression caused by creating temporary file eagerly
`spark.yarn.executor.launch.excludeOnFailure.enabled`) to ban such problematic nodes. You can check the configuration [here](https://spark.apache.org/docs/latest/configuration.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the
[GitHub] [spark] sarutak commented on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala
is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
[GitHub] [spark] sarutak edited a comment on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala
layout is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
[GitHub] [spark] sarutak edited a comment on pull request #34356: [SPARK-36554][SQL][PYTHON] Expose make_date expression in functions.scala
layout is not broken. See also http://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-documentation-changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
[GitHub] gaborgsomogyi commented on issue #23348: [SPARK-25857][core] Add developer documentation regarding delegation tokens.
about considering to put this in somewhere like https://spark.apache.org/docs/latest/running-on-yarn.html after refining it? We have Kerberos chapter https://spark.apache.org/docs/latest/running-on-yarn.html#kerberos` This is an auto
Re: [PR] [SPARK-47912][SQL] Infer serde class from format classes [spark]
ing process. This doesn't seem to be relevant to the current PR, you can get more help from https://spark.apache.org/community.html. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
[GitHub] [spark] pan3793 commented on pull request #42493: [SPARK-44811][BUILD] Upgrade Guava to 32+
oved in Spark 4?](https://www.mail-archive.com/dev@spark.apache.org/msg30708.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: rev
Re: [PR] [SPARK-46111][DOCS][PYTHON] Add copyright to the PySpark official documentation. [spark]
/licenses/LICENSE-2.0";>Apache License, Version 2.0. Review Comment: FYI: I follow this copyright format from [Apache Spark official web page](https://spark.apache.org/). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub
Re: [PR] [SPARK-46351][SQL] Require an error class in `AnalysisException` [spark]
block" ] }, + "_LEGACY_ERROR_TEMP_3050" : { +"message" : [ + "Cannot modify the value of a static config: " +] Review Comment: Can we use this? - https://spark.apache.org/docs/latest/sql-error-conditions.html#cannot_modify_config -- This is
[GitHub] [spark] zhengruifeng commented on pull request #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics
zhengruifeng commented on PR #36926: URL: https://github.com/apache/spark/pull/36926#issuecomment-1161256434 since it is only exposed in the `.mllib` side, and it is not listed in the [official documents ](https://spark.apache.org/docs/latest/mllib-evaluation-metrics.html#ranking
[GitHub] [spark] dongjoon-hyun commented on pull request #36069: [SPARK-38767][SQL] Support `ignoreCorruptFiles` and `ignoreMissingFiles` in Data Source options
dongjoon-hyun commented on PR #36069: URL: https://github.com/apache/spark/pull/36069#issuecomment-1190997582 FYI, Apache Spark 3.4 code freeze was March 15th, 2022. This patch simply arrived one month later after code freeze deadline. - https://spark.apache.org/versioning
Re: [PR] [SPARK-45273][CORE][UI] Support for set the access host in http header [spark]
srowen commented on PR #43169: URL: https://github.com/apache/spark/pull/43169#issuecomment-1752893775 How does this arise? Use priv...@spark.apache.org if needed. I am not clear what attack you have in mind or whether it can affect spark, so, no this would not be useful unless there'
Re: Spark Language / Data Base Question
ng else ? > > > Please help me > > > Thanks > > > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
Fwd: use S3-Compatible Storage with spark
amazon, is there >> a way I can specify the host somewhere ? >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > -
Re: Save DataFrame to HBase
t;> wrote: > >> Has anyone found an easy way to save a DataFrame into HBase? >> >> Thanks, >> Ben >> >> >> ----- >> To unsubscribe, e-mail: user-unsubscr...@spar
Re: Save DataFrame to HBase
to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> > >
Re: broadcast variable not picked up
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > ----- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Allowing parallelism in spark local mode
You’ll want to setup the FAIR scheduler as described here: https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application From: yael aharon mailto:yael.aharo...@gmail.com>> Date: Friday, February 12, 2016 at 2:00 PM To: "user@spark.apache.org&
Re: How to delete a record from parquet files using dataframes
. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional
Re: Dataframe constructor
> > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-constructor-tp25455.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > ---
Re: Where is Redgate's HDFS explorer?
>> http://apache-spark-user-list.1001560.n3.nabble.com/Where-is-Redgate-s-HDFS-explorer-tp24431.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> ----
Re: RDD String foreach println
as why 1) behaves differently on job submit and shell? > > Best, > Patcharee > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > Fo
Re: Effects of persist(XYZ_2)
ding network transfers? Assuming I’m doing joins or other >> shuffle operations. >> >> Thanks >> >> >> ----- >> To unsubscribe, e-mail: user-u
Re: Upgrade to Spark 1.2.1 using Guava
rk. -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional
Re: multiple sparkcontexts and streamingcontexts
contexts-tp21876.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >
RE: Is SQLContext thread-safe?
text to do Select operations in different threads at the same time? Thank you very much! - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.
Re: [ANNOUNCE] Announcing Spark 1.3!
ot on-list). > > Thanks to everyone who helped work on this release! > > [1] http://spark.apache.org/releases/spark-release-1-3-0.html > [2] http://spark.apache.org/downloads.html > > - > To unsubscribe, e-ma
Re: Spark Job History Server
r > > What class is really needed? How to fix it? > > Br, > Patcharee > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
Re: How to merge two dataframes with same schema
mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To
Re: Mapping with extra arguments
in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Mapping-with-extra-arguments-tp12541.html > Sent from the Apache Spark User List mailing list archive at Nabble.com > (http://Nabble.com). > > -
Re: Spark Screencast doesn't show in Chrome on OS X
> https://spark.apache.org/screencasts/1-first-steps-with-spark.html > > The embedded YouTube video shows up in Safari on OS X but not in Chrome. I’m using Chrome 36.0.1985.143 on MacOS 10.9.4 and it it works like a charm for me. Cheers, Michael -- Michael Hausenbla
Re: nested rdd operation
ter exception , > can somebody explain why i cannot have a nested rdd operation ? > > --pavlos > > ----- > To unsubscribe, e-mail: user-unsubscr...@spark.apa
RE: Cassandra connector
pache-spark-user-list.1001560.n3.nabble.com/Cassandra-connector-tp13896p13897.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For addit
Re: Spark on Raspberry Pi?
> Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For a
Re: Accuracy hit in classification with Spark
html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
Re: Stable spark streaming app
er of nodes, events per second, broad stream processing workflow, config highlights etc? Thanks, Tim - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-ma
Re: K-means faster on Mahout then on Spark
List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsu
Re: SparkContext.stop() ?
ntext-stop-tp17826.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For addit
Re: Matrix multiplication in spark
2562p18164.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional co
Re: saveAsParquetFile throwing exception
8929p18935.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.
Re: spark-sql broken
r-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift server
st mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org --
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in task never being executed.
ults-in-task-never-being-executed-tp18969p20957.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional comma
Re: acquire and give back resources dynamically
http://spark.apache.org/docs/latest/running-on-yarn.html Spark just a Yarn application > 在 2014年8月14日,11:12,牛兆捷 写道: > > Dear all: > > Does spark can acquire resources from and give back resources to > YARN dynamically ? > > >
[jira] [Created] (SPARK-32191) Migration Guide
: Documentation, PySpark Affects Versions: 3.1.0 Reporter: Hyukjin Kwon Port http://spark.apache.org/docs/latest/pyspark-migration-guide.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To
[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20188 Actually in R setCheckpointDir method is not attached to the SparkContext; Iâd leave it as ânot setâ or ânot set in the sessionâ https://spark.apache.org/docs/latest/api/R
[GitHub] spark issue #21207: SPARK-24136: Fix MemoryStreamDataReader.next to skip sle...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21207 @arunmahadevan, not a big deal but mind if I ask to fix the PR title to `[SPARK-24136][SS] blabla`? It's actually encouraged in the guide - https://spark.apache.org/contributing
[GitHub] spark issue #21767: SPARK-24804 There are duplicate words in the test title ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21767 yeah, please avoid PRs that are this trivial, it's just not worth the overhead. But I merged it this time. Also please read https://spark.apache.org/contributing
[GitHub] spark issue #21828: Update regression.py
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21828 @woodthom2, if you have some plans to update this PR quite soon, please see https://spark.apache.org/contributing.html and proceed. Otherwise, I would suggest to leave this closed so that
[GitHub] spark issue #19714: [SPARK-22489][SQL] Shouldn't change broadcast join build...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19714 LGTM Thanks! Merged to master. Could you submit a follow-up PR to document the behavior changes in migration section of Spark SQL? https://spark.apache.org/docs/latest/sql
[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19343 @squito Thank you! Instead of changing the source codes, could we just update the document https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#hive-tables ? This might be
Re: Supported Hadoop versions for Spark 3.3
adoop” package from https://spark.apache.org/downloads.html? <https://spark.apache.org/downloads.html?> - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: udf StructField to JSON String
Have you looked at DataFrame.write.json( path )? https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriter > On Mar 11, 2016, at 7:15 AM, Caires Vinicius wrote: > > I have one DataFrame with nested StructField and I want to convert to JSON > S
Re: I want to unsubscribe
to unsubscribe, send an email to user-unsubscr...@spark.apache.org On Tue, Apr 5, 2016 at 4:50 PM, Ranjana Rajendran wrote: > I get to see the threads in the public mailing list. I don;t want so many > messages in my inbox. I want to unsub
RE: unsubscribe
Ohh ! I thought you're unsubscribing :) Kapil Malik | kma...@adobe.com | 33430 / 8800836581 -Original Message- From: Matei Zaharia [mailto:matei.zaha...@gmail.com] Sent: 12 March 2014 00:51 To: user@spark.apache.org Subject: Re: unsubscribe To unsubscribe from this list, p
Re: Sample date_trunc error for webpage (https://spark.apache.org/docs/2.3.0/api/sql/#date_trunc )
gt;> >> On Sun, Jul 7, 2019, 2:22 PM Chris Lambertus wrote: >> >>> Spark, >>> >>> We received this message. I have not ACKd it. >>> >>> -Chris >>> INFRA >>> >>> >>> Begin forwarded message: >&
Re: May we merge into branch-1.3 at this point?
: > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=4aaf48d46d13129f0f9bdafd771dd80fe568a7dc > > Open season on 1.3 branch then... > > On Fri, Mar 13, 2015 at 4:20 PM, Nicholas Chammas > > wrote: > > Looks like the release is out: > > http://spa
Re: [VOTE] Release Apache Spark 1.5.1 (RC1)
am wrote: > Hi Spark Developers, > > The Spark 1.5.1 documentation is already publicly accessible > (https://spark.apache.org/docs/latest/index.html) but the release is not. Is > it intentional? > > Best Regards, > > Jerry > > On Mon, Sep 28, 2015 at 9:21
Re: [ANNOUNCE] Announcing Spark 1.5.1
t; >> Spark 1.5.1 is a maintenance release containing stability fixes. This >> release is based on the branch-1.5 maintenance branch of Spark. We >> *strongly recommend* all 1.5.0 users to upgrade to this release. >> >> The full list of bug fixes is here: http://s.apache.org/spa
Updating docs for running on Mesos
es - (same as current) Running Alongside Hadoop - (trim this down) Does that work for people? Thanks! Andrew PS Basically all the same: http://spark.apache.org/docs/0.6.0/running-on-mesos.html http://spark.apache.org/docs/0.6.2/running-on-mesos.html http://spark.apache.org/docs/0.7.3/
Re: Spark spilling location
nce, ~/temp > instead of /tmp. Will this be the location where all data will be spilled > to? I assume it is, based on the description of spark.local.dir at > https://spark.apache.org/docs/latest/configuration.html: > "Directory to use for "scratch" space in Spark, incl
Re: Question about SparkSQL and Hive-on-Spark
gt; From: "Yi Tian" > To: dev@spark.apache.org > Sent: Tuesday, September 23, 2014 2:47:17 AM > Subject: Question about SparkSQL and Hive-on-Spark > > Hi all, > > I have some questions about the SparkSQL and Hive-on-Spark > > Will SparkSQL support all the
Fwd: [ANNOUNCE] Announcing Apache Spark 2.1.0
OUNCE] Announcing Apache Spark 2.1.0 To: "d...@spark.apache.org" , user < u...@spark.apache.org> Hi all, Apache Spark 2.1.0 is the second release of Spark 2.x line. This release makes significant strides in the production readiness of Structured Streaming, with added support for event t
[jira] [Updated] (SPARK-18279) ML programming guide should have R examples
[ https://issues.apache.org/jira/browse/SPARK-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18279: -- Description: http://spark.apache.org/docs/latest/ml-classification-regression.html for
[jira] [Resolved] (SPARK-25082) Documentation for Spark Function expm1 is incomplete
For: 2.4.0 > > > The documentation for the function expm1 that takes in a string > public static > [Column|https://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/sql/Column.html] > expm1(String columnName) > ([https://spark.apache.org/docs/2.3.1/api/java/org/apache/spar
[jira] [Commented] (SPARK-19445) Please remove tylerchap...@yahoo-inc.com subscription from u...@spark.apache.org
east I am not sure who the admin is. It is a question for dev@ not JIRA > Please remove tylerchap...@yahoo-inc.com subscription from > u...@spark.apache.org > > > Key: SPARK-19445 >
[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example
iven input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/de
[jira] [Commented] (SPARK-25795) Fix CSV SparkR SQL Example
k/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/sql-programming-guide.html#manually-specifying-options > - > http://spark.apache.org/docs/2.3.2/sql-programming-guide.html#manually-specifying-options > - >
[jira] [Commented] (SPARK-25795) Fix CSV SparkR SQL Example
k/blob/master/examples/src/main/r/RSparkSQLExample.R > - > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc3-docs/_site/sql-programming-guide.html#manually-specifying-options > - > http://spark.apache.org/docs/2.3.2/sql-programming-guide.html#manually-specifying-options > - >
[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example
ot resolve '`name`' > given input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R > - >
[jira] [Updated] (SPARK-25795) Fix CSV SparkR SQL Example
sql.AnalysisException: cannot resolve '`name`' > given input columns: [_c0];; > 'Project ['name, 'age] > +- AnalysisBarrier > +- Relation[_c0#97] csv > {code} > > - > https://github.com/apache/spark/blob/master/examples/src/main/r/RSparkSQLExample.R &
[jira] [Updated] (SPARK-17794) 2.0.1 not in maven central repo?
Put the following into pom.xml as shown here: https://spark.apache.org/downloads.html {code:java} org.apache.spark spark-core_2.11 2.0.1 {code} Version 2.0.1 does not seem to exist in the Central Repository: https://repo1.maven.org/maven2/org/apache/spark/spark
[jira] [Comment Edited] (SPARK-3044) Create RSS feed for Spark News
0 PM: -- Hi Michael, I don't know if the site itself is open-source. We might need someone from Databricks to update it. [~pwendell], [~rxin], or [~matei] - Is it possible for contributors to contribute to the [main Spark site|http://spark.apache.org/]? was (Author: nchammas): Hi Mi
[jira] [Created] (SPARK-32095) [DataSource V2] Documentation on SupportsReportStatistics Outdated?
wants the operators pushed down [2]. Is the documentation for SupportsReportStatistics referring to something other than [2] or should it be updated? [[1]https://spark.apache.org/docs/2.4.6/api/java/org/apache/spark/sql/sources/v2/reader/SupportsReportStatistics.html|https://spark.apache.org/
[jira] [Created] (SPARK-29830) PySpark.context.Sparkcontext.binaryfiles improved memory with buffer
) differs from the Scala implementation (see pyspark here: [https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/context.html#SparkContext.binaryFiles). |https://spark.apache.org/docs/2.4.0/api/python/_modules/pyspark/context.html#SparkContext.binaryFiles] In Scala, Spark returns a
[jira] [Commented] (SPARK-27972) Move SQL migration guide to the top level
hese guides from the spark.apache.org site's release notes pages! I've talked to a few folks who were unaware that these guides existed and linking them more prominently from the release page could really help with discoverability. > Move SQL migration guide
[jira] [Created] (SPARK-28085) Spark Scala API documentation URLs not working properly in Chrome
following get redirected to a generic "root" package page: [https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Dataset.html] [https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset] Here's the URL that I get : [https://spark.apache.
[jira] [Updated] (SPARK-32186) User Guide - Debugging
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules
[jira] [Updated] (SPARK-32186) User Guide - Debugging
[ https://issues.apache.org/jira/browse/SPARK-32186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32186: - Description: 1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules
[jira] [Updated] (SPARK-7096) Java example for Streaming on site uses map instead of mapToPair
[ https://issues.apache.org/jira/browse/SPARK-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Sargisson updated SPARK-7096: Description: https://spark.apache.org/docs/latest/streaming-programming-guide.html Here
[jira] [Commented] (SPARK-7255) spark.streaming.kafka.maxRetries not documented
question, first. Some stuff is undocumented on purpose as it's experimental or not really to be changed by most users. I don't know about this one. It only exists for the direct, Kafka-based stream. I thought the config docs (which are at http://spark.apache.org/docs/latest/confi
[jira] [Comment Edited] (SPARK-12694) The detailed rest API documentation for each field is missing
1 AM: - Does this still need someone to work on? I checked it at http://spark.apache.org/docs/latest/monitoring.html it seems to be fixed. Last commit is commit 4a4c3dc9ca10e52f7981b225ec44e97247986905 Author: jerryshao Date: Fri Jan 6 10:07:54 2017 -0600 was (Author: ehsun7b): Does
[jira] [Commented] (SPARK-20199) GradientBoostedTreesModel doesn't have Column Sampling Rate Paramenter
fication can’t be done 5. Offset: present in H2O 6. Choice of distributions do not include Gamma, Tweedie, Poisson 7. Generates classes, not probabilities (they said later version will take care of this) http://spark.apache.org/docs/latest/api/scala/in
[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...
, including a programming -guide, on the project webpage at <http://spark.apache.org/documentation.html>. +guide, on the [project web page](http://spark.apache.org/documentation.html). This README file only contains basic setup instructions. ## Building Spark -Sp
[GitHub] [spark] dongjoon-hyun commented on pull request #34195: [SPARK-36939][PYTHON][DOCS] Add orphan migration page into list in PySpark documentation
guide/pyspark_3.2_to_3.3' ``` In short, although SPARK-36939 landed at 3.2.1. `pyspark_3.2_to_3.3` doesn't show up in Apache Spark 3.2.1 documentation becuase it doesn't exist in branch-3.2. Let me revert this from branch-3.2. - https://spark.apache.org/docs/3.3.0/api/python/migrat
[GitHub] [spark] itholic commented on a diff in pull request #39820: [SPARK-42249][SQL] Refining html link for documentation in error messages.
t found an encoder of the type to Spark SQL internal representation. Consider to change the input type to one of supported at https://spark.apache.org/docs/latest/sql-ref-datatypes.html."; + "Not found an encoder of the type to Spark SQL internal representation. Consider to change th
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33435: [SPARK-31907][DOCS][SQL] Adding location of SQL API documentation
documentation. + * at SQL API documentation (https://spark.apache.org/docs/latest/api/sql/index.html). Review comment: the problem is that the link became stale. For example, in Spark 3.0.0 release documentation, https://spark.apache.org/docs/latest/api/sql/index.html will point out the latest
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33435: [SPARK-31907][DOCS][SQL] Adding location of SQL API documentation
documentation. + * at SQL API documentation (https://spark.apache.org/docs/latest/api/sql/index.html). Review comment: the problem is that the link becomes stale. For example, in Spark 3.0.0 release documentation, https://spark.apache.org/docs/latest/api/sql/index.html will point out the latest
[GitHub] [spark] dhruvildave opened a new pull request #33107: [SPARK-35909] Fix broken Python Links in docs/sql-getting-started.md
dhruvildave opened a new pull request #33107: URL: https://github.com/apache/spark/pull/33107 ### What changes were proposed in this pull request? The hyperlinks in Python code blocks in [Spark SQL Guide - Getting Started](https://spark.apache.org/docs/latest/sql-getting
[GitHub] [spark] risinga commented on a change in pull request #34359: [SPARK-36986][SQL] Improving external schema management flexibility on DataSet and StructType
], schema: StructType): DataFrame = { Review comment: @HyukjinKwon I've sent an email to d...@spark.apache.org, as well as dev-subscr...@spark.apache.org and I've got no response. I can find the subject on the mailing lists http://mail-archives.apache.org/mod_mbox/spark-dev/2
[GitHub] [spark] panbingkun commented on pull request #42425: [SPARK-44729][PYTHON][DOCS] Add canonical links to the PySpark docs page
panbingkun commented on PR #42425: URL: https://github.com/apache/spark/pull/42425#issuecomment-1676556063 > Here is an example of a documentation page for a specific version: https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html >
[PR] [SPARK-46464][DOC] Fix the scroll issue of tables when overflow [spark]
yaooqinn opened a new pull request, #44423: URL: https://github.com/apache/spark/pull/44423 ### What changes were proposed in this pull request? https://spark.apache.org/docs/3.4.1/running-on-kubernetes.html#spark-properties https://spark.apache.org/docs/latest
RE: Spark Version 3.0.1 Gui Display Query
Hi Attila, I checked the Section < https://spark.apache.org/docs/latest/monitoring.html#web-interfaces> and Web UI Page What document is saying that if I want to view information only for the duration of the application, then I do not need to generate the event logs and do not need
Re: number of partitions in join: Spark documentation misleading!
yKey and join, the largest > number of partitions in a parent RDD." > https://spark.apache.org/docs/latest/configuration.html > > And the Partitioner.scala comments (line 51) state that: > "Unless spark.default.parallelism is set, the number of partitions will be > the same a
Re: Configuring Ports for Network Security
ote: > > Hi > > We want to use spark in a secure cluster with iptables enabled. > For this, we need a specific list of ports used by spark so that we can > whitelist them. > > From what I could learn from - > http://spark.apache.org/docs/latest/security.html#configur
Re: StateSpec raises error "missing arguments for method"
arguments for method mappingFunction; > follow this method with `_' if you want to treat it as a partially applied > function > val spec = StateSpec.function(mappingFunction) > ^ > I followed the current documentation
Re: Stop Spark Streaming Jobs
msung.com] > Sent: Wednesday, August 03, 2016 11:24 AM > To: 'Pradeep'; 'user@spark.apache.org' > Subject: RE: Stop Spark Streaming Jobs > > Hi. Paradeep > > > Did you mean, how to kill the job? > If yes, you should kill the driver and follow next.
RE: SparkR Error in sparkR.init(master=“local”) in RStudio
ny hint in the console output -Original Message- From: akhandeshi [mailto:ami.khande...@gmail.com] Sent: Tuesday, October 6, 2015 8:21 PM To: user@spark.apache.org Subject: Re: SparkR Error in sparkR.init(master=“local”) in RStudio I couldn't get this working... I have have JAVA_H
RE: SparkR Supported Types - Please add "bigint"
these two type name representations. -Original Message- From: Exie [mailto:tfind...@prodevelop.com.au] Sent: Friday, July 24, 2015 1:35 PM To: user@spark.apache.org Subject: Re: SparkR Supported Types - Please add "bigint" Interestingly, after more digging, df.printSchema() in raw
Re: how to integrate Apache Kafka with spark ?
Please see below links depending on version of Spark 2.x http://spark.apache.org/docs/latest/streaming-kafka-integration.html Spark Streaming + Kafka Integration Guide - Spark 2.0.2 ...<http://spark.apache.org/docs/latest/streaming-kafka-integration.html> spark.apache.org Spark Str