Mailing lists matching spark.apache.org
commits spark.apache.orgdev spark.apache.org
issues spark.apache.org
reviews spark.apache.org
user spark.apache.org
Re: Is storage resources counted during the scheduling
, this might be another question). Thanks. Best, Jialin > On Apr 11, 2016, at 3:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > See > https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application > > <https://spark.apache.org/docs/latest/job
Re: Where to set properties for the retainedJobs/Stages?
2016-04-01 18:58, schrieb Ted Yu: You can set them in spark-defaults.conf See also https://spark.apache.org/docs/latest/configuration.html#spark-ui [1] On Fri, Apr 1, 2016 at 8:26 AM, Max Schmidt <m...@datapath.io> wrote: Can somebody tell me the interaction between the prop
Re: SPARK-13843 and future of streaming backends
epository.apache.org. Exceptions to that must be approved by VP, > Infrastructure. > -- > Sent via Pony Mail for dev@spark.apache.org. > View this email online at: > https://pony-poc.apache.org/list.html?dev@spark.apache.org > > -
[jira] [Commented] (SPARK-17560) SQLContext tables returns table names in lower case only
[ https://issues.apache.org/jira/browse/SPARK-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495906#comment-15495906 ] Aseem Bansal commented on SPARK-17560: -- Looked through https://spark.apache.org/docs/2.0.0/sql
Re: Spark streaming completed batches statistics
Ok it looks like I could reconstruct the logic in the Spark UI from the /jobs resource. Thanks. https://richardstartin.com/ From: map reduced <k3t.gi...@gmail.com> Sent: 07 December 2016 19:49 To: Richard Startin Cc: user@spark.apache.org Subject: Re:
Re: EXT: Multiple cores/executors in Pyspark standalone mode
mode, you need standalone Spark cluster<https://spark.apache.org/docs/latest/spark-standalone.html>. It requires a master node (can be started using SPARK_HOME/sbin/start-master.sh script) and at least one worker node (can be started using SPARK_HOME/sbin/start-slave.sh script).SparkConf
Re: Welcoming Hyukjin Kwon and Sameer Agarwal as committers
>> > >> Hi everyone, > >> > >> The Spark PMC recently voted to add Hyukjin Kwon and Sameer Agarwal as > >> committers. Join me in congratulating both of them and thanking them for > >> their contributions to the project! > >> > >> M
[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...
"chara #' model, \code{predict} to make predictions on new data, and \code{write.ml}/\code{read.ml} to #' save/load fitted models. #' For more details, see -#' \href{http://spark.apache.org/docs/latest/ml-classification-regression.html#random-forest-regression}{ -#'
[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...
ict} to make predictions on new data, and #' \code{write.ml}/\code{read.ml} to save/load fitted models. #' For more details, see -#' \href{http://spark.apache.org/docs/latest/ml-classification-regression.html#gradient-boosted-tree-regression}{ -#' GBT Regression} and -#' \h
[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...
, path #' model, \code{predict} to make predictions on new data, and \code{write.ml}/\code{read.ml} to #' save/load fitted models. #' For more details, see -#' \href{http://spark.apache.org/docs/latest/ml-classification-regression.html#decision-tree-regression}{ -#' Deci
[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19485 Thanks for taking a look for this one. Actually, I thought we should add a chapter like http://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets And, add a link
RE: Adding Custom finalize method to RDDs.
I want to delete some files which I created In my datasource api, as soon as the RDD is cleaned up. Thanks, Nasrulla From: Vinoo Ganesh Sent: Monday, June 10, 2019 1:32 PM To: Nasrulla Khan Haris ; dev@spark.apache.org Subject: Re: Adding Custom finalize method to RDDs. Generally overriding
[GitHub] [spark] wangyum commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive
`. It seems `--jars=one.jar,two.jar` is not a standard usage. http://spark.apache.org/docs/latest/spark-standalone.html#launching-spark-applications http://spark.apache.org/docs/latest/running-on-yarn.html#adding-other-jars http://spark.apache.org/docs/latest/rdd-programming
[GitHub] [spark] aof00 opened a new pull request #30376: change 'spark.sql.adaptive.skewedPartitionThresholdInBytes' to 'spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes' #SPARK-33451
aof00 opened a new pull request #30376: URL: https://github.com/apache/spark/pull/30376 JIRA Issue: https://issues.apache.org/jira/browse/SPARK-33451 In the 'Optimizing Skew Join' section of the following two pages: 1. [https://spark.apache.org/docs/3.0.0/sql-performance
[GitHub] [spark] gengliangwang opened a new pull request #31525: [3.1][INFRA][DOC] Change the facetFilters of Docsearch to 3.1.1
. ### Why are the changes needed? So that the search result of the published Spark site will points to https://spark.apache.org/docs/3.1.1 instead of https://spark.apache.org/docs/latest/. This is useful for searching the docs of 3.1.1 after there are new Spark releases
[jira] [Comment Edited] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid
s > Priority: Major > > [http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html] > links to: > [https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html] > which links to: > [https://spark.apache.org/docs/latest/api/python/sql/arrow_pandas
[jira] [Commented] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid
Issue Type: Bug > Components: Documentation >Affects Versions: 3.2.0 >Reporter: Thomas Graves > Priority: Major > > [http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html] > links to: > [https://spark.apache.org/docs/latest/api/pytho
[jira] [Commented] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid
Project: Spark > Issue Type: Bug > Components: Documentation >Affects Versions: 3.2.0 >Reporter: Thomas Graves > Priority: Major > > [http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html] > links to: &
[jira] [Resolved] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid
s Graves > Priority: Major > > [http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html] > links to: > [https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html] > which links to: > [https://spark.apache.org/docs/latest/api/python/sql/a
[jira] [Updated] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid
s Graves >Priority: Major > Fix For: 3.2.1 > > > [http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html] > links to: > [https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html] > which links to: > [https://spa
[jira] [Created] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid
Components: Documentation Affects Versions: 3.2.0 Reporter: Thomas Graves [http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html] links to: [https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html] which links to: [https
[GitHub] [spark] zhengruifeng commented on pull request #43011: [WIP][SPARK-45232][DOCS] Add missing function groups to SQL references
zhengruifeng commented on PR #43011: URL: https://github.com/apache/spark/pull/43011#issuecomment-1728581850 @allisonwang-db I am not sure, I don't see document for FROM clause, you may check 3 places: - https://spark.apache.org/docs/latest/api/sql/index.html#explode - https
[GitHub] [spark] zr-msft commented on pull request #35561: [MINOR][DOCS] Fixed closing tags in running-on-kubernetes.md
zr-msft commented on PR #35561: URL: https://github.com/apache/spark/pull/35561#issuecomment-1148923071 @dongjoon-hyun I've periodically checked the docs site and I'm not seeing any changes show up based on commits i've added from this PR: * https://spark.apache.org/docs/latest/running
[GitHub] [spark] HyukjinKwon commented on pull request #38470: [CONNECT] [DOC] Defining Spark Connect Client Connection String
HyukjinKwon commented on PR #38470: URL: https://github.com/apache/spark/pull/38470#issuecomment-1299418917 Maybe it's better to have a JIRA. BTW, wonder if we have an e2e example for users can copy and paste to try. (e.g., like most of docs in https://spark.apache.org/docs/latest
[jira] [Updated] (CALCITE-6241) Add a few existing functions to Spark library
for other libraries. Spark Functions Link：[https://spark.apache.org/docs/latest/api/sql/index.html|https://spark.apache.org/docs/latest/api/sql/index.html#rtrim] Add function List： * DECODE was: Add Spark as a supported library for functions that have already been implemented for other
Re: acquire and give back resources dynamically
http://spark.apache.org/docs/latest/running-on-yarn.html Spark just a Yarn application 在 2014年8月14日，11:12，牛兆捷 nzjem...@gmail.com 写道： Dear all: Does spark can acquire resources from and give back resources to YARN dynamically ? -- *Regards,* *Zhaojie
Re: spark won't build with maven
You are running a Continuous Compilation. AFAIK, it runs in an infinite loop and will compile only the modified files. For compiling with maven, have a look at these steps - https://spark.apache.org/docs/latest/building-with-maven.html Thanks, Visakh -- View this message in context: http
Re: LDA example?
You can check out this pull request: https://github.com/apache/spark/pull/476 LDA is on the roadmap for the 1.2 release, hopefully we will officially support it then! Best, Burak - Original Message - From: Denny Lee denny.g@gmail.com To: user@spark.apache.org Sent: Thursday, August
RE: resize memory size for caching RDD
AFAIK, No. Best Regards, Raymond Liu From: 牛兆捷 [mailto:nzjem...@gmail.com] Sent: Thursday, September 04, 2014 11:30 AM To: user@spark.apache.org Subject: resize memory size for caching RDD Dear all: Spark uses memory to cache RDD and the memory size is specified
Re: Spark SQL - Exception only when using cacheTable
I am using the python api. Unfortunately, I cannot find the isCached method equivalent in the documentation: https://spark.apache.org/docs/1.1.0/api/python/index.html in the SQLContext section. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL
How to calculate percentiles with Spark?
Hi, What would be the best way to get percentiles from a Spark RDD? I can see JavaDoubleRDD or MLlib's MultivariateStatisticalSummary https://spark.apache.org/docs/latest/mllib-statistics.html provide the mean() but not percentiles. Thank you! Horace -- View this message in context: http
Re: Rdd of Rdds
on an RDD of RDDs. Depending on one's needs, one could also consider the matrix (RDD[Vector]) operations provided by MLLib, such as https://spark.apache.org/docs/latest/mllib-statistics.html - To unsubscribe, e-mail: user-unsubscr
Re: Use RDD like a Iterator
Call RDD.toLocalIterator()? https://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html On Wed, Oct 29, 2014 at 4:15 AM, Dai, Kevin yun...@ebay.com wrote: Hi, ALL I have a RDD[T], can I use it like a iterator. That means I can compute every element of this RDD lazily
StreamingLinearRegressionWithSGD
Hi Gurus, I did not look at the code yet. I wonder if StreamingLinearRegressionWithSGD http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/regression/StreamingLinearRegressionWithSGD.html is equivalent to LinearRegressionWithSGD http://spark.apache.org/docs/latest/api/java/org
[jira] [Created] (SPARK-5409) Broken link in documentation
Reporter: Mauro Pirrone Priority: Minor https://spark.apache.org/docs/1.2.0/streaming-kafka-integration.html See the API docs and the example. Link to example is broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332
Re: Upgrade to Spark 1.2.1 using Guava
confused about what config you're changing. I'm suggesting using: spark-submit --conf spark.executor.extraClassPath=/guava.jar blah -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional
Re: SparkStream saveAsTextFiles()
Structure seems fine. Only need to type at the end of your program: ssc.start(); ssc.awaitTermination(); Check method arguments. I advise you to check the spark java api streaming. https://spark.apache.org/docs/1.3.0/api/java/ Regards. -- View this message in context: http://apache-spark
Re: spark sql, creating literal columns in java.
This should work from java too: http://spark.apache.org/docs/1.3.1/api/java/index.html#org.apache.spark.sql.functions$ On Tue, May 5, 2015 at 4:15 AM, Jan-Paul Bultmann janpaulbultm...@me.com wrote: Hey, What is the recommended way to create literal columns in java? Scala has the `lit
Re: Performance tuning in Spark SQL.
Please see below link for the ways available https://spark.apache.org/docs/1.3.1/sql-programming-guide.html#performance-tuning For example, reduce spark.sql.shuffle.partitions from 200 to 10 could improve the performance significantly -- View this message in context: http://apache-spark-user
Re: How to set environment of worker applications
Check for spark.driver.extraJavaOptions and spark.executor.extraJavaOptions in the following article. I think you can use -D to pass system vars: spark.apache.org/docs/latest/configuration.html#runtime-environment Hi, I am starting a spark streaming job in standalone mode with spark-submit
Re: Create RDD from output of unix command
You may want to look into using the pipe command .. http://blog.madhukaraphatak.com/pipe-in-spark/ http://spark.apache.org/docs/0.6.0/api/core/spark/rdd/PipedRDD.html -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Create-RDD-from-output-of-unix-command
Re: Python Kafka support?
Hi Darren, Functionality like messageHandler is missing in python API, still not included in version 1.5.1. Thanks Jerry On Wed, Nov 11, 2015 at 7:37 AM, Darren Govoni <dar...@ontrenet.com> wrote: > Hi, > I read on this page > http://spark.apache.org/docs/latest/
Re: Ranger-like Security on Spark
r a > >> Ranger plugin or something similar to Spark. The target is to > >> differentiate users and their privileges when reading and writing data to > >> HDFS? Is Kerberos my only option then? > >> > >> Kind regards, Daniel. > >> --
[jira] [Created] (SPARK-12661) Drop Python 2.6 support in PySpark
Reporter: Davies Liu 1. stop testing with 2.6 2. remove the code for python 2.6 see discussion : https://www.mail-archive.com/user@spark.apache.org/msg43423.html -- This message was sent by Atlassian JIRA (v6.3.4#6332
[jira] [Created] (SPARK-15966) Fix markdown for Spark Monitoring
Components: Documentation Affects Versions: 2.0.0 Reporter: Dhruve Ashar Priority: Trivial The markdown for Spark monitoring needs to be fixed. http://spark.apache.org/docs/2.0.0-preview/monitoring.html -- This message was sent by Atlassian JIRA (v6.3.4
Re: Does filter on an RDD scan every data item ?
Looks like this has been supported from 1.4 release :) https://spark.apache.org/docs/1.4.1/api/scala/index.html#org.apache.spark.rdd.OrderedRDDFunctions -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Does-filter-on-an-RDD-scan-every-data-item
Re: udf StructField to JSON String
Have you looked at DataFrame.write.json( path )? https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameWriter > On Mar 11, 2016, at 7:15 AM, Caires Vinicius <caire...@gmail.com> wrote: > > I have one DataFrame with nested StructField and I
Re: Spark ML Interaction
Could you create a JIRA to add an example and documentation? Thanks On Tue, 8 Mar 2016 at 16:18, amarouni <amaro...@talend.com> wrote: > Hi, > > Did anyone here manage to write an example of the following ML feature > transformer > > http://spark.apache.org/docs/latest/a
Re: please add Christchurch Apache Spark Meetup Group
(I have the site's svn repo handy, so I just added it.) On Wed, Mar 2, 2016 at 5:16 PM, Raazesh Sainudiin <raazesh.sainud...@gmail.com> wrote: > Hi, > > Please add Christchurch Apache Spark Meetup Group to the community list > here: > http://spark.apache.org/community.ht
Re: Where to set properties for the retainedJobs/Stages?
You can set them in spark-defaults.conf See also https://spark.apache.org/docs/latest/configuration.html#spark-ui On Fri, Apr 1, 2016 at 8:26 AM, Max Schmidt <m...@datapath.io> wrote: > Can somebody tell me the interaction between the properties: > > spark.
Re: yarn-cluster
Hi, this is a good spot to start for Spark and YARN. https://spark.apache.org/docs/1.5.0/running-on-yarn.html specific to the version you are on, you can toggle between pages. - Neelesh S. Salian Cloudera -- View this message in context: http://apache-spark-user-list.1001560.n3
[jira] [Resolved] (SPARK-15228) pyspark.RDD.toLocalIterator Documentation
l > > There is a little bug in the parsing of the documentation of > http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.toLocalIterator -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To
[jira] [Created] (SPARK-15228) pyspark.RDD.toLocalIterator Documentation
Type: Documentation Reporter: Ignacio Tartavull Priority: Trivial There is a little bug in the parsing of the documentation of http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.toLocalIterator -- This message was sent by Atlassian JIRA (v6.3.4#6332
[GitHub] spark issue #16816: Code style improvement
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16816 @zhoucen please close this PR and read http://spark.apache.org/contributing.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well
[GitHub] spark issue #16863: Swamidass & Baldi Approximations
Github user uncleGen commented on the issue: https://github.com/apache/spark/pull/16863 Please review http://spark.apache.org/contributing.html before opening a pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub
[GitHub] spark issue #16676: delete useless var “j”
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16676 Merged to master. Please read http://spark.apache.org/contributing.html for next time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well
[GitHub] spark issue #8632: Update README.md
Github user packtpartner commented on the issue: https://github.com/apache/spark/pull/8632 Hi @srowen , where is the Github repository to feature books on http://spark.apache.org/documentation.html ? --- If your project is set up for it, you can reply to this email and have your
[GitHub] spark issue #16638: spark-19115
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16638 Could you follow the title requirement in http://spark.apache.org/contributing.html? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub
Re: CSV escaping not working
lto:ko...@tresata.com>> Date: Thursday, October 27, 2016 at 12:49 PM To: "Jain, Nishit" <nja...@underarmour.com<mailto:nja...@underarmour.com>> Cc: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apa
RE: Spark SQL join and subquery
unsubscribe -Original Message- From: neil90 [mailto:neilp1...@icloud.com] Sent: Thursday, November 17, 2016 8:26 AM To: user@spark.apache.org Subject: Re: Spark SQL join and subquery What version of Spark are you using? I believe this was fixed in 2.0 -- View this message in context
[GitHub] spark issue #17309: same rdd rule testcase
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17309 See http://spark.apache.org/contributing.html I'm not clear this adds any value? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well
[GitHub] spark issue #17556: [SPARK-16957][MLlib] Use weighted midpoints for split va...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17556 http://spark.apache.org/docs/latest/building-spark.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have
RE: RDD functions using GUI
Ping... wonder why there aren't any such drag-n-drop GUI tool for creating batch query scripts? Thanks From: Ke Yang (Conan) Sent: Monday, April 17, 2017 5:31 PM To: 'dev@spark.apache.org' <dev@spark.apache.org> Subject: RDD functions using GUI Hi, Are there drag and drop GUI (cod
[GitHub] spark issue #18836: Update SortMergeJoinExec.scala
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18836 You didn't read the link above, I take it? http://spark.apache.org/contributing.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub
[GitHub] spark issue #18414: Update status of application to RUNNING if executors are...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/18414 please fix up the PR title: http://spark.apache.org/contributing.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your
user-unsubscr...@spark.apache.org
user-unsubscr...@spark.apache.org From: 萝卜丝炒饭 [mailto:1427357...@qq.com] Sent: Sunday, May 21, 2017 8:15 PM To: user <user@spark.apache.org> Subject: Are tachyon and akka removed from 2.1.1 please HI all, Iread some paper about source code, the paper base on version 1.2. they
[GitHub] spark issue #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC connection t...
? Thanks! https://spark.apache.org/third-party-projects.html --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19433: [SPARK-3162] [MLlib][WIP] Add local tree training for de...
://spark.apache.org/contributing.html) -- have we stopped using the WIP tag? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19429: [SPARK-20055] [Docs] Added documentation for loading csv...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19429 When I opened a JIRA, I thought a chapter such as https://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets. This chapter, `Manually Specifying Options`, looks describing
[GitHub] spark issue #19154: Fix DiskBlockManager crashing when a root local folder h...
://spark.apache.org/contributing.html --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19343: [SPARK-22121][SQL] Correct database location for namenod...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19343 @squito Thank you! Instead of changing the source codes, could we just update the document https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#hive-tables ? This might
Re: Welcoming Tejas Patil as a Spark committer
Congratulation Tejas! Kazuaki Ishizaki From: Matei Zaharia <matei.zaha...@gmail.com> To: "dev@spark.apache.org" <dev@spark.apache.org> Date: 2017/09/30 04:58 Subject:Welcoming Tejas Patil as a Spark committer Hi all, The Spark PMC recently added Teja
[GitHub] spark issue #19485: [SPARK-20055] [Docs] Added documentation for loading csv...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19485 @jomach and @HyukjinKwon I did not generate the doc. I think we should follow what we did for JDBC. http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other
[GitHub] spark issue #19714: [SPARK-22489][SQL] Shouldn't change broadcast join build...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19714 LGTM Thanks! Merged to master. Could you submit a follow-up PR to document the behavior changes in migration section of Spark SQL? https://spark.apache.org/docs/latest/sql
Re: [01/51] [partial] spark-website git commit: 2.2.1 generated doc
tation pages under > http://spark.apache.org/docs/latest/ (that is still 2.2.0). Is this being > worked on? > > http://spark.apache.org/docs/2.2.1 is available and shows the proper > version, but not http://spark.apache.org/docs/latest :( > > Pozdrawiam, > Jacek Laskowski > > >
[GitHub] spark issue #19996: [MINOR][DOC] Fix the link of 'Getting Started'
by tab). About 75-100 broken links are related to "http(s)://spark.apache.org/docs/latest". --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20219 `NullType` is not well supported in almost all the data sources. We did not mention it in our doc https://spark.apache.org/docs/latest/sql-programming-guide.html cc @cloud-fan
[GitHub] spark issue #19290: [SPARK-22063][R] Fixes lint check failures in R by lates...
://spark.apache.org/docs/latest/index.html#downloading --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20334: How to check registered table name.
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20334 Hey @AtulKumVerma, questions should go to mailing list usually. See http://spark.apache.org/community.html. I believe you can have a better answer from there. Pull request from
[GitHub] spark issue #20254: [SPARK-23062][SQL] Improve EXCEPT documentation
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20254 @henryr Since Spark 2.3, Spark SQL documents all the behavior changes in [Migration Guides](https://spark.apache.org/docs/latest/sql-programming-guide.html#migration-guide). Hopefully, this can
[GitHub] spark issue #21961: Spark 20597
is one of the PR categories shown at spark-prs.appspot.com and Title may be the JIRAâs title or a more specific title describing the PR itself. Take a look to this helpful document : https://spark.apache.org/contributing.html
[GitHub] spark issue #22177: stages in wrong order within job page DAG chart
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/22177 Please change title to "[SPARK-25199][Web UI] XXX " as we described in http://spark.apache.org/contributing.html. ``` check the DAG chart in job page. ``` Could yo
[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21589 AFAIK, we always have num of executor and then num of core per executor right? https://spark.apache.org/docs/latest/configuration.html#execution-behavior maybe we should have
[GitHub] spark issue #22339: SPARK-17159 Significant speed up for running spark strea...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22339 Hi, @ScrapCodes . Could you do the followings? - Update the title to `[SPARK-17159][SS]...` - Remove `Please review http://spark.apache.org/contributing.html ` from PR description
[GitHub] spark issue #22367: [SPARK-17916][SPARK-25241][SQL][FOLLOWUP] Fix empty stri...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22367 Usually we merge into master and backport to other branches when it's needed. https://spark.apache.org/contributing.html > 5. Open a pull request against the master bra
[GitHub] spark issue #20618: [SPARK-23329][SQL] Fix documentation of trigonometric fu...
Github user misutoth commented on the issue: https://github.com/apache/spark/pull/20618 @felixcheung, I have started a mail thread on d...@spark.apache.org with title _Help needed in R documentation generation_ because I did not feel it is directly related to this PR. Thanks for your
[GitHub] spark issue #17466: [SPARK-14681][ML] Added getter for impurityStats
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/17466 @shaynativ Sorry for the inactivity here. Btw, for the JIRA & PR title question above, I'd recommend checking out http://spark.apache.org/contributing.html Since @WeichenXu123 op
[GitHub] spark issue #21057: [MINOR][PYTHON] 2 Improvements to Pyspark docs
. Usually, JIRA is made first. See also https://spark.apache.org/contributing.html. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20897: [MINOR][DOC] Fix a few markdown typos
} -## `spark.mllib` supported models --- End diff -- backquotes in mds cause display problems (see http://spark.apache.org/docs/latest/mllib-pmml-model-export.html) --- - To unsubscribe, e-mail: reviews-unsubscr
[GitHub] spark issue #20893: [SPARK-23785][LAUNCHER] LauncherBackend doesn't check st...
Github user sahilTakiar commented on the issue: https://github.com/apache/spark/pull/20893 Ok, I'll work on writing a test for `SparkLauncherSuite`. The test added here was meant to cover the race condition mentioned [here|https://spark.apache.org/docs/2.3.0/sql-programming
[GitHub] spark issue #20889: [MINOR][DOC] Fix ml-guide markdown typos
of the later issue, so i found it. [http://spark.apache.org/docs/latest/ml-guide.html#breaking-changes](url) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h
[GitHub] spark issue #22852: [SPARK-25023] Clarify Spark security documentation
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22852 I think these are good changes. In a separate PR for the versions-specific docs, we could add a similar note to https://spark.apache.org/docs/latest/spark-standalone.html as much of the security
[GitHub] spark issue #22840: [SPARK-25840][BUILD] `make-distribution.sh` should not f...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22840 @srowen . It's a documented feature. - http://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution I know that you're not against it, but Spark 2.4.0
[jira] [Created] (SPARK-25933) Fix pstats reference for spark.python.profile.dump in configuration.md
: Spark Issue Type: Documentation Components: Documentation Affects Versions: 2.3.2 Reporter: Alex Hagerman Fix For: 2.3.2 ptats.Stats() should be pstats.Stats() in https://spark.apache.org/docs/latest/configuration.html for spark.python.profile.dump
[GitHub] spark issue #22822: [SPARK-25678] Requesting feedback regarding a prototype ...
Github user UtkarshMe commented on the issue: https://github.com/apache/spark/pull/22822 I did send the proposal on d...@spark.apache.org mailing list (twice). But unfortunately, I got no response so I opened a JIRA ticket about it about 20 days back and now opened a pull request
[jira] [Created] (SPARK-25991) Update binary for 2.4.0 release
Feature Components: Spark Core Affects Versions: 2.4.0 Reporter: Vladimir Tsvetkov Archive with 2.4.0 release contains old binaries https://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v7.6.3#76005
[GitHub] spark issue #22606: [SPARK-25592] Setting version to 3.0.0-SNAPSHOT
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22606 You mean http://spark.apache.org/versioning-policy.html and the reference to 2.4? I think that's still valid. When 2.4 is released, I'd propose to change that to refer to 3.0 being released .. I
[GitHub] spark issue #22593: [Streaming][DOC] Fix typo & format in DataStreamWriter.s...
Github user niofire commented on the issue: https://github.com/apache/spark/pull/22593 From https://spark.apache.org/docs/2.3.2/api/java/org/apache/spark/sql/streaming/DataStreamWriter.html 