After this change:
[SPARK-12681] [SQL] split IdentifiersParser.g into two files
the biggest file under
sql/catalyst/src/main/antlr3/org/apache/spark/sql/catalyst/parser is
SparkSqlParser.g
Maybe split SparkSqlParser.g up as well ?
On Thu, Jan 28, 2016 at 5:21 AM, Iulian Dragoș
wrote:
> Hi,
Does this mean the following Jenkins builds can be disabled ?
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-MAVEN-SCALA-2.11/
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.11/
Cheers
On Sat, Jan 3
w.r.t. protobuf-java version mismatch, I wonder if you can rebuild Spark
with the following change (using maven):
http://pastebin.com/fVQAYWHM
Cheers
On Sat, Jan 30, 2016 at 12:49 AM, Yash Sharma wrote:
> Hi All,
> I have a quick question if anyone has experienced this here.
>
> I have been tr
The following jobs have been established for build against Scala 2.10:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-MAVEN-SCALA-2.10/
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.10/
FYI
On Mon,
SPARK-12624 has been resolved.
According to Wenchen, SPARK-12783 is fixed in 1.6.0 release.
Are there other blockers for Spark 1.6.1 ?
Thanks
On Wed, Jan 13, 2016 at 5:39 PM, Michael Armbrust
wrote:
> Hey All,
>
> While I'm not aware of any critical issues with 1.6.0, there are several
> corne
w.r.t. running Spark on YARN, there are a few outstanding issues. e.g.
SPARK-11182 HDFS Delegation Token
See also the comments under SPARK-12279
FYI
On Mon, Feb 1, 2016 at 1:02 PM, eugene miretsky
wrote:
> When having multiple users sharing the same Spark cluster, it's a good
> idea to isolat
For #1, a brief search landed the following:
core/src/main/scala/org/apache/spark/SparkConf.scala:
DeprecatedConfig("spark.rpc", "2.0", "Not used any more.")
core/src/main/scala/org/apache/spark/SparkConf.scala:
"spark.rpc.numRetries" -> Seq(
core/src/main/scala/org/apache/spark/SparkConf.scala:
Assuming your change is based on hadoop-2 branch, you can use 'mvn install'
command which would put artifacts under 2.8.0-SNAPSHOT subdir in your local
maven repo.
Here is an example:
~/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.8.0-SNAPSHOT
Then you can use the following command to build Spa
Congratulations, Herman and Wenchen.
On Mon, Feb 8, 2016 at 9:15 AM, Matei Zaharia
wrote:
> Hi all,
>
> The PMC has recently added two new Spark committers -- Herman van Hovell
> and Wenchen Fan. Both have been heavily involved in Spark SQL and Tungsten,
> adding new features, optimizations and
Do you mind pastebin'ning code snippet and exception one more time - I
couldn't see them in your original email.
Which Spark release are you using ?
On Tue, Feb 9, 2016 at 11:55 AM, rakeshchalasani
wrote:
> Hi All:
>
> I am getting an "UnsupportedOperationException" when trying to alias an
> ar
sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
> at
> org.apache.spark.repl.SparkIMain.l
gt; |arrayCol|
> ++
> | [0, 1]|
> | [1, 2]|
> | [2, 3]|
> | [3, 4]|
> | [4, 5]|
> | [5, 6]|
> | [6, 7]|
> | [7, 8]|
> | [8, 9]|
> | [9, 10]|
> ++
>
>
>
> On Tue, Feb 9, 2016 at 4:52 PM Ted Yu wrote:
>
>> How about chang
Hdfs class is in hadoop-hdfs-XX.jar
Can you check the classpath to see if the above jar is there ?
Please describe the command lines you used for building hadoop / Spark.
Cheers
On Thu, Feb 11, 2016 at 5:15 PM, Charlie Wright
wrote:
> I am having issues trying to run a test job on a built ver
Have you seen this thread ?
http://stackoverflow.com/questions/24402737/how-to-read-gz-files-in-spark-using-wholetextfiles
On Tue, Feb 16, 2016 at 2:17 AM, Deepak Gopalakrishnan
wrote:
> Hello,
>
> I'm reading S3 files using wholeTextFiles() . My files are gzip format but
> the names of the fil
The referenced benchmark is in Chinese. Please provide English version so
that more people can understand.
For item 7, looks like the speed of ingest is much slower compared to using
Parquet.
Cheers
On Mon, Feb 22, 2016 at 6:12 AM, 开心延年 wrote:
> 1.ya100 is not only the invert index ,but also
When you click on Create, you're brought to 'Create Issue' dialog where you
choose Project Spark.
Component should be MLlib.
Please see also:
http://search-hadoop.com/m/q3RTtmsshe1W6cH22/spark+pull+template&subj=pull+request+template
On Mon, Feb 22, 2016 at 6:45 PM, Pierson, Oliver C wrote:
>
In hbase, there is hbase-spark module which supports bulk load.
This module is to be backported in the upcoming 1.3.0 release.
There is some pending work, such as HBASE-15271 .
FYI
On Fri, Feb 26, 2016 at 8:50 AM, Renu Yadav wrote:
> Has anybody implemented bulk load into hbase using spark?
>
Looking at
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
*WARNING* Generating the caller class information is slow. Thus, use should
be avoided unless execution speed is not an issue.
On Sat, Feb 27, 2016 at 12:40 PM, Prabhu Joseph
wrote:
> Hi All,
>
> Whe
Since majority of code is written in Scala which is not analyzed by Coverity,
the efficacy of the tool seems limited.
> On Mar 4, 2016, at 2:34 AM, Sean Owen wrote:
>
> https://scan.coverity.com/projects/apache-spark-2f9d080d-401d-47bc-9dd1-7956c411fbb4?tab=overview
>
> This has to be run man
suggesting anyone run it regularly,
> but one run to catch some bugs is useful.
>
> I've already triaged ~70 issues there just in the Java code, of which
> a handful are important.
>
> On Fri, Mar 4, 2016 at 12:18 PM, Ted Yu wrote:
> > Since majority of code is written
ashCode
>
> On Fri, Mar 4, 2016 at 2:52 PM, Ted Yu wrote:
> > Last time I checked there wasn't high impact defects.
> >
> > Mind pointing out the defects you think should be fixed ?
> >
> > Thanks
> >
> > On Fri, Mar 4, 2016 at 4:35 AM, Sean Owen w
Please stack trace, code snippet, etc in the JIRA you created so that
people can reproduce what you saw.
On Sat, Mar 5, 2016 at 7:02 AM, Dhaval Modi wrote:
>
> Regards,
> Dhaval Modi
> dhavalmod...@gmail.com
>
> -- Forwarded message --
> From: Dhaval Modi
> Date: 5 March 2016 at
Josh:
SerializerInstance and SerializationStream would also become private[spark],
right ?
Thanks
On Mon, Mar 7, 2016 at 6:57 PM, Josh Rosen wrote:
> Does anyone implement Spark's serializer interface
> (org.apache.spark.serializer.Serializer) in your own third-party code? If
> so, please let m
com> wrote:
> Looks like the other packages may also be corrupt. I’m getting the same
> error for the Spark 1.6.1 / Hadoop 2.4 package.
>
>
> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz
>
> Nick
>
>
> On Wed, Mar 16, 2016 at 8:28 PM
On Linux, I got:
$ tar zxf spark-1.6.1-bin-hadoop2.6.tgz
gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
On Wed, Mar 16, 2016 at 5:15 PM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:
>
> https:
:48 PM, "Nicholas Chammas"
> wrote:
>
>> Looks like the other packages may also be corrupt. I’m getting the same
>> error for the Spark 1.6.1 / Hadoop 2.4 package.
>>
>>
>> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz
>&
Do you have performance numbers to backup this proposal for cogroup
operation ?
Thanks
On Mon, Mar 21, 2016 at 1:06 AM, JOAQUIN GUANTER GONZALBEZ <
joaquin.guantergonzal...@telefonica.com> wrote:
> Hello devs,
>
>
>
> I have found myself in a situation where Spark is doing sub-optimal
> computat
>From the error message, it seems some artifacts from Scala 2.10.4 were left
around.
FYI maven 3.3.9 is required for master branch.
On Tue, Mar 22, 2016 at 3:07 AM, Allen wrote:
> Hi,
>
> I am facing an error when doing compilation from IDEA, please see the
> attached. I fired the build process
The warning was added by:
SPARK-12757 Add block-level read/write locks to BlockManager
On Sun, Mar 27, 2016 at 12:24 PM, salexln wrote:
> HI all,
>
> I started testing my code (https://github.com/salexln/FinalProject_FCM)
> with the latest Spark available in GitHub,
> and when I run it I get th
I guess you have looked at MemoryManager#pageSizeBytes where
the "spark.buffer.pageSize" config can override default page size.
FYI
On Mon, Mar 28, 2016 at 12:07 PM, Steve Johnston <
sjohns...@algebraixdata.com> wrote:
> I'm attempting to address an OOM issue. I saw referenced in
> java.lang.Out
Hi,
Based on master branch refreshed today, I issued 'git clean -fdx' first.
Then this command:
build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6
-Dhadoop.version=2.7.0 package -DskipTests
I got the following error:
scala> sql("explain codegen select 'a' as a group by 1").head
org.
.
>
>
> On Sun, Apr 3, 2016 at 9:38 PM, Jacek Laskowski wrote:
>
>> Hi,
>>
>> Looks related to the recent commit...
>>
>> Repository: spark
>> Updated Branches:
>> refs/heads/master 2262a9335 -> 1f0c5dceb
>>
>> [SPARK-14350][
gards,
>
> Herman van Hövell
>
> 2016-04-04 12:15 GMT+02:00 Ted Yu :
>
>> Could the error I encountered be due to missing import(s) of implicit ?
>>
>> Thanks
>>
>> On Sun, Apr 3, 2016 at 9:42 PM, Reynold Xin wrote:
>>
>>> Works fo
bq. the modifications do not touch the scheduler
If the changes can be ported over to 1.6.1, do you mind reproducing the
issue there ?
I ask because master branch changes very fast. It would be good to narrow
the scope where the behavior you observed started showing.
On Mon, Apr 4, 2016 at 6:12
; >
>>> > On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky
>>> wrote:
>>> >>
>>> >> I just experienced the issue, however retrying the download a second
>>> >> time worked. Could it be that there is some load balancer/c
on't you wipe everything out and try again?
>
> On Monday, April 4, 2016, Ted Yu wrote:
>
>> The commit you mentioned was made Friday.
>> I refreshed workspace Sunday - so it was included.
>>
>> Maybe this was related:
>>
>> $ bin/spark-shell
>&g
Looks like the import comes from
repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoop.scala :
processLine("import sqlContext.sql")
On Mon, Apr 4, 2016 at 5:16 PM, Jacek Laskowski wrote:
> Hi Spark devs,
>
> I'm unsure if what I'm seeing is correct. I'd appreciate any input
> to
Raymond:
Did "namenode" appear in any of the Spark config files ?
BTW Scala 2.11 is used by the default build.
On Tue, Apr 5, 2016 at 6:22 AM, Raymond Honderdors <
raymond.honderd...@sizmek.com> wrote:
> I can see that the build is successful
>
> (-Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phi
The next line should give some clue:
expectCorrectException { ssc.transform(Seq(ds), transformF) }
Closure shouldn't include return.
On Tue, Apr 5, 2016 at 3:40 PM, Jacek Laskowski wrote:
> Hi,
>
> In
> https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/st
Josh:
You may have noticed the following error (
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/566/console
):
[error] javac: invalid source release: 1.8
[error] Usage: javac
[error] use -help for a list of possible options
On Tue, Apr 5, 2016 at 2:14 PM, Josh Ro
Looking at recent
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7
builds, there was no such error.
I don't see anything wrong with the code:
usage = "_FUNC_(str) - " +
"Returns str, with the first letter of each word in uppercase, all
other letters in " +
Mind
i
>
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Tue, Apr 5, 2016 at 8:41 PM, Ted Yu wrote:
> > Looking at recent
> >
> https://amplab.cs.berkeley.e
Front
>>> (i.e.
>>> >> the “direct download” option on spark.apache.org) are also corrupt.
>>> >>
>>> >> Btw what’s the correct way to verify the SHA of a Spark package? I’ve
>>> tried
>>> >> a few commands on working pack
The broken build was caused by the following:
[SPARK-14462][ML][MLLIB] add the mllib-local build to maven pom
See
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/607/
FYI
On Sat, Apr 9, 2016 at 12:01 PM, Jacek Laskowski wrote:
> Hi,
>
> Is this me or the build is
Sent PR:
https://github.com/apache/spark/pull/12276
I was able to get build going past mllib-local module.
FYI
On Sat, Apr 9, 2016 at 12:40 PM, Ted Yu wrote:
> The broken build was caused by the following:
>
> [SPARK-14462][ML][MLLIB] add the mllib-local build to maven pom
>
&
I see the following code toward the end of the method:
// Unpersist the RDDs hidden by newly-materialized RDDs
oldMessages.unpersist(blocking = false)
prevG.unpersistVertices(blocking = false)
prevG.edges.unpersist(blocking = false)
Wouldn't the above achieve same effect ?
Gentle ping: spark-1.6.1-bin-hadoop2.4.tgz from S3 is still corrupt.
On Wed, Apr 6, 2016 at 12:55 PM, Josh Rosen
wrote:
> Sure, I'll take a look. Planning to do full verification in a bit.
>
> On Wed, Apr 6, 2016 at 12:54 PM Ted Yu wrote:
>
>> Josh:
>> Can you ch
I assume you tested 2.0 with SPARK-12181 .
Related code from Platform.java if java.nio.Bits#unaligned() throws
exception:
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPa
unaligned);*
> }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
added
>
> Cheers,
>
>
>
>
> From:Ted Yu
> To:Adam Roberts/UK/IBM@IBMGB
> Cc:"dev@spark.apache.org"
> Date:15/04/2016 16:43
> Subject:Re: BytesToBytes and unaligned memory
> --
>
&g
unaligned memory access on a platform where unaligned memory access is
> definitely not supported for shorts/ints/longs.
>
> if these tests continue to pass then I think the Spark tests don't
> exercise unaligned memory access, cheers
>
>
>
>
>
>
>
> From:
I had one PR which got merged after 3 months.
If the inactivity was due to contributor, I think it can be closed after 30
days.
But if the inactivity was due to lack of review, the PR should be kept open.
On Mon, Apr 18, 2016 at 12:17 PM, Cody Koeninger wrote:
> For what it's worth, I have defi
filtered for non-mergeable PRs or instead left a comment
> asking the author to respond if they are still available to move the PR
> forward - and close the ones where they don't respond for a week?
>
> Just a suggestion.
> On Monday, April 18, 2016, Ted Yu wrote:
>
>>
en but the cost to reopen is approximately zero (i.e.
> click a button on the pull request).
>
>
> On Mon, Apr 18, 2016 at 12:41 PM, Ted Yu wrote:
>
>> bq. close the ones where they don't respond for a week
>>
>> Does this imply that the script understands re
ing people look at the pull requests that have been inactive
>>> for a
>>> > long time. That seems equally likely (or unlikely) as committers
>>> looking at
>>> > the recently closed pull requests.
>>> >
>>> > In either case, most
s; a higher barrier to contributing; a combination
>> thereof; etc...
>>
>> Also relevant: http://danluu.com/discourage-oss/
>>
>> By the way, some people noted that closing PRs may discourage
>> contributors. I think our open PR count alone is very discouraging. Und
Corrected typo in subject.
I want to note that the hbase-spark module in HBase is incomplete. Zhan has
several patches pending review.
hbase-spark module is currently only in master branch which would be
released as 2.0
However the release date for 2.0 is unclear - probably half a year from now.
On Tue, Apr 19, 2016 at 10:23 AM, Marcelo Vanzin
wrote:
> On Tue, Apr 19, 2016 at 10:20 AM, Ted Yu wrote:
> > I want to note that the hbase-spark module in HBase is incomplete. Zhan
> has
> > several patches pending review.
>
> I wouldn't call it "incomplete
bq. create a separate tarball for them
Probably another thread can be started for the above.
I am fine with it.
On Tue, Apr 19, 2016 at 10:34 AM, Marcelo Vanzin
wrote:
> On Tue, Apr 19, 2016 at 10:28 AM, Reynold Xin wrote:
> > Yea in general I feel examples that bring in a large amount of
> de
se's input
> formats), which makes it not very useful as a blueprint for developing
> HBase apps with Spark.
>
> On Tue, Apr 19, 2016 at 10:28 AM, Ted Yu wrote:
> > bq. I wouldn't call it "incomplete".
> >
> > I would call it incomplete.
> >
&g
'bq.' is used in JIRA to quote what other people have said.
On Tue, Apr 19, 2016 at 10:42 AM, Reynold Xin wrote:
> Ted - what's the "bq" thing? Are you using some 3rd party (e.g. Atlassian)
> syntax? They are not being rendered in email.
>
>
> On Tue,
gt;
> While you're at it, here's some much better documentation, from the
> HBase project themselves, than what the Spark example provides:
> http://hbase.apache.org/book.html#spark
>
> On Tue, Apr 19, 2016 at 10:41 AM, Ted Yu wrote:
> > bq. it's actually in use
too
> many dependencies for something that is not really useful, is why I'm
> suggesting removing it.
>
>
> On Tue, Apr 19, 2016 at 10:47 AM, Ted Yu wrote:
> > There is an Open JIRA for fixing the documentation: HBASE-15473
> >
> > I would say the refguide li
> On Tue, Apr 19, 2016 at 1:59 PM, Ted Yu wrote:
>
>> bq. HBase's current support, even if there are bugs or things that still
>> need to be done, is much better than the Spark example
>>
>> In my opinion, a simple example that works is better than a buggy
:
> On Tue, Apr 19, 2016 at 11:07 AM, Ted Yu wrote:
>
>> The same question can be asked w.r.t. examples for other projects, such
>> as flume and kafka.
>>
>
> The main difference being that flume and kafka integration are part of
> Spark itself. HBase integratio
Interesting.
For #3:
bq. reading data from,
I guess you meant reading from disk.
On Wed, Apr 20, 2016 at 10:45 AM, atootoonchian wrote:
> Current spark logging mechanism can be improved by adding the following
> parameters. It will help in understanding system bottlenecks and provide
> useful
Interesting analysis.
Can you log a JIRA ?
> On Apr 21, 2016, at 11:07 AM, atootoonchian wrote:
>
> SQL query planner can have intelligence to push down filter commands towards
> the storage layer. If we optimize the query planner such that the IO to the
> storage is reduced at the cost of run
Zhan:
I have mentioned the JIRA numbers in the thread starting with (note the
typo in subject of this thread):
RFC: Remove ...
On Thu, Apr 21, 2016 at 1:28 PM, Zhan Zhang wrote:
> FYI: There are several pending patches for DataFrame support on top of
> HBase.
>
> Thanks.
>
> Zhan Zhang
>
> On A
Interesting.
bq. details of execution for 10 and 100 scale factor input
Looks like some chart (or image) didn't go through.
FYI
On Mon, Apr 25, 2016 at 12:50 PM, Ali Tootoonchian wrote:
> Caching shuffle RDD before the sort process improves system performance.
> SQL
> planner can be intellige
Here is the body of StreamFileInputFormat#setMinPartitions :
def setMinPartitions(context: JobContext, minPartitions: Int) {
val totalLen =
listStatus(context).asScala.filterNot(_.isDirectory).map(_.getLen).sum
val maxSplitSize = math.ceil(totalLen / math.max(minPartitions,
1.0)).toLong
> Hi Ted,
>
>
>
> I have 36 files of size ~600KB and the rest 74 are about 400KB.
>
>
>
> Is there a workaround rather than changing Sparks code?
>
>
>
> Best regards, Alexander
>
>
>
> *From:* Ted Yu [mailto:yuzhih...@gmail.com]
> *Sent:* Tu
On May 2, 2016 12:09 AM, "Ted Yu" wrote:
>> Using commit hash 90787de864b58a1079c23e6581381ca8ffe7685f and Java 1.7.0_67
>> , I got:
>>
>> scala> val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List
gt; Created issue:
>> https://issues.apache.org/jira/browse/SPARK-15062
>>
>> On Mon, May 2, 2016 at 6:48 AM, Ted Yu wrote:
>>
>>> I tried the same statement using Spark 1.6.1
>>> There was no error with default memory setting.
>>>
>>> Suggest logging a
Have you tried the following ?
scala> import spark.implicits._
import spark.implicits._
scala> spark
res0: org.apache.spark.sql.SparkSession =
org.apache.spark.sql.SparkSession@323d1fa2
Cheers
On Tue, May 3, 2016 at 9:16 AM, Koert Kuipers wrote:
> with the introduction of SparkSession SQLCont
PR #10572 was listed twice.
In the future, is it possible to include the contributor's handle beside
the PR number so that people can easily recognize their own PR ?
Thanks
On Fri, May 6, 2016 at 8:45 AM, Hyukjin Kwon wrote:
> Hi all,
>
>
> This was similar with the proposal of closing PRs bef
I assume there were supposed to be images following this line (which I
don't see in the email thread):
bq. Let’s look at details of execution for 10 and 100 scale factor input
Consider using 3rd party image site.
On Sun, May 8, 2016 at 5:17 PM, Ali Tootoonchian wrote:
> Thanks for your comment
Please see this thread:
http://search-hadoop.com/m/q3RTt9XAz651PiG/Adhoc+queries+spark+streaming&subj=Re+Adhoc+queries+on+Spark+2+0+with+Structured+Streaming
> On May 11, 2016, at 1:47 AM, Ofir Manor wrote:
>
> Hi,
> I'm trying out Structured Streaming from current 2.0 branch.
> Does the branch
In master branch, behavior is the same.
Suggest opening a JIRA if you haven't done so.
On Wed, May 11, 2016 at 6:55 AM, Tony Jin wrote:
> Hi guys,
>
> I have a problem about spark DataFrame. My spark version is 1.6.1.
> Basically, i used udf and df.withColumn to create a "new" column, and then
Which release of Spark / Hive are you using ?
Cheers
> On May 18, 2016, at 6:12 AM, JaeSung Jun wrote:
>
> Hi,
>
> I'm working on custom data source provider, and i'm using fully qualified
> table name in FROM clause like following :
>
> SELECT user. uid, dept.name
> FROM userdb.user user, d
Yash:
Can you share the JVM parameters you used ?
How many partitions are there in your data set ?
Thanks
On Fri, May 20, 2016 at 5:59 PM, Reynold Xin wrote:
> It's probably due to GC.
>
> On Fri, May 20, 2016 at 5:54 PM, Yash Sharma wrote:
>
>> Hi All,
>> I am here to get some expert advice
The following line was repeated twice:
- For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
Did you intend to cover JDK 8 ?
Cheers
On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun wrote:
> Hi, All.
>
> I want to propose the followings.
>
> - Turn on Travis CI for Apache Spark PR
- For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>
> Thank you, Ted.
>
> Dongjoon.
>
> On Sun, May 22, 2016 at 1:29 PM, Ted Yu wrote:
>
>> The following line was repeated twice:
>>
>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`
and amend your commit title or messages, see the Travis CI.
> Or, you can monitor Travis CI result on status menu bar.
> If it shows green icon, you have nothing to do.
>
>https://docs.travis-ci.com/user/apps/
>
> To sum up, I think we don't need to wait f
Can you tell us the commit hash using which the test was run ?
For #2, if you can give full stack trace, that would be nice.
Thanks
On Mon, May 23, 2016 at 8:58 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.ma...@inria.fr> wrote:
> Hi
>
> 1) Using latest spark 2.0 I've managed to run TPCDSQueryBe
>>> And, don't worry, Ted.
>>>
>>> Travis launches new VMs for every PR.
>>>
>>> Apache Spark repository uses the following setting.
>>>
>>> VM: Google Compute Engine
>>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
>>
Please log a JIRA.
Thanks
On Tue, May 24, 2016 at 8:33 AM, Koert Kuipers wrote:
> hello,
> as we continue to test spark 2.0 SNAPSHOT in-house we ran into the
> following trying to port an existing application from spark 1.6.1 to spark
> 2.0.0-SNAPSHOT.
>
> given this code:
>
> case class Test(a
Congratulations, Yanbo.
On Fri, Jun 3, 2016 at 7:48 PM, Matei Zaharia
wrote:
> Hi all,
>
> The PMC recently voted to add Yanbo Liang as a committer. Yanbo has been a
> super active contributor in many areas of MLlib. Please join me in
> welcoming Yanbo!
>
> Matei
> --
See the following from
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.10/1642/consoleFull
:
+ SBT_FLAGS+=('-Dscala-2.10')
+ ./dev/change-scala-version.sh 2.10
FYI
On Mon, Jun 6, 2016 at 10:35 AM, Franklyn D'souza <
franklyn.dso...@shopify.
With commit 200f01c8fb15680b5630fbd122d44f9b1d096e02 using Scala 2.11:
Using Python version 2.7.9 (default, Apr 29 2016 10:48:06)
SparkSession available as 'spark'.
>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import IntegerType, StructField, StructType
>>> from pyspark.sql.
Have you tried the following ?
Seq(1->2, 1->5, 3->6).toDS("a", "b")
then you can refer to columns by name.
FYI
On Tue, Jun 7, 2016 at 3:58 PM, Alexander Pivovarov
wrote:
> I'm trying to switch from RDD API to Dataset API
> My question is about reduceByKey method
>
> e.g. in the following exa
I built with Scala 2.10
>>> df.select(add_one(df.a).alias('incremented')).collect()
The above just hung.
On Tue, Jun 7, 2016 at 3:31 PM, franklyn
wrote:
> Thanks Ted !.
>
> I'm using
>
> https://github.com/apache/spark/commit/8f5a04b6299e3a47aca13cbb40e72344c0114860
> and building with scala-2
Please go ahead.
On Tue, Jun 7, 2016 at 4:45 PM, franklyn
wrote:
> Thanks for reproducing it Ted, should i make a Jira Issue?.
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Can-t-use-UDFs-with-Dataframes-in-spark-2-0-preview-scala-2-10-tp1
I think the second group (3 classOf's) should be used.
Cheers
On Wed, Jun 8, 2016 at 4:53 PM, Alexander Pivovarov
wrote:
> if my RDD is RDD[(String, (Long, MyClass))]
>
> Do I need to register
>
> classOf[MyClass]
> classOf[(Any, Any)]
>
> or
>
> classOf[MyClass]
> classOf[(Long, MyClass)]
> cl
Docker Integration Tests failed on Linux:
http://pastebin.com/Ut51aRV3
Here was the command I used:
mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6 -Psparkr
-Dhadoop.version=2.7.0 package
Has anyone seen similar error ?
Thanks
On Thu, Jun 16, 2016 at 9:49 PM, Reynold Xin wrote:
> P
You can use a JIRA filter to find JIRAs of the component(s) you're
interested in.
Then sort by Priority.
Maybe comment on the JIRA if you want to work on it.
On Fri, Jun 17, 2016 at 3:22 PM, Pedro Rodriguez
wrote:
> What is the best way to determine what the library maintainers believe is
> imp
Running the following command:
build/mvn clean -Phive -Phive-thriftserver -Pyarn -Phadoop-2.6 -Psparkr
-Dhadoop.version=2.7.0 package
The build stopped with this test failure:
^[[31m- SPARK-9757 Persist Parquet relation with decimal column *** FAILED
***^[[0m
On Wed, Jul 6, 2016 at 6:25 AM, Sea
bq. we turned it off when fixing a bug
Adam:
Can you refer to the bug JIRA ?
Thanks
On Fri, Jul 8, 2016 at 9:22 AM, Adam Roberts wrote:
> Thanks Michael, we can give your options a try and aim for a 2.0.0 tuned
> vs 2.0.0 default vs 1.6.2 default comparison, for future reference the
> defaults
Found a few issues:
[SPARK-6810] Performance benchmarks for SparkR
[SPARK-2833] performance tests for linear regression
[SPARK-15447] Performance test for ALS in Spark 2.0
Haven't found one for Spark core.
On Fri, Jul 8, 2016 at 8:58 AM, Michael Allman wrote:
> Hello,
>
> I've seen a few messa
I assume you have enabled Zinc.
Cheers
On Fri, Jul 22, 2016 at 7:54 AM, Mikael Ståldal
wrote:
> Is there any way to speed up an incremental build of Spark?
>
> For me it takes 8 minutes to build the project with just a few code
> changes.
>
> --
> [image: MagineTV]
>
> *Mikael Ståldal*
> Senior
There was SPARK-12008 which was closed.
Not sure if there is active JIRA in this regard.
On Tue, Aug 2, 2016 at 6:40 PM, 马晓宇 wrote:
> Hi guys,
>
> I wonder if anyone working on SQL based authorization already or not.
>
> This is something we needed badly right now and we tried to embedded a
> H
901 - 1000 of 1643 matches
Mail list logo