com> wrote:
> Looks like the other packages may also be corrupt. I’m getting the same
> error for the Spark 1.6.1 / Hadoop 2.4 package.
>
>
> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz
>
> Nick
>
>
> On Wed, Mar 16, 2016 at 8:28 P
Josh:
SerializerInstance and SerializationStream would also become private[spark],
right ?
Thanks
On Mon, Mar 7, 2016 at 6:57 PM, Josh Rosen wrote:
> Does anyone implement Spark's serializer interface
> (org.apache.spark.serializer.Serializer) in your own third-party
Please stack trace, code snippet, etc in the JIRA you created so that
people can reproduce what you saw.
On Sat, Mar 5, 2016 at 7:02 AM, Dhaval Modi wrote:
>
> Regards,
> Dhaval Modi
> dhavalmod...@gmail.com
>
> -- Forwarded message --
> From: Dhaval Modi
gt; - bad equals/hashCode
>
> On Fri, Mar 4, 2016 at 2:52 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > Last time I checked there wasn't high impact defects.
> >
> > Mind pointing out the defects you think should be fixed ?
> >
> > Thanks
> >
>
va code. I'm not suggesting anyone run it regularly,
> but one run to catch some bugs is useful.
>
> I've already triaged ~70 issues there just in the Java code, of which
> a handful are important.
>
> On Fri, Mar 4, 2016 at 12:18 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
Since majority of code is written in Scala which is not analyzed by Coverity,
the efficacy of the tool seems limited.
> On Mar 4, 2016, at 2:34 AM, Sean Owen wrote:
>
> https://scan.coverity.com/projects/apache-spark-2f9d080d-401d-47bc-9dd1-7956c411fbb4?tab=overview
>
>
Looking at
https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
*WARNING* Generating the caller class information is slow. Thus, use should
be avoided unless execution speed is not an issue.
On Sat, Feb 27, 2016 at 12:40 PM, Prabhu Joseph
In hbase, there is hbase-spark module which supports bulk load.
This module is to be backported in the upcoming 1.3.0 release.
There is some pending work, such as HBASE-15271 .
FYI
On Fri, Feb 26, 2016 at 8:50 AM, Renu Yadav wrote:
> Has anybody implemented bulk load into
When you click on Create, you're brought to 'Create Issue' dialog where you
choose Project Spark.
Component should be MLlib.
Please see also:
http://search-hadoop.com/m/q3RTtmsshe1W6cH22/spark+pull+template=pull+request+template
On Mon, Feb 22, 2016 at 6:45 PM, Pierson, Oliver C
The referenced benchmark is in Chinese. Please provide English version so
that more people can understand.
For item 7, looks like the speed of ingest is much slower compared to using
Parquet.
Cheers
On Mon, Feb 22, 2016 at 6:12 AM, 开心延年 wrote:
> 1.ya100 is not only the
Hdfs class is in hadoop-hdfs-XX.jar
Can you check the classpath to see if the above jar is there ?
Please describe the command lines you used for building hadoop / Spark.
Cheers
On Thu, Feb 11, 2016 at 5:15 PM, Charlie Wright
wrote:
> I am having issues trying to run a
Do you mind pastebin'ning code snippet and exception one more time - I
couldn't see them in your original email.
Which Spark release are you using ?
On Tue, Feb 9, 2016 at 11:55 AM, rakeshchalasani
wrote:
> Hi All:
>
> I am getting an "UnsupportedOperationException" when
ethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
> at
> org.apache.spark.re
>
> ++
> |arrayCol|
> ++
> | [0, 1]|
> | [1, 2]|
> | [2, 3]|
> | [3, 4]|
> | [4, 5]|
> | [5, 6]|
> | [6, 7]|
> | [7, 8]|
> | [8, 9]|
> | [9, 10]|
> ++
>
>
>
> On Tue, Feb 9, 2016 at 4:52 PM
Congratulations, Herman and Wenchen.
On Mon, Feb 8, 2016 at 9:15 AM, Matei Zaharia
wrote:
> Hi all,
>
> The PMC has recently added two new Spark committers -- Herman van Hovell
> and Wenchen Fan. Both have been heavily involved in Spark SQL and Tungsten,
> adding new
Assuming your change is based on hadoop-2 branch, you can use 'mvn install'
command which would put artifacts under 2.8.0-SNAPSHOT subdir in your local
maven repo.
Here is an example:
~/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.8.0-SNAPSHOT
Then you can use the following command to build
For #1, a brief search landed the following:
core/src/main/scala/org/apache/spark/SparkConf.scala:
DeprecatedConfig("spark.rpc", "2.0", "Not used any more.")
core/src/main/scala/org/apache/spark/SparkConf.scala:
"spark.rpc.numRetries" -> Seq(
w.r.t. running Spark on YARN, there are a few outstanding issues. e.g.
SPARK-11182 HDFS Delegation Token
See also the comments under SPARK-12279
FYI
On Mon, Feb 1, 2016 at 1:02 PM, eugene miretsky
wrote:
> When having multiple users sharing the same Spark cluster,
The following jobs have been established for build against Scala 2.10:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-MAVEN-SCALA-2.10/
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.10/
FYI
On
Does this mean the following Jenkins builds can be disabled ?
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-MAVEN-SCALA-2.11/
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-master-COMPILE-sbt-SCALA-2.11/
Cheers
On Sat, Jan
w.r.t. protobuf-java version mismatch, I wonder if you can rebuild Spark
with the following change (using maven):
http://pastebin.com/fVQAYWHM
Cheers
On Sat, Jan 30, 2016 at 12:49 AM, Yash Sharma wrote:
> Hi All,
> I have a quick question if anyone has experienced this
gnoreCase("string");
>
> String tsColName = null;
> if (iTimestamp >= 0) {
> tsColName =
> jobConf.get(serdeConstants.LIST_COLUMNS).split(",")[iTimestamp];
> }
>
>
>
> -- 原始邮件 ------
> *发件人:* "Jörn Fran
After this change:
[SPARK-12681] [SQL] split IdentifiersParser.g into two files
the biggest file under
sql/catalyst/src/main/antlr3/org/apache/spark/sql/catalyst/parser is
SparkSqlParser.g
Maybe split SparkSqlParser.g up as well ?
On Thu, Jan 28, 2016 at 5:21 AM, Iulian Dragoș
For the last two problems, hbase-site.xml seems not to be on classpath.
Once hbase-site.xml is put on classpath, you should be able to make progress.
Cheers
> On Jan 28, 2016, at 1:14 AM, Maciej Bryński wrote:
>
> Hi,
> I'm trying to run SQL query on Hive table which is
Cycling past bits:
http://search-hadoop.com/m/q3RTtU5CRU1KKVA42=RE+shuffle+FetchFailedException+in+spark+on+YARN+job
On Sun, Jan 24, 2016 at 5:52 AM, wangzhenhua (G)
wrote:
> Hi,
>
> I have a problem of time out in shuffle, it happened after shuffle write
> and at the
I would suggest trying option #1 first.
Thanks
> On Jan 13, 2016, at 2:12 AM, Maciej Bryński wrote:
>
> Hi,
> I/m trying to run Spark 1.6.0 on HDP 2.2
> Everything was fine until I tried to turn on dynamic allocation.
> According to instruction I need to add shuffle service
There is no annotation in TestingUtils class indicating whether it is
suitable for consumption by external projects.
You should assume the class is not public since its methods may change in
future Spark releases.
Cheers
On Tue, Jan 12, 2016 at 12:36 PM, Robert Dodier
I logged SPARK-12778 where endian awareness in Platform.java should
help in mixed
endian set up.
There could be other parts of the code base which are related.
Cheers
On Tue, Jan 12, 2016 at 7:01 AM, Adam Roberts wrote:
> Hi all, I've been experimenting with DataFrame
+1
> On Jan 5, 2016, at 10:49 AM, Davies Liu wrote:
>
> +1
>
> On Tue, Jan 5, 2016 at 5:45 AM, Nicholas Chammas
> wrote:
>> +1
>>
>> Red Hat supports Python 2.6 on REHL 5 until 2020, but otherwise yes, Python
>> 2.6 is ancient history and
Right.
Pardon my carelessness.
> On Dec 29, 2015, at 9:58 PM, Reynold Xin <r...@databricks.com> wrote:
>
> OK to close the loop - this thread has nothing to do with Spark?
>
>
>> On Tue, Dec 29, 2015 at 9:55 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
Hi,
I noticed that there are a lot of checkstyle warnings in the following form:
To my knowledge, we use two spaces for each tab. Not sure why all of a
sudden we have so many IndentationCheck warnings:
grep 'hild have incorrect indentati' trunkCheckstyle.xml | wc
3133 52645 678294
If
>>
>>
>> format issue I think, go ahead
>>
>>
>>
>>
>> At 2015-12-30 13:36:05, "Ted Yu" <yuzhih...@gmail.com> wrote:
>> Hi,
>> I noticed that there are a lot of checkstyle warnings in the following form:
>>
>>
For #1, 9 minutes seem to be normal. Here was duration for recent build on
master branch:
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 10:44
a processes separate from Spark processes, so you can
>>>> monitor, debug, and scale them independently. So consider streaming data
>>>> from Akka to Spark Streaming or go the other way, from Spark to Akka
>>>> Streams.
>>>>
>>>> dean
>>>
Do you mind sharing your use case ?
It may be possible to use a different approach than Akka.
Cheers
On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava
wrote:
> Hi,
>
> I wanted to know how to use Akka framework with Spark starting from
> basics. I saw online that Spark
Hi,
You may have noticed the following test failures:
org.apache.spark.sql.hive.execution.HiveUDFSuite.UDFIntegerToString
org.apache.spark.sql.hive.execution.SQLQuerySuite.udf_java_method
Tracing backwards, they started failing since this build:
I found that SBT build for Scala 2.11 has been failing (
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/SPARK-branch-1.6-COMPILE-SBT-SCALA-2.11/3/consoleFull
)
I logged SPARK-12527 and sent a PR.
FYI
On Tue, Dec 22, 2015 at 12:10 PM, Michael Armbrust
getMissingParentStages(stage) would be called for the stage (being
re-submitted)
If there is no missing parents, submitMissingTasks() would be called.
If there is missing parent(s), the parent would go through the same flow.
I don't see issue in this part of the code.
Cheers
On Thu, Dec 24,
Running test suite, there was timeout in hive-thriftserver module.
This has been fixed by SPARK-11823. So I assume this is test issue.
lgtm
On Tue, Dec 22, 2015 at 2:28 PM, Benjamin Fradet
wrote:
> +1
> On 22 Dec 2015 9:54 p.m., "Andrew Or"
In Jerry's example, the first SparkContext, sc, has been stopped.
So there would be only one SparkContext running at any given moment.
Cheers
On Mon, Dec 21, 2015 at 8:23 AM, Chester @work
wrote:
> Jerry
> I thought you should not create more than one SparkContext
Ran test suite (minus docker-integration-tests)
All passed
+1
[INFO] Spark Project External ZeroMQ .. SUCCESS [
13.647 s]
[INFO] Spark Project External Kafka ... SUCCESS [
45.424 s]
[INFO] Spark Project Examples . SUCCESS [02:06
Allen:
Since you mentioned scheduling, I assume you were talking about node label
support in YARN.
If that is the case, can you give us some more information:
How node labels are setup in YARN cluster
How you specified node labels in application
Hadoop and Spark releases you are using
Cheers
>
k 1.5.0, what happened to me
> was I was blocked to get the YARN containers by setting
> spark.yarn.executor.nodeLabelExpression property. My question,
> https://issues.apache.org/jira/browse/SPARK-7173 will fix this?
>
>
>
> Thanks
>
> Allen
>
>
>
>
>
>
mailto:sai.sai.s...@gmail.com]
> *发送时间:* 2015年12月15日 18:07
> *收件人:* 张志强(旺轩)
> *抄送:* Ted Yu; dev
> *主题:* Re: spark with label nodes in yarn
>
>
>
> SPARK-6470 only supports node label expression for executors.
>
> SPARK-7173 supports node label expression for A
.6 is pretty close to master,
> I am wondering if there is any environment related issue.
>
> On Sun, Dec 13, 2015 at 3:38 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> Thanks for checking, Yin.
>>
>> Looks like the cause might be in one of the commits for build #4438
since 4438 and 4439 were failed
> way before the thrift server tests.
>
> On Fri, Dec 11, 2015 at 10:27 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> Hi,
>> You may have noticed that maven build against Hadoop 2.4 times out on
>> Jenkins.
>>
>> The last
Hi,
You may have noticed that maven build against Hadoop 2.4 times out on Jenkins.
The last module is spark-hive-thriftserver
This seemed to start with build #4440
FYI
-
To unsubscribe, e-mail:
I tried to run test suite and encountered the following:
http://pastebin.com/DPnwMGrm
FYI
On Wed, Dec 2, 2015 at 12:39 PM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:
> -0
>
> If spark-ec2 is still a supported part of the project, then we should
> update its version lists as new
+1
Ran through test suite (minus docker-integration-tests) which passed.
Overall experience was much better compared with some of the prior RC's.
[INFO] Spark Project External Kafka ... SUCCESS [
53.956 s]
[INFO] Spark Project Examples . SUCCESS
If I am not mistaken, the binaries for Scala 2.11 were generated against
hadoop 1.
What about binaries for Scala 2.11 against hadoop 2.x ?
Cheers
On Sun, Nov 22, 2015 at 2:21 PM, Michael Armbrust
wrote:
> In order to facilitate community testing of Spark 1.6.0, I'm
Should a new job be setup under Spark-Master-Maven-with-YARN for hadoop
2.6.x ?
Cheers
On Thu, Nov 19, 2015 at 5:16 PM, 张志强(旺轩) wrote:
> I agreed
> +1
>
> --
> 发件人:Reynold Xin
> 日
See this thread:
http://search-hadoop.com/m/q3RTtLKc2ctNPcq=Re+Spark+1+4+2+release+and+votes+conversation+
> On Nov 15, 2015, at 10:53 PM, Niranda Perera wrote:
>
> Hi,
>
> I am wondering when spark 1.4.2 will be released?
>
> is it in the voting stage at the
Please take a look at http://www.infoq.com/articles/tuning-tips-G1-GC
Cheers
On Sat, Nov 14, 2015 at 10:03 PM, Renu Yadav wrote:
> I have tried with G1 GC .Please if anyone can provide their setting for GC.
> At code level I am :
> 1.reading orc table usind dataframe
> 2.map
; It only runs tests that are impacted by the change. E.g. if you only
> modify SQL, it won't run the core or streaming tests.
>
>
> On Fri, Nov 13, 2015 at 11:17 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> Hi,
>> I noticed that SparkPullRequestBuilder complet
I was able to access the following where response was fast:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45806/
Cheers
On Thu, Nov 12, 2015 at 6:21 PM, Yin Huai wrote:
> Hi
Please consider using NoSQL engine such as hbase.
Cheers
> On Nov 9, 2015, at 3:03 PM, Andrés Ivaldi wrote:
>
> Hi,
> I'm also considering something similar, Spark plain is too slow for my case,
> a possible solution is use Spark as Multiple Source connector and basic
>
Why did you directly jump to spark-streaming-mqtt module ?
Can you drop 'spark-streaming-mqtt' and try again ?
Not sure why 1.5.0-SNAPSHOT showed up.
Were you using RC2 source ?
Cheers
On Sun, Nov 8, 2015 at 7:28 PM, 欧锐 <494165...@qq.com> wrote:
>
> build spark-streaming-mqtt_2.10 failed!
>
>
+1
On Sat, Nov 7, 2015 at 4:35 PM, Denny Lee wrote:
> +1
>
>
> On Sat, Nov 7, 2015 at 12:01 PM Mark Hamstra
> wrote:
>
>> +1
>>
>> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin wrote:
>>
>>> Please vote on releasing the
Created a PR for the compilation error:
https://github.com/apache/spark/pull/9538
Cheers
On Sat, Nov 7, 2015 at 4:41 AM, Jacek Laskowski wrote:
> Hi,
>
> Checked out the latest sources and the build failed:
>
> [error]
>
Would the following change work for you ?
diff --git
a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala
index 61b5a4c..c330d25 100644
---
Since maven is the preferred build vehicle, ivy style dependencies policy
would produce surprising results compared to today's behavior.
I would suggest staying with current dependencies policy.
My two cents.
On Fri, Nov 6, 2015 at 6:25 AM, Koert Kuipers wrote:
> if there
;
> Regards,
> Dilip Biswal
> Tel: 408-463-4980
> dbis...@us.ibm.com
>
>
>
> From:Ted Yu <yuzhih...@gmail.com>
> To:Dilip Biswal/Oakland/IBM@IBMUS
> Cc:Jean-Baptiste Onofré <j...@nanthrax.net>, "dev@spark.apache.org&
ng able to find
> com.google.common.hash.HashCodes.
>
> Is there a solution to this ?
>
> Regards,
> Dilip Biswal
> Tel: 408-463-4980
> dbis...@us.ibm.com
>
>
>
> From:Jean-Baptiste Onofré <j...@nanthrax.net>
> To:Ted Yu <yuzhih...@gmail.com>
> Cc:"de
See previous discussion:
http://search-hadoop.com/m/q3RTtPnPnzwOhBr
FYI
On Thu, Nov 5, 2015 at 4:30 PM, Stephen Boesch wrote:
> Yes. The current dev/change-scala-version.sh mutates (/pollutes) the build
> environment by updating the pom.xml in each of the subprojects. If you
Looks like SparkListenerSuite doesn't OOM on QA runs compared to Jenkins
builds.
I wonder if this is due to difference between machines running QA tests vs
machines running Jenkins builds.
On Fri, Oct 30, 2015 at 1:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> I noticed that the Spa
Please take a look at first section of spark.apache.org/community
FYI
On Sun, Nov 1, 2015 at 1:09 AM, Chenxi Li wrote:
> unscribe
>
On Linux, I got the following test failure (with or without suggested
change):
testChildProcLauncher(org.apache.spark.launcher.SparkLauncherSuite) Time
elapsed: 0.036 sec <<< FAILURE!
java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at
per-job basis (this doesn't
> > scale that well).
> >
> > thoughts?
> >
> > On Fri, Oct 30, 2015 at 9:47 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >> This happened recently on Jenkins:
> >>
> >>
> https://amplab.cs.berkeley.edu/j
nal
>> aggregate functions not supposed to be used or I am using them in the wrong
>> way or is it a bug as I asked in my first mail.
>>
>> On Wed, Oct 28, 2015 at 3:20 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>>> Have you tried using avg in place o
unsodh...@gmail.com
> > wrote:
>
>> Also are the other aggregate functions to be treated as bugs or not?
>>
>> On Wed, Oct 28, 2015 at 4:08 PM, Shagun Sodhani <sshagunsodh...@gmail.com
>> > wrote:
>>
>>> Wouldnt it be:
>>>
>>>
You can use the following link:
https://issues.apache.org/jira/secure/CreateIssue!default.jspa
Remember to select Spark as the project.
On Thu, Oct 22, 2015 at 9:38 AM, Richard Marscher
wrote:
> Hi,
>
> I'm working on following the guidelines for contributing code to
See this thread
http://search-hadoop.com/m/q3RTtV3VFNdgNri2=Re+Build+spark+1+5+1+branch+fails
> On Oct 19, 2015, at 6:59 PM, Annabel Melongo
> wrote:
>
> I tried to build Spark according to the build directions and the it failed
> due to the following error:
From
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=spark-test/3846/console
:
SparkListenerSuite:- basic creation and shutdown of LiveListenerBus-
bus.stop() waits for the event queue to completely drain- basic
creation of StageInfo- basic
for `SPARK_MASTER_IP`, amazingly, does not show it
> being used in any place directly by Spark
> <https://github.com/apache/spark/search?utf8=%E2%9C%93=SPARK_MASTER_IP>.
>
> Clearly, Spark is using this environment variable (otherwise I wouldn't
> see the behavior described in my
bq. Access is denied
Please check permission of the path mentioned.
On Thu, Oct 15, 2015 at 3:45 PM, Annabel Melongo <
melongo_anna...@yahoo.com.invalid> wrote:
> I was trying to build a cloned version of Spark on my local machine using
> the command:
> mvn -Pyarn -Phadoop-2.4
Some old bits:
http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
FYI
On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:
> I’m setting the Spark
Please see
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
On Tue, Oct 13, 2015 at 5:49 AM, _abhishek
wrote:
> Hello
> I am interested in contributing to apache spark.I am new to open source.Can
> someone please help me with how to get
http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss
> On Oct 11, 2015, at 10:45 PM, Niranda Perera wrote:
>
> Hi all,
>
> is there a way for me to get the heap-dump hprof of an executor jvm, when it
> goes out
You can go to:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN
and see if the test failure(s) you encountered appeared there.
FYI
On Mon, Oct 12, 2015 at 1:24 PM, Meihua Wu
wrote:
> Hi Spark Devs,
>
> I recently encountered several cases
in _get_connection
> IndexError: pop from an empty deque
>
>
>
> On Mon, Oct 12, 2015 at 1:36 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > You can go to:
> > https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN
> >
> > and
; On October 12, 2015 at 2:45:13 PM, Ted Yu (yuzhih...@gmail.com) wrote:
>
> Can you re-submit your PR to trigger a new build - assuming the tests are
> flaky ?
>
> If any test fails again, consider contacting the owner of the module for
> expert opinion.
>
> Cheers
&
Interesting
https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/job/Spark-Master-Scala211-Compile/
shows green builds.
On Thu, Oct 8, 2015 at 6:40 AM, Iulian Dragoș
wrote:
> Since Oct. 4 the build fails on 2.11 with the dreaded
>
> [error]
]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 17:49 min
FYI
On Thu, Oct 8, 2015 at 6:50 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> Interesting
>
>
> https://amplab.cs.be
In root pom.xml :
2.2.0
You can override the version of hadoop with command similar to:
-Phadoop-2.4 -Dhadoop.version=2.7.0
Cheers
On Thu, Oct 8, 2015 at 11:22 AM, sbiookag wrote:
> I'm modifying hdfs module inside hadoop, and would like the see the
> reflection while
As a workaround, can you set the number of partitions higher in the
sc.textFile method ?
Cheers
On Mon, Oct 5, 2015 at 3:31 PM, Jegan wrote:
> Hi All,
>
> I am facing the below exception when the size of the file being read in a
> partition is above 2GB. This is apparently
hadoop1 package for Scala 2.10 wasn't in RC1 either:
http://people.apache.org/~pwendell/spark-releases/spark-1.5.1-rc1-bin/
On Sun, Oct 4, 2015 at 5:17 PM, Nicholas Chammas wrote:
> I’m looking here:
>
> https://s3.amazonaws.com/spark-related-packages/
>
> I believe
I tried to access
https://repo1.maven.org/maven2/org/apache/spark/spark-streaming_2.10/1.5.0/spark-streaming_2.10-1.5.0.pom
on
Chrome and Firefox (on Mac)
I got 404
FYI
On Fri, Oct 2, 2015 at 10:49 AM, andy petrella
wrote:
> Yup folks,
>
> I've been reported by someone
oct. 2015 20:08, Ted Yu <yuzhih...@gmail.com> a écrit :
>
>> Andy:
>> 1.5.1 has been released.
>>
>> Maybe you can use this:
>>
>> https://repo1.maven.org/maven2/org/apache/spark/spark-streaming_2.10/1.5.1/spark-streaming_2.10-1.5.1.pom
>>
>>
too (did not get it before). Maybe the servers are
> having issues.
>
> On Fri, Oct 2, 2015 at 11:05 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > I tried to access
> >
> https://repo1.maven.org/maven2/org/apache/spark/spark-streaming_2.10/1.5.0/spark-streaming_2.10-1.5.
What version of hadoop are you using ?
Is that version consistent with the one which was used to build Spark 1.4.0
?
Cheers
On Mon, Sep 28, 2015 at 4:36 PM, Renyi Xiong wrote:
> I tried to run HdfsTest sample on windows spark-1.4.0
>
> bin\run-sample
Which Spark release are you building ?
For master branch, I get the following:
lib_managed/jars/datanucleus-api-jdo-3.2.6.jar
lib_managed/jars/datanucleus-core-3.2.10.jar
lib_managed/jars/datanucleus-rdbms-3.2.9.jar
FYI
On Tue, Sep 22, 2015 at 1:28 PM, Richard Hillegas
xml-apis-1.4.01.jar
> commons-math-2.2.jar jaxb-impl-2.2.3-1.jar paranamer-2.3.jar
> xmlenc-0.52.jar
> commons-math3-3.4.1.jar jaxb-impl-2.2.7.jar paranamer-2.6.jar xz-1.0.jar
> commons-net-3.1.jar jblas-1.2.4.jar parquet-avro-1.7.0.jar
> zookeeper-3.4.5.jar
> commons-pool-1.5.
I cloned Hive 1.2 code base and saw:
10.10.2.0
So the version used by Spark is quite close to what Hive uses.
On Tue, Sep 22, 2015 at 3:29 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> I see.
> I use maven to build so I observe different contents under lib_managed
> dire
You can use broadcast variable for passing connection information.
Cheers
> On Sep 21, 2015, at 4:27 AM, Priya Ch wrote:
>
> can i use this sparkContext on executors ??
> In my application, i have scenario of reading from db for certain records in
> rdd. Hence I
Can you clarify what you want to do:
If you modify existing hadoop InputFormat, etc, it would be a matter of
rebuilding hadoop and build Spark using the custom built hadoop as
dependency.
Do you introduce new InputFormat ?
Cheers
On Mon, Sep 21, 2015 at 1:20 PM, Dogtail Ray
Maybe the following can be used for changing Scala version:
http://maven.apache.org/archetype/maven-archetype-plugin/
I played with it a little bit but didn't get far.
FYI
On Sun, Sep 20, 2015 at 6:18 AM, Stephen Boesch wrote:
>
> The dev/change-scala-version.sh [2.11]
Looks like you didn't specify sparkr profile when building.
Cheers
On Sat, Sep 19, 2015 at 12:30 PM, Devl Devel
wrote:
> Hi All,
>
> I've built spark 1.5.0 with hadoop 2.6 with a fresh download :
>
> build/mvn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean
See first section of http://spark.apache.org/community.html
Cheers
> On Sep 13, 2015, at 6:43 PM, 蒋林 wrote:
>
> Hi,I need subscribe email list,please send me,thank you
>
>
>
Is it possible that Canonical_URL occurs more than once in your json ?
Can you check your json input ?
Thanks
On Sat, Sep 12, 2015 at 2:05 AM, Fengdong Yu
wrote:
> Hi,
>
> I am using spark1.4.1 data frame, read JSON data, then save it to orc. the
> code is very
;
> Azuryy Yu
> Sr. Infrastructure Engineer
>
> cel: 158-0164-9103
> wetchat: azuryy
>
>
> On Sat, Sep 12, 2015 at 5:52 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> Is it possible that Canonical_URL occurs more than once in your json ?
>>
>> Can you check
101 - 200 of 331 matches
Mail list logo