[no subject]
Hi Everyone, My Security team has raised concerns about the requirement for root group membership for Spark running on Kubernetes. Does anyone know the reasons for that requirement, how insecure it is, and any alternatives if at all? Thanks, Rodrigo
[SPARK-CORE] JVM Properties passed as -D, not being found inside UDAF classes
Hi everyone! I'm trying to pass -D properties (JVM properties) to a Spark Application where we have some UDAF (User Defined Aggregate Functions) who will read those properties (using System.getProperty()). The problem is, the properties are never there when we try to read them. According to the docs, we have two ways of passing -D variables: --conf 'spark.driver.extraJavaOptions=-Dmy.key=value1' --conf 'spark.executor.extraJavaOptions=-Dmy.key=value1' First for "driver", the second for "executors". So the first one is working, because code outside the UDAF can read the property. But the second apparently is not. Can this have something to do with the fact that we're using client mode? '. Or could this have something to do specifically with the UDAF classes? -driver-java-options' also didn't work. To summarise, only the UDAF classes don't see the JVM properties. I'm using Spark 2.2.0. Best, This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. __ www.accenture.com
RE: Spark on Apache Ingnite?
I also had a quick look and agree it’s not very clear. I believe if one reads through the clustering logic and the replication settings would get a good idea of how it works. https://apacheignite.readme.io/docs/cluster I believe it integrates with Hadoop and other file based systems for persisting when needed. Not sure about the details on how does it recover. Also resource manager such as Mesos can add recoverability for at least scenarios where there isn’t any state to recover. Resilience is a feature and not every use case needs it. For example, I’m currently considering Ignite for caching purposes of transient data where we have the need to share RDDs between different Spark Contexts where one context produces data and the other consumes From: Koert Kuipers [mailto:ko...@tresata.com] Sent: 11 January 2016 16:08 To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> Cc: user@spark.apache.org Subject: Re: Spark on Apache Ingnite? where is ignite's resilience/fault-tolerance design documented? i can not find it. i would generally stay away from it if fault-tolerance is an afterthought. On Mon, Jan 11, 2016 at 10:31 AM, RodrigoB <rodrigo.boav...@aspect.com<mailto:rodrigo.boav...@aspect.com>> wrote: Although I haven't work explicitly with either, they do seem to differ in design and consequently in usage scenarios. Ignite is claimed to be a pure in-memory distributed database. With Ignite, updating existing keys is something that is self-managed comparing with Tachyon. In Tachyon once a value is created for a given key, becomes immutable, so you either delete and insert again, or need to manage/update the tachyon keys yourself. Also, Tachyon's resilience design is based on the underlying file system (typically hadoop), which means that if a node goes down, to recover the lost data, it would need first to have been persisted on the corresponding file partition. With Ignite, there is no master dependency like with Tachyon, and my understanding is that API calls will depend on master's availability in Tachyon. I believe Ignite has some options for replication which would be more aligned with the in-memory datastore. If you are looking for persisting some RDD's output into an in-memory store and query it outside of Spark, on the paper Ignite sounds like a better solution. Since you are asking about Ignite benefits that was the focus of my response. Tachyon has its own benefits like the community support and the Spark lineage persistency integration. If you are doing batch based processing and want to persist fast Spark RDDs, Tachyon is your friend. Hope this helps. Tnks, Rod -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Apache-Ingnite-tp25884p25933.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments.
RE: Scala 2.11 and Akka 2.4.0
HI Jacek, Yes I was told that as well but no one gave me release schedules, and I have the immediate need to have Spark Applications communicating with Akka clusters based on latest version. I'm aware there is an ongoing effort to change to the low level netty implementation but AFAIK it's not available yet. Any suggestions are very welcomed. Tnks, Rod -Original Message- From: Jacek Laskowski [mailto:ja...@japila.pl] Sent: 01 December 2015 18:17 To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> Cc: user <user@spark.apache.org> Subject: Re: Scala 2.11 and Akka 2.4.0 On Tue, Dec 1, 2015 at 2:32 PM, RodrigoB <rodrigo.boav...@aspect.com> wrote: > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0. Why? AFAIK Spark's leaving Akka's boat and joins Netty's. Jacek This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RE: Scala 2.11 and Akka 2.4.0
Thanks that worked! I let you know the results. Tnks, Rod From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: 01 December 2015 15:36 To: Boavida, Rodrigo <rodrigo.boav...@aspect.com> Cc: user@spark.apache.org Subject: Re: Scala 2.11 and Akka 2.4.0 Please specify the following in your maven commands: -Dscala-2.11 Cheers This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments.
RE: Scala 2.11 and Akka 2.4.0
Hi Ted,
Thanks for getting back to me and for the suggestion.
Running a 'mvn dependency:tree' I get the following:
[ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve
dependencies for project org.apache.spark:spark-core_2.11:jar:1.5.2: The
following artifacts could not be resolved:
com.typesafe.akka:akka-remote_2.10:jar:2.4.0,
com.typesafe.akka:akka-slf4j_2.10:jar:2.4.0,
com.typesafe.akka:akka-testkit_2.10:jar:2.4.0: Could not find artifact
com.typesafe.akka:akka-remote_2.10:jar:2.4.0 in central
(https://repo1.maven.org/maven2) -> [Help 1]
So it seems somehow it's still pulling some 2.10 dependencies. Do you think
this could be the cause for the observed problem?
tnks,
Rod
-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: 01 December 2015 14:13
To: Boavida, Rodrigo <rodrigo.boav...@aspect.com>
Cc: user@spark.apache.org
Subject: Re: Scala 2.11 and Akka 2.4.0
Have you run 'mvn dependency:tree' and examined the output ?
There should be some hint.
Cheers
> On Dec 1, 2015, at 5:32 AM, RodrigoB <rodrigo.boav...@aspect.com> wrote:
>
> Hi,
>
> I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0.
> I've changed the main pom.xml files to corresponding akka version and
> am getting the following exception when starting the master on standalone:
>
> Exception Details:
> Location:
>akka/dispatch/Mailbox.processAllSystemMessages()V @152: getstatic
> Reason:
>Type top (current frame, locals[9]) is not assignable to
> 'akka/dispatch/sysmsg/SystemMessage' (stack map, locals[9]) Current
> Frame:
>bci: @131
>flags: { }
>locals: { 'akka/dispatch/Mailbox',
> 'java/lang/InterruptedException',
> 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox',
> 'java/lang/Throwable', 'java/lang/Throwable' }
>stack: { integer }
> Stackmap Frame:
>bci: @152
>flags: { }
>locals: { 'akka/dispatch/Mailbox',
> 'java/lang/InterruptedException',
> 'akka/dispatch/sysmsg/SystemMessage', top, 'akka/dispatch/Mailbox',
> 'java/lang/Throwable', 'java/lang/Throwable', top, top,
> 'akka/dispatch/sysmsg/SystemMessage' }
>stack: { }
> Bytecode:
>0x000: 014c 2ab2 0132 b601 35b6 0139 4db2 013e
>0x010: 2cb6 0142 9900 522a b600 c69a 004b 2c4e
>0x020: b201 3e2c b601 454d 2db9 0148 0100 2ab6
>0x030: 0052 2db6 014b b801 0999 000e bb00 e759
>0x040: 1301 4db7 010f 4cb2 013e 2cb6 0150 99ff
>0x050: bf2a b600 c69a ffb8 2ab2 0132 b601 35b6
>0x060: 0139 4da7 ffaa 2ab6 0052 b600 56b6 0154
>0x070: b601 5a3a 04a7 0091 3a05 1905 3a06 1906
>0x080: c100 e799 0015 1906 c000 e73a 0719 074c
>0x090: b200 f63a 08a7 0071 b201 5f19 06b6 0163
>0x0a0: 3a0a 190a b601 6899 0006 1905 bf19 0ab6
>0x0b0: 016c c000 df3a 0b2a b600 52b6 0170 b601
>0x0c0: 76bb 000f 5919 0b2a b600 52b6 017a b601
>0x0d0: 80b6 0186 2ab6 018a bb01 8c59 b701 8e13
>0x0e0: 0190 b601 9419 09b6 0194 1301 96b6 0194
>0x0f0: 190b b601 99b6 0194 b601 9ab7 019d b601
>0x100: a3b2 00f6 3a08 b201 3e2c b601 4299 0026
>0x110: 2c3a 09b2 013e 2cb6 0145 4d19 09b9 0148
>0x120: 0100 1904 2ab6 0052 b601 7a19 09b6 01a7
>0x130: a7ff d62b c600 09b8 0109 572b bfb1 Exception Handler
> Table:
>bci [290, 307] => handler: 120
> Stackmap Table:
>append_frame(@13,Object[#231],Object[#177])
>append_frame(@71,Object[#177])
>chop_frame(@102,1)
>
> full_frame(@120,{Object[#2],Object[#231],Object[#177],Top,Object[#2],O
> bject[#177]},{Object[#223]})
>
> full_frame(@152,{Object[#2],Object[#231],Object[#177],Top,Object[#2],Object[#223],Object[#223],Top,Top,Object[#177]},{})
>append_frame(@173,Object[#357])
>
> full_frame(@262,{Object[#2],Object[#231],Object[#177],Top,Object[#2]},{})
>same_frame(@307)
>same_frame(@317)
> at akka.dispatch.Mailboxes.(Mailboxes.scala:33)
>at akka.actor.ActorSystemImpl.(ActorSystem.scala:635)
>at akka.actor.ActorSystem$.apply(ActorSystem.scala:143)
>at akka.actor.ActorSystem$.apply(ActorSystem.scala:120)
>at
> org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121)
>at
> org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)
>at
> org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52)
>at
> org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
>at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:166)
>at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
>at
> org.apache.spark.util.AkkaUtils$.createActorSy
