India Scala & Big Data Job Referral

Hi Community, I was laid off from Apple in February 2023, which led to my relocation from the USA due to immigration issues related to my H1B visa. I have over 12 years of experience as a consultant in Big Data, Spark, Scala, Python, and Flink. Despite my move to India, I haven't secured a

Spark Scala Contract Opportunity @USA

Hi All, Is anyone looking for a spark scala contract role inside the USA? A company called Maxonic has an open spark scala contract position (100% remote) inside the USA if anyone is interested, please send your CV to kali.tumm...@gmail.com. Thanks & Regards Sri Tummala

Big Data Contract Roles ?

Hi Flink Users/ Spark Users, Is anyone hiring contract corp to corp big Data spark scala or Flink scala roles ? Thanks Sri

Re: sql to spark scala rdd

e: >> >> Hi, >> >> Impossible - see >> >> http://www.scala-lang.org/api/current/index.html#scala.collection.Seq@sliding(size:Int,step:Int):Iterator[Repr] >> . >> >> I tried to show you why you ended up with "non-empty iterator" after

Re: sql to spark scala rdd

)._1,x(1)._2,(x.foldLeft(0.0)(_ + _._2/x.size)),x.foldLeft(0.0)(_ + _._2))).foreach(println) On Sun, Jul 31, 2016 at 12:15 PM, sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi All, > > I already solved it using DF and spark sql I was wondering how to solve in > s

Re: sql to spark scala rdd

l's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 31 July 2016 at 19:49, sri hari kali charan Tummala < > kali.tumm...@gmail.com> wrote: > >> Tup

Re: sql to spark scala rdd

.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Sun, Jul 31, 2016 at 9:23 AM, sri hari kali charan Tummala > <kali.tumm...@gmail.com> wrote: > > tried this no luck, wht is non-empty iterator here ? > > > > OP:- &

Re: sql to spark scala rdd

t;)) .map(x => (x(0),x(2))) .map { case (key,value) => (key,value.toArray.toSeq.sliding(2,1).map(x => x.sum/x.size))}.foreach(println) On Sun, Jul 31, 2016 at 12:03 AM, sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi All, > > I managed to write using

Re: sql to spark scala rdd

intln) at the moment my output:- 75.0 -25.0 50.0 -50.0 -100.0 I want with key how to get moving average output based on key ? 987,75.0 987,-25 987,50.0 Thanks Sri On Sat, Jul 30, 2016 at 11:40 AM, sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > for knowledge j

Re: sql to spark scala rdd

for knowledge just wondering how to write it up in scala or spark RDD. Thanks Sri On Sat, Jul 30, 2016 at 11:24 AM, Jacek Laskowski wrote: > Why? > > Pozdrawiam, > Jacek Laskowski > > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0

Re: spark local dir to HDFS ?

thanks makes sense, can anyone answer this below question ? http://apache-spark-user-list.1001560.n3.nabble.com/spark-parquet-too-many-small-files-td27264.html Thanks Sri On Tue, Jul 5, 2016 at 8:15 PM, Saisai Shao wrote: > It is not worked to configure local dirs to

Re: spark parquet too many small files ?

Hi Takeshi, I cant use coalesce in spark-sql shell right I know we can use coalesce in spark with scala application , here in my project we are not building jar or using python we are just executing hive query in spark-sql shell and submitting to yarn client . Example:- spark-sql --verbose

Re: how to run latest version of spark in old version of spark in cloudera cluster ?

Hi Koert, I am submitting my code (spark jar ) using spark-submit in proxy node , I checked the version of the cluster and node its says 1.2 I dint really understand what you mean. can I ask yarn to use different version of spark ? or should I say override the spark_home variables to look at 1.6

Re: how to run latest version of spark in old version of spark in cloudera cluster ?

and configure it correctly to find your cluster (hdfs and yarn). >> then use the spark-submit script for that spark 1.6 version to launch your >> application on yarn >> >> On Wed, Jan 27, 2016 at 3:11 PM, sri hari kali charan Tummala < >> kali.tumm...@gmail.com> wr

Re: how to turn off spark streaming gracefully ?

Hi Cody, KafkaUtils.createRDD totally make sense now I can run my spark job once in 15 minutes extract data out of kafka and stop ..., I rely on kafka offset for Incremental data am I right ? so no duplicate data will be returned. Thanks Sri On Fri, Dec 18, 2015 at 2:41 PM, Cody Koeninger

Re: spark data frame write.mode("append") bug

Hi All, https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L48 In Present spark version in line 48 there is a bug, to check whether table exists in a database using limit doesnt work for all databases sql server

Re: Release data for spark 1.6?

t; On Dec 12, 2015, at 7:12 PM, sri hari kali charan Tummala < > kali.tumm...@gmail.com> wrote: > > Hi Michael, Ted, > > > https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L48 > > In

Re: Release data for spark 1.6?

Hi Michael, Ted, https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L48 In Present spark version in line 48 there is a bug, to check whether table exists in a database using limit doesnt work for all databases

Re: spark sql current time stamp function ?

Hi Ted, Gave and exception am I following right approach ? val test=sqlContext.sql("select *, monotonicallyIncreasingId() from kali") On Mon, Dec 7, 2015 at 4:52 PM, Ted Yu wrote: > Have you tried using monotonicallyIncreasingId ? > > Cheers > > On Mon, Dec 7, 2015 at

Re: Pass spark partition explicitly ?

Hi Richard, Thanks so my take from your discussion is we want pass explicitly partition values it have to be written inside the code. Thanks Sri On Sun, Oct 18, 2015 at 7:05 PM, Richard Eggert wrote: > If you want to override the default partitioning behavior, you