Thanks Abhishek. I will check it out. Thank you, Naresh
On Sat, Jul 27, 2019 at 9:21 PM Abhishek Somani <abhisheksoman...@gmail.com> wrote: > Hey Naresh, > > There is a `shaded-dependecies` project inside the root directory. You > need to go into that and build and publish that to local first. > > cd shaded-dependencies >> sbt clean publishLocal >> > > After that, come back out to the root directory and build that project. > The spark-acid-shaded-dependencies jar will now be found: > >> cd .. >> sbt assembly > > > This will create the jar which you can use. > > On another note, unless you are making changes in the code, you don't need > to build yourself as the jar is published in > https://spark-packages.org/package/qubole/spark-acid. So you can just use > it as: > > spark-shell --packages qubole:spark-acid:0.4.0-s_2.11 > > > ...and it will be automatically fetched and used. > > Thanks, > Abhishek > > > On Sun, Jul 28, 2019 at 4:42 AM naresh Goud <nareshgoud.du...@gmail.com> > wrote: > >> It looks there is some internal dependency missing. >> >> libraryDependencies ++= Seq( >> "com.qubole" %% "spark-acid-shaded-dependencies" % "0.1" >> ) >> >> How do we get it? >> >> >> Thank you, >> Naresh >> >> >> >> >> Thanks, >> Naresh >> www.linkedin.com/in/naresh-dulam >> http://hadoopandspark.blogspot.com/ >> >> >> >> On Sat, Jul 27, 2019 at 5:34 PM naresh Goud <nareshgoud.du...@gmail.com> >> wrote: >> >>> Hi Abhishek, >>> >>> >>> We are not able to build jar using git hub code with below error? >>> >>> Any others able to build jars? Is there anything else missing? >>> >>> >>> >>> Note: Unresolved dependencies path: >>> [warn] com.qubole:spark-acid-shaded-dependencies_2.11:0.1 >>> (C:\Data\Hadoop\spark-acid-master\build.sbt#L51-54) >>> [warn] +- com.qubole:spark-acid_2.11:0.4.0 >>> sbt.ResolveException: unresolved dependency: >>> com.qubole#spark-acid-shaded-dependencies_2.11;0.1: not found >>> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313) >>> at >>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191) >>> at >>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168) >>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156) >>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156) >>> at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133) >>> at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57) >>> at sbt.IvySbt$$anon$4.call(Ivy.scala:65) >>> at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93) >>> at >>> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78) >>> at >>> xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97) >>> at xsbt.boot.Using$.withResource(Using.scala:10) >>> at xsbt.boot.Using$.apply(Using.scala:9) >>> at >>> xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58) >>> at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48) >>> at xsbt.boot.Locks$.apply0(Locks.scala:31) >>> at xsbt.boot.Locks$.apply(Locks.scala:28) >>> at sbt.IvySbt.withDefaultLogger(Ivy.scala:65) >>> at sbt.IvySbt.withIvy(Ivy.scala:128) >>> at sbt.IvySbt.withIvy(Ivy.scala:125) >>> at sbt.IvySbt$Module.withModule(Ivy.scala:156) >>> at sbt.IvyActions$.updateEither(IvyActions.scala:168) >>> at >>> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1541) >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Thanks, >>> Naresh >>> www.linkedin.com/in/naresh-dulam >>> http://hadoopandspark.blogspot.com/ >>> >>> >>> >>> On Sat, Jul 27, 2019 at 3:25 PM Nicolas Paris <nicolas.pa...@riseup.net> >>> wrote: >>> >>>> Congrats >>>> >>>> The read/write feature with hive3 is highly interesting >>>> >>>> On Fri, Jul 26, 2019 at 06:07:55PM +0530, Abhishek Somani wrote: >>>> > Hi All, >>>> > >>>> > We at Qubole have open sourced a datasource that will enable users to >>>> work on >>>> > their Hive ACID Transactional Tables using Spark. >>>> > >>>> > Github: https://github.com/qubole/spark-acid >>>> > >>>> > Hive ACID tables allow users to work on their data transactionally, >>>> and also >>>> > gives them the ability to Delete, Update and Merge data efficiently >>>> without >>>> > having to rewrite all of their data in a table, partition or file. We >>>> believe >>>> > that being able to work on these tables from Spark is a much desired >>>> value add, >>>> > as is also apparent in >>>> https://issues.apache.org/jira/browse/SPARK-15348 and >>>> > https://issues.apache.org/jira/browse/SPARK-16996 with multiple >>>> people looking >>>> > for it. Currently the datasource supports reading from these ACID >>>> tables only, >>>> > and we are working on adding the ability to write into these tables >>>> via Spark >>>> > as well. >>>> > >>>> > The datasource is also available as a spark package, and instructions >>>> on how to >>>> > use it are available on the Github page. >>>> > >>>> > We welcome your feedback and suggestions. >>>> > >>>> > Thanks, >>>> > Abhishek Somani >>>> >>>> -- >>>> nicolas >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> -- Thanks, Naresh www.linkedin.com/in/naresh-dulam http://hadoopandspark.blogspot.com/
