I realised that the build instructions in the README.md were not very clear due to some recent changes. I have updated those now.
Thanks, Abhishek Somani On Sun, Jul 28, 2019 at 7:53 AM naresh Goud <nareshgoud.du...@gmail.com> wrote: > Thanks Abhishek. > I will check it out. > > Thank you, > Naresh > > On Sat, Jul 27, 2019 at 9:21 PM Abhishek Somani < > abhisheksoman...@gmail.com> wrote: > >> Hey Naresh, >> >> There is a `shaded-dependecies` project inside the root directory. You >> need to go into that and build and publish that to local first. >> >> cd shaded-dependencies >>> sbt clean publishLocal >>> >> >> After that, come back out to the root directory and build that project. >> The spark-acid-shaded-dependencies jar will now be found: >> >>> cd .. >>> sbt assembly >> >> >> This will create the jar which you can use. >> >> On another note, unless you are making changes in the code, you don't >> need to build yourself as the jar is published in >> https://spark-packages.org/package/qubole/spark-acid. So you can just >> use it as: >> >> spark-shell --packages qubole:spark-acid:0.4.0-s_2.11 >> >> >> ...and it will be automatically fetched and used. >> >> Thanks, >> Abhishek >> >> >> On Sun, Jul 28, 2019 at 4:42 AM naresh Goud <nareshgoud.du...@gmail.com> >> wrote: >> >>> It looks there is some internal dependency missing. >>> >>> libraryDependencies ++= Seq( >>> "com.qubole" %% "spark-acid-shaded-dependencies" % "0.1" >>> ) >>> >>> How do we get it? >>> >>> >>> Thank you, >>> Naresh >>> >>> >>> >>> >>> Thanks, >>> Naresh >>> www.linkedin.com/in/naresh-dulam >>> http://hadoopandspark.blogspot.com/ >>> >>> >>> >>> On Sat, Jul 27, 2019 at 5:34 PM naresh Goud <nareshgoud.du...@gmail.com> >>> wrote: >>> >>>> Hi Abhishek, >>>> >>>> >>>> We are not able to build jar using git hub code with below error? >>>> >>>> Any others able to build jars? Is there anything else missing? >>>> >>>> >>>> >>>> Note: Unresolved dependencies path: >>>> [warn] com.qubole:spark-acid-shaded-dependencies_2.11:0.1 >>>> (C:\Data\Hadoop\spark-acid-master\build.sbt#L51-54) >>>> [warn] +- com.qubole:spark-acid_2.11:0.4.0 >>>> sbt.ResolveException: unresolved dependency: >>>> com.qubole#spark-acid-shaded-dependencies_2.11;0.1: not found >>>> at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313) >>>> at >>>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191) >>>> at >>>> sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168) >>>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156) >>>> at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156) >>>> at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133) >>>> at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57) >>>> at sbt.IvySbt$$anon$4.call(Ivy.scala:65) >>>> at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93) >>>> at >>>> xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78) >>>> at >>>> xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97) >>>> at xsbt.boot.Using$.withResource(Using.scala:10) >>>> at xsbt.boot.Using$.apply(Using.scala:9) >>>> at >>>> xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58) >>>> at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48) >>>> at xsbt.boot.Locks$.apply0(Locks.scala:31) >>>> at xsbt.boot.Locks$.apply(Locks.scala:28) >>>> at sbt.IvySbt.withDefaultLogger(Ivy.scala:65) >>>> at sbt.IvySbt.withIvy(Ivy.scala:128) >>>> at sbt.IvySbt.withIvy(Ivy.scala:125) >>>> at sbt.IvySbt$Module.withModule(Ivy.scala:156) >>>> at sbt.IvyActions$.updateEither(IvyActions.scala:168) >>>> at >>>> sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1541) >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Thanks, >>>> Naresh >>>> www.linkedin.com/in/naresh-dulam >>>> http://hadoopandspark.blogspot.com/ >>>> >>>> >>>> >>>> On Sat, Jul 27, 2019 at 3:25 PM Nicolas Paris <nicolas.pa...@riseup.net> >>>> wrote: >>>> >>>>> Congrats >>>>> >>>>> The read/write feature with hive3 is highly interesting >>>>> >>>>> On Fri, Jul 26, 2019 at 06:07:55PM +0530, Abhishek Somani wrote: >>>>> > Hi All, >>>>> > >>>>> > We at Qubole have open sourced a datasource that will enable users >>>>> to work on >>>>> > their Hive ACID Transactional Tables using Spark. >>>>> > >>>>> > Github: https://github.com/qubole/spark-acid >>>>> > >>>>> > Hive ACID tables allow users to work on their data transactionally, >>>>> and also >>>>> > gives them the ability to Delete, Update and Merge data efficiently >>>>> without >>>>> > having to rewrite all of their data in a table, partition or file. >>>>> We believe >>>>> > that being able to work on these tables from Spark is a much desired >>>>> value add, >>>>> > as is also apparent in >>>>> https://issues.apache.org/jira/browse/SPARK-15348 and >>>>> > https://issues.apache.org/jira/browse/SPARK-16996 with multiple >>>>> people looking >>>>> > for it. Currently the datasource supports reading from these ACID >>>>> tables only, >>>>> > and we are working on adding the ability to write into these tables >>>>> via Spark >>>>> > as well. >>>>> > >>>>> > The datasource is also available as a spark package, and >>>>> instructions on how to >>>>> > use it are available on the Github page. >>>>> > >>>>> > We welcome your feedback and suggestions. >>>>> > >>>>> > Thanks, >>>>> > Abhishek Somani >>>>> >>>>> -- >>>>> nicolas >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>>> >>>>> -- > Thanks, > Naresh > www.linkedin.com/in/naresh-dulam > http://hadoopandspark.blogspot.com/ > >
