Not totally sure what exactly I'm looking to see, but I think that the hook libs are already there / symlinked. What I see is...
➜ ~# cd /usr/hdp/current/sqoop-server/lib ➜ lib# ls aircompressor-0.10.jar ant-contrib-1.0b3.jar ant-eclipse-1.0-jvm1.2.jar atlas-plugin-classloader-1.1.0.3.1.0.0-78.jar ... sqoop-bridge-shim-1.1.0.3.1.0.0-78.jar xz-1.5.jar ➜ lib# ls /usr/hdp/current/atlas-server/hook/sqoop/ atlas-plugin-classloader-1.1.0.3.1.0.0-78.jar atlas-sqoop-plugin-impl sqoop-bridge-shim-1.1.0.3.1.0.0-78.jar So it looks like the jars in the dir you recommend symlinking are already in the location they should be symlinked to. What specifically in the sqoop logs would you be looking for in for more info on it hooking into atlas? Thanks On Fri, Aug 16, 2019 at 12:45 AM Nixon Rodrigues < [email protected]> wrote: > Reed, > > Looks like atlas sqoop hook libs are not present while running sqoop job. > logs while running sqoop job would give more insights about this. > > Can you please check if Atlas sqoop hook libs are symlinked to sqoop libs. > > ls /usr/hdp/current/sqoop-server/lib . > > If atlas hook libs are not linked. create sym link for Atlas sqoop hook > lib > > cd /usr/hdp/current/sqoop-server/lib > ln -s /usr/hdp/current/atlas-server/hook/sqoop/ . > > HTH > > Regards, > Nixon > > > > > On Thu, Aug 8, 2019 at 7:15 AM Reed Villanueva <[email protected]> > wrote: > >> Using sqoop on hdp 3.1 to import Oracle DB data into HDFS as parquet and >> expecting it to appear in atlas as an entity (I assume it should be the >> default sqoop_process type). However, after running the sqoop job, >> nothing appears in atlas. Sqoop job looks like... >> >> sqoop import \ >> -Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \ >> -Dyarn.app.mapreduce.am.log.level=DEBUG \ >> -Dmapreduce.map.log.level=DEBUG \ >> -Dmapreduce.reduce.log.level=DEBUG \ >> -Dmapred.job.name="Ora import table $tablename" \ >> -Djava.security.egd=file:///dev/urandom \ >> -Djava.security.egd=file:///dev/urandom \ >> -Doraoop.timestamp.string=false \ >> -Dmapreduce.map.max.attempts=10 \ >> $oracle_cnxn_str \ >> --as-parquetfile \ >> --target-dir $importdir \ >> -query "select a.*, current_date as etl_date from $tablename a where 1=1 >> AND \$CONDITIONS" \ >> --split-by $splitby \ >> --where "1=1" \ >> --num-mappers 12 \ >> --delete-target-dir >> >> Looking in the hdp ambari interface for atlas, I can see that the sqoop >> plugin for atlas is configured[image: enter image description here] >> <https://i.stack.imgur.com/U7qYp.png>(did not initially do any manual >> tweaking to it after installing atlas according to the docs >> <https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/installing-ranger/content/ranger_install_ranger_plugin_settings.html>). >> Adding the property[image: enter image description here] >> <https://i.stack.imgur.com/LRukd.png>based on another Hortonworks post >> <http://community.hortonworks.com/answers/52257/view.html> and >> attempting the sqoop job again still shows nothing in the atlas UI. Anyone >> with more experience with hdp and atlas have any debugging suggestions or >> fixes? >> >> This electronic message is intended only for the named >> recipient, and may contain information that is confidential or >> privileged. If you are not the intended recipient, you are >> hereby notified that any disclosure, copying, distribution or >> use of the contents of this message is strictly prohibited. If >> you have received this message in error or are not the named >> recipient, please notify us immediately by contacting the >> sender at the electronic mail address noted above, and delete >> and destroy all copies of this message. Thank you. >> > -- This electronic message is intended only for the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
