[
https://issues.apache.org/jira/browse/PIG-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630373#comment-13630373
]
Dan Harvey commented on PIG-2927:
---------------------------------
I've had a look over this and I think I get the general approach. We're going
over all the gem's loaded for the ruby file required, and using the locally
installed gem to package in the Jar for the UDF to use?
If so I'm not sure that's the best approach, in Ruby you generally use the
Gemfile with bundler to specify _where_ you want to get the dependencies from
and what _version_, then the require in the code to specify _what_ you need to
use for that code specifically. By loading them locally we end up having lots
of issues with having to have the correct version installed and used by default
locally which is not always the case. You also can't lock the versions use in
committed code, which you can do with a Gemfile.lock file.
What I've done to work with ruby UDFs in Pig at the moment is use Warbler to
package up a Jar of dependencies from a Gemfile then also require this Jar in
the pig script. This way I can quickly change where I pull dependencies from
that being local, github, different branches on github, etc...
https://github.com/EqualMedia/warbler/tree/just_jar_gemfile
{code}
j = Warbler::Jar.new
j.apply(Warbler::Config.new)
j.create("gem.jar")
{code}
As the changes to using jRuby 1.7 are in here too what I think makes sense is
we split out the uploading dependencies from using jRuby 1.7. The 1.7 is
probably quicker to get in? then I could maybe look at adding bundler support
if we think that's a preferred approach?
> SHIP and use JRuby gems in JRuby UDFs
> -------------------------------------
>
> Key: PIG-2927
> URL: https://issues.apache.org/jira/browse/PIG-2927
> Project: Pig
> Issue Type: New Feature
> Components: parser
> Affects Versions: 0.11
> Environment: JRuby UDFs
> Reporter: Russell Jurney
> Assignee: Jonathan Coveney
> Priority: Minor
> Fix For: 0.12
>
> Attachments: PIG-2927-0.patch, PIG-2927-1.patch, PIG-2927-2.patch,
> PIG-2927-3.patch, PIG-2927-4.patch
>
>
> It would be great to use JRuby gems in JRuby UDFs without installing them on
> all machines on the cluster. Some way to SHIP them automatically with the job
> would be great.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
