[
https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-13:
------------------------------
Attachment: GIRAPH-13-5.patch
Here's the latest placeholder. Things are going very well so far, almost done
(if my "faking out Mapper#Context" approach works.) right now I'm trying to get
to a point where I can test that part (might need a fake OutputComitter also)
but I need a better way to get locate my JARs on the local machine. In order to
distribute them w/o a distributed cache, we use the YARN LocalResources, and
that means finding them locally and putting them on HDFS to be downloaded by
all the worker tasks and the AppMaster. The code below is failing on the "URL
is null" case. Any ideas? I want to avoid assumptions about where the jars are.
In this case, I have my jars in $HADOOP_HOME/share/hadoop/... but they are not
showing up using my YarnGiraphClient's class loader to find them. I'll keep
playing with it, but if there's an obvious solution let me know ;)
Of course, I will be removing the hardcoded "giraph-examples" jar (and other
cruft) when I know all this works as well, and we can pick up whatever jars we
desire using the new -yj argument.
{code}
/**
* Utility function to locate local JAR files for trasmission to the
* remote app container where our Application Master will run.
* @param clazz the calling class who's classpath we want to hunt along.
* @param name the file name of the jar, without path information.
* @return the named jar, as a File object.
*/
public static File getLocalJarFile(final Class<?> clazz, final String name) {
java.net.URL jarUrl = clazz.getClassLoader().getResource(name);
if (jarUrl == null) {
throw new IllegalStateException("Could not locate local JAR: " + name);
}
LOG.info("[*] Local JAR path for " + name + " found: " + jarUrl.getPath());
return new File(jarUrl.getPath());
}
{code}
> Port Giraph to YARN
> -------------------
>
> Key: GIRAPH-13
> URL: https://issues.apache.org/jira/browse/GIRAPH-13
> Project: Giraph
> Issue Type: New Feature
> Reporter: Jakob Homan
> Assignee: Eli Reisman
> Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, GIRAPH-13-3.patch,
> GIRAPH-13-4.patch, GIRAPH-13-5.patch
>
>
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop
> trunk, we should think about what it would take to separate out the graph
> processing bits of Giraph from the MR1-specific code so as to take advantage
> of the less-MR centric aspects of YARN, while still supporting both over the
> medium term.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
