date:20140930

Re: Passing options to giraph 1.1.0 - SNAPSHOT, Hadoop 2.2.0

2014-09-30 Thread Charith Wickramarachchi

Thanks that solved my issue.

--Charith

On Tue, Sep 30, 2014 at 11:12 PM,  wrote:

>  The "-D" options should be set just after "org.apache.GiraphRunner"
> Because it is this class that integrates  the ToolRunner interface that
> reads the configuration options set by "-D"
>
>  Hope it helps
> Olivier
>
> Le 1 oct. 2014 à 07:57, "Charith Wickramarachchi" <
> charith.dhanus...@gmail.com> a écrit :
>
>   Hi,
>
>  I am trying to pass some system options into the giraph job so that I
> can access it through the Giraph configuration.
>
>  I am using the following command
>
>  $HADOOP_HOME/bin/hadoop jar
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.examples.LongDoubleFloatTextInputFormat -vip
> /user/charith/input/tiny_graph_neighbour.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/charith/output/shortestpaths_el -w 2
> -Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
> -Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultLongLongMappingStoreOps
> -Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongLongMappingStorePartitionerFactory
> -Dgiraph.mappingLocationHdfs=/user/charith/input/tiny_graph_neighbour_map.txt
>
>
>  But it gives me following exception. "Exception in thread "main"
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
> "
>
>  Basically, I want to pass following properties to the job.
>
>  -Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
> -Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultLongLongMappingStoreOps
>  
> -Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongLongMappingStorePartitionerFactory
> -Dgiraph.mappingLocationHdfs=/user/charith/input/tiny_graph_neighbour_map.txt
>
>  It will be great if someone can help me in this regard.
>
>  Thanks,
> Charith
>
>
>
>
>  --
> Charith Dhanushka Wickramaarachchi
>
> Tel  +1 213 447 4253
> Web  http://apache.org/~charith 
> 
> Blog  http://charith.wickramaarachchi.org/
> 
> Twitter  @charithwiki 
>
>  This communication may contain privileged or other confidential information
> and is intended exclusively for the addressee/s. If you are not the
> intended recipient/s, or believe that you may have
> received this communication in error, please reply to the sender indicating
> that fact and delete the copy you received and in addition, you should
> not print, copy, retransmit, disseminate, or otherwise use the
> information contained in this communication. Internet communications
> cannot be guaranteed to be timely, secure, error or virus-free. The
> sender does not accept liability for any errors or omissions
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>


-- 
Charith Dhanushka Wickramaarachchi

Tel  +1 213 447 4253
Web  http://apache.org/~charith 

Blog  http://charith.wickramaarachchi.org/

Twitter  @charithwiki 

This communication may contain privileged or other confidential information
and is intended exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be
guaranteed to be timely, secure, error or virus-free. The sender does not
accept liability for any errors or omissions

Re: Passing options to giraph 1.1.0 - SNAPSHOT, Hadoop 2.2.0

2014-09-30 Thread olivier.varene

The "-D" options should be set just after "org.apache.GiraphRunner"
Because it is this class that integrates  the ToolRunner interface that reads 
the configuration options set by "-D"

Hope it helps
Olivier

Le 1 oct. 2014 à 07:57, "Charith Wickramarachchi" 
mailto:charith.dhanus...@gmail.com>> a écrit :

Hi,

I am trying to pass some system options into the giraph job so that I can 
access it through the Giraph configuration.

I am using the following command

$HADOOP_HOME/bin/hadoop jar 
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar 
org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.SimpleShortestPathsComputation -vif 
org.apache.giraph.examples.LongDoubleFloatTextInputFormat -vip 
/user/charith/input/tiny_graph_neighbour.txt -vof 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op 
/user/charith/output/shortestpaths_el -w 2 
-Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore 
-Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultLongLongMappingStoreOps
 
-Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongLongMappingStorePartitionerFactory
 -Dgiraph.mappingLocationHdfs=/user/charith/input/tiny_graph_neighbour_map.txt


But it gives me following exception. "Exception in thread "main" 
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: 
-Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
"

Basically, I want to pass following properties to the job.

-Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore 
-Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultLongLongMappingStoreOps
  
-Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongLongMappingStorePartitionerFactory
 -Dgiraph.mappingLocationHdfs=/user/charith/input/tiny_graph_neighbour_map.txt

It will be great if someone can help me in this regard.

Thanks,
Charith




--
Charith Dhanushka Wickramaarachchi

Tel  +1 213 447 4253
Web  
http://apache.org/~charith
Blog  http://charith.wickramaarachchi.org/
Twitter  @charithwiki

This communication may contain privileged or other confidential information and 
is intended exclusively for the addressee/s. If you are not the intended 
recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating 
that fact and delete the copy you received and in addition, you should not 
print, copy, retransmit, disseminate, or otherwise use the information 
contained in this communication. Internet communications cannot be guaranteed 
to be timely, secure, error or virus-free. The sender does not accept liability 
for any errors or omissions

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Passing options to giraph 1.1.0 - SNAPSHOT, Hadoop 2.2.0

2014-09-30 Thread Charith Wickramarachchi

Hi,

I am trying to pass some system options into the giraph job so that I can
access it through the Giraph configuration.

I am using the following command

$HADOOP_HOME/bin/hadoop jar
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.examples.LongDoubleFloatTextInputFormat -vip
/user/charith/input/tiny_graph_neighbour.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/charith/output/shortestpaths_el -w 2
-Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
-Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultLongLongMappingStoreOps
-Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongLongMappingStorePartitionerFactory
-Dgiraph.mappingLocationHdfs=/user/charith/input/tiny_graph_neighbour_map.txt


But it gives me following exception. "Exception in thread "main"
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
-Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
"

Basically, I want to pass following properties to the job.

-Dgiraph.mappingStoreClass=org.apache.giraph.mapping.LongLongMappingStore
-Dgiraph.mappingStoreOpsClass=org.apache.giraph.mapping.DefaultLongLongMappingStoreOps
 
-Dgiraph.graphPartitionerFactoryClass=org.apache.giraph.partition.LongLongMappingStorePartitionerFactory
-Dgiraph.mappingLocationHdfs=/user/charith/input/tiny_graph_neighbour_map.txt

It will be great if someone can help me in this regard.

Thanks,
Charith




-- 
Charith Dhanushka Wickramaarachchi

Tel  +1 213 447 4253
Web  http://apache.org/~charith 

Blog  http://charith.wickramaarachchi.org/

Twitter  @charithwiki 

This communication may contain privileged or other confidential information
and is intended exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be
guaranteed to be timely, secure, error or virus-free. The sender does not
accept liability for any errors or omissions

Compiling Giraph with Hadoop 2.5.1 & YARN

2014-09-30 Thread Nicola Vitucci

Hi all,

I've just tried compiling Giraph (from trunk and release-1.1) after
applying the GIRAPH-947 patch, but I am still unable to compile it
successfully. This is the command I use for compiling:

mvn package -DskipTests -Dhadoop.version=2.5.1 -Phadoop_yarn clean install

and this is the stack trace:



[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-source-plugin:2.1.2:jar (attach-sources)
on project giraph-core: Error adding directory to source archive.
$HOME/giraph/giraph-core/target/munged/maven-shared-archive-resources
isn't a directory. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
execute goal org.apache.maven.plugins:maven-source-plugin:2.1.2:jar
(attach-sources) on project giraph-core: Error adding directory to
source archive.
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error adding
directory to source archive.
at
org.apache.maven.plugin.source.AbstractSourceJarMojo.addDirectory(AbstractSourceJarMojo.java:410)
at
org.apache.maven.plugin.source.AbstractSourceJarMojo.createArchiver(AbstractSourceJarMojo.java:392)
at
org.apache.maven.plugin.source.AbstractSourceJarMojo.packageSources(AbstractSourceJarMojo.java:248)
at
org.apache.maven.plugin.source.AbstractSourceJarMojo.packageSources(AbstractSourceJarMojo.java:232)
at
org.apache.maven.plugin.source.AbstractSourceJarMojo.execute(AbstractSourceJarMojo.java:201)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: org.codehaus.plexus.archiver.ArchiverException:
$HOME/giraph/giraph-core/target/munged/maven-shared-archive-resources
isn't a directory.
at
org.codehaus.plexus.archiver.AbstractArchiver.addFileSet(AbstractArchiver.java:258)
at
org.codehaus.plexus.archiver.AbstractArchiver.addDirectory(AbstractArchiver.java:244)
at
org.codehaus.plexus.archiver.AbstractArchiver.addDirectory(AbstractArchiver.java:232)
at
org.apache.maven.plugin.source.AbstractSourceJarMojo.addDirectory(AbstractSourceJarMojo.java:406)
... 25 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR]   mvn  -rf :giraph-core



Any idea why it is failing? My impression is that there is a problem
with the folders, because I can find
$HOME/giraph/giraph-core/target/munged/munged/maven-shared-archive-resources
(note how 'munged' is repeated) but not
$HOME/giraph/giraph-core/target/munged/maven-shared-archive-resources.

Thank you,

Nicola

Re: Giraph 1.0 | Computation stuck at map 100% - reduce 0% for my algorithm only, at multi-node cluster

2014-09-30 Thread Panagiotis Eustratiadis

Hello Matthew, thanks for the answer.

Oddly enough the job isn't listed under the failed jobs, as it is still
running. The execution never ends unless I kill it from the command line.
And I did check the logs (I always do), but they don't say anything. By the
way I see no syslog, only stdout and stderr.

What I didn't mention in my previous post and it might help, is that the
algorithm executes perfectly on a single node cluster setup. And from the
fact that the SimpleShortestPathVertex runs on the multi node setup just
fine, we deduce that the multi node setup is correct (right?).

2014-09-30 15:24 GMT+03:00 Matthew Cornell :

> I'm new, but in my meager experience when it stops at map 100% it means
> there was an error somewhere. In Giraph I've often found it difficult to
> pin down what that error actually was (e.g., out of memory), but the logs
> are the first place to look. Just to clarify re: not finding outputs: Are
> you going to http://:50030/jobtracker.jsp and clicking on
> the failed job id (e.g., job_201409251209_0029 -> 
> http://:50030/jobdetails.jsp?jobid=job_201409251209_0029&refresh=0
> )? From there, click the "map" link in the table to see its tasks. (Giraph
> runs entirely as a map task, IIUC.) You should see tasks for the master
> plus your workers. If you click on one of them (e.g.,
> task_201409251209_0029_m_00 -> 
> http://:50030/taskdetails.jsp?tipid=task_201409251209_0029_m_00
> ) you should see what machine it ran on plus a link to the Task Logs. Click
> on "All" and you should see three sections for stdout, stderr, and syslog,
> the latter of which usually contains hints about what went wrong. You
> should check all the worker logs.
>
> Hope that helps.
>
>
> On Tue, Sep 30, 2014 at 2:53 AM, Panagiotis Eustratiadis <
> ep.pan@gmail.com> wrote:
>
>> Good morning,
>>
>> I have been having a problem the past few days which sadly I can't solve.
>>
>> First of all I set up a Hadoop 0.20.203.0 cluster of two nodes a master
>> and a slave. I followed this tutorial for the settings:
>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>>
>> Then I set up Giraph, and I built it properly with maven. When I run the
>> SimpleShortestPathVertex with number of workers = 2 it runs properly, and
>> gives me results which I can view from any of the two nodes. Also the
>> jobtracker at master:50030 and slave:50030 and everything else is working
>> as expected.
>>
>> However, when I try to run my own algorithm it hangs at map 100% reduce
>> 0% forever. I looked at SimpleShortestPathVertex for any configurations and
>> it has none. And the weird part is: the jobs at the jobtracker have no logs
>> at stdout or stderr. The only thing readable is the map task info:
>>
>> task_201409300940_0001_m_00 | 100.00% - MASTER_ZOOKEEPER_ONLY | 1
>> finished out of 2 on superstep -1
>> task_201409300940_0001_m_01 | 100.00% | startSuperstep: WORKER_ONLY -
>> Attempt=0, Superstep=-1
>> task_201409300940_0001_m_02 | 100.00% | startSuperstep: WORKER_ONLY -
>> Attempt=0, Superstep=-1
>>
>> Is there anything I'm overlooking? I have Googled the obvious stack
>> overflow solutions for two days now. Has anyone encountered anything
>> similar?
>>
>> Regards,
>> Panagiotis Eustratiadis.
>>
>
>
>
> --
> Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson
> Street, Amherst MA 01002 | matthewcornell.org
>

Re: Giraph 1.0 | Computation stuck at map 100% - reduce 0% for my algorithm only, at multi-node cluster

2014-09-30 Thread Matthew Cornell

I'm new, but in my meager experience when it stops at map 100% it means
there was an error somewhere. In Giraph I've often found it difficult to
pin down what that error actually was (e.g., out of memory), but the logs
are the first place to look. Just to clarify re: not finding outputs: Are
you going to http://:50030/jobtracker.jsp and clicking on
the failed job id (e.g., job_201409251209_0029 ->
http://:50030/jobdetails.jsp?jobid=job_201409251209_0029&refresh=0
)? From there, click the "map" link in the table to see its tasks. (Giraph
runs entirely as a map task, IIUC.) You should see tasks for the master
plus your workers. If you click on one of them (e.g.,
task_201409251209_0029_m_00 ->
http://:50030/taskdetails.jsp?tipid=task_201409251209_0029_m_00
) you should see what machine it ran on plus a link to the Task Logs. Click
on "All" and you should see three sections for stdout, stderr, and syslog,
the latter of which usually contains hints about what went wrong. You
should check all the worker logs.

Hope that helps.

On Tue, Sep 30, 2014 at 2:53 AM, Panagiotis Eustratiadis <
ep.pan@gmail.com> wrote:

> Good morning,
>
> I have been having a problem the past few days which sadly I can't solve.
>
> First of all I set up a Hadoop 0.20.203.0 cluster of two nodes a master
> and a slave. I followed this tutorial for the settings:
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>
> Then I set up Giraph, and I built it properly with maven. When I run the
> SimpleShortestPathVertex with number of workers = 2 it runs properly, and
> gives me results which I can view from any of the two nodes. Also the
> jobtracker at master:50030 and slave:50030 and everything else is working
> as expected.
>
> However, when I try to run my own algorithm it hangs at map 100% reduce 0%
> forever. I looked at SimpleShortestPathVertex for any configurations and it
> has none. And the weird part is: the jobs at the jobtracker have no logs at
> stdout or stderr. The only thing readable is the map task info:
>
> task_201409300940_0001_m_00 | 100.00% - MASTER_ZOOKEEPER_ONLY | 1
> finished out of 2 on superstep -1
> task_201409300940_0001_m_01 | 100.00% | startSuperstep: WORKER_ONLY -
> Attempt=0, Superstep=-1
> task_201409300940_0001_m_02 | 100.00% | startSuperstep: WORKER_ONLY -
> Attempt=0, Superstep=-1
>
> Is there anything I'm overlooking? I have Googled the obvious stack
> overflow solutions for two days now. Has anyone encountered anything
> similar?
>
> Regards,
> Panagiotis Eustratiadis.
>

-- 
Matthew Cornell | m...@matthewcornell.org | 413-626-3621 | 34 Dickinson
Street, Amherst MA 01002 | matthewcornell.org

Re: Passing options to giraph 1.1.0 - SNAPSHOT, Hadoop 2.2.0

Re: Passing options to giraph 1.1.0 - SNAPSHOT, Hadoop 2.2.0

Passing options to giraph 1.1.0 - SNAPSHOT, Hadoop 2.2.0

Compiling Giraph with Hadoop 2.5.1 & YARN

Re: Giraph 1.0 | Computation stuck at map 100% - reduce 0% for my algorithm only, at multi-node cluster

Re: Giraph 1.0 | Computation stuck at map 100% - reduce 0% for my algorithm only, at multi-node cluster

6 matches

Site Navigation

Mail list logo

Footer information