reference document which properties are set in which configuration file
Dear all, while configuring our hadoop cluster I wonder whether there exists a reference document that contains information about which configuration property has to be specified in which properties file. Especially I do not know where the mapred.task.tracker.http.address has to be set. Is it in the mapre-site.xml or in the hdfs-site.xml? any hint will be appreciated thanks Christian 8<-- Siemens AG Corporate Technology Corporate Research and Technologies CT T DE IT3 Otto-Hahn-Ring 6 81739 München, Deutschland Tel.: +49 89 636-42722 Fax: +49 89 636-41423 mailto:christian.kleegr...@siemens.com Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
AW: reference document which properties are set in which configuration file
Hi all, thank you for the answers. Especially the hints from Harsh were helpful. Since I was in a little hurry I tried out the different properties in the different files and finally succeeded. I think my worst problem was that I had a typo in the property definition. So if I take care of this in future I think I do not need a reference document any more. Thanks and best regards, Christian 8<-- Siemens AG Corporate Technology Corporate Research and Technologies CT T DE IT3 Otto-Hahn-Ring 6 81739 München, Deutschland Tel.: +49 89 636-42722 Fax: +49 89 636-41423 mailto:christian.kleegr...@siemens.com Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322 -Ursprüngliche Nachricht- Von: Praveen Sripati [mailto:praveensrip...@gmail.com] Gesendet: Samstag, 11. Februar 2012 04:58 An: common-user@hadoop.apache.org Betreff: Re: reference document which properties are set in which configuration file The mapred.task.tracker.http.address will go in the mapred-site.xml file. In the Hadoop installation directory check the core-default.xml, hdfs-default,xml and mapred-default.xml files to know about the different properties. Some of the properties which might be in the code may not be mentioned in the xml files and will be defaulted. Praveen On Tue, Feb 7, 2012 at 3:30 PM, Kleegrewe, Christian < christian.kleegr...@siemens.com> wrote: > Dear all, > > while configuring our hadoop cluster I wonder whether there exists a > reference document that contains information about which configuration > property has to be specified in which properties file. Especially I do not > know where the mapred.task.tracker.http.address has to be set. Is it in the > mapre-site.xml or in the hdfs-site.xml? > > any hint will be appreciated > > thanks > > Christian > > > 8<-- > Siemens AG > Corporate Technology > Corporate Research and Technologies > CT T DE IT3 > Otto-Hahn-Ring 6 > 81739 München, Deutschland > Tel.: +49 89 636-42722 > Fax: +49 89 636-41423 > mailto:christian.kleegr...@siemens.com > > Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard > Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte > Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, > Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: > Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, > HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322 > > >
AW: How to compile HBase code ?
How do you execute the client (command line) do you use the java or the hadoop command? It seems that there is an error in your classpath when running the client job. The classpath when compiling classes that implement the client is different from the classpath when your client is executed since hadoop and hbase carry their own environment. Maybe tha following link helps: http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath regards Christian ---8< Siemens AG Corporate Technology Corporate Research and Technologies CT T DE IT3 Otto-Hahn-Ring 6 81739 München, Deutschland Tel.: +49 (89) 636-42722 Fax: +49 (89) 636-41423 mailto:christian.kleegr...@siemens.com Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Wolfgang Dehen, Brigitte Ederer, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322 -Ursprüngliche Nachricht- Von: praveenesh kumar [mailto:praveen...@gmail.com] Gesendet: Dienstag, 24. Mai 2011 11:08 An: common-user@hadoop.apache.org Betreff: How to compile HBase code ? Hello guys, In case any of you are working on HBASE, I just wrote a program by reading some tutorials.. But no where its mentioned how to run codes on HBASE. In case anyone of you has done some coding on HBASE , can you please tell me how to run it. I am able to compile my code by adding hbase-core.jar and hadoop-core.jar in classpath while compiling it. But not able to figure out how to run it. Whenever I am doing java ExampleClient ( which is my Hbase program), I am getting the following error : Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration at ExampleClient.main(ExampleClient.java:20) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration at java.net.URLClassLoader$1.run(URLClassLoader.java:217) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:321) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) ... 1 more Thanks, Praveenesh
AW: How to compile HBase code ?
Are you sure that the directory where your ExampleClient.class is locates is
part of the MYCLASSPATH?
regards
Christian
---8<
Siemens AG
Corporate Technology
Corporate Research and Technologies
CT T DE IT3
Otto-Hahn-Ring 6
81739 München, Deutschland
Tel.: +49 (89) 636-42722
Fax: +49 (89) 636-41423
mailto:christian.kleegr...@siemens.com
Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme;
Vorstand: Peter Löscher, Vorsitzender; Wolfgang Dehen, Brigitte Ederer, Joe
Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen;
Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin
Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
-Ursprüngliche Nachricht-
Von: praveenesh kumar [mailto:praveen...@gmail.com]
Gesendet: Dienstag, 24. Mai 2011 11:54
An: common-user@hadoop.apache.org
Betreff: Re: How to compile HBase code ?
I am simply using HBase API, not doing any Map-reduce work on it.
Following is the code I have written , simply creating the file on HBase:
import java.io.IOException;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;
public class ExampleClient {
public static void main(String args []) throws IOException
{
HBaseConfiguration config = new HBaseConfiguration();
HBaseAdmin admin = new HBaseAdmin(config);
HTableDescriptor htd = new HTableDescriptor("test");
HColumnDescriptor hcd = new HColumnDescriptor("data");
htd.addFamily(hcd);
admin.createTable(htd);
byte [] tablename = htd.getName();
HTableDescriptor [] tables = admin.listTables();
if(tables.length !=1 && Bytes.equals(tablename, tables[0].getName()))
{
throw new IOException("Failed to create table");
}
HTable table = new HTable(config,tablename);
byte[] row1 = Bytes.toBytes("row1");
Put p1 = new Put(row1);
byte[] databytes = Bytes.toBytes("data");
p1.add(databytes,Bytes.toBytes("1"),Bytes.toBytes("value1"));
table.put(p1);
Get g = new Get(row1);
Result result = table.get(g);
System.out.println("Get : "+ result);
Scan scan = new Scan();
ResultScanner scanner = table.getScanner(scan);
try
{
for(Result scannerResult: scanner)
{
System.out.println("Scan : " + scannerResult);
}
}catch(Exception e ){
e.printStackTrace();
}
finally{
scanner.close();
}
table.close();
}
}
Now I have set the classpath variable in /etc/environment as
MYCLASSPATH="/usr/local/hadoop/hadoop/hadoop-0.20.2-core.jar:/usr/local/hadoop/hbase/hbase/hbase-0.20.6.jar:/usr/local/hadoop/hbase/hbase/lib/zookeeper-3.2.2.jar
now I am compiling my code with javac command
*$javac -classpath $MYCLASSPATH ExampleClient.java*
It is working fine.
While running, I am using java command
*$java -classpath $MYCLASSPATH ExampleClient*, then I am getting the
following error :
Exception in thread "main" java.lang.NoClassDefFoundError: ExampleClient
Caused by: java.lang.ClassNotFoundException: ExampleClient
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: ExampleClient. Program will exit.
But I am running the code from the same location. and ExampleClient.class
file exists at that location.
On Tue, May 24, 2011 at 3:07 PM, Kleegrewe, Christian <
christian.kleegr...@siemens.com> wrote:
> How do you execute the client (command line) do you use the java or the
> hadoop command?
> It seems that there is an error in your classpath when running the client
> job. The classpath when compiling classes that implement the client is
> different from the classpath when your client is executed since hadoop and
> hbase carry their own environment. Maybe tha following link helps:
>
>
> http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath
>
>
> regards
> Christian
>
> ---8<
>
> Siemens AG
> Corporate Technology
> Corporate Research an
