Re: editing etc hosts files of a cluster
DNS ;) Ramesh.Ramasamy wrote: Hi, I have a cluster setup with 3 nodes, and I'm adding hostname details (in /etc/hosts) manually in each node. Seems it is not an effective approach. How this scenario is handled in big clusters? Is there any simple of way to add the hostname details in all the nodes by editing a single entry/file/script? Thanks and Regards, Ramesh -- *** The 'Last-Chance' Architect www.galatea.com (US) +1 303 731 3116 (UK) +44 20 8144 4367 ***
Re: Datanode Throwing NoRouteToHostException
].getLocalPort() returned 50070 2009-10-17 17:05:40,520 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070 2009-10-17 17:05:40,520 INFO org.mortbay.log: jetty-6.1.14 2009-10-17 17:05:46,612 INFO org.mortbay.log: Started selectchannelconnec...@0.0.0.0:50070 2009-10-17 17:05:46,612 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: 0.0.0.0:50070 2009-10-17 17:05:46,613 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2009-10-17 17:05:46,615 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting 2009-10-17 17:05:46,629 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54310: starting 2009-10-17 17:05:46,634 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54310: starting 2009-10-17 17:05:46,636 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54310: starting 2009-10-17 17:05:46,637 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54310: starting 2009-10-17 17:05:46,641 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310: starting 2009-10-17 17:05:46,642 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54310: starting 2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54310: starting 2009-10-17 17:05:46,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54310: starting 2009-10-17 17:05:46,900 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54310: starting 2009-10-17 17:05:46,916 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310: starting 2009-10-17 17:11:10,967 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.125 2009-10-17 17:11:10,968 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 2009-10-17 17:11:11,518 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll FSImage from 192.168.1.125 2009-10-17 17:11:11,519 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 2 -- *** The 'Last-Chance' Architect www.galatea.com (US) +1 303 731 3116 (UK) +44 20 8144 4367 ***
Re: Hadoop Developer Needed
Hi Alex, I'm a senior Java/J2EE developer/architect. I am American, but live abroad. I've been using Hadoop for the past year, but have many years experience in the field of large-scale data analysis, warehousing, etc. Let me know what you think. Cheers, Lajos alevin wrote: Overview The SQL Data Migration Specialist plays a crucial role in converting new Client's data onto Brilig's service platforms. We are looking for a talented and energetic full-time freelance programmer to work both remotely and onsite at our midtown Manhattan location. The Specialist will work with our clients' technical teams to determine the optimal formats and requirements to create files for subsequent import into a Brilig remote database during the implementation process. This process involves extracting, scrubbing, combining, transforming, validating and importing large data tables into final data sets suitable for loading into Brilig's defined databases. The Specialist will be responsible for creating/editing the database structure and writing of all import scripts and programs. The ability to work on multiple projects simultaneously while meeting tight deadlines is critical. This project will last for 3 months but may be extended or may eventually lead to a full time position in our fun and exciting startup. Must be able to travel to client meetings and work independently. Responsibilities: - Subject Matter Expert on software tools used in the entire data migration process from extraction to validation and load - Design, develop and execute quality data movement processes that are consistent, repeatable and scalable - Streamline testing, audit and validation processes through data scrubbing routines and presentation of audit reports prior to load - Roll out newly developed processes via documentation and training - Maintain and manage a template library of executed solutions to leverage against future opportunities - Identify, clarify, and resolve issues and risks, escalating them as needed - Build and nourish strong business relationships with external clients Please include: - Salary Requirements - Availability Experience - At least 3-5 years experience in the development of java applications - Use of XML and other protocols for data exchange between systems - SQL database design and implementation - Experience with Eclipse, Maven, and SVN a plus - Experience with Htable and Hadoop a big plus - Excellent communication skills with both technical and non-technical colleagues - Upper management and client facing skills - Interest in keeping up with technology advances PLEASE NOTE: US citizens and Green Card Holders and those authorized to work in the US only. We are unable to sponsor or transfer H-1B candidates. Contact: Alex Levin, COO Brilig ale...@brilig.com -- *** The 'Last-Chance' Architect www.galatea.com (US) +1 303 731 3116 (UK) +44 20 8144 4367 ***
Re: Hadoop Developer Needed
Whoops guys, sorry. Hit the reply too soon ;) Lajos Last-chance Architect wrote: Hi Alex, I'm a senior Java/J2EE developer/architect. I am American, but live abroad. I've been using Hadoop for the past year, but have many years experience in the field of large-scale data analysis, warehousing, etc. Let me know what you think. Cheers, Lajos alevin wrote: Overview The SQL Data Migration Specialist plays a crucial role in converting new Client's data onto Brilig's service platforms. We are looking for a talented and energetic full-time freelance programmer to work both remotely and onsite at our midtown Manhattan location. The Specialist will work with our clients' technical teams to determine the optimal formats and requirements to create files for subsequent import into a Brilig remote database during the implementation process. This process involves extracting, scrubbing, combining, transforming, validating and importing large data tables into final data sets suitable for loading into Brilig's defined databases. The Specialist will be responsible for creating/editing the database structure and writing of all import scripts and programs. The ability to work on multiple projects simultaneously while meeting tight deadlines is critical. This project will last for 3 months but may be extended or may eventually lead to a full time position in our fun and exciting startup. Must be able to travel to client meetings and work independently. Responsibilities: - Subject Matter Expert on software tools used in the entire data migration process from extraction to validation and load - Design, develop and execute quality data movement processes that are consistent, repeatable and scalable - Streamline testing, audit and validation processes through data scrubbing routines and presentation of audit reports prior to load - Roll out newly developed processes via documentation and training - Maintain and manage a template library of executed solutions to leverage against future opportunities - Identify, clarify, and resolve issues and risks, escalating them as needed - Build and nourish strong business relationships with external clients Please include: - Salary Requirements - Availability Experience - At least 3-5 years experience in the development of java applications - Use of XML and other protocols for data exchange between systems - SQL database design and implementation - Experience with Eclipse, Maven, and SVN a plus - Experience with Htable and Hadoop a big plus - Excellent communication skills with both technical and non-technical colleagues - Upper management and client facing skills - Interest in keeping up with technology advances PLEASE NOTE: US citizens and Green Card Holders and those authorized to work in the US only. We are unable to sponsor or transfer H-1B candidates. Contact: Alex Levin, COO Brilig ale...@brilig.com -- *** The 'Last-Chance' Architect www.galatea.com (US) +1 303 731 3116 (UK) +44 20 8144 4367 ***
Re: Map Recude code doubt
Shwitzu, Why can't you just use query thru a Filesystem object and find the file you want? Lajos shwitzu wrote: Hello All, I was wondering if our map reduce code can just return the location of the file? Or place the actual file in a given output directory by searching based on a keyword. Let me make myself clear If there are 100 image files in my HDFS and I want to extract one image file. If I give a keyword which matches the name of a file in my HDFS will my mapper and reducer code be able to locate that file and put back the original file in a given location?? Please let me know if you have need more information. Thanks -- *** The 'Last-Chance' Architect www.galatea.com (US) +1 303 731 3116 (UK) +44 20 8144 4367 ***
