Task Tracker not starting
Hi, Im trying to build a Fresh hadoop cluster. Im using cloudera manager 5.5.1 and CDH 5.5.2. Hdfs is up and running. When i start Map Reduce (MR1) job tracker is up but all task trackers are not starting. error: Command aborted because of exception: Command timed-out after 150 seconds Any help? regards, siva
Spark streaming
Hi,
I'm pulling some twitter data and trying to save the data into
persistent table.This is the code written.
case class Tweet(createdAt:Long, text:String)
twt.map(status=>
Tweet(status.getCreatedAt().getTime()/1000, status.getText())
).foreachRDD(rdd=>
rdd.toDF().saveAsTable("stream",SaveMode.Append)
)
When I go to spark-sql an check , i can see the table created. When im
trying to retrieve data im getting below error.
* java.lang.RuntimeException:
file:/user/hive/warehouse/stream/_temporary/0/_temporary/attempt_201602101609_0383_r_14_0/part-r-00664.parquet
is not a Parquet file (too small)*
Is this the correct way to store the streaming data into a persistent table?
Any help?
Thanks in Advance
Siva.
spark streaming
Hi, Im new to Spark. I need to pull Twitter data based on some keyword using scala and store them in spark sql table. Can anyone help me out to work on this? Im using scala 1.6.0 Thanks and regrads, siva
spark streaming
Hi ,
Im trying to pull data from twitter and do some processing out of
it. I found this code
import org.apache.spark.streaming._import
org.apache.spark.streaming.twitter._import
org.apache.spark.storage.StorageLevelimport scala.io.Sourceimport
scala.collection.mutable.HashMapimport java.io.Fileimport
org.apache.log4j.Loggerimport org.apache.log4j.Levelimport
sys.process.stringSeqToProcess
/** Configures the Oauth Credentials for accessing Twitter */def
configureTwitterCredentials(apiKey: String, apiSecret: String,
accessToken: String, accessTokenSecret: String) {
val configs = new HashMap[String, String] ++= Seq(
"apiKey" -> apiKey, "apiSecret" -> apiSecret, "accessToken" ->
accessToken, "accessTokenSecret" -> accessTokenSecret)
println("Configuring Twitter OAuth")
configs.foreach{ case(key, value) =>
if (value.trim.isEmpty) {
throw new Exception("Error setting authentication - value for "
+ key + " not set")
}
val fullKey = "twitter4j.oauth." + key.replace("api", "consumer")
System.setProperty(fullKey, value.trim)
println("\tProperty " + fullKey + " set as [" + value.trim + "]")
}
println()}
// Configure Twitter credentialsval apiKey =
"x"val apiSecret =
"xx"val accessToken =
"xx"val
accessTokenSecret =
"x"configureTwitterCredentials(apiKey,
apiSecret, accessToken, accessTokenSecret)
import org.apache.spark.streaming.twitter._val ssc = new
StreamingContext(sc, Seconds(2))val tweets =
TwitterUtils.createStream(ssc, None)val twt =
tweets.window(Seconds(60))
case class Tweet(createdAt:Long, text:String)twt.map(status=>
Tweet(status.getCreatedAt().getTime()/1000,
status.getText())).foreachRDD(rdd=>
// Below line works only in spark 1.3.0.
// For spark 1.1.x and spark 1.2.x,
// use rdd.registerTempTable("tweets") instead.
rdd.toDF().registerAsTable("tweets"))
twt.print
ssc.start()
But getting the below error
import org.apache.spark.streaming.twitter._
:19: error: object twitter is not a member of package
org.apache.spark.streaming
import org.apache.spark.streaming.twitter._
I using spark 1.3.0
The same worked for spark 0.9.0 . Any help?
Re: failed to start namenode
Hi Daniel, My host is up and running. On 11/20/15, Daniel Haviv <danielru...@gmail.com> wrote: > Are you sure the host is up? > > On Friday, 20 November 2015, siva kumar <siva165...@gmail.com> wrote: > >> Hi Sandeep, >> Im tryning to start using cloudera manager . This >> is the error message im getting. The log is not getting generated in the >> log directory. >> >> Supervisor returned FATAL. Please check the role log file, stderr, or >> stdout. >> >> >> HTTP ERROR 502 >> >> Problem accessing /cmf/process/54/logs. Reason: >> >> No route to host >> Connection may be blocked by host's firewall. >> >> Log Details >> HostC15742 <http://172.20.98.12:7180/cmf/hardware/host?hostname=C15742> >> Change... >> <http://172.20.98.12:7180/cmf/process/all/logs/context?timestamp=1448021817565=9000=C15742=%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-NAMENODE-C15742.log.out=96#>RoleNameNode >> - Change... >> <http://172.20.98.12:7180/cmf/process/all/logs/context?timestamp=1448021817565=9000=C15742=%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-NAMENODE-C15742.log.out=96#>File/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-C15742.log.out >> >> >> - >> TimeLog LevelSourceLog Message >> >> No log messages at the specified URL >> >> >> On Thu, Nov 19, 2015 at 11:39 AM, sandeep das <yarnhad...@gmail.com >> <javascript:_e(%7B%7D,'cvml','yarnhad...@gmail.com');>> wrote: >> >>> Its surprising that no logs are created. How are you trying to start >>> Name >>> node? If you are starting using cloudera manager then logs can be seen >>> found out on screen as well. >>> >>> On Thu, Nov 19, 2015 at 11:36 AM, siva kumar <siva165...@gmail.com >>> <javascript:_e(%7B%7D,'cvml','siva165...@gmail.com');>> wrote: >>> >>>> Hi Sandeep, >>>> The log is not getting generated for the name >>>> node. >>>> >>>> On Wed, Nov 18, 2015 at 5:53 PM, sandeep das <yarnhad...@gmail.com >>>> <javascript:_e(%7B%7D,'cvml','yarnhad...@gmail.com');>> wrote: >>>> >>>>> At least share some excerpts from log of name node log file. >>>>> >>>>> On Wed, Nov 18, 2015 at 5:46 PM, siva kumar <siva165...@gmail.com >>>>> <javascript:_e(%7B%7D,'cvml','siva165...@gmail.com');>> wrote: >>>>> >>>>>> Hi Folks, >>>>>> I'm trying to install a fresh hadoop cluster. But >>>>>> then,namenode is not is not starting up because of which hdfs service >>>>>> is >>>>>> not started during my first run. Can anyone help me out? >>>>>> I'm trying this using parcels(CDH-5). >>>>>> >>>>>> Any help? >>>>>> >>>>> >>>>> >>>> >>> >> > - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Re: failed to start namenode
Hi Sandeep, Im tryning to start using cloudera manager . This is the error message im getting. The log is not getting generated in the log directory. Supervisor returned FATAL. Please check the role log file, stderr, or stdout. HTTP ERROR 502 Problem accessing /cmf/process/54/logs. Reason: No route to host Connection may be blocked by host's firewall. Log Details HostC15742 <http://172.20.98.12:7180/cmf/hardware/host?hostname=C15742> Change... <http://172.20.98.12:7180/cmf/process/all/logs/context?timestamp=1448021817565=9000=C15742=%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-NAMENODE-C15742.log.out=96#>RoleNameNode - Change... <http://172.20.98.12:7180/cmf/process/all/logs/context?timestamp=1448021817565=9000=C15742=%2Fvar%2Flog%2Fhadoop-hdfs%2Fhadoop-cmf-hdfs-NAMENODE-C15742.log.out=96#>File/var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-C15742.log.out - TimeLog LevelSourceLog Message No log messages at the specified URL On Thu, Nov 19, 2015 at 11:39 AM, sandeep das <yarnhad...@gmail.com> wrote: > Its surprising that no logs are created. How are you trying to start Name > node? If you are starting using cloudera manager then logs can be seen > found out on screen as well. > > On Thu, Nov 19, 2015 at 11:36 AM, siva kumar <siva165...@gmail.com> wrote: > >> Hi Sandeep, >> The log is not getting generated for the name >> node. >> >> On Wed, Nov 18, 2015 at 5:53 PM, sandeep das <yarnhad...@gmail.com> >> wrote: >> >>> At least share some excerpts from log of name node log file. >>> >>> On Wed, Nov 18, 2015 at 5:46 PM, siva kumar <siva165...@gmail.com> >>> wrote: >>> >>>> Hi Folks, >>>> I'm trying to install a fresh hadoop cluster. But >>>> then,namenode is not is not starting up because of which hdfs service is >>>> not started during my first run. Can anyone help me out? >>>> I'm trying this using parcels(CDH-5). >>>> >>>> Any help? >>>> >>> >>> >> >
failed to start namenode
Hi Folks, I'm trying to install a fresh hadoop cluster. But then,namenode is not is not starting up because of which hdfs service is not started during my first run. Can anyone help me out? I'm trying this using parcels(CDH-5). Any help?
Re: failed to start namenode
Hi Sandeep, The log is not getting generated for the name node. On Wed, Nov 18, 2015 at 5:53 PM, sandeep das <yarnhad...@gmail.com> wrote: > At least share some excerpts from log of name node log file. > > On Wed, Nov 18, 2015 at 5:46 PM, siva kumar <siva165...@gmail.com> wrote: > >> Hi Folks, >> I'm trying to install a fresh hadoop cluster. But >> then,namenode is not is not starting up because of which hdfs service is >> not started during my first run. Can anyone help me out? >> I'm trying this using parcels(CDH-5). >> >> Any help? >> > >
Hdfs service failed to start
Hi Folks, My first run fails while installing CDH-5 (parcels) using cloudera manager. Below is the error message . Service did not start successfully; not all of the required roles started: Service has only 0 NameNode roles running instead of minimum required 1. Any help?
Re: unknown host exception
Hi Ted Yu, Im using hbase 0.96 Hi Anubhav, yes.i can ping the machine.also,i have added the IP in the hostfile.Im able to telnet to the zookeeper port to the host machine. Thanks On Fri, Sep 25, 2015 at 7:32 PM, Anubhav Agarwal <anubha...@gmail.com> wrote: > Can you ping to that host? Maybe your computer is unable to resolve that > hostname. Add the IP to your hosts file. > > On Fri, Sep 25, 2015 at 9:02 AM, siva kumar <siva165...@gmail.com> wrote: > >> Hi Folks, >> Im trying to write some data into hbase using >> pentaho. Im facing issue in connecting to hbase using hbase outpout step. >> >> com.google.protobuf.serviceexception: java.net.unknownhostexception: >> unlnown host : x >> >> >> Any suggestions? >> Thanks in advance. >> siva >> > >
unknown host exception
Hi Folks, Im trying to write some data into hbase using pentaho. Im facing issue in connecting to hbase using hbase outpout step. com.google.protobuf.serviceexception: java.net.unknownhostexception: unlnown host : x Any suggestions? Thanks in advance. siva
cassandra + tableau
Hi Folks, Can anyone suggest me the open source connector to connect tableau to cassandra database. ? Thanks, sivakumar.c
Why Only Mongodb suits MEAN techstack
Hi folks, MEAN (Mongodb,Express,Angular.js,Node.js) techstack is the recent buzz in the market for building Web Applications. My confuison is, why only Mongodb suits the best when compared to other NoSql databases like(hbase,cassandra). If Mongodb is best suited, then why ? Also, can we use hive with angular.js and node.js ?. Can someone help me out with your best answers?. Thanks and regards, sivakumar.c
Re: hadoop yarn
Thanks Rohit. Do we have any examples on MR2 other than wordcount, bcoz i dnt find much difference for word count example for both MR1 and MR2. Im new to yarn, so if you suggest me any example programs on MR2 it could help me out in a better way. Thanks and regards, siva On Tue, Jan 20, 2015 at 11:45 AM, Rohith Sharma K S rohithsharm...@huawei.com wrote: Refer below link, http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html Thanks Regards Rohith Sharma K S *From:* siva kumar [mailto:siva165...@gmail.com] *Sent:* 20 January 2015 11:24 *To:* user@hadoop.apache.org *Subject:* hadoop yarn Hi All, Can anyone suggest me few links for writing MR2 program on Yarn ? Thanks and regrads, siva
