in mainframe you can have 3 type of datasources
--flat files
--VSAM files
--DB2/IMS
DB2 or IMS supprt the export utilities to copy the data into flat file
which you can get through ftp/sftp
VSAM file can be exported using IDCAMS utility
flat files can be get using the ftp/sftp
:::
Hi,
I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
Created table in HBase.Inserted records.Processing the data using Hive.
I have to show a graph with some points ( 7 - 7 days or 12 for one year).In
one day records may include 1000 - lacks.I need to show average of these
1000 - lacks re
Hi Balamurali,
As per my knowledge, there is nothing in the hadoop which does exactly as per
your requirement.
You can write mapreduce jobs according to your functionality and submit
hourly/daily/weekly or monthly . And then you can aggregate the results.
If you want some help regarding Hive,
At those columns, I am using uint type. I tried to cast them via sqoop
option still it gave the same error.
For other columns having type int, text etc, I am able to import them but
I have hundreds of data in uint type that I need.
While looking at some solutions, I saw that sqoop does not suppo
Hi,
I have 2 data set one of them contain string text, and other table contain
string patern (which is searching in text), id.
I have create volatile solution, to create two external hive table and full
join of them
and after full join,I use regex function in where case. but, it takes too
long. b
You can try writing the mapreduce job for this. In the Job, you can filter the
records in Mapper based on the where condition regex and then perform the join
in the Reducer.
Please refer the classes present in hadoop-datajoin module to get an idea how
to implement the join job.
Thanks
Devaraj
For scheduling mechanism please take a look at oozie.
Cheers
On Jul 22, 2013, at 10:37 PM, Balamurali wrote:
> Hi,
>
> I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
> Created table in HBase.Inserted records.Processing the data using Hive.
> I have to show a graph with some points
Ok thanks
On Tue, Jul 23, 2013 at 3:02 PM, Ted Yu wrote:
> For scheduling mechanism please take a look at oozie.
>
> Cheers
>
> On Jul 22, 2013, at 10:37 PM, Balamurali wrote:
>
> Hi,
>
> I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
> Created table in HBase.Inserted records.Proces
Thanks for your reply guys ,
i am looking for open source do we have any ??
On Tue, Jul 23, 2013 at 12:53 PM, Devaraj k wrote:
> Hi Balamurali,
>
> ** **
>
> As per my knowledge, there is nothing in the hadoop which does exactly as
> per your requirement.
>
> ** **
>
> You can write ma
I think you will have to write a custom code to handle this.
Regards,
Shahab
On Tue, Jul 23, 2013 at 3:50 AM, Fatih Haltas wrote:
> At those columns, I am using uint type. I tried to cast them via sqoop
> option still it gave the same error.
>
> For other columns having type int, text etc, I
Hi There,
First of all, sorry if I am asking some stupid question. Myself being new
to the Hadoop environment , am finding it a bit difficult to figure out why
its failing
I have installed hadoop 1.2, based on instructions given in the folllowing
link
http://www.michael-noll.com/tutorials/runnin
Hi,
I am just wondering whether I can move data from Ftp to Hdfs via Hadoop
distcp.
Can someone give me an example ?
In my case, I always encounter the "can not access ftp" error.
I am quite sure that the link, login et passwd are correct, actually, I
have just copy and paste the ftp addres
Maybe the conf file is missing or no privilege to access or there is
something wrong about the format of your conf files (hdfs-site, core-site,
mapred-site). You can double check them. Also probably the typo of the
tag or something like that.
2013/7/23 Ashish Umrani
> Hi There,
>
> First of al
Hi,
You might have missed some configuration (XML tags ), Please check all the
Conf files.
Thanks
On Tue, Jul 23, 2013 at 6:25 PM, Ashish Umrani wrote:
> Hi There,
>
> First of all, sorry if I am asking some stupid question. Myself being new
> to the Hadoop environment , am finding it a bit dif
Hi Sandeep,
I think Apache Oozie is something you are looking for. It provide workflow
management on Hadoop (and Pig, Hive, etc.) jobs and support continuously run
jobs in specific time period. Please refer: http://oozie.apache.org/docs/3.3.2/
for details.
Thanks,
Junping
- Original M
Hey thanks for response. I have changed 4 files during installation
core-site.xml
mapred-site.xml
hdfs-site.xml and
hadoop-env.sh
I could not find any issues except that all params in the hadoop-env.sh are
commented out. Only java_home is un commented.
If you have a quick minute can you ple
Hi Devaraj,
Thanks for the advice. That did the trick.
Thanks,
Max Lebedev
On Wed, Jul 17, 2013 at 10:51 PM, Devaraj k wrote:
> It seems, It is not taking the CutomOutputFormat for the Job. You need
> to set the custom output format class using the
> org.apache.hadoop.mapred.JobConf.setOutp
Hi,
I see that most of the tasks have only log.index created in
/opt/hadoop/logs/userlogs/jobId/task_attempt directory.
When does this happen ?
Is there a config setting for this OR this is a bug ?
Regards,
Ajay Srivastava
Hi Ashish
In your hdfs-site.xml within tag you need to have the
tag and inside a tag you can have , and
tags.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-Original Message-
From: Ashish Umrani
Date: Tue, 23 Jul 2013 09:28:00
To:
Reply-To: user@hadoop.apache
Seems *hdfs-site.xml has no property tag.*
2013/7/23 Ashish Umrani
> Hey thanks for response. I have changed 4 files during installation
>
> core-site.xml
> mapred-site.xml
> hdfs-site.xml and
> hadoop-env.sh
>
>
> I could not find any issues except that all params in the hadoop-env.sh
> are
Hi Ashish,
Please check in hdfs-site.xml.
It is missing.
Thanks.
On Tue, Jul 23, 2013 at 9:58 PM, Ashish Umrani wrote:
> Hey thanks for response. I have changed 4 files during installation
>
> core-site.xml
> mapred-site.xml
> hdfs-site.xml and
> hadoop-env.sh
>
>
> I could not find any is
Thanks Jitendra, Bejoy and Yexi,
I got past that. And now the ls command says it can not access the
directory. I am sure this is a permissions issue. I am just wondering
which directory and I missing permissions on.
Any pointers?
And once again, thanks a lot
Regards
ashish
*hduser@ashish-HP
Its warning not error...
Create a directory and then do ls ( In your case /user/hduser is not
created untill and unless for the first time you create a directory or put
some file)
hadoop fs -mkdir sample
hadoop fs -ls
I would suggest if you are getting pemission problem,
please check the foll
Hello Ashish,
Change the permissions of /app/hadoop/tmp to 755 and see if it helps.
Warm Regards,
Tariq
cloudfront.blogspot.com
On Tue, Jul 23, 2013 at 10:27 PM, Ashish Umrani wrote:
> Thanks Jitendra, Bejoy and Yexi,
>
> I got past that. And now the ls command says it can not access the
> di
Hey,
Thanks Shekhar. That worked like a chimp. Appreciate help from you all.
Now I will try to put files and try the word count or similar program.
Regards
ashish
On Tue, Jul 23, 2013 at 10:07 AM, Shekhar Sharma wrote:
> Its warning not error...
>
> Create a directory and then do ls ( In yo
Thanks,
But the issue was that there was no directory and hence it was not showing
anything. Adding a directory cleared the warning.
I appreciate your help.
Regards
ashish
On Tue, Jul 23, 2013 at 10:08 AM, Mohammad Tariq wrote:
> Hello Ashish,
>
> Change the permissions of /app/hadoop/tmp t
After starting i would suggest always check whether your NameNode and job
tracker UI are working or not and check the number of live nodes in both of
the UI..
Regards,
Som Shekhar Sharma
+91-8197243810
On Tue, Jul 23, 2013 at 10:41 PM, Ashish Umrani wrote:
> Thanks,
>
> But the issue was that t
Try..
*hadoop fs -ls /*
**
Thanks
On Tue, Jul 23, 2013 at 10:27 PM, Ashish Umrani wrote:
> Thanks Jitendra, Bejoy and Yexi,
>
> I got past that. And now the ls command says it can not access the
> directory. I am sure this is a permissions issue. I am just wondering
> which directory and I
Jitendra, Som,
Thanks. Issue was in not having any file there. Its working fine now.
I am able to do -ls and could also do -mkdir and -put.
Now is time to run the jar and apparently I am getting
no main manifest attribute, in wc.jar
But I believe its because of maven pom file does not have
hadoop jar wc.jar inputdata outputdestination
Regards,
Som Shekhar Sharma
+91-8197243810
On Tue, Jul 23, 2013 at 10:58 PM, Ashish Umrani wrote:
> Jitendra, Som,
>
> Thanks. Issue was in not having any file there. Its working fine now.
>
> I am able to do -ls and could also do -mkdir and -pu
Hi All,
Do any of you have or can refer me to some sample Java code that get the
tree structure of a HDFS directory, similar to the file system?
For example: I have a HDFS dir, called /data, inside data, there is
/data/valid and /data/invalid, and so on, so I would need to be able to get th
Thanks Shekhar,
The problem was not in my building of the jar. It was in fact in execution
I was running command
*hadoop -jar* input output
The problem was with -jar. It should be
*hadoop jar* input output
Thanks for help once again
regards
ashish
On Tue, Jul 23, 2013 at 10:31 AM, S
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
and
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)
Basically you can write your own function, p
Hi,
A common use case one want an ordered structure for, is for saving counters.
Naturally, I wanted to save the counters in a Mapfile:
for (long ix = 0; ix < MAXVALUE; ix++) {
mapfile.append(new Text("counter key of val " + ix), new
LongWritable(ix));
}
This however looks a bit
Hi,
If you intend to use those counters in further functions then I think Hadoop
will take care this by itself, you can explore combiner for the same. In
Sequence file mapfile and index files has its own functionality and as per my
understanding index file expect offset to move over map file and
Uhm...
You want to save the counters as in counts per job run or something? (Remember
HDFS == WORM)
Then you could do a sequence file and then use something like HBase to manage
the index.
(Every time you add a set of counters, you have a new file and a new index.)
Heck you could use HBase f
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)
On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham wrote:
> Hi All,
>Do any of you have or can refer me to
It could either mean that all those task-attempts are crashing before the
process itself is getting spawned (check TT logs) or those logs are getting
deleted after the fact. Suspect the earlier.
Thanks,
+Vinod
On Jul 23, 2013, at 9:33 AM, Ajay Srivastava wrote:
> Hi,
>
> I see that most of t
Hi Vinod,
Thanks. It seems that something else is going on -
Here is the content of log.index -
ajay-srivastava:userlogs ajay.srivastava$ cat
job_201307222115_0188/attempt_201307222115_0188_r_00_0/log.index
LOG_DIR:/opt/hadoop/bin/../logs/userlogs/job_201307222115_0188/attempt_201307222115_
Ah, I should've guessed that. You seem to have JVM reuse enabled. Even if JVMs
are reused, all the tasks write to the same files as they share the JVM. They
only have different index files. The same thing happens for what we call the
TaskCleanup tasks which are launched for failing/killed tasks
Yes. That explains it and confirms my guess too :-)
stderr:156 0
syslog:995 166247
What are these numbers ? Byte offset in corresponding files from where logs of
this task starts.
Regards,
Ajay Srivastava
On 24-Jul-2013, at 12:10 PM, Vinod Kumar Vavilapalli wrote:
Ah, I should've guessed
41 matches
Mail list logo
