Hi ,
How to copy datasets from Mainframe to HDFS directly? I know that we can
NDM files to Linux box and then we can use simple put command to copy data
to HDFS. But, how to copy data directly from mainframe to HDFS? I have
PS, PDS and VSAM datasets to copy to HDFS for analysis using
Hello Sandeep,
You don't have to convert the data in order to copy it into the HDFS. But
you might have to think about the MR processing of these files because of
the format of these files.
You could probably make use of Sqoop http://sqoop.apache.org/.
I also came across DMX-H a few days ago
in mainframe you can have 3 type of datasources
--flat files
--VSAM files
--DB2/IMS
DB2 or IMS supprt the export utilities to copy the data into flat file
which you can get through ftp/sftp
VSAM file can be exported using IDCAMS utility
flat files can be get using the ftp/sftp
Hi,
I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
Created table in HBase.Inserted records.Processing the data using Hive.
I have to show a graph with some points ( 7 - 7 days or 12 for one year).In
one day records may include 1000 - lacks.I need to show average of these
1000 - lacks
Hi Balamurali,
As per my knowledge, there is nothing in the hadoop which does exactly as per
your requirement.
You can write mapreduce jobs according to your functionality and submit
hourly/daily/weekly or monthly . And then you can aggregate the results.
If you want some help regarding Hive,
At those columns, I am using uint type. I tried to cast them via sqoop
option still it gave the same error.
For other columns having type int, text etc, I am able to import them but
I have hundreds of data in uint type that I need.
While looking at some solutions, I saw that sqoop does not
Hi,
I have 2 data set one of them contain string text, and other table contain
string patern (which is searching in text), id.
I have create volatile solution, to create two external hive table and full
join of them
and after full join,I use regex function in where case. but, it takes too
long.
You can try writing the mapreduce job for this. In the Job, you can filter the
records in Mapper based on the where condition regex and then perform the join
in the Reducer.
Please refer the classes present in hadoop-datajoin module to get an idea how
to implement the join job.
Thanks
Devaraj
For scheduling mechanism please take a look at oozie.
Cheers
On Jul 22, 2013, at 10:37 PM, Balamurali balamurali...@gmail.com wrote:
Hi,
I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
Created table in HBase.Inserted records.Processing the data using Hive.
I have to show a
Ok thanks
On Tue, Jul 23, 2013 at 3:02 PM, Ted Yu yuzhih...@gmail.com wrote:
For scheduling mechanism please take a look at oozie.
Cheers
On Jul 22, 2013, at 10:37 PM, Balamurali balamurali...@gmail.com wrote:
Hi,
I configured hadoop-1.0.3, hbase-0.92.1 and hive-0.10.0 .
Created table
Thanks for your reply guys ,
i am looking for open source do we have any ??
On Tue, Jul 23, 2013 at 12:53 PM, Devaraj k devara...@huawei.com wrote:
Hi Balamurali,
** **
As per my knowledge, there is nothing in the hadoop which does exactly as
per your requirement.
** **
You
I think you will have to write a custom code to handle this.
Regards,
Shahab
On Tue, Jul 23, 2013 at 3:50 AM, Fatih Haltas fatih.hal...@nyu.edu wrote:
At those columns, I am using uint type. I tried to cast them via sqoop
option still it gave the same error.
For other columns having type
Hi There,
First of all, sorry if I am asking some stupid question. Myself being new
to the Hadoop environment , am finding it a bit difficult to figure out why
its failing
I have installed hadoop 1.2, based on instructions given in the folllowing
link
Hi,
I am just wondering whether I can move data from Ftp to Hdfs via Hadoop
distcp.
Can someone give me an example ?
In my case, I always encounter the can not access ftp error.
I am quite sure that the link, login et passwd are correct, actually, I
have just copy and paste the ftp address
Maybe the conf file is missing or no privilege to access or there is
something wrong about the format of your conf files (hdfs-site, core-site,
mapred-site). You can double check them. Also probably the typo of the
property/property tag or something like that.
2013/7/23 Ashish Umrani
Hi,
You might have missed some configuration (XML tags ), Please check all the
Conf files.
Thanks
On Tue, Jul 23, 2013 at 6:25 PM, Ashish Umrani ashish.umr...@gmail.comwrote:
Hi There,
First of all, sorry if I am asking some stupid question. Myself being new
to the Hadoop environment , am
Hi Sandeep,
I think Apache Oozie is something you are looking for. It provide workflow
management on Hadoop (and Pig, Hive, etc.) jobs and support continuously run
jobs in specific time period. Please refer: http://oozie.apache.org/docs/3.3.2/
for details.
Thanks,
Junping
- Original
Hey thanks for response. I have changed 4 files during installation
core-site.xml
mapred-site.xml
hdfs-site.xml and
hadoop-env.sh
I could not find any issues except that all params in the hadoop-env.sh are
commented out. Only java_home is un commented.
If you have a quick minute can you
Hi Devaraj,
Thanks for the advice. That did the trick.
Thanks,
Max Lebedev
On Wed, Jul 17, 2013 at 10:51 PM, Devaraj k devara...@huawei.com wrote:
It seems, It is not taking the CutomOutputFormat for the Job. You need
to set the custom output format class using the
Hi,
I see that most of the tasks have only log.index created in
/opt/hadoop/logs/userlogs/jobId/task_attempt directory.
When does this happen ?
Is there a config setting for this OR this is a bug ?
Regards,
Ajay Srivastava
Hi Ashish
In your hdfs-site.xml within configuration tag you need to have the
property tag and inside a property tag you can have name,value and
description tags.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-Original Message-
From: Ashish Umrani
Seems *hdfs-site.xml has no property tag.*
2013/7/23 Ashish Umrani ashish.umr...@gmail.com
Hey thanks for response. I have changed 4 files during installation
core-site.xml
mapred-site.xml
hdfs-site.xml and
hadoop-env.sh
I could not find any issues except that all params in the
Hi Ashish,
Please check property/property in hdfs-site.xml.
It is missing.
Thanks.
On Tue, Jul 23, 2013 at 9:58 PM, Ashish Umrani ashish.umr...@gmail.comwrote:
Hey thanks for response. I have changed 4 files during installation
core-site.xml
mapred-site.xml
hdfs-site.xml and
Thanks Jitendra, Bejoy and Yexi,
I got past that. And now the ls command says it can not access the
directory. I am sure this is a permissions issue. I am just wondering
which directory and I missing permissions on.
Any pointers?
And once again, thanks a lot
Regards
ashish
Its warning not error...
Create a directory and then do ls ( In your case /user/hduser is not
created untill and unless for the first time you create a directory or put
some file)
hadoop fs -mkdir sample
hadoop fs -ls
I would suggest if you are getting pemission problem,
please check the
Hello Ashish,
Change the permissions of /app/hadoop/tmp to 755 and see if it helps.
Warm Regards,
Tariq
cloudfront.blogspot.com
On Tue, Jul 23, 2013 at 10:27 PM, Ashish Umrani ashish.umr...@gmail.comwrote:
Thanks Jitendra, Bejoy and Yexi,
I got past that. And now the ls command says it
Hey,
Thanks Shekhar. That worked like a chimp. Appreciate help from you all.
Now I will try to put files and try the word count or similar program.
Regards
ashish
On Tue, Jul 23, 2013 at 10:07 AM, Shekhar Sharma shekhar2...@gmail.comwrote:
Its warning not error...
Create a directory and
Thanks,
But the issue was that there was no directory and hence it was not showing
anything. Adding a directory cleared the warning.
I appreciate your help.
Regards
ashish
On Tue, Jul 23, 2013 at 10:08 AM, Mohammad Tariq donta...@gmail.com wrote:
Hello Ashish,
Change the permissions of
After starting i would suggest always check whether your NameNode and job
tracker UI are working or not and check the number of live nodes in both of
the UI..
Regards,
Som Shekhar Sharma
+91-8197243810
On Tue, Jul 23, 2013 at 10:41 PM, Ashish Umrani ashish.umr...@gmail.comwrote:
Thanks,
But
Try..
*hadoop fs -ls /*
**
Thanks
On Tue, Jul 23, 2013 at 10:27 PM, Ashish Umrani ashish.umr...@gmail.comwrote:
Thanks Jitendra, Bejoy and Yexi,
I got past that. And now the ls command says it can not access the
directory. I am sure this is a permissions issue. I am just wondering
Jitendra, Som,
Thanks. Issue was in not having any file there. Its working fine now.
I am able to do -ls and could also do -mkdir and -put.
Now is time to run the jar and apparently I am getting
no main manifest attribute, in wc.jar
But I believe its because of maven pom file does not have
hadoop jar wc.jar fully qualified driver name inputdata outputdestination
Regards,
Som Shekhar Sharma
+91-8197243810
On Tue, Jul 23, 2013 at 10:58 PM, Ashish Umrani ashish.umr...@gmail.comwrote:
Jitendra, Som,
Thanks. Issue was in not having any file there. Its working fine now.
I am
Hi All,
Do any of you have or can refer me to some sample Java code that get the
tree structure of a HDFS directory, similar to the file system?
For example: I have a HDFS dir, called /data, inside data, there is
/data/valid and /data/invalid, and so on, so I would need to be able to get
Thanks Shekhar,
The problem was not in my building of the jar. It was in fact in execution
I was running command
*hadoop -jar* jar filename qualified class name input output
The problem was with -jar. It should be
*hadoop jar* jar filename qualified class name input output
Thanks for help
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
and
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)
Basically you can write your own function,
Hi,
A common use case one want an ordered structure for, is for saving counters.
Naturally, I wanted to save the counters in a Mapfile:
for (long ix = 0; ix MAXVALUE; ix++) {
mapfile.append(new Text(counter key of val + ix), new
LongWritable(ix));
}
This however looks a bit
Hi,
If you intend to use those counters in further functions then I think Hadoop
will take care this by itself, you can explore combiner for the same. In
Sequence file mapfile and index files has its own functionality and as per my
understanding index file expect offset to move over map file
Uhm...
You want to save the counters as in counts per job run or something? (Remember
HDFS == WORM)
Then you could do a sequence file and then use something like HBase to manage
the index.
(Every time you add a set of counters, you have a new file and a new index.)
Heck you could use HBase
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)
On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham pha...@yahoo-inc.com wrote:
Hi All,
Do any of you have
It could either mean that all those task-attempts are crashing before the
process itself is getting spawned (check TT logs) or those logs are getting
deleted after the fact. Suspect the earlier.
Thanks,
+Vinod
On Jul 23, 2013, at 9:33 AM, Ajay Srivastava wrote:
Hi,
I see that most of the
40 matches
Mail list logo
