Hi Manisha
In mapreduce if you want to change the name of output file you may need to
write your own OutputFormat.
Renaming files in hdfs is straight forward
hadoop fs -mv oldFileName newFileName
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: Manisha Gayathri <mani...@wso2.com>
Date: Fri, 27 Jul 2012 11:56:57
To: <user@hive.apache.org>
Reply-To: user@hive.apache.org
Subject: Rename an output file in hive {was: Re: Possibility of defining the
Output directory programmatically}
Hi all,
Thanks alot for your help.
With your advice, could successfully pass the needed parameters and
generate the output file as required.
In my use case, the output file needed to be a .gz file. When I run my
query with the following commands, the file that gets generated is named as
*000000_0.gz*
set mapred.output.compress=true;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
Is there anyway that I can rename this file or generate the file in a name
of my choice from hive it self?
Thanks again.
Manisha
On Mon, Jul 23, 2012 at 3:35 PM, Nitin Pawar <nitinpawar...@gmail.com>wrote:
> manisha,
>
> set is variable key value pair, it does not execute any functions
>
> if you want to use it like that then I would recomment write a shell
> script which will generate all required things to a file and then
> execute that file with hive -f option
>
> On Mon, Jul 23, 2012 at 3:27 PM, Manisha Gayathri <mani...@wso2.com>
> wrote:
> > Thanks Vinod.
> >
> > I tried concatenating variables. But that is also not possible as I see.
> > set pqr = concat(foo,bar);
> > set file_name= home/user/Desktop
> >
> > Then the file_name I am getting is NOT home/user/Desktop/foo_bar
> > But what I am getting is, /home/user/Desktop/concat(foo,bar)
> >
> >
> >
> > On Mon, Jul 23, 2012 at 3:04 PM, Vinod Singh <vi...@vinodsingh.com>
> wrote:
> >>
> >> We generate variables dynamically and then create a final script file by
> >> concatenating variables (SET commands) and Hive queries. Then final
> script
> >> is executed. Probably you can adopt something similar approach.
> >>
> >> Thanks,
> >> Vinod
> >>
> >> 2012/7/23 Manisha Gayathri <mani...@wso2.com>
> >>>
> >>> Thanks again Vinod.
> >>>
> >>> Will try to find a way to pass the directory URLs from outside then.
> >>> And it would be grateful if you can direct me to any guide or
> >>> documentation that describes how to pass values from outside of Hive.
> >>>
> >>> Thanks
> >>> Manisha
> >>>
> >>>
> >>> On Mon, Jul 23, 2012 at 2:08 PM, Vinod Singh <vi...@vinodsingh.com>
> >>> wrote:
> >>>>
> >>>> SET commands are handled differently and UDFs can't be invoked there.
> >>>> IMO you need to pass the directory location value from outside of
> Hive. That
> >>>> is how we do.
> >>>>
> >>>> Thanks,
> >>>> Vinod
> >>>>
> >>>> 2012/7/23 Manisha Gayathri <mani...@wso2.com>
> >>>>>
> >>>>> Hi Vinod,
> >>>>>
> >>>>> Thanks for the prompt reply.
> >>>>> Understood your point and sorry for not providing the complete code
> >>>>> segment earlier.
> >>>>>
> >>>>> I have the getFilePath function which should return a URL like this.
> >>>>>
> >>>>> home/user/Desktop/logDir/logs/log_0_testServer_2012_07_22
> >>>>>
> >>>>> The defined function works perfectly if I put getFilePath(
> >>>>> "0","testServer" ) into the select statement.
> >>>>>
> >>>>> But I want to get that particular URL as the local directory name.
> (The
> >>>>> requirement is such that this should not be hard-coded in the hive
> query.
> >>>>> Rather should be generated in the custom UDF. )
> >>>>>
> >>>>> So can I do something like I v shown below?
> >>>>>
> >>>>> set file_name= getFilePath( "0","testServer" ); //Define a
> >>>>> parameter.
> >>>>> .................
> >>>>> ..............
> >>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
> >>>>> //Assign the above parameter as the file URL
> >>>>>
> >>>>> I tried this way. But the directory name is returned as
> >>>>>
> >>>>> file:/getFilePath( "0" , "testServer" )
> >>>>>
> >>>>> Does that mean I cannot use UDF to define the local directory name?
> Or
> >>>>> am I doing anything wrong in here?
> >>>>>
> >>>>> Thanks
> >>>>> Manisha
> >>>>>
> >>>>> On Mon, Jul 23, 2012 at 1:47 PM, Vinod Singh <vi...@vinodsingh.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> The output path in this query is already parameterized-
> >>>>>>
> >>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
> >>>>>>
> >>>>>> Though UDF is not going to be invoked here.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Vinod
> >>>>>>
> >>>>>>
> >>>>>> 2012/7/23 Manisha Gayathri <mani...@wso2.com>
> >>>>>>>
> >>>>>>> Hi
> >>>>>>>
> >>>>>>> Is there any possibility of defining the output directory of a hive
> >>>>>>> query using a Hive UDF?
> >>>>>>>
> >>>>>>> In my UDF, I am passing 2 parameters (as follows) and this
> generates
> >>>>>>> a file-system URL
> >>>>>>> getFilePath( "0","testServer" );
> >>>>>>>
> >>>>>>> Can I use the above getFilePath( "0","testServer" ) value, as the
> >>>>>>> Local Directory location as follows?
> >>>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
> >>>>>>>
> >>>>>>> If above is not the correct way, could you please show me how to do
> >>>>>>> it?
> >>>>>>> Your help is much appreciated!
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> ~Regards
> >>>>>>> Manisha Eleperuma
> >>>>>>> Software Engineer
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> ~Regards
> >>>>> Manisha Eleperuma
> >>>>> Software Engineer
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> ~Regards
> >>> Manisha Eleperuma
> >>> Software Engineer
> >>>
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > ~Regards
> > Manisha Eleperuma
> > Software Engineer
> > WSO2, Inc.: http://wso2.com
> > lean.enterprise.middleware
> >
> > mobile: +94 71 8279777
> >
> >
>
>
>
> --
> Nitin Pawar
>
--
~Regards
*Manisha Eleperuma*
Software Engineer
WSO2, Inc.: http://wso2.com
lean.enterprise.middleware
*
*
*
*
