Re: Organising Hive Scripts

[Elliot West]

Hi Charles,

You can also split out column level logic using Hive macros. These also
allow re-use of said logic:


hive> create temporary macro MYSIGMOID(x DOUBLE)
> 2.0 / (1.0 + exp(-x));
OK

hive> select MYSIGMOID(1.0) from dual;
OK

1.4621171572600098


Cheers - Elliot.

On 11 September 2015 at 20:51, Charles Mean  wrote:

> Great Dmitry,
>
> It will certainly help me a lot.
> I will give it a try, thank you very much for your help.
>
> On Fri, Sep 11, 2015 at 4:34 PM, Dmitry Tolpeko 
> wrote:
>
>> Charles,
>>
>> Not sure what you can do in Hive CLI right now, but consider a new Hive
>> HPL/SQL component that will be included to new Hive versions and that
>> currently you can compile and run separately, see
>> https://github.com/apache/hive/tree/master/hplsql or www.hplsql.org
>>
>> It supports include files, user defined procedures and functions,
>> exception based exception handling, flow of control statements and so on.
>> It is new, under active development so any feedback is welcome.
>>
>> Dmitry
>>
>> On Fri, Sep 11, 2015 at 8:32 PM, Charles Mean 
>> wrote:
>>
>>> Hello,
>>>
>>> I am working with a huge hive script that I would like to improve its
>>> organisation to a better maintenance in the future.
>>> Looking into this issue, I did not found any kind of include or
>>> something to split my script into some smaller parts.
>>> So, Is there some sort of pattern that is implemented to improve script
>>> structure ?
>>>
>>
>>
>

Re: Organising Hive Scripts

[Daniel Lopes]

Thanks Erwan,

I needed this too, and works fine for me.


*Daniel Lopes, B.Eng*
Data Scientist - BankFacil
CREA/SP 5069410560

Mob +55 (18) 99764-2733 
Ph +55 (11) 3522-8009
http://about.me/dannyeuu

Av. Nova Independência, 956, São Paulo, SP
Bairro Brooklin Paulista
CEP 04570-001
https://www.bankfacil.com.br


On Sat, Sep 12, 2015 at 12:11 PM, Erwan MAS  wrote:

> Hi,
>
> HIVE have a the source keyword , so you can split you big huge script in
> multiple part .
>
> Sou you can have a script_part.hql and inside you call all you small part .
>
> source  script_a.hql ;
> source  script_b.hql ;
> source  script_c.hql ;
>
> --
> Erwan MAS
>

Re: Organising Hive Scripts

[Erwan MAS]

Hi,

HIVE have a the source keyword , so you can split you big huge script in 
multiple part .

Sou you can have a script_part.hql and inside you call all you small part .

source  script_a.hql ;
source  script_b.hql ;
source  script_c.hql ;

--
Erwan MAS

Re: Organising Hive Scripts

[Charles Mean]

Great Dmitry,

It will certainly help me a lot.
I will give it a try, thank you very much for your help.

On Fri, Sep 11, 2015 at 4:34 PM, Dmitry Tolpeko  wrote:

> Charles,
>
> Not sure what you can do in Hive CLI right now, but consider a new Hive
> HPL/SQL component that will be included to new Hive versions and that
> currently you can compile and run separately, see
> https://github.com/apache/hive/tree/master/hplsql or www.hplsql.org
>
> It supports include files, user defined procedures and functions,
> exception based exception handling, flow of control statements and so on.
> It is new, under active development so any feedback is welcome.
>
> Dmitry
>
> On Fri, Sep 11, 2015 at 8:32 PM, Charles Mean 
> wrote:
>
>> Hello,
>>
>> I am working with a huge hive script that I would like to improve its
>> organisation to a better maintenance in the future.
>> Looking into this issue, I did not found any kind of include or something
>> to split my script into some smaller parts.
>> So, Is there some sort of pattern that is implemented to improve script
>> structure ?
>>
>
>

Re: Organising Hive Scripts

[Dmitry Tolpeko]

Charles,

Not sure what you can do in Hive CLI right now, but consider a new Hive
HPL/SQL component that will be included to new Hive versions and that
currently you can compile and run separately, see
https://github.com/apache/hive/tree/master/hplsql or www.hplsql.org

It supports include files, user defined procedures and functions, exception
based exception handling, flow of control statements and so on. It is new,
under active development so any feedback is welcome.

Dmitry

On Fri, Sep 11, 2015 at 8:32 PM, Charles Mean 
wrote:

> Hello,
>
> I am working with a huge hive script that I would like to improve its
> organisation to a better maintenance in the future.
> Looking into this issue, I did not found any kind of include or something
> to split my script into some smaller parts.
> So, Is there some sort of pattern that is implemented to improve script
> structure ?
>

Organising Hive Scripts

[Charles Mean]

Hello,

I am working with a huge hive script that I would like to improve its
organisation to a better maintenance in the future.
Looking into this issue, I did not found any kind of include or something
to split my script into some smaller parts.
So, Is there some sort of pattern that is implemented to improve script
structure ?