Re: MRv1 vs. MRv2

[Gunnar Tapper]

Thanks.

On Tue, Feb 2, 2016 at 2:36 PM, Hans Zeller <hans.zel...@esgyn.com> wrote:

> No, unless a user explicitly puts it into a UDF, none of the built-in UDFs
> and SPJs use MR.
>
> Hans
>
> On Tue, Feb 2, 2016 at 1:21 PM, Gunnar Tapper <tapper.gun...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I think it's a configuration choice. For example:
>> http://www.cloudera.com/documentation/manager/5-0-x/Cloudera-Manager-Managing-Clusters/cm5mc_mapreduce_service.html
>>
>> Are there UDFs, SPJs, and similar functions that use MapReduce, too?
>>
>> Thanks,
>>
>> Gunnar
>>
>> On Tue, Feb 2, 2016 at 2:19 PM, Qifan Chen <qifan.c...@esgyn.com> wrote:
>>
>>>
>>> Hi Hans,
>>>
>>> I think Hive uses MapReduce to sort the data during table population,
>>> after disabling YARN. This is observed on a workstation.
>>>
>>> Thanks -Qifan
>>>
>>> On Tue, Feb 2, 2016 at 3:12 PM, Hans Zeller <hans.zel...@esgyn.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> That decision would be made by Hive, not Trafodion. For people who use
>>>> install_local_hadoop, we recently changed that setup to use local
>>>> MapReduce, not YARN, see
>>>> https://issues.apache.org/jira/browse/TRAFODION-1781.
>>>>
>>>> Hans
>>>>
>>>> On Tue, Feb 2, 2016 at 12:58 PM, Gunnar Tapper <tapper.gun...@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Suresh:
>>>>>
>>>>> Thanks for the information.
>>>>>
>>>>> Given from what you write, it seems that YARN with MRv2 is required
>>>>> for full functionality.
>>>>>
>>>>> MRv1 is a separate install in current distributions, which is why I am
>>>>> asking about it. How does Trafodion decide to run the MapReduce job as 
>>>>> MRv1
>>>>> vs. MRv2 if both are installed?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Gunnar
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 2, 2016 at 1:50 PM, Suresh Subbiah <
>>>>> suresh.subbia...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I don't think Trafodion requires YARN for most activity.
>>>>>>
>>>>>> For Hive table access, Trafodion uses Hive metadata access Java API
>>>>>> and libhdfs to actually scan the data file. Therefore YARN is not needed
>>>>>> for Hive access.
>>>>>> YARN is not needed for native Hbase or Trafodion table access too.
>>>>>> YARN is needed for backup/restore, since the HBase exportSnapshot
>>>>>> class Trafodion calls, used MapReduce to copy large snapshot files 
>>>>>> to/from
>>>>>> the backup location.
>>>>>> YARN is also needed for developer regressions as some vanilla Hive
>>>>>> commands are executed during the regression run.
>>>>>> For the last 2 lines I think both MRv1 and MRv2 is supported.
>>>>>>
>>>>>> Thanks
>>>>>> Suresh
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 2, 2016 at 2:36 PM, Gunnar Tapper <
>>>>>> tapper.gun...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Does Trafodion require YARN with MRv2 or is MRv1 supported, too?
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Gunnar
>>>>>>> *If you think you can you can, if you think you can't you're right.*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>>
>>>>> Gunnar
>>>>> *If you think you can you can, if you think you can't you're right.*
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards, --Qifan
>>>
>>>
>>
>>
>> --
>> Thanks,
>>
>> Gunnar
>> *If you think you can you can, if you think you can't you're right.*
>>
>
>


-- 
Thanks,

Gunnar
*If you think you can you can, if you think you can't you're right.*

MRv1 vs. MRv2

[Gunnar Tapper]

Hi,

Does Trafodion require YARN with MRv2 or is MRv1 supported, too?

-- 
Thanks,

Gunnar
*If you think you can you can, if you think you can't you're right.*

Re: MRv1 vs. MRv2

[Suresh Subbiah]

Hi,

I don't think Trafodion requires YARN for most activity.

For Hive table access, Trafodion uses Hive metadata access Java API and
libhdfs to actually scan the data file. Therefore YARN is not needed for
Hive access.
YARN is not needed for native Hbase or Trafodion table access too.
YARN is needed for backup/restore, since the HBase exportSnapshot class
Trafodion calls, used MapReduce to copy large snapshot files to/from the
backup location.
YARN is also needed for developer regressions as some vanilla Hive commands
are executed during the regression run.
For the last 2 lines I think both MRv1 and MRv2 is supported.

Thanks
Suresh


On Tue, Feb 2, 2016 at 2:36 PM, Gunnar Tapper 
wrote:

> Hi,
>
> Does Trafodion require YARN with MRv2 or is MRv1 supported, too?
>
> --
> Thanks,
>
> Gunnar
> *If you think you can you can, if you think you can't you're right.*
>

Re: MRv1 vs. MRv2

[Hans Zeller]

Hi,

That decision would be made by Hive, not Trafodion. For people who use
install_local_hadoop, we recently changed that setup to use local
MapReduce, not YARN, see
https://issues.apache.org/jira/browse/TRAFODION-1781.

Hans

On Tue, Feb 2, 2016 at 12:58 PM, Gunnar Tapper <tapper.gun...@gmail.com>
wrote:

> Hi Suresh:
>
> Thanks for the information.
>
> Given from what you write, it seems that YARN with MRv2 is required for
> full functionality.
>
> MRv1 is a separate install in current distributions, which is why I am
> asking about it. How does Trafodion decide to run the MapReduce job as MRv1
> vs. MRv2 if both are installed?
>
> Thanks,
>
> Gunnar
>
>
>
> On Tue, Feb 2, 2016 at 1:50 PM, Suresh Subbiah <suresh.subbia...@gmail.com
> > wrote:
>
>> Hi,
>>
>> I don't think Trafodion requires YARN for most activity.
>>
>> For Hive table access, Trafodion uses Hive metadata access Java API and
>> libhdfs to actually scan the data file. Therefore YARN is not needed for
>> Hive access.
>> YARN is not needed for native Hbase or Trafodion table access too.
>> YARN is needed for backup/restore, since the HBase exportSnapshot class
>> Trafodion calls, used MapReduce to copy large snapshot files to/from the
>> backup location.
>> YARN is also needed for developer regressions as some vanilla Hive
>> commands are executed during the regression run.
>> For the last 2 lines I think both MRv1 and MRv2 is supported.
>>
>> Thanks
>> Suresh
>>
>>
>> On Tue, Feb 2, 2016 at 2:36 PM, Gunnar Tapper <tapper.gun...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Does Trafodion require YARN with MRv2 or is MRv1 supported, too?
>>>
>>> --
>>> Thanks,
>>>
>>> Gunnar
>>> *If you think you can you can, if you think you can't you're right.*
>>>
>>
>>
>
>
> --
> Thanks,
>
> Gunnar
> *If you think you can you can, if you think you can't you're right.*
>

Re: MRv1 vs. MRv2

[Amanda Moran]

I have done many installs without Yarn or MapReduce installed at all.
Trafodion runs fine :)

On Tue, Feb 2, 2016 at 2:39 PM, Hans Zeller <hans.zel...@esgyn.com> wrote:

> No, it is not required in the build environment.
>
> Hans
>
> On Tue, Feb 2, 2016 at 1:54 PM, Gunnar Tapper <tapper.gun...@gmail.com>
> wrote:
>
>> Does this mean that MRv1 is now required in the build environment?
>>
>> On Tue, Feb 2, 2016 at 2:12 PM, Hans Zeller <hans.zel...@esgyn.com>
>> wrote:
>>
>>> Hi,
>>>
>>> That decision would be made by Hive, not Trafodion. For people who use
>>> install_local_hadoop, we recently changed that setup to use local
>>> MapReduce, not YARN, see
>>> https://issues.apache.org/jira/browse/TRAFODION-1781.
>>>
>>> Hans
>>>
>>> On Tue, Feb 2, 2016 at 12:58 PM, Gunnar Tapper <tapper.gun...@gmail.com>
>>> wrote:
>>>
>>>> Hi Suresh:
>>>>
>>>> Thanks for the information.
>>>>
>>>> Given from what you write, it seems that YARN with MRv2 is required for
>>>> full functionality.
>>>>
>>>> MRv1 is a separate install in current distributions, which is why I am
>>>> asking about it. How does Trafodion decide to run the MapReduce job as MRv1
>>>> vs. MRv2 if both are installed?
>>>>
>>>> Thanks,
>>>>
>>>> Gunnar
>>>>
>>>>
>>>>
>>>> On Tue, Feb 2, 2016 at 1:50 PM, Suresh Subbiah <
>>>> suresh.subbia...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I don't think Trafodion requires YARN for most activity.
>>>>>
>>>>> For Hive table access, Trafodion uses Hive metadata access Java API
>>>>> and libhdfs to actually scan the data file. Therefore YARN is not needed
>>>>> for Hive access.
>>>>> YARN is not needed for native Hbase or Trafodion table access too.
>>>>> YARN is needed for backup/restore, since the HBase exportSnapshot
>>>>> class Trafodion calls, used MapReduce to copy large snapshot files to/from
>>>>> the backup location.
>>>>> YARN is also needed for developer regressions as some vanilla Hive
>>>>> commands are executed during the regression run.
>>>>> For the last 2 lines I think both MRv1 and MRv2 is supported.
>>>>>
>>>>> Thanks
>>>>> Suresh
>>>>>
>>>>>
>>>>> On Tue, Feb 2, 2016 at 2:36 PM, Gunnar Tapper <tapper.gun...@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Does Trafodion require YARN with MRv2 or is MRv1 supported, too?
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>>
>>>>>> Gunnar
>>>>>> *If you think you can you can, if you think you can't you're right.*
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>>
>>>> Gunnar
>>>> *If you think you can you can, if you think you can't you're right.*
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>>
>> Gunnar
>> *If you think you can you can, if you think you can't you're right.*
>>
>
>


-- 
Thanks,

Amanda Moran