Hive JVM initialization - slf4j

[Rajat Jain]

Hi folks,

It seems to me the initialization of Hive JVM takes ~10s and most of this
time goes into initializing the slf4j library. Is there a known reason why
it takes so much time. Can this be reduced somehow?

Thanks,
Rajat

Question regarding HIVE-6888

[Rajat Jain]

This is regarding the fix that was incorporated in HIVE-6888
 (commit

).

The fix was issued because the MapWork objects were being leaked due to
having multiple AMs. However, there are cases when this fix clears gWorkMap
prematurely and it is populated (and cleared) again. For example, when
HiveInputFormat.getSplits() is called from HiveSplitGenerator.initialize().

Here, gWorkMap is cleared when getSplits() is called, and populated again
when splitGrouper.generateGroupedSplits() is called. gWorkMap is finally
cleared in the 'finally' block of HiveSplitGenerator.initialize().

In our codebase, we do some modification to MapWork in the getSplits()
call, and those changes are negated when clearMapWork() is called inside
HiveInputFormat.getSplits(). I'm wondering if this call is really required?

Re: Running Hive Unit Tests from IntelliJ and Datanucleus

[Rajat Jain]

Tried that too. Still didn't work for me. Anyway, I'm past that for now.

Thanks for the help,
Rajat

On Wed, Jun 17, 2015 at 7:44 PM, Ruoxi Sun  wrote:

> Hi Rajat, I used an alternative instead of Datanucleus plugin in IntelliJ.
> Try create a run configuration as the following pictured shows. And make
> sure you have datanucleus in your module's dependencies.
>
> Hope it can help.
>
>
> *孙若曦*
>
> 2015-06-18 3:32 GMT+08:00 Rajat Jain :
>
>> Hi,
>>
>> I want to run Hive unit tests from IntelliJ but am unable to do so due to
>> datanucleus issues. I tried a lot of options but always seem to be getting
>> the same error.
>>
>> 1. Datanucleus plugin:
>>
>> I installed Datanucleus plugin on IntelliJ, enabled the enhancer but got
>> error of the type:
>>
>> Caused by: org.datanucleus.exceptions.ClassNotPersistableException: The
>> class "org.apache.hadoop.hive.metastore.model.MVersionTable" is not
>> persistable. This means that it either hasnt been enhanced, or that the
>> enhanced version of the file is not in the CLASSPATH (or is hidden by an
>> unenhanced version), or the Meta-Data/annotations for the class are not
>> found.
>> at
>> org.datanucleus.ExecutionContextImpl.assertClassPersistable(ExecutionContextImpl.java:5698)
>> at
>> org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2123)
>> at
>> org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
>> at
>> org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
>> at
>> org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
>> at
>> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
>> ... 66 more
>>
>> When I checked the Datanucleus settings, I noticed that only MDatabase
>> class was registered there and no other classes (like MVersionTable) were
>> registered. Is that the issue? If not, then any way this can be resolved?
>>
>> 2. I tried pushing up the hive-metastore dependency over Module Source as
>> suggested here
>> <http://qnalist.com/questions/5105293/running-tests-in-intellij>. Didn't
>> work either. I tried this option while both enabling and disabling
>> datanucleus plugin.
>>
>> Let me know if someone has any ideas. I have attached sample screenshots
>> for reference.
>>
>> Thanks,
>> Rajat
>>
>>
>>
>