[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations

2016-05-05 Thread Vikram Dixit K (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273320#comment-15273320
 ] 

Vikram Dixit K commented on HIVE-13656:
---

After talking with Sergey, it looks like this is what we have to do for max 
direct memory to be used properly. +1 for the patch.

> need to set direct memory limit higher in LlapServiceDriver for certain edge 
> case configurations
> 
>
> Key: HIVE-13656
> URL: https://issues.apache.org/jira/browse/HIVE-13656
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13656.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations

2016-05-05 Thread Vikram Dixit K (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273319#comment-15273319
 ] 

Vikram Dixit K commented on HIVE-13656:
---

Don't you want this

{code}
long maxDirect = (xmx > 0 && cache > 0 && xmx < cache * 1.25) ? (long)(cache * 
1.25) : -1;
{code}

to be:

{code}
long maxDirect = (xmx > 0 && cache > 0 && xmx < cache * 1.25) ? (long)(cache * 
1.25) : xmx;
{code}

> need to set direct memory limit higher in LlapServiceDriver for certain edge 
> case configurations
> 
>
> Key: HIVE-13656
> URL: https://issues.apache.org/jira/browse/HIVE-13656
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13656.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations

2016-05-05 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273312#comment-15273312
 ] 

Sergey Shelukhin commented on HIVE-13656:
-

[~hagleitn] [~vikram.dixit] can you review instead?

> need to set direct memory limit higher in LlapServiceDriver for certain edge 
> case configurations
> 
>
> Key: HIVE-13656
> URL: https://issues.apache.org/jira/browse/HIVE-13656
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13656.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations

2016-05-04 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271787#comment-15271787
 ] 

Sergey Shelukhin commented on HIVE-13656:
-

No, it sets the value as is in bytes

> need to set direct memory limit higher in LlapServiceDriver for certain edge 
> case configurations
> 
>
> Key: HIVE-13656
> URL: https://issues.apache.org/jira/browse/HIVE-13656
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13656.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations

2016-05-03 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268926#comment-15268926
 ] 

Siddharth Seth commented on HIVE-13656:
---

Does this need to factor in xmx being set in MB in the java part of service 
driver, and without in the python bit. (maybe that's already being done and I 
missed it).
ie. daemon_args = " -XX:MaxDirectMemorySize=%s %s" % (max_direct_memory, 
or daemon_args = " -XX:MaxDirectMemorySize=%sm %s" % (max_direct_memory, 



> need to set direct memory limit higher in LlapServiceDriver for certain edge 
> case configurations
> 
>
> Key: HIVE-13656
> URL: https://issues.apache.org/jira/browse/HIVE-13656
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13656.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13656) need to set direct memory limit higher in LlapServiceDriver for certain edge case configurations

2016-05-01 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266044#comment-15266044
 ] 

Hive QA commented on HIVE-13656:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801543/HIVE-13656.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 9957 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/146/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/146/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-146/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing