Failed: TEZ-3813 PreCommit Build #2599

2017-08-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3813
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2599/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 339.63 KB...]
[INFO] Total time: 56:38 min
[INFO] Finished at: 2017-08-04T02:45:40Z
[INFO] Final Memory: 84M/1401M
[INFO] 




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12880300/TEZ-3813.001.patch
  against master revision 614937c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/2599//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/2599//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2599//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
5155c76e148351702861a63a604a4f0169e8d611 logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs

2017-08-03 Thread TezQA (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113843#comment-16113843
 ] 

TezQA commented on TEZ-3813:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12880300/TEZ-3813.001.patch
  against master revision 614937c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/2599//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/2599//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-library.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2599//console

This message is automatically generated.

> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a 
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is 
> used).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Success: TEZ-3803 PreCommit Build #2598

2017-08-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3803
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2598/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 339.47 KB...]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 53:28 min
[INFO] Finished at: 2017-08-04T02:38:45Z
[INFO] Final Memory: 94M/1394M
[INFO] 




{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12880310/TEZ-3803.004.patch
  against master revision 614937c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/2598//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2598//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
d4ec1efff89a94b12f66409282810ac987a611e6 logged out


==
==
Finished build.
==
==


Archiving artifacts
[Fast Archiver] Compressed 3.51 MB of artifacts by 34.7% relative to #2597
[description-setter] Description set: TEZ-3803
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (TEZ-3803) Tasks can get killed due to insufficient progress while waiting for shuffle inputs to complete

2017-08-03 Thread Kuhu Shukla (JIRA)

 [ 
https://issues.apache.org/jira/browse/TEZ-3803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kuhu Shukla updated TEZ-3803:
-
Attachment: TEZ-3803.004.patch

Revised patch that waits with a set timeout (un-configurable) for simplicity. 
We could go to a different value or move this as a variable to ShuffleUtils if 
this approach seems ok. Also changed the test run time. This patch modifies the 
if block to a while block essentially. Will wait for precommit before further 
review requests.

> Tasks can get killed due to insufficient progress while waiting for shuffle 
> inputs to complete
> --
>
> Key: TEZ-3803
> URL: https://issues.apache.org/jira/browse/TEZ-3803
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Critical
> Attachments: TEZ-3803.001.patch, TEZ-3803.002.patch, 
> TEZ-3803.003.patch, TEZ-3803.004.patch
>
>
> In a scenario where a downstream task has no slow start and gets started 
> before all its shuffle inputs are done, the task can timeout as the wait does 
> not notify progress( set the "progress is being made bit") like it does in 
> MapReduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs

2017-08-03 Thread Muhammad Samir Khan (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113454#comment-16113454
 ] 

Muhammad Samir Khan commented on TEZ-3813:
--

Tested with filterLinesByWord and compared output before and after.

> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a 
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is 
> used).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs

2017-08-03 Thread Muhammad Samir Khan (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113451#comment-16113451
 ] 

Muhammad Samir Khan commented on TEZ-3813:
--

*JOL Dump:*
+Before:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]

Instantiated the sample instance via public 
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)

org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object 
internals:
 OFFSET  SIZE 
TYPE DESCRIPTION   VALUE
  0 4   
   (object header)   01 00 00 00 (0001  
 ) (1)
  4 4   
   (object header)   00 00 00 00 (  
 ) (0)
  8 4   
   (object header)   7a 12 01 f8 (0010 00010010 
0001 1000) (-134147462)
 12 4  
int FetchedInput.id   0
 16 8 
long FetchedInput.actualSize   0
 24 8 
long FetchedInput.compressedSize   0
 32 4 
org.apache.tez.runtime.library.common.InputAttemptIdentifier 
FetchedInput.inputAttemptIdentifier   null
 36 4  
org.apache.tez.runtime.library.common.shuffle.FetchedInput.Type 
FetchedInput.type (object)
 40 4   
org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback 
FetchedInput.callback null
 44 4 
org.apache.tez.runtime.library.common.shuffle.FetchedInput.State 
FetchedInput.state(object)
 48 4
org.apache.hadoop.io.BoundedByteArrayOutputStream MemoryFetchedInput.byteStream 
(object)
 52 4   
   (loss due to the next object alignment)
Instance size: 56 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
{code}

Footprint:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]

Instantiated the sample instance via public 
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)

org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput@215be6bbd 
footprint:
 COUNT   AVG   SUM   DESCRIPTION
 11616   [B
 23264   [C
 22448   java.lang.String
 13232   
org.apache.hadoop.io.BoundedByteArrayOutputStream
 12424   
org.apache.tez.runtime.library.common.shuffle.FetchedInput$State
 12424   
org.apache.tez.runtime.library.common.shuffle.FetchedInput$Type
 15656   
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput
 9 264   (total)
{code}

+After:+
Internals:
{code}
# Running 64-bit HotSpot VM.
# Using compressed oop with 3-bit shift.
# Using compressed klass with 3-bit shift.
# Objects are 8 bytes aligned.
# Field sizes by type: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]
# Array element sizes: 4, 1, 1, 2, 2, 4, 4, 8, 8 [bytes]

Instantiated the sample instance via public 
org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput(long,long,org.apache.tez.runtime.library.common.InputAttemptIdentifier,org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)

org.apache.tez.runtime.library.common.shuffle.MemoryFetchedInput object 
internals:
 OFFSET  SIZE 
TYPE DESCRIPTION   VALUE
  0 4   
   (object header)   01 00 00 00 (0001  
 ) (1)
  4 4 

[jira] [Updated] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs

2017-08-03 Thread Muhammad Samir Khan (JIRA)

 [ 
https://issues.apache.org/jira/browse/TEZ-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Muhammad Samir Khan updated TEZ-3813:
-
Attachment: TEZ-3813.001.patch

Replace using BoundedByteArrayOutputStream with a byte array in 
MemoryFetchedInput.

> Reduce Object size of MemoryFetchedInput for large jobs
> ---
>
> Key: TEZ-3813
> URL: https://issues.apache.org/jira/browse/TEZ-3813
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3813.001.patch
>
>
> Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a 
> BoundedByteArrayOutputStream that is not used (only the underlying byte[] is 
> used).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (TEZ-3813) Reduce Object size of MemoryFetchedInput for large jobs

2017-08-03 Thread Muhammad Samir Khan (JIRA)
Muhammad Samir Khan created TEZ-3813:


 Summary: Reduce Object size of MemoryFetchedInput for large jobs
 Key: TEZ-3813
 URL: https://issues.apache.org/jira/browse/TEZ-3813
 Project: Apache Tez
  Issue Type: Bug
Reporter: Muhammad Samir Khan
Assignee: Muhammad Samir Khan


Same as TEZ-3752 for the unordered case. MemoryFetchedInput has a 
BoundedByteArrayOutputStream that is not used (only the underlying byte[] is 
used).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized

2017-08-03 Thread Muhammad Samir Khan (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113369#comment-16113369
 ] 

Muhammad Samir Khan commented on TEZ-3809:
--

Request for comments [~rajesh.balamohan]/[~sseth]

> The buffer size allocated for InMemoryMapOutput can be optimized
> 
>
> Key: TEZ-3809
> URL: https://issues.apache.org/jira/browse/TEZ-3809
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3809.001.patch
>
>
> Related jiras: TEZ-3752 and TEZ-3732.
> -When shuffling input to memory, the decompressed length is used to create 
> the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 
> bytes less (the IFile header). These 4 bytes can optimized and, in an extreme 
> case of 10,000,000 fetches, can save ~38 MB (TEZ-3732).
> -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers 
> to allocate the new InMemoryMapOutput. However, each input has two 
> EOF_MARKERs while only two are needed at the end.
> -InMemoryWriter wraps the output BoundedByteArrayOutputStream in 
> IFileOutputStream which will write checksum at close. This creates an 
> inconsistency between the primary input buffers which don't have checksum and 
> the merged buffers which do. IFileOutputStream wrap can be removed to save 4 
> bytes per merged buffers.
> -InMemoryWriter does not account for two EOF_MARKERs written at close() in 
> its accounting so that the getRawLength() method is off by two bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (TEZ-3809) The buffer size allocated for InMemoryMapOutput can be optimized

2017-08-03 Thread Muhammad Samir Khan (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113366#comment-16113366
 ] 

Muhammad Samir Khan commented on TEZ-3809:
--

Also tested with memory-to-memory merger to check if the output is the same. 
For the unordered case, used filterLinesByWord and compared output before and 
after.

> The buffer size allocated for InMemoryMapOutput can be optimized
> 
>
> Key: TEZ-3809
> URL: https://issues.apache.org/jira/browse/TEZ-3809
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Muhammad Samir Khan
>Assignee: Muhammad Samir Khan
> Attachments: TEZ-3809.001.patch
>
>
> Related jiras: TEZ-3752 and TEZ-3732.
> -When shuffling input to memory, the decompressed length is used to create 
> the InMemoryMapOutput object. However, IFile.Reader's readToMemory reads 4 
> bytes less (the IFile header). These 4 bytes can optimized and, in an extreme 
> case of 10,000,000 fetches, can save ~38 MB (TEZ-3732).
> -Memory-to-memory merge sums up the sizes of input InMemoryMapOutput buffers 
> to allocate the new InMemoryMapOutput. However, each input has two 
> EOF_MARKERs while only two are needed at the end.
> -InMemoryWriter wraps the output BoundedByteArrayOutputStream in 
> IFileOutputStream which will write checksum at close. This creates an 
> inconsistency between the primary input buffers which don't have checksum and 
> the merged buffers which do. IFileOutputStream wrap can be removed to save 4 
> bytes per merged buffers.
> -InMemoryWriter does not account for two EOF_MARKERs written at close() in 
> its accounting so that the getRawLength() method is off by two bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (TEZ-3812) race condition in ssl shuffle

2017-08-03 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113350#comment-16113350
 ] 

Sergey Shelukhin commented on TEZ-3812:
---

cc [~sseth] [~visakh_nair]

> race condition in ssl shuffle
> -
>
> Key: TEZ-3812
> URL: https://issues.apache.org/jira/browse/TEZ-3812
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> ShuffleUtils does the following:{noformat}
> if (sslFactory == null) {
> synchronized (HttpConnectionParams.class) {
>   //Create sslFactory if it is null or if it was destroyed earlier
>   if (sslFactory == null || 
> sslFactory.getKeystoresFactory().getTrustManagers() == null) {
> sslFactory =
> new 
> SSLFactory(org.apache.hadoop.security.ssl.SSLFactory.Mode.CLIENT, conf);
> try {
>   sslFactory.init();
> {noformat}
> It is possible for a thread to get sslFactory that has been assigned but not 
> initialized. It could result in e.g. the hostnameVerifier being null:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: no HostnameVerifier specified
> at 
> javax.net.ssl.HttpsURLConnection.setHostnameVerifier(HttpsURLConnection.java:265)
> at org.apache.tez.http.SSLFactory.configure(SSLFactory.java:219)
> at org.apache.tez.http.HttpConnection.setupConnection(HttpConnection.java:98)
> at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:137)
> at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:123)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:340)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:260)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:178)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:191)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:54)
>  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (TEZ-3812) race condition in ssl shuffle

2017-08-03 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/TEZ-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated TEZ-3812:
--
Summary: race condition in ssl shuffle  (was: race condition in ssl 
shuffle?)

> race condition in ssl shuffle
> -
>
> Key: TEZ-3812
> URL: https://issues.apache.org/jira/browse/TEZ-3812
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> ShuffleUtils does the following:{noformat}
> if (sslFactory == null) {
> synchronized (HttpConnectionParams.class) {
>   //Create sslFactory if it is null or if it was destroyed earlier
>   if (sslFactory == null || 
> sslFactory.getKeystoresFactory().getTrustManagers() == null) {
> sslFactory =
> new 
> SSLFactory(org.apache.hadoop.security.ssl.SSLFactory.Mode.CLIENT, conf);
> try {
>   sslFactory.init();
> {noformat}
> It is possible for a thread to get sslFactory that has been assigned but not 
> initialized. It could result in e.g. the hostnameVerifier being null:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: no HostnameVerifier specified
> at 
> javax.net.ssl.HttpsURLConnection.setHostnameVerifier(HttpsURLConnection.java:265)
> at org.apache.tez.http.SSLFactory.configure(SSLFactory.java:219)
> at org.apache.tez.http.HttpConnection.setupConnection(HttpConnection.java:98)
> at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:137)
> at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:123)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:340)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:260)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:178)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:191)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:54)
>  
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (TEZ-3812) race condition in ssl shuffle?

2017-08-03 Thread Sergey Shelukhin (JIRA)
Sergey Shelukhin created TEZ-3812:
-

 Summary: race condition in ssl shuffle?
 Key: TEZ-3812
 URL: https://issues.apache.org/jira/browse/TEZ-3812
 Project: Apache Tez
  Issue Type: Bug
Reporter: Sergey Shelukhin


ShuffleUtils does the following:{noformat}
if (sslFactory == null) {
synchronized (HttpConnectionParams.class) {
  //Create sslFactory if it is null or if it was destroyed earlier
  if (sslFactory == null || 
sslFactory.getKeystoresFactory().getTrustManagers() == null) {
sslFactory =
new 
SSLFactory(org.apache.hadoop.security.ssl.SSLFactory.Mode.CLIENT, conf);
try {
  sslFactory.init();
{noformat}

It is possible for a thread to get sslFactory that has been assigned but not 
initialized. It could result in e.g. the hostnameVerifier being null:
{noformat}
Caused by: java.lang.IllegalArgumentException: no HostnameVerifier specified
at 
javax.net.ssl.HttpsURLConnection.setHostnameVerifier(HttpsURLConnection.java:265)
at org.apache.tez.http.SSLFactory.configure(SSLFactory.java:219)
at org.apache.tez.http.HttpConnection.setupConnection(HttpConnection.java:98)
at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:137)
at org.apache.tez.http.HttpConnection.connect(HttpConnection.java:123)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:340)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:260)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:178)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:191)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:54)
 
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (TEZ-3811) Tez UI: Add timezone to all timestamp columns

2017-08-03 Thread Prasanth Jayachandran (JIRA)
Prasanth Jayachandran created TEZ-3811:
--

 Summary: Tez UI: Add timezone to all timestamp columns
 Key: TEZ-3811
 URL: https://issues.apache.org/jira/browse/TEZ-3811
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prasanth Jayachandran


To match app logs and UI easily, it will be good to print TZ information in all 
timestamp fields. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)