[jira] [Commented] (HIVE-14071) HIVE-14014 breaks non-file outputs
[
https://issues.apache.org/jira/browse/HIVE-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343863#comment-15343863
]
Hive QA commented on HIVE-14071:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812304/HIVE-14071.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10247 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-leftsemijoin_mr.q-bucket5.q-and-1-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/216/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/216/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12812304 - PreCommit-HIVE-MASTER-Build
> HIVE-14014 breaks non-file outputs
> --
>
> Key: HIVE-14071
> URL: https://issues.apache.org/jira/browse/HIVE-14071
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14071.patch, HIVE-14071.patch
>
>
> Cannot avoid creating outputs when outputs are e.g. streaming
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14071) HIVE-14014 breaks non-file outputs
[ https://issues.apache.org/jira/browse/HIVE-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343056#comment-15343056 ] Pengcheng Xiong commented on HIVE-14071: I had a discussion with [~sershe]. The main draw back of current solution in this patch is that we call "createBucketFiles" and we call "closeWriters" to close it immediately. The reason behind is to keep the client from endless waiting. We explored and discussed different other solutions. As far as we know, it seems that we could not find anything better. I would suggest that we could put some comments about this drawback and some "TODO" comments for future improvement in the code. +1. ccing [~ashutoshc] > HIVE-14014 breaks non-file outputs > -- > > Key: HIVE-14071 > URL: https://issues.apache.org/jira/browse/HIVE-14071 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14071.patch, HIVE-14071.patch > > > Cannot avoid creating outputs when outputs are e.g. streaming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14071) HIVE-14014 breaks non-file outputs
[ https://issues.apache.org/jira/browse/HIVE-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342976#comment-15342976 ] Sergey Shelukhin commented on HIVE-14071: - This is writer; this is server pushing data to client. If the server never pushes anything or disconnects, the client just sits around forever waiting for data. At least the close() call should be performed, so that the notification is sent to client and/or the socket is closed. > HIVE-14014 breaks non-file outputs > -- > > Key: HIVE-14071 > URL: https://issues.apache.org/jira/browse/HIVE-14071 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14071.patch, HIVE-14071.patch > > > Cannot avoid creating outputs when outputs are e.g. streaming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14071) HIVE-14014 breaks non-file outputs
[ https://issues.apache.org/jira/browse/HIVE-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342782#comment-15342782 ] Pengcheng Xiong commented on HIVE-14071: [~sershe], thanks for your fast patch and response. So basically, when the file (to be streamed) is empty, your patch will still create the file (the streaming pipeline)? Is there any other way we can improve that so that we do not create the pipeline at all? If I understand correctly, the root cause of the problem is that streaming client is writing data to streaming server but server may wait infinitely when client does not have anything to write (file is empty). Is it possible to use "push" model for the client to push data to the server rather than current "pull" model for the server to pull data from client? Thanks. > HIVE-14014 breaks non-file outputs > -- > > Key: HIVE-14071 > URL: https://issues.apache.org/jira/browse/HIVE-14071 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14071.patch, HIVE-14071.patch > > > Cannot avoid creating outputs when outputs are e.g. streaming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
